Synergy and ThreadX: shared resources and Mutex

In our previous articles we saw how to create threads on Synergy/ThreadX and took a look at the RTOS basic structure. Now it is time to talk about a very common mistake when programming in multi-thread systems which is the concurrent access to shared resources which can potentially lead to race conditions. But what is a shared resource? Basically, any peripheral which is used by more than one thread is a shared resource! If two or more threads write to the same I/O port, that port is a shared resource, if two or more threads write on a display, the display is a shared resource and so on. If the operation on the shared resource could not be performed atomically (by a single instruction), then this operation can fail if the resource is shared between two or more threads.

Let’s take as an example an LCD display: suppose two threads (A and B) are writing data on the display, thread A is writing on line 0 and B is writing on line 1. Now let’s suppose that while A is writing the string “Voltage = 5.02” on line 0, it is preempted by the kernel having written only “Vol”. Thread B is resumed and starts writing the string “Current = 1.08” on line 1, but it is also preempted while having written “Current =”. Now the kernel will resume thread A, what is going to happen? Unfortunately, thread A neither  the shared resource (display) can realize there are two concurrent operations ongoing! The result is that when A is resumed, it will continue the write operation on the display (which now is in line 1 with “Current =” on the screen) resulting in something as “Current =age = 5.02” to show in line 1!

This example, despite silly, demonstrates how important it is to carefully deal with shared resources. Some people might argue the chance of such situation happening is pretty low, but it is important to realize the problem exists and can lead to a potentially catastrophic failure! Our example makes use of a display which allows easily and visually identifying the failure caused by concurrent access, but is some cases, failures due to race conditions can be very tricky to catch and stay hidden for days, months or even years! And believe me, according to Murphy’s law, these failures will probably show up on a presentation to your boss, to a customer or in field!

The following code example demonstrates this failure in the real world by using the SK-S7G2 board. We have two threads writing to the onboard graphic LCD. We designed a graphic library with primitive functions for drawing on the display. For demonstration purpose-only we created a special function for drawing characters on the display by using static variables. Note that using static variables in multi-thread applications is an error and should be avoided, but in this case, it helps us to demonstrate how failures from accessing shared resources can occur!

Code for the two threads is shown below, as well as the resulting image on the SK-S7G2 display. If you want to try the code, it is available in the repository as MultiThreadGLCD1.

 

Figure 1 – Thread 0
Figure 2 – Thread 1
Figure 3 – Artifacts due to LCD concurrent access

Notice the presence of artifacts (garbage) just below the lines where the threads are writing data (“O”, “.”, “X” or ” ” depending on the thread). That is due to the threads being preempted while running glcd_printChar_TEST function! Since that function makes use of static variables, their state is not saved on context change (because static variables are not stored on stack), so that when the thread is resumed, glcd_printChar_TEST context is not the same as it was! The same situation would occur if the shared resource was a hardware peripheral shared among multiple threads!

So how can we avoid such failures? The ideal answer is: do not share global resources ever! In our examples, instead of two threads writing concurrently on the display, you could direct all display operations to one thread and rewrite the other one to send messages to the display thread with the desired content to be displayed!

On the other hand, there are cases where you can’t avoid using shared resources and then some kind of mechanism is needed in order to prevent shared resource corruption. The easiest way to protect a shared resource would be to avoid preemption while a shared resource operation is ongoing, that could be done by disabling interrupts while the operation is taking place, but that is an RTOS heresy  and shouldn’t be done ever, since it will compromise other’s thread response time and kernel operation! So how protect such critical code section?

The definitive answer is the use of Mutex, an RTOS object which ensure mutual exclusion, so that only one thread can have a Mutex at a time, excluding the others until the Mutex is released by the thread which first got it. A thread can get (or try to get) a Mutex (as long as it is free) by calling tx_mutex_get() function and once the operation on the shared resource is completed, free the Mutex by calling tx_mutex_put().

A Mutex example

The following code snippet shows how to use Mutex in order to protect a shared resource:

tx_mutex_get(&mutex,wait_time);
// shared resource operation here
tx_mutex_put(&mutex);

Notice all Mutex calls make use of a pointer to the Mutex and, regarding tx_mutex_get() function, it is also necessary to specify a wait time for when the Mutex is held by other thread. Possible values for wait_time are:

  • TX_NO_WAIT (or 0x0) – do not wait for the Mutex to be released, return immediately;
  • TX_WAIT_FOREVER (or 0xFFFFFFFF) – wait indefinitely (the thread is blocked) until the Mutex is released;
  • values between 1 and 0xFFFFFFFE – wait time (RTOS ticks) while waiting for the Mutex to be released;

So how can we use a Mutex to protect our LCD operations? All it is necessary is creating a Mutex object (by using Synergy Configuration) and request the Mutex by calling tx_mutex_get() before using the shared resource and release the Mutex, by calling function tx_mutex_put() after the operation is done.

Figure 4 – Creating a Mutex on Synergy
Figure 5 – Mutex property

The following figure shows thread 0 code using Mutex.

Figure 6 – Thread 0 with Mutex

Don’t forget to change thread 1’s code as well and include Mutex calls before and after any use of a shared resource (the LCD in this case)!

Once both threads are modified, running the application will result on a screen as shown in figure 7. Code for the example is available as MultitThreadGLCD2.

Figure 7 – Thread concurrently using the LCD with Mutex

Now we can see the artifacts (garbage) below the lines being written by the threads are gone! Thanks to the Mutex, only one thread is allowed to use the shared resource at a time, making the other thread trying to use it at the same time to get blocked until the Mutex (and thus the LCD) is released!

Priority inheritance

Using Mutex in an RTOS can sometimes suffer from issues related to thread priority levels. Suppose an application with four threads (A, B, C and D, with A the highest priority thread and D the lowest priority thread) and also that A and D both access a shared resource protected by a Mutex. Now suppose that at a given moment, D got the Mutex and started to do a time-consuming operation with the shared resource, but soon after starting, context is changed, A is resumed and also needs to use the same shared resource. When A tries to get the Mutex it fails because D is holding it. So A is suspended until D releases the Mutex. The system will continue to operate, but as threads B and C have higher priority than D, D will take a long time to release the shared resource and the Mutex, making A to wait a long time until it can access the shared resource. So we a have a high priority thread (A) waiting for a low priority thread (D) do release a resource and that probably is not admissible by the application!

To solve such situations, ThreadX includes a mechanism named priority inheritance which can make a lower priority thread holding a Mutex to assume temporarily the same priority level of a higher priority thread which is trying to get the Mutex!

In the previous example, when A tries to get the Mutex (created with priority inheritance enabled) held by D, ThreadX will temporarily rise D’s priority level to A’s level, so that D will be able to release it without being disturbed by C and D! As soon as D releases the Mutex (by calling tx_mutex_put() function) its priority level will return to its previous original level!

To enable priority inheritance on Synergy, all you have to do is to enable the option on the Mutex properties as shown on figure 5.

In our next post we are going to talk about thread communication, see you there!

References

Leave a Reply