 Hi, everyone. Thanks for tuning in for my presentation. I'm Grazian Krishan, and today I'll be talking about using conditional variables in real-time applications, some of the problems we've encountered using them in real-time applications, and some of the solutions we've implemented in Libar TPI. To give you a bit of context about me, I work for NI. We recently got rebranded from National Instruments to NI, and I make hardware and software for test measurement and automation markets. I've been with the Real-Time OS Group for the past decade or so, and we're using pre-empt RT-based Linux kernels on our Real-Time hardware. We're proud to support the Real-Time Linux collaboration project, and we use that on ARM and X86 architectures. We use Open Embedded and Yachter to build our distribution. For the past couple of years, I've been the maintainer for the Linux kernel shipping on our RT hardware, and in that capacity I often debug nasty real-time issues, and too often they're related to locking primitives and conditional variables, which is the subject of today's presentation. In terms of agenda for today, I'll very quickly cover just some real-time concepts that are relevant for today's presentation, and then some details about conditional variables and the associated concept of monitors, and then I'll go more depth into the problems we've encountered. We're using conditional variables in the libp thread implementation, our re-implementation on Libar TPI, and then I'll close out with some future ideas and questions. Please submit your questions in the chat window. I'm doing this presentation live. I probably won't be able to answer you necessarily during the presentation, but I'll make sure we have enough time at the end to answer questions, and if we don't, then we can move over to the Slack channel and continue the discussion. Very broadly in very general terms, a real-time system is a system that needs to interact with the real physical world, and if you have a system that needs to interact with the real physical world, then you must have a way to synchronize with it. One of the most important attributes of a real-time application is what's called having a deterministic response to a stimulus. If an event happens and then the time it takes between the event and the action taken by the system, we measure that as latency, and we want that latency to be predictable, but even more importantly, we want that latency to be bounded, to have an upper bound, because if it's bounded and we can know in advance what the max latency is going to be, then we can design our real-time application in such a way to meet the demands of synchronizing with the real physical world. Traditional real-time applications tend to be pretty simple things. You usually have a desired value, you're trying to control the system output for, and you just have a simple control loop, sometimes open loop control, sometimes closed loop control if you have a sensor in the loop. But the point I was trying to make on this slide is traditional applications used to be simple things, and that's not the case anymore. Real-time applications in today's world have become these complex things that involve basically autonomous driving and landing rockets and controlling smart grids at the country level. So you have complex actuators and you have complex sensors like cameras and GPS and radar, and you have to do sensor processing and the complex filters and image recognition and so on. Oftentimes within your real-time application you have to construct a model of the world, and you have planners and executors, and oftentimes you have AI and ML in the loop or hardware in the loop. So the point I'm trying to make on this slide is well, number one, preempt RT and Linux. It's ideal for solving this type of real-time applications. It enables this type of applications because with preempt RT and Linux you can take advantage of the rich ecosystem you find in Linux of drivers and libraries and databases and ML libraries and so on. But for the purpose of this presentation the other point I wanted to make is there's a lot of data movement that needs to happen between all these components in these systems. So basically what the pattern you see implemented over and over again is solving this bounded multiple producer-consumer problem but with RT constraints. So you have consumers waiting for data to become available from sensors and you want the consumer that's running at RT5 for 50 your real-time thread to have access to that data first when that happens. And one of the fundamental questions when you solve the bounded consumer-producer producer-consumer problem is how can a thread wait for a condition to be true? How can one of those consumers waiting for data know when new data is available in the share queue? There's multiple ways to solve this. One way you could do it is by having the thread spin until the condition becomes true. That is very inefficient and it actually can be actively detrimental if you have an RT thread spin because it can live-lock a CPU. A much better solution would be to have an explicit queue where threads can put themselves on and wait for some state of the execution to become to happen. So that's called waiting on a condition and then some other trial when it changes the state it can wait one or more of these waiting threads by signaling the condition. So one of the most common ways to solve this problem is basically by using conditional variables. So very briefly a conditional variable it's a synchronization primitive that provides a queue for threads to wait for a resource. And the operations you can do on a conditional variable are wait which adds the calling thread to a queue and puts it to sleep potentially with the timeout. And then the converse operations are signal which removes the thread from the queue and wakes it up or broadcasts which wakes up all the threads that are waiting on that way queue. What you'll encounter in practice more often is something called a monitor which is a synchronization construct that allows to trust to have both mutual exclusion and the ability to wait for a certain condition. So a monitor is composed of a lock object which provides the mutual exclusion that's a mutex and one or more conditional variables which provide a queues for a thread to wait on for a condition to become true after atomically releasing the mutex that part is important. Higher level languages like C sharp and D support monitors natively. In C and C++ they must be constructed from mutex and a conditional variable. So a bit more about monitors. There's a design rule associated with monitors. You can have multiple conditional variables associated with the same mutex but not vice versa. If you think about it the mutex is there to protect the integrity of the underlying shared resource, the queue, the buffer, whatever but you can have multiple wait queues. You can have consumers wait on the empty condition if the queue is empty and when a producer adds more data in it will signal this empty condition and a consumer knows hey more data is available I can go read it. Conversely you can have a full condition for producers to know when more space is available in the shared resource and they can do their job. In theory there's two styles of monitors. One is called Hore style monitors. This is what you'll find in most theory. In this style of monitor the signaler passes the lock to the waiter when it does the signaling and the waiter can run immediately. And because of this handoff the waiter, the condition is guaranteed to hold while the waiter runs and the waiter can do his job and give the lock back to the signaler when it exits the critical section or wants to wait again. What you'll find in practice a lot more often is something that's called a Mesa style monitor. Most real time users implement this because it's more practical to implement and in this style of monitor the signaler keeps the lock while it's doing the signaling the waiter is simply put on the ready queue for the associated lock and might have to wait for the lock again before it can run and because of that it must recheck the underlying condition. If we express Mesa style monitors in pseudocode it will look something like this for the signaler making a resource available aside, the signaler side you take the lock you make the resource available let's say if you're a producer thread you add more data into the shared queue and then you signal the associated condition and then you unlock the associated mutex pretty straightforward on the signal impact signal impact sorry on the waiting path on the waiting for a resource path things are a bit more complicated what you need to do is you lock the associated mutex that protects underlying resource you check if the resource is available or not and if it's not you then call wait and this wait call will automatically release the mutex and wait on that conditional variable and after you wake up you have to recheck that condition because previous wakeups are allowed with Mesa style monitors and assuming everything is okay and resource is available you can do your work and unlock the associated mutex there's an additional constraint if you use monitors and conditional variables in real time design and that is you want these various threads waiting on these various conditions and these various wait queues you want them to wake up in their priority order so if new data gets added by a producer into the queue and the consumers were waiting on that empty condition we want that artifact for 50 consumer to wake up first and have first dibs for that data before the other schedule other threads for example that are waiting on the same condition the other important concept in real time design when we're talking about threads and locking is the concept of priority inversion so this is a situation that can occur if you have threads of different priorities that have a shared lock so what I have on this graph I've represented on the Y axis priority and I have three threads in there T1, T2, T3 the filled up boxes represent portions where that thread runs and the empty boxes represent portions where that thread is scheduleable and is run for whatever reason so in this example our lowest priority thread T1 starts running at some point nothing else is running on the system so even though it's the lowest priority thread on the system it can get scheduled in and start running and let's say this thread at some point at time T1 acquires lock L starts doing his job accessing the shared resource but in the meantime T3 becomes runnable and then because it's a higher priority it pre-ends our T1 thread so it starts running, does some work and then it tries to access the same shared resource so it's trying to access the underlying and lock the underlying lock L but because lock L is still being held by a thread T1 it cannot do that so it blocks and the scheduler has to put this thread to sleep waiting for that lock to be released now in the meantime an intermediate priority thread T2 can come in and because it is a higher priority than our lowest priority thread T1 it will get scheduled T3 cannot run because it is blocked on waiting on lock L but T2 can get scheduled in and it can run for an unbounded amount of time it's not connected to our T1 T3 threads it doesn't need shared lock it can run forever and it can pre-end basically our high priority thread from doing its job because it's waiting on the lock L and it pre-ends the lower priority thread to finish its work and it cannot release lock L so only when the intermediate thread T2 releases releases the CPU the scheduler can our lower priority thread T1 finish its work, release lock L and then T3 can our high priority thread can finally run so you can see in this diagram that T3 had an unbounded amount of time where it was preempted to run and it created an unbounded latency which is a bad thing in real time design so the way this is solved in Linux with preempt RT and it's basically this algorithm that's called priority inheritance so in this scenario as before T1 starts running it requires the associated lock L it gets preempted by T3 but then at a point where T3 blocks on the lock L the priority of our low priority thread that's called in that lock gets boosted to the priority of our high priority thread so the priority of T1 now becomes the priority of T3 T1 runs at T3's priority and because of that it cannot be preempted by this intermediate priority thread T2 so it can finish its work quickly, release the lock and at a point it releases the lock it gets deep boosted and our high priority thread T3 can acquire the lock and start running the algorithm behind this is fascinating to go in the kernel code and kernel documentation and read the bodies the important point I wanted to make for the purpose of this presentation though is that having locks with our priority inheritance is really bad for real time design because it can create these unbounded latency situations so we get to our first problem that we encounter with conditional variables when you use P-Tread it was discovered back in around 2009, Dan Hart and a bunch of other real time folks apologize I'm not naming here I wasn't involved at the time so I don't remember everybody so they discovered that P-Tread conditional variables has the potential of priority inversions, they are not priority inheritance aware the problem was with an internal lock that's used to protect internal structure associated with the conditional variable and that internal lock even if you use the user mutex with priority inheritance that internal lock did not have priority inheritance and it could create these priority inversion situations and unbounded latencies so Darren and the other real time folks involved at the time work with the kernel folks and solve this on the kernel side and on the side they put some patches together they are attached to this bug and they started working with the Julia Seafolk trying to upstream this fix so on the kernel side two new feedback operations got added, I'll talk more in depth about them later in the presentation but just briefly feedback compare EQPI can be used for signaling and a feedback wave EQPI can be used for waiting and they both implement this priority inheritance algorithm I talked about so they started working through trying to upstream those patches at some point I came in and I tried to help too I migrated the patches to the latest Julia Seafolk at the time and recently needed them they was backing forward about getting the right copyright assignments in place with the free software foundation and we worked with lawyers and we solved all that but in the meantime there was another bug that was filed against LibPetrate and conditional variables and it turns out that at the time it was possible for a PetraCon wave call that happened after a PetraCon signal call to consume that signal and I linked the associated bug here and the slides are going to be available online so you can go check it out for yourself there was a long discussion that spent I guess over five years if I look at the dates on this bug it's more like six or seven years it was a pre-heated discussion I find the underlying situation correctly the issue was caused by some sequence counters that were used in Julia Seafolk to keep track of how many signals were sent and how many threads were woken up LibPetrate and Julia Seafolk were trying really hard to avoid spurious wakeups even though they are allowed per mesamonitor design and per POSIX for that matter and because of this counter spurious wakeups it was possible for a PetraCon wave to consume a signal so the discussion on this bug got really heated and then it spilled over into POSIX defect that got filed with the Austin Group and there was a lot of discussion there happening there too for a while it looked like there was some consensus on clarifying POSIX to describe when it's a thread considered to be blocked with respect to the PetraCon signal or PetraCon broadcast and I think the consensus at the time was like PetraCon signal or PetraCon broadcast will determine at some point during their execution which threads are blocked on a conditional variable and they will make sure to wake up one of those blocked threads and not a thread that comes in and waits on the conditional variable after anyway the POSIX issue is actually still open that last update you see there is from 2016 and it's me actually asking a question if there's a consensus on this but regardless Julie C went ahead and fixed the bug that I mentioned earlier by implementing this algorithm that I'm trying to explain here so what this algorithm in the current implementation of Julie C and Petra tries to do is make sure that that happens before relationship it seems really enforced between waiters and signalers so in this algorithm all waiters start in this non-eligible group called G2 so you have two waiters there in those diagrams in the upper left corner W1 and W2 they come in they get put into non-eligible group non-eligible group so only when all the previous waiters there were in the active eligible waiter group that's group G1 have been signal the next signal that comes in will take all the waiters there were in the non-eligible group in the group G2 and move them to the group G1 and the signals will start signaling waiters in this eligible group in the group G1 so if a new waiter comes after the signal S1 and now we are looking at the diagram in the upper right corner on this slide if a new waiter W3 comes in it will get in this non-eligible group G2 and a new signal that comes in after that S2 will continue to signal the waiters that are in the G1 group and only after everybody was waiting in the G1 group is signal then our W3 waiter that got in queue in our G2 group gets moved into the active group G1 and it can then get signal by signal S3 so this scheme guarantees fairness in terms of waiters that started waiting earlier they'll consume signals before other waiters that come in later and definitely enforces the happens before relationship but it has a huge problem when it's used for RT design for designing RT applications and that is if you have a new RT priority waiter that comes in and you already have a bunch of waiters waiting to be signaling in the eligible group in the group G1 our WRT waiter our high priority waiter will get in queue in this second group in the group non-eligible group in the group G2 and you can already see the problem here this is a priority inversion basically because our high priority RT waiter even though came in before signal 2 and all the way to signal N it won't get signal until everybody else got signal that came before in the real-time application what we would like to happen is if a new RT waiter comes in as to signal there we would like that to wake up a new RT even though there were other waiters that waited longer we care about the priority of the threads in a real-time design so this creates a priority inversion by design and this is a big problem if you're using JLPC and the EP thread pass version 225 and this puts our T folks in a bind because basically if they want to use conditional variables and have proper behavior now you're in big trouble if you want to upgrade your JLPC pass version 225 there's another issue with the current implementation of conditional variables in EP thread and that is for signaling and for broadcast operations on the slow path they use a few text wake operation and what that creates is basically a thunder in heart effect because you're on a like let's say you do a broadcast they will all wake up at the same time and raise to acquire the associated immune text and you have this thunder in heart effect which is bad again for RT use cases so nevertheless Torval Regal implemented this new scheme in JLPC and he presented on it at the RT summit in Berlin I believe in 2016 we raised our objections but even with that the condition got merging and the last comment on the bug that we care about, the one about conditional variables not having priority inheritance the last comment there says there's no known solution on how to achieve priority inheritance with the current design of JLPC given the current constraints JLPC is also constrained in terms of changing the associated structure that's associated with conditional variables because the structure size is part of the JLPC API so there's no known solution at this point for how to achieve PI support given the current constraints we have available few types of supposing requirements and so on so to make matters so much worse if you look at the other locking primitives available in LEP Dread you'll notice that with the exception of PITRA and MUTEX where you can enable priority inheritance via a MUTEX attribute and in that case it uses PI primitives like FUTEX lock PI and NILock PI all the other primitives locking primitives in LEP Dread and JLPC use FUTEX operations that are not they don't have priority inheritance they're not RT friendly so you cannot use conditional variables and you don't have the option to use semaphores or something else to implement your preconceiver patterns or whatever other reason you might want to use conditional variable for so I presented on this problem at the RT summit in 2017 this was in Prague I linked a video there if you're curious to see my presentation back then and basically lay this out basically hey if you're an RT and you have properly behaved conditional variables you cannot upgrade JLPC past version 224 without running into this unbounded latency issues with conditional variables so Sebastian put together a meeting with Darren, Peter, Julia me and hopefully I'm not forgetting other people some of the interested RT folks that were the RT summit at the time and we talked about it and basically said okay we've been at this now for almost 10 years we we tried to upstream our solution to JLPC because different requirements and POSIX is moving in a different way that's contradictory with the real-time requirements we have maybe it's time to take a step back and try a standalone implementation that narrowly solves this problem for our T application and lets people upgrade their JLPC and use conditional variables in parallel with the outdated JLPC you don't want to be stuck on an old JLPC obviously for security reasons and so on so Darren put together the initial spec and the Github project that I linked there and Sebastian and Julia flashed it out at summit on the summit in 2018 and then Darren and Julia presented a status update that I linked there at Linux Plumbers in 2018 and in the intervening time I've been working with Darren to add more test cases I've imported the JLPC test cases that are used for conditional variables and we talked about tweaking the API at last ELC in 2019 in San Diego because we had a current case that wasn't completely solved by the API we picked related to process-shared conditional variables I added more tests and some fixes and so on and here we are today I'm presenting our current status and the progress we've made you might notice this is conference driven development so our design goes behind Libar TPI where first and foremost we want the priority inheritance by default that guarantees us basically bonded latencies and good real-time behavior to be welcomed in priority order a deviation from POSIX we mandated that the signaler must hold the lock POSIX only mandated it's optional if you use POSIX conditional variables if the signaler side actually holds the lock while it's doing the signaling but if you aren't properly ordered and behave real-time applications you want to order the signalers with respect to waiters so it's important to hold the associated lock we wanted to avoid tundering the fact so use PI prefix PI primitives we decided to default to clock monotonic for time weights if you're familiar with GLC and PTRAT conditional variables they use clock real-time for time weights and that can actually be problematic in real-time applications despite the name clock real-time because in reality that's wall clock time and it can go backwards it can jump and that's bad for time-outs so you're much safer as a default to use clock monotonic we decided to use opaque data types to allow for future expansion we didn't want to be painted in the same corner GLC got painted in where they couldn't make improvements because the structure size is part of the API and we decided to keep the API as close as possible to POSIX to facilitate porting but we intentionally didn't overload the existing PTRAT com API we didn't do some LD preload tricks or anything like that we wanted the porting to live our TPI to be explicit and you can use live PTRAT comvars and live our TPI comvars in parallel not combined together but in parallel they can be used in the same application we decided to allow for that use case in terms of license, build and test live our TPI is licensed under LGPL we decided to do that because it makes it possible to link and reuse GLC code and we've done that by importing the GLC tests and some of the code we had already contributed to GLC and it's also broadly usable in the industry it's what everybody uses in terms of live PTRAT Darren set up an auto tools build system for this project and thread the CI for running tests on github in terms of porting the code from POSIX code to PI mutex API let me back up just a bit I have two primitives in there PI mutex and PI conditional variable if you're porting POSIX code to the PI mutex API unfortunately there was some highlighting here that doesn't show up on this intrado platform I'm presenting on but basically if you're familiar with the PTRAT API the only difference you might notice on the left side is PI mutex in it takes a flags attribute when you're initializing a mutex attribute object we found that it's really cumbersome to deal with creating that mutex attribute object and then setting flags and usually to initialize the mutex and then destroy it it seemed unnecessary because from the get go we wanted this implementation to be very narrow and narrowly scoped so we don't expect a lot of flags that will get applied to a mutex so we can accommodate just a bit flag attribute that we passed to be in it the destroy log, try lock and unlock operations have the same basically API as the PTRAT PTRAT mutex lock, PTRAT mutex try lock, PTRAT mutex unlock one difference you might notice on the right hand side we have decided to provide some macros and some dynamic allocation of API calls for allocating a PI mutex and that has to do with the fact that we wanted to underline PI mutex the data type to be an opaque data type so we provided allocators for it in terms of implementation it's actually pretty straightforward and I have a link to the source code the github source code at the end you can go check it out but it's not much more than what you see on this slide we're going to talk about the PTRAT mutex lock basically we're using the futex lock PI futex operation and we did futex up what you need to do if the mutex is uncontended in user space you can try to do an atomic compare and swap between the futex word and our PID and if that succeeds it's good we own the mutex if not the kernel has to put us to sleep and do its priority inheritance algorithm and so on so basically we try the compare and swap with PID if we don't see we go into the kernel and kernel handles everything else for us the unlock implementation it's basically the converse of the lock implementation first we try an atomic swap between our PID and zero there's some error checks before that to make sure we actually own the lock but we try this atomic compare and swap between our PID and zero to unlock the mutex and if that fails we just go into the kernel and like the kernel handle everything via the unlock PI futex operation which does priority inheritance and so on so yeah in terms of mutex implementation in terms of the actual convar which is the topic of this presentation again the api looks very similar to the posix lippi thread api again one of the differences is api comd init takes a flags attribute as opposed to a comd comd at ATTR comvar attribute object for the same reasons we only have a couple of flags and this makes things easier destroy comd weight comd time weight operations have the same parameters that you will expect in a ptrell implementation one difference that you might see here compared to the ptrell implementation of convars is we're passing in the associated mutex that's associated with a monitor associated with a comvar to the signal and broadcast calls and why does need it will become obvious in a couple of slides when I talk about implementation and again for the same as with mutex we provide some static and dynamic allocators to allocate api comd object in terms of implementation let me start first with a few text calls that are used to implement this conditional variable because it will make it easier to explain the implementation so the mutex is called use for signaling is this mutex operation called mutex compare qpi and what this operation does it basically looks at all the waiters that are in queue on the first parameter that's past there that's the you add there that is our non-pi futex and the parameters are queued on that's our wait queue for the conditional variable and then it wakes up one of them that their parameter, their val it's always one for signal or broadcast we want to wake up at least one and then the four parameter val we pass in zero for signal we just want to wake up one and we pass in max for the broadcast case the number of threads we want to re-queue we want to re-queue all the threads in a broadcast situation on the fifth parameter there which is you add there two which is our user mutex so what this system call does is if there's waiters waiting on you other one wakes up one of them and it re-enqueues the rest of them on you add there two and the wake up is done in such a way as to respect priority inheritance to wake up the highest priority thread that was waiting there so if we go back to the implementation of the PI count signal it's not very complex the underlying structure has two fields in there the count field, the count count that is our fifthx word that we're rating on and I'll come back to wake ID that's used to detect races between signalers and waiters and that's the current sequence count that's stored in our count field and then all we do is call this fitters compare QPI that does all the work for us in the kernel and if everything went well the system call will return zero or number greater than zero that represents the number of threads that got re-enqueued if there was an error we retry on it again and otherwise we just return the error pretty simple in terms of the the the waiting side of the equation uses this futex way to re-QPI futex operation which is another operation that pretty much does everything we need so what this operation does is it will enqueue the current thread on you other one and that's our wake queue the third parameter the value field is just used to detect races because if you remember we messed up the monitors we have to release the associated mutex before we start waiting so that value lets us basically detect if there was a race and then the four parameters the timeout if we have a wait with the timeout and the fifth parameter there you add is our PI futex where we want to get enqueued on wakeup the implementation for our wait basically what we do we again increment the count field that's used as our wake queue and as our sequence counter we record what the wake ID was before we release the associated mutex and then we just do the sys call that puts us to sleep on that futex that's associated with the condition variable and our return if everything went well we owned the mutex and we can just return if there was an error we have to reacquire the associated mutex because the positive semantics for waiting says that the wake call basically atomically resets the mutex weights and then reacquires the mutex even in error cases so we reacquire the mutex we retry the again case unless we've raced with a singular and that's important for real-time design because if a waiter raced with a singular we want that waiter to stay awake and check the underlying condition associated with a Kanbar because that waiter might have been a real-time tread and we don't want to miss a wakeup and the way we detect that is by checking the locally recorded wake ID to the wake ID that's set in the Kanbar that's incremented every time the Kanbar is used basically to signal so that's pretty much it if you look at in my github repo that's what you'll see in terms of the implementation in terms of current status the julepc test supported it's about I think 20 tests or something and the API change have emerged in Darren Hart's github repo that's our upstream for LibRTPA I still owe Darren some poor requests from my branch that I linked there I have some low locking fixes there's a PI mutex fix I need to upstream related to the processor use case I've simplified the sequence counters the initial implementation and Darren's repo has an internal private mutex that protects this internal state of the Kanbar because we mandated in the API that the mutex is held for waiting and signaling we can actually just use the user mutex to protect our state so we don't need that internal mutex so I simplify that I've had a clock real time support just to make more tests pass I've had a cancellation support to make all the julepc tests pass even though that wasn't part of the initial design but hey so there's some more general clean outs although I still owe Darren about 25 commits that you can check in my branch and that's linked in the slides here so the current status LibRTPA it's a very small library it only implements these priority test Kanbar it can be used in parallel with LibP thread it's only 34 kilobytes all test pass has proof here I posted a screenshot of the test with a total 25 tests or so I couldn't ask the risk on that because there's a testing here that's a bit flaky and it has to do with cancellations so once everything is merged and sorted out if you notice here the LibRTPA version is 001 so we should probably do an official release it is currently our version of this is using production NI so I encourage you basically and this is my last slide I encourage you to test it to start using it to contribute I've linked here the two repos and in terms of future I would like to open it up for discussion I'm interested if you're interested in putting together a user space type toolbox for real time design what I found in my experience is a lot of people reinvent the wheels and for example one of the most common things I've seen reinvented is having a shared memory queue that it can use between trends of different priorities so maybe it's maybe it's worth doing another small library that implements that in an RT set way other things that might be useful for RT so really at this point I'm going to open it up for questions and I'm looking forward to your ideas and questions so let's see let's start going through here the first question is does other LibC implementation have the same issues as LibC like muscle no I believe muscle actually has a better implementation for con bars the thing with using muscle is especially in like complex applications that I have already existing pre-existing software that's linked against LibC you're kind of stuck with glibc which is kind of the de facto zero on time but if you can afford to move to muscle or some other implementation then yeah I don't think it has the same kind of priority in version issue as glibc but to be honest I haven't personally used it so I cannot guarantee that it's just here say from other LibC muscle contributors the next one is there a performance difference between this implementation and the LibP thread implementation I haven't benchmarked it it's kind of hard to benchmark condition variables because by their nature they're just waiting on a condition to happen I guess you could benchmark how well the or how quick the wake up is I suspect because the LibR TPI implementation is much simpler and pretty much it just calls to the kernel and doesn't do much else that it should be quicker but I have not tried it I have not benchmarked it yet let's see another question is there a timeout variant of the trilog I'm assuming we're talking here about the PI mutex implementation no not well there's no current implementation LibR TPI for trilog with the timeout but that will be really easy to add and for requests are welcome please try it out and contribute it if you think it's something that's really useful we can think about adding it let's see what else do groups still exist in LibR TPI are the threads and then get other ones in two groups yes you can use you yeah you can use get other or Skype or whatever other scheduler you want with your threads and it should work with LibR TPI you do get advantages obviously when you are interested in RT5 for threads and getting the proper wake-up behavior but not all the threads need to be running at a schedule or anything like that there are hearts looking forward to the PR so I need to make sure I do that I I I'll start sending you more PRs your way Darren thanks for your help it's been very helpful having your review the changes I've made let's see will the first person ask will the first release be already be ready enough to use in real-time products and I have to hope that the answer is yes because like I mentioned on my previous slide this is actually currently shipping on and I products for about a year or so now as far as I know there hasn't been any big fallout so it's been testing production I guess so go for it then Roberta will you say it's a good step to start contributing to RT we know prior Linux kernel development experience Roberta while I suggest is basically find an issue that bothers you or something you're interested in and start picking on it start looking through the source code and start small and start basically get your feet wet and just find an area that you're interested in in the kernel and the RT world and just try it out and start contributing then Randy mentioned that Libertipia is using production are there any open source apps that you can point to no I don't think he has wide adoption yet in open source and has this presentation I'm trying to raise awareness about it the people that go to RT Summit are deeply embedded in the real time community know about it but I don't expect there's wide audience for it yet so I don't know of any open source product that uses it yet but I would love to if there's more users and we can make this really solid and widespread and including in these things like that and I think we're basically out of time I encourage you guys to head over to the Slack channel for the embedded track I'll be watching there and try to answer some more of your questions thank you very much for your time and it's been a pleasure presenting to you guys