 All right everyone ready how many of you came from Stevens firehose. I mean talk All right, those of us that has seen Steven present for years know how overwhelming it could be at times So my name is Julia Cartwright. I work at National Instruments. I'm also one of the stable real-time Maintainers as Steven had mentioned the last presentation I've been involved in Linux for about ten years involved in the RT project for About six years I think at this point I Maintained one of the stable release actually two of the stable releases as as Steven mentioned But I've also followed the Linux RT users list. So there's a mailing list feature kernel at org This is a user list where a lot of people talk about the issues that they've run into lots of bug reports happen there And from this information gathered over years It's pretty clear to see that there are different classes of bugs That our users the preempt RT users are running into and reporting That are issues in actually issues in device drivers So the purpose of this presentation is to present these Issues as they exist and I'll talk about some of the features Steven mentioned like local locks and so on as we go along So how many of you are actually running preempt RT in production right now? I would have expected more honestly. So how many of you are like evaluating it as a potential future RTOS platform About the same. Okay. That's interesting. All right. Well, maybe I hope you will find this interesting How many of you work on upstream device drivers within the community? Great. You are kind of my target audience. So if anything I say is unclear you have any questions Please ask So I'm gonna talk about a class of systems that we all run into if you if you're divining Designing an embedded system that has Yeah, oh No, they don't see my slides. I don't know how to Have we not switched? Aha. I'm that one Nothing over here though Okay All right, great. So now we have them. Yeah. Thank you So I'm gonna describe a class of what I'm calling mixed criticality systems So most complex embedded systems might have might run two different types of tasks You're gonna run tasks that have some real-time requirements and latency bounded latency requirements Then you also might run some non-time critical stuff You know doing basic monitoring or disk or maybe not even any IO or you have some throughput tasks, whatever it is Those two need to run together. Maybe need to communicate with one another. It's mixed criticality in the sense that there's two different Degrees of time-sensitiveness So one way you might actually design this is actually split it up into two different hardware devices You might have Like an arm series our core on a board and you deploy all your super time critical stuff there Or maybe you're using like an FPGA or CPLD You're pushing all your super time-sensitive stuff onto a separate hardware device and then on unlike an arm a series Chip, maybe you're you're running Linux a full Linux stack So this is this is one of the models. However, it has some downsides It's expensive per unit cost now you produce a board. You have to have You know two different sets of hardware accomplishing two different things If there is interaction between the mixed criticalities like crossing the the domains Now you have to define some communication channel between the two hardware Pieces that's up to you to define But you do get really good isolation. So the stuff that's happening on the non real-time Linux side isn't influencing your time critical stuff So some might choose this different set of trade-offs if you look at something like Xenomai I don't know if you are all in Jan's talk today. He was talking about Xenomai and it's it's it follows this Hypervisor, I think they call it the co-kernel approach It's effectively like you can you run a separate operating system within the realm of a I'm going to call it a hypervisor here. It doesn't necessarily have to be a hypervisor There's amp solutions where you can partition like okay these these cores are going to run free our toss and these are going to run Linux Again, there's there's trade-offs to be made about usability And communication between these mixed criticality systems Pre-empt RT sort of has chosen a different path here Systems that run preempt RT are sharing the kernel. They're sharing a scheduler. They're sharing a device Stack all the subsystems are being shared So you you don't get as much isolation But you get a lot of usability benefits Applications can just use standard Linux IPC mechanisms to communicate It's effectively like thinking of that The traditional Linux process isolation model is good enough to isolate real-time tasks from not in real-time tasks and But like I said the driver stacks are shared between them and therefore the drivers can misbehave and so I'll talk about ways that they can misbehave But first I want to talk about latency. This is what I'm going to call the delta. This is what we all care about If you care about real-time guarantees you have A time at which an external event occurs. Maybe it's an interrupt firing a timer or some IO device saying hey I need to be serviced the time it takes from that event to occur Until a the relevant real-time task actually executes This is my bound of defining Naming Delta Most of you should already know this if you care about real-time systems Real-time systems will try to characterize and bound this in some meaningful way So this is how this is one of the example techniques techniques that the RT community especially the preempt RT community uses to characterize this delta So this is a super simplified version of a test. We call cyclic test So sick to cyclic test is pretty simple All it does is take a time stamp and it sleeps for a fixed duration say like 10 milliseconds And then takes a time stamp when that thread is woken up And if you take the difference between the those two time stamps I've collected That's the amount of time that the thread actually slept And then I'm expecting to sleep for the duration so I can subtract that out and that difference is that delta And so if you plot that in a histogram it can look something like this So in the purple is actually just a I ran this on my laptop. This is like the regular config config preempt Setting in a mainline kernel. This is what you would see the delete and see samples Look like you can kind of see the distribution has a really long tail I think I only ran this for like a minute and I saw that 342 microsecond spike. It's not a real-time operating system. So, you know, that's to be expected And the green is on the same hardware system This is the preempt RT patch you can see that it has and that same duration I don't remember how long I ran it You can see that it has a max bound of 16 microseconds much more well behaved for real-time use cases So if you think about this delta and what is Contributing to this delta I break it down into two different phases. So there's a IRQ dispatch latency This is the amount of time it takes between the hardware event actually firing to the interrupt for the relevant thread to actually be woken up to for the the interrupt dispatch to have occurred and the threat the scheduler the thread scheduler being told This thread needs to run that latency. I'm calling this lowercase delta IRQ dispatch And then the second one is scheduling latency. This is the amount of time It takes now that the scheduler has been made aware that this high-priority task needs to be run the time It takes from that moment to the CPU being given That task to actually execute it being scheduled in on that CPU So these are the two phases that I'm defining for the purpose of this Conversation and I'm going to dive into each one of them individually. So I'm going to start looking at IRQ dispatch latency specifically and sort of talking about all the causes of this Latency and issues you might might see So this is mainline model. So what one of the main contributors to this is actually long-running interrupts and mainline so and mainline because Interrupt handlers are executed in hard IRQ context They are implicitly executed with interrupts disabled And so here's this diagram the scheduling diagram like you might have seen before. This is all mainline again X-axis of time we have some thread executing in user mode at some point There's some interrupt that fires say for example, it's like network and you don't even care about network in your real-time application But it happened interrupts happen And then at some later time well at the time that that event occurs The CPU is going to vector off into hard interrupt context and start executing the handler associated with that network device or whatever device it is And say during the duration of that interrupt handler a High-priority event that I care about I'm here and calling this external event that I care about more say that it fires Well, it's not actually able to be scheduled on the CPU until that low-priority interrupt Is done executing so if you if you in any this this delta here between this external event firing and that red task Executing that's a direct contributor to The that IRQ dispatch latency as I mentioned And it should be obvious without the real-time patch if you want to try to define bounds on this Well, it would be a mess Because the bounds would be the bound of the longest running Interrupt handler in the system the longest running interrupt handler is going to be informing what your worst case IRQ Dispatch latency will be hopefully that's made clear And then at the point your task or you the interrupt handler execute and wake up your real-time task and everything's good This is mainline. So on On RT when you when you apply the RT patch and actually enable the RT patch And and Steven mentioned this as well. What we do is actually force IRQ threads So what this means is that there's actually very little code that we actually execute in hard interrupt context very little They're just little shims that execute because they have to and all those little shims do are wake up these threads that are that are Going to be executing your handler If that makes sense. So you might have some low priority task running You have some not unimportant IRQ that fires But only a very small portion of time has actually spent just waking up the associated handler thread And the duration of that shim can be well understood the bounds of it can be well understood for real-time purposes And it provides other guarantees Such that because it's now running in executing in a thread those threads can now be preempted So if I then have my higher priority RT critical interrupt firing That IRQ thread can be scheduled immediately So that actually reduces that IRQ dispatch latency. Does that make sense or provides a more Easily characterized bound So that's how that works now for you as a device driver developer I have some good news. This is just gonna happen Like we are going to be When you when you enable RT kernels gonna forced IRQ threads So Steven alluded to this thread IRQs actually exists in mainline right now You can boot a kernel and pass the thread IRQs parameter and it will thread all your interrupts So this a significant portion of this infrastructure is in place the things that aren't in place for RT is a forced enablement So that's good. It's good except when it's not So for and there are cases where it's not okay It's not okay when Your drivers are invoked as part of interrupt dispatch So if you actually are invoked in the process of delivering an interrupt, then you can't be threaded So the things that fall into this category are IRQ chip implementations Those are important to not be threaded So if you're a device driver author working on our Q chips, I keep that in mind And so yeah, that's all I really had there It's also not great in the case where you Can be invoked by the scheduler Directly, so there are a class of device drivers in the kernel tree right now that can be invoked by the scheduler itself Things like CPU freak These things cannot be threaded. So if these drivers are requiring interrupts Then you need to be registering these interrupt handlers with this extra flag So if you use request IRQ request threaded IRQ, there's a flags parameter as IRQ F No thread is what you can pass to actually cause the interrupt core to not actually thread your interrupts So keep that in mind. The fortunate thing is the vast majority of you I would hope are not working on CPU freak drivers not working on IRQ chip implementations if you are then I'm willing to help you out a few problems. So please send me an email or send an email to the Linux RT users list So that is talking about Implicit interrupts being disabled during interrupt Handler execution There is a different another class of interrupts being disabled that is explicitly Disabling interrupts using local IRQ disable or local IRQ save Device drivers should not be using these things But they are in fact, I think I did a grep. There's about 80 users of these APIs and the drivers portion of the kernel tree Which is I think manageable. It's something we can actually go through and audit each of these and figure out what they're doing But to give an example of why this is problematic I would hope you know at this point, but it's problematic in the case of the external event Can actually once it's fired it can actually execute the interrupt handler because interrupts are disabled And so it doesn't actually execute until interrupts have been re-enabled. Yes question. Yeah, so that's That's a question. So it's a great question. The question is what is my definition of external event? I should have been more specific. What I mean by this is a hardware event Whatever that means it could be a timer IRQ that fires. It could be an IO device that says hey, please give me a sample Depends on what you mean by priority. I'm not sure I understand the question So when I when you call local IRQ disabled the all the interrupts are disabled in the local CPU, correct? I'm sorry The same level. Okay, so you're referring to a hardware prioritization of interrupt features, right? So I don't actually have a complete answer to that question I'd be interested to hear about how you're using that feature because Talk to me afterwards. I'm interested in that question. But yeah, I'm not assuming a hardware prioritization of interrupts This is all purely a software dispatch IRQ dispatch But great question So there is a lot of questions I have like I told you there's like 80 Instances of this being used and the driver's portion of the kernel tree as a 414 I said don't use it Of course, I can't make that like really strong statement never use it There are situations where it's actually required again if you are Absolutely going to be executed in hard interrupt context or you're involved in interrupt dispatch in some way. That's GPIO IRQ chip implementations or IRQ chips Those sort of things it makes sense to use okla IRQ disable maybe But It could also be that it's just buggy or for example the drive the device driver is Wasn't written with SMP in mind. And so there's assumptions being made That the interrupts always going to be delivered to the local CPU and so it might just be buggy I'm hoping that there's very few of these. I haven't actually done a complete audit, but it needs to be done at some point And probably the most common usage here and Steven alluded this as well is that it's kind of a heavy-handed way to prevent the tasks being executed from being ripped away from the CPU so if that task is is Needs to access some per CPU data This might be one way that the driver uses that and there's a solution for those people that's using local locks So there's a question about local locks in the last section This is what local looks local locks look like currently in the RT patch. Now. I should be I should make this clear This is a feature in the RT patch currently that in the next few months will be proposed for mainline inclusion But here's what the API looks like you use this define local IRQ lock declarator macro And then within your function you can replace local IRQ disable And local IRQ enable with these local lock IRQ and local unlock IRQ And if you don't build on RT, these will end up redirecting to local IRQ Disable and when you build with RT It provides it the following guarantees. So writing your device driver. These are the semantics you should be Expecting from these API's Semantics are as follows basically What it prevents is? two different tasks running on the same CPU From running in the critical sections at the same time This is a we looked at the the semantics that that drivers are typically Requiring for these this sort of synchronization model And that appeared to be what is really required for a lot of people per CPU accesses and so on and local IRQ disable and so If these are the semantics that are good enough for your driver and what guarantees you need Then local IRQ lock is a good fit. I'm not going to go in each of these in details but under contention There's a mutex in place. There's proper priority inheritance and so on Again, if you have questions about this, you can send an email to me or Stephen or the Linux RT users list Because sometimes the semantics can be a little complicated But it might be a good fit So some of you might be wondering okay Well, I don't see a lot of local IRQ disable local IRQ enable calls and the drivers tree But what I do see is a lot of these functions. I see spin lock IRQ. I see spin lock IRQ save What's going on with them? Well, it turns out when you enable RT if you do spin lock IRQ Interrupts are not disabled. We just ignore it Use it like it is a spin lock or in the RT case. It's mutex And we just ignore the hint that says also please disable interrupts and we can do that because interrupts are threaded And I'll talk about spin locks a little bit more when I talk about scheduling latency So now I enter into a different class of problems that still caused this IRQ dispatch latency and so this is a unique situation that we've we continue to run into run into our products and So I'm making device driver authors aware So to give you a little bit of background I was running a cyclic test test on my on my desk And I had an ethernet cable running to this box and it was one of those ethernet cables where the RJ 45 connector like the little plastic pin was broken off, you know You've always dealt all that with this I'm sure and the test was running and I bumped the cable out I got up to go get a cup of cup of coffee or something I bumped the cable out and I plugged it back in and there was just this massive latency spike And I was like what the hell happened like I just wanted coffee and So we did a lot of debugging and it was very confusing But we eventually figured out what the root cause was And it's something that you might not actually think about and I'm asking you to think about it so Here is the basic the basics of what was going on So the device when it was being initialized when I was unplugging the cable and replugging in the cable as soon as I plugged in the cable Did the network interface went up and down as soon as it went up or went down and up when the interface went up There was just a bunch of register rights to like some configuration block and this particular hardware device And then there was a read and the read was like, okay Well, I wrote all this configuration data now. Tell me when you're done doing your configuration, whatever Now On this particular device there was a pretty It was a PCI device and PCI Rights are posted so they're effectively buffered in the interconnect to be delivered to the to the device And that read that comes in is gonna push all of those rights out of those those store buffers in the interconnect until they get to the device and then It's gonna return the the read response back to the CPU But the thing we've encountered is that The CPU actually cannot dispatch interrupts until it gets that read response back So then you see something like this where you see okay I've seen this thread it did a bunch of right right right right right right and then did a read and it just like Was waiting for those buffers to propagate down to the device and for the device to do its thing and then to come back And in that interim time The timer had fired for the cyclic test thread and it can't be serviced because architecturally it's required that Interrupts are delivered in that particular instruction Order so there's some order architectural constraints in the hardware that requires this ordering and so therefore interrupts won't be let won't be delivered until the read completes and So and not when once that read completes then the interrupt can vector off But you can measure that latency and so I did So this is one this is the test prior to me bumping this cable out And so you see after this test run you have a max worst-case latency 17 microseconds not that bad And then I wrote a test case that all it did was just do right right right right like it picks a PCI device A memory bar that I knew was gonna be safe and just like pile down a bunch of rights and follow it by a read And then and then measure the latencies And this is what I get So this spiked all the way up to a hundred. That's like a tin tin X spike here to 174 microseconds That's pretty atrocious So the lesson is don't but my ethernet cable I guess But the question is what do we do about this And what am I asking you as device driver developers to consider when you're authoring your drivers? What I would ask of you is to consider this case where there's buffering in the interconnect What we actually did to solve this is actually follow each Right by a read back and that actually prevents the the rights from being stacked up on one another in the interconnect It's kind of a crappy solution. I wish there was a better one But it's what we have available If you have any other ideas, that'd be great there I've heard mumblings that in some of these CPUs you can have some Buffering at least in the CPU complex itself Construct timeout control timeouts and the store buffers and so on But that just seems like a lot of architectural black magic. So question. Yeah, I agree This is where the throughput and latency considerations just I Mean they they come at one another and I don't know what the right solution is Maybe the right solution is to provide a higher level Accessor that's like a ride L with read back and then like if you're building an RT then do the read back Otherwise, don't do it if you're not building for RT to bound that latency. I mean, I don't know we have options I don't know where they are. Yeah, the enemy is hardware. Yeah So the the solution to this class of bugs is to take your take the drugs away from the hardware people Yeah Yeah, speaking of drugs and hardware people Some additional MMI woes as I called him We've seen this Not not just buffering in the interconnect. We have one device That we've seen that actually will enter into some lower power state and then when when an MMI O transaction happens It will like exit from that low power state and that exit latency can actually be measured on the CPU side So again take the drugs away or at least ask the hardware people to provide some knobs that you can take and say Oh, don't go into this low power state, please Secondly Another thing we run out and run into was like this. I think it was like some spy EEPROM. It was just memory mapped It's a terrible idea, but it was done and don't do that, please So that's all I had on on hierarchy dispatch latency any questions there otherwise I'm going to talk about the other phase Okay, so at this point You have a thread that has been told this you've told the scheduler Please execute this real-time thread and From now on I'm accounting that the time spent from that point to the youth to that RT thread actually being scheduled in as the scheduling latency I'm going to talk about causes of of potential causes of the scheduling latency So to show you one of one such example This is very similar to the local IRQ disable local IRQ enable Case, but this is instead the set of accessor is the disabled preemption on the local CPU So just to give you what that looks like in the diagram So you have some thread executing it does a preempt disable And an external event happens and the interrupt interrupts aren't disabled So it just it executes immediately the CPU vectors off into this interrupt handler The interrupt handler is going to wake up this high-priority higher priority thread But it can actually be scheduled in on return from interrupt because preemption is disabled and So you have to wait until that thread has done a preempt enable and at that point you can schedule in the higher priority thread So it should be clear that usage of preempt disable preempt enable are going to Impact that that delta any questions about that The only reason that I can come up with after after thinking about this and maybe you can think of some other other ones, but is that Why a device driver should be using preempt disable and that is if you need to synchronize with the act of scheduling itself If you don't fall into this category then Probably should not be using preempt disable and you can use local locks as an alternate solution here And I did a grep. It wasn't that bad. I think it was only like 60 users in the driver's tree. So Not that bad. It's not like we're talking about analyzing, you know, 100,000 spin lock critical sections or something So it's manageable auditable Speaking of spin locks, however on mainline when you do a spin lock spin unlock on mainline Preemption is actually disabled So if you look at a mainline execution trace and you want to look at these these latencies You see the same sort of behavior the way that we saw them the RT case is make spin lock critical sections preemptable And we do that by turning turning them into PI aware mutexes And disabling migration is a little semantic quibble But in that case when spin lock when a spin lock is being held by a thread it can be preempted by a higher priority thread and so therefore you can Bring in that that outer bound Any questions about that that's a that You have to think about this for a while think about the spin locks sleeping for a long time before it's like How is that even okay? And if you combine sleeping spin locks and forced hierarchy threading you're gonna have to have both of those Pieces together and think about them together to understand how they actually Work and are successful for primed RT. So This is unclear to you. I'm happy to talk about it afterwards Because it's definitely Confusing and again the story is pretty clear. So if you are riding a device driver And you're using spin locks, then you're not really going to impact a RT thread priority Or RT priority thread So you're just lucky now again it works until it doesn't work And so if you're in the situation where you do need to synchronize with Some code running an interrupt dispatch Or things that can be invoked in the scheduler. So example for example like the spin the the scheduling core All of its spin locks internally are raw spin locks And raw spin locks are traditional spinning locks under contention and If you use the the raw spin lock API which looks pretty much exactly like the regular spin lock API by design Before you consider doing so Think about the cases where you might need to I need to do this because I am invoked from hard interrupt context I'm involved in interrupt dispatch. I'm involved in scheduling. That's a good good enough justification But think about it when you were doing it Because you need to think about the critical sections themselves and you want to ensure that they are doing the absolute minimal amount of work possible and Especially that it's bounded because any critical section is going to impact that worst case Delta, so please keep that in mind if you're going to do this in the future and Again, we're always happy to review things or at least I am So in conclusion these are the four things I hope you walk away with So the first one is if your drivers aren't involved in interrupt dispatch And you probably shouldn't be using local IRQ to say a bully probably shouldn't be using local IRQ save Look at using local locks Secondly just consider MMI O axis is I don't have a good story here just to think about it and see if you might get away with doing readbacks That might be suitable look at your own applications to see if you can Yeah, I don't know. There's I don't have a really good story here, but just think about it If your drivers aren't involved in scheduling then you probably shouldn't be using preamp disabled and Instead use it use a per CPU accessor. So the RT patch also introduces a Different set of per CPU accessors called get local CP get local Get local CPU and get local or something and put local local var Yeah, yeah get see yeah local var and put local var and those are access primitives that allow you to do per CPU access that only Disable pre only disable migration and don't disable preemption So that might be an option for you If your drivers are involved in interrupt dispatch and scheduling then you need to use raw spin locks You can't nest a regular spin lock within a raw spin lock That's bad In fact, we've discussed it for a long time about making that trigger locked up splats right now in mainline So we can catch these issues. We haven't done that yet But it's possibly coming in the future or certainly coming in the future. I just don't know when So I ended a little early and I have plenty of time So if you have questions, please ask me if you have questions afterwards or want Clarification you can reach me at julietini.com or juliet kernel at our work or my Twitter handler or Linux R2 users list. That's actually a great resource also great resources that pan linux or TNO if you see so We have a All the cool kids hang out. I'm told I don't know. Do I fall in that category? I guess Question up here. Yeah, so that definitely falls into the category of sure. Yeah, so Yeah, yeah, so this this gentleman mentioned that he had also run into what appeared to be some MMI Oh, what or in my woe as I like to call them And where he was saying you trigger an array cycle for a flash and then when you read the status register It blocks in the pipeline until it's delivered. Yeah, that's certainly definitely falls into that same category thing And I don't know what the right answer is But yeah, that would and he said power PC also suffers from from this pipeline stall So, yeah, I don't know what the right answer. Yeah, I'd be interesting to talk if you had solutions for that I don't know what you did or not you did. Okay. I like to talk about that Maybe afterwards. There's a question in the back here if you can get the slides Yes, they will be up on the on the scheduler page after this talk Also on my Twitter feed if you need them before that question so So the question is and and the question is how does IRQ threading work? So let me describe a little bit more about how it works and hopefully that answer your question And then if it doesn't please ask again In order to support IRQ threading it requires the use of the IRQ chip primitives is implemented So what actually happens is when the interrupt fires we jump into that little shim as I showed The shim actually will mask the interrupt at the interrupt controller Right, so the device is still asserting its interrupt because it hasn't been handled yet, but it's been masked, right? And then we wake up the the associated IRQ thread And if that IRQ thread happens to be the next highest priority thread to be executed Then the scheduler is going to execute it and on the tail end of that handler after the associated handler is executed Right before the thread goes back to sleep. It actually will do the unmask Okay, so the question is what happens if you have to real time like real-time events happening or interrupts happening at the same time Well it if that occurs then it's up to the user or system integrator to properly prioritize those interrupts Because you're now in the realm of scheduling you have all the tools available to you in that realm like adjusting priorities Or adjusting scheduling class even You can adjust these to be like round robin or FIFO or deadline even I'm sure I don't know Oh, yeah, so there's there is a Function called request threaded IRQ. So there's two main registration functions. There's request IRQ and request threaded IRQ So on mainline how this works is is the traditional ones request IRQ the handlers that invoked and hard interrupt context If on mainline you use request threaded IRQ you actually specify two handlers one of those handlers is actually what's executed in hard interrupt context and Then that handler can return IRQ wake thread and IRQ wake thread will actually wake the associated IRQ thread that will actually run in thread context So that's how it works in mainline today. So RT the way this works is that actually will force thread the hard IRQ handler and the IRQ thread in mainline Maybe I didn't get your point look at That line Yeah, so threaded IRQ is is also kind of a Generic work deferral for interrupt handling. So it falls into that category. So there are mainline uses where it makes a lot of sense As well, so thank you Steven. There's a question in the back corner. Okay, great questions Yeah, yeah, actually, yeah Yeah, yeah, I don't know if that's true. I think we'd have to look I believe this was changed In the past several releases because there were device drivers that we're doing a trying to acquire a spin lock in their top handler I don't know that we did we can look at I will absolutely look afterwards if you want to but Or Steven's gonna look right now, but I believe that even the top handler is threaded In RT as it exists now. Yeah Yeah, there is yeah, so tasklet is is one IRQ work. So the question is about is there anything special about tasklets Not really only that if you raise the task at soft IRQ, it's associated with the thread in which you've raised it and So when that thread is done executing the handler, then it will start executing the software cues that were raised in that thread So therefore in that thread context, right? You have your IRQ thread. It's executing It will execute the handler that has been force-threaded that handler is gonna raise the the task that software cue and at the Termination of the handler. It's still in that thread. It's gonna run the task that right then at the very end In the context of that IRQ thread. Yes And that's true of all so not just task list, but it's true of all software software cues that are raised in thread context Yeah, yeah, so the question is doesn't this break Any device drivers that might be involved in the real-time portion of this application of an application the answer is It depends You might see higher latencies because now there are more context switches But you should get better bounded behavior For those and you have to also do all the configuration necessary to actually get these things prioritized with respect to VRT application But maybe I would hope not. Yeah Yeah, that's a pretty tight bound depending on your on your system four microseconds is pretty Pretty tight. I don't know if you'll hit that Yeah, so even on the highest power like Xeon system that I've tested on I'm still seeing like 10 microseconds worst-case execution now in the arm systems we have we see maybe 25 30 microseconds, so Yeah, I mean and and like I said like for those really tight bounds and Yon mentioned this before in his presentation is if you need those tight bounds You might choose a different strategy. You might go to using Xenomai You might spin up and push all this stuff into another hardware unit and you can offload those might be solutions Or maybe it's not possible on this particular button. I don't know it really depends Any other questions Olaf was this was this just a general statement because I agree Okay, so for debugging this. Yeah, so first of all I'll repeat the statement that Olaf made he says that debugging priority inheritance problems that exist between a real-time application and a interrupt thread Can be painful to debug what techniques can be used to debug them? I typically just use the standard sketch timer trace points There's sketch a timer and IRQs or the typical ones I enable but it's really just slogging through some Traces and figuring out what's going on. That's not an ideal situation There was a conversation yesterday. I had it was a little bit interesting Which was that it might be possible to leverage something like? Well, let me let me back up a little bit. There are Unzini my apparently I don't I haven't used in a my much but for a an application You can have it such that if that application if that real-time application actually invokes a system call or something that that exhibits known unbounded Unbounded execution behavior it is sent a signal and then that task can either figure out what it wants to do a dump stage or Core whatever has an application debugging tool It should be conceivable to implement something like that using sec comp in Linux So I'd like to pursue that as like a usability to bug ability tool Especially for people that are taking applications from non Linux platforms to Linux But yeah, I would agree with you in general. There's a it's there's a debugability problem. Yeah Yeah, yeah Yeah, yeah So the comment is even if we chose some sec comp a solution It's going to be difficult to maintain because there's going to necessarily be a blacklist or whitelist that we use to determine whether or not a System call is allowed or not allowed, but exhibits be certain behaviors or doesn't and that's going to be a difficult thing to maintain Maybe it's going to be good enough for just application to bug ability might not be useful to actually enable in deployment So I think there's there's some trade-offs we can consider there any other questions Otherwise again, I'm happy to Answer questions via email how to review patches. Let me know Great. Thanks