 So hello everybody, and welcome to the Real-Time Linux mini-summit. This is our first year of Group 12, but we invite everybody to participate as if we were all together in the same place. And this year, we try to put more introductory topics, trying to invite more people to participate to join on our community and to make connections, right? And our agenda this year starts with Stephen with his famous presentation introducing the Real-Time Linux, explaining who needs it and who also doesn't need it, right? Then we'll have Joe Kacer presenting the Real-Time Test Suite and RT Evolve, which is the tool that we as developers use it every day, and he'll explain us about the past and the future. We will see a very nice use case of the Grand RT on the determinist networking to control on the robot arm by Herrick, and we will finish it with a panel discussing what are the plans of Real-Time Linux with Thomas Flexner and Kate Stewart. And I invite everybody to participate, making questions in the chat, and we will try to address during the presentation or at the end of each presentation. So with that, I pass the word to Stephen who will start his presentation now. Let me see if I can share my screen. Can everyone see it? Okay. Okay, so let me get started. Hi, I'm Steve Rosted. I'm open source engineer at VMware. One of the original developers of the Real-Time patch and I currently lead the effort in maintaining the RT stable trees. So I only have a half hour, so let me get right to it. So first, this is an introductory kind of presentation I'm having here. So I'm going to ask, what is Real-Time? It's kind of like saying, what is your favorite color? Blue. No, red. Anyway, the term I feel is ambiguous to say the least. So if we were to go search on the internet and look up the definition of Real-Time, let's start off with UrbanDictionary.com. You can find some really interesting definitions here, but I just searched Real-Time and there I found what Real-Time is, is instantaneous. Taking place at once as other things are also in progress. When I surveyed the situation in Real-Time, there were only four people who met the qualifications. It doesn't really tell you how much. Let's try another place. Let's go look at whatis.com. And here it says Real-Time is a level of computer responsiveness that a user senses as sufficiently immediate or that enables the computer to keep up with some external process. For example, to present visualizations of the weather as it constantly changes. Real-Time is an adjective pertaining to computers or processes that operate in Real-Time. Real-Time describes a human rather than a machine sense of time. Still doesn't really answer my question. So let's try Google. Here I find it says the actual time during which a process or events occurs. Along with much of the country, he watches events unfold in Real-Time on his TV. It has something about computing too, which comes to relating to a system in which input data is processed within milliseconds. So that is available virtually immediately as feedback. For example, in a missile guidance or airline booking system. Well, missile guidance in milliseconds, okay. Real-Time single processing. But for us, for this talk, for what preempt RT, we're going to say Real-Time is basically doing what you expect to be doing in the amount of time you expect it to be done. What does that mean? Determinism. I always say that I hate the term real-time operating system because of this ambiguous ambiguity or whatever you call it ambiguous term. And I like to think it should have been called a deterministic operating system. DOS. Okay, this joke never gets old. At least doesn't get old for me or some of us old phobies. But actually, DOS was actually a very good Real-Time system. It would always do what you expected to do. It would crash reliably. So what does Real-Time operating system give us? Faster. I've always had a lot of people come to me and say, hey, I can't wait to run the Real-Time operating system or the preempt RT patch and see how fast my system is running. Is it faster? No, they're usually up for a surprise. In fact, sometimes it will be slower, but I don't like to advertise that Real-Time is slower. It's not quite. So what Real-Time does not mean fast. It means real. It means actual time. So a lot of times people think Real-Time is fast. I don't know why. It's just because it's real doesn't mean it's fast. But what does it give us? Again, determinism. But being a deterministic, what does that give us? Repeatability, reliable results, a known worst case scenario. You know what your worst scenarios will be. You could actually calculate it in theory. When there's something that you have to get done within a deadline, and if you don't get done within that time, the system fails. Period. You have to start over, crashes, whatever, good or bad. You can't have any case where, you know, hey, if this race condition happens, we're going to have this outlier that's going to fail or break our deadline. In that case, the system is failed and it's not a real-time system. So you also have known reaction times. And going back to about saying Real-Time is slow. I don't like to say that a Real-Time operating system is slow. I like to say it's the fastest worst case scenario operating system, which means that compared to non-Real-Time systems, a Real-Time system will be faster when everything goes wrong than any other system. So if you have another system, you know, you can say, hey, this is fast 99.99% of the time. Well, that .001% of the time, it could be really, really slow and fail your system because of it. Here's another little pet peeve of mine. And a lot of times when we first started developing PRAMPS RTE, a lot of people said, Linux can never be hard Real-Time. The reason why is because in the old days, hard Real-Time, there was no other operating system. The only time you had a real hard Real-Time operating system, you had to create it from scratch. So a lot of money had to be invested in doing such a job. So what they say back then is mathematically provable. You could prove that it will never have an outlier. So because this operation is sort of NP complete, the bigger the code gets, it becomes almost impossible to prove its correctness. Another thing that hard Real-Time gives you is that bounded latency. There's going to be latency, but you could know what that latency is and you could mathematically calculate that this is the worst latency that you have to deal with. On the other hand, what does soft Real-Time give us? You know, you heard the term soft Real-Time. Well, it allows for outliers. As long as the average is predictable, what they have soft Real-Time is defined as is that you can prioritize tasks. You could have this task, this process be a higher priority than this other process and majority of the time, this process will act deterministic, but it might have outliers. It's still reliable, but it could have an unbounded latency that you don't know about. So let's go back to some Real-Time examples. Engine control systems. My first started working in the fields. I worked for General Mortars and I did Real-Time. We worked on the C-17 engine control system and that was a hard Real-Time system. It wasn't Linux, obviously, but it was one of these things. They spent a lot of money in developing a system that was reliable and it made all its deadlines and it took basically 10 years before it could be deployed because of all the regular testing it had to go to was really controlled. So a lot of people think of that as a hard Real-Time system and that's the mind I'm trying to get people out of because there's other use cases. Robotics. Robotics, like an assembly line. If you have an assembly line that has robotics that are doing machine work, industry and such, it needs to be Real-Time because, like I said, when the assembly line is moving, that's Real-Time. So when a product comes by that the robotics has to work on, it can't be late. It can't let the product go by and then go, the arm comes down. That's on this field in the system. So that's dealing with Real-Time controls. In that case, you need a hard Real-Time system. But believe it or not, music comes in this case as well. It was surprising for me, but actually music requires Real-Time. Hard Real-Time. I'll get back to that. Some soft Real-Time examples. Video. When you're watching a video, if you're watching a lot of time with Netflix and everything, there's a lot of video streaming going on. And that's a Real-Time system as well, but it's soft Real-Time. It's, it could miss an outlier. If you miss a few things, it's more or less, you could have outliers and it's kind of a spectrum. Less outliers you have, you don't even notice it. When it gets to be more outliers than it gets annoying, you have frames freeze, that's a soft Real-Time system. Telecommunications, same thing. If you're talking on the phone and you have a glitch or if you have bad communications or there's network, that's also a soft Real-Time. Well, if you're talking about sound in a telecommunication system, well, what about music? Why is that hard Real-Time and not the telecommunications? Well, when we first started working on the pre-empt RT patch, I was surprised that musicians were actually one of our first testers. I got an email once, you know, a bug report from this guy and I sent him a patch and said, can you try this to fix it? And the guy said, well, no, it doesn't quite work. And I looked, I, oh, I made a mistake. Could you do this trivial change? The guy's like, whoa, whoa, whoa, I don't know how to change code. I'm like, wait, you can apply a patch, but you don't know how to change code. What do you mean? He's like, I'm not a computer developer. And I was shocked. I said, what do you mean you're not a computer developer? Why are you using preempt RT? He says, I'm a musician. I'm a guitarist. And I'm like, really? He says, yeah, I just know how to apply patches. That's all I know how to do. I don't know how to touch the code. And what he told me, because I started talking to him back and forth, and he said that what happens is when he does a recording, the preempt RT kernel was the only system that he could afford. Like I said, hard real-time systems, you have to create from scratch, but Linux provided with the preempt RT patch, we provided a hard real-time system that he said that when he did his recording, if there was an outlier, it would come in as a scratch for those that don't know what scratches are like from a vinyl record, static. It'll be like a static in the recording. Now, if you're doing music and beautiful thing, any little static or glitch like that kills the song. You've got to start from scratch again. If you're doing a recording, you get a scratch, you stop. That's a system failure. That's why music requires hard real-time because it comes down to if there's an outlier, if there's something that fails, is it going to destroy or fail the system? And that goes underneath music. This conference is soft real-time. As we probably found out throughout the week, there's a lot of times things glitched up. Yeah, it's annoying, but it doesn't fail the systems. You continue. There's workarounds. That's soft real-time basically for you. Real workarounds. Because you'll allow a few missed deadlines as long as it doesn't crash. And FYI, Linux without the real-time patch is a soft real-time system. It allows for you to prioritize various aspects of the kernel or the tasks to go in. And in fact, it's become quite a very high quality soft real-time system because of the work of the preempt RT patch. Preempt RT patch has been going into Linux for the last decade or so. So there's so many features that the real-time vanilla Linux got that came directly from the real-time patch. That vanilla Linux works so well that I've talked to people who use recordings that they said vanilla Linux, most of the time, works good enough. They don't have to even use the preempt RT patch because vanilla Linux is close to being a real-time system that they don't have those latency of scratches and such in their systems. But what about the real-time patch? What is it? If it's hard build time, it has to be mathematically provable and whatever. But let me say it's a soft real-time OS. I'm like, we don't allow outliers. The real-time patch does not allow any outliers. And we don't have unbounded latency. So it's not a soft real-time system. But is it a hard real-time system? And the thing is, it's too big to be mathematically provable. And that's why I said it because I'm empty, complete. You can't quite actually prove it to be correct. But what is it then? I like to tell people it's hard real-time designed. Like I said, it's too big to be mathematically provable. Although some people are trying to do so. Right, Daniel? All design decisions are to make it hard real-time. So every design that we do inside the kernel, we look at it says, can this little section of code that we're working on, does it have a worst-case scenario that we could define? We calculate and say, this could take so long and we calculate a worst-case scenario. Now, remember, some people say hard real-time means that I need to have, you know, reaction times within one microsecond. And I, first of all, Intel architecture will not give you that. And I'll talk about that a little bit later. The thing is, though, hard real-time just means, can you define the worst-case scenario? Now, yes, there's going to be different operating systems. You have to make it really, really small and run on special hardware if you are having microsecond reaction times that you need. But if you, a lot of cases, you only need millisecond extra times. Heck, that one definition said milliseconds. And something like Linux with the pre-emptory patch, milliseconds is like a lifetime. I mean, it takes, that's forever. We don't have any issues with making millisecond reaction times. We're usually looking at less than 100 microseconds. So, yes. The only thing that breaks this in our design is considered a bug. So, again, hard real-time sometimes comes down to quality. When people say hard real-time, it's the quality of the code, not really the code itself or making it real-time. It has nothing to do with real-time. It's more about quality. So, if everything works, if there was no bugs in Linux, and we wish for that. If there was no bugs in Linux and we had the design decision, we would be, we would easily be defined hard real-time if we could mathematically prove it. Another thing is, there are exceptions. There are some places in the code that we know that can cause a unbounded latency or missed outlier, but they're well-defined. We could say that avoid doing this and your system will run perfectly fine. And if you document it, again, hard real-time is all about the system itself. I've talked before in another presentation saying that when you do need real-time, it's from bottom-up. You need everything to be real-time. So, when you design your system and you know that this operating system that you're using has paths that can make it so that's not, that you might miss your deadline, you can avoid those paths if they're well-defined, well-documented. And there are some cases. I mean, there are some places in the file system that you could cause you to have missed latencies. This file system is writing to disk and such like that. And also, if you're doing some tracing or debugging operations, I could kick off some trace features that are known to break if you could compile them out, too. So, if you're worried about them, just compile them out. What about latency? When you look up the definition of latency in Wikipedia, you say it says, latency from a general point of view is the time delay between the cause and the effect of some physical change in the system being observed. Actually, that's not really a bad definition. I like to call it latency is simply the time between when an event is expected to happen and when it actually does happen. Think about when you're sitting in the traffic and you look at the traffic light and the light turns green and you, the time between the light turning green and the car in front of you moving forward so you can move forward. That's a latency. Types of latency for what we're worried about. The first one is interrupt latency, and I like to say that there's two types of interrupt latency. Now interrupts, as you should everyone should know, but if you don't, is basically what happens when a network packet comes into your network interface card or when you type on your keyboard. The device needs to tell the CPU there's something that it needs to work on because the device just says it receives a networking packet. Now it tells the CPU go processes networking packet. So it sends an interrupt to the CPU and the CPU has to stop what it's doing to and jump to some memory location where the interrupt handler exists and will execute that code to process the networking packet. But that could take a long time as well and, well, your process if you have a high priority process running and say on that same machine that you're running your, you know, robotics system, you have your child doing playing video games using the same machine and it's the video or over the network and it's just bombarding the system and sending a bunch of packets every time that interrupt comes in it stops your high priority process from moving and you'll miss your deadlines and if you have that donut hole puncher it's way make off centered donut holes. So you don't want to do that and so I call the time when so if you get a lot of interrupts coming in. I, the time interrupted by the interrupt is a latency how long that your system is has to wait for something else to run. And then you decide if your system requires a stimulus from a device and like if you want to have you have a sensor or a networking packet that you want to respond to real quick. Well the CPU can turn off interrupts. And this is when interrupts are disabled. So the CPU could go on doing whatever it wants, but it disables the interrupts so all these devices have to wait for that CPU to say hey okay I'll listen to you again. That's another type of latency that could happen I call it interrupt disabled latency. And then of course there's shared resources between lower priority tasks and higher priority tasks and they're sharing the same resource, you have to usually have locking between it or to protect it. And when a high priority process has to wait on a low priority process to get out of that critical section so it can get in there, there's that latency as well. So talking about the interrupt handlers. When a high priority process running and it gets preempted or interrupted by the interrupt that comes in, and it has to wait for that handler to completely finish before you go in so this whole time frame happens to be the interrupt latency for that interrupt. And preempt RT solved the situation because hardware interrupts is a hardware feature this that we can't change the design of the hardware and how to handle this. So we still need to take the interrupt, but we can make that interruption much smaller and calculate it. So this is what it looks like in preempt RT so instead of actually doing a true handler handling the interrupt. It will just acknowledge to the kernel saying hey, this interrupt came in deal with it and the kernel could go back to running its high priority process, and it has a dedicated interrupt thread that will actually handle that interrupt. And because it's a thread that means that gives users access to modify the threads priority you can make a high priority process or thread because of your high priority. If the interrupt task requires a device to access, you can make that device is interrupt handlers high priority so it could run over everything else, or if you have a bunch of things going on that you don't care about. It will make those interrupt handlers lower priority and let your higher prior process run without being interrupted I mean interrupts once for the interrupt comes in, but then the interrupt is disabled until the thread could run. So you won't get it again, which means that you could calculate what that worst case scenario is. If you preempt RT patch, and you do a PS grab IRQ, you'll see that there's a list of interrupt threads that you have control of. So latency always happens. Nothing is instantaneous, no matter how good an operating system is, there's always going to be a case where a response comes in it's going to take time before you could reply to that response. This is when something that should be running has to wait for for something that shouldn't be running. If a high priority process wants to run, or it needs to run and is a lower priority process that's a way for that's called priority inversion. This to always happens, unless you could dedicate a system that doesn't have any shared resources whatsoever and you just have a single thread running without ever being interrupted and there are cases that there are designs that they do that, where you won't have a priority process. So once you have anything that needs to interact with each other, there's always going to be a cases where something is a shared resources being involved, something has to lower priority process will wait for the higher priority process to get in. What we care about is bounded priority inversion. This means that you know how long that higher priority process will wait always, and you can calculate it in theory that you can always say I could actually calculate what the worst case scenario be as long as it's I'm fine. But we don't want is unbounded priority inversion, which means that you don't, you can't calculate it there's no way to know what this is. And this is the priority version that we have to watch out for. And this is my favorite slides I show this and every single RT talk I've ever given I always have to throw these slides in just to show them anyway but for those that see my talks before you know what I'm going to talk about. Yes, there is a Mars rover or something like that back decades ago ever that was going up to Mars and hit this priority inversion situation previous exactly, and they reboot and they actually had to disable one of the threads to keep this from happening again. But the way priority inversion works is, let's say you have three processes process a which is the highest priority process process be median priority and see the lowest priority process doing some just logging or whatever. Process C is going to share some resource with process a so there's there's some mutex around it. So when process C is executing it grabs the lock goes into this critical section and a wakes up preempting C. So a runs now a wants to go into that same critical section that C isn't so it grabs the lock that C has, but because it has the lock it's got to wait so it gets blocked and goes back down and see runs. In the meantime before I could release that lock be wakes up which is higher priority, higher priority than see, and now it goes runs one by blocking C. It also blocks a which is of a higher priority process in itself and this is where you have unbounded priority inversion. Let's say that the process B has to. Let's say process B only stops when a stops. That means it will never stop and just goes on and is dead. Of course, well, the way that the preemptor T solves this is what we call priority inheritance. And how this works is the same situation happens sees runs goes once again to the critical section grabs a lock a wakes up once again to the same critical section grabs a lock block. Well, here's the difference. When a blocks on the lock that C has C will then inherit the priority of that lock, or of the process that's waiting on that lock. So it inherits a is priority. Now see, while it's running holding a lock that a is blocked on. It will be running at the priority of a. So now when B wakes up, since B is lower priority than a and see is currently running at the priority of a, it can't wake up and see gets to finish this process. If it releases the lock, it now will lose its priority, but now a can grab that resource and it will wake up and run and still keep B from running, but it could finish and then tells be okay you're ready to go. So this is how we handle a priority and inheritance. And before I finish the talk here, I said in the beginning that it's more than just a kernel. And I don't want to make sure it says this to that you go. No matter how good the operating system is, if it's running on hardware that's not good for you, or it's not reliable not deterministic. Your system will not be reliable and it will not be deterministic. It's funny because a lot of people care about real fast. And when you care about real fast, you don't need real time. If you ideally you want both, but it's a balancing thing you have to do you either you have to maybe have to give up some speed to make sure that you're deterministic if that's what you need. And like memory cache memory cache inside the kernel is our time to computer is something that makes the system run fast because the CPU runs faster than the memory bus so it's always pulling stuff in and every so often if you have a like you need to access a memory that's not in your cache you have to basically stall to pull in the memory, and that's unreliable and that's, it's makes things fast, but it makes it so it's not deterministic we can't predict when this is always going to happen or it's very very difficult to predict it always. And what they can do is it could turn off memory cache but your system will now run so slow that maybe you won't be able to make your deadlines and you know that the actual worst case scenario somewhere between memory off and cash completely on. So it makes things more difficult and say they have TLB misses, when a page, when you pull in your page and you have to access memory that's not in the, in the TLB and then the system has to go and walk the page tables again. That causes a non deterministic latency, although you kind of calculate it, but it can happen it's very hard to get these things right. System management interrupts is really the, you know, the Achilles heel to all real time systems, if you have a system that runs like a system management interrupt is when the bias is doing something for example a lot of laptops will have thermal controls that's controlled by the bias, because it doesn't trust the operating system and for good reason, because the bias or the vendor knows exactly how the thermal controls work on your system to make sure that your laptop doesn't burn up. So it will kick off code within the bias and that's called the SMI which stops the kernel from running to execute this special code in system management mode that the kernel has no control over. So no matter how good your kernel is it can't stop these SMIs. So the OS again can only be as deterministic as the hardware is running on. And with that, I hope you enjoyed the show and I, I'm going to stop sharing so I could see the rest of the screens down the what's going on. So, we have two minutes for questions. And the questioner be hello by chat. Yeah, I think there's one question in the chat already. I'm from Pavel saying, is there, is there a list of the unbounded operations in the RT kernel somewhere. Not yet. I think we're looking at that I think we're trying to I believe some of the issues was I believe comes up the real the read write lock that we have to look at, because I'm sure Thomas might be able to answer this too. I think the reader writer locks is something that's of a issue because you have a situation of multiple processes that could be back at own a lock. So if you have a bunch of readers running, you know writer comes in priority in priority inheritance becomes almost exponentially more complex believe me I tried to do this a few times, and it all in failure. So, let me it works but gets buggy. Basically, ideally, if we try to avoid paths that have the reader writer locks which I believe is in the file system such so you really have to analyze the system. No matter what you have if you're using a driver, we haven't analyzed every single driver driver might be doing this as well. So, there isn't quite a list but we, once we, once we start pushing this and if you're going to do something this is why you hire people like Thomas collection or to go and analyze your system and see what you're doing to make sure that you don't have to avoid these paths so it may not be well documented right now, but we it's it's not hard to find out these areas. I guess that's the best. So, maybe people have more questions receive if they want to put them in the chat we'll circle back, or he can answer in the chat to us, we got one more general rule of thumb for ranges of latency it's. It really comes down to hardware. And I think we said I, when I run my machines I just have a typical I guess I seven I run on and I do tests on this and I can easily get below 150 or 100 microseconds almost every test stress I do but in theory, the worst case scenario if I believe Daniel's done a lot of work in I guess some papers that read about this that I believe it's more like 100 in actuality Daniel correct me if I'm wrong with like 150 microseconds like on a normal like Intel processor and such. Yes, yes, that's a very safe value. But yeah if you have a real time. If you're doing a slow like 32 bit one gigahertz arm. You know it's probably 100 microseconds I'm sure it's probably good because I also I bet you those arm systems are not as non deterministic as Intel is. Okay, so our time is a little bit short in this version of the meeting. So, Stephen can continue answering the questions we have chat. And we'll have now the talk by John case or about the RT tools. So John case or it's our time. Okay, I'll start by unmuting that's always a good idea. Share screen here. My screen there. Yes. John case or I work for red hat. I'm on the real time kernel team but I mostly focus on a variety of tools for measuring the performance of the real time kernel. The same tools that you can use to test. First of all, whether the real time kernel is working correctly as expected, and then seeing if the latencies are within correct kind of measurement. These same kind of tools are also useful for application developers. Because as you know with real time where we give you enough rope to hang yourself. So you can screw up in ways that you can't screw up with normal Linux operating system. So if you're writing bad code which is taking over the machine in ways you don't want it to then these kind of measurement applications will also let you know where your code needs to be fixed. Okay, so I'm going to talk to you a little bit about RT tests and RT eval. I'll give you a quick overview of everything. But I only have 15 minutes, but part of what I really wanted was input from the RT community at large so I don't care if that's in the form of emails after or chatting and stuff but I'd really like to hear, you know, if everything's working fine and you know what people would like to see from these tools in the future, or maybe everyone's just happy works well enough. What is RT tests. Well, it's a suite of tools but it started off with a cyclic test and cyclic test to this day is as far as I can see it's the main tool by which the kernel developers are testing the latency of the real time kernel. So, Thomas wrote it and Clark started adding some stuff. Clark Williams started adding some tools to it and I think Thomas put the, gave the maintenance of it over to him and then Clark had me fix a few threading problems in various tests and after that, I sort of got addicted to working on it and I've sort of taken over the maintainership now. It's been around for a really long time. I don't even want to say how many years Thomas could tell more accurately. I bet it's 13 anyways if not 15. After cyclic test was written, Thomas and Karsten Mda. They using the model of cyclic tests they wrote a whole bunch of tests and most of them aren't used is to the same degree that cyclic test is but they have the same kind of general model for testing. And some of them are quite interesting and they don't always get the attention they deserve so. I have a list here roughly in the order that I think things were developed. Just remind people they're there. Have a look. So the PMQ test is measuring the latency of inter process communications with positive messages, and then PT SEM a test does the same thing only with new taxes. It's the best over the years to to maintain these extra tools in there but they don't get the same attention that cyclic test does. But I, yeah, that's something that could still be improved in the future, just by sharing more code that would be a good way you know when things are fixed that it's fixed everywhere. There's a signal test that measures a round trip between a signal, the single weight test which is measuring the latency of sending and receiving a signal that system five semaphore test, which is measuring the latency of system five semaphores. Okay, so those tests that I were that I mentioned briefly before they're all sort of modeled on cyclic test. In the meantime, Clark had written this thing called PI stress, which would purposely create priority inversions. And if the real time kernel wasn't working properly, you could potentially have a deadlock. So this would test also the correctness of the kernel. So when Clark started maintaining the RT test tools, then he added this. And then I was talking to Thomas once and he was telling me about you can actually use, whoops, how do I go back there previous. I have a priority inheritance without using POSIX threads and he told me how to do that so I wrote a little test that demonstrated that and that's the PIP stress was sort of modeled after PI stress but without using POSIX threads. So what else that we have in there, which is still important to this day is the hardware latency detector. It originally used a kernel module by John Masters, and, and Clark wrote the Python code. The idea was to measure any kind of latencies that were outside the control of the kernel, such as a system management interrupts in firmware. And there's still this is still one of the reasons why certain hardware is not suitable for real time today. We need to use that kernel module anymore. Thank goodness we can use the built in F trace hardware latency detector and connect to that. That's from Steven Rostett. And then we got a bunch of tests by Steve, and they're also modeled roughly like cyclic tests. And they measured the, the, at the time the newly added schedule deadline. So there's a cyclic deadline and the deadline test. I also wrote this could almost be a separate slide because it's not in the same category but the RT migrate test, which ensures that if you have priority tasks that they're that if there's an available CPU it's running on that CPU. And then came the queue latency written by Marcelo, and it simulates a network queue and to check for latency violations and packet processing. Okay, something that we've added quite recently is something called OS lat. And what OS lat came from our real time plus KVM team. They were using cis jitter. And they needed something to do a sort of pole model test to detect OS level interruptions. They liked cis jitter but they were very unhappy with the output and the way it process the statistics so this is the reason if you're wondering why they didn't just use cis jitter this is the reason that they wrote their, their own version so it's OS lat. And it's kind of cool because it also supports multi threading and F trace interactions and so that's what OS lat is. So, um, quite a while ago I think it was maybe 2016. I thought it was time to say, you know, we've got a done version of cyclic test we'll call it 1.0 and I'll continue to maintain that version but like development had slowed down on it quite a bit so I thought, um, let's spin off a so stable branch where people won't be afraid to break the API's. So that, you know, should allow people to be more creative and they don't have to be so careful let's try a bunch of new things. But what happened is I wasn't receiving really any patches for the stable version of cyclic test. So, but I was receiving lots of patches for the so called unstable branch and um, it seemed like the different Linux distributions were pretty much just using the latest unstable branch as as a stable version so I am kind of considering 1.0 it's long, long ago dead and the unstable branch is just a stable branch. I still have a rather confusing way that these branches are named I apologize for that I'm going to clean that up. So the first things we did after we spun off the unstable version was we removed all the F trace code and it was quite a bit of code and confusing kind of code. And the reason that we did that and we're able to do that is because of Steve Russ that's tool called trace command. We were able to actually just leave in a few break points in the cyclic test code and but allow the trace command tool to do all the actual work for tracing which was great. It simplified cyclic test quite a bit. Now I know that Steve is creating libraries now. This trace command is using and I'm curious about how this is going to change the way tools like cyclic tests use F trace in the future. So, we've also, you know, removed, there was a huge alphabet of options and cyclic tests we tried to simplify it quite a bit. And the last thing we did was you don't specify new many more it's just detected automatically. So quite a bit of work was done to improve how new more works and fix problems with CPU affinity. And this was important for tools like RTV Val which use cyclic test. We actually added a snapshot utility which just allows if you have like a, imagine you're running cyclic tests for a week and you want to get a status report to see if it's worthwhile to keep running it or whether you need to stop it. You can just send it a little signal and get the status and it keeps running. So I hope that's useful to folks. Boy, I'm running out of time here. So, people see hack bench and say what the heck is hack bench doing in the suite. Well, we have our own version of hack bench, because we thought, how can we screw up cyclic tests let's let's create some noise in the background and try and mess with what's going on so All right, it's used by RT Val, which I'm going to talk about next briefly. So RT Val is a program to basically evaluate how the real time kernel is running on various hardware. I know that that's typically telcos and they're running large machines with maybe hundreds of CPUs and new machines and so on. It's a Python code and it runs. It can run multiple instances of cyclic test on various nodes, NUMA nodes or it can just run the threads of cyclic test if it's just one node on various CPUs. It does that concurrently while it's running hack bench and doing kernel builds so we're just trying to create background noise that can screw things up. We also use it for our customers to evaluate whether their hardware is suitable for real time and will certify whether it is or not. This was ported to Python three I did this myself. The older Python two version is still maintained. I see a question in the chat there. RT Val is not on GitHub. It's on kernel.org. I guess I'll post somewhere the location later. There's a lot of great tools that have been being used recently for measuring real time type performance and other OS level things. One of them was StressNG. I also recently added StressNG as sort of a load to RT Val. It's not the way we normally run RT Val but if you want to sort of run any of the tests from StressNG and then have cyclic tests doing the measurement of how well those tests are doing, then you can use RT Val to do that. We use this with customers on embedded systems. We don't use RT Val for embedded systems. My company doesn't do much embedded but I'm very aware that how important the RT test suite and cyclic test is to embedded people. So I'm very careful not to bloat it and make sure that it will still work well for embedded folks. So what to do in the future here? Like I personally believe that cyclic test works well enough and you know we've been told before we could rewrite it and stuff but I've always needed to maintain this old version and we've never gotten around to it. It seems to work well enough but there's a huge bunch of things that we need to do in cyclic test in RT Val and other tools. Probably people need to do over and over. So it'd be nice to sort of extract this functionality and put it in libraries for other people to use. So I've got bugs in RT Val for doing things like parsing CPUs to see which CPUs are online and I found like crazy things like three implementations in the code and plus two more that were ad hoc. And then, so it's just ridiculous. Okay, so this cyclic test options to output JSON or XML, these things would all be nice. Also, I mean we're already doing stuff like this but the code doesn't always work very well with containers and partitions because this kind of code assumes that it has the whole machine. This simple example that's probably pretty fixable is a lot of cool code has to access the debug file system, which is usually hidden from a partition. But just the idea that we might have a certain area of the machine, which we want to use for real time applications and another area of the machine which doesn't care about that. So one of the things that the application the measurement software could improve on just brainstorming some more wild ideas like I kind of like to this is maybe vaporware until it's actually done, but I'd like to see some sort of work bench that could be a good time programmers that, you know, would tie in various software maybe in a nice gooey or web interface. Yeah, and like I said, I'm always open for patches and ideas and from the RT community. So, I've gone over time here I haven't answered all the questions yet. I can answer questions in chat. Well, we move on to the next fella. Okay, we have always this check here and I think it's more for practical know to answer the questions in the chat. Okay, thank you, John. Thank you. So now I passed the word to Eric with the using different RT and deterministic network. So it's our time. All right. So shit. All right. So talking about using Linux to do some fun stuff. This all started last year when I switched jobs and discovered to my happy happens dance that the room next door had the robotic arm. And as all other things that was a solution looking for a problem. Yeah. So I'm currently working at Sintef where research nonprofit foundation with a lot of people. I'm currently based in Trondheim in the division of reliable automation. My background is engineering cybernetics. I've been using Linux and real time for quite a while now working with it on a day to day basis since 2011. Did real time audio and Cisco for the telepresence codex. Currently since August last year, I've been at Sintef in Trondheim doing reliable automation. Where I mainly work on a satellite project a year project where we're going to make an on orbit satellite servicer basically one set by grabbing hold of another one and do refueling and repairs on satellites in orbit to do life extension. Sorry. And then I also do various industry projects at Sintef's mostly small projects where I just look at the real time problems look at traces, figure out what is wrong, or if it's good enough. And I also run a small TSN lab in Trondheim, which is also playing a role in this project. Before moving on to the talk itself, I would be good to just have a short introduction to AVB and TSN. I'm not sure how familiar people are with this. So I added some slides. I'll just go through them quickly. And you have them for reference for later. So AVB started as an audio, an audio protocol to extend audio video systems with microphones and speakers, mainly pro audio. And that was later extended to time sensitive networking where they decided they could use use this for other things than just AV. There are terms like bridges and stations, talkers and listeners basically just names for the actors in the network. And TSN and AVB talks about streams, which is basically a deterministic stream from one talker to one or many listeners. And through TSN you get guaranteed delivery and bounded latency on the network traffic. And this is important in real time systems. Steve talked a lot about that earlier. And this is no different in the networking world when it comes to TSN. You talk about stream reservation clauses, typically ANB, which where A is the highest priority, and it gives you a upper bound on latency through the network. And a clause A stream will give you a two millisecond latency guarantee. And all of this is coupled with PDP to get accurate timestamps. So there's a separate profile for PDP in AVB called the generalized PDP. And finally, all these AV endpoints were considered to be really small systems, so a microphone or speaker, you don't want to implement the entire IP stack. So for that reason, AVB and TSN is on the layer two in the network, so ether frames basically. So it's fairly low level. That's because of the history from AVB. And yeah, and then moving on to TSN because the moment you had deterministic networking for audio, people started putting everything except audio through those streams. So then they forked out TSN and added some more standards to get even better time guarantees. And even though they want to put this into robotics and industrial IoT and also fancy words, there's still no official tag for pro-control data in TSN yet. But you can use the experimental format if you want to roll your own streams, as I've done later in this talk. Alright, so what I do at work is I do satellite stuff and TSN, but there's other groups that do advanced robotics. And basically, if you can think of anything crazy you want to do with a robot, there are people working at it. And one of the problems they're currently working on is to use image computer vision to detect parts hanging on an inverting conveyor belt. So basically suspended underneath and then grab those parts with a robotic arm. That's actually a really hard problem because you first have to detect the part you want to grab. And then you have to find exactly where your robotic arm is, where the end effector is. And then you need to find the pendulum motion of the part because they are suspended underneath. And calculate where the part will be, where the end effector is, and then create a path to that place in time in the future. And then move the arm to that place and then grab hold of it and lift it off the belt. And this requires a significant amount of computational power. And most robotic controllers attached to robotic arms are not strong enough or powerful enough to run these algorithms. And it's not easy to include cameras and lidars and pattern or 3D point clouds, stuff like that to this as well. So it will be very useful if you could move the computational algorithms away from the robot itself. So the arm located in my office next door is a universal robotics UR10E, which is a 60 degrees of freedom arm. And it can lift 10 kilograms. And for those Americans, that will be about pi half stones. So you know how heavy that is. It's also what they call operator safe, meaning if it detects a very sharp increase in power to move the arm, it will stop. So typically if it tries to bash through your head or through a wall, it won't do that. It has a fairly nice user interface. It has a simulator, which is handy. And they also have a way to upload scripts from outside. So you can upload, you can write the script on a computer and it's really easy to upload it to the controller. And they also have this thing they call the real time data exchange, which is interesting. And what that is is basically a very simple protocol where you can define recipes for state information from the robot and input to the robot that you can send in both directions. So you can define an output recipe for, say, joint angles or temperatures or speed and velocities for all the motors. The downside is you can only have one output recipe. Input recipes, on the other hand, are a set of general registers you can write to. And for some reason you can have multiple input recipes from multiple endpoints. So a lot of people can talk to the robot at the same time. That being said, you can only have, you cannot have overlapping recipes, meaning two remotes come right to the same registers. And you need a script running on the robot to actually read those registers and use to something. You can't write that directly to the angular speed of the base joint, for instance. And all of this runs with two millisecond slots. So you have outgoing frames for robotic state every two milliseconds, which gives you a four millisecond round trip if you want to write something back. Or if you're unlucky and you miss the two millisecond slot, you have six milliseconds. And all of this runs the TCP protocol, which is a bit problematic, but we have a way around that as well. So setting up a recipe is a really, really naive setup. Basically, you just write in text the registers and values you want and you get a reply with the size of those registers and the recipe ID. And then the controller will start pumping data to you or expect you to send it data. So it's fairly easy to set up. So I've wrapped all of this in a library. I call it liburx. That handles uploading scripts. It handles setting up the RTDE streams. And it cross compiles to ARM64 as well, because that's what we're using on the satellite project. It's currently in Mozilla public license too, but I haven't been allowed to officially release it yet. We're waiting for some wrap up in the Eros project, but it will be handled soon. And then everything is nice and tidy and we can make a nice profit. That was the wrong way. Yeah, so as an example, this is the code you need to actually implement PID controller for the robots using liburx. So it's fairly tight. You don't need that much code to connect to the robot. Okay, so here's the idea of the scenario we wanted to do for the inverted pendulum approach. We wanted to take a control logic and path planner and connect it to the EUR controller over the network. And the problem with that is that you introduce a lot of latencies and uncertainties. So for instance in the network, you will have packet loss. TCP will give you lossless transmission, but it will do retransmit and that will result in latencies. You will also have queues on the switches that will introduce latencies or frame drops. And finally on the control box as Steven noted earlier, you will have scheduling latencies, you will have interrupt interference. You'll have to traverse the networking stack. And you will also have issues with the PCIe backplane. So our setup currently is basically two Intel boxes with some RAM with an M2 disk and dual I210. The Intel I210 card supports TSN and it has good support for that in the kernel. It's been there for three, four years now I think. And the network, we have a couple of catalysts and two small motors switches. And we have the universal robotics arm and also the simulator, which is useful when you develop the protocols. So to combat the issue with latency through the network, we added a proxy mode to LiveURX. Basically we just take the RTDE stream and we wrap that in a TSN stream. And we do that just to make sure that the traffic through the network does not interfere and causes TCP retransmits. And TSN should provide guarantees enough for you not to worry about losing frames. On the endpoints, on the controller and on the machine we attached to UR10, we run preempt RT to try to reduce the latencies and jitter as much as possible. And since this also runs with a very, very tight two millisecond period, we use the deadline scheduler that has proven to be, that has worked quite nice. And of course we put that in yet another library, which we called LiveTSM, which is just a tiny layer on top of LiveAVTP, which is AVENUS transport layer library for AVTP streams. So that will be dropped alongside LiveURX once, here it says finalized. And you have to configure the network card as well. You have support for this in the kernel. You just have to tell MQPrior, what priorities you want to use for which hardware queue. Now there's a link in the reference at the end to some documents describing how to do that. So now we have a slightly different setup. So basically we connect directly to the robot with one box to avoid interference and packet loss. And then we set up a TSN stream between one box and the controller and pipe that through the network with some catalysts that also supports TSN. So you have power preservation and queuing reservations and everything to avoid losing, sorry, losing frames along the way. So yeah, you minimize network jitter by having a direct link, use preempt RT and schedule deadline to avoid latencies through the networking stack and scheduling issues. Approve that with the schedule deadline even further. And then you define that as a TSN stream to traverse the network fast as well. So sorry, I've been talking fast I see. Yeah, preempt RT and TSN worked really well for this. It allows us to move the computation away from the robot either because the controller is not powerful enough or because it's in a hostile environment, which is also important in robotics. Another nice side effect of this is when you have PTP sync time between the machines, you get really accurate logs. So you can actually detect small variations a lot easier because you know the time on one machine is the same as the one on the other machine. And that's really useful. UR has a simulator that runs being delivered in a VM, which is really nice. You can mess around with it at home without actually breaking the wall in the office and also debug the networking interface. Since it's a VM, it's not exactly real time, so it runs at approximately 388 hertz on my machine compared to the 500 hertz. Some systems tuning is required, of course. And then configuring the catalyst. That's an interesting experience. Even though I worked in Cisco for seven and a half years, I never got around to actually learning to do catalyst config. And now that I have a catalyst and I wanted Cisco to help me configure them, I realized that's actually not that easy. And of course, why settle for a multi-coded line system with indirect priority inversions, SMIs and all the fun when you can throw some network on it as well. So yeah, why not. Yeah, and remaining steps, as I said in the beginning. IEEE 1722 is the IEEE standard that defines the AVTP stream format, which is basically the protocol for the TSM streams. And it has no format for control data. So I just grabbed experimental format for now. But this is something that needs to be fixed. And also the library doesn't really do signaling that much. So you have to configure both parts before you can get this to work. So it's not that user friendly at the moment. And I also need to really run my tests with some heavy crosslink traffic and look at the results. And also, you need to be careful with bash history because the universal, the robotic arm, you can just send it a script. Basically, you can just use netcats to send it a new script. So this script here will just put the arm in the straight upright position and you can send it wherever you want. And if you set the A value, the acceleration to a high value, the arm loads really fast, which I discovered because it nearly knocked me senseless at the lab one night. And one final thing, the GPDP profile, I promise you one microsecond accuracy. But if you actually try to get the time from the I210 on the test system, it actually takes 19 to 38 microseconds. Fairly distinct bands, but there are some issues with you are not being prioritized through the PCIe express back playing. So that's something I should look more into. But that's an interesting, I think I have a plot of that as well. So the top row is just reading clock monotonic. This is a really simple test, no tuning, nothing. So there's probably interrupts happening in the high spikes. The top line is clock monotonic and it averages on about 2200 cycles on the CPU. And the bottom one is reading the PTP time from the networking card. So it's basically 19 microseconds. And the standard deviation is about 2.4 microseconds, I think. You see there's two bands of this, basically something stopping the traffic on the bus. But yeah, and the rest of the resources that there will be. I'll drop the slides later on. So that was my talk. A bit ahead of schedule even. That's really good. Thank you. So people have questions for Henrik? Yeah, I'll just get this shot. Oh, hang on. Oh, that was a lot. So, yeah, so to elaborate on Steven's comments, I actually almost knocked myself senseless. The arm moved really fast and I looked up and suddenly it was an arm like this close to my head. So it was like that moved fast. Worst latency you can tolerate on the full cycle. So I actually don't know that you are controller is fairly. It is really helpful. So basically, I, when I do the control, I just set the angular velocity of each joint until I have the desired end position. And all those velocities will be translated into, I think they do an internal PID regulator on the torque on each joint. So they will make sure that I don't do anything stupid. And there also I wind down in the UR script views as well. So if we don't send an update to the speed, it will just gradually decrease the speed. So if suddenly the link goes down, the robot will just gradually stop. It won't reach the destination, but it won't rip itself apart either. So I don't know is the answer really. So what you want to make sure is that you don't have too much variance in the latency. That is far more dangerous, I think, than having a 10 millisecond constant latency versus a two millisecond constant latency. If you have somewhere between two and 10 doing a PID regulator for that is going to be really, really difficult because it will have on linearities and all sorts of crazy things. Are there any more questions for me? Did I miss some questions? Give me a shout out. I think I answered. Thanks again. It's fascinating to hear that. So thank you for pulling that together for us, definitely. Shaded. And if people like keep asking questions in the chat, I think maybe what we'll do now is maybe turn it over to a Q&A with Thomas. And so, Thomas, thank you for having me. Well, thank you. Thomas, if you want to unmute and... Oops, Thomas is asking questions. He's got one more for you. Why don't you just unmute Thomas and ask him. We can turn into discussion and others can ask questions too. I have already unmuted. Now we hear you. Do you want to just ask Henrik your question that you just typed in the chat? Oh yeah. Henrik, did you look at the IEEE standard 6802, which is TSN for industrial applications? Not in detail. There are so many standards and IEEE are so fond of writing long texts. So no, I haven't put it on my to-do list. Yeah, that one at least goes into the right direction for robotics and stuff like that. Oh, okay. Excellent. Thanks. Okay, so Thomas, do you have any comments to start us off with today before we get the questions typed in? I don't have comments, no. Okay, well then I've got a question or two lined up while we're waiting for others to sort of go. So for people who are just starting out right now, where do you see the best source of information? So for people who are trying to work on writing drivers to be effective when configured to run with real time? Where do you recommend people start to go these days? That's a really good question. I have no idea. So I mean, let me ask a question. What kind of drivers do they want to write? Just a regular device driver, which is a driver which is specific to a real time application usage? I'd say, why don't you answer for both? Because we're going to have both places out there. So a regular device driver is just write them so they are fitting into the kernel and do the right thing on their subsystem and they are good. But if you do the right stuff with just what everybody else does and what the maintainers tell you to do, then you're probably good. So there's not much you can screw up. So the typical things or let me disable interrupt here just because or let me disable preemption just because. I mean, that's usually not necessary in drivers. And if people use proper locking, then all of that is very well hidden behind. So it's not an issue. If you want to do really device drivers for real time usage, you should better know what you're doing. Yeah, but how did they learn by doing it? Kind of work on them as a part of a community, right? In terms of fun. It sounds like that's a gap that we should start to figure out how to fill this next year. I mean, yes. Documentation is always a problem. Okay. What sort of we sort of talk this with John about John already, but I'm just wondering, do you have any other additional recommendations about test infrastructure, or that people should be doing and testing regimes that they should follow when they're wanting to check their code is working properly before they submitted upstream. The long list in documentation what you should do with your code before submitting it upstream. Unfortunately, most people ignore that. Okay, but what about for the real time specific aspects to make sure that it won't cause issues. You like to scene run things like that. Right now, for most people who develop just drivers or some random feature in the kernel, real time testing is hard to do because you have to apply extra patches. You have to know what you're doing. But once the stuff is upstream, it's just going to happen for them. So the continuous integration systems will lack them over the head over time. And I mean, this depends because if it's something which is a esoteric hardware driver, which is not available on everybody's board. Yes, then it's going to be tough to be covered by continuous integration until unless you run your own, which you might do. Actually, once RT is better available in the mainline, I want to try to extend the testing recommendations to, hey, you should at least run once with an RT enabled kernel on that thing and see what explodes. Yeah, so that we keep it all running. Are there. So one of the question I've got is, you know, how can the wider community get involved in helping to fill the remaining gaps. Where are you looking for documentation documentation. The other stuff is working on the remaining gaps. I mean, it's getting really hard now because we're just looking at this easy future migrate disabled. We had it in RT for, I don't know how long everybody hated it. Everybody knew it was coming. And so it got ignored for a very long time. Now we have it. So now Linus wants to use it for something else, which we need also for RT to have some of the K-Map atomic of the high mem support working on RT, which I don't care about. But other people care about. And so it starts to get really into the thing. Either you know where to help or if you have to start to catch up games here, we are probably way ahead before you even figure out what to do. It's a little bit tricky now. Okay. The easy ports are gone. Okay, so it's documented. I mean, help John, help John getting print K done. Oh, yeah, that's going to be interesting. I really took only one and a half year to get the ring buffers merged. Well, in Turkey, using the word merge here obviously asks, okay, what's the status. And it sounds like there's not too much people can do to help at this point in time, other than potentially do some reviews and things like that. Right review testing. So what we're currently doing, basically, some of our engineers are working on various bits and pieces. And then we throw it over to Sebastian, who is trying to integrate it into RT, see what explodes, expose it to the people in form of an RT release. And then we shuffle gradually the stuff over to people in various areas, scheduler locking and other things in that range. Unfortunately, there's a contention problem because it ends up at the very same people having to review and deal with that. So, so reviewing stuff. Yes, it's always helpful testing stuff, telling us where it explodes where we have to fix things where we where we took the wrong turn. Yes, that's that's truly helpful. I think Daniel's got a question. I'm not seeing one. That's okay. There's this increasing demand of running Linux on safe critical systems and certainly the brand 30 is a building block for it. So, I would like to know what are your thoughts about using Linux on safety critical system and how do you think that we can make progress in this regard using the brand 30. Using Linux and safety critical system has been done 20 years ago. It's just a matter of how you design your safety critical system, what kind of system design decisions you take. If you have diverse systems with a simple voter, you can do it because then you turn the the operating system more or less into a black box and you don't have to care worry about it. Of course, people are trying to create silver bullets with preempt or G and auditing itself and trying to avoid all the extra hardware. Let's see where that goes. I mean, a lot of that stuff is wishful thinking right at the moment. I mean, there's some some interesting outcome of that as well because those people are actually looking not very, very carefully at all the compiler warnings which are suppressed by by our warning levels. So, they use more, more tools on it, which is a good thing. And so they they have ideas and also common interests with other people. So if you look at the security folks, they want to have unmapped data for their applications in order to so that nothing else can actually look at their secret source. That's something the safety people want as well because that's helps them to argue to argue better towards freedom of interference. So there's a lot of things which which or help through what they are doing, whether that's at the end going to be something where you have Linux certified at the certain safety integrity level or not, that's still an interesting question. And it's at the very end, even if you could find a way to to argue it, then it's, you still have to find hardware which actually fulfills the requirements, what the software depends on. Okay, so voting and as as Henry said on the chat and voting systems can get you a long way. Yeah, and you need libraries and you need, especially need hardware which is reliable. For the other presenters feel free to turn on your camera and ask questions right now, or if there's other questions in the chat. We'll turn this into a discussion for the last few minutes we've got if you'd like. Sure. Okay. So, let's see. We've got a question in from Ahmed saying isn't freedom of interference a harder problem to solve than with monolithic kernels. Of course, it is. Yeah. Freedom of interference is not the problem freedom freedom of interference is the really hard problem. And that's, that's one of the reasons why they want to do, for example, unmapping of data in the kernel space so so that the kernel doesn't have a direct map for for the data sections of the application because they can, they can argue that no accidental whatever stray pointer in the kernel is ever going to write over their user space data. It's still hard to argue, but it's it's it makes it easier. So there's a lot of stuff which you, which is really hard to to argue in a in a kernel as complex as the Linux kernel is so that's why most really certifiable system operating systems are very, very tiny. Okay. So anyone else here have some questions. I'm just wondering, for instance, for john, how are you keeping in sync with the test suites with what Thomas is, you know, what's emerging upstream and catching up upstream how's that happening. So I think one of the things that's really interesting is that the use of cyclic tests is actually pretty stable and people haven't been requesting any changes in it to keep up with any features like keeping up with a new kernel is actually done through different tests. Not seeing a chain not seeing too much changes need is what you're saying. Cyclic test relies on the user space interface and the user space API is not changing. So it, it would, it just detects when we screwed up with the latency is nothing else. So, and it pretty much covers would be covers what we need for a very long time. Okay. I guess Steve, once everything is sort of stream and we've got our stable back ports. Does this mean each of these stable kernels is pretty much going to, you know, just go and then eventually the task will go away. We wish. No, what we tend to do is obviously everything is long term stable is going to still be long term stable and doesn't have our T yet so we're going to be doing that till then, but eventually when the oldest long term stable happens to be real time. What we plan still doing is we have to be involved because we have to make sure things still back or back ported that are real time bugs. And, you know, we can't just expect the stable teams that are working right now to understand the real time requirements so there might be things are fine but we still have to monitor and say hey this, this caused the latency or this cause unbounded latency or unbounded private version it's been fixed that needs to be back ported and just slapping stable on it might not be good enough we have to make confirm it so we still plan on having the stable team monitoring to make sure that the long term stables and such still that's the tests that we run like RT eval and everything. So that still needs to work. Excellent. Well, if we don't have any more questions from the panelists, we're just about at the end of our time. So, so is this the year of real time let it be for next year don't want to do it on 2020 has to be next year. This is definitely been a year. Well, again, thank you every very much everyone and thank you Thomas. And with that, I think we will stop recording and wish you all a good weekend. Thank you very much. Thank you. Thank you. Bye bye. Bye bye.