 Okay. Hi. Hi. My name is Ilana. This is a joint presentation with Shua from the Linux Foundation. It arose out of work as part of Eliza, which is a project which we are involved in for enabling Linux in safety critical systems. And particularly recently we've been focusing on the potential of RT Linux and the power for safety critical systems rather than, as in the past, having to rely on proprietary RT OSs. So I will be giving the first part of this talk. Shua will be continuing for the second half. She's been doing work on actually testing on real workloads and she'll present her work as well. Okay. And what's important is this is very much an early work. It's work in progress, trying to investigate the potential and the limitations, the challenges of having to comply with both safety requirements and real-time constraints and how to do that in an effective and appropriate way for safety critical systems. About myself, I work at Mobileye and as a system software safety architect and this has implications for our work, but also in general for the work in Eliza to bring awareness of RT Linux to the safety critical community and to foster communication so that the potential can be met and the challenges can be dealt with. So, one second, let me see. Oops. Okay, so, okay, I hope it's... Okay, you see my screen moving ahead, the slides, yeah? Yeah, it's okay. Are you okay with... Okay, I did not put on the camera because I tend to have problems with bandwidth with the video, so I apologize for that, but I will be in Prague from Wednesday and anybody who wants to meet up to follow up on different aspects of this talk, reach out and we'll be glad to speak. Okay, so, on the first part, I will be providing a brief overview of some features of RT Linux, which I have been starting to look into, but with a new point of view with the focus of trying to understand how they can be used to support safety critical features and to deploy safety critical systems. Okay, and this is where, again, the feedback and the input is important because it's an analysis from a different angle and how, you know, we can best move ahead. Okay, so, this is the first part of the talk and what we're trying to achieve here is an awareness of the existing and potential RT Linux features which can be used to support the deployment of deployment in safety critical systems, generating the interest and communication and helping to actually develop and test such systems. Okay, what we are not dealing with here and that I want to make it very clear, I'm not a safety expert, I'm not dealing with safety certification, qualification or anything like that. I'm talking about actual Linux features which can be used to support safety requirements and real-time requirements. Okay, so from an engineering architecture point of view, what's not in scope, I think this should be clear to the people in this group that we'll be giving the same talk to on Friday there's a summit on safety critical software, Linux and safety, and I think it's important to understand the limitations. There's no silver bullets, there's no magic, nobody is going to drop in any features and expect that they will meet all potential safety or real-time requirements. It's obvious that everything will have to be specifically defined according to a specific use case and as always a special in any software deployment but much more so even for real-time systems, testing is always necessary to prove the compliance with the requirements and what we are trying to do is to raise awareness of the features and to find guidelines which point out potential usages but it is obvious that for any specific feature different options will be relevant for different use cases and it's always going to be the responsibility of the specific integrator, the specific architect, the specific user to demonstrate compliance or any given use case. Again, I think this is more obvious to the people who are involved in the actual development of RT Linux but it's something which we have to make clear to the general public and the general audience who will be using or maybe relating to this material. I'll go very briefly over some guidelines and the real scope here is what we would like to be doing is to go to deep dive into with time into the different features and to provide more specific guidelines. So in general there are high-level scheduling policy recommendations for RT user threads, schedule deadline as a specific use when you want to grant the highest user controllable priority in the system and then you can grant specific priorities and different levels for different threads or groups of threads and then the question is how do you decide when you are defining or architecting the system what particular priority you define to specific types of threads. For non-RT threads we use schedule other, the default policy but for RT threads if we want to be able to comply with some strict real-time and safety requirements how do we assign the different priorities and to take this into account in the architecture of the system meaning that we can group together those threads which should have again as well defined in the architecture have higher priorities than others. So these are the kind of considerations that we can take into account when we design a system which will use RT Linux and these are features which exist and which can help us to support our safety architecture. Memory management and again these are the general rules and these are the types of requirements which we are, this is ongoing work trying to understand how we evolve these into more specific requirements for safety critical systems. It's clear that memory allocations introduce latencies in general and it's if possible we try to limit allocations during initialization which is sort of axiomatic and to try to resolve symbols at system startup and we can also if again it's whenever relevant we can lock pages in memory after allocation. So these are features which RT Linux provides for us and can be used in appropriate ways and again we have to analyze the system which we are trying to use and we can see in which cases the features are most relevant and most supportive of what we are trying to develop the system architecture that we're trying to develop. Okay, resource management which is also a very fundamental architecture decision. We can define CPU sets, we can define use them to support separation architecture, we can use them to define access permissions and to isolate CPUs dedicated to sensitive RT workloads. In safety architecture there's especially in the automotive domain there's a basic concept of freedom from interference which basically means that we have to design the architecture so that software components which should not be interfering with each other we have some provable way of demonstrating that they will not and cannot interfere with each other and one of the tools which can be used one of the features which Linux provides which can be used to support that are CPU sets and in general C groups and again I'm raising here the same type of questions how we can use these features to define the safety architecture to support FFI. Okay, so going back to what I said in the introduction the real point of what I'm trying to bring here is to introduce that RT Linux comes with a rich, very very rich set of features which grew out of the underlying obvious Linux kernel and features which are specific to RT Linux and the goal here is to foster this type of communication to understand how RT Linux is powerful and can actually help us to develop strong and demos and provable or that we can verify that they are safe as well as that they meet safety requirements as well as real-time requirements for real systems. Okay, and to move away from the mindset which has always been in the past that for safety critical systems we inevitably must use RT OSs which at the end of the day will not have the same rich set of features that can be used to support more modern and complex systems which are developed for safety critical use. Okay, there are different features which are there are different configuration settings which can be enabled to basically features of the kernel they can be enabled to allow to give us some type of support for workloads for which there's a more strict requirement for jitter-free CPU execution. Okay, one of these features is by enabling RCU NoCB CPU which basically allows the kernel to offload the management of those callbacks on RCUs. Again, I'm assuming people here are more familiar in this audience are familiar with the features so I'm not going into the details of the features themselves but there's a balance here and the balance here is that we have this overhead and another potential feature is RCU Boost. RCU Boost allows us to boost priorities and again there's a potential benefit and there's a lot of material which can be looked into for seeing the different backgrounds. One of the things which and again Shua will be talking about her work is testing on actual workloads and again what we would like to come to is some kind of more specific guidelines generic guidelines for making such decisions and deciding if and when the features are relevant and what the benefit I get and what type of both real-time and safety requirements can be met by enabling the different features. Okay, there are a variety of features which are used for controlling the kernel timer ticks. We can, from all kinds of different possible options and which are again relevant for multiple a wide variety of different use cases. Okay, from one extreme is the periodic where we keep the tick running periodically to concentrate even if we don't need it and again this is usually not recommended for modern kernels but again there may be instances in cases where this is relevant and to the no-hc-full option in which stops the timer tick whenever possible. Okay, and again these are general recommendations and what we're trying to do is to take this further and break it down into more specific guidelines focusing primarily on again on safety critical systems. Okay, system tuning, the TSC from Intel-based superior systems comes highly recommended and highly marketed. It comes with its own portions and as a general rule it implies a certain dependency on Intel architecture and again the advantages versus the disadvantages have to be weighed, okay. And we can do use F trace to help to choose the specified clock source but in general ensuring that we have a reliable time source is an important decision and ensuring that the timer remains calibrated and accurate on any specific system. Okay, system tuning, power management, okay. There's recommendations, recommendations which I not from which were from different sources. I gave the sources, not all of these. Again, everybody has to see what's relevant in specific use cases, okay. Config CPU freak allows you to change the clock speed on the fly and to save power but it has its caveats, its warnings and in general power management for real time systems there's a recommendation that it should be disabled at boot but again that comes at a price and it may not be relevant in specific cases where power management is essential and cannot be disabled. So there are features which let you do that for example here for the Intel features but again you have to be careful when you test the system and before deploying the system that you have the necessary latency guarantees. Okay, as a general rule and I think the last bullet is more like a axiomatic almost that unused peripherals should in general be disabled because any overhead will unnecessary overhead will impact performance and again this has to be carefully investigated and looked into. Okay, as a general rule there are all kinds of built-in system safety nets which exist. Okay, if you want to remove any such anything on this slide and I think the next slide too if I remember correctly with caution with a huge warning sign that anything has to be well tested. Okay, you can disable RT throttling and instead you say we're going to take it on ourselves and rely on a watchdog to manage an unblocked starved processes but again that means you take that responsibility and you have to demonstrate that you can actually manage that effectively and well. Okay, you can also enable the RT runtime greed schedule feature and there's a new feature which I only became aware of when I saw Shua's work and I'll leave her a little bit to speak about it later in her work about deadline schedule and deadline service, work from Daniel who spoke before me. Okay, so there are many features which are possible which help you to either disable or to manage RT throttling and as a general comment at the end there are some instances where we just say let a process starve until the load has somewhat subsided and that may be legitimate even as a simple solution again, in specific use cases where it can be demonstrated that that's the most simple and effective solution. So again, Linux provides us with this rich set of options and the awareness can help any system architecture or any user to decide what is the most appropriate. Okay, other safety nets which again remove only with caution disabling soft lockup and other lockup detectors. Okay, that's not normally recommended for safety critical systems, but again, with caution if you can test sufficiently and demonstrate that there is no negative impact then, you know, that's legitimate. Hardware errors which are self corrected by hardware should be ignored and that should be some guarantee that we get from the hardware vendors and security mitigations. Again if you want to disable them, you can improve performance but that's only good for production systems where most commonly if you don't have high risk, high security risk external interfaces and as always after careful testing. Memory safety nets, I'm getting close to the end of my time to try to be quick here. Avoid memory over commit because it the kernel gets involved here to perhaps kill some process in order to meet the necessary allocation and it may not make the correct decision which process should be killed because it may not have sufficient understanding of the architecture and so it's something again but this is a general warning but again if it's used correctly it can be very very effective and helpful. Okay, you can set priorities to control which processes will get killed or should never get killed and it's always a good policy to define system reaction when out of memory rather than leaving it to system defaults or for the kernel to decide. Okay, interrupts. I'm going to go quickly over this. I'm not going to dwell on this because we don't have really time. Priority inversion is a very important issue and it's something which we should support in the best way possible and again it's something which Linux has various multiple features which can be able to help us to control again with no guarantees in a specific system and but those features can be used in a judicious way to help control and manage the system. Priority inversion debug tricks I gave a couple of hints and some references here but at the end of the day the real recommendation here is to test yes, hello. Somebody is saying something. Hello. I hear once in a while like something comes up as if somebody is trying to say something. Okay, sorry about that. And then there are a couple of recommendations for kernel configurations but I think I'll leave them for afterwards and again it's only a sampling. Everybody knows there's a long list of possible kernel configurations and recommendations with new ones being introduced all the time which again give us so much power and so much ability in RT Linux to do so much to support both real time and safety features that truly amazing and again we would like to see more and more of this deployment in real safety critical systems. Okay, so Shua I'll let you take it from here, Shua. Yes, Ilana, thank you so much. Would it be easier if I share? No, I'm going to stop sharing and you can share your room. Okay, thank you. I think that'll be easier for you. So I hope you can all see my screen now. Yes. Following up on Ilana's talk about what would be the guidelines for SAP critical system share what I have done in conjunction with Ilana's work. Okay. Okay, so I took my goal is to run a generic workload which I primarily used RT tests and RTLA to run experiments on development branch on the kernel.org. That's what I took and then I simply enabled preemption preempt RT kernel and disabled RT group scad I think I might have run into issues with one of the RT tests. Quickly just a run-on of my kernel configuration what is different from the vanilla kernel. Let's get into what I have done as a quick run-down of preempt RT versus vanilla kernels. I picked a set of commands to look at, cyclic test with Mlock all. So that's something we want to recommend to RT workloads, safety critical workloads that it's good to lock all the memory that's necessary and allocate and lock at the beginning and then a deadline test and then the idea is to run it on both preempt RT kernel and vanilla kernel and compare the results. Then I also ran a cyclic test duration one to get a feel for how it's doing and quickly that's a matter of gathering all the data at this point. These are my results from deadline test when I ran deadline test on preempt RT kernel. Some of the things that I paid attention to I would like to get feedback from this group on something else I can look at. One of the things I looked at is I failed to perform tasks within runtime that's the one I looked at in terms of because if you were to miss the deadline that would be of interest to forever close if we are missing the deadlines. I also paid attention to missed deadlines and over here to look at how many deadlines are missed and so on so that if that can be tuned. Let's keep going and I have done this on vanilla kernels as well if there is any discrepancy what I am expecting is that I would at this time I'm just comparing data in terms of what I'm getting not too getting not too to evaluate more than that because what I am trying to get to is what I can tell people to look at on the system as they are running the workload. Then sickly deadline test one hour I have run that and then both on vanilla kernel and regular kernel to see how things are looking. This is kind of gathering my baseline really. The second goal is doing all of this work is to come up with a process and a guidelines for system integrators that are looking to use preemptive kernels on their safety critical systems and to see how preemptive kernel would work together with their system as well as the workload they are using. The guidelines for that is the first thing is understanding your system. I have measuring latency in terms of I have used RTLA system. I'm not going to go into a lot of details about what I found on my system because that's not really important. I'm just using an indie desktop but that won't be the one that safety critical system integrators would use. I will create a system as something I do not know details about and then same thing with the workload. I do not know the details of the workload. Just giving guidelines on what system integrators can do and tools and methods they can use to gather data to understand the system. Guidelines on measuring operating system noise I have played with OS noise and OS LAT on my system and also hardware noise. Those are the guidelines saying take your system and measure your latency on your system, measure operating system noise on your system and measure hardware noise as well. Using the tools that we have in the kernel and also the tools that might be necessary. I primarily focused on RTLA and RT tests as a means to gather disinformation. And the next step is understanding the workload. Is the workload following guidelines in terms of allocating and locking memory ahead of time so that during runtime you don't run into allocation stalls and waits due to waiting for memory. If it is a hybrid workload that consists of RT and non-RT threads then how are you are the priorities the workload tuned well to work together in a RT and non-RT threads working together well. The priorities of individual threads work together. The other scenarios where RT threads wait for non-RT thread to do something then you might not be able to achieve the RT goals deadline goals you are looking to achieve. Another thing I'm recommending is that run continuous hours of operation tests on the workload to gather more information on how it's behaving over longer periods of time as opposed to a small snapshot of a short duration of time how is it behaving. So the second, while you are understanding the system and workload recommending that recording system stats before the workload starts to get a baseline on how workload the system is doing without workload running and then also while workload runs as well as after workload stops. Take a snapshot of all of the steps. So let's see what I'm recommending here to gather information is majority stats. Some of the ones are fragmentation and VM stats VM stats and general health interrupts. This is all coming from just looking at the system and what kernel is recording. And then to see key users to see who is occupying taking up the time and then get stats as well. And then looking at the different stats in terms of proc interrupts. I think I already said proc interrupts earlier. Pressure. What is the system pressures looking like? CPU, IO, memory and so on. Memory stats wise take the EXT frag and look at that and then allocation stats in terms of what to look at. What kind of stats are we looking at in terms of migration? Are we looking at migration failures? And then any of the if we can look at the stats we're seeing normal stalls, Alex stalls on devices and then compact stalls. If the compaction stalls are failures and then how many times compact daemon is and then any of the pages that are marked for migration are we seeing failures on actual migration happening? And then also look at MLock and our MLock to see how many pages are locked. Ideally if an RT workload wants to get a good timing, probably it would lock more pages, allocate and lock more pages. Any questions at this time or any okay, I'll keep going. Pay attention to what I'm asking people to do is system integrators to do is pay attention to memory fragmentation after workload runs and then also what kind of allocation stalls are they looking at compaction stalls. If they are seeing any then they might have to go to look at what's happening on their system and are there any page migration failures. Compare one thing to compare is migration success and failure stats. Looking at if there are if number of pages K compact de-isolated for migration are there any failures or anything to fail, anything failing there. Look for a page migrate fail of about less than 5% of compact isolated so that you are seeing there will be some failures but there shouldn't be you shouldn't see a huge spike of failures as your workload is running. Really the bottom line is that you should be seeing more successes than failures, fewer failures and more successes. And then how many times compact Damon woke up? It does wake up and you have to see it actually tells you if you see the number of count going up it tells you that compact Damon is healthy and if you are seeing the counts not incrementing that probably indication that maybe compact Damon is not healthy on your system. As I mentioned before what we are looking for with memory locked pages is we are looking for a substantial number of M lock pages, higher number depending on the workload size. Since system integrators would know their workload they would know relatively how much, how many pages they should potentially see. So kind of look at it and see it lines up M lock lines up and our M lock lines up with their expectations. And then it's a click deadline test with six cardinal compiles and then I quickly took a look at compact my stats, the stats that I have seen on the system. So you can kind of see that the page compact isolated is the range versus how many page successes and then yeah there are few failures. I'm comfortable with what I'm seeing considering I was running six cardinal compiles at that time running a cyclic deadline in parallel. So that's the kind of number you should be seeing, you should be seeing a lot more migrate successes than failures. Also look at sketch stats and do you see MCEs, a lot of machine checks and interrupts impacting workload performance? Keep an eye on that, that would be a hardware part of it you have to understand. Pressure, CPU pressure stats also look at CPUIO while you are running the workload and before of course and then after stopping align that with your workload resource usage. Does it that is in line with the workload resource usage you would expect? And then pay attention to the severity coverage to see if you can tune any of the actions that need to be taken when MCEs happen or any of the system severity conditions occur. Pay attention to deadline tests, using deadline tests is your workload missing too many deadlines. Tune the hardware based on suggested vendor suggestions hardware and firmware to see look for any suggestions they might make. To be able to use in a RT for the best RT performance, whatever that means for the hardware and firmware. And I think Ilana talked about this a little bit. Doing power management, turning off power management NMIs that always need to be looked at in conjunction with workload as well as platform, firmware, all of those things into conservation adjusting P states, all of these again to look at the overall system holistically including the workload running based on the expectations from your workload. That is primarily my core of the presentation in terms of what are the recommendations to system integrators to be able to gather So where I am coming from my assumption is that system integrators will know their platform hardware and firmware and they will know about their workload they are familiar with the workload the part that they are not familiar is the cardinal part what they can look at what kind of information can they look at from the cardinal individual, if they want to say hey is my workload for example missing any deadlines or they can gather that after running the workload but some of the stacks where can they get the stacks to gauge how the cardinal is working together with their workload so that is the primary objective So couple of tips I am leaving for the scheduling part of it this is general philosophy of scheduling, most everybody in the audience would know this I think that we already throttling fix in when already tasks 95% by default is allocated which means throttling could kick in and the premise being that a well behaved workload does not exceed 90% of the CPU time So and then I am just leaving couple of tips on what kind of deadline scheduler differences between different scheduling philosophies and schedulers If your RT workload you want to focus on deadline type scheduling as opposed to normal POSIX real time classes then you want to go with the deadline scheduler to play with on your system on the cardinal I also played with a little bit with the two deadline scheduling server works with that is happening, the patch that is happening in progress and then I jot down some information on what the work that is happening and then I think Daniel is doing this work in terms of delaying enabling deadlines on work, control the enablement using watchdog timer, I played with it a little bit, I plan to continue to play with that to see if I can develop any guidelines to look for any information I can ask we can ask the system integrators to look at, so what would be a good one to look at I am kind of following this work that is happening and I think that is the end of my slides and I think we are going to develop these are the references I have looked at some of the work I have done and figuring out what to gather, what kind of information to gather from the system and so on it would be great if you can give me feedback on this and if you can suggest any other ways to maybe gather information on the system for system integrators so that they can go and understand their system, any other stats that would be relevant to look at when they are tuning their workload or just understanding the workload how their workload is working together with their platform and the pre-emptory, thank you very much there are questions from the room, I will pass the microphone this is Tim Bird one of the stats you had I am a little confused on PROC key users I hadn't seen that before is that related to real time so the PROC key users tells you the user groups are users that might be taking up, who is running the users that are running so my thinking is that it will tell us if you have, sorry the place that I am on my idea is that if you were to look at that stack it could tell you, if you have an RT on your system, your workload has different user groups and you have set your RT groups and then non-RT users then you can tell looking at it, you can tell if the stack is in line with what you were expecting I guess the man page is off there because it appears to be related to users who have items in the kernels key rings, it appears to be a topical related thing okay let me when I looked at the stack last time I thought that could be useful it gives you information on my system it tells you based on user groups that's the information that tells me is the man page off a little? I can add, I'm sorry go ahead a question related to the RT throttling I'm not sure but is there is no recommendation to put there a minus one to disable all the other RT while there is no to solve it under the design, under a good realtime design you didn't suggest to put the minus one right? no should I be doing that? I think if you don't want to if you just want the realtime task to be scheduled right I didn't do that I wanted to I didn't want to just run realtime tasks I am assuming that the workload might have a mix would that be a right assumption to make? I think still if you are running also non-RT and RT and put the minus one maybe in good realtime design you have time for those non-RT tasks I will recommend that I like that recommendation thank you the RT throttling is more safe to measure than something that you use to tune your system even with the out server the fact is that Linux cannot run 100% of the time on realtime tasks because there is eventually a key worker that will jump on that CPU and if you stall it you can put your system down so disabling realtime throttling actually in a safety system if you hit the conditions for running throttling it is probably your workload that is broken and you should check that right if that didn't come across that is my intent they need to check that if you are getting close to the throttling so I will add that that didn't come across I think I can add that is that what you are saying that I should call that out that if your workload you are reaching throttling that means there is something wrong with the RT workload you need to give some space for eventual work that will work on the CPU we still don't have full isolation support but there is a question about RT it depends on your workload if you do something periodic like for example you do automatic lighting control on your video stream or control injector in auto engine or something like this if you do even minimal realtime throttling you will break your workload but if you do have periods where you don't care about realtime maybe it will work but again I don't know we do system where we control camera for every frame we try to put some 5% or 2% of non realtime it just break all the system so it's a work what I am hearing is that that is workload dependent so that's kind of what I am saying that you have to understand workload you have to tune the system for the workload I don't see anything in the chat there is one more question in the audience here absolutely great thank you I am Pawel Mahik from Dengs I would like to ask do you have some kind of guidelines where Linux should not be used like who is not comfortable with seeing Linux in their breaks no I don't I haven't focused on that I am looking at that is a good question to say where it should be used also we are not making recommendations on where it should be used either or where it should not be used either we are assuming that that is a decision that is made by the people that are interested in using it and for people that are interested in using it we are providing guidelines on how best they can analyze their workloads on their platforms with the preemptive Ilana would you like to address that is there any more you want to add to that no I would be very very hesitant to make any recommendations I do believe that we are only at the beginning of an investigation and I am always optimistic because we are seeing Linux being used in more areas than in the past but I think we are still seeing non-linux more proprietary classical RTS is being used in control systems in very areas where there is very specific very strong safety requirements mainly because of safety standards which need to be dealt with and as Shua also said we cannot make specific recommendations we can only try to broaden the usage and perhaps even broaden the awareness and even the acceptance by testing and additional features of RT Linux to see it being used in more and more areas but I don't think at this point anybody can really make any more specific recommendations than that so this is Tim again I just like to make an observation there are this last year I have been looking at the space industry and some of the standards they have are not exactly standards around things like specific things like realtime but they have process standards that Linux cannot possibly meet like the process for adding code to the kernel is just way too open compared to some of the process standards they have at least in the space industry I don't know about other industries and so that's a problem for us but my overall impression is technically you can get Linux to do what needs to be done but it's quite a bit more complicated than with an RTOS you don't have like all these extra layers you don't have like you know a bunch of config knobs you have to turn or to tune you have to understand the system just as well but there is a challenge there's a perception from other industries that the RTOS are just simpler to validate and that's I think again technically Linux can do the job but it's more complicated at least that's the impression that people have right I don't know one is more complicated than the other it being open comes with its benefits that there is no more more of a knowledge base to tell you what can be looked at and what you can benefit from and like you're saying there is a perception in your right in your opinion is that perception incorrect yes I am glad to hear that glad to hear that perception is incorrect I think sometimes people shy away from yeah I guess you heard me even though I didn't have the mic which is good yeah the perception is incorrect but you know things like in the space industry they're using radiation hard processors that are running like 250 kilohertz and it blows their mind when they get on something that's running gigahertz and they have so much processing power that they really don't even know what to do with it they can hit all their real-time deadlines no sweat even only 50% of the CPU but it's a total mind shift for them so anyway absolutely I'm with you I have been in the industries where in the past when I did RT work on proprietary operating systems you had verticals the entire vertical was controlled by one entity so there is that without being able to control the entire vertical probably the comfort factor as well thank you Tim that's great any other questions let me answer on that one let me answer on that one some more it massively depends what you want to do I mean if your job is just blinking a LED every 5 minutes 64 bit CPU with gigahertz it's a microcontroller it's good enough so anyone with the same mind would choose a microcontroller I mean there are enough insane people out in the world so some people use 64 bit machines to flip a bit but no seriously if you go whatever it is real-time system safety whatever you do you look at I need to get done and then you choose your components so that explains because a lot of you look in your car why would I put a 64 bit CPU into that thing because it's cheaper no it's not cheaper because you have to look at both the production cost and the engineering costs if your engineering costs are so high that you can save money then nobody with the same mind will go there I mean if you have high quantities and it can spear 5 cent on production costs but then this is going to be a lot of money for engineering but is it enough and what's the risk you're taking with that I mean that pulls into the money side as well because your insurance fees will go up into astronomical heights so there's seriously for safety critical things and also for real-time operations where you don't need to have the extended safety aspects you really look at it what do I need and what's my engineering effort to get it actually done but why are people looking into preemptive or safety critical systems because they want to do insane things on safety critical environments like autonomous driving with 500 sensors and cameras and whatever the hell and you can't freaking do it with a 16 bit microcontroller for which you get all the nice things including pre-certified operating systems and whatever but even for that case if you get the pre-certified OS from whatever the small RTS vendors for your safety critical system then you integrate it still into your product and you add your application on top and you do this overall safety assessment that port underneath you get from your OS vendor is just where you build up upon but that's it it's not guaranteeing that just because you use it your system will be safe and that's where we really have how we really have to look at it but coming back to Tim there are RTS systems up in space oh yeah that's true too but I know at least a large amount of RTS systems flying around up here and it frightens the hell out of me just adding on top of that when we think on safety critical system we are always or many times you are trying to think on the most critical component of my car like the ABS for example but I don't think that people in the automotive are targeting for those components they are targeting like Thomas said for the autonomous driving or on the top of safety they have other mechanisms to back them up and they are targeting a lower level of safety for example it's not AUD for automotive but AUB things that Linux can achieve with some work so some realistic we cannot try to focus on thinking Linux will never work on the ABS of my car that's not what most of the people are trying there they are not trying AUD people know that they want AUD but a lower level of safety like AUB where I can have a backup system to back me up but still I can use Linux with all this new cool stuff so when people think on Linux on automotive for safety critical they are almost never thinking on the components that already exist they are building a platform for the new components that will come that some components are bringing to the car but things 10 to 15 years ahead and that's why they are interested in Linux and most of these after 40s is being backed by companies that have an idea and have a need for Linux so that's what based their reasoning and their effort it's not for the space but it's for a big pile of software stuff that the drivers around Yelana you have your hand up on Zoom you have something to add? Actually I want to thank Thomas and Daniel that the world is not black and white and there are definitely certain areas where we won't have, we will have to focus on those let's call them legacy and standard RTOS which gives those high qualification standards and whatever but for the very, very tremendous range of applications in which Linux is being used I think you know Thomas and Daniel actually expressed it very, very well and that's where I'm trying to foster this interest to see how we can expand that use and expand that awareness and see which features are most relevant and grow on that, okay? And I think that's also why we are involved also in Friday's Safety Critical Software Summit and I will be in Prague from Wednesday very happy to follow up with anybody on these discussions because of the great need for Linux in the many, many areas of the software stack which are not as highly Safety Critical but definitely are still related to both safety and real-time requirements to see how we align and move ahead from there, okay? So thank you, Thomas and Daniel and I hope we can continue this communication and conversation and the work with Shua and to grow together and see more usage. Thank you everybody, this is great feedback and then we are presenting, Ilana and I are presenting this talk, the Critical Software Summit on Friday, June 30th and then we will take all this feedback that you have given us and put that into our talk when we present this. Just one last technical feedback so many of the options that I saw there like mem compaction and those things, we have disabled Red on the Red Hat Enterprise Linux for real-time and I think that you all can use the config option from the Red Hat for real-time as a solid starting point for the current configuration because we have been tuning that configuration for more than a decade now. Ah, good to know. And there's also a tool that is tuned that has profiles that it automatically sets your system with well-known knobs for example for low latency for network and there is one option which is real-time. And on that profile they are able to enable all the basic stuff like setting up the performance governor, removing the exit from idle latency and all the many of the bugs that are just easy to pick. So these two things, the kernel config file from Red Hat and the tune the profile for real-time and speed up your process because they are fixing all the basic stuff. And so RHRT config and then profiles that's the one that you are saying take a look at. I will. Thank you. I will talk to you offline Daniel to get any specific documents that I have linked. I did refer to one of the Red Hat document that goes over RHRT and I will connect with you offline to get this information. Thank you so much. Thank you. Thank you. That's good.