 Today talk about reproducible multi-element system composition with Linux Xen and Zephyr and before I get into it I shortly would like to show the environment in which I'm working. So The project I'm in we call herself embedded IoT Linux at Bosch It's a lot of embedded topics and I just took some sample projects which we make use of and what we do with this project We see what is the best for different Bosch divisions And when we talk about people with people sometimes they say Bosch This is I have some home equipment or if something in my car, but there are many different business units in Bosch, which make use of Linux which creates systems based on Linux and that's basically where I'm Located in and where we try to foster collaboration between the different entities I'm called technical business development manager Which basically means I go out look for partners topics to collaborate with inside external So this is the main topic. I got to this position because I spent many years into the ELISA project got some traces into AGL and other projects and recently I also became a advisory board member of Linux Foundation Europe was shortly addressed by Jim this morning We have been my personal Linux history. Oh, this is something. I pressed the wrong shortcut just like this Where do we go? All right, I Started 2006 So I guess maybe some in the room here have longer history with Linux than I have my start was Actually with Ubuntu and we I was an exchange student and set up old or took old PCs From the university which were no longer use put Linux on it because was free available and maintainable also at this time and Gave it to extra exchange students who didn't have a laptop notebook at this point in time So I brought this in there and this was my resume really kickoff of Linux ignoring the Unix time before Regarding automotive. I'm doing this for more than 10 years. Our first product within Car multi-media area was on a 2.6 28 kernel I guess Actually, this is also the head unit which runs on it So there's really if you know what you're doing you can still get the root access for it and the shell to it There was a different time at these days But even if you have you cannot do anything because all the critical things are running on an RTOS Which sounds like a little more complex composition and for this let's now take a look into the ELISA work and the reproducible system architecture I Have a little bit of disclaimer in the beginning, which is important to understand I always start that well if you have done the previous session also was going about how system requirements work System properties and assessing whether a system is safe always requires to understand the system sufficiently This means it doesn't and if you are interested in Linux doesn't mean it ends with Linux If you're interested in Zafire or a hypervisor, it doesn't end with it You need to see the surroundings you need to bring it into a context and what we do in ELISA as we try to understand Linux within that system context I'm going to bring it together and how also Linux is used in the system and if you Reach this point you will see Linux consists of a lot of components and features And these need to be evaluated for safety are they relevant not can they interfere or that the next level of looking into it And when you look into it you're most likely identify gaps because Well originally Linux also of other open source of it was not written with safety in mind with the intention to use it in functional safety use cases and There's in the handout. There are some bullet points. Also. I will not mention them all here What I wanted to mention is that if you compare Linux to a traditional safety critical OS there are fundamental differences So if you think about a commercial art of safety certify You typically think about a full development process it has been qualified maybe to an ISO 26262 an IC61582 or whatever standard and Yeah, it brings hard real-time capabilities for Linux. We're much closer. There are the pre-empt RT patches getting in there You can see that the patch level is going down. So we are in a good stabilization phase for this most likely still Even then you don't get the hard real-time depending to whom you talk to you They say even who's an artist you don't get hard real-time. You only get it with bare metal, but Still this is one part but what you see is you see a large challenge upcoming So you hear about software defined vehicles software defined industry everything gets larger You have more computation power. So you get more and more elements in and for this Suddenly you see the Linux world Maybe from the one side and see okay, I have Talented engineers and their kernel maintainers discussed. I can bring it to almost every hardware which I want I will find board support drivers packages. I have a massive amount of infrastructure skilled engineers for free more or less because they bring it in and I don't have to pay for it and That makes it attractive to use it for systems, but you see on the other side this strong part about Here's my arches safety certify What will I do if you're coming from the safe artist side? You also see this challenge because you see oh, I see how I have multi cores multi-level caches I have variants of my devices and I need to do the development again and again and again This morning had a short discussion with us Oh, we make big business currently with porting GPU drivers to Real-time OS because there is an up-growing demands from customers to have it and they don't want to go to Linux And now we can port all the existing GPU drivers from Linux on artists. So this is one part in there and There are also Some limitations which we will never achieve was in the elisa project, which others may do so what we would say is When we work and all the tools we provide We will not be able to guarantee that the system which you create will be safe because you have your environment in which you operate and We can also not guarantee that the processes which were described are fully set up Also, the third item is maybe very important. We will not create an out-of-dream module because there's so much going on So many patches which are in you need to have a continuous certification pass You cannot just say here is my version. There is an urban legend I haven't found the reference so far someone said there was a company in UK I guess who did a qualification of the 2.4 kernel it was before there was a safety standards as such But they did this 2.4 kernel work They were close to sell the product and then there was a 2.6 kernel and nobody wanted to have the 2.4 kernel anymore So they couldn't make the business out of it because certification policy were taking too long and yeah so this means you have still a lot of Reliabilities and responsibilities obligation, which you need to follow and the elisa project just is a way to provide a path forward to Collaborates with peers and these peers are Set from different industries. We have a lot of automotive partners in there. We have strong driving aerospace With Intel and reddit also some more wider partners, which are acting in multiple industries We collaborate with the automotive grade linux I come through this also later and a little bit with the civil infrastructure platform These are partners in here and if you get so many people together you need to Come to a certain mission which you bring in there and say this is to what we subscribe and these are set of elements Processes tools, which are later on amenable to safety certification. So all what we are doing tries to bring this path forward and You soon figure out that you don't do it in a full group you will split up into several working groups and We have horizontals in there The horizontals basically will cover various use case areas So one is the safety architecture here people will look into the linux kernel analyze sub systems one of the first parts were for example looking into the watchdog subsystem, which we were using for Reference use case this has done the safety architect They also doing certain tools in there for visualization of dependencies and so on Then in the linux features that is more a work group which looks Like he would treat security. So if you look about security, there's C group namespaces macdac all these kind of features Nobody tells you how to really do this and how to combine them and how it guarantees to be secure But these are elements in there and we know there are people who are skilled and safety They are skilled in linux and they can Yeah, have benefits by getting search and search such features what this group currently does is they improve the work on the preemptive patch of the real-time capabilities of the linux kernel and Update documentation try to bring this forward Tools investigation code improvement. This is a group which will look on the processes Which are using like system theoretic process and all these also how we work inside elisa that we make a Traceability design work and so properly they also brought us closer with the system theoretic process analysis So STPA if you haven't heard about it's a very nice tool for complex systems Because I'm not covering it here, but there's a bunch of material from elisa also outside and it's Came originally from the MIT right then I Think I just tied the open source engine process for tools investigation code improvement This is one major element from this presentation together with the systems work group. So we were setting up a CI system and went into testing I'm bringing different use cases together Also, there were topics like fuzzy testing code checker and so on which were general so we have a set server for the setup and Yeah, that's about this this goes into verticals the vertical the newest one is the aerospace Aerospace is not that easy yet. So there's still a lot of work going on Getting use cases together see where it's in use We know that there's Linux and use in aerospace But some things are more confidential on the NDA and we now draw the direction in there then for automotive Yeah, we concentrate on the instrument cluster from a GL that's the image which you can just be It's the second use case. It's not bound to it We basically had something where we need to derive our requirements demands to the system because we cannot simply create everything out of context We need to have some Sumtion on where we work on the third group is on the medical devices This is an open APS use case, which they were analyzing. It is an open source based artificial Pancry system. This was developed based on scriptings on Raspberry Pi saying simplified work without Really being by a medical device manufacturers driven by open source And this was really nice because you could you're not bound was NDA is like typically for automotive or so you could really openly work together right and Then if you see this work groups, which brings us closer to the system composition, which we had They contribute to different elements and most of the work groups contribute in this kind of red area So it's directly on the Linux and assume that People work in this environment work on Linux and at some point of time they would say I'm done now I just looked into my Linux subsystem element and then I bring it into the world and the world There's not a natively running Linux anymore these days. You will see that there are container technologies involved There may be a cloud connection which I haven't drawn here, but at least even if you don't have this involved There's typically always a microcontroller. There could be interference of hardware There could be another operating system. It could be an artist could be an Android system with your own infotainment domain And especially an automotive more more things get centralized, right? There's tooling around this and So all these need to be covered also by a proper open source engineering process So that you say these are all going in hand You can re-identify things and to really bring this Into use into comparison then you have the use cases around it. So so it's a collaborative work and We were not doing it alone with Elisa. We also added other projects in there So the collaboration which we have is with the Xan project the fire project which is There in the beginning, but we also reached out to a GL I mentioned but also to the Sophie Part and arm initiative on the software-defined vehicle. They work on mixed criticality use cases. So they also have Artists involved visualization involvement similar also for software-defined vehicle of eclipse foundation touching it from different Perspectives, but on a similar level Last outreach was on yoke to football tooling the SPD X as predicts is there We figured out that we need also software bill of material for safety parts and we had a special interest group created Yeah, it's for completion. We also discussed with Leonardo and we all did this because say Yeah, if you have an apple and I have an apple and we both exchanged the apples and each of us has still one apple but if we exchange an idea and we have at the end two ideas and Yeah, what we learned in this discussion was it started basically a year back reproducing these systems That's problematic. I don't know who of you had listened to the Google Keynote this morning there was the topic about testing and that you need to automate your build and even if you are Automating your build someone using your component is not Able to build it and you need to do all the trials and I think that's where we are a year back it came like this that Stefano Stabilini went up with Demo in Austin what's the open source summit what he had is I said I'm presenting features since a long time But the people asked me how can I use these features and then he created a demo Session where it's a little bit like a tutorial recording saying you're doing this and that and this is the way how he Experienced features with then and in the setup he put exactly these architectural elements of Linux Linux so real-time Linux that firing son But he didn't care on how the Linux was generated He just said grab you a be okay to generate it or whatever image grab your root file system Great a criminal image get a sapphire because his focus was also on the Xan sub system You just wanted to bring things in if you're interested in this part is a very nice one It was also with a lot of so it's not enough to just look into the slides It makes sense to watch into the recording because he was going into the system. He was doing is on his PC with Q emu and That brings us to the next point a Product will run on real hardware and whenever you go somewhere and say well, this is my PC He runs a VM this may be nice for the server guys But if you go to embed it if you go to someone you want to touch something you want to have a single board computer Or whatever you just want to touch and feels this is where we went on and said okay let's look where we can go to and We did a hardware selection We put in a lot of efforts and We were close to Having a reproducible documented and so on This was prepared by our developer first and then we brought it forward. We put it into the tools group They set up to started to set up the CI and suddenly Reality hit us What we had is we got went to hardware the hardware was not available to everybody and For the hardware selection said what is the best hardware to choose? So we were looking for a hardware where there's sense support where there's the fire support where there's What Linux support is almost everywhere? Then we saw the real-time Linux which Stefan was doing, but we wanted to get some visualization in there So we had graphics interface with the instrument cluster and so on and we use tools brought things up Another critical part come there were proprietary graphics driver And this ended up that well whenever you have this hardware at hand this was an automotive hardware if you have it at hand You can get these drivers, but still you need to download them You need to sign an evaluation agreement parts of it may be under NDA and if you want to have it fully open Then it's hard to achieve so and this is say well we had to take a step back and Yeah, my original intention when I handed this in when I did the submission I thought I can present you the full setup of this hardware But it would be cheating because nobody here you can reproduce what I have presented there, but we came close So we added different operating systems. We had graphics running We had all this achieved to our proof and proof of concept at a local desk work But maybe we start with a little bit less in the beginning So what we see us less is if you look at kind of a production line We have a setup now where we use github for sources. There is a cold meter Eliza it's based from the automotive work group and it gets built regularly on a git lab si runner and This si runner also interacts with our tool server Which also does code checking for the testing and so on and when the system is built it went into an open QA Testing to make sure that whatever is built on a daily base Gets really checked as well. So you see this little icon there below I guess I've also Separate slide on this cluster comparison and what makes it I mean that's something which everybody of you most likely Aware of what you have done Maybe you know from other examples so setting this up is Nothing special maybe but where I see it little difference in what we are doing is that the things are tightly coupled. So basically The meter Eliza layer is a description and the description of the meter Eliza How you go step-by-step is a map to while the docker file is constructed the docker file is there to Create the docker image and it's not only that you can create a docker image out of it This exactly image from the docker file is the input also for the git lab build image So whenever you would change something in the docker file or the docker image because of Either an update in the system the environment this becomes visible by the build image so you see it directly the next day we build it once per night and Then we move on We use also as state cache for your to build because otherwise it would take you a long long time to build It's a hundred gigabytes or whatever and by this we're much faster. We don't trust in it So we also do Rebuilds of the full sources because you could say what could go wrong over the time It's like when you take a Dibby in image try to rebuild it from scratch rather than just doing the incremental packages approach you most likely hit something which you didn't expect and The good thing about this reproductional part is we have different people coming in into the group and they come from a different Perspective so there's maybe someone who said oh you generate this QM or image I put it on my PC because I want to do an analysis of a work or whatever directly Inside my synthesizer system, but I don't care about the previous steps While we also just recently have a new joiner said I said oh the easiest way for you would be to start with the docker file He said no no I start the hard way I start with the description from the beginning and I know that most likely something will be different to what I have in my Environment because that's just logical, but I will update then the description and let you know So you figure out that there was one yok-to-ling we didn't update it because they changed the structure of their web page and so on so this is the element in there and The open QA also became quite important. So the open QA testing grabs the generated QM or image Was to mention that it updated every day, but the link remains more or less the same But you would always just get the latest image and what happened on 7th of April more or less It was a second time that it hits us We had a build we had a booting system, but our modification Which is basically rendering a danger sign for if the warning as a warning sign into a cluster didn't come up anymore And what we did a longer analysis and then it was not properly put in the logs We saw that there was a client server certificate expired So while this is failed we raised a ticket and it was quite soon then also resolved of two days later The full setup Description of this there is a blog post on this slides are already uploaded So you can just grab this and this gets you then the full enablement of the items This also mentions one more element, which I would like to say as a benefit Last I was last week discussion. There was one of the engines that oh, I wanted to try out something since a longer time and This trial was just some new features in it, but I miss I have computation performance But I miss the hard disk space currently I have no space on my device to do another yachter build of the hundred gigabyte and said, oh, do you know that? You can simply just branch off create Pull request or just even you don't even need to pull because you just need to have a branch And then you go to the git lab change a line for the CI and it builds whole thing for you And you can just see even if the test is still pathing and this is like a low entry hurdle for those to say Oh, I don't bring the computation performance. I have something tried out bring it just forward so this is something which was really in there and Yeah, that tears the open QA testing that also come then with Boots lock so you can see how the system booted it you get a serial console output because we have an Interface on serial con command line interface. You also get a video recording of what has been done over the time that you Yeah, also can see if something went wrong, right so Overall This means we have a bunch of starting points So we can reproduce the image and the nice thing is this was the one part We have the second part and the system a complex architecture and this more complex architecture We had the first run which worked locally So now it's like bringing the things together even the CI was prepared for this multi setup until the point of now I create and flash an image and Yeah, so this is basically what we see I mentioned here also that part of the direct starting point Right so Then Now we say we start all over again our new hardware and I'm also happy if someone is here and say oh That's a bad choice go for another one We will go currently for the XU 102 That's signings hardware. It's Also a little bit in the ages. It's not the latest one, but it brings very close arm reference implementation to us This is a benefit for being more transferable to other architectures because if we would go for a proper Atari Let's say system MMU implementation or whatever then it's not as much wide usable as when you have a very close to reference implementation part what we see as a major drawback of this part is That one the GPU level the GPU is not as good as Like on more recent automotive hardware which is done for which would be much better for a cluster or Infotainment whatever where you want to handle it, but the nice thing about this hardware is currently that There is a successor version of it the Kria hardware setup and this is for AI applications for robotics and This gives a chance on a very low-cost one This XU is a very expensive hardware selects three three thousand dollar plus or so if you want to purchase it It's available but expensive and this will not scale on a community base later on So the community scale will be boost a lower cost hardware like an ultra 96 board In the other work group there was discussion on orange pie But we wanted to have also something where as a microcontroller there where we can all Send support or virtualization support and that's where we start with Another benefit of this is that this is available to also send maintainers and it can also Support Zephyr. It's very much closer to the demo of Stefan from the past and The main reason we didn't choose it was because it was no reference hardware from a GL and we wanted to have more graphics performance But we also see the movement towers telematics other systems where you typically don't need the graphics performance for Rendering things and by this we can move forward have a wider use case No Possibilities we can migrate the hardware with good PSP support the Korea's brand new But as I said, we are not fixed to it if we get a better approach if there's another hardware So we were looking like for an XP Ti apart What for us is important? It has an open GPU open source GPU driver that we don't fall into the same issue again we can discuss with other maintainers from open source GPU drivers and don't need to talk directly to GPU providers and Yeah, this gives a start on this hardware. I hope that we are good in presenting this in June So I give another try at the embedded open source. I'm at a proc this time from my colleague who is doing a lot of engineering work there and Yeah, we're just on this part What's comes also in there as some next steps? I guess as bomb is Something which is important and attractive to many so we do the s-bomb generation what we also were considering to switch on reproducible builds elements. I haven't listed this in here, but this was important for us then Yeah, there was the original reproduction. This is the ongoing activity There is an RT Linux cell in there, but we will go There and document this part so that you can really just go with step by step wise You can say I just start from source. I take a Docker file. I go to the git lab. I go. I just take binaries I just bring it on the hardware. I don't have a hardware So if you a more option, well, this is basically the step where we continue if you get a first want to get a first free check the work in progress pull request, but it's on a very Initial phase because it never reached something like we did before where an engineer starts a second engineer takes over the results improve the documentation someone both engineers look at it After a month, I really recommend to look at your documentation after months If you haven't done things then bring it over to someone new and this is basically how we approach the whole thing But we couldn't do we'll see proprietary graphics driver We will go for other Linux distros because although a bunch of bear with York to you may come with the Debian part So that's why a partisan CIP I'll just sit here as well. We did actually this already on the old demo So this worked also to replace the Domain the the Linux domain was another one and still see graphics and some this was on the other hardware We thought we believe it will work also on the new one and Yeah, the last important part is if we have it on this zoo one or two Which is a 300 dollar boards. Nobody will buy it if you don't have it in your company but we want to be open for other communities and Therefore we look for a community hardware now could actually be also a good time to just say Oh, I know the perfect hardware which has zafi xn or let's say our toss a virtualization and Linux support And it's low price and available then This would be also chance to just swap but for now, I guess we go with this one and This basically can also show you an evolutionary pause So you see that all the work which we're doing is not directly related to safety And the aliza project mainly covers the topic of enabling links and safety application and that's where it started but if you want to experience a use case you need to put it again in this wider system context and This is what we see here And even if you just have a smaller use case that's called telltales for this is these are the warning signs Within an instrument cluster This has safety implication because it's important that your check engine sign goes on that your gear indicators Properly if the oil pressure is not correct as these are all warnings Which need to be there need to be shown and if they wouldn't pop up you have an issue and this can become Life critical in the end. Although a bunch of us may ignore the first appearance of this measure message And if you think one step further what happens in there is that? You could extend the use case to go for park distance control for example, then you don't have Maybe your engine which records put on this light But you will have an ultrasonic sensor which sends a signal and say make the beep and from the beep Then you can add the camera you get more real-time capabilities and we get a scaling use case and The funny thing about it is even if you start with this first small use case. This will already run in systems where Suddenly virtualization comes into place container technology come into place artist comes into place because there's almost no Automotive ECU which runs the microprocessor, which will not have microcontroller next to it which Interoperate and we see these use cases also from other topics like a lawnmowers industrial robots and We just did have more and more computation performance is needed Over time and this brings the composition of complex systems and you need to face these Complex system architecture need to reproduce this right You can get involved from many different points within the project So we have a bunch of mailing list almost with every part from the mailing list You will also find a working group meeting some are us-friendly like the systems work group then if you Being in Asia the automotive part could be more interesting and it's good overlap of 50% So that you're updated with both sides and you can also contribute to both right here by this I would conclude this presentation. I Want to say don't miss the next session. So after lunch, there's the session from Stefano Stablinian zento Kumar. They Talk about the safety certifying of Xen I guess it's also a follow-up of our last year's discussions and We're looking forward to this by this. I guess we can go over to questions Anyone brings a question? You get it You're assuming that the yacht of you just completely reproducible, but it's not so yeah It's in a good state I mean if you see what you can do you cannot the reproducible builds project is also not fully completed in there But we are much closer to it. There are just a few things and it doesn't need to be the yacht of part in there and also Mainly or what could be a large issue if we get it fully reproducible This gives us a much better argumentation for the tools we have in use and you have a harder argumentation as long as you build once And you build a second time. It's not exactly the same But at least we're much closer on this and it's also not about making here a safety certification Was in the Lisa but to help others who work with linux and they may not go with yacht Or they may go with others or they have their way around it or they are much further than the reproducible But we don't see anything Katie wants to add something Yeah, I'd actually be interested in having you talk chat with Richard Purdy because actually yachto is fully reproducible To the extent that they at least that's what Richard's been telling me and so To the extent that they even factor the time sense and they're using a whole estate System for making some of this stuff possible. So there's a lot of work that's gone into it So I'm curious why you say it's not fully reproducible And so but I'm not the expert to have that discussion with as well I recognize that so I think Understanding why are you seeing a limitation? I think we have good follow-up discussion It's basically a simple concept that's in if you actually using exactly the same computer exactly the same distribution to build the base of yachto Then the same compiler then you can get the resulting parts then you can say that's reproducible But if you get the same layers and you build in a completely different distribution You'll not get exactly the same result because the compiler would change the definition So it's partially reproducible if you keep the same thing This is the same issue that we have for the safety qualification if you change it left or right is not the same anymore That's also quite funny to mention because I Can keep it on so what I saw was that when you take the a gl yachto also It's downloading the version of the compiler to the version which the compiler version also fully matches So there are a lot of these elements in there which try to make it better If and what you all said if you exchange it to somewhere else but I'm pretty sure when you create something for your safety certification will do also in your environment as an integrator and this makes your selling point and Then it's not also not intended that someone There is the obligations from licenses which you may have to a certain extent But it's not intended that someone builds exactly the safety product again on it Yeah This was bright It was a very good talk. Thank you Clearly knowing stuff in in aerospace When we want reproducibility and stuff like that We actually have to qualify the tool by a DO 3 3 0 Normally, we don't do that for like Dell D&E will do we just we don't care That doesn't even we don't even care about reproducibility actually we just care that functional test passes And then for CB and a we just have to do All the way up to MCDC coverage and those things so reproducibility in that situation is really Not about reproducing the binary and even even with the binary one threat's gonna finish fast to the end of your root of fast You can't just check something. It won't work that way. No matter what there's no such thing as true Shaw 256 summary produce ability that will never happen unless you have you know everything marked up But at least you can say all my tests pass and I all my requirements pass So I guess that's a long-winded way of saying in like two six two six two and all that is it the same way Where you just have to show that your tests pass or do you actually have to go further back and show that your environment is Compliant Yeah, correct, it's a it's a it's a functional test in the integrated environment So does that also apply in automotive? Yeah, so for the automotive part if you you typically have the tool qualification And you will do a tool qualification and then the topics like reproducible build projects and other will be very useful Because this is actually where salt get modified and the argumentation gets incredible hard if you would like to say well, you know I have here something which is Certifiable, but every binary will look different Right. Yeah, that's that's not possible I heard once a nice story where for railways where they had to construct something and they were very close and they thought Figured out that they had to they had to skip the drawings from 13 things and they said okay How do we get it so they sold out the different pieces together and then they put it into a box and said okay When someone sees what we've just sold at this will not be good So they and they need to be robust in the railways so they put in Some epox seeds whatever in there and then they put a cover around it The best thing about it was in the certification phase It's X-rayed this and now later on every model had to be x-rayed And having there is a number was quite low But the x-ray picture had to look in the same way because has to be reproducible from the hardware setup And it was some just some last-minute fix which was equivalent But it's very hard to argue that this is equivalent if it just looks a little different right even if the circuit It's the same and that's what we have with the software bulls then as well I want to add one point for the octopi which are Some people say I go with Davion because there the wrist is Much lower when you have something set up and you will just build your component on top because you will go from a binary base Because never underestimate what a developer could do what a system integrator could later on do and Yeah, but then even then this doesn't avoid to have maybe binary patches in there But this will be then you have a certified one and you can see that something change and the changes are properly. Yeah Okay, another question Thank you So I kind of started to because you know first time to here, um, Eliza and very in a very good very impressive and it's kind of Ready to our our company that you know, it's kind of safety critical But in on top of safety, we also kind of you know Needs need to be real time. So I'm wondering yeah, you know, yeah So there's something when we started the use cases We don't have the real-time Part in there from the automotive because if we start when you start to your spec alter the RT patches in the Linux kernel one It depends on your use case how Much real time real time you need right if you say I need a one microsecond guarantee it can be also Potentially better to also have the system architecture where Linux gets some product and where you add other components because you cannot give a 100% guarantee under below microsecond with the Linux kernel As far as I know, I might be someone else things but What we did already we are currently improving the documentation part of it one of the work groups really looks into the RT part We will most likely also evolve on use cases and I guess Are the real-time patches enabled in the open APS? Because I saw some art no, I'm cause I thought I saw some RT patches And But basically means depending on your use case what you bring in may just see how much real-time real-time you need So I was talking to people also an artist is not a real-time real-time just it there when you do bare metal programming and But I see other industries which said from oh finally we can't use Linux and get rid of our alternative solution because Real-time is enables in Linux and it depends then on your product because they came from industrial Tracks, and then they are coming a hundred microseconds millisecond range and say but we need to guarantee this frame and suddenly There is real-time enough for their production steps. So this is Is earning other open source? No platform or a community that is kind of supporting This, you know, like hard root time You know how the real-time definitely with the xanthorx in there later on for the virtualization topic and the Arthos The sapphire the sapphire project is also in and both that's what why we collaborate is both have a safety track a safety certification Passed they're on the way that all of us all this three project not always take exactly the same approach Troubles safety, but we have a lot of overlap. So I recommend the sapphire project as an Arthos which has a Huge board support and it brings you the pass towards also linux usage because you have Kconfig in there you have device trees and since you will if you know a little bit about Linux You will get much more Confident with the work and fast ramp up in that fire as our way around if you start with the sapphire artists It is you the way Towards Linux and it's because many mechanisms are in there and they have a safety part They are Arthos, but it may end up with certain complexity If you have a very complex product then you would say, okay I may go into certain rendering or a certain workload parts. I put on my linux system I can argue that they are just quality managed means they have no safety in strategy level Are you can also argue with the Admitation say this is not the RT critical part. I have it in there and I use watchdog I would monitoring that's how we do from our use case We cannot guarantee the scheduling on real time as we have now because therefore we put a watchdog which Triggers and say, okay. I figured out scheduling was not achieved and Therefore I can go in safe state But this also depends on a use case of course if you're allowed to go into safe state if you have a drone maybe for example and Then just switching off the drone may not be good because it can harm people anyway It's just while for other example with the instrument cluster you can switch it off turn it black reboot And there will be not really a harm or anything happening So this depends really on use case we can discuss and see how far we can get where we can help Then I guess it's close for lunchtime Thanks a lot