 So hello everyone I welcome you on the automotive panel discussion My name is Martin Perina and I'm working as a manager in the automotive part of our organization and Today I would like to Introduce you my colleagues who I'm working if so here we have like Pierre Yves Chibon He's principal software engineer and the product owner for the container on wheels team I have Rachel Sibley who is senior principal QE engineer and he's working as a QE automotive lead for the for the automotive effort and also we have Daniel Walsh senior distinguished engineer and the lead architect for the containers runtime platform in Red Hat So before we will get some kind of our question, of course we will give Talk to you, but at the beginning let me pass the microphone and my colleague will tell you what they are working on currently Good morning everyone, so I'm a Pierre Yves Chibon also known as Pingu more known as Pingu more normally I mean the as Martin say I'm the prep tenor for the containers on wheels teams Which also stands for the cow team which gives us the opportunity of doing a bunch of bad pen jokes It's been the team has been created about a year and a half and we still keep container moving and doing the move every week So yeah, we're pretty good with bad pun jokes The responsibility of the team is to do to look at everything that has to do with running containers in the cars What are the challenges? How do we how do we run them? How do we manage them? What are the challenges at you see to do that? So it's a it's a pretty interesting area and I ever talk about it at Chew I believe this afternoon in this room to talk about earter and we'll go into more details into that here Hi, good morning. I'm Rachel Sibley. I'm the QE technical lead for the in vehicle operating system and automotive I'm also the product owner for testing enablement team and our goal is to migrate existing rail tests That relate to the safety scope And run them in an automotive environment. So Yeah, I have a talk later on if you want to learn more about that in functional safety and what that means and It's at 415 today in this room Yeah, by the way, this microphone does nothing other than for the thing so everybody's thinking so we're gonna try to talk loud So my job nine months ago. I moved into the rivals team I consider myself my main job is to get containers running everywhere inside of Red Hat That's my so my focus and when it comes to automotive I'm looking at how we can use containers in the car how we use the technology to satisfy a lot of the Requirements for us to get approved as an operating system that can run inside of a moving vehicle Okay, so Dan so I will ask a question for you because like we have published several blocks in the redhead block and one of the Most let's say controversial or most visible thing was that we don't believe that the Kubernetes is good to be running On your car so done. Could you please explain? So we'll start out by saying I love Kubernetes I want to open shift to be successful and but Kubernetes has some key issues when we talk about putting The reason we bring up Kubernetes in a car is most of the automakers You know, that's sort of the first thing out of the mouth is we want to get into this cloud native world and you know this whole buy-in to Kubernetes and There are going to be multiple computers in a car So why not just use Kubernetes to move you know different containerized workloads around the vehicle? Well, we're gonna be talking By the way, I didn't mention I have container buff. I think at two o'clock this afternoon same time Yeah, we're competing so if I get more numbers on him. Yeah Mine is just gonna be general it you know similar to this just general questions about containers And I'll be showing I potentially show some cool stuff, but and then tomorrow at To what time to 2 o'clock 2 30. I'm doing to talk containers on wheels or containers in cars So it'll be a I'll show a lot of the technologies and talk about it But our her days is one of the key ones anyways Going back to Kubernetes in a car So quick when you drive in a car One of the key things in any type of moving vehicle is what they call functional safety so functional safety means basically we want to make sure that the You don't do anything to injure someone with with a moving vehicle So it's it's a somewhat similar to security in that you want to make sure that the system works correctly But in functional safety, we want to make sure for instance You might have an app that's let's say Applying the brakes and then you have another app that is running Netflix Well, you want to run make sure the Netflix app is not like dominating in such a way that the brake app cat fire function fire off in in the world of Kubernetes So basically when you want to have an app execute that app has to execute. Okay, so be to be functionally safe In Kubernetes, they they have the concept of eventual consistency, right? So you want to we eventually your app to come up and eventually the environment to be up and running So obviously that is not the same thing as the app has to be up and running so we can't use Kubernetes and orchestrate the other thing is trying to get to functional safety. You have to really look into the code really examine the code and Explain how the code always works on time when you get into multi-threaded applications that are written totally in something like Golang It becomes a lot more difficult. So Lastly Kubernetes applies a heavy workload. So in the Kubernetes environment, there's always a lot going on, right? This is a heavy heavy The kubelet Cryo, they're always doing stuff. So they're always using CPUs. They're always Performing stuff. So there's a lot of reasons not to use Kubernetes in the car and for our Low overhead Orchestration is her day, which is what he'll be talking about this afternoon And I'll be talking about a little bit in mind tomorrow Thanks, Dan. So as you mentioned the functional safety is one of the biggest challenge or the biggest Problem that we are trying to solve within the automotive effort. So Rachel, could you please talk about like, you know How do we handle and how do we test functional safety? So for testing the in-vehicle operating system we need to well functional safety or FUSA requires 100% requirements test coverage So that can be a bit challenging Especially the way we inherently do it with rail QE. We were not designing tests 100% functionally testing A an API for example. We take a lot of what's upstream. We rerun it So part of this is running the tests against our requirements, which are the APIs Running them identifying where our gaps we use code coverage analysis decov to assist us with that Developing new tests to ensure that we have that 100% coverage So a lot of what we're doing with testing is we want to leverage Existing tests from rail. We don't want to fork their tests or have duplication there So we're rerunning all of the existing rail tests and then adding new tests for any of the gaps that we're finding and a Lot of these of course are tailored around the safety scope And we're adapting them to an automotive environment because the tests weren't designed to run In an OS tree environment. They're you know designed to run against an RPN traditional compose So there's some tweaks or changes to the test to get them to run an OS tree and then migrating them to an updated test framework depending on where we're running them so Yeah for for testing. Yeah, it's definitely quite challenging to also establish the traceability We need to have requirements down to the test cases The executed runs the logs the failures every failure needs to be linked to an existing issue All of this needs to be traceable. We have to have the evidence. We have to have retention policies in place So we have a lot of work to do I Just want to point out. There's a key word that she mentioned multiple times there evidence So in to me functional safety means Eventually if a vehicle causes an accident or a machine causes an accident and you go ahead in front of a court of law You have to have evidence that you did everything to make the system as functionally safe as possible So that's you know, so that's really what we're trying to do is what this will be the first time Linux Operating system has ever achieved functionally safe to to describe Linux in terms of being as safe as possible Right doesn't mean accidents again. I'm not going to happen But it would eventually you might have to get in a court of law to prove that we did everything possible Or our partners did everything possible. So to prevent an accident No to follow up on that a little bit one of the one of the example I like to take is You know you certify an API and that API can be something as simple as you know open open a file and write content to it and The idea of the the functional safety is to make sure to guarantee that you know If you use that function what it will do is open a file and put content into it If there are very specific use cases, there are very specific conditions under which you know if I give it a 42k Buffer frame and I'm pointing to a very specific place in the file system in under these conditions The function does not behave as it should that's a problem. That's that function is no longer functionally safe So it's it's going a lot through the code examining it and ensuring that it behave the way it does that doesn't mean that the The way you can you can't use the phone the function to actually do something bad, but it means it means that you know It's the example I do is you have a gun you shoot yourself in the foot the gun has worked the way it's supposed to be You pull the trigger it fire the bullet if you the fact that you are aiming it at your foot is your Responsibility the gun they did what you did so the function to open a file and write content to it Does what it did if you use that function to mess with the Canon parameter that send that landed with a crash of the Compute unit that triggered the accident then you know the function is what it was supposed to do The fact that you used it wrong is your responsibility and a lot of foosers to do with this it's you know Where does the responsibility lay and can we track that risk that? responsibility and you know So it's it's a lot to go going through all the codes lots of doing through all the test cases And that's where the test cases are important and the traceability of them are very important because they are to prove They are the evidence that the function does what it's supposed to be doing and some functions are very easy Open a file write a file some functions sorting functions For example becomes a lot more a lot more challenging and there how do we end all exceptions? How do we and those DH cases and all of these are becoming back to retracial was saying testing is important there Thank you as you all said like you know the the functional safety is one of the core Problem that we are trying to solve and that also like comes with the where the Linux can move on right? It's kind of very well established on the computers and servers It's probably most probably also very well established in the edge devices now. We are trying to get the Linux to the cars, which is like, you know completely new level regarding this functional safety So Pierre do you think there are kind of other? Categories where we could do it or other areas when after we will get into the cars it This work will not be specific only to cars, but we can expand From discussions we've had recently there is definitely an interest in what we are doing There is the the entire automotive industry is curious of what we can do and what we can offer with it The idea about being able to update systems being able to have a life-cycle You know a single soft-trace tag that is Applicable that is maintainable across multiple generations of cars is something that is very appealing In the same way that you know, it's nice to be able to run well night on a different generation of servers And not having a specific version of an operating system for a specific version of a server So then the automotive industry is very much interesting in what we're doing, but they are not the only ones looking into us we have had this recently discussions with industries that are less regulated that automotive but that are still sufficiently critical that they actually rely on softwares and software and software stacks that are used in automotive industries But without necessarily the set of functional certificate one one example we've met recently is a mining company that operates, you know heavy machinery and They are not as critical from a functional safety perspective You know if it's a if the engine breaks in the mine It's probably going to be less dangerous than you know if you computer crash on the autopilot in the highway But it is still sufficiently important for them that they're actually looking at these kind of stacks But then the line. There are also other areas that are heavily regulated But that will have certifications similar to the one that we are working towards with automotive You can think like autonomous trains They'll have a different set of requirements But there will be a lot of overlaps with what we are looking for in the automotive industries Medical devices are not another area where there are very strong very very complex You know certification you can achieve but there is also a lot of overlap with what we do in automotive So there is Industries, you know if Linux manages to enters in the functional safety areas of automotive It is the footstep in the door to a number of other areas for earlier regulated that may and that may involve That may look at what's being done in automotive that may look at the The way the standards is evolving because there is also work that's been done at reddit to make to make the standards Evolve the standard has been written I don't know 23 30 years ago something like that and hasn't really evolved since and the you know The IT industry has changed tremendously since The core fundamental of how the standard was written is still applicable today But the way we do it and especially the way we do it in open source software where you know Every single open source software was started by a highly descriptive written requirement document That is highly implemented and covered in 100% test coverage It's like I'm sure every open source project in the room is Fulfilling all of these requirements No, really, you know you have highly described requirement documents and really person could test coverage So but no, but that doesn't mean your your project is not actually You know following the spirits of some of these documents even if you don't realize them today So it's one of the things that we are trying to also influence is the the ISO community the standards to make to be able to Make them amendable to other approach to software development that relies on you know crowd sourcing reviews highly available test test devices the test suites you know The links kernel is probably one of the most Tritonized piece of software the number of industries a number of people looking into it It's a very complex one But it's also one of the most reviewed piece of software and yet it does not today satisfy the the ISO Standards, but that doesn't mean this period what the ISO standard wants to do is incompatible with the way the kernel is being developed So there is work being done there and that's probably going to open other areas in the future to have these discussions again Okay, thanks, so any questions from you that we would like to ask No Okay, so so I will try to repeat the so the core question was like, you know If there are any other areas other than the functional safety, which we need to take care about I my kids used to play a video game called need for speed and so in in a vehicle there are Requirements like you turn the car on within two seconds These are legal requirements within two seconds of being has to happen and noise has to be made that tells you to put your seatbelt on If you put the car into reverse the reverse camera has to come up within two seconds So there's there's lots and lots of requirements So I think it was a talk yesterday on turning on the backup camera within as fast as possible the Linux kernel So imagine going from a cold stat You know the operating system not and being booted so that it makes a sound within two seconds So there's lots of stuff going on we in the in the pod man team. We spent probably about eight ten engineers to try to Optimize pod man for starting a container and when we started out We were about two seconds for on a really low-end Raspberry Pi So we wanted to make the machine as low-end as possible and we were able to get it up to about point three seconds So we six times faster starting now for human being if I hit pod man hit character turn like you know The container stats within one second It's not you don't even notice it, but When you're worried about Starting up what our partner and this is General Motors and they want us to go negative time So they want us to create a time machine. Yeah Other things the the user experience of The you know the central system in a car is a lot different from what we're used to in in a data center Dan mentioned the the boot time requirements, you know is you you start your car You don't want to have to wait multiple seconds for the system to be ready. You want to get in your car start the engine and drive Same thing for the application the starting the pod man. You know when you click the button, you don't want you don't want to to wait You know if you take a a cabinet is cluster Data center server that will take minutes to start You know if pod man's text a few seconds to start the application after that you don't really care You've been waiting five minutes for the server to reboot anyway So, you know a couple of second is not hot the the entire user expectations. There are challenges We are speaking about an edge device that we need to protect Physically because the user has access to the device physical access to the device We need to protect it from actually the user tempering the device but we also need to protect the software once it's running from How to say from Non-human triggered Random event that would curb the software corrupting the file basically if Again going back to functional safety. You're driving along and all of a sudden you got a corruption and on a desk These the system has to realize that so you can't you can't cause an application to miss miss Misrun right so if a file or an object on the file system gets corrupted We have to know so we were looking at like DM Verity and and different functions. So the big effort has been this I think all composed of s which is a Composable file system that we can actually apply some of the rules of DM Verity and and FS Verity On to the file system basically so the kernel will while you're reading a file the kernel will know that that file Corrupted and so then we can realize it and basically put the car into safety mode right pull the car over to the side of the road into the breakdown lane and Call on a US thing called triple A and get a tow truck out the other toe of the car There's also the concept of freedom from interference So we need to ensure if there's a cascading failure that's happening from the quality management side of things Which is non-safety aspect code if that's cascading into or affecting or Preventing a safety function from delivering its capability then that would be bad So we need to ensure that we for a testing perspective. We have freedom FFI test cases to ensure that There's isolation between the A's will be code safety code and the non safety code aspect to ensure. There's no interference there so the question So the question is what do we what happens if we get a kernel panic or a sec foot? That is the that is the that is one of the core element of the functional safety aspect is Which we're not supposed to be able to get a sec fault but That's that's one of the idea functional safety You ensure that the function be of the way it's supposed to be and you know the people using the functions are also due to Do functional safety on their application? So it's not only we the functional safety basically is a layered approach where we build on the we are we build the operating System which is built on top of a functionally saved C certified firmware and then functionally safety certified applications will be built on top of a functionally safety certified operating system Which means you can't have a functional safety certified Application that does not run on this functionally safety certified operating system So it's it's really every every layer. No step. It needs to be certified for the top level application to be So one of the thing is you you have to document all of this the in part of the functional safety is like okay The question it was what happens if there is a if there isn't a you know file system this permission error when you try to write a file the the functional safety Documents all the all the work that Rachel is doing and documenting in tests are comes we basically a pretty big user manual and we basically get to tell the OEMs, you know RTFM and Make sure that your codes is compliant with the way we describe how it works So we do not only are the documentation going to tell you how to use the code But also what what are the different exceptions that will be raised and all the code, you know What what will be triggered so that the code can live can handle such an exception? Yeah, so I mean So the if if something goes wrong in the car So the goal is for the car to know that something went wrong So it's again if you run out of disk space on the car or a point Yeah, actually segmentation fault happens then we have to relay that information up to You know the monitoring program that's running on to see what kind and the monitoring program will take action So a classic example will be you're in self-driving mode on the highway And it'll notify the driver of the car to take over that the human being has to take over I'm dropping out of self-driving mode because something went wrong So from an operating system point of view our job is to not only describe what's happens You know if you write code like this you could have a chance of running out of disk space Right, so that's us describing what functional safety, but if an app on the third node blows up System D is going to realize it right system D knows that the service went down System D will then tell her to the monitor the tool that we have our Kubernetes right our lightweight orchestrator will then tell her to will then send that a message to The main node that's monitoring the system that Hurt a on that system will then relay that to the monitoring app from the vendor So the car company to a software and then the car company software will take action like Notifying the human being you have to take over driving or tell the self-driving car to pull the car over into the breakdown line Right, so that all has to but that all has to be realized right. It's not You know just something happened and the car keeps going right so so Before we got to that question one of the things I wanted to to precise that they are there are basically four different levels of Functionally safety certifications you it's basically called a seal and it goes from a to D And D. He's the highest level of certifications And a is the lowest we are aiming for a seal B And it's important to know because things like as much as we like the break example It's actually a bad one because the brakes are not going to be running in an a still be environment They are going to be in a still D environment because these are critical systems and these kind of certifications are It's no longer an operating system. It's a microcontroller And those microcontroller eyes LD and they are very embedded So even if you know if we were to blow if our if the node in which we run would blow up the car Would still be it will remain in a state that it can be drive by you someone, you know all the as you'll be functionally safe certified Systems will keep on running so you still be able to steer the car You'll be able to drive the car you'll be able to you know move to the emergency lane pull break and safety ends Keep the safety of the the integrity of the of the passenger intact So we're we're not looking we like the brake system because it's something we can easily understand But it's actually not the best example But it is it's easy it makes it easy to to to grasp some of the challenges there And I'm sorry to come back to your question there was the does that I'm sorry, okay There was a I'm sorry Yeah, I mean the entire operating system is gonna be based on top of rel right we're not we're not building a brand-new operating system Yeah, the goal is to take rel and prove its functionally safe We are gonna be modifying slightly the Linux kernel. We're running in real-time kernel and We're modifying the init ID for quick boot up things like that But pretty much we have to fall back on you know red hat 30 years of experience in an operating system and building a rel and all the all the software, right? We're not building our own g lip see we're not building anything special You know for the most part this is rel so it's it's red red hat enterprise Linux For a car is what I would say Are we looking at other standards for certification? No, we're only looking at ISO to six to six to we're also considering a spice Which is has a lot of overlap with ISO to six to six to but just looking at different aspects of that as far as our quality management system, but So far just ISO to six to six to Maybe I will add to this that like you know We have a working group within the ISO who is trying to improve the current version of the standard to be more Like let's say aligned with the modern approaches that we are trying to do right So it's it will not be like in the past that when you want to have some kind of functionally safe operating system You will need to print out like bunch of books and the wilderness stamps But it will be much more based on the on the continuous certification to be able to to align with the latest changes Right. So so that's one of the part is that we have a people within the ISO standard who is working on improving The ISO standards to be more aligned with the best practices as we have now So the question is if you're targeting a CB what kind of functional safety software are we speaking about? the So infotainment the question about the infotainment is infotainment is not functionally safety certified It's called QM and that's quite team-managed and that's basically the you know the real city infotainment the GPS to some extent So what the stack that we are mostly looking into is the adas stack. So the advanced driving assistance systems basically autopilot being able to Sorry No, Netflix. No, no, no Yeah, it's another example that we like to use because it's easy to grasp But Netflix is not going to be running on top of you know, right at is going to run on Android Yes, so in yeah, it there'll be a separation as will be We actually support a ZLA which I'm not even sure what that is But a ZLA and a ZLB applications, but that could that could be your entire self-driving capabilities in the vehicle So it's it is there's a lot It's a matter of fact There's very little parts of the system there I usually see an A's will be just because it's so expensive to so it's a large framework QM is is Quality managed which basically means it doesn't have to live up to the standards functional safe Right if if your Netflix app or you're in when I'm really talking about here Is will be running Android inside of a VM If that if that VM fails then we just put it aside now What we have to guarantee is that the entire QM System will not interfere with the self-driving car. So the self-driving car gets all the priority over the No No with they would they would they want to combine Basically three or four computers total in the car and then have sensors talking to all the rest of the systems Traditionally It's like it's gonna be like your cell phone it's gonna get up. No That that's a There's lots of question that we're working back and forth with the vendors on that idea So they want to have over the over the air updates So the certain parts the operating system system itself will be updated maybe Twice a year so that the base operating system But the containers that are running inside of the operating system will be updated on using over the air updates So imagine you and my part point my thought would be you pack the car in the garage and it hooks on the Wi-Fi and downloads But the the automakers want to be able to update the applications as you're driving around now not Replacing the running applications. So imagine you're doing a podman pull as you're driving along the highway Okay, but not doing a podman restart. Okay to give you to make it make it as simple as possible So they want to have the same experience you have on a cell phone where they can update But they also want us to We just had something come up this week where they talk about how they can protect a USB They want to update the entire operating system via a USB So you take it to the dealership and again, I don't know how this is if this is US centric or not But you have to take the vehicle into the dealership and the dealership will reflash the operating system from a USB stick And they want to make sure the USB stick is You know certified somehow and so there's things like that we have to deal with as well So Let me know if I get a question correctly are there any other features that we are foreseeing in the future about What we run on top of all is that correct? So are there was a feature that are driven by the automotive use case that will be used outside of the automotive use case? I think composite fs work from Alex is definitely going to be interesting for the entire edge ecosystem Because the entire edge ecosystem is going to have the same so it doesn't have the safety element But he still has the security elements of the same requirement than the cars like it's an edge device That is somewhere far away from a data center and therefore the user has physical access to it And we need to protect the system from the user or you know, it's environment So there are definitely work on a couple of fs that are interesting it Quadlets is is something that landed in podman for two And that is that was directly driven by the automotive use case quad let is an is an easy way to To manage system this to manage to start container from system the services. It's basically if you if you do To proper to properly start container using system D You have to enter in the exact start line a command that is about three or four lines long depending on the widths of your terminal Podman makes these quadlets sorry makes these two lines in a text file and It automatically generates the correct way of invoking podman runs in your exact start on system D from a template That's the quad that file and that's something that makes it a lot easier to to run containers from system D one of the advantage of course is One of the other advantage of using that is if in the future we optimize podman To be you know to run in the system use case You know we do other optimizations for podman from to run container from system D You will automatically get these optimizations in quadlets and you don't have to remember to go and edit all your service files System the service file to add the new parameters or remove the old ones and so you you keep that in quadlets And you ultimately benefits from it. Yeah, I'm going to give it to her because There's a lot of functional safety Stuff that's funneling back into the rel test Yeah With functional safety, there's All of the tests that we're taking are derived from the requirements and having to take those and rerun them In in vehicle OS. So yeah, there's a lot of rigor depending on how complex the API is but essentially the APIs are Basically based on man page requirements and then QE has to go and then decompose the man page into Low-level requirements of taking something that's ambiguous and breaking it down into functional pieces So there's a big Yeah, and part of that is reviewing the man pages and ensuring that the man pages are doing what the implementation is supposed to be doing Yeah, but there's a lot of rigor a lot of rigor that's going on in the testing aspect with verifying these man pages Also, there's a lot of effort going on right now to get Android to work well inside of You know eventually you'll see it'd be easier to run Android inside of fedora for instance as a VM. So there's a lot Basically, yeah, there's I mean RIVOS is not a huge organization But there's there's about a hundred people in and RIVOS at this point and everything We're doing is feeding back into other parts of the vehicle. I mean have the operating system Okay, unfortunately, we are out of time. So thanks a lot for coming here. Thanks a lot for the great question And and if you have any and you have any additional questions Feel free to come to us and ask and we will let to answer you. Thanks a lot