 Welcome to Fedora, the vehicle for Automotive Linux. We are Allison King and Steven Smugin. I'm Allison King. I'm a technical project manager within the Automotive program, which means I work with engineering teams, both within Red Hat and partner engagement teams to identify customer requirements, build upon them, and then ultimately shape the future of the in-vehicle operating system, all within the agile framework. And to put it in other terms, I'm a cat herder. My name is Steven Smugin. I'm a long-time Red Hat Fedora employee who's done many different things and different roles. Most of them get titles after I've completed them. So today we're going to take you through how we are building a lean and agile team to plan and execute on requirements for the Red Hat in-vehicle operating system. With Red Hat's already established presence in the upstream Linux and cloud communities, we want to take the time today to explain a little bit more about how we'll be establishing a presence in the Automotive community through the Automotive CentOS SIG, as well as other public-facing communities such as Fedora. What we will not be covering is since neither of us are lawyers or speculative fiction writers, though my status reports do say otherwise, anything we would say on legal questions would quickly veer into science fiction of how we would like the world to work versus how it would actually do so. We also will not be covering how you might be able to install Fedora on a car's computer. Most of the current computers that the Red Hat in-vehicle operating system is aimed at are still in late prototype development. As with many new systems, again, it is very speculative how this would work in reality. So just to catch everybody up on what Red Hat in-vehicle operating system is all about, it's really, it's no secret that cars are becoming more and more dependent on software and technology or that there's a push for electric vehicles. At the same time, many automobile manufacturers are starting to realize that these classical methods of proprietary operating systems just aren't scalable and they won't work for the long term. So the Red Hat in-vehicle operating system is being developed to enable and accelerate many of these current trends in the automotive space in ways that traditional proprietary systems just can't handle. So the goal is to move the automotive industry into a more scalable and nimble way of design. So our original development plans went something like this. We would work with various center sigs and center stream and mirror how rel was developed. However, we actually went through this painful journey. We realized it glossed over a lot of important parts that we had to go further back on. It's like watching a cooking show and having the cake pulled out of the oven, but you're not told how anything was put together or where the eggs came from, the leavening and spices, and a lot of our work still needed to grind flour, find chickens and grow a vanilla tree. So I'm going to use an image that Alexandra Fedorovic Alentus that she's done for her talks to give us a better idea of how this works. Packages are accumulated into Fedora repositories and then snapshot it into regular releases of Fedora ELN. That gets broken into a release at some point like Fedora 34 or into a branch like Centostream. The Centostream is then eventually cooked further and becomes a Red Hat Enterprise Linux internally after some testing and other things. We then take a subset of the Red Hat Enterprise Linux and we do further testing, documentation, and certification all during this whole timeframe we're working on this stuff, catching up with things. And we are also, in a way, this graph doesn't show, pulling things backwards from things. So this is really complex. How do we make this happen on top of all the customer requirements? We do that the same way you need a six foot diameter pie. We break it up into smaller bites. And I'm just realizing that we really must have been hungry when we were doing this presentation because we've made multiple references to food. I mean, starting before everything started and now it slides in. But, you know, got to do what you got to do. But we do handle development the same way that we would handle eating this six foot diameter pie. We break it up with so many unknowns and so much discovery. We need to break things up into manageable chunks each sprint that are capable of delivering an increment or in other terms a bite. And doing this means more changes to Fedora and CentOS and it means those changes come to you faster. So yes, that does mean we're using agile methodologies to deliver Red Hat in Vehicle OS to operating systems and the community. And as many of you have heard at Osium, agile is an iterative approach and it involves and empowers customers and self organizing teams to solve problems faster than traditional development styles. And since none of you came here for a lecture on agile, I did add a picture of a dog to keep your attention as I got through just one more slide because we did feel it was important to mention how we're developing because it impacts the speed and quality of the software changes flowing into Fedora. We have joint development work that really keeps the customer involved early and often and the teams that are working with the customers have the autonomy to discover and solve problems all with a continuous improvement and systems thinking mindset. So why do you still care now that the puppy's gone? In the classical format, each part of this layered cake you see is narrowly defined with large amounts of pre-planning and documentation built and written that can take years to complete without any results actually ever being to flippered or defined. Generally each tier in this layered cake has minimum interactions with other layers to improve or change things. With agile and Red Hat in Vehicle OS, these changes are moving in a faster fashion depending on the software parts. The communication of requests may skip levels going up and down. Instead of large waterfall solutions where you would need to buy a new car every couple of years to get a different application set, they will happen in steady iterations so that a 2027 car may still be getting software improvements up until 2037. So agile, Fedora, and automotive development, how the heck do all these things tie together? And it's not a bad riddle and don't worry, I'm not going to call on any of you to answer it. But we're here to answer that today and take you through our process. So as we mentioned earlier, we do, we involve customers early and often, and then we solution based on their needs and requirements that are determined in these joint interactions. And from here we frequently push changes into Fedora that are available for all of you guys to see and collaborate on. So buckle up and we'll take you through the process of how we handle this in the next few slides. So we'll start with the generic layout that will describe how we go from customer needs to Fedora. And it all starts with requirements. And just a reminder, this is based on agile and lean solutions. So that's why you're seeing the customer being involved in so many facets of what we're doing and talking about today. But we start with the requirements that are sourced through these customer interactions. And then we take the parts of the problem and work on them and interdependent teams. We have different problems and we break them all down into different stages. And some of these stages look something like this. Hmm, what about this? Oh, all right, that works. Let's make it better now. And okay, now we need to see if this works with other things. And as time goes on and these solutions are refined, teams feed these solutions into various upstreams like OS build, kernel, lib camera, et cetera. And these smaller solutions are merged into Fedora where they can be used by workstation or other groups needing similar functionality. And last but not least, this is the slide that you didn't ask for because it's not Fedora specific, but we did want to show it to you because this final stage shows how these solutions are then farmed up into the automotive stream to allow customer solutions to be worked on immediately. So now we're going to kind of go a quick overview of one problem we've had to deal with. One of the biggest and ongoing problems we're having to figure out solutions for is getting a camera to a screen within two seconds. It is a very complex issue currently being us worked on by Eric Curtin at Red Hat, which we're not going to be able to give a lot of detail on to the level he could, but it is a good example of how we're making things work all the way through. So we started with the problem, which is that various regulations in the EU and the US require that for a rear view camera to be operational within two seconds of the car being started from ground state. Currently in new cars, combustion vehicles, this is done through dedicated hardware solutions which bypass a lot of stuff so that it can just be done. But how does a manufacturer do this with a software defined vehicle where everything's being run through via central computers? Well, we had to go figure out a lot of different spikes to figure out what things were, how do we make the boot go faster? How do we get an image up in Plymouth? What kind of daemon do we use to show images from the rear view camera? What kind of camera is it? Where's the network? Once the spikes have been fully explored and merged upwards, we started upstreaming the solutions to various open source groups, which impacted the kernel, lib camera, jacuz, system D, and several other places that had improvements silently put in, not saying that they were from this. The changes are then worked through those upstreams down in the fedora, either during release or in raw hide to future releases. Some of the solutions are also back ported back in the center stream automotive deliverables. Those can then be evaluated, be included in RHEL or RIVUS over time. They're also delivered to customers for them to look at. So what else might be working on and landed on from fedora from the automotive SIG? We are looking at CAN bus utilities. CAN bus shows up in a lot of tooling. Your refrigerator and your washing machine probably have a CAN bus attachment in it that your technician would hook up to to look at things and find out why the washing machine is not working. Beyond CAN bus, we are looking at SAE J1939. This is more industrial equipment and would actually show up more into edge devices. But it is everything from giant chemical machines to dump trucks and 18 wheelers. HTTP over UDP libraries. Many car parts are actually talking to each other using HTTP over UDP. It's a faster protocol in some ways, but it doesn't work exactly the same way as HTTP does. Various tooling that we're finding from the Connected Vehicles Assistance Alliance. We are pulling things out and porting them in. DLT Daemon, VSUM I3P, things like that. And also various utilities from QT because most of automotive is C++ based and is QT based. So it is a lot of various things are being tied in and pulled in from there. So, what hardware did we do a lot of our initial development on? As everyone knows, there's a lot of things going on and we found that we had one solution. And we've gotten some choice quotes from developers to help explain what those things were. Hello, I'm Justin Curse and I found this hardware to be very slow. I eventually switched to buying a four-year-old used Android phone, put a container on it and boom! A week's worth and done it in an hour. I leaned over here. Most of the automobile manufacturers are looking to use boards with completely different processors. I mean, who would ever even think about using this? Well, Joe King couldn't make it his flight to Nest, so I'm filling in. We would not have been able to succeed on this hardware without the extra help and care we got from Peter Robinson and Pablo Greco. We thank them very much. Well, I definitely can't top the slide from our friends on the quotes that our friends provided on the slide above. And big thank you to Eileen Dover, Justin Case and Joe King for putting themselves on the line to say these great things. So after hearing all this, why the heck did we choose the Raspberry Pi 4 as our hardware development platform? Well, there's a few reasons. Aside from the chip shortage and the community support, we did find some other benefits to using the Raspberry Pi 4. So many of our partners have very strict security requirements, which do not allow them to use cloud AR64 and AWS or other places without years and years of audits. Others had hardware, which was slow enough that made the Pi seem fast. But all in all, the slowness of the Pi is something that allowed members of our team to focus on parts of the boot, which were actually too fast on virtual machines or snappier hardware. At the time we started working on this about two years ago, we found many of our partners were using out of date operating systems using tools aimed better at 32-bit embedded solutions versus a 64-bit workstation class hardware they were actually aiming to use. We worked in multiple stages to get their solutions working with rail and showed that it could be worked cleanly. The original two sets of partners we had then wanted to explore a more native rail solution and we quickly transitioned to using EL8 and OS Builder. However, we also began running into issues with the EL8 systems. It had been designed around 2019 and Fedora 29, which was much newer than the 2014 OS that the partners were using, if not the 2012 one. However, the various applications they were working on actually needed stuff that was not in EL8 at all yet. At the time, EL9 was still very early in development with ELN only being worked on in some stages. This is where we began really pulling a lot of stuff out of Fedora into quick deliverables in order to meet the customer expectations and then feeding back whatever problems we were running to back into the Fedora solutions. So what were we feeding back to Fedora and how are we doing it? We were working together in short sprints with these partners to create solutions to address their concerns and their needs. So we had worked together on various QT libraries and compilers and other tools were even recompiled for EL8. Work was also done to get ELN and then CentOS Stream 9 working on the test hardware that we had. And last but not least, we worked with the upstream and Fedora OS tree teams to get tooling fixes in place. So that was really the path forward with Fedora. And I know we've taken you through a lot of different Fedora examples today, but that really brings us to the end of what we wanted to discuss. But we wanted to share with you where else you might run into our team or some of the solutions that we talked about today. So as everybody knows, Red Hat is involved very heavily in the community and this slide just outlines some of the places that we are involved. And there's, honestly, there's plenty of ways to join us on the ride of a lifetime for anyone who's interested. All right, I think we're down to doing questions. I apologize for not answering them during hands, but I have the same problem. Matthew Miller mentioned in his talk, looking at the chat while I'm trying to do a talk means I scroll. And, well, sorry, scroll. All right, so we will now do some questions. I apologize for if people needed to do questions beforehand and stuff. I will try and get through things backwards as possible. So let me do one more thing. There we go. So Matthew Miller asked, so in this model some things might go from Fedora to Centosig but not to Centostream and current rel. Yes. In that some of these things are not. Some of the things that we are building as sample applications that are in Fedora, but will not be in the normal release of the OS. This allows a partner who may have their own version of the vSum3 IP that they've tested and dealt with that they're going to go and use that instead for but they need to have something that they can show with their partners in an open area that we would be building. So the QT stuff and the other little daemons will probably go there into a SIG, but they wouldn't show up in stream or rel. Other things like fixes to jacoot and kernel items may show up in the real time kernel or they may show up in other in stream and rel as we improve the overall experience of fast boot. I miss this one. I apologize. Ricardo. I just want to jump in real quick before you move on from that Steven. Just a shameless plug that we do have auto SD. So automotive stream distribution is the automotive equivalent to the CentOS stream distribution for rel. Yeah. So some of the automotive specific things will show up there as well. I cannot answer your question on this record. Grossman Nielsen. I'm not a BU person. I'm an enterprise person. So I do not know our business model and how it is done. It's we I think we are doing direct things because we an automobile developer dealer doesn't want to have a free OS. They're going to be supporting they are legally on the hook to support this car for 20 years in some cases. They need to make sure that whoever they're paying for the software is going to be around for 20 years. And that changes the whole dynamic of business models and what type of operating system they can put into it. Beyond beyond that I can't answer because I'm I'm not a BU person. And I think the functional safety certification aspect of the red hat and vehicle OS is something that adds a little more complexity to this. Yes, as well. Functional safety is a huge talk on its own. And I'm not an expert enough to even start on it other than the fact that every part of what we ship has to be go through a whole bunch of documentation and testing and certification by outside vendors, which is actually part of the reason why what it. There's only specific versions that will be certified and certifications do not transfer and all those kinds of things that we get back to that slide about legal questions. Okay, does this mean we'll see Raspberry Pi support incentives rail at some point or is it only going to be hidden in the corner of the center stream. I have no idea. Not my it's a view that's a view question. Sorry. I can't answer that. There is. We will support it in our group as we need to. But other those are other product lines and things that have other decisions to make, sort of like we're not supporting automotive on the power PC. And we're not, we're not going to port our stuff to be on a nest 390. Not going to happen. So, that's our decision. And unless you really want to have a, I mean, we probably do it on 18 wheeler if you had the, the, the wheeler contain the s 390 and the parts but it be hard. Will that aim for it toward a higher level of driver customization. Are we talking about Colonel drivers or car driver customization. Right to Colonel driver so I'm going to go with Colonel drivers for this. Kernels are. Oh, Huey Huey drivers. No. Huey's are owned by the manufacturer. The Huey is going to be that is an experience that the manufacturer is delivering as an add on thing that they want to have to be their stuff. We haven't done anything with that at the moment other than we'll drive your, your monitors for you. And how it looks and what is done in that is, you know, Ford is going to make it so that Ford looks like a Ford and GM looks like a GM. And we are going to look at doing things that will be. Let me see if I missed any questions up here. We, there is talk of doing a, what is it the red ribbon car. Yeah. In the auto SD, if there are people who would like to build a Huey for a car and work on something like that, that would be, that would be a place where that could be tested out and played with on a Raspberry Pi or something, but it's not, it's not our currently our current goals. Most of them are just getting, making sure the hardware works in a safe and defined fashion. To go off of a little bit more of what Steven said too, it's the UI is definitely not in our scope for what we're developing now but the red hat in vehicle operating system is creating, I think a more standard platform for the developers at these automotive companies to develop on so I think, although it's not one of our key goals or something we are necessarily doing, it's something that's going to be impacted. I think the developers will have a more standard platform to, to design different user interactions and user experiences on top of red hat and vehicle operating system. So it is, I think ultimately that's something we'll see as a byproduct of our product, but it's not something that's at the core of what we're, we're trying to provide as Steven said, functional safety and stability are the core of what we're trying to, to provide here. Well, it is coming up to the top of the hour. Are there any more questions. One more just came into chat. Okay. Well, we've mentioned quick, which was pretty surprising given the RFC is released a, and makes me curious about this part of the stack and quick pretty much equals to boring SSL. Red Hat crypto team wasn't planning on adding to that. So how's that going to be handled? Will we see relevant parts in Fedora? Well, it would be first worked on through Fedora. I mean, it's not something we're, we're being asked to show that it can be done. Most of the vendors are going to have their own mid level libraries that they're going to use at this time. They've already paid for them. They've got their own certifications and programs and all this that they're going to do with their applications. However, they're wanting to show that it can be done in a general sense. So the quick I mentioned, because it is a form of UDP, HTTP over UDP, it may end up being a different version that is being looked at in the end. It may be slow. I don't know. But anyway, I don't have a full answer to that. I can try and look it up and get back to you. And with that, people do need to leave for other meetings. We're going to say goodbye. Thank you very much. We hope you had a good day. No, not Bobby. Oh, God, poor Bobby. I can't watch. Bye bye. Have a good day. Thank you all. We appreciate your attendance and all the questions.