 Welcome to another different session. I hope you are enjoying it so far. So I would like to welcome our next speaker, Eric Curtin, software engineer at Red Hat in Automotive, who will talk about quick booting of software defined rear view camera. There will be a few minutes at the end of the session dedicated to our questions. And without any further ado, I give you the floor. Hi, everyone. It's a mic working. It's on you for the streamers. OK. Yeah, hi, everyone. I already got introduced. It's my first time at DevConf. The venue is very nice, I must say, and thank you for attending the talk. I think it's about 28 slides, which is probably a lot. And I'm going to go through them rather quickly. But if there's anything that I was, I'll speak louder. The microphone is for the streamers on YouTube. I'll speak louder. Thanks for letting me know. So yeah, 28 slides. I'm going to try and go through them rather quickly because I want to leave time for questions at the end. So if anything I said was really unclear or people weren't sure about, feel free. Any questions as possible? So I was already introduced. I'm Eric Curtin. I'm a software engineer at Red Hat. I work in the automotive org on all kinds of different things, I guess. So the scope of this talk, the scope of optimization can be quite large. But for this talk, we're going to focus on optimization techniques from Linux, kernel, and in-it-RD boot onwards. So for example, something that isn't covered is anything before that because firmware, et cetera, it's generally owned by the automotive board vendor. So the optimization of this area is out of the hands of an OS vendor, such as us. And obviously, the firmware optimization is just as important. It's just something that's not my responsibility. So I do not go through optimization of the camera stack at the kernel level either because we use the USB camera for this, rather than an actual ISP SOC from a real automotive board. Although the same optimizations here would apply to an actual automotive board. Another thing I want to cover is there is a kind of a balance between keeping an OS generic and making it optimized to run some specific things early. So you could trim down a kernel and user space to only display camera frames and do nothing else, but it wouldn't be a very useful automotive OS then. So yeah, I kind of split the talk in two. One is just generically starting an early boot system of the service. And then the second half, I'll describe how we applied that to a camera application. And yeah, the techniques in this talk can be used to start other services early. In automotive in particular, there's an expectation to start all sorts of services kind of early. Our rear view camera is just one of these. And so here, I did a little ASCII diagram of how kind of a Linux OS is booted from a certain perspective. So generally, the firmware loads into some sort of boot loader. And normally, that's where the compression of the kernel and init RAM disk occurs, even though that's not always the case, but normally. And then the first file system mounted is the init RAM FS. And that loads system D and does a bunch of things. And then at some point, we changed the root file system from the init RAM FS to the normal root file system. So you have two options at this point. If your system D service, if it doesn't need to start super fast, like it needs to start after three or four or five seconds, maybe starting in the root FS is just fine. But if you want to start something super early, like in the case of a camera, you have to start it within the inner RAM FS, sometimes called init RD as well. They mean the same thing pretty much. So the reason why there's two options at this point is there's actually kind of a trade-off here. So when you start a service in the init RAM FS, if that requires you to add more binaries, libraries, et cetera, into the init RAM FS, it actually has a negative effect on the overall boot time. Because, but anyway, I'll go through that in a while. But we're going to use that technique for this talk, because in the case of a camera, you have to kind of start it super fast. These are some generic tips just to boot things quickly. They're probably obvious, but sometimes they aren't. So keep your kernel as small as possible and as modularized as possible. So anything that can be a kernel module probably should be, and keep the init RAM FS small. But the kernel and init RAM FS must be small, as I said earlier, because you actually pay a penalty for every single byte, because every single byte has to be decompressed before you can use it. So they're just generic tips. So if you want to start a service really early under a system D-based operating system like all the Red Hat operating systems, this is what it looks like. The one line that's probably important is that default dependencies equals no, because system D services have a default set of dependencies. And the default for that is yes. And if you have that set, it's yes. Your system D service won't start until a couple of seconds because it must start all the default dependencies first. So in this case, we don't want that. This is just a Draco file used to pack things into an init RAM disk. And it just says, put this file in the sim link in the init RAM disk. And if you do that and run a service like this, it'll actually start within about half a second. This is just how you rebuild an init RAM disk. So that technique is useful to know, because using system D and using that technique I just described, you can start things within half a second. So if you need to start something within that time frame, that's a good technique to use. Now we're going to apply some of this to a camera application. This has already been done in automotive in different ways, actually. But the existing approaches, they use firmware-based cameras and displays to launch and display a rear view camera. And this works perfectly fine. But it's difficult to maintain. It's hardware specific. And it's not software defined, so it can be difficult to develop for. And in fact, it's not even Linux. So it's just custom code. So this was my first technique I tried to start a camera quickly. It was something that exists already in Red Fridge called kiosk mode, which is a mode to auto-starter a graphical application. I won't play that video. But basically, if I played that, that was when I started. I had the camera would start after 14 seconds, which is clearly too slow. So we have to do something a bit more custom. So then I moved on to an optimized approach. So I was kind of discussing it with the team. And the first thing that came to mind is, what's the first graphic that Linux displays? And of course, it's the boot splash, which is, Plymouth is the tool that launches the boot splash. So the technique I use for this, it's actually takes a lot of influence from Plymouth, except the main differences. In this case, not only are you just displaying graphical data, but you're also processing graphical input from a camera. I suppose that's kind of the key difference between this and Plymouth. So there's two main components to this. One is libDRM, which is part of MESA. And the other one is libCamera, which powers the camera. Yeah, so I've gone through that already. So this is libCamera. The kernel guys that are involved at the kernel camera stack are pushing this library hard. It runs on Chrome OS and all various Linux distributions. I think it even sports Android. But don't quote me on that. It's definitely running on Chrome OS and all the other distributions. It aims to be the MESA of the camera stack. So if you're familiar with MESA, MESA is basically the Linux user space graphic stack. So it supports various different types of cameras across multiple CPUs, SOCs, ISPs, et cetera. And it has decent support for things like DRM, KMS, PipeR, QT, SDL. And yeah, it's basically the compatibility layer for all these things. So I wrote a little reference application where I tie all these things together to make it work. I called it TwinCamp. So as I said, it's very similar to a boot splash, what we do here. It doesn't use a compositor or a window manager. Instead, we use the DRM, KMS subsystem to write to the sped up buffers directly. And you definitely should not do this most of the time. Like you should run your graphical application on top of Wayland and all of that. It's just for this use case, Wayland has so many dependencies. You could never start a graphical application under Wayland at the moment, at least, in around a second or two. It's just really hard. So in this case, there's a lot of custom code against DRM, KMS to do it quickly. So yeah, Plymouth is another example of a tool that does things this way. But yeah, it's not recommended at all, unless you need performance and need to start something quickly. So this is how you normally install a camera application on a Linux system. So the Internet or the Internet RAM FS normally initially loads all of these things. And the camera application is typically stored in the root FS somewhere else. And that's what you should do if you don't have to require and start things really quickly. So to start the camera quickly, we basically move to anything that the camera requires into the Internet RAM FS. Yet then we wrote a little system, the unit file that looks a little like this. It's very similar to the last example I showed you. I won't go through every single line, but another important line there is to ignore and isolate equals true. Because other system D units can run in this isolate mode, but basically what that means is it can stop other system D services, and we don't want that to happen. But yeah, not everything in this file is strictly required, but they're the important ones anyway. Default dependencies equals no and ignore and isolate equals true. And here is just the Dracood module.setup file to add things to the Internet RAM FS. So the first function says, make sure you have twin cam in the Internet RAM FS. This depends call. It just tells the Internet RAM FS that we have a requirement on this graphic stack, so make sure that's there. Instamods, driver media, that's the media subsystem in the Linux kernel. It's like where the camera kernel stuff is. So if you add that, you have kernel support in the Internet RAM FS. And the final install is basically the twin cam binary, lib camera libraries, lib event library, and the C++ library. Skip the slide. So when you do all this, yeah, you can basically stack the camera within two seconds rather than 14, which is the point where I started. So I'll play a little demo of that. I have recorded just to prove that I'm not talking nonsense. So what I tend to do is I'm at, sorry. Sorry, I had it on the wrong screen. So this is grow bootloader. I normally pause it at grow bootloader because this is an old dusty Acer laptop I have at my house. And I'm not interested in optimizing Acer laptop firmware because it's for an automotive use case, not for a laptop. So I'll hit Enter here on the keyboard, and you'll see the camera displayed really, really quickly. And there is me cycling a bike. So that's pretty much it. Yeah, so if you take the timings there, that starts at about 1.8 seconds from kernel boot. And actually, what's more important, because often in the automotive use case, sometimes the node processing camera frames is different to the display. So what's more important, if you actually check the timings, we start receiving camera frames at about 1.4 seconds, which is pretty decent. This concludes the talk with just some shameless self-promotion here. If you're interested in automotive or ARM enablement or any of this space, I'm doing another talk tomorrow on Fedora Sahu remix, which is essentially Fedora and Apple Silicon. And the reason I'm involved in this is because ARM enablement, it's important for automotive, because almost all automotive boards are ARM based. So it provides me and many others in the community an affordable, well-upstreamed, and powerful hardware to develop on and out very quickly. So that's about it. Thanks very much. I'll take any questions if there are any. This is a good question that we have. Thanks. The question is, do I just keep on playing the frames indefinitely, or at some point, do we transition to Wayland? This is something we have to work out. It's a good question. I've spoken to an internal team that works on digital cockpits and that kind of thing. At some point, you should switch to Wayland and the long-term compositor. I just didn't cover that for the scope of this talk. But yeah, 100%, you don't want to be writing frames directly to the frame buffer like that in the long-term. It's just to achieve the goal of initially seeing it really quickly, because that's a requirement. Yeah, go for it. So the question is, did I only measure the times on an Intel laptop, or did I also run this on some ARM hardware that is more similar to what you see in automotive? So the answer is yes. At the start, all I had available to me was a Raspberry Pi. And yeah, the Raspberry Pi was much slower, as you'd expect. But there's a huge difference between a Raspberry Pi and an automotive board, because the Raspberry Pi costs 100 Euro and an automotive board is like in the thousands. So going back to my last slide on the Apple Silicon thing, I've also ran the same thing using my Mac Mini, and I achieved roughly the same time. So it's roughly similar. And actually, most automotive boards, there isn't Apple Silicon running on any vehicles at the moment, at least. But an M1 is roughly similar in performance to what you would see in an actual automotive board. But I don't want to dismiss that point too much, because the hardware is important. Like everything is important. Is the hardware fast? Is the storage fast? Is the camera stack fast? Everything matters. Is the firmware fast? Everything is important, of course. I have, so the question is, I referenced the In-N-R-D, which is compressed. And have I tried other techniques, such as not compressing the In-N-R-D or maybe using a different compression algorithm? I did play around with that stuff. And I also tried not using an In-N-Ram disk at all, which is another option. But it didn't make much of a difference, because I found the blocker was, for that laptop in particular, and other machines I tried. The blocker was generally the kernel spinning up its camera framework. That was normally the slowest part. When I tried those different techniques, I got pretty much the same time. So I didn't see the need to just change things for the sake of it. Like, earlier on in the talk, I referenced, you can actually stash a system D service as quickly as half a second. So it's not actually a huge issue. Another question back? So the question is, have I also tried this with encryption turned on and things like DM Verity switched on? I haven't, but we have to look at that at some time. I will say on DM Verity in particular, I work on Red Hat's automotive operating system. For DM Verity in particular, I'm not actually sure we're going to actually use that for our verification. There's something called Compose FS that Alexander Larson on our team is working on. And I think, don't quote me on this because it could be wrong, I think he's using FS Verity instead. But yeah, no, you definitely have to rerun things with encryption and all these chain of truth things turned on because it will have an impact yet. Yeah, hard to. At the same time, I wonder why I'm thinking about the car. I'm in this dismay and whether you consider the automotive industry because you're starting the electronic unit in the rearview sooner in the moment I open the door, and silently start and then switch on the display when I corner on the car, probably give me five or 10 seconds. Yeah, yeah, yeah. I have to change the fractions of a second. Yes, so the question for everyone online is this is a really cool technique. But the question is kind of like maybe we could cheat a little and start to boot when someone opens the door and this kind of thing, and you might not need this at all. And that's a very good point. Why this is still important is automotive companies definitely look into those things, like optimistically starting a car if you think somebody is going to start driving it soon. So first, to first answer the question, they look into that also. The second part of the question is it's actually a certification requirement to start it really quickly from cold boot. So cheating on it gets you so far, you might pass all the certification requirements. And on the third reason, if you just auto start a graphical application like this out of the box using Wayland and all this stuff, it's around 14, 15 seconds. So 14, 15 seconds is just way too slow. You can do all the cheating you want in the world, but 15 seconds is way too slow. Question from? Absolutely. Actually, a follow on to that point is what some automotive companies have looked into is just kind of resuming from suspend and a hibernate state. But that has a couple of issues. For example, if it's suspend and you leave it in a low power state, you're draining battery. But if it's more hibernate and you're loading from disk, there are still issues because do you want, would you feel safer in a car that clean booted from scratch or a car that woke up from a suspended state? A clean boot is arguably safer, but you could debate about those things all day. Yeah, I know. I'm not actually against that technique. But some people don't like it because it's not a clean boot. And they would say is that system safe after suspending all that hardware and reactivating it. I don't know. I'm not very opinionated on that stuff. Question here? Yeah, yeah. That was a long question. But to repeat on the microphone, it was basically, do you need to stack quickly in every case? My shorter answer is you kind of do because it's actually a requirement for some certifications and such. And if you fail the test, you fail the test, though. They will literally, a tester will come in and he'll step into the car, turn on the ignition, start a timer on his watch if he deems it to be sufficient pass or fail. Any other questions? I'm just going to check the metrics really quickly. And if there's no questions there, cool. Thanks very much, everyone. Thanks for attending. That's it. Thank you.