 So, let's start this thing. Hi, my name is Aigars and I'll be trying to answer a very, very important question for us today. Will there be a debbie in your next BMW car? Spoilers? No. But I will try to tell you what will be and why and how that whole thing comes together from a perspective of a person who am I? So, I've been a debbie developer since the year 2000 and now I'm actually closer to 12 years of developing exclusively in Python. I've worked for Nokia to help them when it was still good and help them create configuration systems for like, oh, we'll do lots of stuff, a lot of different phones. I worked for BBC to help them create the UU setup box and actually test that. And then at DevCon 15, BMW was sponsoring that event and they had a job fair and they were selling like, well, we kind of meet people with the experience in Linux systems with a lot of Python knowledge and who know how embedded software works and how to test embedded devices. And I was like, you've just described my entire CV. So, soon after that I talked to them more and joined BMW car IT and moved to UU in southern Germany. I did not have any auto-specific knowledge before that. So, basically this presentation is the story about all the interesting and non-confidential things that I have discovered since then and thought all of you might be interested in. So, in theory, a car is extremely simple thing. It's just like some framed couple of wheels preferably for a steering wheel, as I mentioned something, seats, maybe a radio. That's it. What could be really complex? What could be software about the car? But if we take a look just from the driver's seat of a modern car, we start to notice screens everywhere. You can see, not even highlighted, in the very middle there is the head unit. That's the part where the maps and navigation show up. There's a heads-up display. There is an instrument cluster. There is now even displays for the climate control system and a display on your key as well. That's just the way it wants you visible right now. And that is just the very, very edge, just the top of the iceberg because behind each of those there is a computer. And behind that computer there are multiple networks that connect them together. So, in practice modern cars are extremely complex and advanced network systems. In a modern car, this is an old slide. This is a couple years ago. It has become even worse than that. In a modern car you can find up to 80 electronic control units, basically computers, that are connected by up to eight different types of networks. And some of them are more generic, not really specific hardware controllers. Some are more generic, almost normal computers that mostly communicate over ethernet. But others are tiny, tiny, tiny real-time components with a code banked into the chip itself and they communicate with others over hard real-time networks like Flexray or something even less comprehensible than that. We have a lot of specialization, a lot of different components and even automakers recognize that this kind of stuff is kind of unsustainable if you make it box by box. So they started several years ago to standardize on high-level architectures of how the code would be built, what would be the buses. So a lot of what is in the car is the same across almost all the manufacturers. So that is getting a bit better over time. But then we hit the future and the future is bright. In the future the computational and functional demands of the computers in the cars and the cars themselves will only rise. We'll need more computing power and more computers to do that kind of computations. So we can see that, especially with autonomous driving, it's not just the computers that do the autonomous driving that will be there, it's the computers that will entertain the passengers while the car is driving themselves. And yeah, with the more computing power required, car manufacturers are starting to consider that having 80 computers or now 150 computers in a car might be a bit too much. And instead there are ideas about consolidating many of those small, hard, real-time computers into larger computers, which are more generic. So instead of tiny boxes running assembler instructions, you would have one larger box that is running a normal Linux operating system with a bunch of applications on it. That is the direction that these boxing things are just going right now. But for us at this point, at this point in time in the computer and car industry, what we would like to look at more from the Debian perspective, from the Linux perspective is the few computers in current cars that look very much like normal computers that we're interested in. And that's basically the head unit display and in some higher-level models the rear-seat entertainment. Clusters that entertain the passengers in the back. These are basically almost like normal computers. They have Ethernet. They have Bluetooth. They have SSDs and hard drives. They have USB connections. Those don't look anywhere near normal USB cables. They're round and weird. But they're secured against interference mostly. And the hardware design in large broad strokes is similar to what we see in phones. So you have basically a motherboard with everything baked in. And that goes through several iterations until it gets to something that can be actually released. But there are different requirements. So, yeah, the computer in a car will need to work for several decades. It will need to tolerate extreme vibrations. It will need to tolerate a lot of dust, possibly even oil getting onto it and all kinds of weird conditions that in a phone world it's perfectly fine for the phone to actually break if you drop it on the concrete. But not in a car world. And functionally some of those parts are very close to what we can get on the phone. So you have like navigation. You have like media playback. Internet. You get some system settings. In this case the car settings. Information. And even you get some applications installed onto those systems. And yet these systems are trying to differentiate themselves from the phones by being tightly coupled to the car. So they have access to things, to the sensors that phones do not have. For example, wheel rotation sensors. So in a tunnel when your GPS loses signal an in-car navigation can continue to show you a precise location when you are based on the wheel rotation and where your steering wheel is pointing to. In a phone you cannot quite do. And a computer in a car can interact with the car suspension for example. Sometimes that's not a good thing but it can be useful. Or it can show you a surround picture of what is happening around your car from parking sensors. Pretty useful when you try to park it. Or when you left the car behind and you want to see if somebody is approaching your car. If we go deeper there is a lot of stuff going on in there. Just don't try to read that. It's just an illustration that there is a lot of layers and a lot of libraries a lot of things that are being done inside even just the one electronic computer control unit that controls the head unit the navigation cluster. In total when we build the software we can get up to 9000 individual packages just in that one system. If we are going from the top down so the GeneVy is like a generic collaboration between the car manufacturers to create in vehicle entertainment or information systems. It has an API to allow communication between software whether it's on the same computer or different computers or remote access calls through the common API and some IP. Then we use Yopto. So we use the open embedded framework and the BitBake tools to actually compile the software that we are delivering. Underneath all that we control everything with Git we build everything with Jenkins and we review every code with Garrett. That's already getting closer to what we're getting used to. There are components that are relatively new that we now know will be in the next cars. I'm happy to say that we will be shipping systems with SystemD and Wayland for the next cars. That is kind of a future that is coming to life even if our desktop systems are not quite ready for that. We can control the environment so we can actually have all the applications written in Wayland from scratch so we don't have the problems of backwards compatibility with that. The good stuff from this whole approach of building the source packages is building from the source. You've seen in previous talks here what BitBake kind of looks like. In BitBake files we basically declare to get LibGTK we want to go to that Git repository. We want to check out this particular Git revision. Then we want to apply these patches and we want to compile it with these options. We can customize everything on that stack. We can customize all the build options. We can apply whatever patches we really require. All the contents of the images that we have are controlled from our Git repository. These BitBake files in Git repository fully control what will end up in the end image. There is no change somewhere upstream that would in any way influence the end image on our side. We have to build a lot of images so there are different targets. We need to build SDKs and we need to also build packages that developers could install to bring extra functionality for the software. In time development for example. It's a big process. When we build the whole system from scratch it can take more than 8 hours to fully compile the system. But incremental builds are definitely much faster because everything is cached. One thing I wanted also to touch is licensing requirements. In this phase the licensing requirements are taken extremely seriously. Every BitBake build when it identifies that the license is one of the licenses that require software distribution there is an additional build artifact created which contains all the sources of all for example GPL packages that go into this particular build. And that is happening for every build including just developer builds that are happening several hundred times a day. So that's a good practice so that we have the source all the time and we make sure that it appears. And if there is some package that tries to include something and doesn't declare a license specifically it will be blocked from execution. It will not pass into the software. So every piece of software that goes in must declare a license and license requirements are taken seriously by the tools automatically so we don't have to worry about that later on. And of course we actually cross compile everything. So that's a good practice whether your target environment is the same as your base environment or not we just use cross compilation the whole time. What is not so good in this process of having everything through BitBake? We kind of need a lot of base packages so a lot of software that we have minimal changes or almost no changes from what the upstream or the VIN ships but they still need to be maintained because we kind of want to be very specific about okay we want this particular git comment of the software to be in our systems and unless we update that particular hash nothing will change. By being this specific we have the power but we also have the responsibility of maintaining these versions which basically leads to the situations where they get frozen at some point. And then the internal security team needs to go and search for whatever security vulnerabilities are happening and track that down. In some cases things like runtime dependencies can trigger massive rebuilds of the whole thing so at one point we updated that system D and because everything depends on system D to get their services running everything got rebuilt. That was not fun. Anything that a developer needs either during development or during a test needs to be included in this here or here packages into the SDK. That means that the SDK that we are giving to the rest of the developers can grow really, really big very, very fast so it can exceed several gigabytes of the SDK package and that is a problem that is pretty hard to manage and we'll try to limit the dependencies that developers are trying to bring in. When you have several hundred builds happening every day you probably will not do the same server. And when you have multiple servers and you are trying to cache intermediate compilation results the cache management starts to become extremely tricky how it happens and yeah the caches are in tens of gigabytes and they need to be synchronized and yeah that's a lot of fun. At this point the builds that we are having are not reproducible. There is a bunch of stuff that we are doing that is bad and should change and that was one of the goals that my employees sent me here hey find out everything about reproducible builds because for example it would be nice to figure out if we have a car, we have a car that was in an accident especially if it was an autonomous driving accident we kind of need to be able to verify what binaries it was running what was the source of those binaries what were the compile options that were produced them we need to actually verify that sometimes in court and that is important. The other thing that is of more immediate importance is updates so we compile the whole system from scratch and if we then do an update of one particular function somewhere in navigation it would be nice if only the navigation binary changed both from an auditing perspective like okay this changed, this didn't, that's fine but also from the incremental updates perspective so the update is much smaller and that goes into not just reproducible builds as a whole but okay can we do something about having the locality of the changes inside the 200 megabytes binary that's another problem that we're trying to solve BitBake has interesting connotations for example when you try to run a big project with a 9, 10,000 BitBake tasks well each of those tasks has sub tasks like do fetch, do configure, do compile, do install the problem is the BitBake sometimes really really eagerly schedules the same type of tasks simultaneously so you have 16 threads running but all of them are stuck on do fetch they are all downloading source code from somewhere and when they've done that they all execute do compile and some of those compilers they take more memory than the system would have if all of them are running in parallel yeah we have some crashes BitBake is also extremely prone to escalating complexity because basically every layer, every file in BitBake could modify anything else it could layer on top, below, in the middle of any other process or layer or anything it is super useful for example we have the security team specifying compile options for the whole project in one file super cool except then other developers would kind of need to understand the rest, the whole of the project to really know what options are used when compiling their software so the complexity can really escalate without organizational needs for that so where is the beginning, the whole of this at this point, not so much there is an initial build environment that is used to compile GCC and libc for the start of this whole process and that is, well currently it's actually bootstrapped from Ubuntu but it will be bootstrapped from Debian soon but there are other things that could be done that I could see from the inside and from Debian perspective that could be done to bring these worlds closer together for example there is a lot of auto specific tools that are kind of not really auto specific so the tools for managing the logging from embedded devices so a DLT is kind of like a binary log format for outputting information out of a device that doesn't have enough storage to write its own logs locally and that kind of thing and other tools and libraries could be packaged for general Debian consumption and then maybe propagate to other embedded areas as I said we really really love reproducible builds having them perfect and having the documentation about how other developers in their own projects can do reproducibility of builds especially with language specific logs like okay if you're doing it in C go through this list do these options, do not do these options definitely that will be a big help not just to us but to other communities as well there are tools that we are using there are libraries that we are using that by improving them yeah you are basically helping improve the software that will be going into cars and other devices eventually as well currently a lot of the packages that we are using in the cars we are not checking out anything the source from Debian we are checking out the source from upstream Git repositories so the more code, the more fixes go upstream the better it would be to developers that do the same kind of thing there are some things that also were a problem like the bootstrap well we really like to use the bootstrap more except we were faced with GPL compliance issue because when we generate an image with the bootstrap we don't get a target of sources next to it and that would be very, very useful because that's the way how a company with white policy would set that yeah the easiest way to comply with GPL is just shake the sources right next to the binaries and we don't get that from the bootstrap it's pretty annoying so we will probably be trying to do that kind of thing as a feature but if somebody pieces to it that would be amazing and one other thing that has also come up in this morning's talk was that ok, Bitbake basically is a thing in embedded community so it would be nice if we had an easier way to use Debian source packages, Debian binary packages from a Bitbake recipe so I would probably want to say ok I want to get this particular Debian package with its from source or from a binary and if it's from source I want to add this particular patch on top of that and have that as an easier way to reuse the word that is already in Debian to targets that are created by Bitbake but in general it is easier to get these whole things together if there are more developers with our mindset inside those things so, yeah, there's a link Thank you Questions? First of all, thank you I'd like to know if you'll be back porting all this cool stuff to my 2011 335D but the answer is probably not My real question is, I'm sorry you weren't able to be at my compliance talk because there's a commonly overlooked other prong in the fork of what the GPO requires and that is the installability of modified versions of the code Doing source code disclosure is critical and it's wonderful that you guys are doing that when I was at Cisco we spent a lot of effort trying to make sure our engineering teams would actually comply with that and it's all too rare but there is that other aspect even of GPLv2 that says the end result needs to be installable and GPLv3 requires this too I think a car qualifies as a consumer device at least for things that are commercial fleets Yes, due to this reason the software that is GPL is actually not allowed to be on the car so there are tools that are packaged in the whole framework but they are in the SDK so that's the thing that is not shipped with the car and therefore this thing kind of gets worked around Why we actually care about this compliance tool with this concept because the SDK and the other development things get distributed to other companies and that is where the legal problems start So there is no copy left at all that is in the devices on the car? No Wow, okay, that's too bad I was really opening for an angle there Yes, there is a legal team that is working on that So just to clarify that the Linux kernel does not appear in the car at all anywhere The Linux kernel doesn't have its license enforced, so it's okay A busy boat Anyway, that may be something to consider for the installation structure The big part, we actually thought about doing these as individual packages and actually patching the packages and using the Debian build tools and then using archives and things like that The problem is that this whole process needs to work on hundreds of different machines completely independent from each other So we have developers We have developers that are building these things that are completely remote from the systems that we have So the thing is they need to build these images completely locally and we have a lot of build servers that are creating those local images without then publishing anything So the versions that they are actually compiling are different They have different patches installed, they have in progress development So I've thought about this when you proposed this in the morning And I'm pretty sure it would be very hard to work out because it would require way more moving parts working together to get this into a working order But you would just make a disability and you would get sharing of the bind-based model and you wouldn't have the cash model We would still want to have the developers and other teams rebuilding the other binaries with their updated versions of libraries so that they could actually check that stuff works automatically There is an idea to do that, to have a base layer that is less changing The problem is we also changed the base layer quite a lot So we changed, even like we changed compilation settings In some cases we inject libraries that do runtime audit of the binaries that are running on the system So there is quite a lot of changes that are in the base layers and they evolve over time as well The other thing is that I have actually written some code which does the obtaining of the source code packages, the DSCs and stuff After when in deep bootstrap it's relatively simple to actually go through if you've got, as you've got the sources found in deep bootstrap at the end then it's relatively easy to actually then tune in change the, or add a depth source and go through the package and get the source in So that's one way around deep bootstrap so it's like, you get the full depth incomprehensible It would be better if we just have an extra the bootstrap option, like, oh produce also a thing here that has all the sources but to get that we could do the search as well It would be the same code whether you do it as deep bootstrap runs afterwards, you're not actually going to save any time during deep bootstrap because deep bootstrap doesn't care about depth source normally so you're not actually going to be saving any time when you do it during or straight up Yeah, we can do it like that So Toshiba is building a deep end source package using BitBake, are you aware of this work? I personally am definitely not aware of that and would like to look at how that would be useful Yep, hi, okay, so I would like a clarification on the whole is Linux accurate on the car thing and also sort of, because my original question was what happens if system D spikes and the car freezes? Is that a thing, is system D running on the car? The kernel and the system D are definitely running on the entertainment cluster of the car So it's not in the engine control unit which actually controls the car So in the worst case scenario, even nowadays what happens is that there are separate watchdogs that monitor the processes, the key processes and if any of them stops responding, freezes, crashes, whatever happens basically what happens is the system reboot Sometimes, well let's say that most of the time we hope that users will not actually notice that because it can take less than two seconds But the whole reinitialization of the UI is the same state as it was before Well that's handy But there is a Linux bit that's plugged into the sensors like reads the sensor network or something? It reads the network which includes the data from the sensors Okay, cool, thank you Okay so you said that it reads the network I wasn't here for the start of the talk So does the Linux have access to the canvas? Yes, the Linux has access to the canvas That sounds a bit dangerous There are multiple canvases It speaks only to the entertainment canvas I guess? Yes, basically there are up to eight different types of networks and each of them has multiple instances inside the car So you only have the buses that are required for the communications that are requested And on top of that there are also firewall rules for many of those networks like you might be able to communicate with the tire pressure sensors from the entertainment cluster because you need to show the pressure indicators but there are some other data that you don't need so that is actually blocked from them and it's available through a different network to a different issue Alright, thank you Hi, thanks for the talk I just wanted to make a little comment Earlier this week I gave a talk about secure software updates and several people asked me why New York University who developed the framework hadn't talked about it earlier and so I just wanted to drop another project that's actually a spin-off of the secure updates framework that's called Optane and it's specifically designed for updates in cars and for the special requirements of the tiny ECUs So I don't know if you guys are already aware of the project but it might be interesting I guess Let me say it this way There is a team that is working on the remote updates and software updates in general and they're a very isolated team for security purposes and so I don't specifically know what they are using but that is information that I would definitely pass on to them the concepts that might be useful would be nice to see Okay, thank you Okay, thank you very much