 My name is Ulrich Mateik. I started working as a security consultant for a company called ETAs. Most of you have probably not heard about that. It's a subsidiary of Bosch. We do a lot of stuff for the automotive world. But since I'm part of Bosch, I can also work on some interesting IoT Linux projects they are doing. And this is why I'm here with my colleague, Philipp. I just shortly jump in. I'm the sidekick of Ulrich. I'm not the security guy. I'm working more in safety-critical systems, but I would like to tell a personal story from what I've experienced with security and automotive in the past. So I guess we jump over to this. So as said, ETAs is a subsidiary of Bosch and supports us. I'm mainly in an activity called Embedded IoT Linux at Bosch. Linux was strong in automotive, but we also find it in many other devices. And we see that many, many projects are utilized. So if someone wants to grab a sticker, we have some of them available. But what may be more interesting is that we see that a Linux system comprised of many, many different elements. And even if you don't share the same code base to any extent, you often share the same problems. Problems are not only within Bosch, but they are also within other areas. You just get a very rough idea on what kind of products we are building within Bosch, because some know this from maybe from your home appliances or from some power tools which you're using. But actually we are in a vast amount of industries. And even if the pictograms here of the products not all run Linux in this explicit product, there's always Linux running in these devices. And device classes are also sometimes an artist, but all of them get connected to the network. So it's all an IoT device. And bringing this together, we see that, well, in the past 10 years back, we just built products which were there standalone. But IoT devices are connected, right? And this means you even have to secure them in a better way because the attack surface gets much longer. Now it comes to my personal story. So I'm mainly working for Bosch since 15 years. And when I really started in a platform, we came to first time have a multi-core processor. And in this one, we wanted to get in Linux to do our navigation, certain animations, had a 3D GPU rendering. And this was in the range of 2010. And who of you have worked on things like secure boots in 2010 already? That's very good. We would have loved to have you in here. And I guess you also said in your previous talk, if I would have known, would have been smarter 10 years back, right? And that's where we came from. We were completely new to Linux. We got support from other companies in there which were in the field of Linux products. And we started off with the 2634 kernel, put it on a device. It was not a connected device. And the good thing is we already had certain things in mind to say, how do we make things proper handover? What are real time? What are important system functionalities? We had some security ideas or prevention in mind that you don't get into a system. So we have a real time or as separated from the real operating system, Linux, the infotainment part. But what I didn't expect is how much creativity people bring to just try to hack such a system. And what I became aware of this, that I still find people post on Reddit from 2021 or so where people just hacked a device which we developed in 2010 or the base hardware software for it. And that actually, I put the timelines here for the different kernels. So you can see that there are cars around released in the age of 2018 with a 2634 kernel because we never thought about updates and so on. And if you would like to see what is all possible, I mean, it still means you need to dismount this device from the car, open it, attach certain parts. So it's a multi-level intrusion part on it. And then at the end you need to solder some wires to get to the terminal. But when you're basically done, so if you look for the password of the root, we found it easier for developer for not having one. So you just have a root user. You don't have password in there. You just get to the terminal. That's how we started development. And we never evolved from this. And at a long time as it goes, it was 2010. And I guess Ulrich would give us some ideas on what we're doing these days and our days. So Ulrich, back to you. Thank you. Yeah, I don't think I need to tell you about the importance of secure boot, of a verified boot process. For Bosch it's mainly about warranty issues and having their reputation ruined by people executing doom on a wall scammer or something. But even though there's a lot of IoT projects going on, still basically each of them has to reinvent the wheel when it comes to securing their boot process. And recently I was involved in a project where we tried to give them a head start, give them a blueprint they can follow to implement secure boot so they don't have to research the options themselves, decide which approach to take. And basically what I'm doing today is just taking you along on this journey on the different options we evaluated and what basically is in our toolbox that we try to provide to the projects to help them focus on generating the actual functions they want to implement and not on securing the boot process. Yeah, okay, I already told you that. When you look at a boot process, usually what you want to do is you want to build a chain of trust that starts somewhere in your hardware in your system on a chip, microcontroller, microprocessor, whatever that verifies the boot loader and then starts executing it. You want to make the boot loader verifies and executes the kernel or maybe even your initial RAM FS. And then you want to extend this chain of trust into the root file system so you can be sure that the actual software running on your IoT system is that which you ship with it. A little bit disconnected from that you see, you also need to do updates especially in the connected world of today. So any system we come up with for securing the boot process also needs to be somehow compatible with whatever update system the project decided to use. When we were talking to the projects we noticed that they want to have strong control over what they were doing with their system on a chip so all we could do there is provide them some guidance on how to implement the root of trust in that system on chip and also all of them were using the Apertis Linux distribution they wanted to use an upstream kernel without any modifications so this was also something we could not touch. So our blueprint basically focuses on the boot loader and how to secure the file system itself. The first building block, the boot loader basically all projects were already working with Uboot we heard also a little bit about that yesterday it's immensely popular, it has a large community it supports a huge amount of features but if you haven't done it is also maybe a little bit challenging to configure it securely but the most important part is that it supports something called signed fit images which is basically a file containing the kernel, your initial ramfs, other configuration data and you can pick a number of these and sign them so you can be sure that they haven't been tampered with. However we also wanted to provide another boot loader for the projects to use so we took a look at RustBoot the probably up and coming star on the boot loader firmament which just has a very minimal feature set it can just load and execute the kernel or the initial ramfs along with it but it needs to come from a signed fit image so if the fit image isn't signed or the signature doesn't verify RustBoot will just not load and execute a kernel and since it also supports doing updates in a very simple fashion we thought this might be interesting not to projects that are already on the way but maybe to future projects so we included that in our blueprint. Now the second and more important part was how to make sure your root file system hasn't been tampered with the most obvious solution is somehow make it read-only but that also creates additional challenges because the Linux kernel would very much like to write at least some of the directories, thank you very much and obviously having a read-only file system prevents you from doing updates might not help you against offline attacks where somebody just tampers with a file system while your device is switched off. Another measure is the Linux integrity measurement architecture that stores authentication data or integrity data in extended file attributes so it's wonderfully compatible with update systems that work on a file base but it did not seem immensely popular and so we looked at the third option which is the de-inverity kernel module as the name already implies it uses the device mapper to provide you with read-only access to a file system but each time you try to read a block from that device de-inverity actually performs integrity measurement by hashing that and comparing the value to that stored in a hash tree and effectively tie the contents of the block you're reading to one root of your hash tree so you just need to provide the hash tree itself and the root hash to de-inverity to secure your whole file system and this third option was the one we included in our blueprint so we told our projects to use that and for doing that basically you have two phases during the preparation phase which usually happens somewhere in your build pipeline you take the file system, you compute the hash tree and from the root of that hash tree you can take the root hash and when your device is running and an IO request is processed by the kernel it's passed on to the de-inverity kernel module which then uses all the data the block read from the file system your hash tree and the root hash to basically decide has this block been tampered with and if so then you can perform some other error handling like rebooting your system, freezing your system or just logging an error otherwise the read proceeds as you know it the interesting question and this is what we had to cover in our blueprint is how do you get from the preparation to the operation phase and for this they're also basically two to three options the first one is to just include the root hash into somewhere in the initial RAMFS or in the kernel parameters so just put it somewhere in your fit image where it is signed and also protect against tampering and in this case you can just for example put it into the kernel boot parameters and then the kernel will take that data set up de-inverity, mount the root file system and you have extended your chain of trust basically from the boot loader via the fit image straight to the file system unfortunately when you're working with the router's distribution that requires modifying the kernel configuration which was not an option for our projects so we could not use that approach the second approach would be to store for example your root hash somewhere in the initial RAMFS and use that for mounting the file system that brings some penalty in terms of boot time with it but it does not require modifying the kernel and in the end this was what most of our project chose this is similar to the first approach in that you also extend the chain of trust straight from the boot loader to your root file system because the root hash to which the authenticity of the file system is tied is verified by the boot loader because it's the boot loader that checks the fit image and authenticates either the kernel command line or the initial RAMFS where this data is stored there's also the third option in which you just sign the root hash with the private key and you store the public key somewhere in your kernel in the trusted key ring and then when loading when setting up the invariity the invariity uses this key from the kernel key ring to authenticate the root hash and then load the file system this is conceptually different a little bit from the first two approaches because in this case you have an additional link in your chain of trust you do not extend the chain from the boot loader straight to the file system but to the kernel and then from the kernel to the file system itself and for this there are also two options either you do just by passing the correct parameters to the kernel or you use an initial RAMFS however as I already told you since our projects didn't want to use a customized kernel this whole approach was not an option for us but anyway no matter which way you take you end up with an authenticated root file system that however is read only so in the end what you need to do is at least add a couple of overlayFS mounts to enable write access to the kernel but of course for example if you are using an initial RAMFS you can while setting up these mounts you can always also clear your overlayFS petitions so you always start from the same state again alright so these were the options just to summarize we have basically we have two different boot loaders we have two ways for authenticating our root hash namely placing it somewhere in the fit image where it's authenticated or placing it somewhere else but adding a digital signature and also for mounting the root file system we can either let it do the kernel or we can do it in an initial RAMFS however the only way we saw in use so far was from you boot to using a fit image to setting up the invariity in the initial RAMFS and what we basically have right now is we have two different boot loaders that we support we have a set of scripts and tools that support the projects in generating and signing the fit images we have a couple of scripts that they can add to the initial RAMFS that do the mounting and error handling because when using an initial RAMFS you also need to start thinking about other ways the system can get compromised for example if there were an error in mounting the root file system or something else goes wrong in the initial RAMFS you need to be sure that you are not dropped into a shell where someone can interact with the system in this early boot stage we have a couple of scripts that can be integrated into the build pipeline for computing the root file hash and for placing it into the initial RAMFS basically that's what we are currently providing the projects of course there's a ton of stuff that we still want to do basically the real treasure was not the files we created but the knowledge that we gained along the way because as I said right now there's just one happy path that projects are using but we want to also document and enable all the other options because this is like one of the first questions we get we also would like to provide a comprehensive performance comparison between all those options especially when it comes to you boot sorry to Rust boot however while we're working with Rust boot we discovered a couple of kinks that provided prevented Rust boot from running at full speed so any performance figures we could have gained from that would not have been really accurate we still need to evaluate some update systems see how they integrate with our solution so the projects don't have to do that every time again and also provide a lot more documentation so integration goes much more smoothly with less consulting going on all that is mostly internally right now but once we've reached a state where everything looks consistent and works more or less out of the box we can also open source that somewhere probably on Bosch's GitHub and when that time comes I will be happy to share a link with you if you get in touch with me All righty, that's it for me, any questions? Yes? So one of the problems with secure boot is basically you're assertive that secure boot's name So one of the problems with secure boot is effectively asserting that secure boot's enabled and many SoCs that's a matter of whether they're fused or not into having that enabled Do you have any insight into best practices for verifying that a particular device actually shipped with secure boot's enabled as opposed to that somehow being skipped during manufacturing and then it being possible to just bypass everything that way? That would be lovely However, at this point we need to place a lot of trust in whoever manufactures these devices so we just rely on them doing everything properly Thanks Just to follow up on that there are some hardware root of trust projects and chips, the open compute project has an architecture that we're familiar with in the cloud side of things so I think that was one of the thoughts I had was how do you know that you're coming up so I think there's like an assurance and a root of trust issue there that is slightly out of the scope of this Hey Excellent talk Have you thought so far in your research at all about guaranteeing the integrity of a potentially persistent overlay of those persistent overlays with something like deep integrity for example? Not yet That's also one of the many things we have in our roadmap However, most of these devices we're working with they're like very low powered and all of the security stuff is just an annoyance to the developers so it'll be a long way till we have that by default Thanks, yeah, I'm working on similar problems But so far anything that might get persisted using DM Overlay Overlay is maybe some configuration data some settings of your wall scanner so it's nothing we would execute without looking at Alrighty Thank you