 Good morning everybody Yeah, good morning. So thank you for attending our presentation. So I'm Milan and with my colleague Conta, we will present you the secure boot A short presentation. So we are immediately new engineer at bootlin formally free electrons, but due to some Trademark dispute we change our company name, but that's it. We just change the company name We are still the same engineer the same team and still those we have a strong open source focus For a customer we implemented a full chain of trust on a custom IMX6 board and We wanted to share it in case it can help other people This is what we are going to talk in this presentation Short disclaimer, we are definitely not a security expert And we are presenting you one way to verify both on the board based on specific family of SOCs, but some parts can be applied to other board and And Conta will tell you when there is some specificity on IMX SoC. So first in an introduction Who wants to verify the boot sequence and why So there is product vendors they want to make sure that The device are used in the same way. They should be and not for a different purpose Also, not for running inappropriate software. So for example company may have some software limitation and Will need License for a customer to have a fully Featured software so this it can be used with the chain of trust to be sure that the user are using the The fully feature software with a license and of course to protect customers Without having some malware on your system There's also end users. They may want to make sure that the system has not been tempered with So basically it's to make sure that the binaries you are trying to load both are execute Were built by a person that you trust. So how does it work? Everything is based on digital signature verification, which is different from encryption So Conta will show you in next slides the difference between Signature verification and encryption So in fact the first element in the boot process Authenticate the second one the second one will authenticate the third one, etc And that's why it called a chain of trust because you have this chain mechanism And if any element is authenticate authenticated, but not sufficently locked down. So for example, if you let console access in bootloader or work access in user space The device is not verified anymore. And so you will break the chain of trust. So This is how The chain of trust look like so you have the warm card it has a public key it use this public key to perform a signature verification on the bootloader then the bootloader have a public key and it will perform Signature verification on the kernel and the kernel will have a digest to perform a hash verification on the root file system So every component is verified using its digital signature and a public key The root file system integrity is verified using a hash mechanism. We see Later in next slide how it works and our experience. So as I said before we implemented it on a custom IMX6 board Content work on the chain of trust from one code to the kernel and I worked on the root file system Okay, so now a little a little bit of well Explanation on what are the differences between encryption and signature. So as Milan told you The verified boot sequence or a secure boot as it called as it's called is based on signature and not encryption These are different things, but they are compatible. You can actually have encrypted data that is also signed So here is the monetary Alice and Bob example about encryption. So the point of encryption is to Well be able to send some data that is unreadable for anyone else That isn't supposed to receive for example the mail. So here we want to be sure that the mail Alice is sending it to Bob What it can only be read by read by Bob So you have as input the data and the public key of Bob Which is the input to a math function. So we don't care what is it, but it's math function and Then it outputs the encrypted data. You send it over the internet. Nobody can read it except Bob Bob receives the encrypted data and It put it into a math function along with the it's his public private key, sorry which is The only one to owns of course, it's his private key Into your math function and then you have the decrypted data And you're sure Bob is the only one who is able to read this data The next one is actually the signature Verification so here is a different purpose Here you want to be sure that Alice so the one who sent the mail to Bob is actually Alice and Not somebody else The data is an encrypted so everybody can read it So basically Alice will take her Private key and her data put it into a math function. We'll output a signed data It's readable by anyone as it seemed very important concept. Everybody can read it and Go through the internet and Bob receives it you can read it and to attest that Alice is the one who sent the mail Well, we'll use the public key of Alice to actually Verify the signature goes into a math function again a different one and well the data is verified or not So we can attest it's actually Alice who has sent the mail So that's it for the differences between encryption and signature So now you have to know that well the secure would sequence it's It has consequences, of course it's closely in terms of well putting it Setting up the whole architecture for managing your kids creating your kids Signing all your binaries etc. So of course, it's a bit longer to Handle all these cases those cases You have some added complexity to your workflow for developers because well once your platform is locked down It can only boot verified Binaries you have to sign those binaries to test them and Of course the boot time is well increased because well you have a bunch of authentication along the way to the prompt boot to the Linux prompt sorry and well last but not least you really have to be careful with your private keys because if one gets stolen or leaked and Well, you're pretty much screwed up because well anyone can sign your Your binaries and then it's not secure anymore So that was it for the introduction and now we can go to the first element of the chain of trust the rom-card or the It's called the root of trust. So this is the really the main point of your chain of trust It's very specific to the SOC so it depends on your vendors not all SOC supports secure booting and Well as I told you before you need the public key to be able to verify the signature of binary so you need a way to securely store it somewhere that the rom code can look at and Well, that depends on every vendor implementation and it's Definitely embedded in the rom code. So you don't have access to it. You have to trust your vendor and There are definitely different vendors that implement some kind of sexual boots. So these are the main one I would say xylings to grad mel and xp etc. I'm missing some surely So now you have to have a way to store your public keys So this is a very the specific way for an XP to handle the public keys so basically, it's a non-volatile memory of course because you need it to be Up when the boots from start An XP uses OTP. So one-time programmable fuses. So basically a fuse if use also is well, we could illustrate it with a wire and When the current path through the wire, then it's a binary one and you apply a very very very high current to it And it disrupts the wire and then you can get a binary zero That's how it works and it's irreversible. So once you have blow the fuse then well never gets a word again OTP fuses are really really expensive in terms of size occupied on your PCB and Whereas the public is already at least one kilobyte and sometimes up to four kilobytes so Maybe a good idea would be to actually store the hash of this key, which is way way smaller into well the OTP and Have the public key in another place for example in the binary you're going to load That way you just have to check the hash of the public key that is embedded somewhere else And if it matches then you're good to go well Now we have plenty of space. So we're not using maybe Four keys so that we can actually revoke some if some gets stolen or well Lost so a lot of way you can just revoke one key and just say Well, this one we don't want to use it anymore. We can use Any of the three remaining one? So that's it For the next slide. Yeah, thank you The rom code so basically the process is to load the bootloader in a secure Space to avoid physical physical attacks. So the point is here If you load your bootloader and authenticate it from a space that isn't secure The problem is once you have to actually boot execute the binary well, this is a time frame where an attacker can actually change the binary and Make it like it was authenticated while it was definitely not You then it loads the embedded public key, which was in which is in the you would bootloader binary, for example It checks the hash of the public key, which is embedded in the binary of the bootloader against the one which is in the hash table of Made of different OTPs fuses You check the public key, then you execute the bootloader binary if it matches and on NXP for IMX etc So C family, it's called HAB for high-answerance boot So how do you set it up for your board it's again very specific to your vendor Here you use code signing tools so CST For creating the keys so the private private keys and certificates you flash the fuses using Working and verified you boot Again using an XP specific code, which is fuses flash or something like that fuse prog I think With the fuse table written by CST You sign the bootloader using again CST and you check the status of the bootloader HAB underscore status from the bootloader from so from U-boot with a signed U-boot Which is also an XP specific and if you have reached this step Successfully, then you can decide to lock down your bootloader to actually restrict the loading of the bootloader to one which is authenticated so verified and Well, if you just screwed one of the previous item and you you blow the locking fuse then you screwed up Be sure you're doing the right thing So this is the two example of The results of HAB status from U-boot the first example you have the first line Which is actually disabled. This is because I haven't blown actually the locking fuse When it's blown then it's secret would enable if you have any HAB event Following this line. It means that your bootloader hasn't been authenticated. So it failed In the next example, you can see no HAB events found while you successfully authenticated your bootloader So that was it for the ROM code part to authenticate the well the bootloader, but now you have to authenticate your kernel So Trust no one believe me trust no one So there's no point of having a sexual bootloader if it's not authenticated by the ROM code so by that I mean If you can actually switch the bootloader with something that isn't authenticated then there is no point actually Taking the time to authenticate it because well with another not authenticated bootloader You can boot whatever you want after that The bootloader has to be sufficiently locked down. So it depends on the bootloader binary you're booting and this is There is a very specific use case well example for U-boot in mainline U-boot in mainline has Environment variable called verify if it's said to know U-boot stops to verify the signature of images and That's well, not what what we want So we definitely do not want any user to access the console to set the environment variable from there So you have to disable the console at all Well very important point point you should I don't know Circumstances trust anything that isn't in an authenticated binary. So it means that well environments are often on other mediums But there you can't trust them because you cannot authenticate them So you should use the one which is definitely in the U-boot binary that was authenticated by the ROM code Well, that doesn't really work with update Software from Linux kernel because they always they often interact with the U-boot Environments to say you should boot this image or this image So I have a pending patch for you boot to only load a handful of variables as you can control from the U-boot embedded environment binary and It can load from whichever medium you want to load the environment from So yeah, that's it So now how it actually works you're gonna Use a device reblog which is basically the same as it's in the kernel It's used to actually embed the the the public key within the U-boot binary and This public key will be used to check the signature of the kernel images and Because it's appended to the U-boot binary. It's also authenticated by the ROM code So you can actually trust the public key which is within Well, the U-boot binary So now to be a bit more practical and well simple to use To check the signature of the different images you have two choices Either you load one image and then its signature and then the second image and its signature, etc Or you use the fit image which is basically container Storing a lot of different binaries and there are hushes and MK image so the tool to actually compile the fit image has already built-in support for signing in Well doing the the binary is hushed. So well, we'll knock so better use this tool so well now in the process to get you to the next chain In a chain of trust you have to create actually your keys To verify the kernel signal to actually sign the bootloader The sorry to sign the kernel and verify it from the bootloader So the first line is to create an RSA 4k key, which is called my underscore key This is a private key and then you create a certificate certificate. This is really close to a public key So it's really MK image requires the use of certificates and not public keys. Well, it's a choice and It's really really important that your certificate and your private key are named the same Otherwise MK images in the work at all So this is an example of a source source file for the well the device 3 used to embed the Public key within the you would binary So you have the signature device we know if your ball already has device 3 Well, you should be able to use it and just add the signature device. We know Something to note is that key name hint and the suffix to key dash DT node has to be the same Value or name as the one you have given to your key So in the previous slide was my underscore key. You have to well use the same There is also a property which is called required. It's it's it's either image or configuration it depends on which level you want to actually do your signature verification and You can find the well the explanation in the Documentation here in the you would project So what's a fit image? I say it's basically containers. So you have a set of images different one one two three kennels, whatever The same for device trees in it from a face. You have also support for other kind of files like scripts and And then you have also configurations so configurations are a way to force the user to Load Binary together. So for example here you have conf at one so the Red box which force the user to load the first canal the third device tree and the first in try fast So that's a way to force your user to boot a certain configuration a certain configuration it supports different architectures OSCs Image types and a lot of other things and you can find all of them in the source code in command slash image. Let's see So this is really a quick example of a source file for fit image So you have the image images device 3 well node with all the different binaries where we can find them where you can load your binaries Which key to use to sign each of the binaries and you have the configurations on the on the on the right And there is an important line, which is default equals Confat one and this will be important in the next upcoming slides and You can see that the confat one has kernel is equal to kernel at one So it will be using the first kernel and the device reads which is also the first one so There is a little bit of Tricks to do before compiling your uboot binary. You have to actually compile your Device through which contains the embedded public key outside of your uboot well source Repo history and this is because we need to use MK match which is a tool from uboot To actually embed the public key before we are going to build the uboot binary so You use DTC to do well the first compilation of the uboot public key the DTB Then you compile all the tools from uboot your run MK match to actually Build the fit image and at the same time embed the public key within the device 3 and Then you compile your uboot binary using the external device 3 blob, which is called uboot under scope of public key These are the Well the configuration you have to put to enable the fits the fit image signature verification on In uboot to say config of control is for embedding actually using actually the Device 3 blob support in uboot and the first two items are an xp specific So this is actually what it looks like to boot a fit image with all the verification steps Etc. So provided you have a fit image loaded at 0x15. Let's say so you do boot M 0x15 pound Confat one but since the default in the configuration configurations Node in the source file of your fit image, you know that configure confat one is the default one So you can actually just keep the part after the pound. So with them 0x15. It's fine You can see that it's using confat one. It's trying to load the kernel at one It does all of those things and verify the hash integrity which says well, it's okay. So let's get to the next binary in the In the fit image, which is the device 3 blob. It checks it. Well, it's fine. So let's get started with the kernel but then someone touched the device 3 blob and well, hopefully it will get it and Stops the woods and that's what happened in the lower part of the this slide It's verifying the hash integrity. They say no, I cannot verify it So well, I'm stopping there and I'm crashing you it you shouldn't stop stop here So I think that was it for the part Up to the root file system. So now million very will present her part on the root file system Yeah, so it's the final part of the chain of trust To have a very fast root file system, we have chosen the following solution It's you can have other kind of solution, but this is what we choose So first of all, we have we wanted to have an Inalteable file system. So we choose to have a read-only file system So it's impossible to modify it So we use a squash FS image, which is the type for read-only file system So this part is not really part of the secure boat process, but it was important for us and for our customer So the Authentification of the root file system. So it's really the Part of the chain of trust we use them verity. So them verity It's an infrastructure to check if the word file system is the one we are expecting to have so in fact, it's the authentication of the squash FS image and Damn verity needs some user space application to authenticate the system And so you need to have these tools available So that's why we choose to have to have in it from FS as a first file system and in this in it from FS you will have the application tools to perform the Authentication of the root file system The in it from FS is built in in the kernel. So thanks to that it doesn't break the chain of trust Because as Compton told you before the kernel is already in the chain of trust using the fit image So now the MVVT so the M It stands for device mapper. It's an infrastructure in the Linux kernel to create virtual layers on block devices So device mapper verity It's provide integrity checking of block devices using kernel crypto API Damn verity could hash the whole block device and then compare the Ash with an expected ash, but it's not doing that. Otherwise the boot time will be very very long so instead of that it use a cryptographic ashtray and Your block device will be split in small block and each block will have it ash and It will create a ashtray until the last ash which is called root ash we will use this root ash in the authentication mechanism, so we will see it in next slides and This block are Asht only on access so instead of having a longer boot time We will just have a smaller Long access on block device And as I said before the MVVT needs some user space application So crypt setup package provide different tools such as verity setup So we see next slide verity setup is a tool to create the Ash tree and also to authenticate the device So the MVVT in our case So we have our kernel Damn verity has some Configuration in the kernel so you need to enable this configuration to have them damn verity and As I said before we have a need from FS and we created a in script that will use Verity setup tool to perform the verification on our block device For our use case we use UBI block device, but it can be other kind of block device if the verification is Successful, then you can use mount command for example to have access of your SquashFS that you verified and If it's not the case, then it will fail to Mount the SquashFS and it will stop the init here So now let's see how to create the Ash tree that we have seen in previous slide So very this setup can Create the Ash tree on device or images for in this example It's on images, but you can do it on device node So we have our SquashFS image we perform very to set up format command The resulting image is Ash image and it's containing the Ash tree that we will use to authenticate the device and It also prints on your screen the root Ash And we will you need to keep it because we will use it to authenticate the device to be sure that the SquashFS image that we Create the Ash tree right now. It's the correct one that we want to authenticate So by default the dem verity and verity setup expect to have Block device with data in there and another block device with the Ash tree So the command used is verity setup format then the data device and the Ash device But in fact in our use case we wanted to have only one device So our block device will have the SquashFS image and also it will contains the Ash tree For that we can contain the Ash image at the end of the SquashFS image to have at the end final SquashFS containing also the Ash tree To do that verity setup tool Have as an option Ash offset and to to say how to To locate the Ash area in the same device or image so Here's the example of the command so verity setup format then we set The offset so where we can find the Ash tree in the same Device or image Currently in our example Because Ash tree is at the end of the SquashFS image the offset is the size of the SquashFS image and Then the data device so in our use case. It's the SquashFS image and the final Resulting result. So it's the SquashFS image and also the Ash image So now we have created our Ash tree We have the root Ash and we will see how to authenticate the device So now we are on our platform. We have flash flash All the different images and we are we have our block device That contains the SquashFS and the Ash image We use verity setup create command that will need the offset to know where to find the Ash tree and also the root Ash If the Ash correspond to the one that is Available in the Ash tree, then you have a mapper device on The SquashFS verifier. So at the end you will have a slash dev slash mapper slash something and On this device mapper you can use the standard mount command to then have access to your verified SquashFS So the Common use is very tight very tight verity setup create Then the name of the device mapper. So in our example, it's a root file system It can be another one We again we use the Ash offset Option of verity setup to say where to locate The Ash tree in the device block Then there is a data device and the Ash device in our use case. It's the same one and finally the root Ash So because we use the same block device and it contains both data and Ash Image we use UBI block 0 twice Of course, if you use the default mechanism. So if you have for example, the data on UBI block 0 and Ash tree on UBI block 1 then you don't use Ash offset Parameters and you will have block 0 and block 1 in your command verity setup create Yeah, so here you can see that we have the offset and the Ash parameters and We need a way to have it available in the kernel. So in fact, we use both args And for that we created a Uboot environment script But this new component can be attacked so to not again break the chain of trust we add it in the fit image So in the fit image, we have the Uboot environment script with the Ash With which has the signature of the Ash of the binary then Uboot check the signature of this Uboot environment script If the signature is Matching then you can source this script and in this script we have some variables to To retrieve the offset and the Ash and also of course it sets the boot args Thanks to that we can retrieve it in the kernel So it's the final Mechanism so as I said before we have Uboot it check the signature of The Uboot environment script if it's not the correct one it will fail to source it If it's the correct one it will source the script Thanks to that in the boot args. We will have the offset and the Ash Then we have our kernel with them verity enabled Our init1fs with the init script that will use the verity setup command To do the verification on the UBI block device Of course as we have seen before verity setup command needs the Ash and the offset so thanks to the boot args you can retrieve it and If the verification is successful then you have a device mapper and you can do Switch route so in our use case we use switch routes instead of mount because we wanted to have To switch the root file system from the init1fs to the verified squashfs. Otherwise Our system will always be on the init1fs and we didn't want that and to complete the Chain of trust so if as we added a new component So you use a Uboot environment script for example we add it in the fit image to not break the chain of trust Thanks minute So now we have actually well completed our chain of trust So basically now to Summarize what has been done yet until now So into the in the rom code you have up to four hash of public keys Which are used to verify the public key which is embedded in the binary of the bootloader Then when it's verified the public key Then you can use it to verify actually the signature of the bootloader once that's done You have another public key in the Uboot binary Which is used to verify the signature of the kernel and the dTB Well basically the fit image the whole fit image with the Uboot script I mean and presented just the slides before And once that's done you have the digest within the fit image which is used to verify and authenticate the root FS So yeah, basically that's done But now we really want some kind of I would say distribution As a root FS and we don't want to well do it manually We can use build routes yokto whatever build system you want But in our case it was yokto, but yokto has a really specific way to handle to create beat image fit images and currently It's the kernel recipe that actually in a hit the kernel fit image class and that means that well It's done after The root FS Sorry before the root FS has been created and I think the point is that actually people want to have Fit image the kernel or whatever within the slash boot of the root FS But our script needs the root FS to be actually compiled built so that we can Compute the hash and put it in the Uboot script but well We want it in the fit image which is created before the root FS So basically that's all you end up with a dependency loop in yokto and that's really hard to debug So well, we throw away the kernel fit image class and wrote a new image and class recipes to work around this issue It's really I would say the level of draft or proof of concept, but that works so that was it for us so well it was Very specific example we presented today to have only read read only file system But well, you definitely May want to have some read write file system Besides your read only file system So if it stores not critical data while just mounted Just well alongside your read only file system. That's fine You can so blogs your data. Well, you don't care. Well, it's if it's critical. Well We haven't really looked at it But a hint would be to look at IMA EVM. We have provided a bit of well toolings to look at it I don't know how it works Just have a look at it So to well end this talk Remember I told you to trust no one. Well, you cannot even trust the vendor of your SOC so an exp had Publicly disclosed about a year ago that they had secure with vulnerabilities in our in their HAB verification code for IMX 6 and the other family SOCs But I think right now it should be fixed. So you should ask them if you want to base your product on it and I attended well yesterday a Good talk on Introduction to reverse engineering by Michael Anderson and it was really interesting because well nothing is 100% secure. So What do you want to protect yourself from? So well, I'll end it with this look at team team. Sorry his talk to Well, see what can be done to actually break your thing So that was it. Thank you for your attention. If you have any question, I'll be in the audience to pursue the microphone Hello, okay. How long did it take your team to implement all this? so actually from the beginning to actually look at all the software everything would say for both of us to like Six month but not full-time. So I'd say month is differently Two questions. One of all I saw that you're using Shaw one. You probably shouldn't use that as a hash algorithm anymore Just just it is it totally broken Number two, where is your code that you actually talked about that works for doing the switch route thing? So you don't get the dependency problem. Well, it's a bit different The problem with the depends loop is within your toe It does not have something to do with the finer root of s So it's only when you actually build the root of s. There is the problem. Well, it's a new Well recipe, so that's how your toe work So the question is is the recipe in your toe yet? No, it's well in our yoke. Oh, yeah Yeah, no, but well Definitely would want to work on it So I have two two questions to firstly. How many boards do you actually break doing this and Secondly, is it emulatable and something like QMU? So good question the first one have how many boards are we broken yet so none Yeah, there are a lot of tutorials on the internet so Luckily, we didn't have to break any boards. So well, we're happy with this And the next question was QMU support, I haven't I yeah, I don't know. Sorry You mentioned the performance impact of the verification. Did you actually measure how? How much it affected the boot time in your case? so I Think there's not really a point measuring it because if you need a secure device Well, the latency you don't care really on the root of s part They pretend that the IO takes so much time compared to the hash computation that all You can ignore it Any reason you do very efficient verification of your root of s and kernel and not you put So for example, can you boot read this block device? so the question is where the verification of the root of s is Why not in your boot? There is no support for squash FS first and second is that all Oh That was it. It's on two different mediums for Media for our clients. So the first one is used to store the kernel, etc And the root of s is on another medium so There is a really interesting remarks and I forgot about it It's actually done only on the read access of your data So you cannot actually use the bootloader to verify your ROOT of s because it's done only when you access data, which is done only by the kernel You have four public keys stored in your in your boot image. How do you revoke one of one's bad? The same way you actually flashed the public key. So you blow a fuse and This fuse just say well, this one is revoked. That's it. Oh from from you boot So with a new boot with your update scripts or whatever you ask it to Blow the shoes, but yeah, if it's already too late, then it's good anyone She have an observation on an earlier comment One time verification of the squash FS is not good enough Whilst squash FS itself is read only it does not cache all your data So if you're sneaky you can go in and rewrite the data that's in the squash of s We found this because one of our clients accidentally ran their upgrade system against their company running squash FS This makes the kernel very unhappy And crashes the system When will the URL work? I tried it in it returns file not found The URL on up there. I tried it Yeah, it's not up yet. Well, it's on the sked.org Website, so it's gonna be available. Hopefully next week dogs Okay, any other question? Well, thank you and bon appetit