 Thank you for being here. I'm quite overwhelmed that this hall is almost full even if it's short after the lunch Thank you very much I'm Mark Leinebutter. I'm with spangotronics and I will tell you something about how to boot an embedded system arm system here from rom code to user space and in this boot process Every single stage will be very verified so that we can be sure after we've booted completely Into the user space that we have an un-tempered system that nobody has done anything about this So why do we want to do a verified boot? Because the Linux system is the Linux ecosystem is big. There are a lot of Targets on the internet a lot of systems that are very attractive for hacking Linux systems for example, have you read in the news some some weeks ago? There was a big DDoS where they used some surveillance cameras that run Linux so Linux systems are critical. They control critical stuff on the internet critical stuff in the industry medical Automotive you name it and we want to protect these embedded systems Compared to servers traditionally on embeddance on embedded systems. There are not They are not that well maintenance as servers, but as I've noticed here on the On this weekend there are a lot of talks about updating and updating concepts. So I hope it's getting better that no old Stainless systems with Linux will maybe out there in the wild and We want to have verified boot because it's a complex software system a Linux a Linux stack the user space the proprietary application that might be on top and It's for sure that they might be some it's for that. They are some Undiscovered vulnerabilities in a Linux system. So you want to build defense in depth that means They build different layers different shells around your Linux system to protect it and a verified system that boots verified and nothing is tampered is One part of it. So I will focus on this in this talk and it's quite easy to do this Ourselves, we don't have to rely on some some vendor or so because there are a lot of sucks around that have the hardware support for doing verified boot and it's at least in somewhere in us it's documented openly and All the software components that are needed for verified boot system are available as open source software So what do we want to protect? I expect a lot of you guys have booted an embedded system. Just give a hand sign who has booted an embedded systems Keep your hands up. Who has booted a? Bootloader that is somehow verified or encrypted Okay, and then next step the chrono Yes, okay, it's getting As people and then who does a verification of the file system And do you do read write file system that is still verified? Okay, not so many. This is what this talk is about So we want to protect the whole chain bootloader kernel find system Programs configuration application data and even be able to do read write stuff on this file system because the etica Can in theory if he has access to the system manipulate of the data That is thought on the system and we want to take tempering of this So as I outlined The boot stages of an typical embedded system it starts with a rom code. It's typically Burnt into or on the silicon itself So processor when it's gets powered on it starts Executes a rom code decides where I want to boot from do I want to be put from EMMC or network USB or whatever Then comes the bootloader. This is typically you would bear box something like this then the device with a kernel and then finally the root file system and we want to secure Verify the whole stages Every single stage the first stage on our systems is usually vendor dependent for this example, we use a free scale or now nxp imx 6 and The hardware the proprietary extension they use in the rom code is called High assurance boot or short hub in in version 4 and it uses a standard crypto like Shah and RSA to do the secure of the booting For the bootloader we use on bear box So the rom code will verify the bootloader and only started if it's verified the bootloader comes with another key in it and will load the Image of the kernel the device turns in a rumfs will check it and will only start this verification succeeds and in this fit image here there is Another public key available that will verify the files in the root file system and if you go this chain from top to down The anchor here is in the sock The next anchor is in the bootloader next anchor for verification In the ramfs or in the kernel and then we booted this we can be sure That the system is unmodified and no one has tempered any files Okay, let's talk with the bootloader If you can change the bootloader of your embedded system, you have Complete control over the system. This is usually what is done on the mobile phones If you want to route it you first put in your your own bootloader If you have it could usually comes from a source that is unprotected like emmc none SD card USB whatever and On modern arm systems the bootloader is not started The very first thing in the first the first thing that runs is a rom code So the rom code has to verify the bootloader before it starts a bootloader and This is done on the IMX series By signing the bootloader with a proprietary tool from free scale and The public key that is used The corresponding certificate is burned into the IMX chip itself so the rom code Verifies first The certificates that comes with the bootloader and then the bootloader is verified With a certificate and the bootloader contains the public key for the next stage So this is how it looks here's a sock the rom code runs first Introduction you burn the fuses into Your sock and the fuses verify the public key that comes with a bootloader. This is a bootloader blob here first the fuses or the rom code verifies that the pub key that comes with this is correct, so You are allowed to boot this and the pub key verifies It's used to verify the signature and the signature goes over the bootloader itself So this is our barebox or you would or whatever, but we use barebox in this example and We have another pub key here in the bootloader that is used to verify the next stage. This is a kernel Because we have to do some configuration in the early user space We make use of the fit image. This comes usually originally from U-Boat this is a Image containing a kernel a device tree or device trees and the inner drum FS and this is all In this one image and in this image there can be several configurations So one kernel and several device trees so that you can use one fit image on a variety of boards if your bootloader knows what system you have it can pick the right configuration from that fit tree from that fit image and and boot this and The fit image can be stored on a untrusted media Like separate partition or even in the root file system if your bootloader has access to the file system because The bootloader will first pick the configuration and then check if the configuration is valid against the public key that is included in the bootloader and then we'll check the individual parts of the configuration and then boot into the kernel and load in a drum FS and The kernel it contains the next public key the key that is used For the root file system Here an overview. So the bootloader has been previously been verified by the rom code the bootloader brings the public key with it and Loads this fit image from an untrusted media It first checks If the signature of a configuration that it chooses is valid Only if this is valid Here a lot here are three hashes of the kernel the device tree in the inner drum FS It checks the hashes if these are valid so If this is valid the public keys valid the signature and the hashes, you know, these have not been modified and You see here each configuration is signed So if you have several Configurations in your fit image, you cannot pick this kernel a second device tree and maybe leave out the rum FS Because the configuration is signed and you cannot modify this So we would the kernel was in it rum FS and the device tree Then the kernel comes up the in it rum FS configures the early parts and Starts the Crypto stuff in the kernel We want to secure Each individual file of the root file system So you can later in the next step do a read write mount of it the root file system Needs to be or needs support for extended attributes to store some signatures In this example, we use x4 for a customer. We used ubi FS. So if you have a flash Chip naked dump chip with ubi and ubi FS you can use it or you have a block media Emc that is represented as a block media. You can make use of x4 We want to protect Every file in this root file system. So what we do and we make use of the integrity measurement architecture IMA of the kernel. This is Mainline in the meantime and The IMA Contains a hash of every file So the contents of every file is hashed and then Stored onto your disk or your flash chip or your media and then extended attribute. It's a security IMA attribute so far if you change the contents of the file you can Detect modification of that file because the signature doesn't match or the hash doesn't match When the kernel accesses a file open Execute or mem app or whatever it will load the extended attributes hash the file and see if the hashes are the same So it can detect modification of it First step, but if an attacker has access to the system it can modify a file and just Recalculate the hash and write it to the media But we want to protect it Don't want to give the attacker a chance to do this So the next step is to make use of the extended verification module in the kernel EVM so we create a signature over all security attributes and This is done on your development PC during creation of the image. So you take again your private key Create your root file system sign every file Yeah, we extended attribute which contains a hash and so you can be sure That the file has not been modified and the checksum of the file has not been modified and The EVM has to be verified against the public keys that comes with the corner from the first stage So it looks like this and an overview in the kernel as a public key The public key verifies the EVM signature The EVM signature verifies the IMA hash and the IMA hash verifies the contents of the file so if you put in our kernel with So far trusted and verified system We can see if this EVM signature is unmodified and then see this hash is unmodified and then see If the file itself the coordinates of the file itself has not been modified This is the first step But if you wanted to read write You cannot recalculate RSA signature because the RSA keys the private keys are obviously not on the system if they were you can as an attacker just take the private key from the embedded system recalculate everything and You're in the system. No one will detect it So what EVM then does is make use of Sharm HMAC. This is a clever way of hashing things And you can then guarantee integrity and authentication of contents but you need a Different shared secret for this for each system because if the attacker opens one system does a modification and The HMAC gets used you don't want that the attacker can transfer a modified file from one system to another system So you need a different shared secret for the HMAC on every system and And we do this on the IMX6 and I tell you later how it is done and The funniest thing is an HMAC because it's just a bunch of hashes can be verified faster than an RSA operation This is preferred if you want to boot fast Once the kernel Opens every file or touches a file it will recalculate HMAC based Verification and write it down Because it's much more faster than an RSA one and The Attica cannot calculate A new HMAC when he has not access to the secret that is used in the EVM How we do this so we have our sock that has been fused and on the IMX6 You have access if you have booted your bootloader correctly signed you have access to a unique key That is unique for every IMX6. So you cannot Transfer data with it encrypted from one system to another This this only works if you boot your system With a secured and verified bootloader Extended verification module. It's a part of the Linux integrity subsystem It's mainline The question was What's it about the fuses how it is done in the manufacturing and what other fuses contain The fuses contain a Hesh of the certificate that's used or corresponds to the secret key that's used to sign the bootloader so you have a secret key and a public key and a digest of the public key is burned into the sock and then It's secured that you cannot change the public key anymore and That the system only boots if your bootloader verifies against that public key Does this answer your question? Yes, you can the question was if you can change the fuses You can only burn the fuses from one to zero or zero to one or whatever I'm not sure but you can only do it once and then you even Have another fuse to disallow the burning of more fuses So you cannot change the remaining bits that can be possibly be flipped you even disallow this so you can only burn that once and even Turn off burning of more fuses even and this is done in the production Of your board in the first steps of testing or you can even automate this in the bootloader You sign your bootloader During building of your board support package then you need the free-scale tool and and access to your private keys You can burn the fuses whenever you want You can after you've created a public-private key pair You create a digest of the public key and burn this into your sock. So this is no Material that needs to be secured only the private key has to be secured. What's going into the fuses? It's just Just a public key or digest of a public key So if we a if we have burned off uses signed our bootloader booted with a signed bootloader the system comes up in a secure way and then you can make use of the unique key that is unique to each mx6 and with this key you can Encrypt a shared secret that is that is used for the EVM and And this is stored somewhere on your on your media You can only decrypted on that system that has been used to encrypt it and you can only Decrypt it if you have booted with a bootloader that has been signed properly So your system comes up in your bootloader System is secure and you have access to this unique key You can take the blob and in the inner drum FS you can Decrypt the blob have access to this EVM secret this EVM secret just stays into the kernel and the root file system if You want to do read write you need this EVM secret that sees if the HMAC is correct and the IMA hash is correct and then the File is still correct. Yeah The blob is the question was how do you secure the blob the blob is Encrypted with a unique key and the key is unique to one special IMX You have only access to this if you have booted with a bootloader that has been signed The question was if I can if you can modify the system if you have replaced the blob You cannot Decrypt the blob Well, if you just use some random data for the blob you cannot encrypt it because you can only encrypt it with a unique key Yeah Yes Then You you create the blob in your factory and you only can create the blob if you have a bootloader that has been booted On a secured system So there needs to be a bug in the bootloader or somewhere else that you have Access and can It's generated in the blue bootloader and then written somewhere and you cannot Recreate it without having a secure system So you cannot just boot over USB then you cannot write a The blob because you have not access to this unique key It's only available if you're on a secure system Secure the Because I can change it then I can try to use the cyber side by text attack to Modify the block to fit The IBM sequence I The question was how the block is secured or is it secured? Yes, the blob is secured by the crypto engine of the of the IMX And It is secured you can only access it if you have You cannot Access You cannot recreate the blob that is No, because the blob itself is encrypted with a unique key Yes, but okay, so what you it's not a it's not a RSA. It's not a yes encryption It's more complicated No It's not secured it's not it is secured by the hardware crypto Yes It's The blob itself is I think it uses an HMAC that runs with a By No, it's uses the the red blob or black blob message This is done by the by the rom code or by the crypto engine so you you cannot if you switch a single bit for example in your blob the Crypto engine would say you have tempered with this blob. It's not a valid blob anymore. Okay, so it's Yes So it's not an AS Yes We can look at the data sheet after the talk of the of the crypto engine of the free scale, so it's not just as encrypted it's a Even if you encrypt with the same keys the same secret data the blob looks different each time Yes, yes, I had to be more precise. Thank you You have access to the blob but you cannot access the unique key You can only tell the crypto engine, please use this unique key to encrypt or decrypt stuff Yes, it's encrypted and To encrypt the blob Signing is just This is the shared secret of the so We decrypt the blob and get the evm secret. It's a shared secret for this HMAC Yes Yes, this mainly protects against offline modification, so Programming probably properly your application on doing user space hardening is a different aspect of the defense in-depth But if you have physical access to the system, you usually just replace your bootloader and are happy and You can store a secret data with this properly. Yes With making use of a hardware random generator hardware random number generator and it's done in the factory still so and The same concept having a blob and this unique key To decrypt something you can put this together and make use of ecrypt FS Ecrypt FS is used to do file level encryption this has The aspect that it works on both NAND Storage naked NAND with UB and UBI FS as well as on block devices because it works layer In an upper layer when it's working with the files So every file in the encrypted system When in the encrypted file system corresponds to a file in the unencrypted to a lower file system The file names and the file contents is both encrypted but the directory layout and the permissions they are in clear text and This requires two different shared secret for each system because of the same aspects with a HMAC you don't want to transfer encrypted data from one system to another and and I'm a and Eva M is not needed because we use or the encrypted as uses a yes in Gallo counter mode where you have both integrity and Encryption in one thing so this Almost looks the same as if uses the unique key a different blob this creates your encrypt FS a equipped of a secret Then the equipped of s does the as of the files in the unencrypted file system and this generates you On a sub tree your Encrypted files and you can for example store your proprietary application your configuration or your database or whatever or even secret keys that you don't want to have in your In your system lying around Question the question was if you can use equipped of s for your whole file system Yes, it is possible You can do this in the early user space in your inner term FS and then switch to your Unlocked an encrypted file system. Yes, it's possible You have to talk to your lawyer about GPL Aspects of this But it's possible yes Yeah, yes It's an overhead The slower your system the bigger is the overhead We make use of the crypto engine of the I'm x6 here It brings I think in boot time like 10 percent Overall, but you get probably more CPU cycles left when you use a hardware On an MX 25 we did some Analysts and it was like Like how much was it 10 seconds from 30? On I mix six it's not that that bad and when you have Re-signed your system from our RSA to HMAC it's it's even faster But you have an overheads Because you boot first in the inner term FS and then switch to your route When you can directly boot in your route, it's the system is faster. So it's Takes some time Doing boots you get a lot of IO because if an application for example system these starts and Just maps in your all the libraries Because the kernel accesses this library normally nothing Not much would happen, but with the IMA every file that is touched gets Completely written completely read in into the RAM and then checked So booting takes Takes longer But we haven't done measurements on I mix six No, I want to show you How it's working On the system here here of my riot watch Hope it's big enough everybody can see and I just plug it in and For simplicity I've put this blobs just here in the environment of the system and I can boot it Put the system and what we see here This is now the inner drum FS and The blobs are read and put into the kernel for decoding and Then here we configure The IMA and EVM so we say Please measure this don't measure this FS don't measure the buck FS so don't check security attributes on all the file systems that are exported by the kernel and Then we say At the end please appraise and measure so check everything and If a signature doesn't match, please Do not allow access to the file Then the system boots and Then we can look in and What I've done. I've created a root file system and X4 X4 file system and there is an application called zip on it and I've tempered it offline and I've Modified the copyright string when I want to execute it Here's some debug enabled it says. Oh The hashes invalid the command was my SH That tries to execute it and it says yeah user bin zip and On which file system it was and it says so the hashes invalid and then you get just a permission denied You can configure what to do here for a customer we implement here It marks a system as compromised that we don't want to try that we don't try to boot it again and Then we just reboot and try different system, this is what the customer thought it would be nice to do so and Yes mounted equipped FS so Encrypted is as well the name of the Unencrypted and as where the encrypted stuff and this is the equipped FS and here Yes, my file and it's called bar It contains through And if I unmount it here's the File the content is bigger because there's Private keys there is some I'm not sure what's what in there. It's a standard equipped FS stuff It's gets blown up to somehow You cannot Get to the original file size to the byte the file name itself is encrypted and the contents As well, I cannot Cannot access it even because it's It has no I'm a hashes When I Crypt it and it's here if you can start it again. Just be mounted again and File is still there. Yeah, that's so far for the demo No, it's up to you. You can try it yourself Get an I mix 28. It should basically work on this on the I mix 53 it should work or Get a QVV get a Q box or a ride board. This is the one I used here and Try to get it yourself working For a customer we have it working on an MX 25 and an MX 26 makes 28 is They decided to not do it. Maybe it comes back again The boot loader we used is just normal barebox current version, maybe the maintainer has Released another one because it's Already 2016 10 maybe It's all made and you just have to Tell the barebox that your boot loader should be signed And you need to download the proprietary Prescale tools to do the actual signing The kernel is a 4.0 with some patches on it and For offline signing of the image. I used e2 of s procs plus patches for the X4 and I'm a EVM who teals. These are the tools and you need some patches for these two the patches for the MTD who teals if you do UBI FS our already mainline and All the stuff is integrated with ptx this but you can do it With your own With your own board support package generator like your door All the components are Really available Yeah, what's missing? They have been patches around or they are still around but not mainline to protect Directories what you can now do is I'm a EVMS you can take a file from one location Of this system and move it to another location of this system Because the directory is itself are not protected these patches are somewhere on the mailing list but not mainline if you want to pursue it Contact us or try it yourself Then we have to do the mainlining of the image crazy process the patches for the e2 of s procs and the EV I'm a EVM oteals and the drivers to Decrypt and encrypt the blobs on the IMX 6 they are not yet mainline and Support for MX 53 is missing should be possible as well because it's generally compatible and maybe some other vendors have Crypto extensions or secure boot extension like this Maybe we need some more documentation you can probably use it's on the on the TI Yeah Now we have been in this project for I think two years and the lessons to learn Just for development sake put your development keys in your board support package that everyone can can build a root file System and a bootloader and just play with it every developer that wants to The keys that I use in production you can Not put it into your BSP because this these have to be protected, but you can make use of Cryptographics stunts API PK CS 11 That all the tools can use to read them You can put them on a smart card or even on a hardware security model. That's Somewhere on the network or make use of some vendors that do That have software Hardware security models that you can access over the internet with complicated schemes or whatever and What's really handy for development is that you have some packages in two configuration For example the bootloader you want to have the bootloader that you can log into have shell access maybe even can read out the lock from the kernel and do booting from from NFS routes just that the developers can work and the guys who want to debug and Just have a different bootloader different bootloader configuration. That's all secured. No interaction. No state. No environment With your production keys Just keep that separate that you can that your developers still can can work and still can be happy and are not hindered by the bootloader and You can do this also for the kernel and maybe one corner that Reboots as soon as a security violation is detected and another colonel just says, oh sorry permission denied there was a problem with the security and During integration what turned out to be really good is that we Regularly turned for every release some of the security features on Just step-by-step by step Just increase the security level of the BSP that was used by our customers and And Because once activated When you get a field return, it's really pain in the ass to debug these you have to be You have to keep this in mind It's it's really makes no fun if the system is all locked down and you cannot access it anymore and What we've seen that you be IFS with email and even it really doesn't like when you have sudden power loss if your customer says oh My my clients they just turn off the power You be IFS in read write mode. It doesn't like it. Maybe it's a bug in the old corner and the 4.0 We have to We have to do updates and tell our customers to do so Yeah, that's it. Do you have any questions? Yes, it's a arm system Maybe on the arm 64 series that are more so far so far great hardware. Maybe they have integrated TPM layer You can still put a TPM on your on your board, but No, I'm watch Okay, yes, this I'm a EVM was originally introduced with Support for TPM So it was the first that it came first up with a TPM support and then All the other stuff that you need when you have no TPM is Was added Yes, you can do this at least free scale of us this proprietary method to do so We take what we get there and the tools are proprietary And the mechanism is maybe ship windows will listen to us someday. Hopefully We have not debugged it yet So Customized very let's say cost sensitive and Therefore decided to use a naked another a management like EMC then it would be some others fault if the file system Degrades on power loss System It's not dependent on the root file system that delay because You only check the files that are used or opened or touched So you can have a terabyte of root file system, but if you don't access the file, it's not checked So it's not checked the whole file system is not checked on a pair block level on on booting From start to end but only the files the corner checks them when when opening them we're accessing them Sorry for the Yeah In the kernel we make only use of the Or the EVM and I'm a uses the crypto framework and we plug in the What's already main line? The crypto engine is plugged into the crypto framework and if you enable it, it's magically just makes use of the crypto engine the the Sahara is Partly supported if it's not supported it will just fail back to the software for Shah and It's No, it's It's not in the user space the blob is Yes It's in kernel space. Yes In in runtime so that you cannot scan the rum No, actually, I mean, okay, if you love this yes, then you decrypt the blob. Yes Some socks have the ability to when they decrypt the blob to put it directly into their Okay, protected Memory, I saw them when you use the crypto engine It doesn't read the key from either kernel space Yeah To just pass around a handle and then make a shortcut in hardware Yeah, I think this is possible, but you need first support in your crypto engine and crypto hardware It's it's not supported. I think but would be fine. Yes