 Good morning. Good morning, everybody. How are you? Is everything okay? So welcome to my talk about secret boot. I will go with a round of interaction But before that, can I ask you to raise your hand if you know what secret boot is? And can you rise your hand if you are using secret boot in production on your devices? Interesting, what the difference? Okay, we see why So first of all, let me introduce myself. My name is Fabio Tranquitella I'm a software engineer. I'm Italian studied at the Polytechnic of Torino I worked as a consultant for almost all my career working in different projects all around the world Today, I'm head of product manager at Norton Tech Though I'm a software engineer, so I understand the technicalities and I'm leading the Mender project For Norton Tech. I'm a Debian developer. I joined the Debian project in 2004 and I have a Lot of history as an open source contributor Norton Tech is a company based in Oslo, Norway We implement products for connected devices and the Mender is our flagship product. It's an OTA solution We update devices, update firmware and of course this is a strict connection with secret boot and securing the devices But we also have an holistic overview of Device life cycle management and how to secure your connected devices. So in this presentation We will talk about secret boot but from a high-level perspective in terms of how it involves Securing the boot in a wider understanding of how securing your devices So this is more or less the topic for today We want to understand how secret boot works with embedded devices and IoT As you know secret boot is not only used in embedded world. It's also basically on every computer desktop laptop you have but Edge devices are peculiar because of their constraints the physical location the use cases So we will try to look at secret boot from the embedded perspective the key takeaways for today are Understanding secret boot so decomposing it into different use cases and technologies The key components that are part of secret boot The security Implications what you can achieve what you can't achieve because secret boot is not a one stop for all the security aspects of your device and the limitations That you have to take into account implementing secret boot We will have a Q&A session at the end of the talk So if you have questions just keep them at the end and we will try to answer them. This is the agenda for main areas during the talk We will discuss about security in general for edge devices Then how to secure the boot process? There are different ways to implement it secret boot is one of the options But it's not the only one different use cases and as I mentioned before limitations and gaps So things you cannot do with secret boot and how you can solve them using other tools at the end We will wrap up everything with the conclusions and the Q&A's So let's start with a general understanding of securing Connected devices and what it means at Norton Tech we Tend to talk about device lifecycle management and that's one of the key concepts when we talk about devices It's not only about provisioning them. It's not only about commissioning them Maintaining them, but it's a world journey from the design of your hardware device The Building of the distribution you will run on it designing how this device will communicate the cloud services How you will keep it up-to-date in case of Mender for OTA proposals and how you will decommission it So in this context Security means securing every step of the device lifecycle management starting from the design till the decommissioning or eventually Recommissioning secret boot is strictly connected to several of these steps Because as we will see later securing the boot process requires the injection of keys into the device if it's an embedded device This means Injecting the keys in the hardware so it involves changes in the manufacturing process But of course also in the design part because you have to ensure that the board you are selecting and operating system You're selecting are compatible with the technology that you want to use to secure the The boot process, but not only that also Maintaining the device is extremely important because secure boot Will secure your boot process, but you want to be able to update the boot components in your device And you have to take it into consideration not to mention decommissioning the device have a limited life sooner or later You will have to decommission it and you have to do it in a secure way So from a holistic point of view we talk about a triangle of trust So every project which involves embedded devices is composed of three main actors the people persons that will use the device Either directly or indirectly. This is both means both users of the device, but also maintainers so the Colleagues that will manage the device in the field troubleshooting it for instance the device itself Which involves its identity and how this device communicates with with a back-end because a device isolated device is not Probably what you are planning for it. So device that will communicate with the cloud services. So there are Credentials there is a way to communicate with the cloud you have Software that is running great and then of course the software itself Which you deliver installed to device and then update these three components people software and device Must be taking into account when you plan the security of your device and as we used to Say authentication authorization integrity are the key concepts for securing the device Now we can go to to secret boot and as I mentioned before secret boot is a specific application But we let's keep in mind that it's a part of a bigger picture So we want to ensure that the software which is running on the device is the software that we trust Because of the integrity because of the authentication because it's signed and we don't want that external Actors can tamper them the device So let's go into the boot process and how we can secure it. So this is a high-level overview of a boot Process of a Linux device. So generally speaking, we can split the the boot components into these categories We have a ROM or firmware which is on the board and it's responsible for starting the board when you power it on But itself is not capable of doing anything special. So it needs a boot loader boot loaders Usually are split into two stages first stage second stage because of hardware limitations first stage is extremely small It's loaded directly from the ROM firmware You can call it SPL or SPL depends on the implementation doesn't really matter And it's responsible for preparing the stage for the second stage boot loader Which is the real bootloader capable of loading kernels and then operating system from there The standard boot process of the operating system starts you have the kernel The kernel itself will need a device tree in order to communicate with the devices you have on on the So on the hardware with the hardware you have on the device So for instance the storage and at one point in time the kernel will load the will mount the root FS And then start the user space. So this is a standard boot process The goal of secure boot or generally speaking the goal of securing the boot process There are different ways we can achieve it is making sure that every single step is happening as we planned We will see that there are different ways to do that. The most common way is using what it's called verified boot This is a category verified boot means Asserting that the software that we are using during the boot process from the firmware to the kernel Is a trusted and the way we assert that the software is trusted is using asymmetric encryption and digital signatures So verified boot keeps track of the checksum and the signatures on the software that you are using during the boot process And that verifies the signatures using a public key It has several advantages It gives total control over the boot process to a signing authority So the signing authority is capable of signing additional software packages So when you want to upgrade the kernel or you want to upgrade the boot loader You can sign it with a Private key from the certification authority and it will be recognized by by the device and it will be used to the boot It requires no external validation So the device itself together with a public key we will see where it's stored It's capable of checking the signatures on the boot components and it will just boot without an external actor and As it's based on the cryptography It's very hard to to defeat it to break it the world is based on a Symmetric encryption and symmetric cryptography There are a few disadvantages though because the signing authority has full control on the Signatures on the boot components It means that if the signing authority is compromised for whatever reason Then it can sign any malicious code and it will run on your device all the device is doing is just checking the signatures So losing control of the CA of the signing authority means Potentially being able to run anything on the device The cryptographic validation is done in the software. So in theory it's possible to defeat it just like any piece of software though the probability are pretty low and There is no Partial validation So the device either will boot because all the signatures are correct across the boot Components or it will stop and hang because the components are not signed correctly There is no evidence that the boot process is correct once the device is up and running So if you look at it from the user space perspective when you have software running on your devices If it's up and running it means that the signatures are okay But you have no way in the user space to actually check what happened You only know that the device is up and running So the validation happens early on during the boot process and there is no way to check the evidence of a correct boot once the device is up and running and As I mentioned before losing control of the signing authority can be problematic and Rotating signing keys with a verified boot It's complicated because the key is on the device and rotating it especially for embedded and connected devices where Everything is in the field Can create an escrow problem? so How does secure boot relate to verified boot as I mentioned before verified boot is a category any process to secure the boot of our device using asymmetric Signatures, it's called verified boots secure boot is a type of verified boot and it's designed to protect systems Against malicious code that can be injected into the device of the boot process This verification starts from the early on the boot stage So rom firmware and it ends at the kernel. It does not protect the user space So any software that you're running on the device. It's not protected by secret boot directly Though we will see later in the presentation that it's possible to establish a chain of trust From the secret boot to the user space and somehow verify that what we are running on the device is still a current What are the key components of secret boot as I mentioned before it provides? Authentication and integrity it can check that only authorized images can run on the device Because they are signed digitally and because of the check sums they are not tampered So you cannot alter any boot component of the device it works using a symmetric encryption And cryptography so private key public key, you know the theory Private keys used to sign and the public key is on the device and can be used to verify the code before running it If it's available on the device it can take advantage of hardware security So usually TPM using the TPM to protocol you can store the cryptographic materials on the device and the leverage hardware security It how it does it work in practice so nowadays specifically on x86 hardware we use ufi and Secret boot is a ufi mechanism It protects against running the malicious code using a signature which is Present on the device and verifying the check sums and the signatures on it in practice Secret boot was introduced by Microsoft back in 2007 2008. It was part of Windows 8 I think introduced with Windows 8 and became mainstream with Windows 10. So virtually any x86 hardware today gets delivered and provisioned with Microsoft keys So it embeds the public key of Microsoft and it accepts any software signed by Microsoft So you could ask me how does it work with linux then? Microsoft signing every single linux kernel every distribution. No, doesn't work like that We have the shim boot shim shim is a special software package which was created by community of developers from different distributions in order to create a minimal Boot loader first-stage boot loader which is shared across all the different distributions in the linux Ecosystem Microsoft signed the shim and that's the only thing that Microsoft is signing Thanks to the shim the boot loader is capable of loading distribution specific kernels or additional boot loaders This is possible because the shim contains the signing keys from the main distributions So the shim is the interface between the Microsoft keys and the keys from The different distributions to thanks to the shim. It's possible to run on secure boot enabled hardware any mainstream linux distribution What if you want to sign your boot components or what if you want to build your own kernel in this case It's not going to be signed by the distribution. It's your kernel. You're building it The shim provides utilities to control your system Of course in theory you could disable secure boot just get rid of it and Boot without it, but it's not what you want to do. You want to control your system. So The shim provides Utility which is called M. Okay. So machine owner key Which gives you the possibility to enroll additional keys into the system We won't get into the technicalities, but using the M. Okay till software package. You can add additional Public key to your shim and to the system to the firmware to make it possible to a run software Which is signed by your own private key So this is what usually what you want to do if you control the hardware You are using using a mainstream linux distribution You have the shim and you want to build your own components using the machine owner keys you have full control of Of the signature on the components on the boot process components and you can build your own As I mentioned before secure boot is not the only way you can secure the boot process It's a type of verified boot just for the sake of completeness. There is another way you can secure the boot and it's called measure boot I don't know if you ever heard about it or you ever used it measure boot It's an alternative to a verified boot. It does not involve Asymmetric Signatures using public and private key, but it measured the boot writing usually writing in the hardware In special registries the final state of computation of the boot components So how it works is pretty simple. It's often called measured lunch You can boot your device in a special mode that will record all the different boot steps and Store these measurements. So basically check sums final check sums in In hardware, that's how it's implemented using some specific registries. They are called PCRs and Ensuring that the next boot the final result of the execution of the boot components will lead to the same check sums If anything differs then the boot process will be stopped It's different from the verified boot because there you have an external Authority certification authority signing the boot components here with measure boot You measure you verify the execution steps of the code and that's what you store into the hardware You have more control on the boot process because at any point in time you can Instruct the device to store the new measurements. So in case of an upgrade for instance There is a special modality to write the PCRs if it's possible depending on the device But there is no External authority signing the components. It has several disadvantages. So first of all, it's not standard It depends on the boot loader implementation on the hardware. So it's not as common as a secret boot and This local Attestation that the boot process was correct is something which happens physically on the device So it usually happens at manufacturing time You do the first boot you store the results of these registries in the hardware and then you seal them So it's not possible to change them. Of course Access to the device early on will make it possible to replace the software and store different check sums but because of the lack of open and standard solutions Measure boot is not really mainstream in the embedded word and usually we rely on verified boot And if it's possible on x86 hardware mainly on secret boot So let's look at Edge IOT secret boot use cases so specifically for the embedded word as I mentioned before UFI is The the facto standard today Replace BIOS in 2007 secret boot was introduced by Microsoft for x86 devices. So it's mainstream It's included by default in Windows 10. It was included previewing Windows 8 and It works very well for x86 It's slightly different for non x86 hardware Because it depends on the implementation on the board specific board So what I used to say is that generally speaking x86 like secret boot is not supported on arm devices It's not working as x86. It's not streamlined as x86. There are alternatives The most famous one is a system ready from arm, which is a certification program to standardize Verified boot across different vendors. It's already there. So there are several boards that support system ready It's not secret boot. It's not the same thing, but it's a similar technology still a verified boot with the cryptographic materials However, because of the peculiarities of embedded devices verified boot on Embedded devices usually involve writing the public key into the hardware into the SOC There are different ways you can achieve that using electrical fuse or OTP registries But storing the key at manufacturing time is extremely important in the embedded world Why it's kind of obvious because the edge and IoT devices Have a different physical location than desktop computers, which are with us in a bag or in the office You often don't have full control on the location of the device and attackers can have physical access to the device It means that it's possible for instance to Access the device remove the storage Replace components and then plug in the storage again into device. You can physically attack the device For this reason secret boot in the embedded world has different requirements than the ones that we have on desktop devices or traditional x86 hardware Let's look at the limitations and gaps of secret boot As I mentioned before secret boot is not going to solve all your security problems in the embedded world or your embedded devices For instance once the device is fully up and running so you have your user space Working your application is working the kernels fully loaded You don't have any runtime security provided by secret boot So secret boot is done the device is up and running and whatever is happening on the device is not secret boot responsibility it means that the device can be hacked malware can insert into the device and Nothing will be stopped by secret boot itself any change in the runtime of the device is not checked by secret boot If the changes are not permanent on the storage So you're not replacing boot components changing the kernel changing any device tree driver the device will just reboot fine next time and if it was vulnerable because of default username and passwords or vulnerable software you're running on the device an attacker can still Attack the device and install malware on the device next time it puts so Runtime security and secret boot are two orthogonal concepts. They are not related to each other Secret boot only verifies Critical components in the boot process as I mentioned before we go down till the kernel But once the kernel is up and running we don't do any check on the user space So nothing will check what is running on the device. However secret boot can Work together with other technologies that can ensure that the user space is also current so one of the most Common combinations is using secret boot with the emberty checking the root file system and These will create the so-called chain of trust So the boot process will be trusted from the ROM firmware till the kernel the kernel will load the emberty And with that it will check that the root FS is also coherent with what we expect It's not that with responsibility of secret boot But it's part of our overall strategy to secure and to end your device from the boot process to the user space Important to highlight that secret boot only protects the integrity of the software So there is no encryption involved in any of the boot components So if one of the concerns on your device is protecting The IP on the device or the data which is present device secret boot doesn't help It's always possible to remove the storage from the device plug it somewhere else and read the file system And you read what is it what is inside secret boot does not encrypt anything It's only doing a verification doing the boot process that the components are signed and can be trusted another problem which is Linked to secret boot is that if something goes wrong then the device won't boot in case of an attacker replacing a boot component The device will be bricked There is no way you can boot the device because secret boot will just post stop the boot process and won't continue On ARM devices we have additional Constraints or limitations as I mentioned before on ARM We are not just there yet in terms of standardization of the verified boot Hopefully system ready Will be the standard but it still takes time on x86 It's very common to use secret boot because it's standard and it's widespread It's already 15 years that we are using secret boot on ARM. We still we still need time as I mentioned before ARM is special and usually involves specific hardware Configurations to make it work and because of the physical location of the devices you want to write the public key on the hardware And these involves in many cases changes in the manufacturing line of your device in order to store the public key early on We can jump into the conclusions so we can wrap up and summarize what we discussed so far In this table, I try to summarize different attack vectors and how secret boot relates to them if it's Usable to protect the device from that attack vendor. So first line obvious infection or replacement of critical components of the boot process boot loaders kernel libraries Is covered by secret boot so secret boot will make it impossible for an attacker without access to your Private key signing private key to replace any component in the boot process Without limiting you to update those components in the future because you can always rebuild a new version and then sign it and then update it over the air for runtime Infection secret boot partially covers the use case because as I mentioned before it's not directly managed by secret boot The user space verification, but as it can be done in the kernel using the mVerti for instance or If you think about Other solutions which are available in the kernel being part of the kernel Which is loaded in the secret boot process This creates a chain of trust if I trust the boot loader Which is loading the kernel and I'm trusting the kernel which is loading the kernel module, which will be responsible for doing user space Integrity checks, then I'm I have a chain of trust that what is running on the device is actually what they want to run for physical Disassembly or replacement of components or extraction of secrets from the storage secret boot doesn't help at all It has no encryption mechanism. It's not going to protect the storage any physical access to the device Removing the storage plugging somewhere else will lead the attacker to the data Which is on on the device you need other ways to encrypt the data if you need it So using luks or the encrypt or any other technology in this case secret boot is not the solution for your problem And any runtime infection as I mentioned before so any vulnerability you have on your device An attacker from the network which access the device replace Components change change the user space software. You're running. This is also not protected by secret boot secret boot It's only responsible for booting the device and from there what is happening is not secret boot responsibility and At this point, I think I'm done with the presentation and I open the floor for questions Risk five devices. I personally don't have experience with the risk by device So I don't know how it works. It's very hard for me to answer this question My guess is that it depends on the again on the hardware implementation Yeah, so the problem is the lack of standardization. So generally speaking x86 covered everything else It depends depends on the hardware so street Collaboration with the hardware vendor is what is required in order to implement secret boot Yeah, okay, so how are how are the security to protecting the system libraries if they are like on the root file system or something Yes, actually, it's not system libraries. I would say it's probably a typo in the in the slides more modules kernel modules So Anything which can be loaded from from the kernel so kernel modules is protected So it's probably a typo in the slides It's not system libraries, but modules which you can then use to do for the verification So the M verity or the M crypt are included in the secret boot because they are part of the kernel audit as module So that's what I what I meant So no, it's not secret boot is not going to protect anything which is in the user space Including shared libraries. These are not covered by secret boot. Yes Yes, sure. So secret boot Is not going to stop you from updating your devices It's possible to replace boot components as long as they are signed correctly Generally speaking OTA is agnostic. So you are not going to touch anything which is secret boot related even if you replace First stage or second stage boot loaders as long as they are signed all the verification starts from the ROM firmware of the device That's why it's strictly connected with a hardware support And it's always possible to replace the boot components using OTA strategy Over the air offline updates doesn't really matter. One thing which is worth mentioning though is that if you're reusing If you're using a binary distribution, so baby and Ubuntu it comes by default with the signing keys of the distributions or shim and The public keys from the distribution. It's possible to enroll your own keys But our experience is that you cannot easily get rid of the keys from the distribution so at any point in time and attack even if you streamline the kernel you can buy your own kernel and you sign it with your keys and then you add the public key using the Moku Tils and External kernel from the distribution signed by Ubuntu or Debian will still be installed correctly because the shim embeds the public keys of the Of the distribution it's possible technically picking to remove the distribution public keys, but Running updates running AP to get this upgrade on the device. We probably resume Reinstall those public keys. So the way we go with secret boots and x86 on embedded devices is usually rebuilding our own boot components and relying on our Public key at the hardware level and not using the shim So that's what I see in the field for specialized advice devices embedded devices But if you're okay with running the shim and trusting the kernels coming from the Distribution that you can just update the components a roll eventually optionally and rolling your key using the Moku Tils and build your own boot components Everything which is happening in the user space is not relevant for for the secret boot. So you can just proceed with OTA as usual Yes secure way Short answer no Okay, it's a big issue. It's a big issue. This is one of the disadvantages of using verified boot in general so if the Signing key gets compromised then it's a mess because you have devices in the field and without physical access to the device There is no way you can replace in a secret manner. So you would have to have physical access Directly For embedded devices usually the key is Stored on the heart in the hardware in a permanent way. So there is no easy way to replace it Okay, you are in trouble. Okay. You don't want to keep your key secure or you see a okay Yeah, you probably want to replace the device. That's all you can do To follow up with this question. I've seen that many device have more than one key So probably you can mitigate this one by having more than one CA or mobile public product key And then a revoke them on field if one of them is waste. I mean This is correct. However, it creates an escrow problem. So how can you trust the key which is revoking the other? So when you have multiple keys, it's not that easy to To say that it's an attacker using the secondary key to replace the previous one So you are just having two possible keys that can be attacked or yeah, yeah, yeah But you can you can separate the key in different story I mean, this is the managing the private keys that the biggest problem of all this And so you can rely to have different places Where you build your your images to store these three four private keys so you Only one can be compromised that if someone attacked for example your build system Yeah, you're right. So managing the keys in this context having them in different physical locations with different Access permissions is Crucial anyway, I agree. This is the biggest problem of all these Secure build stuff. Yeah, exactly. That's the problem with any implementation of the verified boots So not only secure boot but system ready works in exactly the same way So any Implementation of verified boot is affected by the problem of managing the keys We have an online question What if it is possible to modify the boot select pins when opening the device? So it would boot with a different SPL that just won't check anything. Can you then still talk about secure boot? So, can you repeat the question, please? What if it is possible to modify the boot select pins when opening the device? So it would boot with a different SPL that just won't check anything. Can you then still talk about secure boot? Technically speaking, it's secure boot implemented on the device I wouldn't say that device is fully secured in this boot process So if the hardware gives you the possibility with a pin to select a different SPL It's the first stage and then from there you can boot whatever you want Maybe maintenance mode then of course you are not securing your device from a physical attack in the field So again, this goes back to the idea that depending on the use case depending on where the device is located Who can access the device? What are the conditions for your specific project? You have to select the right hardware and the right conditions. So there are boards where it's possible to program the enroll the keys and then Burn fuses, so it won't be possible anymore to change the boot process from the hardware with pins. It really depends on the board It's that's why it's so special for embedded devices with desktop computers We don't have this problem or at least we it's a partial issue with embedded The key is the hardware and the support from the vendor in order to implement a Secure boot strategy. It's not only the technical implementation of secret boot It's more about an end-to-end strategy to ensure the device is booting the right software Hey, can you explain a bit more how the emberity can protect the root file system? It's not going to protect it from Securing the IP you have on the device, but it can verify the integrity of it so the emberity is going to check the check sums of the file system and Ensuring that what you are mounting so the file system It's a read-only file system in this case is the one that you want to load and is not tampered with so any change to the root of s will lead to Failure in mounting it from the kernel so the device won't boot So in this case you are trusting the kernel because of secret boot and all the chain of trust you created till the kernel and then the kernel is responsible for mounting a file system where the Check some let's simplify the check sum of the file system is known and it will refuse to mount anything else So if you change The file system then the kernel should know it and as you can only load the kernel that you trust because of secret boot You have a chain of trust from the boot loader till the user space Doesn't mean that what you have on the file system is verified But you can verify that the file system you're mounting is the one that you want to mount This also requires some machinery doing updates because when you want to change the root of s performing an ab update so moving to New version of your software Delivering an OTA update you want to make sure that the kernel will trust the new content of the root of s But it's all about integrity in this case Okay, thanks I would like to ask if secure boot is used in Mender because you mentioned Mender previously Yes, Mender supports secret boot But as I mentioned before there is nothing special that it's required by an OTA agent to support secret boot So the way I describe is that Mender doesn't mess up with Secret boot that's all you need to do because it's way it's earlier in the boot process It's not going to affect anything which is happening in the user space So apart from signing the components that you're going to install and replace on the device You don't have to worry about anything else secret boot Does not dictate anything special in the user space when the device is up and running It's early on in the boot stage, but yes, it's supported And my second question is how constrained are these devices? I mean the devices you have Checked with secure boot. How constrained they are in terms of memory What devices of real device have you you secret boot is extremely lightweight It's only checking checksums and the final states So it's not going to compute a check a signature check on the wall five system two gigabytes So it's implemented you can implement secret boot on usually any device that can boot Linux So memory constraints or CPU constraints are not a problem with the secret boot implementation Okay, thank you. I think we are over with time. So thank you very much And if you have any follow-up questions, just feel free to get in touch with me