 So welcome everyone to our talk about implementing UAV based secure boot and over the air update for embedded arm devices So First of all, who are we? So my name is Jan Kiska. This is Christian storm. Let's start with the common part. We are both working for semen's technology We are embedded Linux consultant and developers targeting in-house means Not wherever but in quite a few cases where semen's is using a better Linux We are involved at various levels and trying to get things the right way done or a better way done We are also involved in the civil infrastructure platform project So Christian is a part of the CIP work group on software update I'm leading the the kernel work group and I'm also maintaining the either CIP core layer. What that is We talk a little bit about later We are both contributing to open source. Whatever. This is possible during our work I'm also maintaining. I'm doing some other stuff maintenance from time to time. So, yeah, it's all about open source here Regarding the topic of our talk. Well, if you've joined virtually at least The talk of us two years ago. There was something about secure boot and over the air update or something like this as well So again the same story Yeah, that was basically showing the concept and next 86 implementation Today we want to look further on that So where are we coming from? Well, the old story the old days were the wrong firm there Nowadays we do that More flexible. We've come a long way because well connectivity is everywhere security is standard Secure updates are well no longer discussable even mandated in some of our domains by standards and you obviously want to do these updates not by sitting besides of this and picking up the pieces after it went wrong, but you won't do it unattendedly and That also means you have to do it in a robust way When something really goes wrong and you're testing failed for whatever reason you want to roll back and you get all your Devices into a consistent state in the field so having all of that Integrated and also presented in form of a reference implementation via our CIP core layer that has been done in the past years And obviously also a couple of product projects before for x86 based on UAV and now as I said today It's about looking beyond that. What do we have to do on the arm side to come to something similar? So a recap where are we on x86? Well x86 it for practical reason means today you are booting from UAV firmware wherever that comes from You are booting into a bootloader, which is UAV based and if you want to do it securely obviously The firmware is validating your train of trust that the artifacts you are booting where UAV have been signed properly For the purpose of our embedded systems doing a B update We are running primarily with a bootloader which does the AB switching control You could rely on UAV for that But well, if you are more skeptical or if you're dealing with a lot of various implementation of UAV We better go with the known bootloader That also has the task of enabling a watchdog Which is essential if you want to do a rollback when some of pieces of your boot chain Got south and are no longer reacting that watchdog has to be started. That's not the service of UAV So that comes with our if you bootguard firmware for x86 Continuing the secure bootchain there What you normally do is booting assigned kernel image Well, if you only boot the kernel image that way you have a problem as we've learned before in other talks and with the Edit ID for example, you may also have a problem with the kernel command line Therefore the common practice is to use a unified kernel image to combining the kernel and the command line and possibly the unit ID Into one binary in one UAV binary sign that and then boot up from there And then you hand over to the operating system domain Where you want to obviously also validate That your file system you're booting from the root file system is as you shipped it So the read only part of it You may verify and as we learned this is a good choice with the invariate and And the persistent part that you want or have to change in the field You may have to encrypt and that's usually done with the encrypt So then you have all the pieces together. Well for the encrypt you also need a platform key and on x86 This is normally done with some form of TPM may it be physical or Implement with a firmware so to get some Platinum specific secret and decrypt this and well, then you basically have all the pieces for a common secure boot System together which can be also updated AB wise in the field So that's a situation x86. We've done it a couple of times. It works quite nicely No rocket science obviously a lot of pulling of pieces together now How can we benefit from this pattern also on on arm side? So if you look at arm Today, and I know there are other bootloaders as well, but the common piece you often get on the better side is Yeah, some you would enable meant Which can do quite a lot of things and basically can allow you to do the same thing just with different pattern here And with a lot of more custom engineering needed to get things done Well every sock vendor has a different way of validating the artifacts you are booting early to the firmware here You also have a different way of validating the OS images afterwards or the kernel image better Typically done with the fit image sign fit image here You can use your boot for quite a few things as one example you can also do there the AB switching There's a scripting inside it's possible that to do it that way. I think there's a talk these days also about this approach You can use you would obviously also as a driver for watchdogs And and start up your watchdog there, which is a good thing to do But yeah, they love customized engineer needed and furthermore you also needed for The getting out the spectrum secrets so managing the secrets to get in the end your description done So there's a situation today and while we could do it we were thinking about it But in the end the ideal case would to be reusing much more from the previously shown actually six side So and if you look at it how arm is evolving today, we are rather looking into also having you if you there as well You just think of system ready specification there So it's upcoming and with this inside but also with the motivation to unify Not just for secure but rather just for booting and the right side more Alex Raff quite a couple of years ago started the idea of having Mubut as an you if you provider and that was one of the kickoffs for also developing in that direction That we have more unified infrastructure there available still from that point to where we are today That has been a long way, but if we have it that way, what could be done now? So the pattern we see here basically if we have an abated you if you provider is that Everything after that provider basically can be the same as we have done before with x86 And we will look into further details on the right side later on what you also see if you look into the the infrastructure for Trusted platform or you will you have also some sanitization approaches there So with opti being around for more and more of the modern platforms and even some of the 30-bit platforms That's now an option to host the platform secrets and managed basically the platform specific Trusted storage there as well. So there's also option to optimize and to unify that on that area as well so The key element here is as I said then looking into who can provide you you a fee and for your embedded device So they're basically for an x86 two key providers right now on the market as far as we see on the open-source market Well, the standard one you do may come into minus a oil my gosh, and it a k2 You is it's you if a reference implementation and very commonly used or at least a common base on x86 firmware So even if you get a commercial one in the end, there's often indicate two bits included And it has half support for arm and arm 64 as well However It doesn't commonly is doesn't commonly supported by your sock vendor There are some exceptions for that But in the end you end up with something which is not yet directly fitting on your board in contrast to what you get We are the BSP for your board and there's also very limited driver support and upstream at least to what we've analyzed so far So also regarding how do I enable my watchdog now? Even if it's a simple task the infrastructure isn't there So this is something if you want to put it into you if you provide up. There's nothing like that On the other side as I said you would By now is well, it's the defect to standard for firmware if you look for your common Silicon vendors they will probably provide you some enablement for that or someone else with it in a nice way It's so it's vendor backed It's has plenty of drivers for all bits is that you may need also to bring up co-processor stuff like that Which may also in some scenarios be interesting for the features indeed for secure boot here That is all there What is maybe not that recognized? By anyone or by some people is that there is also fairly advanced UAV support by now The only issue is it's as I said not to take a nice. It's likely not broadly used yet So what do you do if you want to do or when you use you boot for you a fee and for secure booting secure a fee So one of the first well anecdote to say basically we were shipping an embedded board Which was booting from you were from you boot and was doing the classic distro boot so fit image booting And we're using default config of the vendor for that and after a while when we looked into our what's possible with your a fee I said oh, it's default on Let's just flip in our image the boot loading process with grab grab you ify and it just booted So we were accidentally basically shipping already UAV enabled system in the field Without knowing it's an open platform. It's nothing that you close down and configure to the minimum. So it was working as supposed and Yeah, at least non-secure booting was already working Now if you look at what is needed for booting securely, well, you need to continue your chain of trust Again, remember the the early stage the piece basically from the ROM loader to your u-boot Or whatever comes before you would on the middle of u-boot that is usually Remaining when a specific but this is what you get usually pre-integrated by your vendor and you can just build upon Now when you're talking further what u-boot wants to load or should load be at the UAV mechanism And should validate and you need some train of trust for that So you the easiest way to do that is to pre-compile the public keys into The u-boot binary so you have a static train of trust you could do the same with fit images Obviously, and this is now done with UAV. That's one way the other way It's also interesting but still under development and has dependency is doing this We are the ify variables and having the ify variables in the secure storage Then you can actually also update the keys in the field Without completely replacing your your firmware binary So that's it's one of the first step you have to do go one way or the other so Then look into hardening. Well, this is a general recommendation It doesn't only apply to your ify-based secure boot and many of the patterns are generic But anyway look into that. I didn't find anything like this described in that form so far There were some indications what to do in general But this actually is something that we compiled over the past years for actually practical doing it So one of the obvious thing is you have to make your boot chain Yeah, no longer interactive Lockdown any kind of interaction with the bootloader Because this usually opens up the door for doing something else than non-secure than security booting You have to or you should Limit your boot to the UEFI part. I mean if you're just keeping on the legacy boot That usually means that you don't boot securely anymore unless you've done special metal on that path as well But well, if you decide to go one way lock out the others so limit your boot procedure to UEFI Another general security recommendation or turn off feature you don't need That specifically is interesting regarding file system. So if the user can Yeah, or the attacker can replace some of the storage, which is one of the physical attack vectors and And place some nice file system on it and you would has an implementation But it's probably not that commonly used and may have some bugs. Yeah, you just opened up the door Obviously also network is something if you want to turn off you don't use network booting on that stage Another important things to remember Well either do a read-only environment because the environment keeps All the variables that could also be used for booting and booting in a different way So either make it read-only or if you have some variables that should be readable should be writable still for whatever reason There are mechanisms now in you boot Not that hand lead to describe but there are some reference that can point to you and to make only those variables Revital and nothing else. So there's been some work we done in the past years and to enable that scenario and Yeah, as I said, this is just a brief list on that if you want to look into some examples We compiled the generic ones But not completely secure ones in the ease us yet the core layer and there's also practical examples for a TI based Processor, this is from these id2050 device of us So I have a look there if you're interested in some examples So next step obviously is then to sign and lock the firmware artifacts. That is again the venus specific mechanism Don't forget to start the watchdog as I said That is in the end then taking over by Linux once it's running So that starting point should be in you boot and not later. Otherwise, you may have some black hole there Yeah, and another very important indication on that obviously test on trust the whole thing. I'm as I said This is still Fresh I'm not saying it's instable, but it can happen and it happened unfortunately There was some bug that we found during the testing recently. It's fixed by now So if you take the latest you would version you have it fixed but it's always good to at least these Basic mechanisms also checked by your system test intensively regarding if there are security holes remaining on that level so next step After you would basically there is the UAV payload starting and the UAV payload for us is our AB switching bootloader if you would got which is Quite easily or was quite easily enable both for for arm and arm 64 bills, but that's available checkpoint However, what is now the challenging part is or what turned to be a bit more challenging part was the unified kernel image So so far in x86 we used the pattern that system D was introducing a little kernel a little loader stop which Then forms together with the kernel and within it ID and a command line string and on arm also a device tree Blob that unified kernel image and that whole thing is being pulled together with object copy It's a PE binary So it's a windows binary an elf binary and an UAV binary in the end and the object copy tools You currently get from common toolchain releases you get via your distributions. They are not ready for that So you can work around obviously by taking the latest bootchain or latest toolchain if you're on yachto What an issue if you're on distribution based a build system that can be an issue There's hope as I said it's fixed But current situation was a little bit of problematic first from a from a handling perspective What turned out to be more problematic to our scenarios was this single device tree story So ideally the device tree comes in this system ready model from the firmware and it's perfectly fine describing all bits You want to have on your device Practically well, there is a life cycle of that piece of the firmware piece of the kernel From when you're having your device a longer time in the field you may start with a PSP kernel You may later on end up with a mainline kernel ideally. You may start with unofficial device tree binding You may up in the end starting or using official binding So there is a certain thing that you have to plan for at least we want to do a long-term update ability of your system in other words this kind of bundling option of the kernel and the device tree which is unfashionable But it's unavoidable practically still for a foreseeable future Now if you have one device tree for one device no problem checkpoint you can use this pattern However, if you have an image which supposed to have only minor variant of devices covered with one image You certainly have a problem. There's only one device tree and you want to address them all So we need support for multiple of those device trees some kind of matching mechanism That's what we did. Well, we first of all solved for us the unified kernel image generator problem by doing a Python based approach pulling this Python these sections together for the PE binary and then we simply implemented a mechanism to choose from the option of a list of device trees which were embedded in this embedded the unified kernel image Choose the one which matches what the firmware was providing before Pretty simple thing. Eventually we will also continue to system reboot That's basically the needed step for that the precondition for that to have this kind of device tree replacement is that the boot the firmware allows you to Forward transport certain dynamic information that the firmware plans into the device tree via a fix up protocol So there's a fix up you if your protocol being defined and implemented in you would You would is working just fine system reboot supports it if you boot got supports it I think that was but currently the situation Edk2 unfortunately does not support it yet It's ongoing work to get this into the EVPR specification So it's an open issue on that topic. So eventually maybe become standard and we become also available for other firmware providers So with that we basically leave the way if you domain and I would also hand over at the stage So now what do we have actually we do have cross architecture commodity in the if you domain Which means you do deploy the very same components across architectures Maybe with a bit of minor configuration for example enabling wash dogs or not Next is the US domain here on the right side and I want to focus on three things here First is how does a unified kernel image first find and then mount its root of s the second is how to integrate integrity protection to this process and The third one is obviously as the title of the dog says firmware update so starting with the firmware update on In modern times updateability is probably the most important product feature and commonly what we see is in a B scheme which means you do have two copies of your root file system and one is actually selected for being the current boot path So this is the currently running system while running your update the spare copy And when you're done You're just test booting into this if this has worked and the images booted up perfectly fine Then this becomes a default image and the this select switches over if not you get a rollback and again You have him we are one which was the initial one and can continue or report the failure or whatever to your bank and in terms of an OTA update This of course needs support from the boot loader as Jan just said which is Available for if you boot guard as well as for wood and for all the woodlands as well But it's a very deep integration in terms of arming the hardware watchdog Which you need for having the rollback in case you for example Deployed an x86 terminal and arm system won't boot. So we rely on the watchdog You need the AP switching and most probably other features as well This whole stuff in terms of using SW update here For example is the flow how it works is pre-integrated into sip core so you can just Clone it don't play with it. It's a free integration as said and What you can also do with this mechanism is once you have it available Means you have a general way of deploying artifacts whatever this means to a device So it must not always only be firmware in terms of for root for systems You can also use this mechanism to update packages or contain images or what else as you have one channel down to your So now having done an update on how does this unified call image find its actual root for system The assumption here is that a kernel and its root for system is one package that belongs together So meaning you have deployed a kernel and it belongs to exactly one of the two A or B root for systems Of course, there is a colonel option. You can simply specify root is death petition whatever But this doesn't unfortunately work in the AB update case Because you cannot know without asking the device in the field what its current boot part is So it has to be AB agnostic and in case of secure boot You're also at the problem that you don't want to feed untrusted information into the boot chain Which means you cannot rely on some configuration lying around somewhere So essentially there are a few options left. You can use the file system So in the inner problem of s while booting you scan the disk and search for the file system UID that matches your kernel because it's engraved and body it in the unit ID and then boot it Fine may work, but not all files system support this and so it's out of the game for a general solution Next one is you can use partition UIDs Works the same way, but it's awesome Always available and you have the the problem that you have to write the partition table on each update Which you should avoid which is not that good So in the end we came to the idea that we added custom tech image UID to the root file systems it is us release and then in the iterate over all file systems mount them look into the it is us release Look if it matches the image UID engraved in the unified colonel image If yes, we have found a root file system and we can continue booting so as the UID is engraved also in the unified colonel image means it's signed So both find a match and the UID is protected by secure boot in the unified colonel Now you have booted a root file system and this through first system is not protected against modification So it happens to satisfy our mechanism of finding the UID and it is US release But you are free to tinker with it and as long as you keep this mechanism in tech you can boot anything so what you want to have here is some kind of root file system integrity protection and for this you need Authenticity enforcement and most probably a read only root file system as we have heard from Jan you can integrate the emeriti here, which is kind of a layering approach you can use development before layering block devices and the emeriti gives you the Integrity protection in terms of mercury means if you modify the lower bits here The changes trickle up to the root hash and then you get a different root hash So what you do is you prepare your file system you calculate the root hash and then this root hash you actually Embody into the end ID which then gets put into the unified colonel image and while booting you again scan the disks and you scan the metadata of the file system of the demeriti block device and find a match between the Root hash embodied in the unified colonel image and the root hash in demeriti's metadata Well, congratulations. You found it and the bonus is that you don't have to Actually try to mount each file system you find along the way because you can read it out from the metadata So as you've seen there are many things here involved that you have to align proper In order to make this work. So in essence, it's not that big of a rocket science, but you have to bring those pieces in a line and Doing this upstream is kind of a codified description of the big shared picture It's shared knowledge you bring upstream so to not make this tribal knowledge bring it out to the open and as it happens to touch many of the Core concepts or working groups of the civil infrastructure platform It's a natural fit to bring this stuff up upstream first to the zip project Which we have done and you can find it in the ISA subcore layer. This is an implementation more or less Done with the help of the ISA meetable system, but the idea scripts contacts Configurations, whatever can just be copied over to any other meetable system doctor, whatever As had multiple times. So we do have a reference integration, which is is a subcore that you can find here There are pre integrations of those concepts presented along the line It's not fully integrated because for example Secure boot is on the technical side the realization on the box on the process side You have to do key management, whatever those things are obviously not to be found there But the box side the technical side of there you can just take it and hook it up into your Companies process and be done with it There's a demo available on chemo obviously for all architectures and one physical target and on top Of this in terms of eat your own dog food We prepared a layer meet IoT 2050 for the Zimatic IoT 2050 boxes and this is essentially the device specific difference between The ISA subcore and this one to get this box up and running. So there you can see What it takes to bring a new box or what the difference is how much work it would be to base on ISA subcore for a Final product. So in some This whole stuff combining secure boot and firmware update over the Arizona rocket science, but it's maybe too much for casual Friday If you don't want to ruin your weekend, so Take your time to implement it Most of the puzzle pieces are open source. So they are available. You just have to find them Qualify them and bring them in line And this is what we have done in terms of the civil infrastructure platform To give you blueprints Documentations or even free integrations. So documentation in terms of code which you can use and just run and The more important part here is the testing the long term maintenance So it's not just a one-shot that it's out there to be copied or to take inspiration from But instead it's meant to be a long-term thing that's tested and measured there So with this join us at the mailing list and let's make this stuff even more commodity than it is nowadays Thanks. Now we have time for questions. I guess Yep This one here So what we did here is we embedded this custom tech into the TCS res right and this UID is generated so it's some random UID and Why booting you iterate over all file systems mount them look into this when you have found it you have found your matching rule for a system of your kernel as you are now and This is essentially a shortcut of this mechanism So the root hash is actually kind of a UID and with this You don't need to mount a deambarity block device or prepare it open it and mount the file system It is US Reese instead. You can take the root hash as This UID and do the shortcut You can also use this mechanism with Deambarity, but then you have to open the block device or the root has just need to match When they don't you can open it so you go to the next one. This is also possible So it's kind of a shortcut an optimization. You can call it premature optimization It doesn't take long Yeah, the boot loader is just an if he binary Sitting on the ESP and of course you can update it So using SW update replacing it with the file a new boot loader and you if you boot guard But that's a sensible operation to do obviously if you crash the boot loader well, then No, there's no AB in the boot loader in this case not in the general concept here You need to have one core boot loader, which you don't touch On top of this you can chain boot for example if you boot girls to if you want to have to but Updating this call this single point is Then still the problem. So you don't solve it by putting two there You always have one single point of failure except for the case that you have Support in the AFI bios wherever to have two different ESP's or something, but this is not the general case So yeah, there are solutions, but not the general one not the one you buy from the typical end Yeah, this one here. So if you write to a petition a new update package, which is a petition image you may write a new you ID to this and In order to recognize this for example in the GPT you may modify the GPT depends on the implementation This is bad and except for apart from this. It's not always available if you don't feel GPT Formatted block device, then it's out of the game anyway Yeah, but if you ruin your partition table, then you again have a problem Right, you have to if it works. It's good. If not it crashes better. Don't touch it And with this mechanism here, you don't have to touch it Again So what we didn't tell is that if you boot guard also has two partitions where it holds this configuration and The the idea is that you don't touch under all circumstances You're currently known to work a boot path. So you always write to these pair other partition, right? And following this idea, you don't want to write GPT or something. So you simply don't want to touch single point of failure That's the idea Yeah, could be So we also Thought about having the configuration if he variables like system debut does We opted for not doing this because we do not know how reliable the backing storage for the variables are So we said, okay But I do it on disc if the disc has failed then it's broken anyway Don't rely on other things. You cannot influence. So it's kind of Doing it really hard The update will you just change the device? Yeah, if you do have the same root hashes in both images then both images are available So you can use both that's um, I said it here that a kernel plus a little systems one update package That depends on the wiring of your update procedure this works so for example, if you have kernel a and root file system a you have kernel b and root file system b and You now update kernel b and this one probes the discs and finds this root file system Although it should use this one Then you have x them This can be a problem too. That depends on how you wire up the update system But in terms of integrity protection, it doesn't matter because both root hashes are equal So both partitions are legible for being used Then you are leaving again the UEFI domain then you're back in the classic wood chain of your wood Yes, you can do that and it has this mechanism and it has these options But that's not the goal as I said we want to be on the on the UEFI side just the same and that's what we are achieving now Maybe even think beyond that arm you can think of risk five and things like this So it that's basically idea behind it having that standardized I mean it comes with a cost obviously you don't do this Submitted second boot time optimization this path it has to hurt complexity in it But that's also in our domain. It's not the primary driver We are more interested in having that path more commodity and more robust and more Unified than having it more optimized. So different problems to solve different sweet spot where you optimize And this is the basic it is a generalization optimization Essentially this mechanism works for example rock as well. You just have to implement it We have to choose SW update to implement it as the blueprint in SIP, but you can use anything else for that purpose So this is Quite agnostic. So you have this one here and The communication to the boot loader. So arming the watchdog a B switching this stuff So the update agent communicating with the boot loader. This has to be done This is perfectly fine integrated test W update if you have this in other or even proprietary update agents That's works as well. So there is nothing that binds you to SW update If you want to have the pre-integrated route in terms of the SIP core, then you get a SW update But you can of course swap it Then you have to re-implement the features But it's nothing that Locks you into this. So since the scheme doesn't use fit image So how do you support the use case of incorporating multiple device trees? The multiple device trees come into the unified kernel image This is kind of the new bits on that piece So the unified kernel image has a header stuff, which is a UAV binary Which loads finds out what artifacts are in this binary blob the kernel in it Rd and then a number of device trees and that it matches from that number of device trees The fitting one and boots that up the classic way So this is out of outside of the UAV domain already But as the whole thing the unified kernel which is the UAV binary we benefit from the firmware Validating that whole binary as being integral or not That's the story behind it So I think we have short question the last one as well Yeah, so the suggestion is to put everything into the kernel there are mechanisms for that Maybe we can evolve from that. I'm not sure how that then the kernel command is addressed So I'm not saying that this is the only way to go forward But it's one way that we're practicing the shoes right now And I'm happy to remove pieces from our chain obviously in the future also regarding the firmware So if our bootloader eventually becomes over superseded fine drop it do it simpler Can as well be implemented in the UAV firmware as well if it has support for it meaning extending the specification The watchdog enablement is currently done with the hardware watchdog We if he does have this software watchdog, which is essentially a timer If you would bring this to the specification Every system would have it and we are happy to ditch if he would got on this one You can now do the same very same thing in terms of writing DX ease so We if he drivers that do the job, but those tend to be specific to the way if he implementation you get for report so You better off load this to some some special thing Which we can't do with if you boot God, but eventually we love to get rid of this It's just for the transition But for this you obviously have the need and do need this Not yet. Yep, internally, but it's not yet published. So the dmcrypt part will come definitely But current not yet The problem is not so much on the technical side the problems on the process side As I said, how do you hook up your whole signature management key management? Whatever into this process So doing it on the technical side is Solved but having the integration and doing this as a blueprint or pre integration is the hard part So it's not a question of is it possible? But instead, how do you? Prepare it for being upstreamed in a way that's reusable for others In terms of not imposing our solutions to the old world, so to say Okay Think we are out of time. So thank you for your attention. We are around still