 Okay, and it's time. So I'm Tim Orling. I'm from work for the Consulco group and today's talk is tales from the crypt and implementing secure boot and disk encryption on Tegra platforms. So let me start off with the straw poll How many people are here because they've heard about secure boot and they need to know more? Okay, how many people are here because they heard about disk encryption and they need to know more? How many people are here just because of the Tegra platform in particular? Very hardware specific. Okay Cool So Consulco group we're a services company specialized in all kinds of embedded Linux stuff We have a lot of very highly skilled staff that can solve a lot of problems for you. So We do hardware software all kinds of things. We're actually based in San Jose, California But we have engineering presence in Sweden, Bulgaria Port Portugal, Greece, United States, Canada, so all over the place. I Always include my abstract just so that people who are writing new Presentations can actually see what it was that got accepted and hopefully it's a Hopefully I'm I stay true to the abstract as to what I actually got done So here's the agenda. So why is it tails from the crypt? Okay, this is a war stories talk, okay? Because I want people out there to realize that even folks that are like really have been around for a long time I'm doing this for you know, 10 20 years Like our life is banging our head against a wall until it breaks and then you run and then you find another wall And you bring it being your head against the wall and it breaks. Okay So the cool thing is once you break that wall down you get to the other side you get to do the happy dance, right? And so that's a little bit about what I wanted to talk about today I will assure you that I'm actually giving you, you know links to fully functional things that will really help you in the case of the secure boot and disc encryption and especially on the the Tegra platforms So we're gonna start off talking about secure boot and then we go into disc encryption and then I'll talk about future work Another thing about embedded space is nothing ever stands still. So secure boot Nvidia has got a particular implementation. I think from a hardware standpoint and software standpoint This is actually one of the better ones out there. I was really pleased when I did the assessment of of how this stuff is working So you've got keys or a hash of a key So you got a hash of your your private key or has hash of the public key for your private key For instance, we'll be in burned into the fuses Burnt into the fuses. That's an important thing to know and then there's also this Encrypted key storage Partition and this has additional keys in it. So that one is unlocked by the ones that are in the fuses, right? So you've got a ROM that then looks for something that's secure and then it opens up Another layer of you know security and another layer of security and another layer of security, right? So there's this trusted operating system At the time that this development was done. It was called trusty I'll get into where they're going with opti in the future after that Once you've gotten into Your init.rd now you have client applications that can start to query those trusted applications For keys, they're going to do things like unlock your disk partitions, right? so this is kind of about four layers in already in terms of Stuff that's secure and encrypted and and trusted before you even got to unlocking the disk encryption So there's a couple of example applications for trusty and also opti the hardware key agent and the lux serve Trusted applications. There's client applications that talk to that In general for secure boot specifically there's multiple levels of pop of Possible security the initial thing you have to have is the the pkc the public key Cryptography so this you cannot flash the device without You know once it's been public Public key encrypted or you know set up for secure boot you can't flash it without knowing what your public key is So you have to do everything over secure communication once you've started down this path The next level is secure boot key. So this is where you actually get secure boot, right? You don't with just the public key encryption. You don't actually have any secure boot All you've really done is kind of protected your ability to flash the device On top of that you can also do user key and this is where you want to start to encrypt your kernel or It's also used in the disc disc encryption. So there's kind of these three layers of progressive Optional, you know security that you can do specifically for secure bit the user key doesn't really come into play too much for Secure boot it like I said it is it is useful if you want to encrypt your kernels or sign your kernels So Secure boot is not one-size-fits-all by any means right it is really really specific to different platforms how it works So most of my experience was on Intel platforms, which is entirely different than what we're talking about here in terms of secure boot What I did in this work, this was customer paid work This was specifically targeting the Jetson a gx Xavier platform or the T194 class of stuff So this was specifically done with this, you know SDK that comes from NVIDIA Jetpack 4.6.1 or Linux for Tegra 52.7.1 those two kind of mean the same thing and this is Yachto base. So this was on the Yachto Duntel LTS release so You know highly recommended that folks work on an LTS release not not some random release We'll make your life a little bit better By the end of the work we I forwarded ported it to Kirkston and a newer jetpack and Before I you know came to give this talk an even newer release came out, which is we'll talk about in future work So about the fuses I Think this is kind of confusing at first. So what the fuses are what's what's there? What do you use them for? So there's this ODM production mode fuse Once you have burned that fuse that one does not allow you you can no longer set any other fuses, right? So this is it if you if you set that fuse only You can't do anything else, right? So In practice, I think for secure boot you really absolutely are going to be focusing on the public key hash So that's you know your RSA 2k bit or 3k bit on this platform Private key you get the public key hash of that and then are the public key from that and they make a hash of that That's a 256 bits that they flash into memory or into into fuses And then there's the secure boot key and this is again what actually allows for secure boot, right? So that that's also needed We've got these other keys. So there's the key encryption key zero and one Those can be used by any of your client applications to talk to the trusted applications to get a key For whatever purpose you need a key for this could be anything, right? You want a key to talk to some cloud provider service, right? That that could be what you use it for those are kind of like user user keys There's also this optioned on the the AGX Xavier platform to do the KEK 256 That's just a combination of the 228 bit keys into one key to give you more more bits of security, right and Then there's the KEK 2 key Which is 128 bits this one is what is used in the default Linux Luxe encryption disc encryption is the KEK 2 key. So that one's kind of important later In the slides, there's a link to where these all came from in the documentation So big big warning, okay, it's called a fuse for a reason All right, it's a fuse Once you turn it from a zero to a one you're done okay, so The board ship with all the fuses at zero except some stuff in the middle that NVIDIA has Put in there. That's plop. That's the device specific, right? But NVIDIA recommends burning all the fuses at once at the beginning so plan ahead, right? So in practice, I would say for development purposes, you're gonna want to have production key or sorry the public key the secure boot key and if you're trying to go secure, I would do KEK 256 right for your user key But if you have maybe two different applications that want to do something with a key You might want to use both zero and one and then the KEK 2 is again going to be used for for disk encryption later Keep your keys in escrow. So by that I mean, you know make a tarpaul of it put it somewhere, right? Keep it safe Even for development purposes, right? You still want to keep it safe somewhere because once you've burned these keys You have to have them already to talk to your board anymore, right? So I read the documentation multiple times I Looked through all the steps. I thought I knew exactly what I was doing I read some posts about things that had gone right or wrong and I still thought I knew what I was doing and I created a lovely brick Okay, so my guess it's just a guess as I was on a Slightly older release that may have had a problem with an RSA 3k key So it may have created too big of a hash and Overwritten one of those magic things that's in the middle that NVIDIA has flashed into the fuses or Somehow as an idiot I burned the fuses twice and so I burned them with one key and then then burned a few more bits On top of that But what no matter what happened what you know what ended up and there's a link to the Forum posts that I did where this board is just not gonna come up anymore. It can never get out of ROM anymore. Just done, right? Not a happy day But again, I wanted to tell you, you know real stories, right? We brick boards You know if you if you haven't bricked a board in 10 years, you're not trying hard enough, right? Please work harder You're not working at all. Yeah, exactly. So So where did I go from there? Well, it turned out I found this repo from one of the meta tegra community folks called secure boot tegra They specifically wrote this Because they were having trouble communicating with the NVIDIA tech folks on the forums to prove that something had gone wrong Okay, so they wanted to make sure that everybody was following exactly the same steps Well, I found that and I said, wow, this sounds fantastic And so it turns out I've modified it a little bit, but you can generate your keys There's scripts that'll burn the fuses for you, right? It does the magic command line incantations to burn the fuses, right? Then you can actually flash in a Binti based OS to your target that is now secure boot ready, right? So you can absolutely prove secure boots working. You have no doubt whatsoever. This is secure booting Okay, which is a nice comfortable place to be before you start going into Yachto land and Ripping everything apart and doing whatever you're going to do with it So paranoia I originally started with a debbie in 11 system this version of the SDK says it supports a bun 2 1804 and that's it they also say don't use a virtual machine and So I ended up pulling out a nook that had and you know in the closet Putting a boot to 1804 on it and from then on all flashing and fusing and everything that I ever did was only on that bare metal system Okay, so that's what I will guarantee works Okay, so links to the console coversion of the secure boot tegra Guaranteed this stuff works. There's documentation in there and on how to follow the steps So a little bit about meta tegra. So if you haven't heard of it, this is a very Vibrant community that has set up the Yachto project layers to to let you build things for your tegra platform So it's really really well maintained and tested like any other open source project. We've got you know issues with Resourcing and so you can only do so much, right? So it tends to be on the development end of things a little bit So in order to do secure boot support you can read more documentation specifically about how to met meta tegra does it but basically it's as simple as Take those keys that you generated from that secure boot tegra repo. I mentioned earlier, right? Take those keys put them in your build directory and add this to your to your local.com So you're going to add the RSA private key and you're going to add your SBK secure boot key to that and The resulting image that's built is installed just like any other thing would be that you've built with meta tegra Because of the fact that you told it these keys It built that into all the scripts that are going to run and so when you run do flash It's just going to flash it for you and so at that point you're doing it with sudo and so Everything is just going to work and and if you've already flashed your fuses You don't have to worry about things anymore as long as you have these things in there So I was really it for it for secure boot, right? It was oh my god there's a lot of information this documentation and then oh my god I've bricked the board and then oh this magic and Kentation works and it always works So the next thing was the customer wanted disc encryption So There's good things and bad things in my opinion about Nvidia's Implementation so they did two really good things right they're using Luxe The Linux unified keys setup and they're using DM Crypt. Okay, so this is really well supported stuff or very common we use stuff They've got a bunch of scripts which are all Very convoluted bash scripts I don't think bash is the right tool for such convoluted things once you've started adding libraries and things like that You should be using a real language like Python, but that's just my opinion So you also have to rip apart your root of s to create an unencrypted boot partition such that you boot or see boot can Can open that up get the kernel running and then Unlock your root FS. That's encrypted Not so happy here, right? So now you got this window where Where you you're not secure really and and You're also able you know you've got this your kernel and everything in an encrypted partition So the Luxe passphrase one of the things I do like about it is that the actual passphrase is derived from keys that are in your fuses and in your EKS Partition that encrypted key storage partition and It uses a unique device ID. So all of this together means that the actual disc encryption key They're gonna be using is Pretty darn bulletproof, right? It's it's got like multiple levels of security already built into it So the decryption is actually done in the unit RD They have a Lux serve app, which is the client application that talks to this Lux serve app That's a trusted application that's running in the trusted OS and It unlocks the device mapper devices and switches over to the full root FS just kind of like normal So My first or my second war story, right? I went down a wonderful wonderful path because I thought Let's create a new image class Let's do all the DM the Lux encryption in this image class And so the resulting tar ball and everything that I've got is just ready to flash and it's already encrypted, right? but pseudo tasks run in In a not really pseudo root space, right? You don't actually have full privileges. It's kind of a fake Pseudo, which is why they call it that So we couldn't communicate to device mapper no matter what I did I tried a whole bunch of hacks and We couldn't ever create the Lux header. So we couldn't actually format the container and we couldn't start Writing encrypted stuff into it So I might be able to do this with QMU possibly The other thing here is once you start having AB partitions now you have to know everything about your entire in all your partitions and everything ahead of time and It's kind of a bad assumption to know What your A and B partitions are going to be right because they're supposed to be interchangeable So yes, the first time you burned this it could have been okay, but how are you going to do upgrades and things like that? So here's a link to you know that whole thing and you can look at a bunch of commits where I did some really crazy things So I had another good idea. All right, so I know I've gone down the wrong path. Let me try something else out So I wanted to keep part of what I liked about the NVIDIA implementation one of the things they had was they have a different Passphrase for every partition. I thought that was kind of cool, right? Rather than like most of our desktop systems when you're doing Lex encryption, you've just got one passphrase for the whole computer, right? So I created another image class to create it to rip apart the root FS and create the unencrypted boot partition and I did a hell of a lot of crazy stuff with var flags to store the UUID for each of the partitions and then I used that to Generate the Crip tab and generate the passphrases and a bunch of stuff like this, right? At one point I even started using XML lint and XML starlet to actually parse and Insert stuff into this flash XML file which describes what all the partitions are actually going to be But you probably already realize this is over engineered like a lot Like way over engineered, which is what I tend to do anyway, but really over engineered It got really hairy trying to do all the changes that needed to happen inside the Etsy Crip tab And I needed the Etsy Crip tab before I created the init in it Rd right the init remfs So it turned out to be this like nested, you know dollar squiggly bracket Variable expansion hell. I mean I spent hours trying to figure out how to properly, you know escape Characters and everything to get it to to work And ultimately still never even got to like Lex encryption I'm doing all this stuff to set up all the partitions and set up Crip tab and all this other stuff And I never actually got there So I've got two links there for for where this crazy stuff went By this point you can imagine I've been working on this for a while. I'd already bricked the board I've gone down to kind of dead ends. I'm getting frustrated, right? So I did find a solution and it wasn't I didn't author it I've augmented it, but what if we don't root rip apart the root FS? Okay, what if we don't require the unencrypted boot partition? Can we do that because that sounds better to me What if we perform the disencryption on the device Rather than ahead of time on the host, right? So if you're doing it ahead of time on the host that means your factory floor has to have the keys and Then they have to create the new root FS for every single device and then flash it That is not production ready, right? This is one of the biggest failings in my opinion of what you know, NVIDIA was putting out as their implementation The other cool thing is that this approach allowed us to sign for secure boot independently of the disencryption Completely a hundred percent divorced you can have either one or both together So bonus, I think it's way more production friendly because you can actually sign send your images off to a secure enclave that has you know your keys stored in a HSM and actually do the signing there and The images can then be installed on the factory floor without needing to know a whole bunch of information and special bonus OTA updates just work Because there's nothing special about your OTA bundle and It's already going to be installed on a on a encrypted system. So I Use Tegra test distro. This is there's another distro called Tegra demo distro They're part of the meta Meta Tegra community So this was created by Matt Madison. He's actually the you know the godfather of meta Tegra And it what it has is a two-stage approach. It starts off by booting into a system installer So it's got a special init remfs that Knows about your flash partition table and it writes those things out creates all those partitions creates the Lex headers it takes your real root FS and writes it on device into the encrypted partitions now and Bonus it installs a bootloader update. And so when this system reboots all signs of the installer operating system are gone Right, so now you're just in your purely secure booting system with everything disc encrypted So how long does this disc encryption take on disk? Like a minute Who cares right? It's nothing. So it's really cool So I really love the fact that it decrypted or say decoupled secure boot from Lux Because everything's happening on the on the device So what do you do to enable that so basically we're abusing a machine override by telling it to be crypt parts Right, so we're telling your machine You have this crypt parts capability and that just stands for crypt cryptographically signed partitions, right? We do need to have the tar ball that we're going to actually install and so Not just the normal tegra flash Which is the normal image class ease to to have the flash scripts and everything to flash to your device But we need the tar ball because that system installer is actually going to have the tar ball present and then it's going to install it All your Partitions that are actually going to be encrypted all you have to do is put a crypt dash in front of it and the scripts are actually going to Look at that and see this needs to be encrypted Add it to the crypt the secrypt file and things like this and it basically just kind of magically takes care of everything for you The one cool thing that I didn't know about at this time You know looking at these crazy flash XML scripts was you could use ID equals number and actually specify which Partition number to use this is kind of important because on a stock system. It's got 46 partitions Anybody want to guess what any one of them does some of them have some names that make some sense some of them don't so it's kind of confusing So once you've set these these things up You just do big big tegra system stalls This creates the special system staller image that has that special init ramfs that does all this auto install for you So this is not all fully upstreamed yet, but I've got Functional branches for Kirkston release and the jet pack 4.6.2 or Linux for tegra 32.7.2 and it's We've got some discussion happening on the actual tega test distro for a poll request a little bit about Little more details here about the eks image and things like that. So You don't need to create your own eks image With the the keys and everything like that until you get to lux disk encryption You just don't you can do secure boot all day long with the stock Eks image you don't have to worry about it The moment you set the moment that you fuse and burn kek2 that you're going to be using for the Lex disk encryption You must create your own eks image. You have to there is no option So the other thing is that the stock eks image is all zeros So the user key that's in there is also all zeros So even if you were to use the stock Eks image you still have to pass the user key and all zeros user key in to Do the disk encryption later on Following What had been so successful with the secure boot? I just added on to the secure boot tegra repo and created another Another branch that went into that adds some more scripts and some more test scripts and things like that to actually like make sure all this Lux encryption is working and again. I used NVIDIA's operating system So not only do you know for a fact you're doing secure boot, but you also know for a fact that you're doing lux disk encryption so you know that before you even started on yakto everything is is As secure and everything as you wanted it to be So my favorite part of this whole project was when I came to the OTA updates Because it just worked I did nothing special So we were using Raoq in this case Also works really well with Mender So because of the fact that the disk encryption is performed on the device Once your device is up and running and communicates to get your OTA bundle You don't have to do anything special right your device was already secure booted There's nothing magic, right? You don't need you just use your normal Raoq signed bundle Mechanism or or signed Mender bet mechanism, right? So you already have the ability to talk to that server get that get that bundle Install it switch over and you're done I can't tell you how happy I was that that was just Done it just worked like it never I never had any problems with it whatsoever. It's just normal procedures for OTA updates so Future work So I think it was like two weeks ago There's a new jetpack 5.0.2, which is the new general audience release and I was really excited about this because they switched to optee So more of an industry standard arm standard trusted operating system They switched the UEFI instead of their C boot bootloader or we had actually hacked things to work with with u-boot But it's just nice to have them standardized on something, right? And I think with system ready arm systems are going to UEFI being required and Also, no more 4.9 kernel it's 5.10, right? It's not 5.19 is not 5.15, but I'll take 5.10 over 4.9 any day of the week, right? Please The cool thing is they also ported the Lux serve trusted application the Lux serve app client application and the hardware key agent and hardware key app And so Those are there. They're ready to for us to you know, do some more testing and everything in the Tegra community but But it's promising that I might actually be able to get all of this to work on the new system pretty simply, you know famous last words So they moved from C boot to UEFI 4.9 kernel to 5.10 kernel and You can at least use it in Ubuntu 20.04 host But you're not stuck on 18.04 because every time you run 18.04 it says, hey, there's no more updates Are you sure you want to run this kind of you know stuff? The only caveat is it only works on these Ajax Xavier Xavier NX and the new Ajax Oran Platforms so the old TX one TX to you and so on those those aren't supported anymore. I Think Nvidia just want you to move only to the new hardware So the other thing that I'm working on right now is actually taking the components that I have in that that feature branch that Totally worked for tip for the test distro and getting some stuff into The demo distro, which is the more community supported thing that goes through CI and that all the time So there's this special the thing the thing that does all the magic in that auto installer is Tegra system stall. So that's gonna have to be added. I Need to add in, you know Into these into Tegra demo distro. We need to add in the Crip in it ramfs You need to make sure we got CI and test actually like making sure this stuff's working, you know as much as possible and In the short term, you know just because I already have it all working, right? I know it works I'm gonna be pushing up the existing work that was fully functional get that out there and then we're in the middle of Working with Langsdale the new Yachto release 4.1 release and the jetpack 5.0.2 stuff Mostly the jetpack 5 is working Meta Tegra community. We're all working on this stuff Testing it out. There's a couple of hiccups here and there, but you know looks pretty good so far There's a couple things that don't quite seem like they should have been claimed to be production But it's not too bad But in particular, I haven't tested any of this stuff out because as a contractor I've moved from project to project. So, you know, I'm not we're not working on this specific thing right at this moment But but all this stuff will be up streamed. So I Really want to thank Matt Madison for all that he does for the open and better for Tegra community and specifically the stuff that I got to piggyback on top of from Tegra Chess Distro and so on and you'll use Chagui who He did a lot of early testing on stuff that I was trying to get out there for the community and Found some typos and things like that. So it was really helpful to have that and I really want to thank the OE Ford Tegra community for Just being really excited that I was working on this and and really being excited to you know Answer questions or test stuff out and everything was just really awesome, you know I think we're all here because this is open source and we're all into community and It's just awesome when that's reinforced by a community. You've never worked with before and They just accept you and and work with you on it and that was just awesome So, I think we've got some time for questions About six minutes. Yes, so the nit ramifest could have been signed But the The way that Nvidia was booting it. It was actually booting From the bootloader Into the unencrypted Boot partition and then from then on it was encrypted, right? so the reason they had to do it that way was the way that they were doing the so they were doing the Encryption first and then signing everything and then then flashing it So because of the fact that we're doing those two things completely separate and doing the encryption on target That meant that the signing Is completely a hundred percent independent so all the secure boot and all that kind of stuff that was You know was needed in order to be able to even unlock the unit ramifest all that stuff's just already there and So basically, you know, see boot just hands straight off to you In a net ramifest which has already got the ability to unlock the the Partitions because it can talk to the trusted Applications, but it's signed Yes, yes, so that partition is signed so that partition could never have been loaded It would barfit you because it wasn't wasn't signed. Yeah So I Will say that I did all I did a lot of the work without ever doing any secure boot I just did pure lux disc encryption. So in that case, yes, I was vulnerable to the attack you're talking about But ultimately once you were doing the full secure boot and everything The init ramifest and everything was signed Yeah, so the question was what is the unit what is the unit system being used and in the unit ramifest So this is one of the things I really liked about Matt's Implementation because it's all it's all system D Okay So I know that people have they grown about system D because it's a little bit heavy on embedded systems I understand that right, but everybody's moving to system D these days and System D really feels nice to me like it's just I understand it really well And I'm not hacking at shell scripts all the time so I've done some really really hairy work on init RD systems that were Sysfi in it and it It was just so much extra work To make the make sure the shell scripts worked It I just don't think it's worth it anymore. So I system D made so you you benefit from a bunch of stuff about crypt tab and FS tab and all this other stuff, you know and the The DM crypt stuff that's already built into system D All of that stuff came along for the ride for free and it made this really much easier to work with Yes Yes, so the question is Is this implementation able to flash directly to an NVMe? So yes, so I did not have an NVMe on the particular board that I was using so I didn't test that path out Yeah, so it's it really doesn't matter what the storage media is so SD card or EMMC or NVMe would work as long as Because basically the most of the magic is is the stuff that's flashed into So Obviously you have to have the fuse is set up and also you have to Have your system set up that it's going to boot to whatever the device is that you want to boot it to But it should not affect the ability to flash So the flashing scripts from meta-tegrist should have flashed the same auto installer operating system onto your NVMe and then once that booted up it knew because of There's there's a service that's similar to what system D has but it knows about the boot devices that are going to be coming up and then it uses that to to do the actual Final Crip tab and the disk encryption. So there's a lot of details here I had to gloss over because it's only so much time in the talk, but I'm I'm quite confident that that would be no problem Couple more questions. Can you repeat one more time? Yeah So I think it all comes down to what capabilities pseudo is allowed to have right and and So I couldn't run LO setup Right, so that's what I had to do LO setup and so yeah I'll just say I spent enough time hacking at it that I thought I was burning customer dollars and so I just moved on I'm pretty sure there are other ways around it. I'm not sure pseudo is going to be the best hack I think it'd probably be better off running Q mu as part of the build system In video did everything in a charade, but you know it just started to get Pretty ugly. I did some pretty pretty heavy hacks to try to get things to work And it just wasn't getting anywhere short of actually patching pseudo so You know, it's it's possible. I'm not I'm not sure To be honest with you. We had at least two others. I'll go to the back. Oh Sorry time is up So I will be at the yachter project booth the next couple of days and you can see me in the hallway And I love talking to people so just you know or find me on Twitter or whatever I'm really happy to talk to people And that's it. Thank you so much