 Good morning, everyone. Let's go ahead and get started. So I'm George Wilson. I'm the IBM Linux Technology Center Security Architect and Security Development Team Lead. And I'm in the portion of what got put into the power organization of the old IBM Linux Technology Center. And we work on power systems pretty much exclusively nowadays. This is not my talk. This is why my name is not first on the slide. This is actually Sudhakar Kapusami's talk. He's a developer on my team. He's the primary secure boot developer nowadays. We have some other folks on our team, Shubham Pandy. Nina Jane's been developing CISFS code and Eric Richter has worked on Secfar Cuddle. There are many people. I have a credit slide towards the end. So since this is not my presentation, I will stumble through it and just ask that you be nice to me. So the agenda today, I'll go through a disclaimer, objectives, an intro to PowerPC secure boot, how we are managing keys dynamically in the very near future, and the tool that we are using from user space to generate updates and submit them. And then there's a little demo of generating the updates, setting the variables, reading them, and actually performing a secure boot. And then I have a few thoughts on the roadmap and futures. So I just have to say that these are the views of Sudhakar and me and not necessarily IBM's. Nothing here may ever make it into a product in the form described. And then people's trademarks are their trademarks. So this is, I think, the third talk in a series of secure boot talks I've given here. About five years ago, we developed a full secure boot solution for OpenPower, which was running directly on bare metal. But we had never developed a guest secure boot solution. And now we are refocused on PowerVM. And we need secure boot in our L-Pars. We also need it for KVM guests. And primarily to meet the requirements of the operating system protection profile, 4.2.1 and later, that the integrity of the OS kernel be validated before handing control over to it, we developed a static key solution. Now as you might imagine, it's a real pain because you have to coordinate key rotations between firmware and operating system. So not where you want to be. So it's been available for over a year in distros. We haven't widely proclaimed it just because it's got a number of limitations, the key rotation and third party module support being real problems. So we're now developing a new secure boot solution with dynamic key management that builds on the static key solution. And it makes use of a new mechanism that we have in PowerVM called the Platform Key Store, which gives us a little bit of authenticated storage where we can stuff secrets or things that are not secret, things that we want to protect like public keys, where we want to protect their integrity. So we're making use of this new mechanism. And we've created authenticated variables somewhat like UEFI, but not UEFI. And that's a really important point, and I'll go through that later in the deck. And we hope that by hearing about some of our problems and solutions on a non-X86 non-UEFI platform that you might find some value in it. So moving on. So the purpose of PowerPC secure boot as you might imagine is to protect the bootloader in the kernel against modification attacks. You might also call it verified boot. We call it secure boot. Sometimes we also separate out the aspect called trusted boot where we perform measurements, extend PCRs and log events, trusted computing style, which we call trusted boot or measured boot. So we are using appended signatures. We don't have PECoff. So we can't just leverage the existing tools and framework. And moreover, we need backwards compatibility with older bootloaders. So using the appended signature allows us just to stick the signature on. If it's an old bootloader, it can ignore that. And this has proven to be a really useful thing. It makes use of the same sign file tool that signs Linux kernel modules. And we used it in open power to sign the kernel. We had a built-in bootloader in open power called Pettyboot that shipped with the firmware. So we didn't have to worry about the bootloader. It was verified separately as part of the firmware. But now we're using it to sign Grubb and the kernel. And it's a PKCS7 message with some metadata and the magic string at the end. Same magic string. And then that can be used to verify the kernel and the bootloader. So this is what it looks like. You might be familiar with this if you've looked at the format of signed kernel modules, where we have the actual payload up front. And then we have the suspended signature at the end with the PKCS7 signed data. And then we have a size and metadata and then the module signature appended at the end. So exactly the same thing. We're using it for Grubb, even though it says module signature appended. All right. So this is the current state of affairs for secure boot. The hypervisor has embedded keys. And well, actually the firmware has embedded keys, partitioned firmware. And the hypervisor brings up the firmware. And then the firmware verifies Grubb. And then Grubb further has embedded keys and an appended signature. And that appended signature is what the firmware used to verify Grubb. And then the Linux kernel further has this appended signature. And Grubb can use the embedded keys to verify it. So yeah, it works. Not scalable. Very painful to manage when you have to do key rotations. And we have had to do key rotations. So how can we improve things? So this is what we're currently developing. Now, users will be able to submit key update requests with a user space tool we call SecFarCuttle through the kernel, through a CISFS interface, and store the key material in the secure storage, which is actually, we call it pks. We just talked about pks. Yesterday, it's a different pks. We have acronym overload. So in the kernel, we call it plpks, powerlpar, platform key store. And once those are stored, then we can, from the bottom, the hypervisor can bring up the firmware. The firmware can read the GrubbDB and GrubbDBx authenticated variables, look at that appended signature on Grubb, verify Grubb. Then Grubb can read the DB and DBx and use those to verify the kernel using its appendage signature. We support nine variables right now. They're all predefined. There is no flexibility to create new variables right now. I'll talk about why in a little bit. But it basically allowed us to simplify assumptions about the code and reuse some of the open power code that we had. And firmware uses the SBAT, GrubbDB and GrubbDBx. And Grubb uses a DB and DBx. The Linux kernel loads DB, DBx, module DB, and trusted CADB into respective kernel keyrings, which I'll talk a bit more about later what those actually are. And the authenticated variable naming maintains compatibility with the existing tooling for open power. So why are we using DB and DBx for the kernel? It's an artifact that we didn't have to worry about bootloader and open power. So it's a little similar to UEFI, but it's not UEFI. So these are our authenticated variables. We have a PK, which is the ultimate authorization and indicator of secure boot state. We have a kek. We have this DB and DBx, but don't read too much into these names. They don't mean exactly the same thing as they do for UEFI. We borrowed concepts and we actually borrowed code from TianoCore since it's got this nice BSD2 clause license. But we don't have a central signing authority. We don't have a shim. We don't have a mock. So you can argue that that's a good or a bad thing. We've seen some sort of escapes recently with the central signing. So maybe decentralized signing is not so bad. And the idea here is also that administrators can easily sign their own kernels and their own bootloaders. So and we deliberately didn't want to get into the business of being a central signing authority. And I talked about the awkward use of DB and DBx. It's just maintaining compatibility with our existing tooling. So we have all these variables, which is kind of akin to UEFI. The GrubDB and GrubDBx are used to verify Grub. We are using SBAT. SBAT turns out to be a really important concept. Because much like on x86-64 systems, we have limited storage, even for this. We'll be expanding the PKS, the platform key storage in the future. But right now we're fairly constrained. And SBAT is just really, really pretty nice that you can just block a whole set of things from the past based on a generation number. So that's a very important thing that we implemented. And as I mentioned, DB and DBx names are artifacts. The trusted CADB and module DB are used to verify third party signed modules. And this is really important because if you're trying to boot a machine and you need to get to the disk, you have to have some way to get to the disk. And you need to do that in a secure way. And not all drivers, unfortunately, are shipped in the distro. And they're not all part of the kernel. So we do have third party modules that we have to handle, so we've got explicit support for that. So here's a diagram of a verifying grub from the firmware. So grub, or the firmware detects the appended signature in an ELF note in the grub binary. So we're using ELF notes to add things to ELF binaries. So we have one called appended signature and type ASIG. And so the firmware is able to see that. Then there's a separate ELF note for the SBAT data. And it's literally just encapsulating a CSV, which is the way it works on UEFI systems. And so the firmware reads these variables into various lists that it maintains. And then it verifies the grub binary using the signature trusted list, distrusted list, and SBAT list. So you can do individual revocations with hashes and certificates if you want to, but really I expect we'll rely on SBAT strongly going forward. So then further verifying the kernel from grub. Grub reads the DB and DBX from PKS into trusted list and distrusted list, respectively. Detects the appended signature from the Winix kernel image and validates the appended signature magic string and then reads the signature and then verifies that using the appended signature, the distrusted list and the trusted list. So the kind of chain you might expect. And then further we're reading trusted CADB and module DB from PKS and putting it into the machine key ring and secondary key ring, respectively. And then it reads the DB and DBX and stores them in the platform key ring and the blacklist key ring, respectively. The kernel then verifies the modules using the secondary key ring. And then it can also verify the next Kexect kernel. So we created a tool. We wanted at first to reuse existing tooling for UEFI and the Shem, but it turned out not to be really easy to sort of map into that. And we thought it would very likely be confusing to have similar names in the same tool that don't mean the same thing. So we created our own tooling out of convenience. And it also gives you a single tool, not multiple tools you have to deal with, to deal with the authentication variables. So it simplifies the process of creating the updates and writing the key material. And it allows easy communication over this SysFS interface with PKS that we created. And it supports these nine predefined variables that I talked about. Yes, we have to manually accept the key material. We need to find variables that I talked about. Yes, we have to manually expand that in the future if we want to on both the tool side and the kernel side. And, yeah, we'll improve that in the future, I hope. So to update and get secure boot variables, we have a specific auth file format. And for the PK, the module DB, the CAC, and the trusted CADB, the X509 public key must be an ESL. So once again, we're actually reusing structs and a little bit of code from TianoCore here. The DB and GrubDB can be an X509 certificate or it can be a binary hash contained in an ESL. For DBX and GrubDB, it can be a X509 public key certificate or the binary hash or a hash of the key. That's prohibited. So we have multiple ways of blocking things we don't want to use. And then, of course, SBAT. Is everybody familiar with SBAT in here? I don't know if you are or not. Well, I really wasn't until we started looking at Grub, but it's a pretty cool way of not having to, basically, the problem became that you need to do a bunch of revocations and now you've got a bunch of either hashes or certificates you've got to store and you can exhaust your, well, in the UEFI case flash memory doing that. So they invented it. The Grub maintainers invented a new mechanism to compress that and basically say I will not run anything older than this generation level. And PE coughs already had generation information in them, so we just sort of borrowed that and brought it over to ELF so others may find it useful. But yeah, SBAT is actually really, really cool if you haven't looked at it. So this is what the auth file looks like. This is what we put our updates in. And it's got an append header that indicates whether the update is a replace or append. Those are the two operations we support. It's got the auth info at the top. That's a signature that authorizes the update. And we didn't show the secure variables, but as you might imagine, there's time stamp and authorization information in the variables. I wish we had a slide on that, but we don't. And then we have the ESL info at the end of it here that actually contains the key material or the SBAT CSV. And it actually, yes, it is a CSV text file. Deletion is a special case. We pass empty data in the ESL. These are what some of the commands look like. You know, kind of what you might expect. You might need arguments to generate the updates and write them out to files. So I'm going to get to the demo section here. I have to go out of fullscreen. There we go. And I don't have control over this as it plays. So, yay. So first demo is going to show creating the PK-KEC DB and other variables, and then it's going to read them back. So let's try that. Well, it may not go. There we go. So here we're creating the PK. Create the KEC, and then we're creating the DB and that's authorized by the KEC. These are all using predefined keys, so sorry. It's a little bit fast, and I can't pause it. So we're just basically generating all our key hierarchy from predefined keys here. And then we can show the contents of them. So we'll see that if we look at the PK, and we have our ESL down at the bottom, and I think he's going to show this for multiple of the variables here. It's kind of what they contain. Same thing for the KEC. This is DB authorized by the KEC, and I wish I had better narration for you, but I don't. And this is ESBAT, and you can actually see the CSV there at the bottom that contains the grub generation data, and we're doing this on Fedora. So that was the first demo. Let's see what we get back to here. There we go. So here's the update scenario. So we actually, there's a flag to the verify command that requests the update strangely enough, and we might want to make that an explicit command, but that's the way we're submitting the updates right now. So this is actually writing through the CISFS interface. It verifies the off file, and then it writes it out through the CISFS interface. That's the new interface that Nina Jane started, Andrew, Donovan, and Russell Curry on our Australia team completed and submitted upstream, and it's now in the kernel. So after these updates get written to the PKS, we can actually read them back, and this is what we're going to use when we actually boot up the bootloader and the kernel to do the verification. And it's not a general purpose. It's not a very nice CISFS interface. We can't create files in there or anything, and we talked about some kind of Baroque mechanisms we might be able to use to create new entries in there, but that's something we'll think about doing in the future. But for right now, we're just fine with having the predefined variables. So we've read the variables back successfully, I think. No, he's just written them now, so he's going to read them back here. So he's basically showing that we get back what we put into it, and I think he'll show the SBOT, and you'll see the SBOT CSV once again, which he's written it out, and finally, yeah, SBOT. All right, I think that's this one. All right, and then we have one more, which is especially exciting because it actually shows the kernel binning. So we've established the keys now using our tool and the CISFS interface and PKS. Now we're going to boot up Grubb, and we'll see that we get some messages here. This is a Power LPAR booting up, what it looks like. And we've actually now got some new messages here showing that we found keys, and Grubb wouldn't have come up had it failed the validation, and we're not showing the negative cases here just in the interest of time, but it does actually fail if you tamper in any way or don't have the keys present or don't have the signature present. So you can also look at the keys from Grubb, and I think this is the last thing he's going to show here. He'll actually list them out here and see them. And then finally, I think he just boots up into the kernel. So we're getting very close and to having something that's reasonably feature-complete and I know we'll have some additional work we have to do to complete it, and I talk about that at the very end on a slide. Let me get done with this. I guess I'll finish in a couple of seconds. There we are. All right. All right, so the roadmap. Ignore the kernel versions on here. We've already missed them. And I now imagine that the kernel pieces are going to largely go into 6.5. But we have a firmware piece. We have a new firmware coming out that should be out later this year that will have the dynamic key management support for a secure boot in it. PKS is already there. It's just we have some additional processing. We needed to add to the firmware to process the authenticated variables. Grubb, we hope to get upstream review in June. Grubb is constrained resource-wise. There are not many people participating in the Grubb community, and that's a real problem for Grubb. It's something we've run into repeatedly as we've gotten down into the space. So we're hoping that around June, Daniel will be able to look at it. And we can get it into an upcoming Grubb release. The distros both... Well, the distros tend to have downstream Grubb repositories which sort of tells you about the dysfunction in the upstream. So things tend to get applied there. So they may get applied into these downstream Grubb repos before they actually get applied to the upstream. But they at least need to go through review first. The PLPKS and PLPKS system, or CISFS interface patches, have already made it into the kernel. So that's available now. The third-party signed module support for the trusted CADB and module DB should be in likely the 6.5 kernel. One of the interesting things that we ran into is we have this thing called the runtime abstraction services. And this allows you to manage your logical partition in various ways. And it's got a privileged user space component that up to now has talked through DevMem. And you can imagine how well that plays along with lockdown. So immediately, you turn on lockdown and you can't see how many CPUs you have. You can't dynamically change anything on your LPAR. And yet, you don't want people to have access to DevMem. So right now, you either break RTAS and can't effectively manage your LPAR. That's one of the uglinesses of our current solution. Or you poke holes in DevMem just for RTAS. And you obviously don't want to do that. But that's all you can do right now. So there's a new message passing interface to RTAS that's being developed by Nathan Lynch in the LTC there on the kernel team. And that's unlikely to be six... Well, it's not 6.4 either. That's probably 6.5 or farther out. But we really need that function to have a nice management experience for system administrators. And the corresponding changes in libRTAS. And arguably, applying lockdown and finding this big, ugly interface that we... I didn't really realize existed was a really good thing. It's just like it breaks things when we lock it down. PsychFarCuttle was made available as part of the OpenPower GitHub. And we're just simply updating that. And it'll handle both OpenPower and the LPAR or KVM Guest cases. And then we have another component that's libSTB.seqFar. This is kind of a confusing one. So we just open sourced this. However, it's included in our firmware. So our firmware is proprietary and we don't, you know, make that available. But we wanted the whole secure variable lifecycle to be reviewable and auditable. So we made this available on GitHub. You can actually build it and you can do testing in user space if you care to do so. And it might have some value for other reasons in the future. But that's actually included in the firmware. So FutureWork. We need to sign and verify the Grub config file. Grub can currently use GPG detached signatures. But you basically have to sign everything with GPG and nobody seems to be really using this. And everybody, rather than signing the Grub modules, they just simply have to build them in. And we ran into this with the TPM2 support when we wanted it on power. It's like, oh, it's not there. Distros, can you please build this module in because we can't load it? So we're not actually planning on using the GPG mechanism, but we're going to use detached PKCS7 signatures very much like a detached Appended Signature to do this in the same style we've used for the bootloader and the kernel. And this is a real important piece in having a full chain of trust all the way up. And it's a deficiency across the board, really, across architectures unless you're using this GPG mechanism or some other mechanism to verify it. We need some way to sign and verify the NRMFS. And I know we talked about this at various points. I think in one of the buffs we were talking about this, how we might go to user space to do this, right? But yes, I want extended attributes in the NRMFS and then we can use IMA to verify the files in it. There are other things we could contemplate. There's the unified kernel image potential there, but right now that is UEFI-specific. And we still need third-party modules for storage and network drivers and maybe even a crypto adapter if you're storing a key material and that that you need to use to boot. So we're not, in my opinion, we're not going to get away from the NRMFS for a while. So we need some way to secure it and IMA provides a really flexible way of doing that if we just have the CPIO support. So I hope we can find some way to make that happen. Kernel SBAT, I don't know if it makes sense to extend the SBAT concept to the kernel. This is something we're thinking about so that we could revoke the kernel generations that may be vulnerable, old and vulnerable. And we're evaluating whether we want to do that or not and we would probably do it in kind of a similar way that we're doing it in Grubb. And then a non-goal is cross-architecture key management interface unification. So when Nina proposed the interface for PLPKS, Greg KH said, great, I don't want to see another arch-specific solution. So could you please come up with some unified solution for this? Talk to the cocoa people. Well, our problem is a little different than cocoa, right? We're not injecting secrets. We've got public keys we need to protect. And so we actually did an implementation of something that would be usable for a generic interface, then it's like, well, you need to have at least three users of this interface for me to bring this into the kernel. And what's the incentive for X8664 to move to a new interface? Absolutely none. Things work. So we went back to the drawing board and said, we're just going to have a simplistic approach and extend the existing open power interface. And we'll have fixed variable names and we will have minimal code required to get this into the kernel. And that's not necessarily a bad thing from a simplicity perspective. So that's where we are now. I'm not sure if it will ever make sense to try to come up with some unified key interface, but yeah, we at least had that discussion a fairly long discussion about it and ended up just doing something arch-specific anyway that worked for us. So that was a little less satisfying a solution than we hoped for, but anyway, that's where we are. So we do stand on the backs of giants and many people have contributed to our work over the years. I would especially like to thank Sudhakar because he took over work when Daniel Axton's left our team. Daniel contributed major portions of the dynamic key solution, so I'd like to thank him. And also Russell Curry and Andrew Donovan on our OSLabs team who've worked with us on CISFS patches. Nina Jane and Eric Richter worked extensively on OpenPower and completed the OpenPower secure boot and have made major contributions to the dynamic key management work and many, many others in various parts of the organization. So thanks for listening today and if there are any questions I'll try to answer them and I'll have Sudhakar on Slack to bug if I can't answer them. Hi. So I have two questions. The first one is so it's sound like the signature of the grub binary is embedded in the grub binary itself in an alpha section. So I'm wondering how you're exactly how you're verifying the grub binary if the signature is potentially located somewhere in the middle of it and the second question is whether you're also authenticating the kernel command line. I was going to ask about the grub config file too but that was mentioned in the future work. Right. So yeah, kernel command line we are not locking down in any way. Hopefully we can mostly rely on the grub config file. What else would we need to lock down on the kernel command line? I'm wondering if we lock down the grub config file. We need to do command filtering in grub is basically what it that comes down to I think. We had talked about doing that limiting the commands in grub so that you couldn't alter the kernel command line potentially but we haven't gotten that far yet. But that's, yeah, we have discussed filtering for the appended signature in grub the elf note we have elf notes that indicate I didn't go into all the details but they indicate the position and length of the appended signature in grub and it's always stuck at the bottom but the firmware has difficulty like you said, figuring out where it is so the firmware actually looks for the elf note to figure out the position and size of the appended signature at the end. And that's how it does it. Hopefully that answers your questions. If the doctor might have a better answer for you. Hi. When you're updating the keys do you use timestamps? Yes. Yeah, we do. We also maintain that state in the secure variables and that's why I said I wish we had a slide that actually showed the actual format of the secure variables in the hypervisor memory so that you could see that but yeah, we do maintain a timestamp so we can prevent replay attacks. Yeah, absolutely. This is a bit of a meta question. I couldn't find the slides for this. Oh, we need to post them. Yeah, some people have said they've posted them that they haven't arrived but just maybe a reminder to folks it's definitely interesting. We need to post them. Yeah, share with people. So you talked about your secure boot story which we haven't completed. This is also work in progress. We have certain measurements and we're not measuring the certificates right now and we know we need to measure the certificates and we haven't done a thorough analysis to see what else we need to measure but yeah, we now at least have the kernel measurements and grub. So we know that the certificates are an incomplete portion of the chain so yeah, we need to do that until recently we didn't have grub so we had a really bad story there so that's incomplete. That's kind of why I'm glossing over that a little bit and then we also need to test it out to end attestation with key lime or something which we haven't really done either so we definitely have more work to do on that portion of it as well. All right? No more questions? All right, thanks everyone.