 Okay. Welcome back to the next session after lunch. This is Debian EFI team about the status of secure boot on Debian. And please introduce yourself. I'm Ansgar Wuchert. I'm one of the FTP masters and did some parts of the archive side of the implementation. Hello and Helen. And I help with, I did part of the signing service implementation. Hi, Steve McIntyre with a little bit of echo. I've done bits and pieces on various packages involved here and we were involved in the discussions about how to do things. Hello, I'm Tola Foggehir. My primary hat here is probably the DSA hat, but like everybody else, I've been a little bit all over the place. Hi, I'm Luke Farone. I'm also one of the FTP masters and I was all over the place but primarily on the signing service. So we are here to present a report what we worked on the secure boot sprint which was in April in Fulda, Germany. And we made a huge progress actually during the sprint because we got something functional at the end. Previously we had several suggestions for the implementation but there was a disagreement but we agreed on a solution. So if you want to check the notes for that sprint you can go to that link, the salsa. And also if you have comments and questions, so please go to Gobi. So the link is there. So the overview, we're going to start with the context then the sign infrastructure that we came up with and the current status of the services. So just as a context, the boot sequence is like the firmware then firmware loads the bootloader and the bootloader loads the system. In our case, most of the cases, we have UVFI as the firmware that loads group, that loads the names kernel and modules and then the user space. So why are we doing secure boot? What's the point? So the point is to stop a remote attacker from tampering with the boot sequence on your computer. There's been a long history of viruses and other malware tampering with bootloaders on systems. And so you get to the point where you can't actually boot your system safely because before you have any chance to interfere, to validate what's going on on your system, the malware is already in place. So this is why secure boot happened was to provide a safe path for guaranteed secure booting locally. This is not. So as you can see, quite a lot of wine, we validate each stage of the boot. The firmware itself has a set of certificates. It can use those certificates to validate that the next stage in the sequence, the bootloader, is trusted. It doesn't necessarily have to be trusted by third parties. You can also include potentially your own signatures here so you can run your own software too. But for most people, this is going to be stuff that is built in as a system ships. Ditto, then the bootloader will have a set of certificates built in as well so it can then validate the next stage, whether that's going to be operating system kernel or potentially other tools as well. So this works really well for remote attacks. It doesn't work for local attacks because fundamentally, if you have access to the hardware, there are all kinds of things you can do. This does not stop that. It's not trusted boot, it's not measured boot, but it gives you control, so you actually have to have reasonable access to the computer. Something happening once you're in Linux is not going to break your boot. So that's the general goal. We only want to boot signed binaries when secure boot is enabled. And in Debian, we want to have some infrastructure, which we're going to go into more details of in a moment, so we can sign specific packages only. We have a whitelist. Not that we don't trust all the DDs, but equally, we don't need to be signing all 28,000 source packages in the Debian archive. There's only about half a dozen. And then the final point of this, and this is actually possibly the real reason why we're going here, is we want to make installing Debian easier for users. Yeah, it turns out that life's not super convenient here. Vendors don't actually ship with our keys installed into them. Getting them to do that is basically impossible. So the current process today for installing Debian consists of either disabling secure boot, which is not that great. Or we need to get people to enroll our certificates, which involves talking to firmware, and firmware generally isn't very user-friendly. None of these is super awesome for newcomers. And of course, it's not really built for remote deployments and automated deployments, because interacting with a firmware from, like, interactively is bad, doing it from software is even worse. But we actually have a friend here, and they're Microsoft, because everybody ships Microsoft CA in their firmware, because LG can't install Windows, and people generally, for some reason, want to install Windows. So users are allowed to install their own certificate, but also Microsoft provides a CA, which will sign a UFI binary, which we provide. And so this should be easier, right? We can just get them to sign Grub. Except it turns out that MS does not actually want to sign GPL version 3 code. Grub is kind of large. It changes, bug fixes, new features, all kind of stuff. And this is a manual process. It's not something where we can just have a buildy do it. It actually requires you to poke a website and pack stuff up in cabs and upload them. And it's less bad than it used to be, but it's still not great. So Matthew Garrett created a few years ago, a book around for this, which is the Shem, which is a really simple bitloader. It allows you to enroll your own keys, but also it contains, in our case, the certificate of WN3CA, which is only used for secure boot. It's not the general CA in any way. It's small. It's audible. It doesn't come out with new versions all the time. So even though we still need to do the manual process of getting this through Microsoft, whenever we want to release a new version, it's less of a problem. So what we have here is we get the Shem, get signed by Microsoft, and Grub and other bootloaders, they will be signed by us, and the key we control. So what we get here is we have UFI checks the signature of Shem, Shem is signed by Microsoft. Shem verifies the signature of Grub, Grub is signed by us. Grub verifies the signature of the colon, the colon is signed by us. And the modules are signed by us and the colon checks the signature. It would be the same for any other bootloader or other non-bootloader-ish things like firmware updates and so on. So this allows us to not disable secure boot and not be too scared to users. And we're reasonably sure that nobody's actually tampered with the boot sequence. So it comes to the actual signing infrastructure. It's very straightforward to implement all of this. If you have complete trust in all of your own infrastructure and you can just make sweeping changes, but in Debian we have buildies all over the world in different sort of security contexts, so we needed to have a little bit of a different approach. So broadly the way it works as we've implemented it is a kernel maintainer or grub maintainer or firmware update maintainer will basically build their source package for say Linux image, or like Linux generic. And then that will be uploaded as usual. And one of the binary outputs will be a, what we call a signed template. That template is then pulled down by the signing service. The template contains some metadata that indicates what exactly is to be signed. Signs it, then sends it over. The source package containing the detached signatures sends it back to the buildee and the end result is we get a Linux signed image generic. That is a bit complicated, so to go a little bit more in detail. The signed source package basically just contains detached signatures. And the goal here was to have an architecture where our signing service doesn't run any actual code. So it will have a build dependency on the already built binary kernel package and when it hits the buildee it'll just do that combination and then publish it again. And this preserves the property of reproducible builds in our infrastructure. And so the way the communication ends up working between the DAC and the signing service, when DAC gets a upload from a buildee of Linux signed image template, that will then go into a queue. That is then read by a signing service which currently is co-located but in the future we intend to separate it out. It processes it, then de-puts it back, it goes through new. And amusingly the signing service is the only key in the Debian archive that has the ability to upload packages that are sourced new and bypass the new queue. So we have both a machine-generated source package that doesn't get DAC-side reviewed. But it's fine because all the codes under our usual Debian infrastructure. Next. And so this template package structure. The reason we went for this template package was to allow the kernel maintainers or other maintainers with packages of being signed to get control over what the result of the signing operation looks like. So they include a, in their binary package, they include a template for this source package that contains some specific metadata that lists the files assigned. And again the signing service only runs commands that do not actually end up executing any code in the package. So we don't have to worry about there being something malicious that we would then sign and then could possibly compromise our signing key. Yeah. So the signing service current status. So the code is available at Salsa. We basically code it during the secret boot sprint. The current state is functional and it's deployed in the experimental suit. Currently it works, it signs the package with the Debian key, it's not the official key yet. And the process is manual for now. And we also have an audit log that will log everything that we signed in a PostgreSQL database with public database dumps. And so we have some work to do there. So we need to improve some tests. We currently have a basic test with that signs dummy package that we put there. And also we need to think about a signature revocation process that we don't have. And we need to start deploying in other suits like stable, testing and unstable. And it would be really nice to have a browser for the audit log so everyone can check which were the packages that got signed. So on the DAX side, well, I'm mostly out of practice, but DAX publishes that list of packages and it will accept those source only uploads. And the key point here is we actually have multiple DAX running in Debian. We have one for a separate one for Security Master. Sorry, that handles a market uploads. And basically the structure of the signing service is that it will essentially break if the version number is be used between Security Master and F2P Master. So don't do that if you work on a barcode packages. So we've identified a smallish list of the packages that we think should be signed. Obviously as the very first thing in the queue, there's Shim. We have the firmware update, pairing of packages. We have Grub 2, which is our standard boot loader. We have the kernel. We might sign system reboot as well. That was something that we've just been discussing literally this morning. It depends on whether or not we consider that useful and whether or not we think it's safe. It's a discussion yet to be had. So quick status update on each of those packages. We have Shim signed by Microsoft Key already with our root certificate. That actually happened quite some time ago. It's currently not reproducible due to the way it's built, but we have patches already to improve that. Obviously this is a key area where we want people to be able to reproduce what we have because we desperately want people to trust what we've done. If you don't trust this set of paths, you may as well not bother at all. We're currently on an old version of Shim. Things have moved on in the last couple of years, so we'll have to get a new version built and signed. Obviously, as we were saying earlier, we don't want to be doing this like every week, so we'll have to work on that. Firmware update. Are there packages that enable secure firmware updates during reboot? The way they do that is they drop an EFI binary into your EFI system partition, and the firmware should go looking for those as it starts. This is how, for example, you can have a secure update of your BIOS, knowing well that it's signed by your system vendor, by Microsoft, whoever. It's not just for the BIOS. It could be for other devices on your system as well. We've had these packages in Debian for a while, but what we've done now is we've added support for building the signed template packages. We've already described that normal binary packages will recommend the signed versions, so you should get those by default. We're not going to make it a hard dependency. It's up to you if you don't want them, and we will install the signed versions of each of these binaries in the system partition in preference. Obviously, if you installed it, you probably want to use it. The most recent uploads to Unstable last week have all the needed changes. This is going to look quite similar to the last one. Now, we have made the decision specifically. This is unlike where Ubuntu went, but we had a long discussion about this. If you're booting with Secure Boot Enabled, we will only boot a signed kernel. We will not fall back to an unsigned kernel and call exit boot services. That is a deliberate security decision. We don't want to change that. The changes to the grub packages we now build a monolithic grub image. What that means is it has all the modules built in in one piece. It's much easier and much safer that way if you're going to have a signed output. Otherwise, we then have to worry about signing all of the grub modules individually, and that's such a mess, it's untrue. Again, we've added support here. Just the same way as the firmware packages. The most recent upload to Unstable, literally last night, contains all the needed changes. Ben's done a huge amount in the kernel. Again, he's added all of the extra support that we need. At the moment, the kernel signing stuff is only enabled for experimental uploads. I'm expecting that at some point in the next few days, hopefully Ben will give us something that will go into Unstable. We have signed everything through experimental. This is one of our tests. Coming real soon. Once we have all of those packages sorted, and again, real soon, we'll then need to play with a few other bits and pieces that then use this output. It would be really nice if we'd gone to all this effort if our installation media also took our signed bootloader. Ditto the live images, our cloud images. We might need to tweak the bootstrap to pull these in by default, because, you know, obvious reasons that that's something to look at next. Well, we did this presentation, this talk to tell you what was decided, but we'd like to mix this talk a bit with a buff format. So if you want to discuss something, yeah, make questions, go ahead. Comments, complaints, criticism. If there's anything that's unclear, please shout. We want this to be understandable and believable for everybody, so talk to us. Hi, thank you very much for working on this. I have a few questions, actually. First of all, how large is the monolithic grub image that you mentioned? Do you know? I can check. Yeah, it's the easiest. Instead of the tiny, just core image, it's going to include all of the modules that are needed. So it's going to be maybe 10 meg or so, I think. It's not huge, but it's bigger. That gets saved to this slash boot slash EFI partition. Exactly, yes. So the standard boot EFI minimum size that DI or whatever will give you is 500 megabytes, because I saw this coming, and it's also recommended by the EFI people. Okay, good. Just like making sure there's no risk of running out of space in slash boot somewhere. Exactly. So the nice thing about doing EFI with grub is you don't have to try and embed grub in a tiny space at the beginning of your disk. There is proper file system space for it, so it should be fine. Okay. I noticed that the Linux kernel is checking for signatures on modules before it loads them. Is that necessary for the secure boot spec or just something we wanted to implement? Let's let Ben answer that one. It's sort of unclear whether it's required. My understanding is that the generic UFI specification does not require anything like that, but the Microsoft's requirements for signing say that you shouldn't load... It seemed to imply that you shouldn't load unsigned code into kernel space. Not every distribution has taken the same interpretation, but I think we're going to play safe here. There's always the option if you want to load unsigned modules, there's always the option by local intervention to turn that off. Okay. Final question. The signing service, does it actually rebuild the kernel from source? No. So the signing service, basically, the operations it performs is fetch the signed template... Sorry, fetch the template that contains the metadata about what exactly we're going to sign, then fetch the already built kernel, extract it, and then for each file that the template says to sign, signs them with the detect signature, puts those signatures in the template. The template build depends on that already built binary kernel that it just signed, and then we call... It changes, sign, and depot. So the important thing is we're actually not running any code specified by the package itself. We're entirely just using standard domain utilities and the signing tools. Okay, very good. Thanks again. Does this interact with init-ram-disk? No. Not at present. We don't have any verifications of the init-ram-fs scripts, but obviously the modules in there do have to be signed. The modules in the init-ram-fs will have to be signed. To expand on that, this has been a case where there are some requirements that are made very explicit, and then there are some things where you'll see communications from Microsoft or elsewhere, but you should consider maybe doing this thing or that thing, and so we're trying to thread that needle, basically. So you pointed out that Secure Boot doesn't help against a physical attacker, and one thing that it's common to do to prevent that kind of attack is put your boot partition on a USB stick like this, which protects against certain kinds of physical attack. Can you do that with AFI and Secure Boot? So you can secure boot off this? Yes. Okay. And then could I see the list of things that have to be signed again, please? So why would system debut have to be signed? It's a bootloader. It's a bootloader. It's the new name for a gumboot. Okay, thank you. Hi, I don't actually have a question. I just wanted to thank all of you for the work that you've done on this and to applaud both decisions, the one that you will only boot, or load and boot, a signed kernel if Secure Boot is enabled, and the decision to only load signed modules. Given all of the choices out there, I think both of those are the right decisions for Debian. Thanks for making them that way. Okay. As I understand, this shim and then boot signing is required for dealing with Microsoft certificates, which is embedded in hardware. But what about this hardware provided by Linux friendly vendors like Purism or System76 or something like this? My understanding there is that standard doesn't really allow you to embed multiple signatures. Or it kind of allows, but it often doesn't work. So it's a bad idea. So in fact, older versions of the UEFI spec are only allowed for one signature at this point. Newer versions do allow for multiple SIGs, but are not aware of anybody actually doing it yet. The last thing, of course, we'd want to do is go and sign our shim with multiple keys and then break on older hardware. So it's difficult. We could ship multiple versions of shim. We could ship multiple versions of shim. If that becomes necessary, of course, we'll do it. Please tell us if you know of any machines that would use it. We'd be happy to talk to any vendors that were interested in integrating Debian keys into their systems. We haven't had any reach-outs yet, but we're more than happy to work with them. And of course, if you can get shim to load, part of it includes mock, which includes support for user's own keys. So you can then add all kinds of extra stuff if you want on your own machine. We're not stuck just using the one secured boot path that we're going to ship by default. Obviously, we don't want to lock down people's own holdware, so they can't do what they want with it. That's not what we're doing. So I'll couch this very carefully because there will be other teams who will have their own views and policies and so on. And I have a vested interest here as well, clearly. In an ideal world, and from your perspectives, is your work feasibly back-portable towards stretch or is it just too different to actually be useful there anyway? So my opinion, although people might disagree, is hell yes. I would love, if at all possible, to have these changes in for next stretch point release, for example. We'll have to evaluate exactly how much work is involved, but it shouldn't be impossible. From our perspective, I think that timeline could be feasible, but it's up to the other teams. So the infrastructure is in place already and it's just a case of adding an extra suite. It'll need the package changes and obviously getting off nice friendly SOMs to agree to those. We'll have that discussion offline. A question. Does any of the signatures have an expiration date in it? Or what about if a signature is revoked and somebody is running a very old version on a very old computer? Paul F knows about this, I think. He's looking for it now. I'm trying to figure out if with such an expiration date, I don't believe we do. I think we did, but it was like 10 years or something. I don't think it's very accurate. You can only check. My understanding is that expiration dates might be set, but aren't actually checked for code signing. Certainly the kernel doesn't check the expiration dates on the certificates for module signing. We understand revocation is a problem. Even Microsoft struggles with it. I believe at one point they shipped an update that used more space than there was EFI variable space available. So Azure Blacklist grows. You're going to run into this sort of problem. It hasn't really gotten to that point because we haven't really figured out under what circumstances we need to Blacklist. That's actually there is an ongoing discussion between some of the distros over exactly how to manage revocation. The various EFI vendors, and Debian is one of those, obviously have a vested interest in that working well. We want to have a consistent story across the different distros. It's happening more news as it comes out. Don't touch that dial. What happens if a user has a DKMS module which needs to go into the init-odd? The disable secure boot. It works automated? No, no, no. The user has to disable secure boot. You have to disable secure boot for that. In principle, they could arrange to build and sign it on some trusted machine with a signing key and enroll the signing key, the certificate for that signing key in the machines where it's going to be used. DKMS certainly has support for building packaged modules and shipping them to another machine, but I don't know that it has any integration with signing. That's not something I think any of us have looked at. In theory, you could run your own signing service and get the NVIDIA or whatever kind of module to provide the template package and then run all the machinery yourself. Experimentation needed, but in theory, it ought to work. Deliberately, we've made a point. Everything we're doing is public and open. There is no magic in anything of what we've done. There's no secret source. If people do want to run their own signing service for a derived distribution or something, all of this is doable. We are not necessarily going to add other people's keys because that defeats the object. But if you want to go through the same process we have, it's all documented. It's all well understood. It's just a matter of time. We also do things like ensure that the state of whether something is signed or not, that state does not live in DAC. It actually lives in the signing service, which means that the manifest exported by DAC saying, please sign this package, that doesn't go away once that package is signed. So it means that if you run your own signing service, then it will actually pick up that, even if you only run that once a week, or you don't end up with race conditions. NMUs of the packages that need to be signed, do those get signed as well, or is it only a listed set of uploaders that gets their packages signed? This was something we discussed. It was something we discussed. At the moment, NMUs will get signed in the same way. If people do try to abuse this, we don't want to lock things down totally that way. If people do end up abusing it, then we will shout at them. We will end up evoking things that are broken if we have to, but of course, it comes down to, we trust EDs in general not to upload bad things. This is an extension of that. Thanks. Okay. So there's no manual review process whatsoever between an upload and the signing? So right now, we're running the signer manually, so we're running the signing service where we do actually review it. That's mostly because we want to gain confidence in that it operates properly, but we do not intend to require explicit human intervention after an upload. So the reason I ask this is because of that last question, the way we implemented this in Launchpad years ago, and we have significantly fewer Ubuntu Core developers than there are Debian developers, and yet our trust level was still just barely low enough that we insist that there's that interim step where a binary is built and we've asked for it to be signed, that gets shunted into a queue where we look at it and go, yup, that looks sane, let it go. Because as soon as that binary is signed and published, you're screwed. You have to revoke your cert because that binary happens to be malicious. I've already been talking to Colin about this. He is very interested in how we've done things. He was suggesting maybe Ubuntu changing some of your infrastructure. Clearly we will want to converge at some point, but I don't know if converging on letting it sail through is what we're going to want to do. Other bits, because we're going to want the packages to look the same, we don't want to carry massive deltas forever up, that's insanity. The other thing I wanted to touch on, because BDAL is absolutely right, you've made the right decision to authenticate all the way through. I just don't want anyone to think that, haha, Ubuntu is super insecure and screwed up and doing it wrong because we took a different decision six years ago because we have changed our policy. The last step is to enforce signatures from Secureboot down which we're changing in the next month. I know that was a decision that was made a long time ago. It was a decision that was made a long time ago. It's been discussed in 2012. Yeah, I guess for right now we can discuss offline, but I think definitely there's a certain amount of time that it takes for everything to go through the process between kernel being built, kernel being signed. Turn around time is hell for all of this. Right. Yeah, I think there could be middle grounds having a delay, so we assign a kernel six to 12 hours after it comes out. The idea is we can avoid blocking people, but we can hash something out. Yeah, delays work for you. They don't work for us ever. Sure. But we also have more people manning those queues and just going and quickly looking to see if something's right. This looks different if you have pits or not. Well, basically I check Providence, make sure that it was uploaded by someone that I reasonably trust. I've also already done a source review. If I haven't done a source review of that kernel upload and it's from a name I don't know, it's going away very quickly. Sure. So of course in terms of the commonality of the packages, I mean the firmware update packages and the grub package between Debian and Ubuntu are incredibly closely tied. You know, there's a lot of overlap in the maintainers. So for now absolutely there are diffs in the rules files or if equals Ubuntu or Debian. We definitely want to make those go away as soon as we can. Yeah, actually a lot of the diffs we've had in grub for the last six years can go away very soon if we change stuff, which would be nice. Sure, cool. Hi. One last question. I think some of the install media contain memtest86, do they? Yes. But that still work if you boot it with secure boot or should that also be added to the list of packages to be signed? Correct. We can't load memtest form a UEFI boot menu. You can only do it from a BIOS boot menu at all at the moment. So it's a big question. Okay, that's a shame. Yeah, it's a shame, absolutely. But until somebody changes it, you won't be able to boot it on an UEFI system at all anyway. Mm-hmm. I don't see in principle why we wouldn't sign memtest86 as well though. Like, in principle. Yeah, absolutely. If it wasn't UEFI binary, we'd want to give it a quick audit and then sign it. Another question though, because all of this was sort of a, you know, clean room redesign, blah, blah, blah, which is great. And it seems very x86eficentric. Have you designed it in a way to be extensible for other types of signatures? Yes. Okay, good. I only ask that because we literally just added PowerPC signing like three months ago. We did consider, so already, we're going to be signing, for example, that we're going through. This is deliberately, exactly, but this is deliberately informed by decisions. I do remember, you saw Matthew's talk last year about signing the rest of the system. We deliberately, we've architected this so we can have other types of signatures. We will burn those bridges when we get to them, but we haven't actually cut off any of that yet. One quick question. You mentioned that there is currently an audit database that has public dump with basically all the stuff that was ever signed by the signing service. Would it make sense to make that data available in, for instance, certificate transparency since it's regular X509 certificates, as far as I know? We haven't looked at doing it that way. If that makes sense. We could certainly take a look at that. Okay. It's hard to check a CT log when your computer hasn't booted yet. I do agree that using existing infrastructure is useful. It's not clear that that versus... We can talk offline. My point was more about it's easier to get people to mirror the entire log if we export it in a format they already understand. It also can become quite a lot of data. Thanks a lot. Any other questions? Comments? Discussions? I think we are done. Thank you very much for all of you.