 So that's like a small disclaimer. The whole process is still learning process for people who know secure boot. And the more you know that the more you know that you know nothing basically. So this would be the agenda and I promise I'll try to keep it in time so I won't hold people for lunch. So I work as a senior cybersecurity officer in the Irish Center for High End Computing here in Ireland. We are a research center that provides a high performance computing services for all research institutes here. We are funded by the government. We are part of the university. I've started my Linux and open source journey around year 2002. Been Linux professional admin for over 18 years now and then I'm volunteering and a part of the Rocky Linux release engineering team. I'm mainly focusing at the moment for secure boots, SIG HPC and SIG Alt-Arch and SIG Kernels. And I'm a world member of the RSF and Rocky Linux project and product also. I'm a little bit into ham radio or two for radio as I call it since I don't eat ham. So let's talk a little bit, take a step back about how secure boots started. So the UFI released a specification which is it used to be called back then if I version two around 2006 and I thank Vincent Zimmer for correcting my information regarding the dates for the history because there is not too much documentation about how this whole thing started. Then Microsoft started pushing a Windows logo program or what used to be called the Windows logo program which required hardware vendors to be able to run Windows 8 to have secure boot enabled. There was some information that's not 100% clear to me. There was discussions that for specific architecture like ARM you can't disable secure boot and then they went back and they disabled this feature so now user can enable or disable secure boot but there is a lot of back and forth here. There was lots of panic maybe until like 2015 lots of people in the open source and Linux community was saying okay basically Microsoft is trying to look down running Linux in any hardware that has secure boot. Fedora 18, CentOS 7, Ubuntu 12.04 was one of the first that starts releasing secure boot and shim. There was a lot of frustration when you have a third party driver that like any of the graphics driver that outside of the kernel tree won't run because you have secure boot and you're option either. Disable secure boot or build your own kernel or sign your own driver load your certificates and we will come to that later. But the current state for how the secure boot community is working is very great. There is lots of collaboration between people from Microsoft, Oracle, open source, Red Hat, Debian, Canonical and there is loads of coordination now between the shim and the developers and Microsoft so there is no more non-visibility of what's coming next regarding to firmware development with the hardware. So what's secure boot? Secure boot basically is a layer that tried to protect you from running malicious code into your operating system, before your operating system so like the boot kits and so on which is great and we will see now how it works. So you have a set of keys that usually shape with your firmware. There is usually one master key that's called the platform key which is usually related more to the hardware vendor and then you have something called kick that might be one key or multiple keys which is key exchange keys. Then you have two type of databases, DB and DBX. DB will have what the firmware is allowed to load and DBX will have what the operating system cannot load. That's in theory most hardware will be shipped with those sets of keys pre-configured and then installed. And then secure boot and the firmware need to start the chain of trust. The whole chain of trust is based on the public key infrastructure within the firmware. So the platform key will sign whatever kicks that there, the key exchange key, the key exchange key will sign a database that includes a specific certificate that then will sign your first boot loader or will sign a database which is DBX that will prevent specific hashes and certificate from loading in your firmware. In practical at the moment the kick database for example within most of laptops that will run Windows will have a Windows certificate and then the Linux community need to submit a specific first loader, first stage boot loader to Microsoft getting signed and then loaded into your operating system. That key exists in the DB of the firmware, the Microsoft keys. We will also get this into details in a little bit. But don't mix the DB and DBX with the actual shim DB and DBX. So shim is what Linux community and open source community wrote to be able to boot a first stage boot loader on your machine when you have Microsoft keys. For one reason, Grub 2 is under GPL3 and it can't be signed unless you would release your signed keys and Microsoft and other people will not be signing the, will not be releasing the actual private part of your public key for the public because that will define the whole idea of having a secure boot kind of concept. Then there was another aspect being added, which is called the SPAT, which is secure boot advantage targeting, which is emitter data that you put in top of your EFI loaders and will have more information about this software that you are running. Like the version, some URL or email, but most importantly is the second column which is a global version. And that also will allow the EFI and secure boot to disable specific loaders or specific version based on their global generation. As you can see here, we have shim version 2 for example. So that's allow mean that we cannot run anything that says shim version 1. And this was used that to revoke malicious or vulnerable, both loaders that's been already released to be running in the same way that you are running any more in your operating system. The DBX and DB within shim introduced as a, how you call it, a reaction to the grub boot hole bug, where basically there was a, I think around eight security bugs in a grub assistant point and that require all grub hashes to be added to the DBX and it basically took half of the NVRAM that assigned in the framework to be able to load or forbid hashes. So shim now has their own DB and DBX. So why should you use secure boot? So security is not a luxury, it's a mandate. You have, as long as there is security you can use, you have to use it. So for end user, it will protect you from bootkits, it will give you control over your, if you bind it to load and that's what happens when organizations basically wipe the framework or load their own sets of keys inside the framework to make sure that you can run only specific if, if, I, that you wrote. Most military will be doing that, they will not leave it with these top keys. Also, you as a developer, you may not want any kernel to load or any grub to load unless you build it and you compile it so you can build your own small secure boot infrastructure and whenever you have your distribution, whatever the distribution is, you can load your kernel, your keys and you make sure that this laptop will only load your stuff. For distros, it makes, we try to make end user life much more easier. Also it allow us to be able to actually prevent vulnerable if I and both loaders to load on the operating system. For example, if now we have a bug in the grub that has been released, we can use secure boot to disable this grub until the user in advance, you need to upgrade the grub because at certain point this grub will not boot anymore. Okay, so let's talk now about the process of how a distro can achieve secure boot. And that's a little bit of a mix between technical and non-technical. Most of the obstacles in secure boot are actually non-technical. So they are all more about process governance and your procedures to actually be able to sign and build secure boot. So we will talk about the technical part first. So in high level, you need to have a FIPS 140-2 level 2 operation environment and there is a little bit of discussion regarding the operation environment because some will tell you enough to have the keys are stored in a FIPS 140-2 level 2 certified. But we prefer if the whole environment is FIPS certified. Then you need to generate your own distro keys as we talk. Some distros will prefer to go with extended validity certificates other than building their own CA on having a self-signed CA. Then you get your CIM and you put your CA within this CIM. Then you split your certificates and put them within your kernel, grab FWD and CIM as well again, and whatever other EFI binaries you are going to load, compile and send your packages without CIM. Now you need to get this CIM signed by Microsoft. Otherwise, that's great. It will boot in your machine if you actually loaded your CIM CA into your framework as we were talking about. Microsoft has some requirements to do that, which is basically, first of all, you need to be registered with their partner program for hardware portal, I think. You need to get your organization to be vetted and have an EV certificate. You need to provide your security contacts. You need to fill a CIM review form, or it's a GitHub issue basically, and you go through the CIM review process. So the CIM review process is being reviewed by a community of Linux developers. Mainly, they are CIM and Grub developers that volunteer their time to do that. They are from most distros, Red Hat, Oracle, Open Sozi, and I don't forget any of that. It's Debian. And they check actually all your code regarding CIM. They only sign Grub at the moment, so they make sure that your Grub is hardened correctly. They require you to have specific fixes for certain CVEs in Grub. They make sure that your kernel has a specific configuration, like lockdown, and that your kernel also has specific patches for CVEs fixes. And then once your CIM review is accepted, you get your CIM, and of course it needs to be reducible. So the CIM review will try to rebuild again your CIM, and they need to get again the same hashes. Usually you provide a container file and maybe a side repo where you actually was using the packages and the tool sets that you used to compile the CIM for. Once your CIM is accepted, you go back to Microsoft Portal, and you combine your multiple CIM, if you are signing for multiple architecture, into a single CAP file, sign that CAP file with your CAP file, and you submit your CIM to Microsoft Portal, because your EV certificate that Microsoft Portal has, unsubmit your CIM to Microsoft. Pray and hope that everything will go well, because sometimes things stuck in the pipeline at the Microsoft Portal. So Microsoft will do lots of processing on the binaries. They will compare to make sure that everything is running fine. Once everything is good, they may ask you a couple of questions. Submitting CIM and this is other than the CIM review. After maybe a couple of weeks, you will get a signed binary from Microsoft, which is basically a CIM, including your distro CA, which is signed by Microsoft key, that the public part of the certificate within the framework of the laptop, or your PC or whatever. You get this CIM binary and you drop it into your RBM, starts running a lot of break, everybody's laptops and nobody will boot. Once you are happy, you can release. So the chain will be, firmware will verify your CIM. CIM has your certificate, your certificate, or your CA, you have your certificate from this CA that actually did sign your kernel grub and FWD, whatever other binary you have, everything will boot. So far so good, but then come the kernel modules. So the kernel modules is not and considered a part of the secure boot. So if you have your kernel look down mode, you won't be able to load any kernel modules, unless they are signed by the kernel. But the only part that's signed from your kernel for secure boot is actually the kernel image, but not the NETRAM, FS, or NTRD, which is why now there is some discussion within API where you need to combine a kernel boot command with the NETRAM, FS, and the kernel image and have one, with a stub and have this signed for secure boot. The keys, most of the distros, like CentOS and Red Hat and others, they use a formal keys while they are building the kernels. So your kernel modules are your spec file basically generating a key, then they are signing the module and then it's gone. But your kernel knows that this key is, know the keys through something called a mock in your machine, which is a machine owner key database that shem generates, including some of the kernel keys that you have while you are building your distro. Then you release. That's so far so good. So the approach how we did it for Rocky, I sadly kind of volunteered myself to do this. I said okay, I'm going to volunteer to look into secure boot, which was a great journey and I'm still learning a lot of things. Then come the most complicated part which is finding actually a FIPS 140-2 level 2 environment. Most of the major cloud providers, they do have an HSM service that is FIPS 140-2 level 2 or level even 3 certified. However, there is no operation environment that can connect to this HSM, that is FIPS 140-2 level 2 certified and that caused lots of issues and we ended up actually finding a data center where they are FIPS certified and we had hardware there and this is how it is. But again, as I was chatting with some people who actually wrote some of these requirements and they were saying, okay, mostly we just require the keys to be in the FIPS environment, not the whole build environment, but there is no confirmation about this. Based on the documentation that issued from Microsoft, the whole environment need to be FIPS certified. And we needed to start installing those, releasing those packages with secure boots, so we went only with a single certificate for Kernel, Grubb and FWD. Everything went fine, we submitted Shem review for 8.5, I mean, we submitted before 8.5, however, until we got the whole process done, 8.5 been released and we finally released 8.5 because it was a signed Shem from Microsoft. We had to go on to another iteration and we released a new Shem because our Shem was based on 15.4 plus some cherry pick, get patches that the Shem review committee required us to have before the Shem review was being released. Now we have Shem 15.6 signed, there is Shem 15.7 already released and I think within a few weeks we will have a new Shem released soon. So we went into two phases. First phase we went with a single certificate as I was mentioning. We were using a UP key FIPS 140-2 level 2 HSM module and we were using PESI in server and client mode to be able to sign the rest of the binaries, the Kernel and Grubb and so on. So PESI will talk to the FIPS HSM, get the keys and sign the binaries. But now we move to phase 2 which is this is actually the current that we have at the moment. We still have the self-signed CA for Shem. We start separating the Kernel, Grubb and FWD certificates. So now basically we have multiple certificate per package and we move to use UPHSM FIPS 140-2 level 2 which will allow us to be able to use more even certificates or keys slots within the HSM module and we are using, we move to use PESI and tools. And so far this works fine and give us less headache and now we have more modular of kind of a networked HSM that we can use even for signing other packages within the secure boot ecosystem that we are building. Okay, so what's next? At the moment in the ERESF we are talking about joining the UFI forums as contributors. We have we are thinking and talking about signing other kernels from Rocky SIG kernels which will include Kernel mainline, Kernel LTS with some configuration patches we are building. We are looking into the power PC and the system secure boot because those are a little bit more complex and they don't do the traditional if I kind of concept of floating packages. Also we are looking at the ARM64. We never submitted an ARM64 shim to Microsoft because Microsoft has different requirement for that but we are going to look into that and submitting this and the aim is to help the SIG Alt-Arch to have secure boot running on most of the system of chip and build hardware if it's applicable. And also we are looking at the UKI from system boot which is as I mentioned the combined kernel with NETRAM, FS and command line and system do beat which is a boot manager. And we need to automate. At the moment there is not too much automation happening within this when this signed which is good and bad because you kind of need to confirm that you are doing things right but it doesn't mean that it can be automated and have more monitoring. In secure boot ecosystem there is a SPAT support being pushed to the kernel patches at the moment and the NX support for kernel patches. The NX support is a little bit tricky because Microsoft from last November require any shim submit to have an NX support and there is no NX support at the moment in the kernel and they won't sign any shim unless it's NX supported. So I know that Debian is the only or decode the last cycle of releasing shim and that's NX supported and I know that they will backport the NX patches to their Debian to their Debian kernels. For the SPAT still under review by the kernel developers. So that's to recap. Any questions? Yes? Okay. Is this SPAT real? It is there as far as I know there are still patches need to go in to be able to actually do the revocation correctly or as far as I know. I'm not 100% but I know that at the moment you can run things below your SPAT version but that should be fixed. I hope within the next release of shim. Which is very interesting because I don't exactly know what the history behind having this SPAT if you can actually use the shim DB and DBX because with DBX you can just prevent all your hashes. So the idea also of the governance here that we were talking about that you need to keep logs of everything they are doing. Everything needs to be logged. All your binary hashes need to be stored so you can actually make it easier for you to revoke. But what I know is that the SPAT and DBX was a reaction to the boathole. Any more questions? This may be a dumb question but you were talking about the fact that kernel modules need to be signed if your kernel is signed obviously if you're wanting to use them. So with the secure boot Rocky Linux release is there a way to do either to sign your own if you have out of tree kernel modules is there a way to load them up without having to go through all of this? Yes there is a way. That's why the kernel module is a little bit confusing because the kernel module is not actually a secure boot feature but if you have kernel look down on some patches it will be a secure boot feature and kernel modules will not load if you have secure boot enabled. And even as far as I remember there were some things that you need to make sure that the kernel can check that the boot load are actually on our secure boot. However, there is a ways that you can load another certificate so you can sign out of tree modules if you want and then load them into tree but then you will have to do that. The way that this works is some distros will have specific key that they will be certificate loaded but it's not used unless it's need to be used for signing an out of tree kernel if they really really audited and they want to run in their distros. Otherwise it's up to the user. User can just build their own kernel and then load this key load the certificate into their mock database into the machine and then it will load. I hope this answers your questions. I guess this is kind of a me catching up question. From what I understand the Shim was originally developed so that you could actually use secure boot with Linux regardless that it wasn't the whole thing signed at the time. Is that correct? That's why it's called the Shim but no. Exactly, not the whole thing. Shim. So going through that process did you it must be like all the open source operating systems are just doing the same thing. And what does Microsoft do? Is there anything like they feedback from them about that or they're just like yeah whatever they don't care? It's not that they don't care. So the idea of Shim as the Neil and I think David mentioned that so you don't need to push your hands and you don't want to do that. That's why the Shim existed in first place to be able to submit something to Microsoft so they can sign and then other Linux desktop can load that. At the moment as far as I know the Shim has hard code only to road grab to and most of the Shim reviews will only advise you that at the moment we are only reviewing your Shim or your Shim. So that's what the main idea of the Shim that you have something simple enough that submit to Microsoft very easily audited and then Microsoft can sign it and then you can take the chain from there loading your own certificate and your own key. Also the same way as when you are loading the keys for like when you put your CA within your Shim you can actually utilize the DB feature of Shim and have multiple certificates that you can use. Okay, why just we don't have one Shim that has all distrust key, sorry? Yeah, there was this discussion to have this kind of concept. Also there is now a side tools that's running within Shim that actually, I think they just changed the name recently it used to be called certimule which basically if we are in a classroom that you want to load your own only the kernel modules or only the kernel images signed by you you can give us your certificate and we will sign this certimule and then it will be loaded. It's kind of similar to DBDBX but the issue is DBDBX that every time we need to update this DBDBX we need to send it back to Microsoft to be signed. So that's why there is this certimule tool exists. Any more questions? I don't have questions are there any more questions? Going once, going twice okay, thank you so much. So lunch is served or will be served in three minutes the lunchroom is on this floor just down the hallway and we will return here or whatever track you want to be in an hour at 1.30 if you come back here we're going to get a hyperscale update hyperscale SIG update followed by an automotive SIG update so thank you and I'll see you at lunch.