 So, hello everyone. My name is Eric Onunes. I'll talk about automating secure boot testing. So, I work for Red Hat. I'm the kernel QE team for hardware enablement. I work mostly with ARM platforms, ARM servers, Red Hat support for ARM servers. And after some time I started picking firmware testing. So, I worked with the firmware update testing, UEFI related things and also secure boot. So, as you most people have worked about, have heard about secure boot so far. So, secure boot is a technology where the system firmware has a database of keys. And it will only boot the next phases of boot if the boot loaders or next phases of the boot are signed by some key that is known by the firmware. So, we have pretty much every major distribution follows the path that is in this diagram. So, UEFI firmware contains database of keys. It will load, in this case, SHIM. So, SHIM is something that we need for secure boot support in Linux distributions because pretty much every computer or laptop that you can buy today will only come loaded with Microsoft keys. So, in order to not have everyone load the Fedora key, the Red Hat key or the Ubuntu key or whatever in our laptop, to make Linux distribution just work. We just have the SHIM that is some small boot loader that is signed by Microsoft and it will contain the keys to load whatever you need to load for the rest of your distribution. So, we load the UEFI firmware, loads the SHIM that is signed by Microsoft. We can load that because Microsoft keys are probably in your laptop already. And then the SHIM contains the keys for distribution. For example, Red Hat keys to load the grub that is signed by Red Hat themselves. Then the Linux kernel that is also signed by us. We don't need to rely on Microsoft signing every release that we do for all of those other components. And some other important mentions that, for example, kernel modules have to be signed to, otherwise you could load unsigned kernel modules that could run arbitrary code in your kernel and secure boot would be moved at all. So, some of the problems that I have to deal with is that it's difficult to automate testing for secure boot. Because pretty much every firmware from every vendor will have different menus, different settings, different names. For example, it can be secure boot or only load signed binaries or something completely different or maybe just a graphical user interface for that. It is not really helpful to do automation for that. So, not only for enabling secure boot, we may also have to do the same thing for loading custom keys that we may need for other steps like testing, for example, that kernel module signed with custom keys work. And that's why before I started working with this, we didn't really have any secure boot automated testing for, for example, CI that will test for every beta release, for example, that we do that secure boot would be working completely. And if anything fails, for example, if a key, some signing step is broken or if we have a grub rebase, for example, that has some broken code that will fail to load the linsear, for example, or verify the signature correctly, then we wouldn't catch that. We would only catch that much later with some manual testing. So, if you load the Red Hat Enterprise Linux installation guide, something that we need to do. And this is part of what I want to automate is that we first need to disable secure boot. So, for example, having a machine where secure boot is enabled at all times and I would just test many times on that same machine is not something that is feasible. We need to disable it and enable it every time and that's part of the problem as well. So, first we need to disable it, install it. By the way, this is for the beta release, it's not for the final release, but that's what I need to test anyway. So, first we need to disable it, then you can perform the installation as normal. Then you need to load in the keys, for example, this is using the mock, which utilizes user space utility for that. We may use that or not, for example, we may just load in the keys directly in the firmware. And then we need to reboot and then we have the secure boot system installed. So, the first solution that we attempt on that is to test on virtual machines using OVMF. The biggest advantage of using OVMF for that is that it stores the variables in the NVRAM file. It's the OVMF vars file. The OVMF vars file is a template provided by OVMF themselves. And basically if you use something like lib-work, it will copy the OVMF vars file template to machine-specific directory. And it will use that. So, it's possible to manipulate that file to have a machine where secure boot is enabling firmware or not, for example. We can do that from the host machine. And then we can accomplish installing with secure boot disabled and then in automated and programmatic way, enable it and then boot the machine again with secure boot enabled. Unfortunately, there's no way to do that. There's no tool to do that with the, like, some user space that will go and tweak the file to just flip the bit that will make secure boot enabled. So, a workaround for that is to have a pre-prepared file of NVRAM file that will have secure boot enabled. So, we can just install it with secure boot disabled and then just copy over a file that has all of the NVRAM settings with secure boot enabled and all of the keys loaded. And then we can just boot the machine again and we'll have secure boot there. I just wanted to mention here that we're using a tool called Beaker for that. It's a Red Hat Community project. So, Beaker is great for hardware enablement testing. It allows you to locate a machine that has, for example, some PCI hardware that you need or that has number of cores or a certain amount of memory or certain architecture and then you can just send a query and it will reserve a machine for you or just run a set of tests that you have for that. So, I have a set of tests that will verify that secure boot is enabled, for example, that we can, and I have plans for extending that. We'll see that later. But basically, I have some Beaker tasks that will do that and some machine that we reserve. It also takes care of installing new systems, which is great and has support for virtualized workflow that we need for secure boot testing virtual machines. So, this is the outline of the solution that I have currently implemented. Kickstart. A machine basically do a Red Hat Enterprise Linux installation on the host and then some gas machine. Beaker will automate that for me. Then we perform some test checks that UEFI is in use and secure boot is disabled that we should make sure for that for the installation for the first time. After that, we switch the gas invariant file and then we boot the gas again and secure boot must be enabled. That's basically the first step and it works. So, it has some problems that, for example, multiple releases of our distribution may have different keys. So, it may need to have different invariant files for every different configuration or release. Beta keys may not be the same as well and still did not solve the problem of adding custom keys, for example, for testing a custom kernel module signing, which is something that we also want to cover. So, a different solution for that is that OVMF also provides this small UEFI application called enrolldefaultkeys.efi. Basically, it's an UEFI application that you can run from the UEFI shell. It will enable secure boot and enable some keys as well like the Microsoft keys. And this is a bit handy. For example, I used to do that to avoid having to have an invariant file pre-prepared. However, it has a hard-coded set of keys. It still doesn't solve the problem of custom keys for custom kernel modules. And some OVMF builds may have the UEFI shell disabled, actually. That's what I wanted to say. So, I may have to use something else like the UEFI shell, which makes the setup a bit more complicated. So, our proposed solution going further is to every time patch the enrolldefaultkeys.efi, add some newer keys, rebuild the application, and then repeat the step that you had for solution one. So, that's pretty much it. For future plans, as we're using Beaker, for example, we can have other Beaker tasks that will do loading out of three kernel modules signed with custom keys. I can do that if I have the patched enrolldefaultkeys.efi application. The mock tool is also not automated as described in the manual. These tools are a bit difficult to automate because they require manual steps. There's no way around that. So, you may have to deal with expect scripts, for example, or pyexpect. It's something that I really didn't want to go there. There's also FWTS, which is a firmware test suite. It's maintained by some people at Canonical. It's a great tool for firmware testing. I'm interested to see how that works in the secret boot environment. There's also ARM64 support. Actually, ARM64 support for secret boot is a bit, I don't know how the state is right now. There are some political issues, for example, with Microsoft not allowing you to disable secret boot for ARM platforms, which is different than the X64 where you are mandated to be able to disable it if you are physically present to the machine. And KXX is something else that we need to test there as well because it wouldn't make sense if you have a secret boot environment that you were able to just KXX to a totally different kernel that is not going to care about secret boot at all. So, that's pretty much it. Thank you for coming. I hope you have some questions. This shim that you have shown in the second slide is that breaking the secret boot as if that's a problem that will also start no matter what comes up. So, Microsoft provides a service where you can send them a binary and they will verify that it complies with the secret boot and then they will sign it and give it to you. So, I assume if you malicious user send them a binary that will be signed and will load something with a different key, you could break that but it will have to be signed by Microsoft anyway. So, shim, for example, the Red Hat build is only signed because it complies with the thing that it will only build other signed binaries. You can't have something signed by Microsoft that will just boot anything. Right, yeah, sorry, I forgot to repeat the question. Both questions about shim actually. So, yeah, shim contains the keys to be able to boot the other stages of booting. Does it mean that Microsoft does a code review on this shim before they sign it? They should do verification that the binaries that they are signing comply with the secret boot. Sorry, I didn't understand the question. So, if new ARM systems will include the Microsoft keys as well, is that the question? So, I'm not sure about that. I think it's not a resolved issue yet. The systems that I deal with are not coming with Microsoft keys, but they are mostly prototypes. I'm not sure you can buy an actually ARM supported server yet, for example, it is supported by Red Hat. So, I think even if they do, there are still some issues, for example, for booting Linux, because it's not totally compliant with booting just anything. I'm not able to answer you better than that. Okay then, thank you.