 All right, hi everybody I'm Chris I seem to have about 30 minutes and I usually talk very slowly, so I'm gonna try to get through it quicker than normal I'm gonna talk to you about firmware today, which is kind of strange for a Linux user space conference But I promise you there's Linux user space involved in firmware And we'll get to that in a second First of all, I want to thank we have a whole bunch of contributors from other companies that we collaborate with on this project unfortunately, none of them are here But You should know they exist The first thing that I actually want to talk about is what in the x86 world firmware usually looks like And then what we're currently doing about it So if you have worked with UEFI before which is basically x86 firmware in most cases You're probably familiar with this picture, which is probably about 20 years old at this point and The individual pieces are not too important, but what I want to point out is the the left two columns the security and the PEI phases they consist usually of Intel binary blobs or Chipset like a vendor binary blobs or a motherboard vendor binary blobs And these are things that you're not likely to see being open-sourced in the near future So this is stuff that we don't that we don't really touch But on the right from there You'll notice that you're going to recognize a bunch of operating system like concepts You have network drivers. You have disc drivers. You have a scheduler that Dispatches those drivers and you can actually also run This is kind of important for later. You can run arbitrary PE 32 executables in ring zero and on On your normal UEFI system what comes together here is usually about 20 different vendors that put together codes that Ends up on your system like 20 to 30 different ones And if I look at some of these driver modules, which are by the way called Dixie's On a five six-year-old open compute platform server called Winterfell You can see there's Some outdated stuff here like the microcom networking protocol, which by the way the server doesn't have a modem It's a modem specific protocol Some ISA bus drivers like that bus from the 80s that isn't used anymore and In addition to that you have a full networking stack like IPv4 UDP TCP all this stuff re-implemented in firmware and Some of the observations you make when you work with this ecosystem is most of this code is closed source You don't really have any way of looking at it Except for some of the reference implementation that Intel open sources as edk2 that tends to be a Small portion of the code actually running as the firmware on your x86 devices though It's all stuff written and see and it all tends to be run in ring zero in one shared address space Between all of these drivers applications and all these things that ufi runs the other thing you realize from this ecosystem is that when someone like Actually, I don't want to name any names when there's a company that builds a new server or laptop or something They will tend to contract out firmware to another what we usually call an independent bios vendor an IBV And those vendors get paid when they deliver firmware that works on that server and it ships So the they're mostly incentivized to get something working that runs on the server based on the constraints that the manufacturer gave them and to ship it and forget about it, right? And the problem with that is as there are like 20 to 30 different vendors involved in shipping firmware because you need firmware for your devices in there you need to coordinate with the board manufacturer you need to coordinate with a The the company actually selling this stuff all these different companies that have to be involved You realize that Even for people that are involved in the process you kind of rob wrapping black boxes around black boxes Not everyone is going to know the entire system and One question I want to ask you about that is what do you think happens with vulnerabilities? Let's say you are a network card vendor. You found some vulnerability in your driver Where does the first fix for that vulnerability go? Usually Linux or Windows something large like that Let's say you also found that vulnerability in your firmware driver What's the likelihood that that fix will end up on your actual end-user system for that network card driver? Even if it's a security fix right and that likelihood is probably point zero zero one percent or something Because you have all these questions like who owns fixing it who actually gets paid to fix it by the way Is it the the the network card vendor? Is it the independent bios vendor that put it all together? Is it the OEM that you bought the system from? Who is actually responsible for getting this done? How do you integrate it and then actually how do you release it? So once you release an update you either you know You put it on the vendors website and you hope that your customers download it or you try to distribute it through Windows Or some Linux distribution, but then you also have to deal with those distributions So the likelihood that any of this happens is pretty low Unfortunately besides the ecosystem not being incentivized to fix these issues You also can't fix it yourself Because as I said most everything is closed source You likely don't even know if you're if your driver that's in firmware is vulnerable or not I don't think they don't necessarily disclose that and Just because you can't like you can't build firmware from scratch for your platform That also means that you don't really have a way of deploying fixes yourself if you want it to So with that I'm going to take the observation that you can run arbitrary pe32 executables in ring zero in UEFI and Tell you that you can compile the Linux kernel as a pe32 executable So What we started with is we took the UEFI shell We scooped out the code section and we've replaced it with a Linux kernel Compiled with config EFI stub. It turns out that this actually works pretty well Even if you remove most of the drivers on the system because Linux knows how to initialize most devices Linux knows has to how to initialize your disc drivers Linux knows has to deal with your network cards In most if not all cases without any interaction that has to happen with firmware beforehand except for ACPI and similar tables that have to be passed to Linux However, those usually come from the driver dispatcher and not from any of the individual drivers So if you take For example the ocp Winterfell server I was talking about earlier and you remove about 90 of the 120 drivers that were on the system and you replace it with Linux What you actually get is a system with drivers that are open Which means to you you can actually get fixes for vulnerabilities and deploy them yourself You also get a system that's reproducible And I think a lot of people here are familiar with the reproducible builds project The same properties apply to firmware and are important there and For those of you that deal with TPMs in some of your Hardware you also know that Linux is measurable. You know what you deployed, right? If I think I've heard way too many stories about new firmware updates somehow ending up on your platform and you Found out about it through a TPM measurement that gets reported because it's not what you thought it was And it's hard for you to tell if you got owned or if it's actually just an update that you didn't know about Through some channel that you didn't know about The other thing that you gain at least when you're a larger company If you Google or Facebook, you maybe have a handful of firmware engineers that can deal with UEFI You probably have hundreds of kernel engineers that know the Linux kernel So it's a way for you to leverage talent that you already have not having to hire people that are very hard to find And a nice side effect of removing a whole lot of Dixie drivers is that your boot time goes down the Dixie driver scheduler is not an especially Useful scheduler in that sense of word. It's more of a trial and error process. I'm gonna just say like this But for the Winterfell platformer we did this we went from eight minutes to 20 seconds so Okay, I already talked about this the one thing I want to point out is you know, Linux has too many contributors to count Compared to edk2 and grub which have a combined 248 Based on a github screen screenshot that I probably took quite a while ago But the point is there are a lot more eyes Looking at Linux and fixing things in Linux than there are in your current firmware and your current bootloader and The idea of putting Linux in firmware is actually not really a new idea This is something that the open power platform has done for the last four years And been pretty happy with and it's the original idea behind core boot Which used to be called Linux BIOS And It's something that works with UEFI it works with core boot and it works with U boot So this is not specific to UEFI and the Linux combination Here if you want to look it up later The thing that I would kind of want to point out here is that the build process is pretty easy If you've ever tried to build edk2 You know that this is probably much easier and by the way it works on Linux too The supported hardware that we currently have upstream is a bunch of OCP boards an Intel reference board and I Believe that Dell R630 these are things where you can actually build the system right now using these four or five commands if you wanted to So Now we have Linux and firmware we have drivers we can leverage What do you put in the nitro MFS to actually do something with that kernel? And this is where we get to a point where we say As a Linux boot project you can put whatever you want in there Right, there are plenty of Linux boot loaders that people have written like pitted boot on the open power platform like system deboot or You can if you want to just put busybox and K exec in there, right? It's kind of up to you but What we've done is come up with a Go-based runtime that we use as our bootloader platform and Google Facebook and a company called it were new currently collaborate on this effort and it's open source. It's all out there And it comes from the realization that we now have the full tool set of Linux applications at our fingertips We might as well use them And we also started with a premise that we would like to start with a somewhat memory safe language something better than C And a language that makes concurrency easy So you root is a project that has a bunch of busybox like tools like all the standard stuff You know of like DDLS CP all that stuff and a bunch of K exec based bootloaders That implement pixie-like or grub-like protocols in file formats and The why go besides the memory safety is Go gives you easy cross compilation In the initial days of your roots the project was actually kind of born out of a frustration with building a Cross-tool chain for build route and for a busy box and all these projects And one of the nice properties of go is also that it is reproducible Without having to patch GCC or do any crazy things in the sea because in the sea ecosystem And then of course you get a bunch of nice static analysis tools similar to the ones that you get in the C++ world Those of you that are familiar with go are gonna say, okay firmware storage tends to be pretty small usually we usually get between four and eight megs of space where we can put a Linux kernel and a runtime and You probably know that the a normal hello world go binary is something like 10 megs 11 megs So we've had to come up with a kind of strange system To take many different go commands and compile them as one binary We do this by taking all the source of those commands and rewriting it in a pretty simple way to be packages basically and Then compiling then generating a main.go file that imports all those packages and compiling that program What that gets us is a compressed binary that goes down to about three megs four megs depending on what programs you actually put in there And it's a system where you can plug in any go commands that you want to Compile into your firmware as well. So this is not not specific to you route itself and It's pretty easy to do you basically there's a make BB command since this is like busy box style and argv zero decides which program you actually run and You just give it some Go import paths to compile into a busy box and from there it should be pretty easy to use I'm gonna skip over that so now that we have a Base of go programs that we can use Let's talk about some bootloaders written in go I Don't know how many of you are familiar with pixie booting and excuse the shitty picture that I drew this morning but If you are familiar with it, you know that there are some strange complicated pieces in there for example Um The the original ufi firmware will talk to your Nick to find out if your Nick has what's called an option ROM An option ROM is just a like I'm just gonna call it a piece of code The Nick can say yeah, I have an option ROM. I would like you to run it So you go to the Nick you get some piece of code you run it on the x86 firmware and What that usually tends to do is pixie The reason for this in the I'm gonna say the olden days is that you didn't really have drivers for network cards in your firmware So the network card had to provide you a piece of code to do that for you That's not really necessary Anymore today since I also said there are drivers in UEFI, but it still tends to be the way it works in a lot of systems And then from there some pixie implementation makes a DHCP request from there gets a file name Downloads a file over TFTP It will run that file and that tends to be another piece of code another I'm gonna call it a microkernel pixie linux dot zero That will actually go and interpret The configuration for the kernel that you want to download and actually go and download it This is a somewhat complicated process for What I don't think should be as complicated and there's some other problems with it first of all pixie booting is At least in many systems still based on TFTP TFTP is a file transfer protocol With which it can take easily 30 seconds to download a 10 megabyte binary and Option ROMs are a problem because There are devices that hand you a random piece of code and you just go and execute it First of all, do you trust that piece of code that comes from the device? Yes. No, maybe right The other thing is that option ROMs tend to be your real mode code 16 bit Which can also pose a security problem for you in some cases so Now the other question is do you go and re-implement? newer protocols and newer standards like HTTP TLS SSL IPv6 Do you go and implement all that in UEFI and if you do go and implement that? Do you get a mature or audited or well tested implementation and the answer is likely no? Which is another good reason for us to do Linux and firmware But even if you have all these things A lot of companies out there have different security models in their data centers now A lot of companies out there have different very different security models on their client machines like laptops and workstations and even if I go and I Say I try to verify a signature on an option ROM or I try to verify a signature on a kernel that I downloaded over pixie Whose keys do I actually trust? in a Current secure boot system those tend to be Microsoft keys because they're the only ones willing to run the CA But does that really give you ownership over your platform and The other thing is what's your actual boot policy if you can't verify Something that you're about to run If you download a kernel and the signature check fails, what do you actually do? Maybe on a on a on a laptop It's acceptable to throw up a scary screen and say this is not safe You can't use your laptop because someone has compromised you But on a server system that presents a nice attack vector as a denial of service attack where you can take down a whole data Center just by serving something. That's one bite off So these are all questions that I don't have an answer for for you But that I want to give you the tools to answer and program for yourself So if you have Linux as a bootloader you can implement pixie booting and go pretty easily the protocols are out there as packages The the configuration file is not that hard to parse You don't have duplicate drivers from EFI anymore, but more importantly, it's trivial to use modern features, right? Go already has an HTTP implementation The Linux kernel already has an IPv6 implementation These are things that are somewhat well tested and audited and that we can reuse now Or if you're not something like Google or Facebook, you might just keep it simple, right? You bring up your your device up to the Linux kernel infirmary and then you just get a DHCP lease You download a kernel with W get You verify it with GPGV and UK exec, right? It can be that easy for your own systems However in in many cases, this is kind of infeasible for for companies that have large infrastructure for example on our servers If a kernel fails to verify their secrets stored in a certain location on our servers that we have to say Any subsequent stage after this doesn't get access to these secrets anymore, right? These are some kind of set of keys that gives you access to production services, so What I'm trying to say is right our security model is not the same as yours And whatever we end up coming up with is not going to be something that you can just take and deploy in your data centers or on your laptops You have to answer some questions about You know Where are your secrets stored? Do you have secrets like that? What do you want to actually happen when kernel verification fails? and You also have to take into account your hardware security model when you design something like this So I would like it if once you answer all of those questions you yourself can write a bootloader and go For your use case in less than 500 lines of code But if we give you all the packages to K exec to deal with kernels, maybe in in Some way deal with the crypto involved Then it should be easy for you to compose these things and write your own bootloader now we have already written some and Our projects for bootloaders a little bit in disarray, and that's another reason that I'm kind of here Like I want to encourage people to do this on their own and also to contribute back to us because We like the users of this project Google and Facebook. We're busy building bootloaders for our own data centers But they're not the things that are going to be out there The same way that we use them in the data center So it would be great if other people were to look at the stuff and see if it's useful to them and to make it more usable So there are a couple different Bootloaders that exist for example with TPM support with DCP v6 But then there's another one that implements pixie as you know it only via dhcp v4 So we're kind of in disarray, and we would love some help And the plan is to hopefully have a shared code base of packages That we can all use and then implement our own commands that are the actual bootloaders So I'm just gonna quickly mention the people involved Which are you know Facebook and Google are testing this on their servers at the moment There's actually a French-American reseller of decommissioned open compute platform hardware that sells servers with Linux boot and you would already on them So you could go spend seven hundred dollars on a server right now to get it And then there's various other smaller people involved at the moment and Kind of to reiterate what I was saying before we want to provide you with tools to build the systems You need not policies right you should be the owner of your platform. You should get to decide How things actually work in firmware and how your bootloader works? and somewhat orthogonal to Linux boot is We would like it if these platforms give you security and user freedom at the same time It seems that a lot of platforms make trade-offs there That they don't actually have to make so if you for example buy a server that has Intel boot guard enabled Intel holds the keys to your I'm gonna say firmware universe We would like there to be server hardware that allows a change of ownership of those keys for firmware I don't care if that's actually an Intel boot guard or is that's an external chip that for example? Google is working on right now and many other companies are too and the the third thing that we want to kind of encourage users to do is Have systems that instead of throwing up a scary screen when something goes wrong and telling you like you can't boot Boot anyway, but don't allow you to get access to secrets or boot anyway and tell you it's not safe But you can still use your computer, which is kind of the Chromebook model With with systems that are available on the market right now like boot guard based server systems What you'll just get is a server that doesn't boot and that's not especially helpful to you if you're trying to debug things One of the things that we're doing is we are engaging with the open compute Community, so there's an open systems from our project Where you have where we have weekly calls actually bi-weekly calls now every two weeks Where a couple of vendors are listening in It's our way of trying to hopefully encourage some vendors to enable the option for people to use this on their system and There are a lot of things in the future, but I have about four minutes So I want to leave some time for questions we would love to see more contributors and Here are a bunch of links and also by the way, I brought stickers. They'll be outside on that table But so time for questions Yeah, I think By putting entire linux into the boot lord, don't you increase the surface attack? Um Yes, and no So one thing you have to realize that is that the UEFI you're running on your system is an incredibly large system That has multiple millions of lines of code of attack surface The other thing that we've realized is you can compile a pretty small linux kernel with only the features you want Start with config tiny config and add some you'll get a kernel that is no more than 800 kilobytes And consists of only a couple thousand or a couple hundred thousand lines of code so The answer is it depends but we hope that linux will be a smaller attack surface that is That has had more eyes on it than UEFI Have you have you worked with the ony or onie project the open network install environment? Which is another open compute project. It's effectively a bootloader for white box routers for sort of advanced pixie They are slightly different from us. I haven't spent a lot of time with that project, but They are not a bootloader that actually lives in firmware and I think the definition of bootloader is getting meddled somewhere in the middle there too, so It that we should talk a little more with them, but I don't think it quite fits together Okay. Yeah. Oh, they're primarily a linux distribution for what again? For Facebook switches fbass I Realize that this is a silly question But do you think that in our lifetime? We will be able to see this on normal consumer laptops I really hope so we do have some contributors doing it on normal consumer laptops But if you want to do it on modern consumer laptops It either has to be a Chromebook or a system where you find a boot guard vulnerability, so We have to work with vendors to Enable users to be able to do this or encourage vendors to do it themselves So I hope soon, but I don't want to promise you anything Mike back there Okay, I have a question as well So if I seem correctly on your slides, so you need some downstream patches for the linux You needed you needed one patch and we're getting rid of it pretty much in a couple of days Okay, so you're working on up streaming this. Yeah, okay. Thanks. Have you It's on have you approached Intel on this? Actually Intel is on those open systems firmware calls and we also meet with them every once in a while Intel is pretty supportive of this project Because you can frame it as a this is Linux as an EFI driver project so They have no problem with this We're actually hoping that they will enable us to To do this on some systems with blobs they provide. I don't know that they will But I'm hopeful. Yeah So if I want to try Linux boot by itself or you root by itself Could I so in the case of Linux boot if I have access to my EFI shell and I can load Dixie modules Could I presumably take the kernel that I already have? And kind of like get it into the EFI shell mod Actually, what we usually start with is we replace the code section of the EFI shell with a Linux kernel Right. So if I have access to my EFI shell like presumably can extract it from my yeah firmware So I can put my existing kernel into it and and try to do what what will I expect to see if I just loaded that module from the Shell is that a way to test it load in that module from the show and so just to execute the Dixie module Oh, it's boot What would I see at that point? It's you should just get Linux actually Like it should just work one of the one of the issues There is you might not have enough space to actually do that without removing some of the drivers that you won't need anymore Right. Yeah, so that that really depends on your system though. Yeah So I'm pretty sure I can do that in my in my shell So then in the case of you root could I already now try that? Go based in it ramfs on just my laptop already today. Yeah, you can use it now There's a QMU environment too. So it Yeah, all right, I think I'm out of time. So thank you very much