 What was traditionally known as the BIOS, now it's usually EFI but we're talking about, do it, it stopped working. Well, that's gonna be annoying. This little guy, this is the one on one of the Intel boards. It's a piece of flash, it's not very big, this one's an 8 mag I think. And although the old days when you were trying to update these, you wound up with something like this, right? And then eventually it got a little bit better and you wound up with this, but it's still kind of crappy because what you wound up doing is that, right? And nobody likes doing this, you wind up spending a day building a floppy disk or faking a floppy disk with a CD, right? So I don't like that very much, you don't like that very much. It's kind of bad. And on your firmware, when you look at it, if you look at your vendor and hit the read me, you'll find something like this with, you know, there's stuff on it. That's bad. You can literally have a root exploit that the person can exploit and stay on the machine after you do an OS reinstall. Maybe. So you might wonder what we can do about it and the answer used to be not very much. But then somebody made this little machine. You probably didn't expect me to say how great Microsoft is or how great the surface is, but it has a feature that is firmware updates organized by the operating system without having to have any special tooling. And, you know, that's a pretty good feature. But we can't use it, right? So they've just got one more feature to sell Windows with. But then they did the right thing, which is really weird. And they wrote EFI specification in a standards body of how it works. And then they started writing requirements in Windows for how it works. And the structure of the PC market is that they produce a lot of requirements that look like this and have very, very, very specific details in order to put the logo on a laptop. And the logo comes with millions of dollars of marketing money. So if the laptop vendor puts the logo on it, then their laptop is profitable. And if they don't, it isn't. So they have to do the requirements and there's audit processes and everything. But that's a tool I can use. And then somebody else made this machine, which is an Intel Galileo. And it gave me something that we can use to inspect what's going on, because it has a lot of debug parts. So that was my desk for a while. There was actually, I looked for a better photo, but oh well. And I learned a lot of things that they say how it works aren't really true. And then wrote a lot of code. And then somebody sent me an actual machine that supported this that I could test on. And so I wrote a demo. And that's my cue to play the demo. And this is hooked up, this is from my phone, obviously. And this is the demo I gave to the Red Hat desktop team two years ago now. And it's just set up the machine to reboot into a firmware updater. And then this is the screen telling you it has no signal. And then that goes away. And for a while nothing happens. And it is a long while. So I'm going to fast forward and then rewind to where something happens. So that's a minute later. And somewhere in this 15 second window, thanks to you streamer. And it says this. And it sits like that for a long, long time. And then eventually says that. And then it reboots again. And so that's not a great UI, right? Nobody wants to see that. You're just looking at it going, what the hell is going on? But this was the first box that anybody could get me that could do it at all. And so about four minutes in it reboots into Linux, which you can see there. So thank you to Harry at Intel and Jeff at InsideBios for actually sending me that machine and getting it working and another one like it. And then Richard Hughes started doing some things that we're going to come back to on the other side of Linux on the UI end. And then I wrote some code to make something better than what you just saw. And then somebody else sent me another machine to test it on. And a while later it looked more like that while you're doing the update, which is a bit better. So thank you to Mario and Jared for sending me a laptop that had the support that I could actually make that work on. So you might wonder how it works. And it's actually really simple. This system firmware builds a table in memory that has a list of every part that has a firmware on it that it knows how to upgrade. It's a GWID and a version number. And a version number that's the lowest number you can upgrade to, which means you can go backwards if it supports it. And then there's a function called UpdateCapsule where we load it into RAM and we call a function. And then we reboot the system and when you reboot the system it applies the firmware update. And you have to do that because during booting it locks off the part so you can't do any writes to it. So the theory is you don't actually want anything running while the OS is running that could change it because that's exploit city, right? And then the Linux parts, it's actually fairly simple as well. I wrote a kernel driver that just sticks all the data in CISFS and I wrote a program that copies the firmware file onto the EFI system partition, sets a boot variable to say, hey, next time you boot, run this program and sets a variable that's in VRAM with some state in it. That's just, you know, hey, by the way, here's what you should look for. And then the stuff Husey was writing. He wrote a whole lot of code. He wrote a daemon that is activated when something looks for it because it's Debuss. And he wrote software into GNOME Software Update so that if your system vendor provides a system of firmware update, it'll show up in the normal software update stream. So you can just click and apply and it does all this and it's beautiful. And he also wrote the Linux Vendor Firmware Service which is a service we're doing for the hardware vendors so they can upload their firmware to us in actually the Windows Capsule file, a CAB file that they would update to Microsoft. It's the exact same file. And we distribute to all the Linux vendors so everybody can use this and we suck it down and just that's how you get the firmware from the vendors so you don't have to go hunting for it on the internet, right? And he's done a fantastic job so if you see him, although I don't think he's here, thank him. That's actually the whole of the talk, but who wants to see me do it? Thanks, Laura. Now, let's see. Right now, the only thing on my machine you can see it's set to just boot Fedora. Yeah, but what did I do? What did I do? All right, I have to have completion for it now. As you can see, I've got a lot of options to choose from because, well, I've done this a lot of times. And now it's set up so there's a second boot item. It's not normally in the list of things to boot, but it is the thing that will boot the next time I boot it up, right? It takes a few minutes to fill it a while. Hopefully it'll sync video otherwise I'm going to hold the laptop up waving it around at you. Okay, well, waving it around, it seems to be. On this one, it takes about four minutes, and Dell has packaged all the firmware for every part in one bundle. So right now it's updating the system BIOS, so it's the main EFI image. In a second, it'll do the embedded controller that drives the keyboard and suspend and all that stuff. Then it'll do the Alpine Ridge, which is the Thunderbolt part. I don't remember if there's anything else there might be. It's just going to keep going. So now it's the USB controller. So I guess we can do Q&A for however much time. I think that's actually the dongle, but maybe not, because I thought I had seen it do it with the display port output, but we don't have a display port here. So on this one, it's all one bundle and you can't choose anything. Now, in theory, you can have anything in the list. You can have a whole list of, it's GUIDs, right? So you've got a scattergather list of physical pages and a GUID, and you say update this GUID with this scattergather list, and you do as many as you want and then reboot. And that's a little scary because you don't know what order to apply in or if it's going to reorder them or if it knows how or if there's a specific order you should use. So Dell bundled it all as one. But in the future, one of the things we don't support yet is PCI option ROMs because none of the vendors have shipped the firmware support for the PCI card to say it has an option ROM you can do, right? So sometime we're going to support that, and that'll have to be two separate ones. Oh also, one of the things that Richard Hughes wrote is from GNOME software you can also do USB devices, which is entirely separate infrastructure, but from the interface it looks exactly the same. And so those would all be separate. We really actually spend a lot of time making it so the Windows cabs would work. So you could take the Windows binary, so we don't want the hardware vendors to have to come and make a different firmware bundle for us. We want them to use the same ones Windows is using for the least friction for them to encourage them to do it. And we've actually been getting a little bit of support and it finished. I would have thought it would reboot but it powered off. There it goes. If it fails I have two fancy screwdrivers, a pile of the pry tools and a firmware programmer in my laptop bag, and a backup from Friday. And that's the, yeah, well that's the interior password to decrypt the disk screen. So don't watch me do that. That's the one thing I, right, he's asked, did the firmware update or use Shim, so is it secure? It's mostly secure, it is secure in that regard. It runs, it's signed by the red hat key and so Shim will load it and like secure boot is on on my laptop right now. It is not running something at all that's untrusted. And in theory, the firmware blob has to be checked for security, that it's valid. In the US there's a publication that's called NIST, the National Institute of Standard Technology, special publication 800-147A, which is a really catchy name I know, that says that any firmware update has to be cryptographically verified with at least shot 256 sums and RSA 2048 signatures before a reply. And it also says that some really kind of dumb things like the key ring isn't something that's allowed to be exposed to the operating system. So we can't verify that it actually does it. So at some point we're going to have it so that we're checking a signature that's on the cab file as well and we're going to be checking that, well we already checked that when you download it but we don't check it after rebooting before applying the update yet. The reason for that is I already have one open SSL fork I have to maintain and I'd like that to be zero, not two. So we're waiting for the UEFI 2.6 has a feature where it gives us an RSA signature verification function but it hasn't rolled out to a lot of machines yet. So once that exists in the wild then we're going to be able to do that pretty securely. So you're asking if when they upload to afterupd.org do they have to authenticate? Right now we're creating accounts for it on a one-on-one basis so it's people we've talked to that can upload and they have to sign in and upload it. And from a practical perspective the thing they're updating is the Windows cab file that's signed by Winqual. So they've done two-factor auth and smart cards at HSMs to sign it to upload it and then they download it again with a signature on it from the Windows CA who are surprisingly fairly good trustworthy people who are very straightforward in how you deal with them. We've had a lot of success working with them actually because of secure boot where we went through a lot of politics and eventually got a pretty good working relationship with them. It's Infador 25. Not up to me. He wants to know when it hits distributions. I can't control what goes in when. I wanted it in 7.3 but it's not. But it's Infador 25. Two or three are talking about it and so I can't really talk about them but there are some that are still talking about it. Dell has really gone out of their way to support this. They've spent a lot of engineering time on it. They spend their patches when something doesn't work. They've really done a good job. Intel has done a fairly good job inside who makes a lot of system firmwares, gave us test equipment really early on or rather we found an Intel desktop board that they'd sent us a firmware for. So we flashed the first firmware on and then do firmware updates from there. That's really the ones that have really gone out of their way to help. All right, well, I went through the slides faster than I expected but... He wants to know what about open source firmware. We could support it if they support it. If you're loading... So one of the things I built as a test rig very early on was I wrote a driver for EDK2-OVMF which is the UEFI reference implementation. I wrote a driver that made it so you could set a UEFI variable with a particular GWID and stick in the header that goes into the table that's the output. Then wrote a driver so when you do the firmware update it rewrites the variable. So I can test the API out without actually testing any firmware update and make sure that the API actually works. So we have that in a branch somewhere on GitHub and the upstream EDK2 which is BSD licensed has support for all of the front-end API and some reference implementation of here's how you flash an SPI flash part but it doesn't have any of the how you control this one machine glue. So you don't know... Like when you ship a machine you basically build that firmware and then add in a binary blob from Intel or from whoever that says here's how you support this chip set and mostly it's like timing data and that sort of thing. So if they have... Like if you loaded core boot and then loaded EDK2 from it and they wrote some back-end for that API then it would work. So his question is is it possible to use our servers on any Linux distribution? Yeah, we don't want in any way to be a Red Hat only thing because it's better for everybody if everybody gets security updates everywhere and that's a big concern for us. So we initially wrote it just because we needed to have something at all and we're working with other distros to try to make sure they don't have concerns like do I get all their user data while they do it or do I know how many of our two machines are updating and they don't? That sort of thing. We're working through that because we really do want this to be something that's applicable absolutely everywhere. So the question is I have my tools for if the flash fails what can I do? Let's take the laptop apart and find the little test clip and all that. Is anybody doing last known good where they've got two firmwares and it'll flash it to the other one that's not in use and then try to boot that? The answer is usually most of the hardware we've seen so far is all laptops and that's actually pretty price prohibitive because if your margins per laptop are in cents the difference between an 8 meg flash part and a 16 meg flash part is about three times the price so you're going from something like 70 cents to something like four bucks and so nobody's ever going to do it. On servers though there's a pretty good chance that you'll have a system where the actual flashing is done by the I've forgotten the acronym but the service processor and so what'll happen is either we use this API or another one depending and we'll upload the flash the new image to the service processor and then it will apply the update and when you reboot it'll have a watchdog timer that if the system firmware hasn't talked to it soon enough then what it'll do is just flash the old one in and then reboot it again. Thank you and I'm hoping we can bring this to all your systems. It works.