 All right. My name is Brian Richardson. It's weird. I actually get paid to talk about firmware. It's an odd job. So the fact that you're actually hearing interested in listening to it justifies my existence in the corporation. So thank you very much. They paid for the plane ticket. So, you know, we have to do that. I put this material together with help from people from Tiana Core Project at Intel, Vincent Zimmer and Mike Kinney. Also, I had invited Richard Hughes, who maintains the LVFS to come and talk. He wasn't able to this time. So I've incorporated some of his slides into the materials. So even though some of those will have an Intel logo in the corner, we didn't make them, and I'll point out where those are. Okay. So, a quick check of the room. Who likes firmware? Or is interested in firmware anyway? If I say who likes firmware, some of the hands will go down. Who has had a great experience updating firmware on a device? Yeah. Great experience. Yeah, okay. I have recently, actually, that the hand has done this over the years, and I used to be paid to write firmware, like an assembly code for money. So updating is hard. It's harder than it should be, and we're going to try to fix that. And it's actually now implemented in a way where it can be done in the open. So I can talk about the entire process and how, if you're doing development, especially on Linux, how you can start working on making sure that that code works as we move forward. So first, let us define the problem space. Tech presentations, do you have infomercials here? They probably, if you do, are you familiar with the infomercial? All right. Who's jet lagged and stayed up in their hotel room and watched a terrible ad about, like, some tape that covers over leaks or a knife that cuts through every... That's called an infomercial. We invented that. I'm very sorry. It wasn't my idea. They didn't call me. But the infomercial always has three things. You have something stupid, a proposal to fix the stupid thing, and how to get the thing that we just proposed to fix the stupid. So first, here's the stupid. In technical terms, there is a low attach rate for firmware updates in the field by end users. In my Southern vernacular from America, y'all ain't an update in your firmware. Now, if you read CVEs, y'all should be updating your firmware. You saw the last couple of talks. Yeah. We probably ought to do that every once in a while. We made it hard for end users to do it. In the old days, the 90s, when I started doing firmware, you got a floppy disk. That's a 3D-printed version of the save icon. You put it in the system because we didn't have USB. And you ran a program. You had downloaded off of an FTP site because we didn't have proper... We just got an overgo for it at this point. And this beard is gray, naturally. And then you would run that program, and you hope you had the right firmware for the right board, and then you hope that there wasn't a magic jumper you had to set to make all that work. So you have to open the system to update the firmware, and sometimes you would get what we like to call a brick, which is something shaped like your computer, but now doesn't boot. The process was designed for nerds like me, who got engineering degrees and learned why we use X or on a register instead of move a zero into the register. So the answer takes one less opcode. Users need to update that thing. Otherwise, every time you go home to visit your family for the holidays, you're doing the updates. You don't get to actually hang out with your family. You get to answer stupid questions and update their laptops. So it creates an environment where we do a lot of work in the industry to fix all these problems and do cool security talks about them and get paid to go to conferences in Las Vegas. And then nobody actually uses the work, which is just entropy. That's bad. All right, in Linux, it's worse. One, because OEMs ship windows. They might not be testing Linux. So if you get a nice shiny ultrabook style thing and you put your favorite distro on it, we're not going to say which one, because if I name one, it'll be wrong, and I'll get thrown out of Boston. So they're not testing that thing. Now, on Windows, typically, it has been a Windows PE or some other type of Windows runtime service to update the firmware, which means in Linux, they didn't implement it. Now, let's think about the technical complexities of doing this in Linux runtime. So on x86, Intel, firmware is mapped to a 4GB space. It's within the 4GB space. It takes up the size of the SPI part and it's mapped below a 4GB register even if you have more than 4GB of memory. Why? Old people reasons. Address mapping. So, okay, you have to, if you're updating the firmware, what you're really doing is writing to a fixed address. Now, anybody who's been protected in programming in the Linux kernel knows you don't write to a fixed address. Things that do write to a fixed address on Windows unless they have special permission to set off virus protection. And in Linux, we're normally done in kernel drivers. Now, do all the OEMs disclose how they update their firmware? They do not, which means it would be a closed sourced Linux kernel driver, which means it would have to be compiled against specific kernels, which means it would always be the wrong kernel because y'all are rebuilding your kernels all the time, and kernels evolve and the drivers would not follow up. So it creates a different environment where the updates that are done are not applied to a subset of users that would love to update their firmware and are probably more technically capable, but can't. So there is now a solution space, which solves a couple of problems. One, it standardizes the delivery format. So you can do things in a standard way. It's written down in the spec. Everybody can read it. And it's OS independent. It's called a capsule. I'll explain that coming up. In infrastructure for update delivery, you must have consistent protocols. You must have a consistent way of doing it no matter the operating system. So if somebody in BSD land decides to implement it, same process. And it's OS based staging. That means the OS delivers the firmware to the system but does not necessarily do the update. And then you have this modular firmware thing. The firmware should be smart enough to update itself. If you don't trust version one of the firmware coming out of the factory, you have a different security problem that I cannot solve for you. It usually involves hiring a very large man with a stick and he makes sure that people don't put, you know, I'm not advocating violence, but that's usually how those things work. So that's a chain to trust problem. Different topic probably already presented here. But let's assume you trust version one of firmware. That's probably the thing that should do the version two update. Version two should update version three and so forth. Again, if you don't trust that guy stick. All right. So let's talk about capsules. So UEFI actually recognized that this was a problem when we wrote the spec. Now I'm, some of that spec is my fault. I apologize. The initial chapter in UEFI 2.0 about USB 2 with high speed. I copy edited. Sorry. So the idea in the spec is that UEFI capsules define a way to encapsulate firmware. This is a standard way you can take a firmware payload and deliver it to a system. Now how that's processed is system dependent, but the capsule should be the same. So it shouldn't matter if it's an expansion card, shouldn't matter if it's in built firmware, if it's the graphics driver, use the same system. So some developer somewhere generates a capsule and they sign it. They sign it. We don't deal with unsigned capsules anymore. And they don't sign it with the test key and they definitely don't put that test key in their shipping firmware. I have to say it because somebody did it. I'm sorry. It's like on the airplane door when it says do not open during flight. That sticker probably wasn't there on the first airplane, but it's definitely there on the second. We're not going to the same thing. So Toledo, let me get out here. No, that's we're not doing that. You go to a distribution system of some type. Now it's nice to have a repository. Windows has who windows update. We're not talking about that here. We're going to talk about Linux vendor firmware system LVFS. This does the same thing because it's all standards based. LVFS works off exactly the same system as Microsoft does for capsule delivery. If you have a windows surface device, it uses capsule that has for since the first surface. If you have a Lenovo laptop, you probably have started seeing more updates available for those in Linux. Those are all using LVFS. LVFS then allows the system to distribute the same capsule to the same model of platform and allows it to locally update. So capsules are in a spec. You're not going to read it. I took part of that and put it here in the spec. By the way, it's not a novel. You don't have to read the whole thing. It's just parts that work for you and apply to your work. I have read the whole thing once and I didn't do it again. The capsule has a standard header that says hi, I'm a capsule. It has other information. FMP, which is firmware management protocol, which is the protocol that actually processes the stuff on the firmware side. Authentication info because we signed our capsules. The payload header which describes what you're dealing with should process it and the actual payload. Now what LVFS cares about in the delivery is actually the capsule header. The capsule header tells the system firmware, hey, this is this kind of firmware. Can you process it? Can you process the update? Forget about the payload. Just look at the basic header. Can you deal with it? And the system firmware has FMP drivers. It has a set of GUIDs. Everything in UEFI runs on GUIDs. So the driver's got a GUID that matches and says, yeah, I can use that. I got a big table called ESRT an EFI system resource table tells me all the firmware in the system. What are its GUIDs? What versions is it running? And then through that process it can verify the keys. They should be signed by the same people who made the first version of the firmware. Again, that whole why we use certificates. So that driver is signed. Same public keys as the original version. And it updates in the flash and you update the ESRT so that somewhere in operating system land you can observe the results of the firmware update. Because way early on the only way you could tell if the firmware updated was to go into the BIOS setup or look for the screen that just blinked past you as the monitor warmed up and figure out, oh, that was version 102. Crap, I didn't write down what version I just updated over. I hope it was 101. Nobody knew. And then the operating system couldn't get that information until we started publishing it in ESRT. Now I am presenting on UE5 firmware, which means I am obligated to show at least one slide based on the progression through the boot. I don't... If you wanted to see, hey, is it going to use the big green H? Yes, I am. Those always show up in UE5 presentations. You can just play UE5 bingo. That's the free square. So read this left to right. You have the phases of the UE5 boot process and then top down how we go through each cycle. So you'd first do a normal boot. The OS has an agent running or you go out and you're on, say, Ubuntu. Sorry, I know it's the wrong distro for most people. And you click on the software update thing and it says, huh, one of your updates, a system firmware. I can go get it for you if you want. Sure, that sounds cool. It gets it stages the capsule. Staging the capsule just means I put it in a magic space in memory and I sent a note to the kernel that says, I put a thing in a magic space in memory. When you reboot, please tell the firmware there's some magic thing in memory so that it doesn't do DRAM initialization and wipe out the thing I just put in memory. So it does the reset. Firmware reboots and goes, ooh, there's a thing in memory. I should probably do something with that. Now there's a couple different ways to load the capsule in most systems. It happens at the boot loader, like a grub or Windows update kind of situation. In other cases, the firmware just directly deals with it on boot. But either way, there is a update capsule routine, which then causes what we call a coalesce. You could have multiple capsules staged in memory from different LVFS agents. It could be the embedded control or the ME firmware. Lots of things can get updated on the system. A modern system is really just 20 coprocessors in a trench coat. It's not a piece of firmware. It's pieces of firmware that interact together. So you coalesce the capsules and go through the loop of processing them until you run out of capsules. And because the firmware is up doing that process, it can say yeah, I know that capsule. It's native firmware. It's the UEFI. I'm going to update that. What's my next capsule? That's an embedded controller. It's an FMP driver for that from the manufacturer. I don't know how to do it, but they do. I'll pass it to FMP. Or in the rare case, something slips through. It goes I'm not supposed to. I don't know why I have this. I'm going to throw it out. It doesn't match anything on the platform or it doesn't validate because somehow maybe the signature got broken. And then you update the SRT and you have updated firmware so that you don't look at a blank screen in the open source implementation. We actually implement two different ways of doing the update process. One is graphical and one is console because some of us still use serial ports. Yeah, I know it's 2020 really? We're still using serial ports? Okay, modular philosophy. So UEFI EDK2, which is the underlying open source project that most people use for UEFI. So that is modular. So we have this modular idea that if we componentize everything in manufacturing then we make it easy to update using capsules when we ship the system. Not that we get very granular with it, but we can at least group into a couple of clusters. There is an open source implementation of this. Intel has gotten, in my opinion, way better about putting out more code for actually running on platforms. So we've come up with an idea called minimum platform implementation. Sometimes it'll show up as MPA. Most people call it min platform. Just means that we put out a reference design, say Whiskey Lake, which is the eighth generation core product. I know that because I looked it up on ARC. We've got open and closed source modules for that. There's an FSP. There's some redistributable ME kind of stuff that goes along with it. We also have a Whiskey Lake RVP reference validation platform model in Tiana core. So you can build a minimum amount of UEFI you really need to build a UEFI thing. Now, that means that you could morph that and say put it on the Aeon Up Extreme available for $299. It's not an ad. It's just letting you know it's a cheap board that you can actually work with in Tiana core. So now this gives you a way of playing with this code and doing some development that say if you were going to make your own device you'd need to write an FMP DXC component. Not the whole driver. The stuff in blue on this slide is standard and open source. It should not change generation to generation you know, very much for supporting, you know, except from bug fixes for supporting the FMP update process. But if you are the device vendor, you want to write a library that implements FMP we have too many acronyms for that device. If you want to be a platform vendor and determine how that is processed, you write a capsule update policy library. That's the policy for the system. Okay, so that gets the basics of EDK2 out of the way. If you have questions hang on until the end I want to make sure I get through this stuff before we hit that level. Alright, now you want to distribute it. And this is the part where I pinged Richard and said, I understand roughly how LVFS works, but what would you want the audience to know? What are the top things you talk about? Because Richard uses basically the reason why you can do this in Linux. Even if I did all this stuff in firmware, if you don't build a distribution system that's consistent and everybody uses it, the whole thing falls apart. So Richard's built LVFS basically because he encountered the same problem on a lot of other kind of embedded and maker style devices. They had weird proprietary firmware and they had kind of weird protocols that weren't documented and he didn't care about disassembling the firmware, he cared about updating it on the system he was using. So sometimes in LVFS you implement a protocol that's completely closed and you black boxed. Sometimes they give you the protocol just enough information to write source code and then they endorse you're using that information, like headers for instance. And in some cases the whole thing's open and you can do it soup to nuts. So LVFS has two main components. One is FWUPD which is the update mechanism, the thing that runs on the system. That's 100% free and open and that is used by the users typically with a GUI. So if you've run the Ubuntu update system and you see the firmware shows up that's running under as LVFS. And it's seamless enough now to where a lot of people just do system updates and don't realize they've taken a firmware update until the reset. LVFS website is the data source. That's again free and open. It's used by the vendors. So the vendors interact with this to upload the information and get their stuff in there. Richard's built a trust model with the manufacturers to make sure that he's only getting the verified binaries. His phrase is LVFS. It's just a website. A terrible slogan for a bumper sticker. It does mean that there is some vendor secrecy for permissions just to ensure the right things go up there. And it can be mirrored. Mirror the whole thing and take off. If you want to build a mirror of it, great. It's just a website. But it's a website that is supported by so many vendors now. This list is a little hard to read. I would have not designed it this way. I always hate saying this is an eye chart. I'm sorry. But this is an eye chart. I'm sorry. The left to right is time. So it's months. And then bottom to top is, I believe number of I can't remember if it's a number of website hits or number of instances of firmware. But you can see the number of vendors going up as they sign the adoption agreements. And the nice thing is because it's the same capsule that everybody else uses, if they are building and signing that for windows, they just take the same thing they gave to Microsoft Windows Update and put it in LVFS. So they're not building a different thing for Linux or a different thing for BSD. They're building a thing and distributing it differently based on the OS target. Now there is some back end processing that goes in. There's another layer of security that Richard does through standard SSL. And there are some verifications run. He checks things on the back end to make sure that things are formatted properly. Like did you follow the spec for the capsule header? Because if the capsule header is trashed, it's not going to do anything good when it gets to the other side. And firmware analysis. This is pretty cool. He's actually checking to see if, oh, I don't know, the Dell firmware doesn't have any attestation checksums. Now, we can't stop Dell from putting it up there. But he can make little green and red check marks of you when you pull it down and understand what you're getting. And then by having these verifications by running things like chip sec, for instance, in the background, you can do checks on, say, a whitelist blacklist. I know there's a grid out there for a driver that's been blacklisted. It's got a known issue. I can flag it if it shows up in my system. LVFS should be its own 20 to 25-minute presentation here. I'm not going to try to cover the whole thing. But he does have a very good long-term roadmap setup and it's something that you need to, I think if you're interested in firmware updates as a problem, look at this because it covers way more than just UEFI. It gets down into lots of different components and different ecosystems. So it's definitely something worth looking at. All right. So I told you about the thing that we're doing that I don't want to do anymore. I gave you a potential solution for stopping doing the thing that I don't like. And now I'm giving you homework. Not like real homework. You won't be graded. Attendance isn't mandatory. But in summary, basically, EDK2 supports an industry standard for capsule update. We have a full implementation of it. It can be used on hobby boards as testing. So the Miniboard Max, that's based on BayTrail, the Aeon UpSquared and UpExtreme boards that we've supported. I don't know how much ARM stuff is using capsule, but I'm pretty sure there's a few of them out there that at least could get it wedged in. The ARM guys are doing this notion with their heads good. But this simplifies authentication, system update pre-check. Did I plug in the power adapter on the laptop and could brick it? You can check for that kind of things and have the firmware decide it doesn't want to do the capsule update at that point and through something like ESRT, again documented in ACPI, notify the system if the firmware update was good or bad. So if you're building a system or you know somebody who is, you should probably sign your capsules and make sure they get distributed. In the U.S. we have a thing called NIST, National Institute of Standards and Technology. And the guidance is please have updatable system firmware in the field. Please have root of trust involved in the update so that you know the thing that's supposed to be doing the update is doing the update with the thing it's supposed to have. I know EU and England have this similar kind of regulations or references to it and if you want to sell to the government, this is in their purchasing agreement. So if it doesn't meet NIST and you want the sweet sweet tax dollars back of buying your product, then you're not going to get it. If you do updates, I'm sorry, when you do updates, update your firmware, put it in LVFS and Microsoft Windows update. I'm not here to plug in Microsoft, but if you have a device and it ships both configurations, make sure both configurations can get the update. You worked hard on resolving those CVEs, make sure somebody actually benefits from it. If you have a device vendor, put it in the requirement to them, and if they sell me a thing, it must have capsules for its firmware update, so I can incorporate it in the manifest for my system. And please have a recovery solution. If you encourage more people to update their firmware, the rate of failure won't go up, but you'll have more instances of that rate of failure. Like 0.3% isn't a lot, but it is when more people do the thing. So it works 100% of the time, except when it doesn't. NIST 800-183 or 193 is the requirement for that, and there is a recovery solution in the Teonikor solution for capsule. If you're a developer, hi, that's most of you, you're working with an open-source community, make sure that you're trying the stuff out. If you make a Linux driver and it's got a firmware impact, go get a board that supports it and see if it works. These are all links. The slides are already up, because I paid attention to the guys in the blue shirt and said poster slides before you get here, so thanks for reminding me. I almost didn't do that. The EDK II threat model, some of the risks in capsule are based on that threat model, and there is some information on Windows Update only because they do actually document some of the process of formatting the capsule fairly well. In EDK II, we tried to make it easy by using a bunch of Python scripts to do some of the things we thought were harder. All right, I saved myself a couple of minutes for questions before we get to that. Thanks everybody for sticking around. I know it's late and we should already be having a beer right now, but we'll solve that in a few minutes. Okay, I'll take this one up here first. Can we leverage, well, do you think that vendors are approachable so that we can use capsule to also supply custom firmware images, not just the vendors, so they may audit and sign them? Okay, so the question is, because we have this mechanism that is open friendly, could we get a vendor to do another version of firmware, like a custom version? So essentially you want the, oh, I'm owning my system now, so I want to put my own firmware down. Potentially. But it's not initially designed for that. Because of the signing mechanism, you would have to have the vendor sign that firmware, or the vendor give you the private key. That probably won't happen on purpose. If you get the private key by accident, don't go signing other people's firmware. That's rude. But it's tough to implement. I think you would have to have a vendor who's friendlier to that sort of situation, but that vendor might also assume that they're going to do a physical presence, i.e. I open the box and put the thing on it, and then you're going to self-phone the firmware by establishing your own key and signing your own stuff. So I think that's a quasi-supply chain issue and quasi-friendly vendor issue. There was a question, I'm going to skip you, only because we can talk when we have a beer later. When you are updating several capsules, what I'm updating is several capsules? Okay, so the question is, when you have multiple capsules, what is the order? The answer is the order is not deterministic. As far as I know, on the current implementation at Tiana Core, and Lafa, correct me if I'm wrong on this, it's just when they show up. So they get put in memory and the firmware has a process to coalesce, which just basically means count all the capsules that are sitting in the magic space in memory. So I don't think you can have a deterministic order to get in and change that policy, but you can't guarantee that everybody has that policy. And there's a sign there, we can talk in a minute, but again, appreciate everybody for sticking around and find me afterwards. I'll be around here tomorrow a little bit or my contact info is on the slide. Find me online. Thank you.