 Hi, can you all hear me? Cool. This is my presentation about firmware updates for Linux. So, who am I? I'm Richard Hughes. I work for RedTap on the desktop team. I've worked on open source stuff for about a decade now and in RedTap for about six or seven years. I maintain a few different bits like package color D, U power, FWD, AppStream, G-Lib, Nome Software, Nome Power Manager, Nome Color Manager, and sort of do drive by fixes to quite a lot of the Nome and desktop stacks. And this firmware thing is kind of the result of something that happened to me. A few years ago, I got a laptop from Dell and the suspend would work most of the time. Like nine times out of ten, it would resume fine. One time out of ten, it wouldn't. Which I kind of put down as a kernel bug but wasn't serious enough to investigate. And the USB 3 controller would most of the time work and sometimes would cause a hard deadlock of the system. Which again, I kind of put down as Linux being not awesome, some sort of kernel bug that I couldn't debug and couldn't find the problem for. And then talking to the Dell engineers, they said we fixed that in a firmware update years ago. So I looked on the Dell website and sure enough, there was seven or eight firmware updates I hadn't applied for my laptop, one of which fixing the resume issue, one of which fixing the USB 3 issue. So if a technical guy like me didn't realise that this firmware update would fix my problems, the end users just haven't got a clue. So if we actually realise how to find a firmware update, usually you have to identify your hardware very specifically. So the same physical Nvidia graphics card that looks identical to a different one could need a different firmware because it's using a slightly different chipset or a slightly different memory speed or something. So actually finding out what hardware you've got so you can find the firmware update is really hard. You might have to take the computer apart and actually physically look at the label on the PCB. You might have to dig around on the vendor's website which might have gone out of business in the meantime. So actually finding out what hardware you've got and if it supports firmware updates is kind of problem number one. Second problem is actually applying it. This is my HP monitor which also has a firmware update and it only has... So if you're like typically it will have a Windows executable just here for Windows 7, which is kind of hard and sometimes if you're lucky you get MacoX as well. Now if the vendor's being super charitable they might provide you with a Linux download as well which is either a statically linked non-free executable that's doing God knows what or they might provide you with a 1.44 megabyte floppy image. Now I don't know about you guys but the last time I had a machine with a floppy drive for several years ago and even if you do the trick where you DD it to a certain format and strip out a header you can put it on a CD ISO but again I've only got one machine with a CD ROM drive nowadays and so even that's kind of hard. And so for my last firmware update I had to strip off a header using a hex editor, DD some random file to my hard disk, open up grubb.conf and do some God awful command and bearing in mind most of these commands that you're running is using non-free code that probably would brick your hardware if you got it wrong. So kind of if it's hard for Windows users to find this stuff out and apply it it's somewhat easier for MacoX because they control the hardware more than we do but for Linux it's just almost impossible. So this situation where we're resorting to USB disks and USB floppy drives and usually just not applying the update at all because it's just too much hassle. Now I guess one of the problems also is the update description. Now a lot of the vendors will give you some kind of update description about what the firmware fixes but most of the time it's rubbish. It's either hiding key details for either business reasons, they don't want to admit that something's their fault because they don't want to be sued into oblivion. They don't want to pay technical writers to write the content so users can understand it and so they just get some bios engineer to put something in American English that people who don't speak English will have zero chance of understanding. Other times they're using technical words like AMT and WMI which most of the people in this room will understand but I can assure you most people on the planet don't know what any of those letters mean. So we have this triplet of problems where we don't know if we can get an update for our hardware because we don't know what hardware we're running on or if it supports updates. When we do find an update on the vendor website we don't actually know what it's fixing and if it's going to fix our problem and then even if we do find the update we might have to do something drastic like install an obsolete version of Windows to actually flash the hardware and so it's kind of no surprise that nobody really bothers doing this stuff. So at least from a Linux point of view we suck at this stuff, you know? Like if I'm a 10 year developer for Linux who can't flash hardware on my computer to fix bugs that make it crash or what hope of real end users got of doing this stuff. So when I talk about firmware, most people think of their system firmware. Now I guess most of you will be familiar with BIOS. BIOS is like your system firmware for older type computers responsible just really for bringing up all the hardware on your computer so that you can run grub and actually boot an operating system. Now BIOS is awesome in the fact that it's been around for ages and it doesn't do an awful lot which means there's not an awful lot that can go wrong and usually with BIOS the things that are going wrong with the BIOS are the new hardware features, things like PCIe, link power down, ASPM type stuff so the new features tend to go wrong rather than the older features. But because BIOS is so simple it doesn't... Even if it does go wrong you can reboot your computer it's not like it's doing networking or anything too critical. Most of the BIOS sitting people know how to use and so the number of things that are fixed by firmware updates for BIOS after about the first year of the hardware release is almost zero because the first few months of the hardware being released the vendor will find loads of bugs, issue a flurry of updates and then after the hardware is about a year old it's EOL and there will be no more updates at all. So then UEFI came to the party. Now for a multitude of reasons BIOS isn't great on today's hardware and so Intel came up with an EFI specification which is now a universal specification but cross vendors where you can have a complete pre-boot environment for running networking and graphics card drivers. You can do loads of cool stuff with UEFI like provisioning and networking stuff but now all of a sudden this UEFI thing is its own OS. There's networking drivers, you can do PXE you can do full access to the hardware because you've got the Intel ME management engine which has got control over everything. So all of a sudden this nice little self-contained BIOS that couldn't cause too much damage you now have the possibility of security of router for your system firmware. You have the possibility of remote code execution with full control of all the hardware on your computer through not applying a system firmware update. So if you're like a bank and you have different compliance issues all of a sudden it's a big deal. Now the other thing we kind of care about is removable devices. So things like iPhones, oscilloscopes that support updates, all the random USB stuff you get that seems every electronic gizmo has a USB connection or power and also firmware updates. And about 10 years ago the USB consortium built this thing called DFU and they speced up a whole specification about it and it stands for Device Firmware Upgrade and it's a specification which basically says to a device switch from your normal mode of operation to a special bootloader type flash operation called DFU mode where you can put a new image onto the device, verify it and then boot it back up into application mode. It's a standardized API that maybe 30% of vendors use. If a vendor uses DFU you get three drivers in OSX and three drivers in Windows and until recently not awesome support in Linux. DFU devices when they are written to the specification completely are awesome because you can automatically do the firmware update without having to do anything. But most vendors either through a full sense of security or not quite reading the specification carefully enough usually support like a hybrid DFU scheme where you have to use various different vendor extensions to do the flashing or to apply different memory maps. It all gets a bit vendor specific. Also vendors sometimes forget that there's two halves of specification so it might be to turn a device into this upgrade mode you might have to hold down like a home button for five seconds while you turn it on which are sort of quirks we have to deal with when dealing with devices. DFU also has the problem where most USB devices aren't high power arm devices. Some are which is awesome and you can do like proper public private key firmware security but most are either like ATML or PIC microcontrollers where you just haven't got the processor or the memory ram all wrong for doing like a proper public private key security. So we do support like just like crypt modes like XT over DFU now. Now you can for the last 10 years you can update software a hardware using DFU Util which is like an old utility developed for OpenMoco but it's very much an old Unix type command line it's not a library we can use. So I spent a bit of time writing LibDFU which is a geobject introspectable library used by the later bits I'll show you in the presentation so we can do reliable, safe upgrading of removable devices. So what do we come up with? We needed like a framework to tile this stuff together because we've got all this metadata that we need to acquire from vendors we need the way of applying the update and we need a way of notifying user space components of any changes. So I built a demon called FW Upd. FW Upd talks to various different subsystems like it'll talk to system D for scheduling offline updates some hardware can only be updated when it's not being used other hardware can be updated live other hardware can be updated only before Linux is booted so you have to do it in a pre-boot environment so we have to talk to system D to schedule that we watch this FS for hardware that comes and goes so we can notify any GUI software of firmware that needs updating. All the metadata which is provided by the vendors themselves is stored in a standardized format called AppStream which is XML description of what the firmware applies to what hardware itself applies to and I'll explain more about that in a minute we use something called so for UAFI for a long long time we've been able to install random system firmware but we haven't had a system where the firmware could report to user space what hardware have I got installed that supports runtime updating so using UFI you could update your system firmware which is the obvious one but you can also update Option ROM on things like I don't know, Infiniband network cards or GPUs etc and UAFI supports that as well we need this table called ESRT which is an extension to the UAFI standard we need that from vendors before we can find out what hardware is available on the system to support updating luckily for Windows 10 Microsoft made ESRT a requirement for their WHQL requirement so now more and more vendors have realised that actually they have to do this as well so we get the support for free now I've kind of put in the thing a vendor provider we support by default the UFI support for system devices like PCIe type devices and system firmware and we support LibDFU for removal devices, USB devices but there's still a ton of hardware out there that does something vendor for instance like Colohug 1 I built a vendor specific HID protocol for updating and so I've had to write a Colohug specific provider for FWUPD which really means it provides FWUPD with what hardware is installed if it supports updating what version it currently has and provides a mechanism for actually blitting the binary file to the hardware itself now there's no reason why you couldn't write a BIOS provider for a legacy BIOS rather than use it if you have like a system that's not UEFI compliant or doesn't ship with that BIOS there's no reason why you couldn't write one for BIOS the issue is when you update a BIOS you normally either do it over SMI or ACPI and if you get it wrong even if you get the timing slightly wrong you can break your system pretty easily so most of the vendors I've talked to were quite hesitant to add BIOS updating to this new framework compared to UEFI updating which is a standardized interface that they can use quite safely so there's nothing stopping random vendors adding random code the only provider it has has to be free software we can't accept some random 2MB statically linked blob which does unknown things on your rail server running on some bank infrastructure so now we've got this nice demon that's providing us with changes and it's kind of insulating user space from all the grotty hardware details we've got an online tool called FW Update Manager now this is primarily used for geeks and people scripting stuff so if you're running like a 10,000 nodes render cluster you don't want to be installing free DOS on floppy disks and going round pressing enter with a VGM monitor attached on every single server and so you want to be applying this stuff automatically using something like satellite or fingers crossed in the future we'll have something like cockpit driving this thing as well so it'll be a point and click we also have clients like known software talking directly to FW Update which means we can provide a very high interface for providing the user an easy way and a safe way to install system firmware offline and also for removable devices yep, that's all I want to say so the biggest challenge I've had in the whole project isn't assembling the code the code was pretty easy assembling all the pieces and writing libraries and inverting engineering hardware that's the easy bit so I wrote this web service called the Linux vendor firmware service the LVFS and with the LVFS it basically allows the vendor to upload firmware files which we can check and validate or verify or QA and then securely we can repackage them and send the metadata to clients and firmware if the hardware matches so typically someone like Dell would have two accounts on this they would have a low permissions account where for their firmware engineers most OEMs actually don't actually produce the BIOS firmware themselves they rely on an ODM original device manufacturer so even if Dell wanted the firmware to be open the ODM wouldn't let them because they'll be underneath the person selling the hardware itself and so the ODM would log on to the system an upload firmware which then another more privileged user could log in to this system as a QA user check out the firmware check that the description's up to snuff check that the firmware applies properly that there's no user interaction required and that the firmware fixes the bugs that it says it does so with the LVFS the vendor uploads a file now one of the compromises with Microsoft we were originally working with Microsoft on a shared specification for the metadata required for this which was going really well when I was talking directly to Microsoft engineers and for various legal and competitive business reasons we kind of had to cut those talks short so we decided to ship it anyway we were launching the LVFS we've provided 100,000 downloaded metadata to unique clients and provided over 1,000 firmware files which I think is not bad considering it's only been running about four months so far we've had about six or seven vendors testing the system which was another major source of frustration loads of vendors initially piled on the LVFS system trying out the different things trying out how the distribution system worked but none of them had the guts to kind of go public they wanted to stay secret a lot of the vendors, ODMs and OEMs are very secretive companies and they don't want to release certain things because it could impact other things and it's all very secretive world and so Dell were the first one to come out and say actually yeah we've been testing this for six months we're confident in what it does we like the system we think it's useful and they released a press announcement a couple of weeks ago basically saying that all their UEFI hardware from now on their laptops and their tablets would now be supporting LVFS at ship date which is just an awesome thing so Dell announced that one day and the next two days there was a flurry of emails from the other vendors who were testing the system kind of prompted them into some sort of action so I can't announce anything yet but do you have to trust me that there are rather vendors that are like this close to doing press announcements the LVFS is kind of a political hot potato inside Retap because it's essentially hosting non-free blobs and so hence it's hosted external to Retap on an OpenShift instance one of the reasons I chose OpenShift was that the moment we're only handling I don't know maybe 2,000 downloads a day so it's not a huge quantity of downloads but in Fedora 23 we added this as kind of like a secret feature in Fedora 23 if you go on to the Updates panel in London Software and manually click refresh you get the distro metadata and you also get the firmware metadata and then for some people there might be a little item that pops up saying you've got a firmware update available now that was like a soft start to the system because we didn't want to enable the automatic mode and have just been besieged with 100,000 connections a day because we didn't know if the service would A be a success with the vendors B scale or C even be interesting to end users because without the vendors it was kind of useless to have all this data and CPU stuff going on so now Dell are on board and with other vendors are super close to making announcements for Fedora 24 we are enabling it as an automatic thing so rather than having to manually click the refresh button in the Updates panel it automatically happens on idle bandwidth when in sessions idle it will download the metadata if your hardware is lucky enough to be new enough to be supported you'll get the notification about firmware updates and be able to install them offline easily that's the cool bit so if you aren't admin you kind of care about the LVFS stuff the low level things and even if you are a paranoid hacker you might care about the low level things we added a feature where we can scan and interrogate firmware on your system that doesn't even support firmware updates which might seem a odd thing to do but if you've upset government agencies with three letters it might be that the firmware on your system is without you knowing about it so we support a verify option which will take a hash of all the firmware on your system which if you rerun it and the firmware has been subsequently changed it will then alert you if it's been changed which fingers crossed will be because you've updated the firmware manually yourself but in other respects might be less awesome most devices support offline updates either through system D or the pre-boot environment when you actually when you actually schedule a new EFI update it actually happens before you even boot Grub so that means inside the firmware package it's actually an EFI executable itself which kind of scares people so the idea from a vendor point of view is you take the cab file cab archive wasn't my choice it was like a compromise when we were working with Microsoft so the cabinet archive inside would contain like the INF file it would contain the cat catalog security file the binary file itself which would be an EFI executable and there'd also be the meta info XML which is the high level translated description for end users so you've got all this data inside the cab file once that file has been uploaded to the LVFS we'll break apart the cab file we'll verify it we'll sign it with the LVFS key we'll extract any metadata and provide the metadata to be downloaded to end clients the actual download of the metadata is done over SSL which is like a first layer of protection the metadata is then signed with a detached GPG signature also from the LVFS automatically vendors have to log in to the system using a password some vendors like the ones with the clue know what GPG is and they also have a GPG signature that they sign the firmware file with and then once it actually gets onto the system assuming the hardware matches something in the metadata we download the firmware which has an internal GPG signature which we then check against the LVFS one and then we deploy the firmware the EFI binary will only run if the EFI binary has been signed by the vendor that built the computer so it's like a triple layer of security because a lot of people are quite scared with the idea of updating something as critical as system firmware as easily in the desktop just clicking an upgrade button there's like three layers of security so when you actually get an update you can have a really high level really high level description maybe the description here is not the best in the world but you can have real descriptions that say this fixes the occasional hang on resume so it's designed as a high level user visible translated description so this is all done using the AppStream data that's provided in the MetaInfo file provided by the vendors which they can also edit on the website as well if their firmware engineers aren't doing an awesome job so this provides like a translated product name translated short description translated long description information about where the updates come from if it's been signed by someone the license it's under, the download size the installed size etc so because of the slight breakdown in comms with Microsoft over the collaboration I would be amazed if they didn't roll out exactly the same system for Microsoft update in the next year or so because all the information, the vendors are now providing this information in this file format which we've almost agreed on so I'd be very surprised if we didn't see exactly the same thing in Windows Update probably end of this year but you can see it's a very high level thing in the UI so the information about DFU devices that don't do the right thing a lot of device hardware I have an oscilloscope at home that requires you to hold down one button at the back and one button at the front whilst you turn it on to go into firmware mode which is kind of it kind of talks DFU when you do this but it's not really in the spirit of DFU you have to physically put it in this special mode so I'm going to say that's more secure I think it's because I misread the spec I neutered the colour hug device to take away the runtime DFU component and just showing that we can put localised descriptions of how to put a specific device into a mode in the AppStream data itself and we can also include a screenshot here as well so the screenshot will be something like just a diagram that shows the user what buttons to press if it's slightly unclear I didn't include it here because it's fairly obvious because it's a very high level thing and for DFU devices removable you can mostly update them online and so you can just do it straight in the software without having to reboot or or anything else now I've deliberately left loads of time for questions because this stuff scares a lot of people and a lot of people are quite worried about the security implications and the vendor issues around this so I'm kind of asking this if anyone can ask any difficult questions um really please go for it okay so yes so the question is can you roll back so UEFI allows the provision of roll back we don't allow it easily in the GUI but we do allow it in the command line there is a slight wring on the right minute sometimes when you update system firmware you might update the Intel ME engine which means you can't go back to the old version and so there's a field in the UEFI firmware which says the previous version has to be at least this so sometimes if we update between say 1.0 and 1.3 we can only update go back to 1.02 or something so we can't always go back to exactly what we had for DFU firmware yeah we can always downgrade there's no limit at all on going backwards but yeah it mostly depends on the provider other than that so in general case yes it's absolutely fine we just can't do it in the GUI go on so the downgrade will only work through the command line tools if the original version is available on the LVFS we can't there's especially with UEFI we're not allowed to access the binary blob from user space we can only do that in the pre boot environment so we wouldn't have any way of saving it for recovery so to speak most BIOSs do include split memory maps so you can have old version and new version which is a really good way of doing it because if that one fails you can always go back it will automatically go back to the first one not all BIOSs do that so I'd love to be able to say I'll recover it manually but it really has to be up to the flashing program itself because some of these vendors like Dell have done a really good job with their EFI binary so that it's a well structured update that's going to do safe things in same ways other vendors have taken their existing SMI based flash tool and just stuffed it in an EFI binary which is going to have exactly the same problems of timing violations and hardware access that the BIOS update mechanism did so I'm yeah no names mentioned but yeah any other questions so the question is does the website use two factor authentication no it doesn't I think it probably in my dear world should I think most vendors have a hard enough time with the password and username so most vendors are like well can I have a password like password so I don't have to remember it and I was like no two or three understand what GPG is have no idea about how to do key security so I don't know if that's a liability or a good thing two factor authentication might be opening the gates to hell I wouldn't be against adding it if any vendor wanted to do it like we've had one of the independent vendors was asking they actually asked for the GPG feature which is awesome I don't think they've got enough clue for two factor yeah I think you think of like these cool web companies and you think yeah they know exactly what they're doing they're on top of the security thing yeah but I wouldn't call a firmware engineer a normal user just from the update descriptions that you read you think these people like board line sort of psycho the crazy people any other questions people could maybe get it at once but what happens if a bad firmware is sent out and all of a sudden thousands of people systems are gripped is there anything that we could do so it's very scary that we're now being involved with this level so the question was basically now we've automated the whole system how do we stop it going wrong very badly very quickly so with the old system is if the vendor uploaded a bad firmware to the website very few people would find it very few people would be affected and they'd have a long time to fix it so with the LVFS we have like multiple user logins so I've kind of told the vendors they need to have one low privileged account for uploading for like firmware engineers and they need a QA account the deal is with the LVFS if they break everyone's systems because it's a massive amount of trust you're putting in a vendor just saying your old system that wasn't automated is now automated we can blit it to a million computers in one day but if you break it you're going to break it hard I'm hoping that the risk of their reputation would be enough to actually QA their updates or that they have some sort of like I said the dual bios system where if one was brick you would go back to the old one it is a concern and it is a point of responsibility one of the things kind of ties into this with the LVFS we're not sure whether to make it a red hat system or not should it be something that's housed inside red hat does that give it some sort of stamp of approval that the firmware has been checked by red hat when it really hasn't it's just the vendor that's been doing it so it's kind of a sticky do we want the red hat stamp on it or not and that's way above my pay grade to decide another question so yeah so we do so when you do a UEFI update you do actually get information from the the first boot of the new firmware whether it booted correctly or not so if it didn't boot correctly we'll actually show a notification on the first boot of probably the old firmware that the update failed there's not so much user kind of a description we can give though we normally give like an error code and a fairly cryptic error because that's all we get from the hardware is that kind of answering the question so yeah that's something that Dell actually wants they want to basically say if I send a firmware update out on the LVFS and it's failing on say 1% of computers can we get a failure rate as a graph on the LVFS my response to that was unless it's going to work 100% of the time with no user interaction it probably shouldn't be sent as an automatic update yeah so that's a valid point to the question would be basically if it was not working for the majority of people automatically pull the update and that's a good idea there's a slight privacy implication which is a slight worry where you'd be sending your system version to the LVFS which would be identifying systems that were insecure if you knew of a known security vulnerability so there's a slight privacy issue there but it's not unfixable we could send a hash or something I don't know if I can do a demo yeah so we save it to both the journal and there's an XML file on the system we see if this works never do an unplanned live demo so we have FWD MGR see if I make that bigger yep just trying how do you do that? control plus so you can see here I've got so I've not done any checks basically all the firmware I have plug in a bit hardware which does have a signature if I just plug in a colour hug device so if I now do if I now do FD update manager you can see that I've got a colour hug device I've just plugged in using DFU you can also see other stuff I've got like um not useful like a camera so the camera is a good example because I can't actually update the firmware itself on the camera this is the internal camera in the laptop but I can return the hardware itself and also the version of the firmware so I'm aware if it changes or the version itself so if I then verify something that I know has got a valid firmware ha so it's probably got a development firmware but yeah so the idea is that if you have a device with firmware that you've got from the RVFS it will display a valid check sum because it has the RVFS later already and if you haven't got say for like the camera you'd have to actually physically do something like this which would then store it locally and also record it in the journal which would then go through the hardware etc and just process all the ROMs create the check sums and then verify makes sense any other questions quickly go for it so so the question is really how do we identify that A. the vendor is who they say they are B. that they only update their own hardware and C. that they don't break other vendors hardware so with the when you look at the list of devices each one has a GUID and the idea of the GUID is that it uniquely identifies hardware so if it's if you have two ARM chips that require two different versions of the firmware because they're from memory timings or something they'd have different GUIDs vendors are only allowed to update things that have the GUID listed in the upstream metadata and so you kind of restrict vendors what they're allowed to update so that a Dell update wouldn't be allowed to update a ThinkPad firmware so that's kind of like the first way the harder social thing is when a new vendor comes to me and says hi I'm so-and-so from some random Chinese company can we upload firmware how do I know that A. they are who they say they are and sort of B. how do I know that it's legitimate from because it might be from a rogue in the company so even if the end of the email address matches the company that they say they want to do how do I know it's not a rogue employee wanting to backdoor or whatever long story short we don't really we kind of have to use the social stuff of ringing phone numbers talking to people on the phone talking to managers sometimes going through the legal because asking someone to get legal sign off using the LVFS is really escalating it in the company so you know that you're talking to the right people so there's no technical way really I think we can do it I think it's a social problem rather than a technical problem any last questions apparently we're all done if you want to grab me for questions afterwards well I'm Googleable thank you very much