 All right. Thank you for coming to hear our panel on open systems firmware. We're not kidding that we are passionate trailblazers in this area. Our passion comes because we are all concerned about the security of systems all the way from the bottom level device, the first time that the electrons hit a device in a system all the way up through the firmware, the hypervisors, the devices, the operating system. So in order to do that, we are here from... We represent competitive companies who are working together and we realize that the security of the systems only as strong as its weakest link and there are no systems today, none, in which the components are all built and controlled by one company. So for that reason we all work together and we are presenting today at least three different efforts that are trying to open up and make secure and more resilient the world of firmware. Our experience covers the whole stack. Collectively we've covered everything from smart cards to servers, the whole bit. So our full biographies are in the conference program if you want to learn more about us. But the format of our session today is to... I'll introduce the panel members. Then each speaker will describe briefly an effort that he's working in. We'll have plenty of time I think for questions and discussion which is why we have this as a panel. And if you all don't come up with questions, I will and they are going to worry because some of those are pretty hard. So I personally work at IBM's Thomas J. Watson Research Center in the Secure Cloud and Systems Group. Most recently I worked on our Power and System Z servers for a secure boot. The reason I organized this panel was that I'm hoping that out there in the audience or some of your friends and colleagues, we might hit other passionate individuals who are also worried about these things and you might want to join in our activities. So the speakers, I guess you'll have to raise your hands. The first speaker is going to be Ron Minnick. He's a software engineer from Platform Security in Google. Ron's the inventor of Linux BIOS. It's also known as Corboot. He's a member of the Technical Steering Committee for Linux Boot as well as co-leader of the Open Systems Firmware Project at the Open Compute Project. The next speaker will be Nate Klein. He's from Google. He's co-chair of the Open Compute Security Project. He's a hardware engineer on the platform's infrastructure team that designs and deploys all of the hardware that powers Google's internal and cloud servers. So Nate's going to tell us about OCP, the security project. And third is Brian Kelly from Microsoft. He's also the co-chair of the Open Compute Project on security. He is the principal firmware engineering manager from Microsoft's Azure cloud server infrastructure team. His team designs and develops firmware that enables hardware solutions in Microsoft's next generation cloud platforms. And Brian will be talking about EDK too. So all three of these are open system firmware projects and we'll go on now to Ron. Oh, you're right. I don't need this mic. No, I'm good. I'm always a little shocked when we do something like this and it works. So this is a project we started at Google in January 2017. It's a Linux boot. Linux is firmware. And we decided in March of 2017 that we really needed to open this up to external participants in part to convince the chip set and board vendors that it wasn't just Google because the first thing you ever get from a lot of these companies when you come in with new firmware ideas, you're the only people asking for this. So by May we brought in Horizon Computing with John Reefer done. They buy today Facebook servers and resell them with Linux boot in them already. Then we have also gotten Trammel Hudson. You may know him from Thunderstrike. He's doing some really awesome work in a system called Heads. Let's see. Are you here, Todd? Okay, so go see Todd Weaver and see the neat stuff that he's doing with Heads if you get a chance. And then of course we have Facebook and Nine Elements and other people at Google. And here in a nutshell is the problem. I kind of been saying this for 20 years. I'm surprised I'm still putting this slide up. But basically if you don't own the firmware, the firmware owns you. That's just the way it works today. Now back in the day when I began this kind of thing we wanted to own first instruction after power on reset. We kind of accepted that that's not going to happen in the x86 world ever again. We used to do it, we don't do it anymore. Actually accepting that, accepting that the box in the lower left is going to be kind of owned by vendors from here on out is okay. That's what turns on DRAM, nasty things like that. It's 10% of the image in firmware. And what we're saying is, well yeah, but we want to own that stuff off to the right. And by the way, this is my nice shirt with a nice Bios Hazard sticker. If you see one of them parked Chris Cock over there, he can give you the stickers if you want to put on your laptop. We all have it now. But basically if you look at that set of drivers there, there are several hundred drivers of various own known origin in a typical server or in a typical laptop today. And that number sounded ridiculous to me until Tram will demonstrate it. And what happens is you start, you go into this intrinsic box which is just a quick call out to this thing called a dispatcher. And the dispatcher runs in that function in every one of the drivers in there. And then it'll often run the drivers more than once because if any one driver depends on some other driver, it just runs the driver over and over again. No matter how many times that driver's been run, some other thing is discovered that depends on the driver. The drivers all get run again. Then we go into a thing called the boot manager which runs a bunch more stuff. And then eventually on a lot of your systems probably is going to run grub. But the boot manager also might do a fun little thing for you which is install a thing called the OS present app. The OS present app is an app that runs along with your OS and shares the machine with you. And a lot of people don't even know that. So all the great work that's been done on Linux and SD Linux and all these things over the years doesn't actually really matter because there's this thing called the OS present app running. So this is a slightly scary situation. I used to give this talk to various people from the Department of Homeland Security and things like that. And the first time I mentioned this in around 2000, the response was, yeah, but we're not going to talk about that anymore from the U.S. government. And then about 2007 it got to God, that's refreshing. Let's not talk about that. And nowadays I can kind of go in and say, yeah, but I can offer you a little bit of hope. And a little bit of hope is this picture. And this is what we are doing today. And that includes all the participants I mentioned on platforms we can talk about and platforms we're not ready to talk about yet. But the OS present app is gone. Many of the drivers are gone. And travel literally is reduced on one platform. 400 drivers down to 80. 80 is 80 too many, but it's a start. And we put Linux in Flash. We don't have grub anymore because Linux in Flash runs. And Linux in Flash does a K exec for whatever thing we're loading. Now that Linux in Flash includes an init ram FS. And I'll talk about that more in a second. But look at all the stuff that's going. You know, it's not all the stuff we want going. This is what we really want. There's a proprietary blob. And I'll mention again, you're not going to get anything but that on an x86. You're not going to get that anything but that on an arm. There's always going to be this thing that's native machine code that runs that you can't get rid of. I find that regrettable as actually someone has been writing these BIOS things for 40 years, but that's just the way life is. We will have Linux in Flash. And then the init ram FS is a different project called Uroot. And Uroot is everything written in Go. So if you think of just take the GNU bin and don't use the GNU bin but have a thing that is not equivalent and all the code is in Go. And our security people love that at Google. You know, if you give them Go code, they're willing to audit it. And if you give them C code, they generally are not. So 90% of what's in Flash is now Linux and an init ram FS. And then we start a system we boot and there's nothing left. There's no OS present app, right? Everything is gone when you're running. You own the platform. And that's not the situation as it is today. On most systems you run. Why do you need it? We want control at the BIOS level. We want things like security and performance and security and control and security and ease of use and security and security. I start to sound like the spam script from MoneyPython, but we're trying to get security. If you really just start googling UEFI security issues and CVEs, it's a never-ending list. I've learned because of Trammel Hudson again about some really terrific ones that are going to come out this fall that will kind of make your teeth hurt once you read them. And this is the scary one. Because UEFI is supposed to update itself, right, you need a new UEFI. All of you own Macs. Sometimes you do an update and it says make sure it's powered in. Don't touch it. It's going to go away for a while. It'll be okay. That's the part where UEFI is updating itself frequently. Well, what that kind of implies is that it would be possible to have an exploit that would embed itself in UEFI and break the UEFI updates itself part and make it look like UEFI updated itself successfully. Now you're done. And your only option at that point is the firmware. You can remove that exploit with a chipper. You don't have a useful machine left at the end, but at least the exploit's gone. So we've got these hundreds of binary blobs. Now what's the last thing that gets done on a new machine? The firmware. The machine's late. The firmware is done in a rush. Is the firmware really all that good? Generally not. And you've got hundreds of other things. But we're actually making it a lot easier. We have a Dell R630. You can get clone a tree if you're interested in coming talk to me about it. I don't have time for the how-to. And in three minutes you've got a BIOS image. Now that generally takes about an hour to do a build with UEFI today for that thing. So three minutes is not half bad. There are 15 machines now that are public and on the website. We can point you to. And there are more machines all the time. And then the machines we can't talk about. You can today buy off-the-shelf OCP systems. They're recycled systems from Facebook. They're refurb and being resold. They're actually a pretty good deal from IT Renew. And they come with Linux boot pre-installed. So what we really do, we take the ROM, we scrape away 90% of it and replace it with Linux. That's the short form. If I'm running, am I okay? I'm good on time. Okay, Lane asked me about legal framework. Obviously, you know, Linux is GPL. Uroot, for a lot of reasons, just kind of a go community convention like BSD, Coreboot, because I first started demonstrating on this in Coreboot in 2014, is GPL and some BSD, things like microcode, Power Firmware, because we've made this work on Power. We replaced Pettit Boot with Uroot stuff. So there that's Apache 2, I think is still Apache 2. And Uroot is GPL 2. We've also done this kind of approach on arms and things like that. But the general rule is it's basically probably going to be GPL if you ask that question. We really need help. We can always use help. We need help at the kernel level. We need help at the writing go code level. You can go to linuxboot.org. You can look at uroot.tk. We have a Slack channel, slack.uroot.com. OpenCube platform that, you know, that initiative is there. We're always very happy to have people come in, especially people who make systems and people who make chips. You know, we have a lot of users. We have Facebook and we have nine elements of openCube platform call. What we don't have enough of, in my view today, is the guys who, you know, companies that write design boards and design chipsets. We're working to bring them in, too. And if you want a fun sticker, come and get one after the talk or there's bioshazard.info. And, you know, I think computer people look at that and generally tend to read bioshazard. We didn't leave enough of a space. So a lot of people look at it, say, what's a bioshazard? Sorry about that. We're going to have version two of that sticker soon. Thanks. My name's Nate. I'm here to talk about the OCP Security Project that Brian and I are co-leads for. And firmware security, it turns out, is a thing. As Ron said, it's not great right now, but this will hopefully help. So currently, the state of things is a really sad balloon. The secure boot is, in general, is very fragmented at best. Every chip vendor has their own different solution. So if you want something that, you know, secure boots universally, you don't have it. And there are also these proprietary black boxes that Ron mentioned that you probably want to make sure that magical black box is the correct magical black box and isn't just some random thing that you're blindly trusting. And unfortunately, as was mentioned in the IoT presentation, the lowest common denominator is no security at all. And I usually like to say the S in IoT stands for security. So the goals of our project are to improve security across the entire computing industry through open standards. We want to make security based requirement for anything, not a differentiator. And that also reduces a lot of redundant effort. And, you know, building your own security snowflake is generally not going to go well. Security is significantly better and more secure when it's open. There are lots of eyes looking at it. And so what we want to produce are some specifications for both hardware and software security implementations. We want to work across all different kinds of IT equipment. OCP likes to use IT equipment for basically anything that runs code. And we want to use, like, existing and emerging standards as much as we can and not have to, you know, reinvent the wheel when we don't have to. So the focal points are basically taking every single piece of firmware or storage that's on any kind of board and securing it. So we want to be able to provision firmware as well. So that includes, you know, secure updates and rollbacks, recover from a bad state successfully so that you're never turned into a brick by going into a bad state. And then the ever so terrifying attestation. So making sure that, you know, you're running on the correct thing. So we also like to standardize interfaces. We want to standardize both hardware, like hardware electrical interfaces and software APIs. And then very importantly we want to support change of ownership. So used gear should also be secure. Something like, you know, a key burned into one-time fuses is great for the first person who owns something. It's kind of useless for the next person unless you want to hand off your private keys to someone else, which probably don't. So the scope of the project, sort of physical security counter measures, or sorry, what's out of scope is physical security counter measures. So, like, disabling JTAG interfaces or things like that would be in scope. Someone hitting your server with a screwdriver is not really going to be our problem. And we're also not going to play with thermite, unfortunately. Tried to sneak that one in. They wouldn't let me. We're also not looking at, like, coding practices or compiler time checks or things like that. Also not really focusing on penetration testing of hardware software. And we do not want to invent new encryption algorithms. We'd like to rely on, you know, existing or currently, like, in the works very well proven technology for that. So we are making progress. In here, there are links to a couple of docs that we've been working on as a group. We sort of started out as our group by looking at all the common security threats that we want to be able to tackle and trying to categorize them and figure out, like, how we're going to tackle them. And then we've started drafts of sort of two of our subsections out of about six, I think. So we have a secure boot section and an attestation section that are both definitely still works in progress. And you should join us. We are an incubation stage OCP project. And we'll hopefully be transitioning into a full OCP project. There's a mailing list. There are weekly meetings that are the most exciting thing ever. Yeah, they are very early in the morning if you're on the west coast, unfortunately. And yeah, that's all I have. Thank you, Nate. Hi, so my name is not Gondrala Devinder Gaut. And I didn't plagiarize these slides. These are his slides. He couldn't make it today, so I'm standing in for him. I wasn't supposed to talk on this topic. But Devinder, as he goes by, is a co-lead with Ron in the open source firmware group in open compute. Open compute, if you haven't heard about it, tries to do what's been done in software in that it tries to take hardware and make a hardware platform open just like any open source software project. Obviously there's differences with the types of collateral that are made open. But a part that has a lot of commonality would be the firmware, because it's all software underneath the hood. This slide talks a little bit about the mission for the group. And their goal is to make the firmware as open as possible and provide more choice, just like all open source software. It's about providing transparency and through transparency you're going to get security. It's also about providing choice. So Ron had already covered a lot of this in his talk about the companies that are contributing to the open source firmware development. There's Intel, Microsoft, Google, Facebook, Lenovo, Horizon, and others. The work streams inside this committee or project are open, EDK2, Linux Boot, Core Boot. Then there's the silicon providers also providing their pieces of the platform initialization. And that's the 10% Ron talked about as always been there just to do that platform initialization. So this part talks a little bit about one of the efforts inside that project that Divinder drives. And that is the open EDK Dixie Core work stream. The goals of the work stream of course are taking the EDK2 code base and making that open source. Supporting firmware security features like secure boot, measured boot and multiple OS boot. Supporting new hardware security modules like Serbress and others. Supporting out of band configuration making sure that the deployment of that firmware is easy to set up and also optimizing it for performance reliability, serviceability and deploying at scale. Of course there's an initial project that's been open sourced and the link is provided below. That takes us to the end. Oh, the end of the presentation. Yeah. So now we'll do Q&A. Who wants to be the first one to come to Q? It's very carefully designed to be impossible to find. Security by obscurity. There. Okay. Well, you've been working on this for a while. Why hasn't it done yet? We need more contributors like you to help us take it there. Great question. Hi, guys. I work on OpenBMC and OpenPowerFirmware at IBM. How do you see the work we do on things like OpenBMC interfacing with your work group and what it's going to look like when the rubber hits the road? Yeah, that's a great question. Let's get that one first. We're also collaborators in OpenBMC. So... Yeah. And actually have deployed instances productized of OpenBMC. And one of the only companies to do that to date is to deploy that at scale. So in OpenBMC and having the OpenEDK this communication that goes between the bootloader and the BMC for exchanging platform configuration data. Sometimes there's some error information that goes into the BMC about hardware errors. All of those interfaces and plumbing we've put in place inside in the OpenEDK. So... This... So you see yourselves as defining the APIs between the host firmware and the management firmware? There are APIs... There are APIs to find there already. I don't think we're defining all of those interactions that need to happen, but we have put some in place already. Okay. And it's based on IPMI. So there's like IPMI exchanges during boot information about the system platform into the BMC. Okay. So you want to create open standards for doing this between open firmware that's running on the host and open firmware that's running on the BMC. Is that the goal of your group? That's not the primary goal of the group but it's definitely one of the things that is discussed in there is what information should be exchanged and what is the payload format that should be exchanged. We'd encourage more people from OpenBMC and... Yeah, I was just wondering if I rock up for the meetings which as an Australian they're going to be at 2 in the morning or something. If I could rock up, what would my project better bring to the table and what would you better help out in? Well, with OpenBMC I know they've already got some IPMI framework as well as Redfish framework that they're putting in place. I think the communication between the bootloader and the BMC the bandwidth for that exchange isn't very large so those payloads are kind of more complementary to binary-based payloads as opposed to web-based payloads and XML. But architecturally that interface is still somewhat in its infancy and physically the interfaces too are a little bit different as you transition architectures and away from maybe an x86 architecture might have one interface to a BMC when you go to ARM and other platforms they tend to have different interfaces into the BMC so you can definitely contribute in some of the driver work there and enablement and APIs that are friendly across all platforms to make it more generic. So we welcome you to join. Let's throw in a slightly different angle on all that too. So IT Renew which used to be horizon computing is buying probably tens of thousands, let's just call it at the moment of Facebook nodes and they installed the Linux boot with the Go-based and it ran FS that I mentioned. They are actually going to be looking at the same model on the BMC and the reason is they have got the boot time on those nodes from 8 minutes spent in UAFI down to 20 seconds spent in Linux boot and what they initially then they immediately discovered that it has taken longer for the BMC to boot than it is for the main processor and so what they want to look at and they asked us to help them and the Go-based sort of and it ran FS boot so fast and we did actually put some design in there to make that boot fast on arm they would like to look at that same identical stack on the BMC that they run on the X86 because that cuts their maintenance tasks in half, right? They've got one stack not two and it's a considerably simpler stack with the Linux kernel and the Go-based and it ran FS. Based on that I'm not sure you know what I would be I would want to try and predict where things are going to go here but the change in boot time on the X86 side and I remember this a long time ago in the early 2000s there was a company I forget their name now anyway we put Linux BIOS on the X86 on the AMD and they had a power PC running hard hat Linux as their quote-unquote BMC and as soon as we had Linux booting in 3 seconds on the AMD we observed that 4 minutes and 57 seconds later the BMC came up and put some text on the LCD-based front panel and said hey man I'm ready to help boot your machine well by that point the machines would join the cluster they were running computing tasks by the time the BMC was ready to join the party so I've had this experience since really the 2000s of throwing away the existing firmware on the X86 putting in something that boots in seconds and then realizing that the maintenance system for the X86 is just barely hanging on keeping up so in my ideal world I was at a really great FOSTEM one year and I pointed out that the hard requirement for an automotive Linux based system to be up and ready and have turned on lights on the front panel is 800 milliseconds and in my heart of hearts what I really think should be the right time for a BMC to be ready and active and communicating and functional that's a good number and we're not even close we're about a factor of 40 off today because the number I'm seeing is 30 seconds so that's my slightly contrarian view of the BMC situation so attestation so one of the fun things about all of this about attestation it goes all the way up the stack and for it to be useful you've got to have attestation frameworks remote attestation serves all that sort of stuff do any of you see it as part of your brief to be interacting with those frameworks to be looking at the sort of big server remote attestation community blah etc what are your thoughts I didn't I didn't want to talk to that or did you want to talk to that this may be my mom it's not my mom can you hear me on this thing so on the attestation side that's one of the things that I'm working on in the OCP security project we first had to define our scope and try to limit what we did because it would take how many years to do this that's the first question right as far as how we're planning to do it for now is that it is a hierarchical plan so that devices within the system will attest to the server itself which may actually turn out to be the BMC in the server and then it will be up to the server to then pass that on or not to a data center so it will be hierarchical for now yes, from the device we are talking up from the device to the server but not beyond for now I just wanted to mention one thing because we the real expert on our plans is over here come and talk to him later if you're interested netboot is so incredibly broken in the x86 world that it's almost impossible to look at it and think of anything but just throwing it all away so that's what we're doing give you an interesting example everybody thinks pixie boot is slow and they're right but then they go from that to conclude that netbooting is slow and there they're wrong netbooting can be very very fast it can do it in a second or two so the plan where we are is you'll do we have a DH client written in Go of course we do a DH client W get and then somewhere linked in all that junk in ways I don't understand because these guys understand and I don't is our attestation framework but you know as long I worry a lot about netboot I'm a former high performance computing guy and that's how we did everything none of this has any use unless you throw away especially things like pixie boot so you really have to scrape the stack clean down to the silicon think it over again or you're just doomed in my view so between core boots sort of linux as an application operating system and open BMC are any of you concerned about lack of diversity in the stack from an attack perspective no concerns what you're using the same kernel for everything right so if there's I had an interesting talk gross used to be our VP of security Google I've known him for decades and I had an interesting talk with him about that the whole diversity thing right and he said to me he'd never actually seen a benefit and I've heard this discussion from security people a couple of times now if you talk about saying well you're depending on linux and firmware and you're depending on linux after you boot they've never seen a case that diversity would make that situation better make it a lot worse yeah we've seen it make it worse yes yeah it pays to fix that's the perfect friend just said it fix one bug not three so it helps you to focus a lot of your resources on the one thing as opposed to putting people and focused on many different things to try and catch exploits and bugs across different varieties of kernels if you can focus on that one and mature it then you're in a stronger place I don't know if you've noticed also from a an industry point of view the lack of security developers at all levels I mean you can see how many of us are in this room right and we found at least working in OCP that we have vendors who admit openly that they don't know security and they are relying on us to come up with the standards because they just don't know it and they don't have the staff nobody cares about security in the end here's the other comment so I had a friend once he worked in architecture office said how come the heating and cooling in every building and he says they always do that in the last week nobody cares about it they just throw it in the plants with all the junior staff who have never done it before that's why it's always broken so that's why my joke about look they build a motherboard and they ship the firmware out as the last afterthought so if you think about what do they even care less about than the fact of boots generally security is the thing everybody when schedule pressures hit that's the thing they throw away which is crazy but that's kind of what I hear over and over again so that's always the problem in the end nobody cares about security even though they say they care about security could I respond to the heterogeneity question heterogeneity is I think as Anain was saying is good if you have if you don't have a scarcity of resources if you have scarcity of resources I think trying to apply those resources is a small number of things and trying to get them as good as possible rather than having a large number of things which are all terrible is pretty bad I mean I agree but we do have scarcity I think that's probably the way to look at it fascinating discussion so this is sadly not unique to this industry but there's an accountability piece here so how do we implement accountability in a way that people will see the stake that they have I'm not quite sure I understood your question but I think having the open source and openness of the whole specifications will certainly help what we don't have in all cases are yet the infrastructure like you have with the Linux kernel and ways to do upstreaming and so forth there are I guess in some projects but not all that you can have all of the source code control and management it's just not there yet yeah just on that I think you do recognize that is a good point some of the projects are run out of Linux foundation like the open VMC others are like Tiano Core based and there's others then that are run inside OCP and there's slightly different governance models across those but I think the problem with trying to make it one way or one thing is it doesn't work for everybody there's never one solution that works for everyone so all of the firmware based projects although we support many of them from open compute there's no one governance model that we can mandate on those projects did that answer your question or no? have you talked about measured boot and attestation have you thought about how specifically with the TPM how are you going to map out PCRs for example in a PC client we in TCG I remember a trusted computing group created a PCR mapping systems have you really thought about how are you going to map for example as you take the measurements in the various PCRs standardizing that set of mapping across all of these verification engines can more easily figure out and do the verification which is why we did it in the first place have you thought that through at all? so there we go so obviously with the EDK2 that closely aligns with the with the TPM model with the Linux boot there's fewer boot stages so it's still the same you just need fewer PCRs firmly right we set up 8 because EFI is complex we think there needs to be fewer for this model and I was just asking the question how do you plan on standardizing that smaller set work in progress I'll give you a hand I'll help you with it I define the ones for the PC client yeah so I'll be happy to help with that I think we've run out of time now we're at the end of the session so thank you all very much for coming