 So I'm Josh triplet and this slightly delayed talk is a firmware transparency open all the way down So I want to start with a brief introduction So I'm Josh triplet. I have worked for Intel have for the last six years Five of those I worked on Chrome OS as Intel's Chrome OS software architect And then as of this year, I've moved on to working on open source firmware and that's what I'd like to talk to you about today So I want to give you a brief overview of what What our stack looks like today Simplistically we've got a user space stack sitting on Linux now whether you're doing an embedded system an IOT system or a full desktop system You've got a user space stack. You've got an operating system under that You've got some sort of BIOS and firmware under that and then you've got the the hardware and historically the point of open source has been roughly speaking here that we Put we have an open source operating system that we all know and love We have user space on top of that that can be as open as you want it to be and then historically the firmware has not Necessarily been quite as open. So that's something that's been improving of recent years we're starting to move that open source line down a little bit and We've been putting a lot of effort into various different areas here One of those is that there was a recent open source firmware conference just last month Intel among others had a strong presence there We were to taking a look at how we can open up more of the EFI Tiano firmware all sorts of bootloader items and similar But beyond that I want to introduce another wrinkle in the stack here, which is that we don't just have CPUs these days We have systems on chip. We have all sorts of different cores and dies All sorts of different cores on the processor die that serve many different functions many of them specialized and many of them separate from the main CPU So given that we want to introduce a new piece of the stack here Which is all of those pieces of hardware have firmware running on top of them This is separate from the existing BIOS firmware and stack This is instead Software that's often running a little event loop or a tiny embedded operating system that is in some ways underneath your BIOS or your firmware And so if we have that new complication in the stack Well, naturally we want to push open source even lower here and have open source much closer to the metal so naturally we think this is a really good idea for all sorts of reasons for Improving transparency for improving security for Ensuring that we have more control of the systems that we actually run So one example of this is that are on modern Intel systems There's actually an audio DSP on the processor die This is where all of the audio is routed through and instead of going just directly from the CPU to an audio card You actually have the capability to do things like wake on voice Process multiple microphones to do spatial audio so that you can do good echo Cancellation and remove background noise doing all sorts of output enhancement so that you can make a mid-range set of speakers sound like an amazing set of speakers and Based on that we we want to be able to drive this in a more open source way So earlier this year at embedded Linux conference not in Europe We introduced the sound open firmware and this is an open source firmware for the Intel audio DSP It's got all of the functions that I mentioned and has the ability for people to introduce their own code their own processing algorithms for Things like how do you recognize the speech for whatever agent of assistance that you're using or how you do echo processing and cancellation and And This is a more general pattern where we would like to have more open source available for Soc component firmware the audio DSP is the first in a pattern here and Based on that I want to talk a little bit about what the best practices for this are Because this is not an area that has historically been particularly open on the contrary It's something that has historically been extremely proprietary and so in the course of opening it up We'd like to talk a little bit more about what we think the best practices best Policies and the ways to make this a truly open project are So first of all that you would hope this would go without saying but part of the point is to make things open and auditable So we want to have full source code available not just for some components, but for everything running on that particular Soc chip so that means everything all the way from the initial Rom or initial boot block all the way up through any bootloader through any operating system that's running on it There are many different embedded operating systems people might use you'd want that to be open as well as well as any processing code now naturally Any vendor beyond that might choose to add their own code to it that may or may not end up being open But at least the baseline starting point should be something entirely open and something people can audit and verify for themselves Now beyond just being open source by license and practice We also it also ought to go without saying that it should be developed in the open So this means that this shouldn't just be a code drop air dropped from orbit into a repository saying snapshot as of this date It should actually be developed open source first upstream first Commits to a public repository now naturally there might be pre-release hardware That's not available yet, and then that might go in as a batch when that hardware is released or close to being released But for the most part features and requests and active development should be happening directly in the repository Now beyond that one that anybody who's done embedded development may be familiar with as a potential issue is There should be a unified code base for all platforms We shouldn't have a copy of the code for this chip and a copy for this chip and a copy for this chip That are all divergent and all different branches and so on It's important that one line of development one mainline branch cover all of those different platforms Avoid having regressions on the old platforms introduce new features for the new platforms You might choose to snapshot a stable release for the particular shipment of your hardware But that doesn't mean you should be that you should be locked to that release and unable to upgrade to a new version Because hey the new version supports the new hardware, but not the old so it's important to keep that code base unified Now here's one that has been a historical problem open source firmware needs open source tooling if you're building firmware and You can trust the open source code that's running on it But in order to build it you need a proprietary toolchain for a processor or a proprietary signing tool or some magic proprietary flash configuration utility that Doesn't give you the greatest of confidence that what you have code for is what's actually running on your system So you need full tooling available for whatever Target you're using now that also includes especially compilers if you're going to have a Cross compiler targeting whatever embedded plot processor. You're running then that should be an open source Compiler toolchain and between GCC and LLVM and Clang We've got more than enough tools to successfully do that But relying on a vendor proprietary compiler as the only option available or even the primary option for the binaries People typically ship again doesn't give you the greatest of confidence in whether what you're running is what you have code for Now beyond that you want to be able to do development for these platforms So based on that development You're going to want to test what you have before you submit patches at least hopefully So it's good to have some kind of emulation layer so that not everything has to be tested directly on the hardware with whatever debugging facilities it might have available So as an example of this the sound open firmware project I mentioned earlier has a great work some great work based on QEMU So with QEMU they have the ability to run two separate copies of a virtual machine One emulating x86 for the main system that you're booting Linux on the other emulating the the DSP processor and Running the audio firmware and it connects the two together the same way that it looks on the real hardware So you can then run the standard Linux audio driver that drives this DSP It will communicate with the other copy of QEMU and the two will Produce a result that tests that firmware as though you are running it on real hardware This is a a best practice that we'd recommend for any kind of SOC component or other device that interacts with a host system Emulating the two in parallel so that you can do this kind of test Now beyond that this kind of firmware sometimes serves a fixed function But oftentimes it's actually Dependent on what you're trying to do with your target system So it's important that this firmware be reasonably light and configurable You want something that on the one hand has a baseline that is as small and minimal and boot up and get out of the way As pop as you possibly can and on the other hand You may want many additional functions that you can add on top of that But that you're not required to you don't want something huge and heavy weight that you must use You want configurable functions for example using kconfig to configure your firmware and having a config for this particular application or usage Now beyond that and this is especially important these days You have to have security available if you're running on this embedded processor some embedded processors still have the capability of doing privilege separation of doing memory protection and These are features that you shouldn't ignore you should actually use them You should have a separation between an operating system kernel running on these devices that has direct access to the various hardware and the equivalent of user space that Moves data around but doesn't necessarily have the ability to touch everything if you're parsing some unusual format Don't do that in the same privilege domain as touching all of the sensitive hardware So as one example of a way to do this and not by any means the only but certainly a path that we'd recommend Intel does a lot of work on the Zephyr embedded OS if you have something that's too small to run Linux Which there are still such hardware out there. I'd love to get Linux running in 32k, but not so much yet but in the meantime Zephyr runs in a Space that small and is quite capable of doing privilege separation of doing security and hardening features of Locking things down so that different components of your firmware don't necessarily remain all powerful Now I mentioned earlier that I want I'd like this to be light and configurable and especially that not every function is needed on Every system so that means that the structure of our Linux drivers needs to change You're no longer driving a piece of fixed functional hardware that does one thing and has a variety of specific functions in it You're driving something that on the other end is fundamentally a piece of open-source software that can do anything that you can dream up So given that we need driver Architectures similar to the way the modern audio drivers work where you're communicating with a configurable device on the other end You have audio routing you have Additional capabilities you have a messaging layer all of these are things that allow you to Be extremely flexible with what applications you might have But that also means your drivers can't just be well I send the audio here and it comes out the speakers It has to be well here are the configurable places. I might route different kinds of audio Here's the distinction for what if I've learned to distinguish three different People speaking based on where they're positioned then I might feedback spatial audio data that says well I've identified a person here a person here and a person here with three different voices speaking into an array of four microphones That's not something the default audio drivers have but the current frameworks make that more reasonable to say Hey, let me give you a few dynamic streams of audio with some tags attached So that's the kind of infrastructure that as we're building drivers going forward We need to think about if we have open source firmware on the other end that can do so many things We can't build a rigid inflexible driver architecture Now I mentioned open and verifiable earlier, but in particular we want this to be independently verifiable and auditable So who here has heard of the reproducible builds efforts? Awesome about half the room. That's excellent So this is an amazing project that was started originally by some folks out of the Debian project and expanded outward to cover so many other parts of the industry and The large goal here is if you take the same source code and build it in the same environment You should get the same binary now that sounds really obvious in hindsight But at the same time that requires a great deal of work to make sure that you eliminate sources of non-reproducibility Everything from what order you read object files to putting time stamps into your binaries to all sorts of things that make the binary not match And if you can ensure that the same source code becomes the same binary whenever you build it You get much more confidence that if you have say a signed image from your vendor that it matches the source code You've been handed nothing up our sleeves. No magic extra pieces of your binary You just have what you built So that doesn't just mean having all of the source code available that also means nailing down Well, what is your tool chain exactly? Because you can't guarantee you'll get the same result if you build with the wildly divergent tool chain or divergent libraries are similar so again as One possible practice not necessarily the only one but something that works conveniently Intel has a Linux distribution called clear Linux That is designed to do atomic upgrades and atomic snapshots not so much install this package remove this package as Here is a snapshot of this variant of clear for this particular release number And you'll have an a binary identical result for any such system So that means that you can simply publish the release number That you built your firmware in and say well, we used version 57902 to build this firmware and then if you take that release number We recently added support in make OSI which is the tool to build a change route based on a particular Linux distribution To install a snapshot of clear without necessarily updating it Then you can change route into it build your firmware and you'll have exactly the same result every time So obviously you could do this with any number of other Linux distributions or with bundled up change routes But this gives you some confidence that you can trace back This is exactly What we built from what the history of this is and of course every clear release number has a history of this was built with The previous version was built with the previous version and so on so you can again get more confidence in what you're actually running So that's a sample of what types of things you may wish to do to have Your users have confidence in the firmware you're providing These are the practices that we intend to follow as we release more SOC firmware components where sound open firmware came from and in general Fully wearing my Intel hat on and actually speaking for Intel for a change This is something we believe very strongly in this is something where we feel that open source should be going all the Way down to the bare metal should be running directly on these various SOC components And this shouldn't be a place where proprietary software still hides. Let's make this more open So based on that I do want to thank a couple of people in particular This is the other members of my team at Intel working on open source firmware mark Herbert and Joe Kono There's also a broader team that's working on sound open firmware on several other components And hey if you're working on embedded development and interested in working in Embedded hardware that's distributed in a wide variety of x86 Intel systems, then please let me know we're hiring With that I'd like to open things up to any questions Thank you. Yes, please come on come on up to one of the mics so that it will record the question and everybody else can hear it and Please feel free to line up at the mics if you have questions So when do you expect to to ship? Open source my Intel microcode updates I mean following a specter and it was a bit frustrating to install None non-free packages understood completely So the question was when do we expect to ship open source Intel microcode updates firmly taking my Intel hat off for the moment that I would really love to see that happen myself Putting the Intel hat back on for the moment. Please watch for low-flying porcine aviation Okay, that probably will not be our first target I would love to see that happen, but there's a variety of reasons why that in particular is not wildly likely That said it's also not the type of firmware that as you would normally expect it This is that's not something where you boot and run a loop that does a variety of things That's more simply a translation layer from instructions to lower level instructions It's still obviously something transparency would be nice in but not something that I can speak for that team in any plans that they might have Next question Well, it's a sort of related question What about the memory reference code which is part of the Intel firmware support package? Are there any plans to open that up? In a reasonable time frame Sure, so What we what plans we might have to open up the memory reference code in the Intel firmware support package So a little bit of background here for anybody who doesn't know the Intel FSP is a Piece of binary that is handed out to boot up and drive specific parts of Intel hardware It's primary function these days is the Intel memory reference code Because any of the other functions that it has are also available in open source in places like Corbuth But Corbuth on modern platforms as well as things like the slim bootloader and even portions variations on Tiano and EFI use the FSP So unfortunately the FSP is currently binary and has binary memory reference code that does the initialization of DRAM That's something that we have opened up for specific platforms for example the reference for the Quark some of the quark platforms and some of the smaller atom platforms We do have available memory reference code in full source form under an appropriate open source license That's something I'd like to see for current hardware and the two things I'd say on that One is that that's not entirely under our control because there are third-party Third-party items involved there for the various DRAM vendors and similar So that's something where we would need to Collaborate and work on that with a few other Engineers across the industry that said I'd very much like to see that and that's one where you could make it significantly more likely if you Speaking for any number of OEMs explicitly asked and said we would like this to be open here Is what products we'd like to ship that would be better if that were open in which case that would make it much easier to Look at doing that in the future. Thank you Alright, thank you very much