 Hi everyone, my name is Grant Likely. I am a senior technical director at ARM and also an architect in the system-ready program. And today I want to talk about a program that I've been working on for quite a while or on standards for embedded Linux and just don't know what we can do to make things more reliable and just greater scale, be able to access more devices. And with that, why don't we just jump right in? So, first thing is, what exactly am I trying to solve? There is, I mean, typically embedded Linux is vertically integrated. We've got the firmware, the operating system and applications all bound up together in a single build. Every platform has slightly different behavior. We deal with this with configurations, bring a new piece of hardware in, we change the configuration, we've got a new image that comes out the back, do it again with a different application, we come up with a new build, new configuration, all those things to go together. So with every change that goes on, it means turn the crank and a new image. If we're careful and we do some, use some of the tools that are available, we can get multi-platform images and support multiple pieces of hardware, but it still is very much enabled per platform enablement to get everything working together. And this has scale problems. I mean, let me be realistic here. The embedded Linux ecosystem, we're very healthy, we're very good at getting products and putting everything together. But it doesn't scale well in that we're stuck in that model of having to turn the crank and change the configuration and then manage all the images that come out of there. Where that doesn't scale is, first of all, with the distros, like with the Linux distributions, it works if you're building thing custom or if you're modifying the OS itself. But the distros who have products to support the IoT or the embedded space, they can't handle that, right? That's just not on. It's in order to change any part of the OS image in order to support the hardware, it's not something that they can do. The way that they operate, it has to be generic images. The platform owner also then becomes responsible for maintaining everything. It becomes difficult to use a third party or OS vendor to keep track of, say, security updates on the kernel or applications if everything's monolithic and they have to touch everything from firmware up to the apps whenever there's a change. And as I said, the distros cannot handle per platform customization. Now, I am being a little bit over the top here because yes, of course, in embedded Linux, we've got the ability to break things down and we do do that. But it's more haphazard and it's more of an exception rather than a rule. So we want to kind of fix the scale problem so that we can start to break up the software ecosystem and have the platform responsible for platform stuff, the OS responsible for OS stuff, and then the applications on top responsible for it to enable more mix and matching, more enabling or bringing new hardware in to existing problems. It also comes up with ecosystem problems where with the vertically integrated stack, there's extra engineering that's required just to bring the platform up. This is a real problem for the ODM OEM market where the device manufacturers, the ODMs, they produce products, they produce a BSP that allows you to bring up Linux and then they sell that onto the OEM market that puts software on it and actually turns it into a product. The problem with this is the OEMs know going into it that they're going to have to do some extra engineering work to bring the preferred OS that they are using or their preferred solution onto the product. And so there's, with the vertically integrated mindset, it's difficult for the kind of ODM OEM market that we have seen other parts of computing to work. It also has the ecosystem problem of in embedded Linux, we're using different tools that are used in server or desktop and other parts of Linux. And so this just means it's to transition from one to the other has costs associated with it. So the whole idea with standards is it's designed to reduce the engineering effort required while at the same time increasing the flexibility. With in the general purpose compute space, we see exactly that where the platform vendor produces everything that's needed to be able to boot and support the OSes that then run on it and the OSes don't need to deal with the specifics. And embedded, we do have all the flexibility of we've got customizable products but there's a lot of pieces of that that still can behave in a standard way. And so we're looking for ways, I've been looking at where are the standards that can be brought in so that we can have less engineering effort required just to enable the boards. And I'm gonna be talking a little bit through this talk of ARM's System Ready program. System Ready is a program that ARM announced about a year ago in October, 2020 that was designed to bring a series of standards for different parts of the ARM ecosystem. And when we were looking at this and trying to decide what standards would actually be relevant, we realized that whether we're talking about server or edge class devices that could be running many different OSes to embedded edge and IoT, and then we were going to need a different set of standards. And so we created this program to have different bands. Specifically, I've been looking at the IR band, which is for the IoT and embedded. And when we're designing these and choosing what standards to do, we need to have a balance between having lots of flexibility, which if we're very custom, we can have, there's this high software maintenance cost that can be associated with that. And then the fix and then standardization where by reducing the flexibility, we're reducing the engineering effort. There's a middle point in that of if we can balance the customization required, as well as the standards, we get a nice middle of place where we're enabling innovation. And that's kind of the goal of the program. So I'll be referencing the system ready program somewhat through this talk. Mostly this is about the technology. The system ready program is relevant though, because it shows how it can actually be implemented in practice in a way that's going to be useful today with your products. So there's two parts that I want to talk about here. One side is what's needed out of standardization and the distros. But before I talk about that, I need to talk about the embedded Linux ecosystem. And this comes to how we choose the standards that we do. The first thing that is important to realize is that major changes to embedded Linux just are not realistic. If we're bringing in standards, the embedded Linux market is mature. We have tools, we have projects that all work very well. And to if standards mean taking those technologies and throwing them out and then coming in with a whole new set of tools, that's just not realistic. It is not realistic to throw a U-boot and replace it with something like EDK2 because that would change all of our processes, it would change the familiarity. So it's important, anything we do must work with the established technologies. And that's U-boot, Trusted Firmware A, Opti in the ARM space, the MetaDistros, Yachto and OpenWRT, these are hugely important, anything we do must work with those projects. Device Tree is the standard language and embedded on how to describe platforms. ACPI, it certainly can describe our platforms on our servers, that is the system description language that must be used, but in embedded, ACPI isn't there. It doesn't have the capability and the flexibility that Device Tree does. So this needs to work on Device Tree systems. Also, re-architecting from the ground up is just not an option. So to bring standards in, it's very important that the standards have a migration path that allow us to add them in to existing products or existing tool chains without being disruptive, without have forcing changes on all the components of the system at once and to minimize engineering effort. In my mind, this becomes really important of anything that's talk to the standards also absolutely has to be implemented in the mainline projects, otherwise it really doesn't exist. So as I said, to be relevant, these standards must work with existing tools and techniques. Switching gears, though, we need to talk about what the distros need. I'm gonna talk about the Linux distros. A lot of the stuff here, though, that is useful or that's important for the Linux distros really does apply to some of the meta distros like Yocto as well. It's more extreme when you're talking about a third-party binary distro because the scale that they operate, the way that they operate really forces the kinds of decisions that get made. So, first of all, embedded Linux, more and more embedded Linux doesn't mean Custom OS. It isn't a requirement to do Yocto to build a Custom OS. There's many products now and more that are wanting to either use, they're either using an existing distro root file system or even better, taking the whole file, the whole operating system itself such as Fedora or OpenSUSE, using that as the base software platform on the box and then loading the applications on top. Reasons for this is the distros, they already have all of the processes in place for maintaining a large body of complex software, keeping up with security updates, having a deployment mechanism for package updates. All of that stuff gets baked in and it means that you can amortize the support of all of that software over a large number of products. But in order for them to support these things, custom-bored enablement, if you're using the generic distro kernel, custom-bored enablement is just, it's not on. If the generic image for them, it must boot. Platform drivers, anything that you need to add to the kernel, it needs to be able to be enabled generically. If it has, if it conflicts with something else, that's an absolute no-go. And ideally, they want to be using the same technologies as the not-embedded platforms. They don't want to have a different boot flow on embedded just because it's embedded. As much as possible where they can use the same technologies, it means that there's less cost and there's less maintenance, less complexity over the entire system. The distros also have a mainline first policy. So for any platform that they support, the drivers and silicon and board support, it really better be in mainline Linux. If it's not there, they're not gonna be able to support it. They're not gonna merge in out-of-tree patches just to support another embedded platform. They've got actually quite a number of platforms already that they can focus on rather than worrying about the out-of-tree devices. So the kinds of standards that the distros need, they need boot order and control. When they install or deploy an image, they need to be able to control how that boots. They need a pre-boot execution environment. And so this is to deal with things like shim and grub or system deboot to deal with loading additional keys, handling secure boot, having a boot menu or a boot selector that the operating system can control at runtime. These kinds of things. And they need that pre-boot execution environment to be able to access storage, to be able to access the networks, to be able to access the console. They need a platform description. They need the platform to be able to tell the Linux kernel what exactly it is. And that needs to be part of the platform, not part of the kernel itself. They need a way to maintain the platform. It's not something that they'll be able to do to at scale, implement a different firmware update mechanism for every single platform. So they need a standard one there. And they need verified boot, right? If we're building secure connected always on devices, those devices, the firmware better be updatable. Otherwise they're not gonna stay secure for very long. But that has to be a standard way of doing that without needing to know all kinds of details on the platform. So what has been landed on in terms of what we can do here is this is in very rough form, the architecture that shows off for standards in embedded Linux. And there's three things that I wanna point out on the diagram. The first one is UEFI. Now UEFI is, I'll talk a little bit more about that later, but that's the interface that provides services to the operating system, both pre-boot and when the Linux is running. And U-boot, or UEFI, is actually now implemented in U-boot. I'll come back to that as well later. The second one is the device tree. And you'll notice that the device tree blob is actually sitting down in U-boot rather than sitting up in Linux. And what this represents is by making the device tree owned by U-boot and provided to the OS by default, it then creates that platform interface so that the OS image doesn't need to know ahead of time what the hardware actually is. And then the third one is firmware update. And these are kind of the three key interfaces that if you put them into the platform, the OS now has the separation that it doesn't need to know the details before it boots on the system. Right, so going back to this diagram for a second, keep that in mind and I'm gonna talk through each one of these items one by one. The first one that I wanna talk about is UEFI. It's really worth noting right now that UEFI, it's a specification. It's not an implementation. Very often when I hear UEFI said, someone's usually talking about the implementation of firmware on a PC. But that's not what UEFI actually defines. UEFI itself is a specification that has a lot of functionality that is described in it. There are multiple implementations of this specification. Tiana Core EDK2, which is what the BIOS on most PCs is based on, that's the reference open source implementation, you can find that on GitHub. It's actually a really great project and it's worth going and looking at. But it's not the only one that's relevant. And in fact, for embedded Linux, the UBoot implementation of UEFI is far more interesting. And it implements already with a small number of config options, everything that's needed to implement what the distros use out of UEFI, right? So just my call out to you, when you're talking about UEFI, if you're talking about the specification, please say UEFI. If you're talking about the firmware project, EDK2 is probably more what you're talking about and that will help to reduce confusion. Second thing about UEFI is, there's really very little in it that is actually required. UEFI, it is a big specification, but it specifies a lot of different things, such as how you handle add-in boards and option ROMs. But if you look at what the operating system is used, if you look at what the Linux distros use, it's really quite nicely contained. And we have a project called EBBR, which can also be found on GitHub. And EBBR defines a subset of the UEFI spec that is suitable for embedded platforms, suitable for embedded Linux. And it includes the things that are required to support the distros. It's all stuff that's already implemented in mainline U-boot and just needs to be turned on. It works with the characteristics of embedded hardware that's important to pay attention to. And it works with device tree, right? So this is UEFI implemented by U-boot and then providing device tree as the system description. A number of the things that has always been controversial with UEFI, and in particular, the whether or not the UEFI is still resonant and able to run once we hit, once the operating system is running, are optional and actually quite contained. The UEFI spec itself does keep the stuff that's available at runtime quite contained anyway. And with embedded, the things that are difficult to implement such as set variable, set time and update capsule, all of those are optional. You don't have to implement them at runtime. And the reason that we don't have these implemented at runtime is in most of these platforms, there's only one flash device. And that flash device is owned by the operating system. So it's very difficult to be able to allow firmware to remain resident. And access storage, access the storage because it would then need to coordinate with the kernel. And that's just really messy. So what we do instead is we make those optional. We make, if any manipulation needs to be done using those features to do it at boot time before the OS boots. And then Linux is actually able to protect itself on the bits of runtime services that are available if it wants to call them at all. None of this is even required. So here's what UEFI defines. And first of all, it defines the ABI. If you've looked at UEFI, this will be familiar to you but there's two execution contexts with UEFI. And that is before execute services and after execute services. The ABI that UEFI defines contains two tables, the boot services table and the runtime services table. And all those are function pointers to well-defined behavior. You can go and see this entire table in the UEFI spec. In the boot services table, we see events and notifications. We see memory management like Alec. We see image loading. We see watchdog timer. There's a few other services. And there's a function pointer to exit boot services. In the runtime services table, there are the pointers to the variable manipulation functions, timers, wakeups and RTC access, system reset and firmware update. These are just, again, these are just tables of function callback pointers. Before exit boot services is called, both tables can be used. When exit boot services is called, firmware is then told to go and shut down anything that happens to have running. Update the runtime services table if it wants to to change the functions. And in fact, UBoot provides an entirely different table. And then pass control back over to the operating system so that it can start loading its kernel, loading its device drivers. And then it's able to access the runtime services table if need be. Before exit boot services, there's also a set of protocols, which is a way that the ABI can be extended with well-defined ABI for network, console, storage, graphics. This can be extended as need be. But this gives, before exit boot services, the ability to add in features to firmware, and then after boot services, the limited stuff, limited functions that the OS will need to use. Finally, there's also the system table. And the system table, again, is just a pointer to a set of data structures that the OS is then able to use. And the kinds of things that are in the system table is there's pointers to data about the system. There's the system descriptions, such as device tree, SM BIOS, ACPI, all three of these would be passed through the system table. And then details with UEFI, such as the ESRT, which is used in firmware update. And the runtime services supported table, which tells the OS in the EBBR context which runtime services are supported and which ones are not enabled on the platform. So it's just a nice discreet place that you can pass all these information to the OS. In a way, that's also extensible. So we're not stuck with fixed structures. UEFI also defines the behavior. And these were things like boot device selection. How do you choose between the devices? If you don't have any defined boot devices, then it has a scheme for how it looks on a file system to find a system partition and to find the removable boot path on that device. And if it finds it, then go and run it. It defines the boot order and boot number variables, which is the boot numbers are, there's one variable for each possible boot target. And then the boot order variable just tells you which one to boot next. And then it also defines the handoff to the OS. So with the behavior that UEFI defines, the platform's gonna behave predictably. It means that an OS, when it's deploying to the platform, it can control what's going to be booted next without having to worry about scripts or variables other than the very well-defined ones that UEFI uses. Back to the runtime services table. It's important to stress that UEFI defines limited functions at runtime. You can't do anything at runtime. UEFI cannot take over the system at runtime. In fact, the Linux kernel constrains the memory map that the UEFI services actually get to run in. So it can only access memory and data structures that UEFI has asked for. It can also ask for access to devices and for firmware to retain control of the devices, but embedded, we generally don't do that. We let firmware, firmware gives up control, all the control to the OS and then the OS almost. And so the key things that are in here, the most important ones are, as I said, RTC, variable control and update capsule for firmware update. Next part that's important is UEFI secure boot. And UEFI secure boot, not possibly the best name because it's more of a verified boot scheme. But once again, it is a well-defined process for delegating authority where the platform has a set of keys or key databases which are hierarchical so that you don't need to rely on a single owner or a single key to control for the signing of everything. And the scheme that's used in UEFI is you've got the root of trust for the UEFI portion, which is the platform key. You've got the key exchange key database, which would be the signing authorities that the platform key would, any key exchange key signed by the platform key is trusted by the platform. And then you've got DB and DBX. DB is the set of keys of images that are actually allowed to be booted. And the DBX database is the set of images or hashes that have been revoked. And so only stuff in DB and DBX that's been signed by the KEK is actually allowed to boot. So this allows the delegate authority element. Say you can, if you are the platform owner, you can say, you know what? I trust Red Hat and I will include the Red Hat key exchange key in my platform and that will allow me to boot Fedora or Red, or, and then finally, firmware update. The last thing, another thing that's implemented in the UEFI spec is something called Update Capsule. And it's a very simple interface, but it's quite powerful because it provides a way to pass a defined data structure to firmware that is intended to contain new images of firmware. And then also the ESRT table, which contains a list of all the updatable firmware components in the system. UBoot implements Update Capsule and UEFI. And it's in UBoot, the implementation is backed by the device firmware update system. And so if you've already got device firmware updates on your system in your configuration, it's quite easy to enable Update Capsule. The capsule image format for UBoot is really just a wrapper around the UBoot's fit images. But by having that wrapper around it, it means that tools like FWFD and LBFS can manage those firmware updates. And so we can use third-party services for handling deployment and management of firmware images to the device. The tools to make these, it's all part of mainline UBoot as well, the Make Image tool to make the fit image and then make EFI capsule will take that fit image and wrap it with the capsule wrapper. In terms of actually applying these, there is the standard UEFI application capsule app. You can self-build it yourself with the SCT and I've got a link to how to do that later in this presentation. Or you can use ARM System Ready Architectural Compliance Suite image that already has it installed. And once you boot that on the platform, you'll be able to go and run Capslap to install the firmware update. So that's particularly useful for testing and ensuring it all works. All right, moving away from UEFI, now let's talk a little bit about device tree. In the traditional way that we do embedded Linux device tree has been tied to the Linux kernel. And this, as I said, this one doesn't scale. If the device tree is tied to the kernel, that means every time we bring a new hardware or we want to bring a new OS to the device, we have to do that enablement activity of adding the system description to the kernel image. The reasons we've done that are well known. Device tree has been, the stability of it has been, depending on which platform you are, there's, it can break for between each kernel version or it's been on some platforms it's been quite stable. But we need to make this, start making this change. And so by default, having the firmware provide a device tree that's suitable to boot the kernel means that we enable a whole bunch more, whole bunch of different operating system images. And as I want to be clear, this isn't great for Yachto either with if Yachto image has to carry around the device tree is in the same predicament of to bring a new hardware you're going to need to deploy a new configuration into that Yachto image. If there's a lot of hardware that you need to support, that means there's a lot of additional data files that need to be carried around with the OS image. So the idea here is if we take that device tree, we put it into firmware and we have it provided to the OS by default, then there is, we've got that ability to boot the OSes without actually restricting how you design systems. So it enables the distro use case, it enables the just works of boot the boot into the kernel, but if in your embedded development, you still need to carry the device tree with the OS or you need to update the device tree, you can do that. There is all of the interfaces provided here allow for if the OS to say, hey, I don't like the default device tree that you're providing, I would like you to instead use this device tree and everything will still work as is. So what this really does by putting that in is it sets a baseline for compatibility. It gives us the a platform that allows us to start testing and really provides some feedback on when device tree changes will break, will break against older versions of the firmware. Right, so that starts with the back pressure of stabilize in the device tree without actually impacting product development. Finally, let's talk about mainline Linux. We've got, in embedded Linux, it's not nearly as bad as it has been years ago. We don't have the embedded Linux problem to the way that we did say 10 years ago. However, it still is an issue of different silicon vendors have or different pieces of silicon have different support in mainline. The Linux distros have a very strict policy. I mean, it is mainline first. If the board doesn't support a mainline, the distros will not support it. And so I think it's time for embedded Linux that we actually take this seriously of putting some pressure on or trying to make some movement to getting all of the silicon into mainline. And I mean, this helps with Yachto too. If we are, by default, if the standards that we're relying on include have support in mainline, that means there's less reliance on customer vendor trees. There's, it means that upgrading the kernel or applying security patches is not so big a deal. And so it just maintain requires, already solves a lot of support problems. Now, I'm pretty sure that all of you, most of you listening to this are already on board with this. So I'm not saying anything new. I'm just looking for ways up for when we bring in standards and we talk about standards, mainline Linux has to be one of the things that we talk about. Now, how do we actually test this? If we are using, right, it's one thing to talk about all of the interfaces, but unless we're able to test them, we've got no idea on how well platforms are actually implementing it. And there's a number, there's quite a few options out there right now that you can go get and verify to look into and to check how compliant or how much of this stuff I've been talking about, your platform actually implements. One of them is the UFI self-certification test. And this is the tool that's been used for years by the UFI community as part of their plug fests to verify that all the UFI interfaces are implemented. This thing runs beautifully on Uboot. The project is open source, it's on GitHub, you can find it in the Tiannecore project. Building it is relatively easy, but there's that second repo there on gitlab.arm.com, the EDK2 test manifest makes it really easy to fetch all the sources and build a zip file that can be loaded onto a flash drive or load onto an SD card and then boot it on a Uboot, an UFI-enabled Uboot platform and just run those tests. Uboot also has internal UFI self-tests. If you enable the config command EFI self-test option, you'll be able to run boot EFI self-test and it will run through a ream of tests. Firmware Test Suite that runs under Linux does some limited testing of the interfaces that are exposed on the EBR platform. Firmware Test Suite is mostly concerned about ACPI, but it does test the UFI interfaces and it's growing some support for device tree. And then finally, just a plug-for-arm system-ready program. Because all of this is important, as part of this program, we've created the system-ready architecture compliance suite which is basically a pre-built disk image. You can find it on GitHub, you can rebuild it yourself or just download and use the images. And this includes the SCT, it includes ARMS BSA and BBR tests and Firmware Test Suite. So with that, you'll go through and it will run through all of the UFI interfaces. It'll boot into Linux, it'll run the Firmware Test Suite and that gives you a pretty good indication on how well the platforms matches the standards that I've been talking about here today. Spending some more time on the system-ready program, this is, as I said, I've been working on this for many years. Over the last year in particular, ARM has been putting together a certification program that actually puts some weight behind the standards. So it's not just the standards that are out there that can point to but we've got a branding program or a stamp program that will test platforms, if the platforms conform to the requirements, then we have a stamp that can be used by hardware vendors to say that, hey, we comply with this, you're more than likely you're going to be able to to run third-party distros or use Mainline Yachto without any difficulty. And so in this program, the things that we're looking for or testing are, first of all, we're looking for conformance with EBBR. So with that UFI subset defined in the EBBR spec, we are looking for the ability to pass all of the tests in the system-ready ACS IR test suite. And you can go there, there is a manual on how to use that. There is also on ARM's system-ready page, on developer.arm.com, you can find documents on how to run all of the system-ready tools. You need to be able to boot at least two unmodified Linux distros. The ones that we're recommending are Debian, Fedora, OpenSUSE and Ubuntu. If you're able to boot two of those without modifying the kernel config, then by and large, the major things that you need to do for system-ready compliant are already there. And then finally, update Capsule. And the reasoning on these is, we wanna make sure that we've got UFI interfaces that are sufficient for supporting basically how all the distros boot and that can be used by Yachto as well. We want to use two unmodified Linux distros because that gives us really good evidence that the platform has supported mainline and doesn't do anything funny that could easily get papered over. And so by doing that, if we've got that evidence that all of these things are implemented, we've got a third-party labs who will verify the testing and then ARM will look at the results and if everything is good, we'll put a stamp on it and we'll issue a certification document. Or if there's still things to be worked on, we'll send it back, work with vendors to get the issues resolved and then go through testing again. More information can be found on developer.arm.com, have system ready, and the contact email address is systemready.arm.com. As I say that, I think that URL is wrong, so I will either modify this video or put something over top to replace that link. Right, so that deals with kind of the big things of the standards that we're looking at, that I've been looking at and that the program's looking at. I really have to take a moment of just a huge shout out to the developers at Lunaro, the Yubu project, Associate Next, NXP, who have done an awful lot of work to really enable this and get this working. We have two certified platforms right now. There are many more that are coming up. In fact, by the time you see this video, there's probably going to be more on the system ready website. But there's actually been a huge amount of interest in this program. It's a great foundation as well. It can be used right now and I recommend going and looking at it. But it's just a start because there's still more things that as we get this baseline of functionality, there are things that can be improved. And here's some of the things that are being looked at. One is boot device selection improvements. Right now, the way Yubu chooses which UVFI target to boot is mostly based on Hush scripts and there's no way to get a boot list or boot menu or choose between those. So improvements there would be really valuable. Console device selection has some problems. Most of the time with what's provided now, the kernel messages will come out no problem but getting the console argument for the Getty still needs some work. And sometimes enabling the frame buffer ends up breaking the console. So there's, and that's kind of, that's more systemic problems of the architecture of how console is provided from Yubu through the boot and through into user space needs to be refined. Want to bring in measure boot, device revalidation. I talked about device tree stability. There are tools for validating device tree schema. Want to start making use of those and to really make this program more robust. And so this is all going to grow. If you're interested in this, please come join us at the EVBR project. There is bi-weekly meetings and so technical input, what you would like to see in the standards would be really, really valuable. And right, so some highlights here. The certification program is running right now. You can go to arm.com system-based certification program for a list of all the certified platforms. That's the correct URL that I was talking about earlier. So, and you can expect more announcements before the end of the year. Join us at the EVBR project and you can contact us to ask about certification. And with that, I am done. So, I think there's questions now. I mean, I'm recording this a few weeks ahead of time. So I will be online to answer questions. But thank you very much for listening and I hope you enjoy the rest of the conference. Bye.