 Hi everybody. So I'm going to talk about kernel image formats, mostly related to Uboot. So a couple of goals for this talk. I'm going to talk about, is my voice loud and clear, everyone? Okay, thank you. So some of the goals for this talk. I'm going to discuss about existing image formats, the limitations, you know, whatever's out there. I'm going to talk about multi-component images. Some support for that has been added where you can embed like a RAM disk, a device tree, all this kind of stuff into a single image. I'll talk about the challenges and the limitations. I'll talk about how some of these problems have been solved. And I'll talk especially about the fit image format, which is what this talk is mainly about. And I'll discuss how fit solves some of those issues in having like an image that has multiple components embedded in it. And of course, a bunch of other advantages of the fit format itself. Then I'm going to talk about applications where fit has been used, you know, especially about secure boot, verified boot, things like that. And a whole lot of other applications. I'm going to talk about the other advantages, you know, of potentially using fit in your applications and talk about future work and all the challenges we're seeing having this stuff accepted. So I'm going to start with the classical image formats. So especially start with the Z image format. And this is what it basically looks like. There's a header on the top there. It contains some magic number and some other fields. There's some code in head dot s that does some initialization. This decompression code that actually has the decompression logic that takes the kernel payload and decompresses it. So this is what it looks like. So this format is like very limited. The way it is, you know, there's hardly any information about the kernel itself, what architecture it has. And, you know, there's really no support out of the box of embedding a DTE device tree meaningfully in this format. You know, there's no check sums for data integrity and, you know, a bunch of other limitations. It has a few kernel, it does have a few kernel decompression algorithms though. And so along this line, you know, there's format called DTB image which was added to PowerPC and does have some support for, you know, embedding a device tree inside of the image. You know, and this has been proved useful for systems that don't support passing a device tree from boot loader or the firmware. So again, this really hardly talks about what the kernel is about and, you know, it's load address. There's no check sums. There's, you know, and all that kind of stuff. So it's very limited in the way it is. And somehow we come to a simple image format which is just like DTB image format but it can be executed anywhere in memory. And basically this is for systems that can't pass anything to the kernel at all. So all the information required for boot has to be present in the kernel already and including the arguments and all that kind of stuff. So this was useful for systems that could pass nothing to the boot loader, not only a device tree but literally nothing. And again, this is limited. So here are a few image formats that, you know, are very limited in what they can do. And I'm going to talk a little bit more about this for ARM. There are some, you know, hacks that basically support appending a device tree to a Z image and, you know, this was added quite some time back. And it has a bunch of drawbacks. Really, you know, the data that is appended at the end of the kernel is really no notion of what it is. Someone who looks at the image doesn't know what it is really, you know. And you can only embed one DT in this way to a Z image. And that makes the image only a single platform once you can't take the same image and boot it across multiple boards. So because you've already, you know, embedded a device tree into it. So and though there is boot support, you know, code has been added to head.s to support this. But to build this, there's really no code. I mean, if the device tree is appended, you know, the kernel can detect that it has been appended and, you know, do what is required. But there's really no as such build support. And you really have to build, append that device tree yourself. So it's a bunch of limitations. So now coming to UBoot's image format. So UBoot's image format is really operating system architecture independent. You can specify those in the header, you know, that information supports a bunch of compression types. It has checksums, CRC, you know, it's really good. You can ensure the integrity of your kernel image has executed in place support. So there's a special type of field that tells you that, okay, once you load the UBoot, directly execute the kernel code directly from within the UBoot image and don't copy it anywhere else. So support for that. There's quite a bit of metadata I'll show in the next slide on what the kernel is about and things like that. The U image format is very efficient to parse when it was introduced 13 years back. Systems were, you know, didn't have much processing power and, you know, having a fixed size header was really meaningful and U image was very lean and, you know, it was efficient to parse. So this is what it looks like. So you have a bunch of, so the stuff in blue is like a fixed size header. You have name, architecture, you know, all that kind of stuff. You have a timestamp that allows you to do things like rollbacks and stuff like that. You have a magic number that tells you what type of image it is. And you have a bunch of other fields where to load the kernel in memory, what the size of the kernel is, where to jump, things like that. And it's not limited to kernel. Of course, you can have, you know, this same image format applies to, like, you can build a RAM disk image or a bunch of other things. So it's not really, though I keep saying kernel, you can use this for many things. So that's what it looks like. So it's how it works. So my clicker works. So this is in a nutshell, this is what, I dropped a bunch of fields because there was no space, but typically in a U image what U would does is it takes the payload and it copies it into a location specified by this field in the header. So it takes this and copies it here. And then it just does a jump to the entry point location and it just executes it. So all this happens with the bootamp command. So you just say bootamp and then bootamp parses this header up here and it does a copy that I mentioned and execution. So, and yeah, this copying is not really required. And for image types that do in place kind of booting, the jump is directly to the payload itself. So it's not really required. And in systems that you have not flash, for example, where you want to directly jump, this mechanism is not used. But I've seen this used quite a bit and kind of copying to a location and then jumping to the entry point is what happens. So yeah, and it's a command that you can use to see what the contents of the U image are, shows you all the details there. So then you come to multi-component U images where we have the notion of embedding more than one component in a U image like a device tree blob, you know, RAM disk and things like that. So what were the limitations of single-component image? Users wanted to embed like more than one component, obviously, all in a single image. There are a lot of use cases and that cannot be handled. And you have a bunch of other use cases like booting through DHCP, you know, you want to pull just one image, you want to like pull like three different components and then, you know, from different locations kind of messy. And then for recovery of systems where, you know, you want to RAM disk, it's kind of nice to have all that inside of the image itself, right? So you don't want to mount your root files, you want to directly go into a RAM disk with busy box and stuff. So that's kind of one of the applications possible. Firmware upgrades, it's not easy to download multiple components. Firmware, like vendors prefer to have just one image that just works. And even security, like there's no support at all, even in multi-component images actually that I'm going to talk about. But fit does have support for security, where it was thought that it would be nice to embed the cryptographic signatures inside of the image itself. And that's just not possible, you know, with these kind of simple image formats. So I'll talk more about that. So the type field I was talking about here, you know, has to really say that, okay, I'm a multi-component image and so that is what this macro is about. And that's how you would know that, okay, you know, there might be more than one component in this image. And another limitation of single component images is that the checksum is only limited to CRC32. I mentioned that it's a good thing for the checksum, but if it's CRC32, it's a small chance that, very small chance that, you know, the image is carved, but the checksum is the same. So, you know, these are the limitations of having a fixed size format. This is what a multi-component image looks like. So what they did is they went ahead and added a little table here in the payload that tells you more about the different, actually this table is non-terminated. So you can have, like, multiple sections in that. And the way it works is, you know, because of the non-termination, you know where the table ends. So, you know what the next component is and you know its size. So you can, in that fashion, you can embed multiple components that way. So, this is how that works. So, you know, I find this approach very ugly because, first of all, it's like, predefined all over in the code that, you know, the kernel has to be in this location. And RAM, it has to be the first thing here. And the device tree has to be here. It's not the whole notion of, it's not flexible, you know. So the code has things like, you know, it makes all sorts of assumptions that the kernel is at index one and, you know, the code is really messy because of that from what I observed. So it's fixed mapping of ID to component type, predefined, you know, all that stuff. So it's kind of messy. So, like I was saying, yeah, so the good thing was that now you can have, you have a single image that you can just DHCP and it has all the good stuff in it. You know, you don't have to... So that was a good thing that happened because of this image type. And, yeah, again, like, there's no... Though you could embed multiple components, you can't really have, like, multiple kernels in an image. So you can have an asymmetric multiprocessing system where we have two different cores and you want to load different kernels. So that was not possible even with this. So the same disadvantages, you know, that the IDs were hard-coded, you know, and associating numbers instead of names is not a good thing. Like, you need to make it a little more... The code has to be cleaner, you know. If you have IDs in the code, it's quite sad. And so it's difficult to maintain. You know, code is very, very horrible. And so, you know, it's like very inflexible, basically. That's what I'm trying to say here. Okay. So again, limited support for only these three components. Multiple kernels cannot be represented and it doesn't scale for future designs. No support for stronger checks and stuff like that. So all that I was talking about till now is the existing image formats and their limitations and all the disadvantages I showed you. It was good 15 years back, but now it's time to move on and use something more powerful and, you know, something more flexible using existing tools. So I'm going to talk about tree-like structures to represent some of these images. I'm going to talk about the, you know, like basically what a tree-like structure can add. You know, you can have a more flexible way of representing a kernel like in the form of a tree. You know, wouldn't it be nice to have something like this, you know, to represent an image however you want it with the fields placed wherever you want, as long as you have those fields somewhere. And, you know, you can arbitrarily arrange nodes in a tree, move them around, trees can have properties. Properties can even be binary images if you want it. So a lot of flexibility added, you know, because of, like, if this fixed-size structure could somehow be made more flexible and trees are a nice way of possibly representing it. So we're going to talk about a little bit of a device tree, what it is and how it has been used for fit. A device tree is basically a structure to describe hardware. I think most people here know that. So the idea is that instead of hard-coding everything in the kernel about the platform, you know, you have all that data, you know, have all that code converted to data and actually use that data to represent, to tell the kernel more about what the platform you're booting on is about, you know, the details. And that kind of makes the code a lot cleaner. So the problem now is that now you have all these device tree sources that is not code, but actually data, but it's still in the kernel tree and, you know, that's adding a lot of churn. I'm sure those who attended a lot's talk yesterday heard them about talking about moving it out of the tree and kernel tree and stuff like that. Anyway, regardless, it keeps the code clean and, you know, there's been a lot of effort in that direction to convert drivers to use device tree instead of using, you know, these hard-coded platform data kind of stuff. So that's what device tree is about. You know, it has a basic describe CPUs, peripherals, memory, all that configuration, kernel output, which serial port, kernel parameters, all that kind of stuff. Can be embedded in nice tree-like format. So this has nothing to do with the image formats that I'm talking about. This is just what device tree is and how it has been used. So the question is, can we reuse the device tree? So it's already in use in the kernel. So why not just reuse it? So tools that build the tree are already part of the kernel. So you want to take existing tools and kind of use that in a more useful way. And device tree compiler recent also has got support for embedding even binaries inside of node properties, you know, property values. So all this good stuff is already there in the kernel tree. You know, we're not really talking about adding anything new. And so, sir, Marion, I think, between 2008 he came up with this whole idea, you know, for PowerPC. And the idea was actually it emerged because they wanted stronger checksums and U-image has fixed field for CRC32 and you want to do MD5 or Shah or something like that and you just have no way to do it. So the idea was, okay, let's Wolfgang and Marion and all these guys got together and decided, let's come up with something that's more flexible and that's where they invented fit. And it really uses, makes use of device tree to build a tree node correspond to image components. So now your RAM disk is a node inside the tree or, you know, your device tree blob itself is a node inside a tree, you know, and stuff like that. It's a perfect use for multiple component images, you know, like, you know, embed multiple kernels, multiple device trees, multiple RAM disks, multiple things you haven't even thought of. And, you know, this format can be adapted to pretty much anything that you want to do and that you can't think of now, but you might want to do in the future. So this was the board that had the first support added and needed stronger sections and, like I was saying, all these details. So any questions on so far? Okay. So I did some research and I found quite a few users already there in U-boot users. So you have these 8-core processor boards exhibited, you know, and they use fit quite a lot. In fact, all the XES boards use fit quite a lot. And most if not all free-scale boards that I've seen in the U-boot sources use it. You know, here are a bunch of other boards. I won't bore you with the details, but there's Neo Free Runner from OpenMoco, the Zinc free-scale again here. And yeah, this is very interesting. Samsung Chromebook actually, they do secure boot and they're planning to use fit to kind of store cryptographic signatures inside of the image tree. And I also have a little bit of a demo on my favorite platform BeagleBone. I'll be showing you, like I just added support for that. So I'll just show you the logs. I try not to do demos because demos usually go wrong at the wrong time. I mean, you know, so I have the logs of what I did and I'll show you those details. And X86 support is not really there, but Simon again has posted some patches on the U-boot mailing list for X86 support and they're being reviewed for what I know. And then you have this soft CPU from Zardinx as well that actually uses fit. So coming back to the Zimage hacks to support appending DT and this was one of our main points to, you know, convince all the maintainers that we really want to do this, that it's kind of really a messy way to, you know, it has a bunch of drawbacks of appending a DT. So I think I brought up these points earlier, but I'll say them again. There's no kernel support for this. You have autotree hacks, kernel build support. Runtime support is there, but there's no build support. You have hacks floating around that you have to apply to actually embed a DT, you know, and sometimes I do like hundreds of iterations and I don't want to like keep, like sometimes my device tree is just fixed. I don't want to pass it, you know, copy from my pass it. I want it to be in my image. I want to build kernel and device tree all in the same image. That's not possible. You need autotree hacks in the kernel code to actually do that build. And again, it makes it a single platform where you can embed more than one device trees, you know, even with this approach. And so, yeah, this is the kernel config option you have to enable in order for the kernel to actually work. So there is runtime support when kernel boots, it can detect that there's device tree appended to it, and it can do all that. You need to enable this config option. So these are just details. This is what the code looks like. You know, you basically cat the Z image with the device tree and you output a new image. So it's very simple. You just slap both of them together and, you know, that's what it looks like. So what I talk about fit and how it can be used, I'm going to show you a little demo in the slides on... You know, I'll show you source code and I'll show you different components inside source code and, you know, I think we did a little short of time to try to keep up. I'll show you the source code, show you how I built the actual image, show you your boot commands that you can use to boot the image, show you boot log after it boots, stuff like that. So my first demo is going to be a single kernel, single device tree blob. We're building an image out of these two and it's quite simple. So that's the first demo. Now, for this experiment, I used UBoat, the latest one, at the time, and kernel 3.8. So everybody knows the big old board. So this is what the sources look like. So basically, you have a device tree source version you have to specify, just for backward compatibility. And then you have the whole tree here and you describe the image and, you know, this is the image node. You start with images node and then you have to put a kernel. You have to have this kernel node here that tells you about the, about what image you want to embed inside your, inside the kernel nodes. This actually gets converted to a binary property. Then you have a bunch of other fields, you know. You can specify them flexibly, up and down, wherever you want. You're not limited by a fixed size header. And this is the most interesting thing, like you can actually embed hashes, multiple hashes for even stronger checks on your kernel, you know. So this node is mandatory kernel. So you have an images parent node and then you have multiple sub nodes. The sub nodes tell you what the different components are. So you have first component is kernel. Then I guess I have, in this demo, I have the next component is device tree, the device tree blob. So you can specify which device tree blob you want to embed and this gets converted to a binary property. And again, you have the same hashes and all that on the device tree blob component of the image. Okay, so then you end the images, the parent images node that encapsulates all these little images. You end it with this and then you have a configuration node. That configuration-based concept is that you have different configurations and you can tell, you can say that for this configuration, I want to take this kernel, this boot loader, this device tree, this RAM disk and boot them all together. So it's a nice way of, you know, you can just tell you boot this configuration and it will just take these components out of the images, out of all these image nodes and it will boot it together. So it's very, very clean, you know, the way it is. Then I built the image using MKImage. Very simple, you know, no fancy properties or anything, just simple. You know, if you look at the kernel sources, MKImage is like three lines. This is just so simple. So I build it like this and then this is the output of the build and this is the, this is the, you know, the ITB, so to speak, image tree blob that came out of the image tree sources that I just showed you. So you can see the checksums and stuff that shows you, you know, the SHA and all that has been embedded inside the image already. So MKImage computed those values for you and stored it in the image. Then we just boot it. So booting it is very simple. I didn't have to pass anything to boot it. No device tree location or anything. I just say boot it and it just works. So I copy the ITB I just built and I just boot it. So these are the boot logs of what it looks like. You boot, you know, detects its fit and it looks at the checksums, it verifies it, makes sure everything is okay. Then it does the same thing for device tree. This is not even possible with, you know, old image. I mean, we didn't have concept of checksums since port device tree blobs itself. So, you know, it does even that and loads the, you know, kernel and the device tree from the load address properties I just showed you in the sources. And then it just starts. So it's quite nice, you know, quite elegant and flexible. So in the second demo, I'm going to introduce the RAM disks. So now I added a RAM disk to here. And I don't specify any load address. Just set it to zero so that you would copy it wherever it likes. You know, and I just added, you know, one hash here for the RAM disk. And I created a new configuration in second demo. This doesn't work. Okay. So yeah, too bright. So the second configuration says that, okay, I want to add a RAM disk to the mix as well, not only a kernel and a device tree. So it uses this node that I just added here to, you know, kind of encapsulates all that together. So then I just build it the same way and it, you know, it created the new configuration. I'm showing you in blue here. Pretty standard. And then I just, then I boot it. So this time, I want to boot the recovery configuration. So it's quite simple. I just have to pass the configuration name that I had here, recovery conf. I just pass it to the boot-em command and it takes the kernel RAM disk and device tree from the fit image and it boots it together. So I actually, you know, booted it and there you can see the new updates. And it added, loaded the RAM disk as well, and now I'm inside of a nice RAM disk file system and, you know, I can, you know, I don't need an SD card or anything anymore. So this is one of the other applications. I think it will be very useful to have something like this. And then there are more use cases for fit, like other than this, like just like what I showed you, I mean, you can have two kernel images and one kernel image can be for production. You optimize it, turn on all the optimization flags and make it compact and all that. And then you can have one for your own debugging where you would turn on everything, all the tracing infrastructure and all that. And you can embed both these kernels in the same image and you can have multiple configuration nodes that say that this is a debug configuration node and I'd like you to boot, I'd like you to pick the debug kernel and likewise for production. So, you know, it's another application for that. So, and the other big application where this will be useful, and I'm trying to, I've been writing code to kind of have multiple device tree blobs embedded in the same image to kind of make it like a multi-platform kernel image. And you have different configuration nodes for different boards and you'd have the right device tree blob in each of those configuration nodes. So, you can really have one image that kind of supports multiple boards. It's kind of a nice way to do that. And what you can potentially do is read the WE from or figure out from switches or something that it's this kind of board and, you know, that I need to, you know, board X or something and I need to boot configuration X. So, something like that. So, we'll see how that goes. And, yeah, this is another big use case. Simon Glass was saying added fit support for his push patches for x86. You know, it's like really useful. Like, this is something that wasn't even thought of when fit was invented few years back. And he went ahead and added signatures, you know, that he, you know, the value of which is computed by using a private key and signed those images and embedded multiple signatures, you know, inside of the, inside of the image itself. So, that's really useful. It's something that is not possible at all with the existing U-boot image formats. It's not at all possible to do this. So, it's really flexible. Something that wasn't even thought of can be added to the U-boot image format and, you know. And then he went ahead and added even more security support. So, the idea was that you have a kernel and you have a signature for that. And that's verified and everything's good. And then you have all these other components and everything has signatures. You don't have signatures for configurations. So, now somebody can easily hack the device tree and boot some random configuration that you haven't even thought of and that exposes a lot of holes and stuff. So, he went ahead and added signatures to, you know, even configurations. So, now U-boot can verify that the configuration node itself has not been tampered with. So, stuff like that. And this is another use case. It still hasn't been done, but we're discussing, I was discussing with Simon that, you know, for, this is perfect for AMP configurations where, you know, the P2020 free-scale SOC, which is what the P2020-RDB board is based on. It's a perfect use case because right now what it does is it takes two different kernel images, TFTPs them, you know, that's how they've written it. And the TFTPs, the device tree blobs separately and then it, you know, takes each of these kernels, manually loads it into each of the, for each of the processors and it passes the right device tree blob and stuff like that. So, there's a lot of, you know, magic that has to be done to make this kind of thing work. And this is a very good use case for FIT, you know, where you can have embedded those kernels directly in the image itself and then just have a configuration that possibly tells you which core to load, what kernel on and what device tree blob to pass it. And it can be done much better in a nice little, you know, tree than doing it the way they do it today. So, what I thought was a very, very good application of that. And yeah, I was saying I upgrade procedures for devices, also some vendors want to just, you know, distribute a single image and they want to give you tons of components, you know, customers can use them in a different way and break their boards and stuff and whatnot. So, I guess this can be applied to this kind of use case too. So, a lot of future work and challenges. So, we need a simple way to extend the, so we already have the DTB's target in the kernel, which what it does is, say, make DTB's and it builds DTB device tree blobs for all the platforms that you have configured. So, you know, they already do that. So, why not just take all that and use some of that, you know, and at the same time build a tree itself that contains all the device tree blobs. So, there was a lot, very difficult to accept these patches. Two, three years back, patches were posted for PowerPC and they were grant-likely nicked them and it was, he had his own reasons and, you know, but recently it has become easier to, you know, there's been more, this idea has been more welcoming, you know, now that we're moving to a multi-platform kernel and stuff like that. So, it's much easier. So, that's what is encouraging me more to pursue this. And generally, the challenge in the community is like, if people don't use Uber, they just hate it. I don't know why. I've seen that a lot. Like, you know, generally speaking, this considered to be something that's very bloated and all that, and I really think that it's not the case. You can make it as small as you want to or as big as you want to and it's a good thing that has so many drivers because you can boot in so many different ways. So, that is, again, a very debatable topic. So, that's where that is at. And you would currently... Yeah, so this has been breaking things all over. I think Tony posted some patches, I guess, and, you know, the whole move to the multi-platform kernel. So, we've seen breakages related to that, you know, where the Uber image itself is not built with a load address. And if you didn't do that, it would break. So, and even fit images are breaking. So, I have a little patch I wrote yesterday night that fixed that problem. So, I'm able to boot a fit image without specifying a load address at all. So, you know, that's... So, I've proof of concept that you don't need a load address to boot a fit image. You can just boot it in place. So, that works. And I've seen... The other challenge I've seen is, like, crypto operations are very slow from Uber. Still looking into it, but it takes a long time to do a SHA and to verify it. And so it adds a lot of time to the boot. So, possibly use hardware accelerators to possibly speed up those crypto operations and stuff like that. So, that's possibly something I would like to look at at some point to speed this up. But generally, yeah, this format, it's going to be quite a challenge to accept it and add this support to the kernel tree. And I think it's a good idea because the tools are already there that do this, you know, device tree compiler, all that is in the kernel. So, I think it's not that bad and just adding this is... So, we're having discussions on the mailing list and we have a huge thread going on this topic. And yeah, it's an interesting thread. So, let's see where that goes. So, any questions?