 All right, I guess we can get started. So I'm here today to talk about a million different ways to provision your embedded Linux devices. Hopefully, that's what you guys are here to hear about. If not, no shame. Feel free to head out. I know we're bumping right up against lunch. So I don't want to get in your way on that. But just a brief overview. The desserts here are kind of to kickstart our attention to lunch. The idea being that there are a lot of choices out there. When you're bringing up your embedded Linux devices, obviously a million is a bit of hyperbole in this situation. But sometimes it doesn't really feel like it. If you've ever worked with a wide variety of boards, you'll realize that there's lots of different ways of getting things on there. So I'll talk a little bit about the anatomy of an embedded Linux system so we can kind of understand what pieces we have, where they need to get installed, and why. And then discuss the storage and provisioning of each of those individual components. Obviously, the bootloader, kernel, device trees, if necessary, and then the root file system. And then map that to some very specific devices with the command lines and that kind of thing to show exactly how for specific devices, the initial images are getting loaded. With the goal of presenting an overview of just about all the different mechanisms you're going to encounter as you work through heterogeneous fleet of devices. And before I go any further on that, I did want to specifically define provisioning in this case. And what I'm referring to when I talk about provisioning is getting the initial bit of code on the device. And it's one of those very overloaded terms that is used in a lot of different ways depending on the exact conversation. But for purposes of this talk, the idea is how do you get that initial bit of code onto the device once you pull the device out of the shipping box? So just real briefly, just a bit about me. My name's Drew Mosley. I've been in the embedded Linux and Yachto space for 10, 12 years now, something like that, and even longer than that in general embedded. My current role is Project Lead and Solutions Architect for the Mender IO over-the-air update client server system. We actually have some t-shirts here. If you want to grab a t-shirt on your way out, feel free, we figured we'd bring some down here because we really don't want to take them home. So if you want t-shirts, we've got at least 10 or 15 of them back there. So what are some of the challenges as embedded Linux developers that you're gonna face when you're bringing these boards up? The first one that you're gonna find out is that there is not one way to do this. Every board has its own little idiosyncrasies. Every piece of media that you're writing to, you have to know yet another command line with some cryptic flags that may mean something to somebody, but generally speaking, when you first get the board, they probably don't mean anything specific to you at the time. Another challenge is that generally speaking, the mechanisms to get the images installed onto your board, they're gonna vary quite a bit depending on where you are in the development cycle. Early in the development, you might cheat a little bit and use NFS and TFTP and a few other things. And as you move closer to production and dealing with CI CD pipelines, things are gonna be quite a bit different. And then when you go into manufacturing, I've got a whole slide on that later, just some considerations for when you go actually into the manufacturing environment. You wanna make it as simple as possible for your manufacturing team to be able to get the initial program loads going into these devices. And additionally, they vary widely between boards, manufacturers, between different boards by the same manufacturer, depending on what the storage media is and who designed the board and whether they had a good weekend or not before they did the final layout. And the one biggest thing probably most relevant to all of us is the initial provisioning, if it's slow and tedious, it can lead to some long development cycles, especially when you're updating kernels and adding new device drivers and you've got to really go back to the very lowest level and get everything redeployed from the very beginning of the system bring up. So this slide's probably review for most folks in here, but just briefly, kind of the 10,000 foot view, the anatomy of the system. At the very lowest level, there's generally a bootloader. This is the code that is responsible for bringing up everything from the very beginning, the initial fetch of the reset vector, initializing whatever hardware is needed. Generally speaking, at a minimum, it's gonna initialize the interrupt controller, make sure there are no pending interrupts, it's gonna set up power rails, it's gonna initialize RAM so that it can load a kernel and that kind of thing. Depending on what functionality you need, there may be more into the bootloader, there may be diagnostics, things like that. But from the Linux perspective, as long as it gets the system up far enough that it can actually load a kernel into RAM, that's the bare minimum. From there, obviously we have the Linux kernel, core operating system, functionality, process control, device management, that kind of thing. Fairly straightforward, I think, like I said, probably everybody understands this. Some architectures require a device tree, which is a description of the hardware that's separate from the kernel itself. And finally, the root file system, which is everything else. All the files, all the data for whatever your particular application is. So looking at the storage overview, specifically for the bootloader, different options for storing the bootloader. So you see the five that are listed here are the most common, at least that I've come across, and I won't read them all out, but you see that there's quite a few there, quite a few different options with some boards describing which ones use them. The kernel and device tree can be very similar. Some systems actually put the kernel in a separate partition, some systems put it as a file in the root file system partition. For Mender specifically, one of the first things we do when we're integrating Mender is move that kernel if it is in a separate partition or is loaded through some other mechanism, we change that so that it's actually a file in the root file system, which gives us the ability to update the kernel over the air as just part of our standard file system update. And obviously the root file system, quite a few options there. You got things like EMMC, an SD card, raw NAND flash using UBI and things like that. And then your typical USB drives, a SATA SSD, and then most systems, most boards can actually use root file system over NFS as well. So first thing I did, I wanted to get a setup using Yachto plus the built-in Kimu machines just to kind of illustrate a few things. The set of instructions here, basically I pulled these all directly from that first link on the bottom, directly out of the Yachto project documentation. The set up of the estate mirrors you see there going into local.com, which means that this whole build from scratch took about nine minutes on my Ubuntu 18, or Ubuntu 16 system. You can get, the bottom link, the bit.ly link, that's actually a GitHub gist link to this exact script. So if you wanna pull this down, it's there, you can actually get this setup and running in fairly short order. Like I said, this was done on my Ubuntu 16, so I make no promises on any other target OS. But the idea with setting this up was just to demonstrate a few things. Once you have this build, there's a script within the Yachto repository that gets checked out called Run Kimu. And this has a lot of interesting options in it. The very first line you see there, we're just invoking it, Run Kimu, Kimu x86, in my case I just used no graphic because I just wanted the text to show up in the bash shell that I was in. This basically uses the Kimu block device and mounts the device directly out of your Yachto build. So you don't have to do anything more than finish your Yachto build and run the command. The next set shows you kind of how to do an NFS based setup with this system. Just with one caveat, I couldn't get it to work this time and didn't spend a whole lot of time troubleshooting it. I know I've done this in the past, so you have to install the RPC bind and configure that. But then basically the Run Kimu script takes care of extracting the archive, creating the exports file entry and doing all the NFS side setup so that then when the Kimu system launches, you're actually running over, with your root file system over NFS. And the nice thing here is of course, there's a lot of complexity going in there and the Run Kimu script really kind of hides all that from you. It's not the same as running on real hardware. Generally speaking, Kimu does not load a bootloader. It loads the kernel directly. Again, there's not normally a DTB and the kernel is direct, in this case is not loaded out of a flash device or something, it's just loaded directly as a file off of your build system. But if you are working on some components of your design that aren't hardware specific, this is a great way to get in and get a system up and running. I know that this system will, generally speaking, run all the graphical user interfaces. I know I've seen the Sato and the XOR GUI, I assume it probably also does Wayland and things like that. So you can actually make quite a bit of progress in your system design with just this mechanism of deploying the image. All right, so now we get kind of into the meat. So this is the first provisioning model that most folks come across and that's just writing everything to an SD card in some format. In this case, my example is the Raspberry Pi 3. You kind of see the image over here on the side of the way the SD card is laid out. You have two partitions, one is a fat partition and it contains things like the device tree binary. The kernel 7.image is the actual kernel image that gets loaded by the proprietary bootloader that's onboard the Raspberry Pi 3 in the case of something like Raspbian or out of the box configurations. Kernel 7.image is generally a Linux kernel but with Yachto you can actually tell it to build with Uboot so in that case that kernel 7.image will be a Uboot image which then uses standard Uboot commands to load a kernel binary which could come off the partition, I guess that should be partition two, that's EXT4 up there. So and that's just a standard root file system. So in this case this image file is generated directly by the Yachto build system. So you actually get a single binary image that has the multiple partitions in it. You use the DD command that I'm showing here and just write that at a low level to the SD card in your host build system. Once that write completes you pull the SD card out, put it in the Raspberry Pi and you power it on and everything goes from there. So I suspect probably everybody in here has done this mechanism of initial provisioning. This is pretty straightforward, like I say the Raspberry Pi, they're everywhere. So this is the one that you will definitely see. So the next one is in the case of EMMC. So this is basically, it's effectively the same media format as SDMMC except it's soldered directly to the board. So there's no mechanism to take that piece of storage media and put it into your host PC. So it adds a little bit of complexity. This was done on TechnXE and Pico Pi, IMX7 based board. Again, the image map that you see there, the U-boot binary is actually stored in a partition space. It's not actually a file in a partition, it's just in raw EMMC flash space as well as the U-boot environment. Additionally, you'll find the fat partition which is where the kernel and device tree are out of the box loaded from and similarly you have the XT4 partition. So it's a little bit more complicated than the previous one, but the complication, it gets even a little bit more complicated when trying to figure out how do you get this into the board since you don't have access to that from your build system. So the image file in this case is also generated directly by Yachto. So basically you get a bag of bits that contains the entire image that I'm showing here. There's, once you have that, then you run the two IMX USB commands you see here. One to load the U-boot SPL and then one to load the full U-boot image. And basically what that does is with the Technixian board connected over USB to your host machine, the Technixian board will come up in gadget mode. The two IMX USB commands are basically poking those binary images into the RAM of the device and then launching it. So you'll actually be running U-boot in that case out of RAM while the device is connected to your PC. Once that RAM-based U-boot starts up, you will then run the two commands you see at the bottom, the USB start and the UMS zero MMC zero. And what that does is that sets USB up in mass storage mode. So it's, I guess, USB device zero and then MMC device zero. So depending on the board, those numbers may change depending on what devices you have on the board. But effectively, from the perspective of your PC now, you've just got another USB disconnected and then you use the DD command from your host PC where you've done your builds and then you just have to make sure that you get the right device node for this particular board that wherever it got mapped to in your device space. Obviously, I've got another error in this slide where it says an RPI SD image, this would actually probably be a WIC file since it's the Technixian and not the RPI. So a little bit more complicated than the previous, but generally speaking, EMMC is a little bit faster, a little bit more reliable and we're definitely starting to see a lot more boards come out with EMMC and not even providing the ability to boot out of SD card. So, and this next one adds even one more layer of complexity. This is the CompuLab IoT gateway machine based on the IMX7. So in this case, we actually have two physically separate devices. EMMC is very similar to what we saw previously except the bootloader is no longer in the EMMC. There's actually a separate SPI flash device that is considered the boot device of the system and that's where your U-boot image goes. So other than that, pretty much everything else is the same. And generally speaking, when you have a board like this, there's gonna be some mechanism in the bootloader or rather the bootloader will come up and from there you can determine whether you're gonna use EMMC or SD if you have those as options. The bootloader, especially if you're using U-boot, you can just say load from this device instead of this device. But you will also generally find that there's gonna be a jumper selection or some means in hardware of selecting whether you're gonna boot out of this SPI flash or if the board provides it some kind of fallback mechanism in the case of this particular board. There's a jumper on board that if you connect the jumper, then you're actually booting fully off of the SD card just like we did with the Raspberry Pi. So basically the way this is gonna work then, you're actually going to take the image and deploy it to a USB key or to an SD card, select the jumper setting to boot off of that device. So now you're booted into a Linux system on the target device and you're actually gonna use the flash erase command and the DD command you see there to write the new U-boot image into the SPI flash and then the DD command writing to the DevMMC device to write the root file system into the EMMC chip on the board. So it's, you know, we're getting a little more complicated. You know, a lot of times the nice thing about this kind of layout is you could, some boards actually provide hardware redundancy for that boot loader so that if you did need to do, for instance, an over the air update of the boot loader, you could do it in a robust fashion because now you have a fallback version of U-boot whereas if you've only got the one and you try to update it over the air, you could end up with a brick fairly easily. And moving on, we moved to the raw NAN scenario. So this one was done with a Toradex Calibri IMX7. Image is a little bit more complicated now. We're actually using UBI volumes instead of partitions. Conceptually, it's very similar. In this case, we actually do have the redundant U-boot partitions or UBI volumes in this case. The U-boot environment is also a UBI volume and then inside another UBI volume, we actually have the UBI FS file system containing the standard root file system. So in this case, Yachta will generate images for each of these components and then you copy those individual images to the SD card. This setup does not generally create a single binary that like we had done in previous. You're actually gonna have multiple binaries that get copied to your SD card which are then deployed using the mechanisms detailed here. So the first command, the IMX USB command, similar to the previous slide, that will load the NAN-based version into RAM, the NAN-based version of U-boot into the RAM of the device and then you'll be running U-boot from RAM on that device and then you can use the U-boot commands you see here, the NANDER race and the load and the NAND write to actually write the new U-boot into the UBI volume. So you basically have taken that U-boot image, copied it to an SD card, a USB key, then you load it into RAM using standard U-boot commands from the command line and then you just write it into the flash and similar for the root file system. So it's very similar to the previous one but it generally requires a few more steps, a few more places where you can type something wrong. So just kind of keep that in mind when you're working with these systems. They can be a little bit tricky. And one more that I see coming up more and more often is using the Android tools on your desktop system to actually write the partitions on the device. So in this case, and this was quite a while ago that I worked with this board, so I haven't actually verified that this exact set of commands is functional with the current version of Yachto but something very similar to this should work. These were specifically pulled directly from the 96 boards. Documentation with the link you see at the bottom of the screen. Generally speaking, there's some mechanism to force the board to boot into fast boot mode, whether it's a switch, a jumper, it's gonna vary widely depending on what the actual board does but in this case, you are gonna be loading the images directly to the board from your host system over USB gadget link. So and if you look, there are actually the fast boot command here, fast boot flash and then the next parameter is the name of the partition that you're gonna be writing to. Some of these boards have 10 or 12 partitions because I guess the Android for some reason needs that many for different things but generally speaking, obviously if you're writing just a general embedded Linux system, generally speaking, the boot and the root are faster than the only things you need. And just a couple other provisioning tools I wanted to mention briefly because they probably will come up. Live installers are a fairly common thing, the Toradex easy installer, you know, whatever USB, whatever Ubuntu disk you have, a lot of systems you can actually use those and boot off of that and then use the actual live installer. Some of them like the Toradex easy installer can actually pull images directly off the net internet so you don't even have to have the image downloaded locally. It reaches out and grabs a JSON file of some kind with a directory of what images are available and it already has everything built into the easy installer to know how to deploy it for your particular board. Some imaging tools that you may come across. This is for things like the Raspberry Pi where you're writing the actual SD card. Instead of using DD, you know, those things like Blana etcher and the Win32 disk imager, those are basically just GUIs on top of those kind of things. They make it a little less likely that you're gonna overwrite the wrong device node because they're pretty smart about figuring out which device nodes are mounted, which ones are not and which ones actually are SD cards so that gives you a little bit more safety. And finally, some other protocols. I mentioned briefly at the very top of using TFTP NFS. I was hoping to have a slide showing an easy three-step mechanism to do that but I think that's probably a talk in and of itself because there's so many mechanisms to actually set up NFS and so many things. It's very dependent on your exact network setup. I actually have a Synology NAS at home with NFS built in so all I had to do was go into the Synology NAS GUI, turn it on, and then it uses Pixie Boot so files of a known format tell it where to find everything. So for me, it's just generally speaking a one or two line thing to get the system up and running. If you're starting from scratch, it's generally gonna be a lot more than that. And then there's some other protocols you might come across. I mentioned the IMX USB tool but there's others. Rockchip has their own mechanism which is very similar as does NVIDIA. They all basically use the mechanism of assuming the device can be a gadget mode, some kind of gadget mode connection to the device where it's able to then go and sometimes poke the images directly in into the flash of the device or sometimes simply run U-boot on the flash of the device like I mentioned earlier. And a few other considerations. So in your product development cycles, some of the things you might wanna think about and how they play together with device provisioning. How are you gonna integrate this with the CI CD system? You have Jenkins builds running Knightly builds. How easy is it to actually get the builds installed onto a set of devices? If you're going back and doing all this low level stuff with the IMX USB tool, you might be writing a lot of scripting around it. If you're using something, some kind of field OTA system, you could actually use that instead. Just have some basic system installed and then your Knightly build just comes in as an OTA update. That's a possibility as well. And another consideration is system developers versus application developers. It's not unique to device provisioning. It's fairly common in the embedded Linux space. Your system developers are the guys that are worried about all this kind of stuff. The app developers, they'll be writing their stuff in Node.js or QT and that kind of thing. So they generally speaking aren't gonna be provisioning a board from scratch. They'll probably be getting a board with a Knightly build already pre-installed. So there might be some other considerations that play into exactly how you wanna handle the provisioning in the system. And where it gets really complicated is when you start talking a heterogeneous fleet of devices. We talked about five or six different devices here and they're all different mechanisms. If you have one device that you're deploying to for your entire application, then you only have one mechanism to worry about. But obviously, things get more complicated. The more targets you're deploying to. And finally, a couple things about manufacturing. You need to make sure when you're ready to hand this thing off to manufacturing that it's as few steps as possible in the manufacturing line. It's as quick as possible and it's as bulletproof as possible. Generally speaking, it needs to be an unattended installation because it's on a manufacturing line. They don't wanna have to somebody popping SD cards in and out connecting cables. The manufacturing folks hate that. So you want to avoid that at all costs. One thing that comes up a lot in discussions I have with folks is, okay, so I'm starting with one binary that they can install on all my boards. But each board will have some per board data associated with it. Whether that's a serial number, SSL certificates that are unique to the device, whatever. So you will also need some mechanism to deploy that per board data and keep track of which board has which. You're gonna have to have some mechanism for doing that. A lot of times you can actually do it as just post install steps when the board boots up for the first time, it generates the certificates. Or if you want to inject that certificate so that you can then take it and use that certificate to pre-authenticate the device with some kind of device management infrastructure. A lot more things you might need to be thinking about in terms of the per board data. And finally, the burn in test versus production images. Generally speaking, the first image that goes in in manufacturing is probably not the image that's gonna go out in the strict grab package to go out the door. You're probably gonna have multiple images. You'll have a burn in test that runs for a while, might be a smaller image and might go quicker. But then when you get to the final production image that's gonna have all your application code. And that's really where you start worrying about things like per board data and that kind of thing. And with that, I think that's the end of my slides. And because I promised somebody I would do this, I'm gonna end with a joke. This joke made my day when I heard it a few weeks ago and every time I tell it, it makes me very happy. And then we can move into questions. So how did the cyber criminal get away? Nobody? Ran somewhere. Come on. That's my favorite joke of late. Okay, so I think we've got a microphone somewhere. Does this one work? Let's see. Well, hello. All right, well, I'll just try to repeat the question then because I'm not sure how this is working. Anybody got a question? Yes. Okay, so the question was do I have any examples of secure boot or trusted boot? The answer is no. And that's probably, thank you for bringing that up. That's probably a good point to mention. Secure Boot adds a whole another layer of complexity here because you got all sorts of certificates and things you have to deal with. My focus here was really more on the early stages of the development cycle where you're bringing up the board for the first time, but definitely Secure Boot is gonna have to come in ideally fairly early, but obviously that's not a first day bring up consideration. Yes. Wait, where was that? Yeah, so for this one, there's actually a jumper on the board. So, well, all the jumper does on this particular board is selects whether it's gonna load the bootloader out of spy flash. Let's see. Honestly, I don't remember. But I think by default, if there's not a bootable image in the EMC, that's the default. Presumably, again, there's a switch or a jumper or something. I don't recall off the top of my head for that specific. And there may be examples where it is something like that UMS command is similar because that's what's actually going, obviously one level higher in the USB stack, but that's what's putting you into USB to mass storage mode. Why is the boot partition always fat? That's a good question. My guess is because for things like Raspberry Pi, there's a lot of people running Windows that want to be able to deploy to them. And they need to be able to edit the files in there. In this case, in fact, in a lot of these cases, it doesn't need to be. Like I say, for Mender, the first thing, like on this board, we would probably get rid of that partition completely because the only things that are in there are the kernel and device tree. We'd move them into the root file system and Uboot's perfectly capable of reading that root file system. Yes? Yeah, so that's another good point. I know there were some boot loaders, at least way back in the day, that didn't know how to read EXT4, so you would have to use fat. But I think anything you're using these days is gonna understand EXT4. So I may be wrong, and I'm sure there's... Res, res, res, res. Fair enough, fair enough. Yes, so on Raspberry Pi specifically, yeah, it's because their proprietary bootloader only knows fat. Yes? Okay, yeah, so basically, there are some hardware restrictions where that's effectively required because of memory remapping and things like that. Anybody else? All right, well, everybody have a nice lunch and don't forget, if you want a T-shirt, feel free to grab it on the way out.