 So, welcome to this talk, I'm Michael Abdenacker, the founder of Butlin and working as an embedded Linux engineer now. Butlin is a software service company focusing on the low-level cool stuff about embedded Linux, like BSPs, build systems. So, what I like in this community is that I'm always happy to, I always can learn from you projects. That's nice. So, that's why I have this one. Yeah, it's nice to be in front of a real audience again. I see no cats, no people in pajamas, I think. Well, the bad thing is that I don't have the ability to change t-shirts between every slide to keep you focused. But you're all locked in with me, so no way to escape. Good. By the way, by Butlin, we have lots of training materials that are freely available. One of the authors of some of those, I'm also the co-mentainer, the maintainer for the Yocto project documentation. So, please send us patches. Also, I maintain the Elixir cross-referencer, not the language, but the tool that you use to browse the Linux kernel, the Uboot sources and all the other ones, many ones like C and C++ projects. That's the abstract, just like for people reading it online. So, actually, this is not a very complex project here. I'm just sharing the knowledge. We had the opportunity to explore system update technologies to prepare slides and lab data for a customer project, for a customer seminar. And we wanted to let participants practice with A-B software updates using the software update solution. And the theory was not so complicated to find, but it was a bit harder to find which practicals these days you could use, you could follow on the Uboot side more than on the Linux side, but how to implement that took a bit more research too. I'm happy to share that with you through this presentation. So here, it's just like one of the cases, we are talking about full image update approaches. So you want to, when you deploy an update to a device, you want to just reflash the device with a full update of the system. This guarantees that what you have on the device is fully tested and it's consistent. Like you don't have like file-by-the-pile updates like as you would have in packages. And of course, with this approach, you can't update the system while it's running. So that's why you need like two partitions, two different partitions to boot on. You have like two cases, you have the A-B scheme when you have two full copies of the root file system and you switch between them, like one is active, one is not active. And another way is to use the rescue system. So you have like a smaller root partition. You can boot on when you want to make, to flash an update, right? And to flash the full copy of the update. But you still alternate between those two and that's interesting. You need some bootloader integration to switch between systems and I'm just sharing the practical details about those. So the typical way you apply the system update is to, you have the system that runs on, like say on active copy A. When you find an update, either local update, like you plug in a USB key or something or an SD card or over the network, the update image will be flashed, written to the copy B, the other partition, that's inactive so far. And if flashing is successfully applied, then you need to update the bootloader configuration so that to reboot the new update, right? Is used by the bootloader, right? And you want to mark the updated copy as the active copy now. So the system will reboot on that active new active copy will try to successfully boot. If it doesn't work, then you can use a watchdog to detect that and reboot and fall back to an old copy if necessary. And there are also a few tricks to know how to detect the fail update. So the main practical questions here that I had to solve is how to identify the active partition. So from new boot first to boot on the right version of the system so that right at boot time, but also from Linux when the data is meant to be deployed, of course deployed in the right partition, not the active one, of course, otherwise that would be an issue. And then another question was how to swap the active and inactive partitions from Linux right after an update. And also from new boot when you fail to boot on the new update you want to fall back to the previous one. So that has to be done on new boot. And how to fall back to the original version too if the update fails to boot properly, like how to detect that and how to fall back. So here we are going to address those questions in a generic way. I know there are bootloader integration solutions in the generic tools that you can have like software, update, route, and vendor to do this. And here that's like solution agnostic. It's like the bare Linux and bare Uboot. So how to switch between root file systems? So one other techniques to implement this is to specify the active partition through adding bootable flag to that partition that's supported by the partition tables to have boot flags. Sometimes you use this to make sure that the ROM code boots to the right. Finds the bootloader there. Depends on the, on USOC. So this is typically done to find out which the active partition is. After, well, to toggle the active partitions. So after an update is applied successfully, on Linux you will have like to do that if you detect that that was successful. So if you're using a GPT partition table, you can use the SGDisk command right here. That will, assuming you have two, one active partition and another one that's not active, it will toggle the partitions. Like it will change the state of partition two and change the state of partition four and partition five. Like it just switches alternate between them. So once it's no longer bootable and the new one that becomes bootable. If you have a NBR legacy partition table, I just found a command to make one partition bootable like the parted command. Sorry. No, you can't use. There's a mistake here because it's talking about, oh yes, yeah, parted works, sorry. Parted works support both NBR and GPT. So with parted you can actually turn on the bootable flag on one partition and all the other ones who would lose their bootable flags if they had one, right? You can't use the SGDisk command here because it would convert your NBR partition to the GPT format. And then that could be an issue if your ROM code doesn't support this kind of flag. If it just supports NBR, like the OMAP SOCs, for example, at least the ones I know, like the big one on the big black, yeah, you won't be able to boot anymore because the ROM code won't recognize this flag. It's partition type. And then before you actually apply the update, you should also detect which is from Linux, you should detect which is the current partition that has the bootable flag. So this is a command you can use. If you use a GPT partition table, you use the SGDisk command I'm proposing here. So that's the bit attribute two of the partition, partition four is in this example. So if it said you get zero, one, and otherwise it's zero. And if you have an NBR partition table, you can use this short command as well that will return the number of the partition which is marked as bootable. So it's very simple practical details, but I hope to save you the research. And then when you boot, reboot, right? You boot should be able to detect which one the bootable partition is. So what I found is that you need to compile your boot with the part list, the part command. And then you can, using this configuration flag, and then you can rent part list, MMC, the MMC device you want to use, like MC zero, which could be like the external SD card, and dash bootable in the name of a variable that will receive the output like, which is the partition with the bootable flag. So at the end, you get the boot part variable that contains that information. And then you can use that in the subsequent scripts that will boot correctly. So it would look like this. The first solution, and then, yeah. So you, from this information, you can load the right kernel, you can load the right DTB, but you also need to adjust the bootaxe to boot on that root flag system. And there's a nice trick in your boot that's called account fix bootaxe subst in case you don't know it. That allows to have a substituable part in bootaxe. So here, what we're gonna do is use this boot part in a variable directly in our bootaxe. So it would be root equal dev MMC block zero p dot a boot part. You could also make this up if you don't have this feature in your boot, just like sometimes boot part bootaxe at run at boot time, set it to some shared bootaxe plus the boot part definition that's also possible. But it's more straightforward. So I just didn't know about this possibility and I find it cool. And then you would set the boot command like this. So part is, of course, to find the bootable partition first, load the kernel from the right partition, load the DTB from the same, the right partition like the currently enabled one, and then you can boot, right? So pretty simple here. There's another technique you can use as well to boot on the right partition with the corresponding bootaxe that you set, the kernel command line that you set for the corresponding partition. You could actually compile Uboot for support for the X Linux way that's nice for distributions typically. So compile Linux, Uboot with CMD ssboot and this way, like the ssboot command, this way Uboot can load the kernel, the DTB, initial MFS if you have one, plus the kernel command line from a specification in file that's in the partition, that's in the active partition, like xlinux.com. And this works on a lot only for local booting like USB MMC, but also through the network. That's part of like PXC boot and DTB. So it's nice because with this access Linux option, you probably saw that, not necessarily without knowing what implemented it, it's you've got a list of configurations to boot on, each one corresponding to an xlinux.conf file. So that's nice. You could have a recovery option here if you want as well. So there's a bit of documentation in Uboot about that. So here's what the configuration looks like when you're using xlinux. So you have a label that shows in the menu of boot time. You've got the path to the image file, the compressed kernel, a path to the device tree. So that can be specific to that partition if you want. And then the kernel command line as well. So this could also be specific to your partition or something generate like here that you build from your build system that never changes. And that just makes a reference to the boot part variable that you figured out before in Uboot. So that's a fixed content here with a variable part. And this way you can boot on the right stuff with the right arguments. So no need to tweak this file from your build system, just build the same thing always. But though you could tweak it if you really wanted, you could tweak it. And then to make xlinux work, you need a bunch of environment variables to be set as well because it needs to know where to load the kernel, where to load the DTB. So you have like kernel address in RAM. The RAM disk address in RAM, if you're using an intramfs, you've got the DTB address like the flatten device tree, now the device tree binary. There's the PXC address in RAM where you would load the configuration like the xlinux.conf file. In boot file, that's the path to the configuration file that you can customize as well. So by default it's slash boot xlinux.conf, but it could be something else. And then to make it work, you will typically, at least in my experiment, I had to run part list to find out the currently active partition, right? And then run sysboot, mnc, the device, the partition I want to use, the active partition. And any is a parameter that represents the file system you want to use. That's part of sysboot. You can specify x2, x4, maybe if you try it, I don't remember. Actually, this could be simpler than that. If you look at Uboot's documentation, it could be made to automatically read xlinux.conf from the current bootable flag. So it could actually look it up by itself and then use that information to load the configuration file from the right partition. At least, this didn't seem to work from a Samify D3 explained for microchip. Maybe there was an issue between the keyboard and chair, but yeah, I had to go the more complex way and this just worked fine, but maybe it can even be shorter. Just mark a partition as bootable and configure Uboot to use xlinux and automatically pick up the right partition, maybe. Not sure yet. Next question is how to recover from a fail update. So if you have a Navy Partition Scheme, there are several cases. So when the update fails to apply, that's easy to manage, that fails, so nothing to do. You detect that and you stick to the current root file system until you get a new update that would correct that. If that's successful, the thing that you have to do is to make sure that you have a new update of course, boot to the new one, but also confirm that the new root file system also works correctly, and otherwise you fall back to the older root partition. If you just have a single root file system and a rescue file system, if the update fails, the system may not boot at all. So you need a mechanism in the boot loader to fall back to the rescue file system and wait for a new update. So the same mechanisms will apply. So there's a nice mechanism that I wasn't aware of in Uboot to detect the failed boot attempts. So you enable this through config boot count limit. And if this is on, then after an update is applied, like from the script, from software update, whatever you do, you should set the Uboot environment variables like upgrade available to one. So you do that from Linux. And boot count to zero. I'll show you how to do that from Linux. So when Uboot starts, when Uboot boots, it checks the value of upgrade available. If it's non-zero, then it will increase boot count and try to boot. And well, except that if boot limit is defined as another variable, that's like the number of boots that you accept before considering that the update will fail to boot. If it's greater than that limit, Uboot will actually run commands that are defined in another boot sequence like not boot CMD. That's the default behavior. It will be boot, out of boot CMD. So that's where that's a sequence of commands you can run to fall back to the other partition. So how does it work? When Linux booted successfully on the new file system, you can have a user space application taking care of resetting upgrade available to zero. So we consider that everything is fine and boot count back to zero, right? And then the next time you boot, there's nothing to do. Uboot won't even count the booting. It's considered as fine. Otherwise, the system will be rebooted either manually because the user sees there's a user there that sees that doesn't work or automatically if you have a watchdog to do that. And then boot count will increase over and over again after every boot until the boot limit is reached. By the way, so I told you about boot limit, sorry, boot count. By default, it's saved in an environment variable, but it's also possible. That may be easier in other non-volatile sources like on an SQLC device. If you have a nice RTC, you could store that information in RTC. That's what they say in the Uboot configuration. In an E-prom on a SPI flash, even on a file on the next two file system, or X4, in RAM, if you have an area that's persistent across resets. So there's a bit of documentation in readme.boot count in the Uboot documentation. I think I remember I contributed to that like to explicit those things. So just some practical details here as well. How to update the upgraded variable and boot count variable from Linux after applying an update to make sure that the boot count mechanism is on. Or to disable it when the latest update booted successfully to completion. You can use actually some scripts or some commands that are provided by Uboot. It's called firmware print as and firmware set as. That's available in the Uboot source code. So you can compile them either manually, right? Or using your build system like Yocto or BitRoot. So Yocto, it's like the lib Uboot end for CP. And there's a particular option in BitRoot to that as well. Yeah, so if you want to know more, you have like details on the documentation as well. And just quick details in case you haven't never used a watchdog yet. What you would do is you can figure Uboot and enable watchdog on your platform at boot time. That's up to the boot loader to do that. And when Linux is booted, well, you don't forget to feed the dog. So regularly you write to slash dev slash what dog. And this proves that your system is still alive. So yeah, run a regular process, a periodic process to do that. Or eventually if you consider that you booted far enough just turn the watchdog off and then you don't have to do that anymore. Otherwise, yeah, the hardware, if it's not fed, the watchdog will get crazy. We will boot the machine and the bootcamp will be increased. There's a reference to the watchdog API in the current documentation for details. So essentially you write to a character device that's pretty easy to do. Now, after you make, you have an access for boot, there you need to fall back to switch the active and active partitions. So I'm giving you some, I found some commands in your boot to do that, to edit the partitions. So there's the NBR command in your boot, like that was new to me as well. It's, you find it in config command NBR. And that NBR takes a command, takes a specification, like those, like all the partitions you have. And one of the partitions can be, given the bootable flag here. And all the other flags are like the type of partition so that was like that, if I remember correctly. And then the other ones were Linux type if you remember those from FDisk. And then you use the NBR write MMC and number of MMC device and that will take the specification and write the changes, commit the changes. And the last thing you can do is run NBR, verify, to make sure that the actual partition layout is in sync with the specification. Or you could do that first, if it's not necessary to change then don't change it. For GPT partitions you have similar commands that's called the GPT write and GPT verify. So why did we use partition flags? You may wonder. So the first reason is that that's what the distro boot functionality of your boot is supposed to use to rely on by default. So that's why we chose that initially. But another reason to, even if you're not using distro boot is that the environment, saving the environment is far from being atomic. Like the environment can be like 10 pages big or something like that. So you have like multiple storage blocks to shuffle to modify as many things could go wrong. So it's better to have make a change that as small as possible, like just toggling a bit in the partition table. Even if of course on MMC it implies rewriting one page, of course. So but you could well find another non-verbal type of storage on your SOC or on your board to store such information. So if you have any clues, let me know. That would be interesting. So some limitations about this approach we're getting close to the end. We have only addressed here systems that are booted from block devices and that's the most common type today it seems. If you're booting from NAND flash or NOR flash, you could store the partition. You could store the partition, the flash partition to use on a specific non-volatile storage somewhere like on one of the flash sectors if you want. And I'm not sure, another limitation of the approach here, I'm not sure whether you boot or even the Linux commands will provide you with a domestic warranty. So some things could be wrong when you applied the update and if you lose the power during the update and think during the toggle, you are not immune to issues here. So at least a good idea could be to use U-boots redundant environment features. So if one of the two copies of the environment is corrupted, U-boot can still use the other one. So it's not only on flash storage that is the most common, that's where it's commonly used but you could also have it on block storage, have two copies of the environment and U-boot will fall back on the, will have a valid environment to boot from. Because here we're using a shared environment from the two updates, the two versions of the file system use the same environment. And then the next question that we haven't addressed either is how to update the boot loader. So yeah, you need to configure SPL, the SPL, the first stage boot loader to load U-boot from the file system so that each file system like A and B could have like different U-boots, that would work fine. But anyway, it's gonna be probably difficult to implement the partition selection and the boot counting approach in the SPL because the space is limited and you may not have this feature in the SPL. So that's not trivial to implement. May not be trivial. So if you're booting from MNC, one thing you could do is flash the SPL and U-boot on a special boot zero and boot one hardware partitions. For example, on Bigable Black, you have those. You have an MMC with those. If you boot NX on those and check slash proc slash partitions, you'll see that those exist. They are defined in hardware. You can't choose the tweet of those partitions but it corresponds to special parts in the MMC that correspond to more reliable storage and maybe faster as well. So that's good to put U-boot and SPL on those. So there's actually a command that's called MMC partconf in U-boot. That's, you can follow the documentation link if you want here. That tells U-boot to use that boot zero or boot one or just a regular storage if you want. So that you can specify after an update to say I want to boot on boot zero or I want to boot on boot one. So that's possible. But you need to, your SOC to support this mechanism like finding which one. So like reading that information on the MMC, which one is the active one. And this, I don't know this. This is probably highly SOC specific but you guys may have some insights about this. So I'll be much interested if you have some. And other ways, it's not advisable to update the SPL on the field. Like keep it on a shared place and find a way to jump to the correct partition from U-boot SPL. But that's probably more challenging. So ultimately, it's probably safer not to update the bootloader on the field and just do that when you get a product back to the lab and you can do that under supervision from your engineers, your technicians. They can handle that of course. So what to remember here, the new, the main new things I learned about U-boot here is that you have a partless command to find the boot part, the block partition that has the bootable flag on SD card or on EMMC. You have NBR and GPT partitions to edit the partitions from U-boot on block devices. And U-boot indeed can keep track of boot failures and you can count the boot failures. So using the boot count environment variable and in that case, you can run an alternative boot sequence through the alt-boot-tmd variable when the maximum is exceeded and that's the boot limit. That's the most things to remember in case you were not aware that those commands existed in U-boot. So thanks to my friend Slava for helping me and giving you good insights, reviewing the slides. Thanks again. For Thomas Petazzoni, for the blog post he wrote on applying, using software update on STM32MP1 so that you can read that and have a practical implementation of that. And my colleague Karim Encent as well for sharing the experience on this topic. All right, quick references here. There's a great presentation. If you're like, there has been various presentations on software update techniques at this conference. And there's another good one at ELC in Austin last time from Leon Navi from Consulco. So that's, there's a good reference, good and exhaustive reference. And a few, there's a few booting blog posts related to using the boot mechanisms such as software update, ROK and Mandarin. We have tried them all, so that's nice. And you won't say have on the in the next weekie, immediate next weekie a collection of talks about firmware updates, like how to replace your devices. So well, that's it. So let's, we have like 10 minutes for discussions, for feedback, like if you have our answers to some of the questions of asking myself, that's my email address. You can find me on Fastodon, I don't have in a, on Macedon. I don't have enough followers yet. So if you have a Macedon account and you're interested, you're welcome. And the slides are available at this URL, both the PDF and the sources, if you want to reuse the sources for yourself or for presentations, they are available in latex format and under free documentation license. Thank you. Thank you.