 Hi, and welcome to this session about practical experience with Linux AB upgrades. My name is Lona Navin, I'm a senior software engineer at Consulco Group and an open source enthusiast. Consulco Group is a services company specializing in embedded Linux and open source software. My colleagues and I have experienced hardware and software build design development and training services. Furthermore, we have numerous upstream contributions through the years to very popular open source projects, including U-boot, the Linux kernel, Yocto, Open Embedded, BuildRoot, Automotive Grade Linux and a lot of solutions for software over-the-air upgrades, actually the core topic for this presentation. Consulco Group is based in San Jose, California with engineering presence worldwide. I have the privilege to be working remotely from my beautiful hometown of Pueblo, Bulgaria. Because of this, this talk is pre-recorded, so please feel free to ask questions in the chat while you're watching it. The agenda of this talk focuses on several topics. First, we'll start with discussing strategies and open source solutions for updating embedded Linux devices. For the past several years I have been actively using the Yocto project, so the majority of the examples will be about Yocto and Open Embedded. We'll focus on two particular solutions for big AV upgrades, Mender and Raug. We'll also discuss a combined strategy for integration containers with these AV upgrades. Finally, we'll wrap it up with some conclusions. There are many things that we should consider when we discuss software upgrades. Are there any limitations of the disk space? Are there any limitations of the network bandwidth? How to manage applications? Do we need containers? Do we need AB or binary data updates? How to upgrade? Should we do an upgrade over the air or should we do it using a cable or a USB stick? Although the majority of devices nowadays are connected devices, Internet of Things, there are still some use cases where either the device is not connected to a network or you don't have direct physical access to the network. So instead someone should go physically with a USB stick to the device, plug the USB stick and upgrade it. Also we should consider that some devices have stable Internet connection pretty much all the time, but other devices, for example wearable devices or vehicles are moving and may lose temporarily Internet connection. In this case, there should be a mechanism to retry updating it. Another question to ask is the device mission critical because very often after an upgrade a reboot is required and this reboot might have a very negative impact if it's done in a bad moment. More things to consider about software upgrades are the technologies we are using. What kind of build system are we going to use? If it's the Octo project, do we have board support packages for it? And so on. Last but not least, how to update a fleet of devices. Not just one device, but many devices and some companies have millions of devices to upgrade. In this case, you need reliable server software to distribute the upgrades to make sure that these upgrades are applied if something goes wrong to try upgrading the device again. There are several different embedded Linux update strategies, all of them are common because depending on the use case, different developers decide to use different strategies. First of all, we'll start with the AB updates. In this case, there is a dual renderer scheme with A and B partitions which are identical. The next common strategy are delta updates. Very interesting because a very small portion only, the delta of the difference between the old and new version is distributed. This saves bandwidth, however, it's a little bit tricky if the device is not working properly after the update as the fallback sometimes might not be possible if the device is not even booting after the update. Another strategy which is quite popular and actually is coming from the world of servers where containers are extremely popular nowadays, this is the container-based updates. Last but not least, nowadays, we see more and more people using the combined strategy where you have a small embedded Linux distribution with AB updates of the base distribution and containers running on top of it. We'll discuss this in more details in the slides later on. Let's start with the AB upgrades. Actually this is the main focus of the presentation. So we have a dual AB identical root FS partitions, partition A and partition B. The idea is that the device is running with one of the partitions as active. When there is an update, the client application which runs on a better device periodically checks to the server for an update. If there is a new software update available, the client downloads this update, installs it to the B partition and after that, using the bootloader, it switches to this B partition. If everything is okay, it uses this partition on which we have installed the upgrade. Otherwise, there is a fallback procedure which may return to the other partition which is known as a good partition and known to work. Sometimes having just AB partitions is not enough because the device may need to and actually most of the time needs to store persistent data which is left unchanged during the update process. Because of this, open source solutions for the AB upgrades also support as a feature to have a data partition. This data partition is left unchanged when you do updates. It's very convenient for storing, for example, configurations which are specific for the device. Let's discuss a little bit more the combined strategy that we had already mentioned. The container technology has changed the way application developers interact with the cloud and some of the good practices are nowadays applied to the development workflow for embedded devices and Internet of Things. Containers make applications faster to deploy, easier to update and more secure through isolation. Furthermore, nowadays, some of the embedded Linux devices are quite powerful. So a lot of companies prefer to have base distribution which performs AB updates and on top of it to have several containers for example, a dedicated container for each application. As you know, application development is quite different from platform development. So in majority of the cases, different teams are developing the applications and the platforms. Application developers are nowadays used to containers because they can have a container only for their application to make sure that this application is isolated from our application and won't interfere with them. This is good for security as well. And also application developers can ensure that the container has all dependency their application needs. And this means that you can perform management of the container and AB upgrades of the base operating system. Because of this combined strategy, it's actually something that I have implemented on several occasions for customers of Consulco Group and it's definitely something worth considering for many, many use cases. Let's have a look at the popular open source solutions for updates. We'll start with the two solutions that we will explore in more details in this presentation. Mender and Raug, both of them are for AB upgrades. However, Mender also supports Delta updates. Mender and Raug are not the only solution for the AB upgrades. SW update is also quite popular. There are solutions for container-based, entirely container-based upgrades such as Belenna or SNAP. There's also a whole group of solutions based on a project called LiboS3, previously known as OS3, which is doing binary Delta diffs of the file system and updates through it. Actualizer lines are applications which interact with LiboS3 to manage updates. For several years, as part of my work at Consulco Group, we were helping ATS Advanced Elemental Systems with the integration of OS3 in Yocto Open Embedded layers called MetaUpdater and their whole solution for binary Delta updates with ATS Garage. Also, technologies such as Qt OTA and Torizen by TorbEx are relying on LiboS3 for doing upgrades over the air with using just binary Delta. Another important question that we've mentioned is the built framework. So, how are you going to build your custom embedded distribution? Nowadays, there are a lot of open source built frameworks that do excellent job. Let's start with Yocto and Open Embedded. I've mentioned that I've been using it for maybe seven years now. So, this is my preferred choice, but also there are excellent alternative solutions such as BuildRoot, PDXDIS, or OpenWRT, which is quite popular specifically for rounders. Of course, there are other solutions as well. And I'm sure that someone will ask the question, can I just use Debian? Yes, of course. Debian is a stable full distribution with tens of thousands of packages available as binary files for installation without the need to cross-compile from source. Debian provides pre-compiled files for most common architectures, including X8664 ARM, ALAIS RISK 5. There are numerous Debian derivatives for embedded devices. Probably the most popular one is the Raspberry Pi OS, which was previously known as Rasbian. I'm also a huge fan of Armbian, which is a distribution, a Debian-based distribution for various single board computers, popular among makers. I know the maintainer Igor, who is doing a great job. The whole topic about using Debian on a better device is a very large. It totally makes sense for makers, for industrial projects. And Chris Siemens had a great talk a couple of years ago comparing Debian to the October project. I won't go into any details about this because it's a huge topic and we don't have time for it. But if this is something you are interested in, I highly recommend you to have a look at Chris's talk, which is recorded and it's available at YouTube. Now, I've mentioned that I have a lot of experience with the Yocto project. I also have upstream contributions to BuildRoot, but Yocto and OpenEmbedded is my preferred tool for building a Linux distribution. The Yocto project is a product of Linux Foundation. It is a collaborative project for creating a custom Linux-based systems for embedded devices using the OpenEmbedded build system. OpenEmbedded build system includes BitBake and OpenEmbedded core. Pocky is a reference distribution of the Yocto project, provided as metadata without any binary files to bootstrap your own distribution for embedded devices. The Yocto project has a B-annual release cycle twice a year. There's a new release. And last year, the Yocto project introduced the so-called long-term support LTS releases covering two-year period. The first LTS release is Dunfel. This is version 3.1 of the Yocto project. It was released in April 2020. And I have to say that all the examples in this presentation are based on Dunfel. And in the next slide I'll explain why this is important. The next release of the Yocto project is scheduled for October, which means next month. And it's going to be version 3.4, Honestor. This release brings a major change in the override syntax of Yocto and OpenEmbedded. So the character that we use will be replaced. Actually, it has been already replaced in all these main branches. And if you want to give it a try, you can try it right now. In October, with the release of version 3.4, it will be officially available. The documentation of the Yocto project has been already updated to cover details about this change. So here you can see a very simple example, how image install now appends Docker to your image. This is an example that I picked up because of the combined strategy that I mentioned previously. If you have already existing players, which are not yet converted to the new syntax, OECore provides a convenient script, which can help you to migrate to the new version. And keep in mind that although this syntax is not available in Dunfel, the current LTS version, it will most probably make it to the next LTS version of the Yocto project. So it's something worth considering. I repeat that this presentation and the examples in the following slides are based on Dunfel, so we'll be using the odd override syntax. Now let's focus on the AB upgrade solutions in more details. First we're going to start with Mender. Mender is available as free open source or paid commercial and enterprise plans. AB updates seems are for the open source use for the community and they're available for all plans offered by Mender. There's also a delta upgrades for professional and enterprise plans in Mender. So although Mender is popular among the community for its AB upgrades, it's technically possible to do delta updates if you subscribe for a professional or enterprise plan. However, in this presentation I will focus on the open source features of Mender with AB updates. Furthermore, Mender provides a back end service. This is the host of Mender, which you can install on a server to help you manage your embedded devices and their updates. Mender is written in the programming languages Go, Python, and a little bit of JavaScript of course. Can you live without JavaScript nowadays? The Yocto and Open Embedded Integration for Mender is done through the Yocto layer, MetaMender, and there are extra BSP layers for the various supported boards. The source code is available in GitHub under Apache 2 license. Here is a short list of some of the supported Mender devices. These are not all of them, but of course some of the most popular. Starting with Raspberry Pi, my favorite single board computer, and probably the most popular single board computer in the world, followed by the open source BeagleBone single board computers. There is support for Intel x86-64 devices, Rockchip, O-Winner, NXP, IMX6, IMX8, and many more. To learn actually all the supported devices, have a look at the MetaMender community layer, where you can find sub-layers for the various supported devices. For example, here we have MetaMender Raspberry Pi. Once again, I repeat, the examples here are based on the LTS release. Dumpo of the Yocto project and Open Embedded. Mender AB updates support two client modes. The first one is the managed, which is the default client mode. The client is running as a demon, puts from the server for updates. This is happening automatically. The second option is for standalone updates, which are triggered locally. This is suitable for physical media. For example, if you need to do an update with a USB stick, you can write the Mender artifact on the USB stick, plug it on the device, and if you have created, of course, previously, a UDAF rule. The UDAF rule can trigger, can detect the USB stick, and can trigger a script, which will apply the update using the same steps as explained here. So the first part of this example is that in local.conf we have to disable the Mender client, which by default is automatically enabled. Then we need to run, this example is for over-year update. Then we need to run a simple HTTP script. I'm using Python in this case, and on the embedded device, we need to run Mender installed, followed by the URL where we downloaded. And of course, instead of a URL, if we're doing the example that I've mentioned with the USB stick, instead, we should use the path to the file. Now let's have a look at the Mender data partition. As I said, by default, we have A and B partition, but the information on these partitions is wiped out each time when we apply an upgrade. So in order to keep some information persistent through updates, Mender has a specific directory, a whole partition, data partition. It stores persistent data preserved during Mender updates. The Mender client itself, which is running on the embedded devices, uses slash data slash Mender to preserve data and state across updates for its own usage. The size of the data partition is defined by a specific variable introduced by Mender. This is the Mender data part size mb, which configures the size of the data partition. By default, it is 128 megabytes. However, if this is enabled, MenderGrowFS data tries to resize on first boot the remaining free space for the data partition. However, keep in mind that MenderGrowFS data relies on SystemDGrowFS, so this is particularly for Linux distributions relying on SystemD. And it is also possible to create an image for the data partition in advance. This means that BitBake will produce this data image, and you can put out of the box some information. In order to achieve this, you need to apply the following configuration to ImageFS. The keyword is data image, and the Mender classes will take care of it, and BitBake will generate this partition for you. This is particularly useful because sometimes you really need to have something on this data partition as soon as the board boots. Let's have a look at the steps to install Mender update on embedded device. So first, you apply the update. After that, you do a reboot. On the first boot, after a successful update, the Mender client will commit the update. This step for committing the update means that it verifies that the device has booted successfully. It is working fine, and by doing the commit, you are saving the configuration, meaning that you are confirming that the update is successful, and on follow-up boots, this partition will be used. If the commit is not done on the next reboot, the old partition will be applied. In the setup where you are performing manual upgrades, don't forget to do the Mender commit step on your own. Now, a very interesting feature about Mender is the so-called MenderSingleFile artifact. This is a deployment of a single file directory or even a container image. This is particularly convenient for application updates. Very often people ask, okay, we're doing this AB updates, and if we have a big partition, this means that we are downloading the whole partition, which is a reasonable file in terms of this size. What happens if we just forget one file or something? The answer is Mender has this feature to deploy a single file. You can create a Mender artifact containing this file and apply it as an upgrade. However, of course, the root file system must be read-write, because later on you see that it's also possible to make these AB partitions read-only. So, yeah, this is an option in Mender, which is quite convenient and covers some critical use cases by saving you the procedure to do a whole system upgrade, and instead to deploy a single file. Let's have a look at some of the interesting Mender add-ons. Mender supports several of them. First of all, remote terminal. This is an interactive cell sessions with full terminal emulation. There's also an option for file transfer. This is again an add-on, which, as the name suggests, allows you to upload and download files to and from an embedded device on which Mender is working. Another interesting add-on is the port forwarding. You can forward any local port to a port on a device without opening ports actually on it. And the Mender configure add-on is for applying configurations to your devices through an uniform interface. So, Mender is not just the AB upgrades, but it has all these add-ons. You have the ability to create your own add-ons hooks and to constantly expand the features of Mender. I know that a lot of projects require some specific attention, because they have a very, very specific use case. And in these use cases, add-ons can be either used, any of these add-ons can be used, or eventually new add-ons can be can be developed. In 2018, Mender added support for Intel and AMD X8664 machines through the Group Bootwalder, because for all our devices, for the ARM devices, Mender is relying on Uboot. So for Intel X8664, we have group support. I have to say that in the embedded world, the majority of the devices as of the moment, based on my experience, are ARM. However, Intel also provides interesting solutions, and Consubo has some customers running embedded devices with Intel, for which we are using Mender with X8664 support. Over time, initially, there were some issues which were resolved, and nowadays, Mender works very, very stable on these devices. Here is a screenshot of a Github pull request that I have created to fix Mender initial installation from a USB on machines with BIOS using the same Mender installation script as for FE. So the whole thing behind this Github pull request is that the initial installation of a distribution on an X8664 machine is most commonly done using a live image on a USB stick, the very first time when you populate the drive of the machine. Once that's done, once you're ready with the initial installation, you can proceed with AB upgrades, just like as we discussed with Mender. Let's have a look at the other solution for AB upgrades, which is quite popular. It's called Raoq. Raoq is a lightweight update client that runs on an embedded Linux device and reliably controls the procedure for updating the device with a new firmware revision. Raoq supports multiple updates scenarios. It provides tools for the build system to create, inspect, and modify update bundles. It uses cryptography to sign update bundles. One of the advantages of Raoq is that it's compatible not only with the Yocto project, but also with PDX disk and build route. Raoq is available under several different licenses because it has several different components. Raoq itself is all of these repositories for Raoq are available at Github. There is a Raoq organization. Raoq itself is available under LGPL license. Matter of Raoq, this is the Yocto and OpenEmbedded integration layer for Raoq. It's available under an MIT license. Raoq Hopebit Updater are available also under LGPL licenses. In order to integrate Raoq on your embedded device and in your embedded Linux distribution, there are several things that you should do. First of all, you should select an appropriate boot order. For ARM devices, for example, it's going to be Uboot and I have created several examples for this using different versions of Raspberry Pi single board computers. Next step is to enable SquashFS in the Linux kernel configurations as well as for a root file system because Raoq does not have any X2 or X3 file system type support. We need to create specific partitions that matches the Raoq slots. We need a slot for the A partition, a slot for the B partition, eventually a slot for the data partition. The data partition in Raoq is something that we're going to discuss in the next slide, but basically it has the same features as what we described in Mender. The idea is to have a data partition on which the information is persistent across updates. The whole thing about partitions has to be specified in terms of Yocto and OpenEmbedded in WKS files. This is the WIC kickstarter file. The next step is actually the most important and tricky step because we need to configure the bootloader environment and create a script to switch between the Raoq slots, switch between the A and the B slots when there is an update which has been applied. Last but not least another important step is to create a certificate and a keyring in the Raoq system conf file to verify the updates. Security is a very essential part of Raoq and because of this we need these certificates and keyrings described. Now let's have a look at the Raoq data partition. The interesting thing about Raoq is that it supports both single and redundant data partitions. For redundant data partitions, the active root FS slot has to mount the correct data partition dynamically, for example using UDefRoo. This makes it a little bit tricky. This means that you have an A partition and B partition for the root FS and you also have an A and B partitions for the data, so the root FS should be taking care about switching between them. Honestly, my preferred choice is to stay with a single data partition. This makes things easier and on the data partition. You can store valuable information which should remain on the device between updates. Last year in 2020 I did my contribution to meta Raoq by creating a layer called meta Raoq community. The idea of this layer is to provide examples of how to integrate Raoq on various machines. On the previous slide I showed you the list of steps and there's so many steps that you should do in order to integrate Raoq, so if you have an example it's always easier to get started. And of course the obvious choice for this first example was to use a Raspberry Pi. So I started this in 2020. I wrote a blog post which was posted on consugu.com. After that I had a talk about it, so please have a look at this talk if you want to learn more details about meta Raoq community. And for my surprise a lot of people were interested in this layer. There were so many contributions. Nowadays there are several active contributions. We have a branch for DUNFEL, but we also have a main branch which is following the recent changes in Yoctoprojected Open Embedded, including recently there was GitHub pull request for fixing the new override syntax introduced with the new releases of Yoctop. Also this summer we moved the meta Raoq community to the Raoq organization in GitHub, so now you can find it in one place. Keep in mind that meta Raoq community is something for providing examples. It's something that helps you get started more easily and faster. The great news again coming from this summer is that a contributor was kind enough to provide support for Raoq on all-winner devices, also known as SUGSI devices, through the BSP layer meta SUGSI. Of course for Raspberry Pi we are using meta Raspberry Pi. So if you are using Raoq on a device which is not currently supported in meta Raoq community, please consider contributing an example for the device and contributions are always welcome. The source code is in GitHub and GitHub pull requests are welcome. Keep in mind that October fast is coming, so it's a good time for making GitHub pull requests. Now let's have a quick look at Raoq example for Raspberry Pi 4. As I said, the majority of the actually all the integration steps that I've described previously are already done in this meta Raoq community layer that we have for meta Raspberry Pi. So meta Raoq community inside has a number of sub layers and there's a one sub layer called meta Raoq Raspberry Pi. We need to add all these layers to the bb layers of our build and then in local.conf we have to specify the machine. Here I'm using system D. I'm adding Raoq to the image X4 for the file system type as explained that Raoq requires X4 and I'm switching to Uboot. By default meta Raspberry Pi is using the standard Raspberry Pi boot loader but in our case we need Uboot because I have implemented a script which switches between the partition db partitions and Raoq relies on the script and we have the W kickstarter wks file which has been specifically created for the dual partition scheme for Raoq on Raspberry Pi. So once this is done and we can build an image for example core image base booted on the system then we can build an update bundle. The recipe for building the update bundle is also available in meta Raoq community and here is an example how to perform manual Raoq update on the Raspberry Pi. Here you can see how the boot loader works. Initially after first installation the A partition will be active. When we execute Raoq install the Raoq will install the downloaded file to the B partition and finally we will do a reboot. So Raoq will take with the assistance of Uboot will take care to select the B partition and if everything is okay our device will boot from the B partition. One more thing about Valid for both Raoq and Mender is read-only file system. This is something that's not specific for the AB update mechanism that you are using but instead it's something specific for the Yocto project and open embedded. There are two options how to create a read-only file system using BitBake. The first option if you have your own image which is the recommended way of course is just to add to its image features the root-only root FS as shown here. Alternatively for example if you just if you just debugging and want to do it quickly through local.conf you can use extra image features and again add the read-only root FS. This is described in more details in the documentation of the Yocto project and open embedded. Beware that there might be practice in the image that expect the root file system to be writable and might not function properly. So why is this happening? Well basically as a very active contributor to meta open embedded I have to say that when I do an update for when I prepare a patch for the upstream of meta open embedded I do proper testing on read-write root FS and certain applications which are wrapped as recipes require on the fly after for example after the first boot of the device to create certain files. So if the system is read-only they will not be able to create these files and there are different solutions to this problem if you run it into it and if you want to have a read-only root file system for your AB updates. One solution is to move these files and directors to the data partition. This is a solution that I actually like because most probably you want to keep these files and directors persistence across updates because if there is something if there are any user specific configurations you don't want to lose them. Another solution is to use overlays and something that's again interesting and can be applied for both Mender and Raug is a combined strategy with containers. So there is a yokto and open embedded layer called meta virtualization it provides support for building send KVM, libvirti of course docker and all associated packages necessary for constructing an old based virtualized solution in other words the dependencies of these technologies. In order to use meta virtualization of course you have to add it to your bb layers but also you have to add to your distro features virtualization. If you have your own distro you can do it there. Alternatively if you are looking for a quick solution you can just add with append the virtualization to the distro features in local.conf. So once you have the layer and the virtualization distro feature enabled you can install to the image for example docker and run bitbake to build it. So the idea here is that there are use cases on power for embedded devices where containers are combined with a b upgrades of the base Linux distribution built with yokto and open embedded. So meta virtualization is your friend if you're looking for such kind of a solution. So let's wrap it up with some conclusions. Obviously there are numerous things to consider when implementing an upgrade mechanism for an embedded device. There are many open source software solutions and I highly encourage you to use one of them. I know that a lot of companies and developers think hey we're so special that there's nothing out there that can help us. Even if this is true keep in mind that you can take an existing open source solution and add only the features that are missing there for your particular use cases and based on my experience I have to say that things like Mender and Raok have quite a lot of solutions and they're covering quite a lot of use cases. So most probably for the majority of use cases that you have they already have a solution. Mender and Raok are very powerful and good solutions for a b upgrades. They have excellent integration for yokto and open embedded. Furthermore Mender also has integration for Debian and Raok has integration for various other alternative build frameworks. Combined strategies for a b upgrades with containers are becoming more and more popular. They have a lot of advantages not only because of the isolation of the application which is good from security point of view but also because application developers can have better control of the dependencies and the container images. Real world implementations of a b upgrades very often require data partition for storing any persistent data which is left unchanged during the update process. Honestly all the implementations that I have created for a b upgrade included such data partition. Thank you very much for watching. I hope you've enjoyed this talk. Here are some useful links and let me know if you have any questions.