 Hello, and welcome to this session. How to choose a software update mechanism for embedded Linux devices? My name is Leona Navi, and I'm a senior software engineer at Consulco Group. Consulco Group is a services company specializing in embedded Linux and open source software. My colleagues and I have upstream contributions to various popular open source projects, including the Yocto project, Open Embedded, Automotive Grade Linux, the Linux kernel, Uboot, various open source solutions for software updates, and many more. The company provides hardware and software build design development and training services. Consulco Group is based in San Jose, California, with engineering presence worldwide, and I'm working remotely from Povde, Bulgaria. In the next about 30 minutes, we'll talk about the following topics related to updating embedded Linux devices. First, we will brainstorm the things to consider when we're picking up a solution. After that, we'll have a look at the common embedded Linux update strategies and we'll have a look at the overview of the open source solutions available that implement these strategies. Then we'll have some practical examples using the Yocto project and Open Embedded with several of the solutions for performing updates. And finally, we'll wrap it up with conclusions. So let's get started with the first thing to consider when we're speaking about software updates of embedded devices, the update size. Are there any limitations of the disk space, the storage on the device? Several years ago, we were used as engineers that hardware is in general available and that over time the price of the hardware most probably will go down. Things change completely because right now we are in a middle of a global chip shortage. It's very hard to find microcontrollers. It's very hard to find memory. It's very hard to find pretty much any components. So disk space is becoming a real issue. The second question is are there any limitations of the network bandwidth for the data transfer? This is particularly important if you're working on a solution that relies on mobile data to download the artifact to the device. As part of the preparation for this presentation, I visited cable.couk and according to this website in the US the average cost for mobile data is 3.33 US dollars per 1 gigabyte. Of course, if you have a better contract, you may have significantly better rates. But still, if you rely on mobile data to perform updates, this is not for free and there is some cost. The bigger the size of the update is, the greater the cost is. Another thing to consider is how you're gonna transfer the data to the device. What are the different ways of transferring updates? We can do over the air using Wi-Fi or mobile data. We can do it with an Ethernet cable. We can do it even the old-fashioned way with a USB stick. Yeah, still for some industrial setups where we have machines working together in a local network that's not connected to the internet. Sometimes there is a use case where a technician has to go, plug a USB stick and perform the update. There are other options to do an update which are less common, but it could be possible to perform an update over Bluetooth or even over serial. So there are a lot of options how to transfer the data and before you choose the solution for update of your embellishing device, you have to think over what is the exact use case for you, how you're gonna install this, if it's just over the air or Ethernet cable or USB stick or something else. Another thing to consider is how you're gonna manage the devices. Do you need to update a single device or multiple devices? Are all devices using the same software stack? Well, this is pretty interesting because sometimes you can have pretty much the same image built for different architectures. It can be built for x8664 machine and ARM machine. As a result, you will end up with two different binary images and different binary artifacts for updating these images. Are all devices online all the time? Do you need to monitor the update of the device and get some feedback to see which are devices are updated? Do you need to update different devices at different times? This is particularly interesting for consumer products which are used in different worldwide and used in different time zones. For example, it's common that these type of devices are updated at night. However, in different time zones, night is a different time. Do you need to do something like this and update the devices depending on the time zone in which they are? Also, how to update and manage fleet of devices if you have gazillion of devices? For example, if there is some kind of an internet of things that you need to update and all these questions are not only related to the embedded device, but also to the backend, to the management system that's performing the update. Nowadays, this system could be in the cloud, so we might be even talking about web development in this particular scenario. The next thing to consider is how are you going to build images? Most probably you have already started building images with some kind of a build system. So what distribution and build system do you use? Is there a board support package for the hardware you use? Is the software update technology compatible with the build system and the BSP? These questions are very important because if there is no support for the solution to perform the update in your build system, this means that you have to spend time and money to integrate it. It's pretty much the same with the board support package. However, most of the open source software solutions for performing updates already have integrations with a lot of hardware. So even if you need to do some kind of an integration of the BSP, it's significantly easier compared to integrating the whole software over the year solution in a build system if it's not already integrated there. And speaking about build systems, what are the options out there? Here is a list of the, in my opinion, popular open source build systems for custom embedded Linux distributions. We're starting with Yocto project and open embedded. I don't want to lie to you. This is my preferred choice and I'm using it on a daily basis. Build root is another excellent build system. I have upstream contributions to build root as well. PDX disk is an alternative. OpenWRT is something designed primary for building images for wireless routers, but it can be adjusted for embedded devices as well. And there are other options as build systems on the market too. A common question is, can I just use Debian? It could be Debian, OpenSUSE, Fedora IoT, whatever. We're talking here about desktop distributions. And yes, you can use it. Debian is a stable full distribution with tens of thousands of packages available as binary files for installation without the need to cross-compile from source. And this is actually an advantage because you don't have to spend time to cross-compile a package each time they're already compiled for the ARM and X8664 architectures. There are numerous Debian derivatives for embedded devices. Some examples are the Raspberry Pi OS and Armbian. The problem with Debian or other desktop distributions is that you need to modify them in order to fit on the embedded device. And if you decide to update it using the package manager, this means that most probably you need to update not only the package you are interested in, but also its dependencies. And if you remember from the late 90s and the beginning of this century, you may run into the so-called dependency hell working with the package manager. So although I use Debian derivatives for some hobby projects at home, I highly don't recommend you to go this way if you're working on a professional device. And I recommend you to use a dedicated build system. This is a very wide and wild topic. And Chris Simons had an excellent presentation on this topic a couple of years ago. So here is a link in YouTube. If you have missed that, have a look at it. So now I would like to focus on the Yocto project. This is something that we'll be using in the next slides for the examples. So the Yocto project is an open source collaborative project of the Linux Foundation for creating custom Linux based systems for embedded devices using the open embedded build system. Open embedded build system includes two things. Bitbake and open embedded core. The Yocto project comes with a reference distribution called Pocky. It's provided as metadata without any binary files so that you can bootstrap your own distribution for embedded devices pretty quickly. The Yocto project has B annual release cycle, which means that we have two releases each year. And now there is a long term support release covering at least two year period. So if you are working on a long term project or on which the research and development will take more than six months, I highly recommend you to pick up long term support release of the Yocto project. Now let's talk about the strategies for updating embedded Linux devices. Basically there are three main strategies. A, B updates where we have dual redundant scheme, basically two identical partitions. We have delta updates. In this case, the update artifact is just a small chunk of binary data, which is the difference between the old and the new image. We have container based updates. In this approach, we're bringing containers which are a household name nowadays for cloud applications to embedded devices. And sometimes where there are more sophisticated use cases, you may need to combine several of these strategies together. Now let's have a closer look and discuss the details of each of these strategies starting with the A, B updates. So we have dual A, B identical root file system partitions. There is also data partition for storing any persistent data which is left unchanged during the update process. Typically a client application runs on the embedded device and periodically connects to a server to check if there is an update. So if a new software update is available, the client application should download it and install it on the other partition and after that switch to it by rebooting the system. There is a fallback in case of update failure because we always have another partition that is known to work, a good partition that we can switch to. The next strategy is the so-called delta updates. In this approach, only the binary delta between the difference of the new and the old image is sent to the embedded device. Basically it works in a git-like model for file system trees. This way we save storage space and furthermore we save connection, bandwidth, because the delta updates are significantly smaller compared to the artifacts needed to be downloaded and installed with the previous strategy, the A, B updates. It's also possible to roll back the system to a previous state. Now let's compare the A, B updates with the delta updates in general in terms of strategy. We're not speaking here about a specific implementation of these strategies. We are speaking in general because there are certain rules that are applied no matter how you implement this strategy. So A, B updates obviously require a larger storage space because we have two identical partitions for the root FS. For the delta updates, the storage space is significantly smaller. The same is valid for the update size. For the delta updates, the update artifact is small because it's a binary diff of the changes between the new and the old image. And for the A, B update, the update artifact is actually something that we're going to install on the other partition. Both the A, B updates and the delta updates support rollback to a previous stage. However, the advantage of the A, B updates is that it is possible to do a fallback to a backup image on the separate partition. This is a huge advantage in terms of reliability because all the time we have a separate partition on which we can boot known to be working image. Now another strategy is using containers. The container technology has changed the way application developers interact with the cloud and some of the good practices are nowadays applied to the development workflow for embedded Linux devices and Internet of Things. Containers make applications faster to deploy, easier to update and more secure through isolation. And in terms of the Yocto project and Open Embedded, layer meta virtualization provides support for building SAM, KVM, LibVerti, Docker and associated packages necessary to construct your own open embedded based virtualized operating system. So there are particular solutions for software over the air updates that are entirely focused on containers such as Belenna, but it's also possible to combine containers with some A, B updates or delta updates. And in the ecosystem of Yocto project and Open Embedded, this can be done with layer meta virtualization. And speaking about combined strategy, this is something kind of a hot topic nowadays, because multiple combinations exist for more complicated use cases. For example, we can have A, B updates with delta updates. This way all the time we have a second partition that is known to work and we can switch to it. Another example for a combination is to combine containers with let's say A, B updates for the base custom embedded Linux distribution. The advantage of this approach is that we can have a really small embedded Linux distribution capable of running containers and all the apps can be running in the containers. This is an interesting approach which sometimes is preferred by application developers because this means that application developers can isolate their apps in containers do all the QA there and ensure that the applications are working exactly the way they want them to work. Now let's have a look at the popular open source solution for updates. So far we've covered all the things that we need to consider. We covered the strategies for software over the year updates and here are actually the open source solutions to do updates. We're starting with solutions for A, B updates such as ROG, SW update, Mender, SW UPD actually Mender is capable of doing some advanced features such as delta updates as well but the main open source feature is A, B updates. We have solutions based on containers such as Balena and Snap and on the second column we have solutions based primarily on libOS tree also known as OS tree which implement delta updates. In order to do it, libOS tree is not enough because we need a second application that acts as a client to check with the server to see if there is an update pending to be installed. Therefore we have applications such as actualizer, actualizer, light. Qt OTA is based on libOS tree as well. Toradex Toryzen is also based on libOS tree. We have a full meta update as well and RPM OS tree which is used in project Atomic also based in libOS tree. Now let's start with a deep dive into three of these technologies which I have personal experience with. Mender is available as a free and open source or paid commercial for enterprise plans. Basically, Mender is doing A, B updates scheme for open source and all plans as well as delta updates for professional and enterprise plans. Mender also provides back-end services. This is the so-called hosted Mender. Basically, Mender is end-to-end update where you have a solution for the embedded device as well as a solution for the back-end, for the server where you can see all the devices and you can manage the devices. Furthermore, Mender has some interesting add-ons. Mender is written in Go, Python and JavaScript depending on which component of the system we're talking about. There is a yokto and open embedded integration through the layer meta-Mender and extra BSP layers known as meta-Mender community. The source code of Mender is available in GitHub under Apache 2.0 license. Now let's have a look at the supported devices by Mender. They cover Raspberry Pi, Beaglebone, Intel X86, 64 machines, Rockchip, O-Winner, NXP, which both includes IMX6 and IMX8, and many more which are the full list that's available in meta-Mender community layer. So here is actually a screenshot where you can see a lot of the machines that are supported. Also, there is support for Tegra devices and video Tegra devices. Now, the second thing that we would like to focus on is RAUK. RAUK is a lightweight update client that runs on an embedded Linux device and reliably controls the procedure of updating the device with a new firmware version. It supports multiple update scenarios and also provides tools for the build system to create, inspect and modify the update bundles. It uses cryptography to sign the update bundles and it's updated and it's compatible with the Yocto project, PDX, Dix and build routes. So basically, it's compatible with these three build systems. The source code is again available in GitHub. Here are the licenses. These are the RAUK integration steps. I have done previously talks dedicated in Talion RAUK where I explain in more details these integration steps. But basically we need to select the bootloader to enable SquashFS. The root file system must be X4 and we have to create identical partition for the root FS that matches the RAUK slots to configure the bootloader, some other configurations. So here is an example where we have A and B partition, actually three examples, three different examples. In all of them we have A and B partitions. The difference is how we handle the data partition. The data partition is the information that is persistent during the update. There are three different configurations that we can make with RAUK for the data partition. This is one of the advantages compared to other solutions, pretty flexible in terms of the data partition. In 2020 I started a Yachtel layer called meta RAUK community. Basically this is a layer that provides examples how to integrate RAUK on several machines. At the moment it supports x8664, Quemo, Raspberry Pi through meta Raspberry Pi layer. These are the all-winner devices through Metasunksy and videoJetson TX2 through the Metategra BSP layer. Contributions are always welcome so if you have experience with RAUK running on another hardware device and you would like to contribute to Meta RAUK community, please do it. So here is an example of how you can integrate RAUK with Yachtel and OpenEmbedded with Meta Raspberry Pi for Raspberry Pi. This is based on the long-term support release of the Yachtel project called DUNFO. These are the configurations that have to be set in local.com. After that we need to build the system. Once it's built here is how it works. I'm just starting a simple web server on the build machine and in this scenario I suppose that the build machine and the Raspberry Pi 4 are in the same local network. After that on the Raspberry Pi 4 I have to log in and to download the RAUK artifact and after downloading it I need to install it. In order for the new version to take effect I need to do a reboot and on a reboot the bootloader switches to the second partition on which I have downloaded and installed the RAUK binary. As you can see all these steps here in this example are done manually and it is possible to do them automatically using configuration and management system for example for Eclipse Hoke Bit. This is a domain independent backend framework for rolling out software updates to constrained edge devices as well as more powerful controllers and gateways connected to IP-based network infrastructure. It's written in Java, it's available in GitHub under EPL-10 license and it's compatible with RAUK SW update. So unlike Mender where we have an end-to-end solution that includes both the web interface for managing the updates in the case of RAUK and SW update we don't have this but with a third party tool such as Eclipse Hoke Bit we can have the same result for AB updates. So in terms of speaking about switching back to Mender you can see that the standard installation in the standalone version is pretty much the same approach as the approach we did with RAUK. Again I'm building an image, I have to disable the system disservice that automatically starts the Mender client and then to run an HTTP server on the build machine and on the embedded device in which we have Mender to install it manually. This is the standalone version however it's highly recommended to use the manage the default approach where a client is running all the time as a demon and push from the server updates and once again I repeat that Mender also provides a server so that you can manage the devices on which you're performing updates. Now the Mender data partition, Mender creates a data partition to store persistent data which is preserved during the update, it supports X4, BitterFS and recently I added support for F2FS file systems. The Mender client actually uses the data partition to preserve data that is using information about the updates. There is a variable called Mender data part size mb for Yocto and open embedded builds which configures the size of the data partition by default it's 128 megabytes but if system D is enabled, Mender GrowFS data service will be responsible to with the feature Mender GrowFS data, the system D GrowFS service will be responsible to resize to the whole available space. It's also possible to implement a neutral MFS disc to to modify the data partition size. Also another thing that's possible with Mender is to create an image for the data partition in advance when we are using BitBake. This is particularly convenient if we need to put certain files in the data partition on first boot with the installation. Here are the steps how to install Mender. In terms of AB updates they're pretty much similar in general as with ROG. We have to apply the update, reboot the device on the first boot after a successful update through the Mender client. A commit must be performed to accept the update by accepting this and by committing this update the user accepts that everything works fine and on the next boot the same image will be used otherwise if commit is not performed the system will roll back on the next reboot and use the old partition before applying the update. So the changes won't be available. Furthermore there are some advanced features for Mender which are worth mentioning. Mender single file artifact this means that you can deploy a single file and this is a solution if you are using mobile data because a single file can be really small for example just if you're missing just some kind of a configuration or you need to make a modification of configuration. Here is the Mender single file artifact option. There are add-ons in Mender which makes things really interesting and more convenient for the bugging and development and QA. The features are remote terminal which is an interactive shell session with full terminal emulation file transfer this means that you can upload and download files to and from an embedded device port forwarding and also a module for an add-on module for configuring apply configurations to your device through uniform interface. Mender supports x8664 devices this was added through Grubb four years ago and initial installation of the distribution is most commonly done using a live image on a USB stick so basically the first time when you install the device you use a USB stick and after that you can perform updates over the air. One of the interesting features in Mender is the delta updates so at the beginning of the slides I told you that Mender is known as a solution for AB updates that's totally true and the open source version of Mender does AB updates but also Mender offers a robust delta update root FS as a module for the commercial Mender plan this is a cross-source implementation even with the delta update reboot is required it supports rollback Mender provides a tool called Mender Binary Delta to create the binary diff by comparing two different Mender artifacts. A mandatory requirement to use this delta feature with Mender is to implement a read-only root FS and how you can do this with Yocto and Open Embedded well in theory it's pretty easy because it's just a single line either in the image recipe or in local.conf you have to add the feature read-only root FS. Yocto and Open Embedded will take care for the rest but be aware that you should make sure that all your applications can run in a read-only mode and if there is something that you need to write through these applications that should stay persistent through updates and reboots you should configure the applications to write it to the data partition. We can combine AB updates with containers through Yocto and Open Embedded layer meta virtualization it provides support for building popular container solutions virtualization has to be added to the distro features through beatbag using distro features append by the way this is the new syntax in the previous examples which were based on release done for the Yocto project always using the old syntax for example adding Docker to embedded Linux distribution is pretty easy because all you need to do once you have the distro feature meta virtualization is in place is to add image install and append and there are use cases on powerful embedded devices where containers that combine with AB updates of the base Linux distribution built with Yocto and Open Embedded for example with Krauk or Mender. Now let's talk about something else about open-source solutions for Delta updates starting with libOS tree libOS tree previously known as OS tree is a shared library and tools for this distributing and deploying file system images as atomic upgrades for the whole file system tree it provides a suite of commonline tools that combine a git-like model for committing and downloading bootable file system trees along with a layer for deploying them and managing the bootloader configuration libOS tree also known as previously as OS tree is very effective in terms of disk space and connection bandwidth it supports rollback of the system to a previous state and it is the core of several solutions for embedded embedded Linux distributions such as Toradex horizon actualizer which was implemented in Yocto and Open Embedded a layer meta updater and actualizer line also cute cute over over the air in order to implement libOS tree it's not very trivial i have done that in in the past libOS tree encourages systems to implement the user move it comes with an optional drug cut and system d integration documentation explains how to do this libOS tree only preserves the var directory across upgrades and var is empty by default and the operating system needs to dynamically create the target of these at boot for example with system d temp files if system d is being used for the particular embedded Linux distribution of course now i would like to mention actualizer and actualizer light as part of my job at consulco group our company was under a contract with ATS advanced alimatic systems and for several years we were involved and i personally was involved in the development of actualizer this is a c++ application that implements the client sign functionality for OTA connect according to the obtained security framework there was there is a Yocto and Open Embedded layer called meta updater which provides all the actualizer and all related dependencies for it also there is a lib actualizer which is a library providing ap cost to perform various steps that are necessary for checking updates validating the downloads files and so on all this is available in github actualizer was initially developed by ATS advanced alimatic systems later on they were acquired by here and unfortunately at some moment the active development of actualizer and meta updater stopped however the community and other companies and individuals interested in data updates stepped in and there are a lot of other solutions and modifications such as actualizer light this is something developed by foundress io and it is a c++ implementation of two OTA update client based on actualizer but without the complexity of obtain and the Yocto and Open Embedded integration for these solutions are available with meta updater for actualizer and meta LMP which brings the necessary changes for actualizer light so let's wrap this wrap up this presentation with some conclusions as you have seen there are numerous things to consider when implementing an upgrade mechanism for an embedded linux device you have to carefully think over all the questions that we started with and to see your device and your if your device in your use case fit into them based on on this you can select from the many reliable open source software solutions to upgrade embedded linux devices basically i have shown you that there are three main strategies and a fourth strategy is to combine them so based on your use cases you have to first to select the strategy then to select a reliable open source solution there is no universal answer so if someone expects me to say use this or that it all depends on your on your use case of course there are certain solutions that are more like turnkey solutions for example Mender because it's it provides a full end-to-end update solution however still you may need to combine it with something and an example for this is to combine an update strategy such as a b upgrades of the base distribution with containers for the application for the applications this is increasingly popular nowadays because of the increasing popularity of containers in general the update process implements implementation depends on the build system and the bootloader basically the board support package often real-world solution required a persistent data partition which is left unchanged during the update process because basically you need to save some configurations over updates some configurations that make this particular device unique so thank you very much i hope you've enjoyed this talk here are some useful links and i would like to hear your questions thank you very much again