 Okay. Let's get started. Thank you for joining me today. And the plan is today it is in my session to take a look at how we can integrate an open source software update solution on your custom board. A quick session overview. I will start off with some background about me and why I'm doing this. And then we'll jump straight to the different projects that are available to us. And on each project I plan to provide a quick info about what the actual project is. Requirements to be able to run it on your custom boards because they have requirements. And I will also provide insights on integration points that these projects have. I will focus my integration insights on the Yocto project because all of these projects that I have here have that as the official integration method. But there is a possibility to integrate these projects either with build route or manually if you're not using a specific build system. And with background I work as a contractor at a company called MDM Technologies. We are based in Sweden so I'm a long way from home at the moment. I am also involved in two of the projects that I listed and I'm going to cover. Mender is one of them where I've been an active community member for a while now. But I've also made contributions to Software Update and I'm involved in various other open source projects. But on my day to day work I usually what I do is help out our clients, deploy embedded Linux devices where I do board support package development basically. And this includes hardware bring up, board support in a boot loader, normally U-boot, board support in the Linux kernel, in some cases some custom device drivers. And normally there is some kind of user space or custom distribution based on Yocto with core. But nowadays this also includes integrating an open source software update solution for me at least. And that is part of my delivery so to say. I have been, I have worked in projects where we have tried to make our homegrown updaters so to say. And I've been down that path and they were normally of poor quality and barely worked and that's when I started to look at the available open source project instead. Instead of building our own because someone already has done a huge amount of the work that you can build upon. And some of these solutions are basically as close as you can get to an out of the box solution. I just wanted to mention some previous talks on this topic because I'm going to focus on the integration of each project and not so much on what differentiates the features of the project. But these are from 2016 and the better Linux conference that does one more head to head comparison of the different projects and how do they do the updates and pros and cons and so on. I also have a site where I've written and I'm writing a series of articles covering these projects that I'm talking about here but more hands on where I am also providing tutorials for how to use them or get started on their reference boards. So if you're interested take a look. The plan is to get straight to it. Mender is an end-to-end open source updater for connected devices. And Mender is an end-to-end solution that is open source. That is you have a Mender client that runs on your device and there is a server component that's also open source that where you can manage your devices and deployments. But the key takeaway here is from an integration point is that the Mender uses symmetric AB image updates which I will talk about a bit more. So if you go into the requirements that are required by Mender, Mender requires that you run your boot as a bootloader. That's the only bootloader that's supported at the moment. I know that they are looking at expanding that but they rely on some features in the bootloader to do the root files system switching. Yeah, there's a feature you would call bootCountM and bootCountLimit that they rely on and that's basically a counter that's in the environment with your arm that needs to be reset once the device boots. If it doesn't, you have an alternative boot command that you can like. If you have a failed boot you can do something else. So let's say Mender utilizes this and your U-boot must support this and this is post 2014 I think. You need to store the persistent storage of U-boot environment. So it has to be stored on EMMC or in a flash and it has to be permanently stored so not in a RAM where it's volatile. Mender also relies on a set of tools FV, setM and getM which are user space tools to manipulate the U-boot environment. As I mentioned, it is a requirement that you have two partitions for your root file system because Mender does and the thing is that you have one active root file partition and one inactive so when you update the inactive one reboot and switch so to say and that way you can always roll back to the previous if the intended one doesn't work properly. Mender also requires one partition for this persistent storage where it will store some logs of the log events of the update procedure and Mender only runs on EMMCs, SD cards and UB volumes and if you start looking on how to integrate Mender it's heavily focused as I mentioned all these projects are focused on Yocto to optimize their seamless integration. There is an option to run there is on its way I think or maybe I'll upstream in build root Mender support as well but there's a metamender layer which is a collection of layers so you have a metamender code and you have some metamender Raspberry Pi, metamender demo which are well metamender Raspberry Pi is the board specific one and what you normally want to do is include metamender core then you have to inherit Mender class which is the Mender full which will bring you all the features that the Mender offers so to say and if you're lucky that's all you need to do because Mender now they support fully automatic reboot patching so they try to automate integration to your custom board but this does have some required requirements so you have to run this is only supported on the Rocco Yocto branch you'll probably need to have something a recent Uboot for this to work and this only works on EMMC SD card layouts and the output from metamender when you build you get a SD disk image which is the image that you use to provision your devices so it's an image that contains all the petitions that are required from Mender so this is provided and the dot Mender file is a Mender artifact this is what you give the to the client to update the devices but it's basically a Tor archive containing a binary root file system with some metadata on top of it but if we take a closer look this automatic Uboot patching if that fails because it doesn't work on all boards you still have to do it then you have to do it manually and the main integration point for Mender is you need to make some changes in Uboot so there are two patches that are board independent that Mender carries that you need to apply to your Uboot and this is basically a set of variables and script and commands that are defined by the board in independent patches there is and then you have to provide one board specific patch and that means integrating the generic Mender commands in your custom board boot command so to say and you also may have to make sure that you have the boot count and limit enabled and you have to make sure that boot and options like is in MMC or flash to store it permanently but if we take a little closer look this is normally what the integration custom integration layer looks like so it's basically one patch and a BB a couple of BB appends this is actually removed Mender bigger bone doesn't exist in the recent releases because the automatic Uboot patch patching takes care of this and for yeah for the custom Uboot forks you have to include to make sure to include Mender specific include files to get the board independent patches because they normally work only on upstream Uboot so if you have a custom Uboot fork you have to have custom BB appends as well but if we take a look at what the actual patch looks like and the changes that are required so basically it's just changing if you have a the boot command in Uboot like run MMC boot so you have to prepend that with Mender setup and upend try recovery and the try recovery is in case of MMC boot fails it's going to run some script trying to like roll back to previous image so to say and one has to remove all hard-coded parts of the MMC and there are specific Mender variables and this is what the Mender setup script actually sets up these Mender variables because we have a dynamic root file system it's either A or B so we have to have some way to switching between them and that's basically it for the integration of Mender that's needed and they do provide a quite extensive well checklist that you go through to make sure that you've done all the integration points but that should get you an overview of it and the next project is called Libuestry mostly explained well described as Git for operating system binaries because it has a very Git like command interface and the key takeaway from this one is that it does image updates but binary deltas so it applies binary diffs instead of because Mender does for example full image updates and Libuestry is able to do delta binary updates and some of the requirements because Libuestry doesn't really require a specific partition table you have one partition for your root file system but the only real requirement is hard links that your file system supports hard links because that's how it does the delta thing but there are some requirements or so to say when you use Libuestry you never boot a physical root FS so you have to boot a RAM disk that will set up the physical root file system and then do a chain root into the physical sysroot. Libuestry everything that's managed by Libuestry is in slash user so you can't you can't have files outside user that that you want to manage with Libuestry and slash user is mounted read only that's a requirement from it and Libuestry also said has a yeah it has basically three catalog directories that are special it's slash user slash var is the persistent state so everything that you want to keep you write slash var because Libuestry won't touch that and then you have slash etc that I will talk about a bit later but generally Libuestry it's quite complex to set up because because of these requirements that it has that you have to have everything in user which brings on some complexity and that we are not used to really but thankfully there is there is actually a yocto layer called meta updater that does all the hard work for you like does the make sure that that you move everything to slash user and make sure that you have get the client on on board meta updater actually comes from a project called Geneva where they developed because there is a client called actualizer which is able to communicate with the server backend as well but you can use meta updater in a standalone node if you want to integrate Libuestry basically so it's simply including meta updater in your bb layers then you have to inherit Suta what you also get from the meta updater your as output is you get a round disk image that you need to boot that will do the setup of the Libuestry physical sysroute and change route to change route into it so you get that as output basically so you don't have to do anything and you get a deployment sysroute which is quite important so every time you do our yocto build it will do commit your changes so to say because Libuestry does delta update you need to keep track of all the changes that are being done all the time so by using meta updater it will set up a repository for you which you should then use to deploy to your device integration point is really simple really all you have to do is make sure that your board boots the round disk that is provided by meta updater and then the round disk takes over and does all the setup which is board independence so to say so this is an example script from they have a Raspberry Pi as a reference board and this is what the integration script looks like and the only parts that are in bold here are basically was tree specific the other things are generic things that you need to boot a Linux system with your boot and yeah did I want to say something more yeah please interrupt me with questions along the way I forgot to say that so please if you have anything yeah and you have you have to also think about these three specific directories which do impact have impact on your system the design actually more and this is written the hard part I guess of integrating Libuestry that you have to well we are restricted in this matter so to say and slash war is empty by default so if you want to have anything there at build time you have to make sure that get something installed yeah and jumping on to the next one SV SVW update is a Linux update agent with the goal to provide an efficient and safe way to update embedded systems so this one is a bit different from the previous two that I covered because this project aims to be a framework and a set of tools more while Mender and the meta updater are more out of the box solutions but this project is instead like you're given a bunch of tools to update systems but there's nothing really there's no restrictions there's no requirement you can basically do anything with it but that also like makes the integration harder because well you have to design your update system you're only given the tools to execute it by this project and then that this makes it a bit hard to provide really insights how you're going to integrate it but they do have some starting points so there is a meta SV update layer where you get there's a client running on your device so you have the recipe for the client and it will also it has some recipes for a recovery US image and there is a software update baby class that enables you to build images that software update understands basically and then there's a sub meta software update boards which is sort of reference implementations on how you integrate with software update and there's three boards that supported at the moment it's bigger bone black Raspberry Pi 3 and Wanda board and all the examples here is also symmetric AB updates that they have in meta software boards and the integration is really through the software description files that software update has so it's a simple text file based on lib config syntax where you have different handlers and tasks that you can execute and if you look at example image this is the bigger bone black image that's part of meta software update boards this is what it looks like so it says it has a copy one and a copy two but if we look closer of these what the copies contains so they can contain an image they contain a script and a input command and the image is basically a total archive where do you want to install unpack it and this is like software update understand the syntax and execute it there's also a script in this particular setup that is run before the update and software update supports Lua scripts and this particular script actually partitions the SD card so you have if you don't have two partitions it will create one extra the first time you do an update and that's why the script is there and then in the boot command that's by basically the integration point here because they need to set which part we are going to boot from and this is done here and basically over writes the boot command with the new part and this is it's a quite good starting point I would also like to mention that there's a talk today I think it's around three where they explain a bit of the possibilities of software update what they have done with it so to say so if you're interested you can go to that as well then we have resin IO which is not a simple updater compared to the previous three that we just had resin IO is a new workflow or a new yeah it's a new workflow for a better system basically so it's a way of running apps in containers on embedded devices so and they have actually two different kind of updates in resin and because they run applications in containers they can utilize the layering feature of container technology to update binaries deltas basically but if you want to update the operating system underlying and container applications they deploy also a B stretch strategy and in that case you don't have any delta updates it's just a full image for the operating system and resin IO works with uboot and group and it's usually done with update hooks in user space where they have where they have the variable that says okay I want to boot from part A or part B is in these text files basically and uboot then imports these files or grab to determine which device part of the boot but to be able to use resin you have to integrate resin us which is a yachtable based distribution so it's a full distribution that you have to integrate on your device and not just a updater because really this is a new concept but it also has dual root file system parts and then they have three extra system parts that you really have to reserve for the resin IO resin us and from what I understand they only support emmc and sd cards so I haven't seen any non support or ub support and now to integrate there is a meta resin layer which is the starting point for the resin us I will not talk so much about npm there I would just but what you need you have to include meta resin and then you have to have create a custom board custom board layer but the key takeaways is that you have to add inherits of resin uboot and inherit kernel resin which will do a lot of configure kernel resin at least does a lot of configuration of the kernel kernel linux kernel to support container technology they have a lot of requirements for that so you basically add inherits and then they should take care of that and you also need to provide an update hook which is a best script basically in your custom layer but if we take a closer look for the operating system updates which is the ab strategy this is very similar to how mender does it basically they have three patches that are board independent and that are applied to uboot this is a set of variables and scripts and commands and then you need to provide a board specific patch to integrate in a similar fashion like in mender to call these resin IO specific commands in your boot boot flow they also have a dependency on command part and partition IDs and unlike mender mender the mender mender ly relies on the uboot firmware it tells to write to the uboot environment where they have the the part in the variable that says which is the active part in resin they have a text file in the boot partition instead which is imported so there's no requirement to have firmware details and if we take a look at the example patch this is for i think for the various site dot six u l board how the what the patch does so it basically removes all the hard-coded stuff like we did in like mender does it and replace it with resin IO resin specific variables and you also have there's only a prepend in this case to the boot command so they don't have a append that tries to recover in this particular case and this is basically they use some resin IO specific variables uh to set up what to boot basically yeah and uh i wanted to mention a bit about initially i had a very high ambition on covering very more projects and raw rock rock was one of them i was planning on also looking how to integrate that but it kind of fell out because i didn't have time to prepare it but also i tried looking at it a bit they have a meta rock layer but that only integrates the client side of it or and you have to get the artifact tool as well but i couldn't really find any reference implementations on how to integrate but rock is very similar to software update because it's also a tool or a framework to enable software updates and devices so it's not a lot of box solution and then there is something called s view up s w u up d which is also software update when you pronounce it i guess there is a meta software update layer but this is not to my understanding used that much uh i haven't really used it in many of my projects at least and uh yeah i'm with the summary already so yeah and these open open source projects have been there for a while most of them which makes them proven and battle tested and and there is it's really a seamless integration if you are using yokto it's of course tougher if you don't use yokto or want to integrate with build root and or manually integrate it on a devian system or something like that but if you are using yokto it's basically including some meta layers and providing a patch and you're done in most of this and and that is why there's no reason to really go home design your own systems or go homegrown completely and instead we can collaborate on these projects that are existing and build upon them and improve them so to say and yeah questions i will ah yeah i'm not familiar that anyone really solves the up update beside resin because they have focused on that all of the ones that i mentioned are really image based so full system updates basically and like you mentioned the voice read us apply deltas but you have to reboot uh flatback i'm not sure i'm fairly familiar with flatback but i'm not sure how they use libu s3 problem well ab copies are or is the most robust solution in my opinion and if you are able to have two copies of the root file system because in some cases you can't can't physically have it because you don't have the flash for it so to say but if you can i don't really really see a better and more robust option to do deploy updates i haven't actually deployed libu s3 in a really harsh production environment to really get that feedback if it's really resilient but by nature libu s3 relies on the hardware and the drivers and the file system to handle atomic updates so it relies on the underlying layers to be resilient to power loss or stuff like that which you don't really have when you do an ab update then you kind of eliminate a bit of the hardware issues yeah because it's a complete operating system that you have to integrate to be able to use the container deltas basically so you have to have a container i know they have a embedded one where we have a variant i think they started out with docker but they have something called bolina yeah that they use but it's still i don't know well these solve the problem that you'd only have you have a top version so it's fully the full image based updates and in the mender case like it updates the root file system and if you want to update the linux kernel and the device tree it has to be in the root file system otherwise it can't update it and the mender for example can't update the u-boot because yeah it's generally unsafe to do that so they avoid that you can with the other project like software update update u-boot as well but but generally this is it's full disk image updates you have a top version and then you can find out what the other versions are for u-boots for kernel for all the packages and you always update everything you know yeah yeah all of them i would say support that i know menders at least support signed images and it's compressed images of course software update supports signed and encrypted images libu s3 is like it's a lib library so you have to have a client on top that which is the actualizer for example that talks with the back end and i would assume i don't really know how they implement the download there but i would assume that they have at least check some checks on the files downloaded well yeah i've been in the same situations many times that you have a small microcontroller that you need to update and it's quite easy when if you do full image updates to have a post post process step like on first boot update the microcontroller as well uh and all all of these projects you have something like that uh like post install scripts that will hooks that you can hook on to to run custom update procedures for your microcontrollers so i would say it's solvable but i think that the one that's probably trying to handle it best is software update that really is trying to focus a bit about distributed updates and trying to update yeah or have a like proxy devices that update other devices and stuff like that yeah and that is the meta updater which is the actualizer actualizer i wouldn't say that i mean yeah you have to raise the full partition that you're updating uh but especially if you use ubi uh files well yeah ubi then you at least spread out ware leveling equally over your whole volumes but it's one array cycle per update or maybe two because you need to update a variable somewhere in a you boot environment or something like that so but one would have to calculate like how many up if you have 100 000 cycles on your man flash or 10 000 cycles yeah the more your size you have you the more ware leveling will you have of course and uh did you have a question no it's more usually when you know like non controllers but then you have the ubi file system that tries to like do the ware leveling for you or the ubi layer of never worked with sorry no not that i'm aware of i know that the emcs have some but there are still normal partitions on emc just called boot one or boot two i guess so i don't think it's a problem to integrate that why everyone has a uboot for it while they over that are yeah they're good problem but what they do is they make sure that their variables exist in the default environment like they enforce that that's really not so if you upstream it you like yeah you could have a config for it to enable it yeah i don't know if anyone has tried if you would have accept something like that like but it's possible they are highly every board is different like i changed this slide slide slide here in a single update system or what in the case of mender they have a ab like for the root file system then they have a data partition which is like where you put the persistent data so a mender update doesn't touch that area at all so if you want to keep data you put it there basically and that's normally what you do when you're an ab you have an area where you're like if you have a database or something that you want to save across updates keep it in the separate partition any more question but thank you