 Hi everyone. I'm excited to see you here. I hope that many of you have heard the previous talk. It was also about software updates. If not, please see the slides afterwards and the end the video. I will talk about a bit different methods of software updating software, namely of OS tree and of our meta-updater layer that integrates OS tree into Yocto open-embedded build process. So I work for company called ATS. What do we see here? It's fine, yeah. Sorry for that. So I work for company called ATS. Our main field of expertise is unsurprisingly software updates for embedded systems, namely for automotive systems, but not only for automotive systems. And until recently, our only concern was secure and robust delivery of update to the target device, but with the launch of our open product web service for updating any IoT device over there, over our virus or wired network. We decided that it's important to care about what we actually deliver to the device, how we actually update them. And we integrated OS tree repository into our ATS garage. So those who heard the previous talk, I hope you are already convinced that we need software updates. I will just say that it is, apart from security and market concerns, it's actually really convenient with regard to CI process for, I think, everyone who tried to make CI on embedded devices were really tired with this continuous reflection on the devices. You need to get physical access to them. And it's, well, it's really cool to do it over there. And until recently, we had, so, now, we have basically three methods of delivering updates. First is package-based, what we use in all our desktop or server Linux distributions. It's nice, so it's convenient. We only download what we want to download. It's very important for over there updates where bandwidth is expensive. But it's not fit for embedded systems because we don't have a human operator. Like, if we are stuck, if our package is not compatible with another package, or we install two packages that are mutually dependent and our installation breaks in between, it's, well, we can't recover from that. So, what the gallery, the Enrico was talking about is that it is, for embedded systems, it is good to have full file system update when we just download the full file system, flash the backup partition, and we are done. We can test it exhaustively in our development process and deliver to our client what is definitely working. But it also means that we have to kind of download the whole file system image which can be hundreds of megabytes, and our bandwidth is slow, unreliable, and our customers are going to pay for it, and we wouldn't be happy with that. So, OS tree, with OS tree, you can deploy virtual bootable file systems, but you only download what you have to download, the difference between two file systems, not the whole file system. So, a pair of words about OS tree itself, it was not developed by ATS, it was not developed by me, of course. I had made a pull request once, and it was declined, but it's developed by Colin Walters, Red Hat developer, and its origin, it was not intended for embedded systems, its original purpose was continuous integration for GNOME team. You can get more info there, there will be links, of course, in the end. And basically, typical memory device of OS tree-enabled system looks like that. You have this boot small boot partition with U-boot, with U-boot configuration, and you have so-called physical sys root, which is essentially not bootable, but it contains OS tree repo. I forgot to tell you that OS tree, that was on a previous slide, OS tree is a Git-like tool, so who knows how Git works, we basically have our object database, a set of objects that are addressed by the hashes. We see file objects, directory objects, we don't see any commit objects, but there are commit objects, believe me. So, yes. So, in our physical sys root, we have this object repo, we have several, one or more deployment sys root, and we have some information for boot loader, so that the boot loader knows which sys root we are booting into. And, well, the order of the boot order, when we start our device, boot loader reads first it reads, well, some decent file, which is inside it, then it gives our info that OS tree provided to him, to it, which is stored in boot slash boot slash loader in physical sys root. And then it gives control to any trauma phase, which does uses power of Linux environment to prepare all the complex stuff. I will talk a bit about why do we need to do this complex stuff, why we need to remount user and all this, yeah. It's a bit more detailed view of these physical sys root. There are a lot of lines and errors, but what's important here is that our deployment sys root basically files in our deployment sys root are not duplicated. They are just hard links to objects in our OS tree repo. That means that we don't have to have repo and then one deployment and then another deployment. We don't have to duplicate all the data. We just store what we need to store. And from what I have said so far, it's maybe self-heavening that to adopt this OS tree. We need to prepare physical sys root. We need to prepare deployment sys root. We need to make boot loader and trauma phase work together to do this loading process. And the good news is that if you use Yocto, if you use open embedded, you can just download our meta update layer that already implements all this complex stuff. But you can't just take your root effect that you are using right now. Maybe. Because there is another concern about mutable state in our system. So what is mutable state and why we should care of it? As I said, our files and deployed sys roots are just hard links and they are shared between sys roots. It means that we can't really write to these files. Because if we modify them, we first break abstraction of independent sys roots for independent file systems. And second, we break our repository which contains hashes of all the files and all this stuff. So if we want our data to be managed by OS tree, we need to store it in a read-only part. And all the read-only data in OS tree should be stored under user slash user partition. And we also can have our slash var directory which, as we can see, it is shared between the deployments. Here is deployment one, deployment two. And here is var that is shared between them. And of course, it's not updated by OS tree. It's written in runtime and read in runtime. Well, here, and there is another directory called etc. There is some problems with that. I will talk about it later. So there is a meta update was originally developed for HL, that is Automotive Great Linux. It's Yachter-based distribution for automotive devices. And there was a very typical scenario when we should care about our mutable state. In OS tree, in HL, apart from full file system updates with OS tree, there is also so-called application framework which is basically does, it's basically a sandbox for applications much like what we have in Android. And it can install applications in runtime. And of course, it has some database for it that should be writable. And the interesting part of it is that some applications can be already pre-installed with our file systems while others can be installed in runtime. So there is kind of a conflict between two update domains. When we perform our OS tree update, what we should do. And when we boot first time, what we should do. There are three ways of solving this problem, not only for HL, but also for, well, every time we encounter this problem, there are three options. We can just ignore what's in initial database, what's in var. And actually it's what meta update does by default. It's saying default option. It will give you a warning, of course, because if you have something in var, that probably means that you have missed something that you forgot about some mutable state in your system. But it was really not an option in HL, not because, yeah, of course we can say no pre-installed apps, no problems. But due to some technical issues with access rights, we needed to deploy it, we couldn't deploy it in runtime because we needed to set the access rights in smack, which is a tricky access rights manager in Linux. So another option, what maybe application developers would want, is to have a smart merger process that will merge what we have in our full file system update with what we have installed in runtime. And it's actually approximately what OS 3 tries to do with ETC. And the problem with it is the same as with a package-based update that we don't have a human operation. And if our merger process produces something inconsistent, which breaks our system, we are stuck, we can't do anything. And a moderately dumb solution is to just populate, however, initial database in our OS 3 deployment and just populate it once on a startup that way we can't update our applications with OS 3, but we can at least update them with application framework and we can at least have an initial set of installed applications. So meta update for those of you who are familiar with Yocto and OpenEmbedded, it's Yocto OpenEmbedded layer. It's integrated seamlessly. You can just add it to your biblayers.conf. You can include, dis-true-include or use our dis-true file and just type bit-bake whatever your image name is and you get an OS 3 commit as a result in image. You have a bootable sys-root as a result and optionally you can push all your newly created commit to some server which has a well-documented open protocol and is by pure chance implemented by our ATS garage. Of course, it doesn't implement what I was talking just right now about population of slash var because it's really application-dependent and it has a number of supported boards but of course if you want your board or your bootloader, you need to write some not-so-large integration layer and if you do please inform us. We really want to support as many boards as possible. And I just wanted to learn about some problems that are still present with this. There is a problem with default ETC merger. What OS 3 does, it tries to merge when you update the file system, it tries to merge your new slash ETC with what you already have and well, sometimes it breaks. So maybe for embedded systems it will be nicer to just take ETC as it is. Second is compared to dual bank system that Enrico was talking about. Physically, there is only one file system and if because of some hardware issue or driver issue or whatever or combination of different factors it breaks, it's broken. You can of course have a boot backup partition but you lose then benefit of economy of disk space but you don't lose economy of bandwidth concepts. So third is a serious issue that when you deploy it in production of course you can test OS 3 before deploying but in CI you frequently run into a situation when you update OS 3 and you break it and you can't update your system anymore because OS 3 is a part of your deployed system which is funny. And if you want to roll back you need to, it's not responsibility of OS 3, OS 2 will just download whatever data you need, it will deploy it, it will tell that next you should boot your newly deployed system but if you want to roll back you need to define, well you need a definition of failure and you need to recognize failure and switch back and ideally it should be done in bootloader or some, yeah. So yeah, here a number of pair of links, OS 3 itself, please you can look into details fortunately with our meta update layer we don't you don't want to need much of details of OS 3. Here's a website of Automative Great Linux, well MetaUpdater was initially implemented for HL and it's called MetaSota there and you can also try it out, download HL, build it with Yocto, Bitbake and get ready or automate OS 3 enabled file system. If you already have your Yocto project please look at MetaUpdater layer itself, it's pretty straightforward for adding it to your build system just as you add any other layer and for those of you who are unfamiliar with Yocto with OpenEmbedded please try out our quick start which is basically well it's quick start for Yocto and MetaUpdater, it has Pokey, it has MetaUpdater, it has Meta Raspberry Pi, it has nice tutorial, you can walk it step by step and quickly have your OS 3 enabled Yocto build system for Raspberry Pi. So that was it's questions please.