 My name is Luborze Miachki and I am an associate software engineer at Red Hat and currently I'm working on the PALP project. And today I'm going to talk about one of the ways of updating edge devices in particular by using OS 3 and PALP. So let's begin with some motivation. What are edge devices? Edge devices are basically devices that are playing on the edge of the infrastructure and can be used for real data analysis and monitoring. And in order to always have these devices up to date, you need to run some kind of upgrade management because when you have deployed hundreds of devices, it's really a tough task to properly have them updated all the time. And there are a bunch of tools that can be used for managing updates and managing repositories that will be used for running updates and one of those tools is PALP. Operating systems for edge devices should be immutable and image-based. What basically means that when you have a running image, you shouldn't be able to apply any changes without rebooting it. And by image-based, I mean that once an operating system is image-based, that basically means that there is a copy of operating system and all data associated with which were created during runtime. Operating systems for edge devices should also support atomic updates. That means that either all update steps finish with success or none of them. For instance, Federa IoT supports atomic updates and rollbacks. That means that you can roll back and forth between different versions of content and repositories. And one of the technologies that can be used for, let's say, managing updates or operating system file trees is OS3, which is some kind of like a git for Linux-based operating systems binaries. And it supports content other objects stored with branches. So for instance, when you are an administrator and you have two different environments running, one for development and one for release, you can create two separate branches and commit separately to all of them. In OS3, they are called reps instead of branches. And this technology is also suited for performing atomic upgrades of complete file system trees. So here's the example of how you can commit changes with OS3. So initially you just create an empty repository, then you can create a new file within a random directory and then just commit it like you are doing in Git. On top of the OS3 technology, there was implemented the RPM-OS3 utility that combines OS3 and accepts RPM packages, which basically allows you to do client-side fetch layering, which is similar to adding browser extensions. So you are adding more and more things to the existing environment. And for downloading updates within edge devices, you can see that you configure a bunch of URLs here that can be used for downloading updates. For example, this one file, etcOS3remotes.t is taken from a federal IoT operating system and from this URL updates are downloaded, which is basically a remote OS3 repository that contains commits for those updates and content. And RPM-OS3 utility is integrated with federal IoT, so when you want to run an upgrade or update your software, you just simply run RPM-OS3 upgrade and reboot the system to apply those changes. And you as an administrator always have to have an overview of what type of content is behind this URL, so repository management for doing such stuff is really important. And thanks to that OS3 itself also supports multiple branches inside one repository, so you can use OS3 for doing so. Besides that for repository management and for high level workflows, you can also use Pulp. Pulp is a platform for managing repositories, so you can sync content from a remote repository, you can manage it locally, you can update it and modify it, and then expose it to end users. So at the left side of this image, you just have a couple of public-active repositories like PyPI and Docker Hub. You can sync content from those repositories to Pulp and do the needful inside your local machine or local machine that runs the Pulp and then deploy content to your different environments as needed. Besides that, Pulp allows you to version content, so once you update, modify, or even delete some content, a new repository version is created for you, and you can decide which repository version you are going to expose to your end users. And in Pulp, we recently added support for the OS3 technology, so you can use Pulp for syncing remote OS3 repositories and manage that content locally. So for example, you can take a Fedora IoT official OS3 repository, you can add more commits to it and publish it as your own. So let's take a look at a quick demo. I'm going to show you how to build a new Fedora IoT image with a couple of commits, and I will show you how to import those commits to Pulp and expose it to an edge device. And after that, I will show you how to perform atomic upgrade and fallback while downloading updates from the Pulp repository. So right now I have two running virtual machines, one is called main that is considered to be, let's say, a main machine for an administrator to work on, and PM1, which can be an edge device placed somewhere remotely. So this is the main virtual machines, and let's say I want to install a nano package inside my PM1 machine. So right now there is no nano package, and in order to create an OS3 commit and to build an image from it, you need to create a so called blueprint file that contains all the packages that you are willing to install. And with that, you are just going to use tool called Composer CLI that creates calls to the OS build Composer server, and you can afterwards create or build new images with the OS to build the server. So, if you want to install nano or create a commit that installs nano package, you just push the blueprint like this, and then you run the start OS3 Composer like this. I'm not going to run it right now because it could take some time, and it could take more time, especially when you're doing live demo, so I prepared all those images, like for installing nano, and also for installing the package beforehand. And as you can see, there are two images that will build, one is containing PIM and one is containing nano. If you want to download those images to your local machine, you just run image and paste here the image ID, and it will download you a tarball file, which basically structured like a simple OS3 repository. So it will contain some configuration extension objects, reps, state, and temporary files. And it will have the same structure as an OS3 repository. So, let's now import those commits to our PAL machine. PAL basically works in the following way, so you will initially create a new repository for an OS3 repository like this, blah blah. And you can import commits to that repository when you specify the path to the tarball and the branch name you want to use. I've already created a repository that will contain all those commits. It's called Fedora IoT and it has two versions. One version contains the PIM package and one version contains the nano package. So, when we want to browse, when we want to see what commit we are currently referencing from the red branch, it's checksum is CE0B and so on. From the version one and from the version two, we have different one, which contains the PIM package. So, I have now a repository that contains two OS3 commits and I want to expose its two end users to the virtual machine one. That's basically the edge device. So I'm going to create a new distribution and this distribution will expose all that content that it's lying within the repository. So, the content should be accessed at this URL. So when we are trying to access it, just let me change this host name because I'm using SSH port forwarding. And when we are going to, let's say, where the repository like this. The last table version number. Well, there's some, well, I'm using HTTPS and I should be using HTTP instead, sorry. And there is a commit that was referencing the second version. So, when I want to expose a different version, which will contain only the nano package, I will just run this division update and then why query the same endpoint, I will receive different checksum. So, now I've exposed the OS3 repository and now I want to install it inside my virtual machine that is considered to be an edge device. So, the configuration for a remote OS3 repository that is used for downloading updates is right here. So, I'm just going to comment these lines and type my, type the reference for my, for my file repository and you'll turn off the GPG verification and specify the ref name. And now, table x64 IoT like this. Right now, if I'm going to run RPM OS3 status, the latest checked commit within the running instance is official commit from Fedora distribution. So, now I'm going to run RPM OS3 update check. You will see that it will try to upgrade for the packages a 71 app. Right now, I do not have the nano package installed and I have built a new Fedora IoT image inside the main machine with OS3 build composer. Now I'm going to install the update that should contain new versions of Fedora IoT and also nano package. RPM OS3, staging deployment, remote connection was closed, then as you can see the machine or edge device is rebooting and applying those changes. As you can see there are two images, you can always pick from those are the two latest images that were installed, and you can just roll back and forth between them. Okay, seems like we are. Okay, machine rebooted. And I have nano package installed now. I'm going to update and modify one to expose the team package. I'm just going to update distribution like this from the administrator's perspective. And I'm just going to run OS3 upgrade check. There are going to be added five new packages. I'm going to install OS upgrades like this. It can take some time. And all this upgrade stuff can be automated so you can just buy the Kickstarter file to set a new update policy that will be regularly run so you do not have to always look into an edge device to SSH or something like that. It can be can automatically pull changes by itself. Machine is rebooting. It's running. I'm in. And now I should have been back to install great. And what if I decide to get rid of the link like I just run the rollback like this. Now I'm going to run the reboot. Machine is rebooting game. And I don't have the impact. And what if I decide to get rid of the power repository and rollback to the repository that is officially distributed by Fedora. I'm just going to remove those lines and command those lines. And from our PMOS tree. Upgrade. It will tell me that the time star timestamps are not in the actual order. So I'm just going to run on PMOS to update a low downgrade. Because I was adding new commits. After that. And now when I run CTL reboot. I should be back on the original Fedora image. Where there is no. None of package or. In packaging store. Okay. No beam. No nano. And. That's all from my side. I'm happy to answer your questions. And. Cool. Thanks. That was a great presentation and your demo work. So there is there's one question from Jan. How long does it usually take to build an updated image via composer CLI? Let's say about 20 minutes on my machine. That's. I have think that the four eight zero. Cool. And. Okay. And you wanted to plug. The ball project. Correct. For discussion after this, if anyone has any further questions. Yeah, you can find us on this course that project that work. I'm not sure if I can. I think we can do it for you. Right on. Well, thanks a lot, Lou Bush. And I just want to remind everyone that you can go to work adventure after this session or the main stages having a keynote soon. And thanks. Thanks a lot everyone for coming to dev comp 2022. And we'll see you around. Thank you. Bye.