 OK, sorry for the short delay. Now the setup is working. So welcome, everyone, to my presentation, secure and save updates for your embedded device with the small subtitle and how to solve it with the ROG Update Tool. My name is Enrique Jerns. I'm an Embedded System Developer and System Integrator working at Pangotronics. So Embedded, IoT, stuff. Well, do we really need updates for this kind of topic? And the answer, of course, is yes, this is why we are all here. And at least the events that took place in October last year when those small IoT devices broke down parts of the internet that showed the last person that security, especially for these devices that become more and more connected to the internet and infrastructure, that security is a topic of high priority. So connect with security. There's always the need for updating. Because if we detect a security hole somewhere, we have to update the device. And what other reasons do we have for updating? It's obviously for fixing bugs. If the software does not perform as we assume it to, or if we want to add new features to an existing device. So a short structure of the talk. The first chapter will be a bit about general challenges in updating and give a short overview of that. Then I will introduce the ROG Update Tool. And then hopefully the time is not over then yet. I'll show a short demonstration, a live demonstration. So I hope it will work. And at the end, I give a short, very short conclusion and summary of what you've heard here. So updating, you might think, is very simple. So you just have a few steps. You have an update and load it on your system. Then you perform the update, right? It changes to your system. And maybe you reboot the system, and everything works and is fine. Well, as usual, reality is not that easy. So especially on embedded systems, you have different devices you want to update. For example, a North Flash or an SD card with different petitions on it. And you have to find out what to update there. And then you don't know who is the author of your update. Is he allowed to update your device? And are you updating the right device with the right update? And you also need some interaction with the bootloader. And if you've done all this step by step and suddenly someone plugs the power cable off and everything breaks down completely. And you don't have real direct access to this. So the main question we try to answer here is, what is so special on updating embedded devices? And why don't we simply write it in a short bash script? So updates you may know from your daily use. You use some kind of packet manager like aptitude or something like this. And you update your system. And in most cases, it will work. And if it does not work, OK, you fix it. So what is the difference when you have an embedded system? Well, on embedded systems, you don't have direct access to the device. In most cases, you don't have any access to this. And the only way you can start your update is either by plugging in a USB stick or by starting a remote deployment via a network. And then all you can do is watch a progress bar. And at the end, it either succeeds or it will fail. And if it fails, you really get in trouble because you don't have any chance to recover it. You don't have any easy access to the device and will cost you time and maybe money to recover the device. So what are the key requirements for updating an embedded system? First of all, the update process itself should not fail. And in case it does fail, the system itself should still in a state where it can be updated. And you also want to recover from runtime issues when your system suddenly stops. Then there should be some mechanism that reboots the device and brings it back into a state where it's still operating. Apart from the safety aspects, you also want some security aspects as these devices normally operate in some areas where you don't have direct access and you can't control who accesses the device. And you have to prevent the device from unauthorized access or updates. And there are some further limitations like you have some sort of storage that is not that easy to handle and limited resources, limited bandwidth, et cetera. So one good rule for updating an embedded system to not make a face to only install well-defined states. So that means the entire state with all components tested together. And the major rule we have when updating an embedded system, we do not want to break the device. We have to do everything to avoid this. So how to avoid this? So let's talk about redundancy. The simplest redundancy mechanism is when you run from a system that has no redundancy. So you perform an update. And if there is a power loss or some issue during the update, then the device might not be able to boot or the update system might not work again. And you have no chance to update it again. So you have to cause a service technician. Better approach is to use an asymmetric redundancy mechanism. This is especially good for devices where you have limited storage capacity. So you have a small, dedicated recovery system where you boot into when you want to perform an update. And you can perform the update. And even if the update of the system itself fails, you can still boot into the recovery system and perform further updates. So it's updatable in case of a failure, but you don't have an operating system if the update fails. So what we really want to have is a full symmetric redundancy system where you have a system A that may update system B and a system B that is able to update system A. And even in case of failure, then you have maybe an old, but you have one system that is still operating. So another thing that is really important when updating is atomicity. The result of the update should either be the update succeeded or the update failed. And you don't want to leave it in an intermediate state where it's not defined. So how can we achieve this? Here's an example. You start your update from your running partition A. And you have to see the entire update as a critical operation. If something fails during this, then you have to draw a rollback or you don't have to pick your device. So what you normally have is some kind of switch where you can say, OK, boot from system A. And then you turn the switch over and say, now boot from system B. This is normally done in the boot loader. And this operation has to be done at the very, very end of the update process, as the last operation. Only there you can be true that your update was performed successfully. And it should be itself an atomic operation, of course. And you also want to security. You don't want someone unauthorized to be able to update your system. There are two basic ways to achieve this. The first of this, you use a trusted transport, like in workplace, for example, SSL, HTTPS. Then it does not matter, really, if you trust the update itself. You know it comes from a trusted source. This works well for the download case, but it will fail in the USB stick case. If you plug in a USB stick, you don't have easy mechanisms to trust the USB stick. So a more general approach is to put the trust in the update itself and sign the update. Then you don't have to trust your connection. And this will work both for downloading and for the USB stick case. And another thing is that you also don't want to inattentively update a device with the wrong software, for example. You don't want to turn your toaster into a printer by an update. And one could think, OK, I just check the hardware if it's the right hardware I write to. But this will not work in all cases, as you fear. Basically, from the out of view, the same hardware. But it works as a different product. So a very simple and most times the best approach is to use an explicit identification that you place on the target and on the update. So here, this is PyTos 2000 and PyPrime Plus. And if I say it's an update for PyTos 2000, then it's clear it's for this. You can check this and not for this. And another task that your update system has to handle is that you have to find out where you update. So you have an image B. And you have to say, OK, is it a bootloader image? Is it a bit stream image? And other image, is it for the first? Or it's for the second redundant partition and the same for the root file system. And this works only if you have some kind of configuration, either only in the update itself or in the update and the system itself so that you can match these partitions. And there are a lot of other things you have to take care of. You have to do good error handling to be sure that nothing in the update process failed. You have to think about, do I store my data? Do I store it in a single partition? Or in a redundant partition, this matters if you deal with fallback or data migration because the system A and B use slightly different data formats and watch the handling and much more. So as a conclusion, you can say updating is not that easy. And this is why we developed Raucy Update Tool and firmware. So short overview, it's licensed under the LGPL 2.1. So it should be easy to use it in your application or in your product. It's a C code built based on G-Lib and packaged with autotools so it should easily build. And Raucy, you can compile as a host tool. Then it is used to create, to sign, to resign, and to inspect updates for your device. Updates in Raucy have called bundles. And you can compile it for the target. Then it operates as the update service on the target that controls the entire update process, the installation, communication with the bootloader. And it also provides a small comment line tool that allows you to easily install a bundle from the comment line or inspect the system or updates. Here's a short overview of how bundles look. You have your images and bit streams, for example, that you want to install. There's some information extracted and put in the manifest for verification. And these all are packed together in a squashFS. We have used a squashFS as it provides good compression and as it is mountable. So you don't have to unpack data and store data capacity. And these are signed with X509 signatures you know well from SSL. And the idea of a bundle is that the bundle describes the entire intended state the system should have after the update and then the Raucy service takes care of how to achieve this. This is a bit about handling in Raucy everything that could be a target for installing something we called a slot. And then you can create in your configuration classes of slots, for example, application file system, a root file system, and instances of the slot. And this is used by Rauc to match what you have in your update bundle where you have an image for a specific class to the real device on your target. That's one of the core algorithms that Rauc provides. Here's a sample configuration. This one is the system configuration. This is normally stored on the target's root file system. We have some basic information about the system, about keyring. And here define the slots that you may update. And then you have here the update manifest. You can here see the compatible that has to match between the target and the update itself and the image description of those images you want to install. For example, here these root file system image would be installed on one of these slots, depending on which is detected as currently inactive by Rauc. If you want to reuse Rauc in your next device, we have a Yocta integration in our meta-pdx layer, which will soon become meta-Rauc. We have pdx-dist integration currently only on the mailing list, not in pdx-dist mainline. And you can also integrate it manually. And here's a short overview to give a basic idea how it's used. This is the Rauc bundle command you use on your host to create the bundle with a directory that includes all the content you want to bundle. And on the target, you simply execute Rauc install. Or if you have an application, we have a D-Bus interface where you can trigger the installation from. So let's check the time. How much may I exceed? Two minutes. Two minutes. The previous talk was exceeded, too, so a bit, I hope. Yes. Thanks. So the demonstration, you may all see are these are six Raspberry Pi currently running a pdx-based image. And this includes the Rauc update tool. And I try to trigger the update before continuing the talk. We don't see your desktop. Yes, I know. But I'm not sure if they. So these devices are currently connected to a Hockpit update. Now I know what you're talking about. So I shortly described the setup. We have currently a pdx-based image running. And these devices here are connected to a Hockpit software deployment server. And on the devices, the Rauc service is running and a little Python script that shows these small pdx-disk images. And the presentation, or the demo, should now show that I deploy a Yocto update on these devices. Yeah, this is what I wanted to show here. But I think this setup did not entirely finished because of the irritation in the beginning of the talk. So if you want to see the demo working, maybe you would like to join me after the talk. I try it in a few minutes. Yeah, this is the basic layout of the root file system on the Raspberry Pi. You have an easy redundant system, and the barebox boot loader with its boot state framework running on it. And yeah, this shows that the update system is really independent from the integration. And it can easily be integrated into two different needs, and it's generic. So it is designed to be adaptable to exactly your needs. I can't switch to the next slide, but it should be the conclusion. So I have to try it manually. So what are the basic facts that we have learned? Updates, especially on embedded devices that are connected, are mandatory. The second one is that updates are not really trivial. Some pitfalls I've shown, and some more I have not shown yet. And if you do updates, only install, well, tested software sets. So this is the best way to not break your device. And, of course, use free and open software update frameworks. For example, I give you an idea, Raoq would be a good choice. So thanks to everyone. And I think the time for questions is come directly to me, and we will discuss it.