 Hi, well, my name is Silvano Cirujano-Cuesta. You can call me Silvano. It's easier. And I'm going to give a presentation about firmware updates for IoT-like projects. I'm a software engineer at Siemens. I work for corporate technology. Corporate technology is the corporate central technical unit. And I'm part of the embedded Linux corporate competent center. We are a team with a lot of open source software contributions with people. We have four speakers in this event. And yeah, apart from that, I'm interested not only on firmware updates, software updates in general, but also on trying to bring the embedded devices to write the containers wave. Well, let's see. This is the agenda for the why updating, an overview of the architecture, the components involved into this solution, and how the firmware updates are done, and a conclusion. So I have here down here, you can see where we are. Each slide. So I would like to present. It's an open source software solution for firmware updates, remotely controlled. So that's why IoT-like projects. And I would like to present this solution. But also I would like to show a success story of a contribution from company into open source software projects. So right now in the industrial domain, we have typically devices that are not connected or in isolated domains. And there are quite often on-site updates, just with a USB stick, for example. Some of these devices have really long life. Sometimes they are difficult to reach. For example, this wind power generation offshore devices. And typically they don't get that often updates. But the internet of things is also coming to industrial domains. So the trend is showing that devices are getting more and more connected, industrial products in industrial domains. And the number of devices getting connected is exploding. It's exploding. It means that you cannot any longer manually update each one of them. So you need to remotely manage, update, remotely manage. On the other side, we have there is an increase on the attack surface due to the network connectivity that we have. So we need more software updates in order to fix those security issues. Of course, there is always a back to fix. And we have in industrial domains, people have a big exposure to technology, smartphones, a tablet, IoT at home. And they have some expectations also for industrial devices, like that it's easy to update the devices to get new features and to fix bags. So in the end, there is a good reason for updates. Sorry, I didn't say it yet. Just if you have any questions, don't hesitate. Just interrupt me. So when we started working on this solution, we were looking for open source software components to get remotely managed firmware updates. We identified two open source software projects, Hockpit for the back end, and SWU Update for the devices. But they were kind of disconnected. So they were not connection between both projects. If you attended the previous presentation from Matt Porter, you already saw a spoiler. So we contributed to SWU Update, what we call Suricata, that was an extension to SWU Update to create a connection with Hockpit so that you can manage the updates on the devices from the back end. OK, let's see the workflow. I don't know, it's too small. Sorry. Well, anyway, I will explain all the steps. So we have, first, the devices are polling. There are many different possible implementations that I'm showing the one that we implemented. In this solution, the devices are polling, requesting for artifacts, firmware updates, and the back end replies either with no updates available or with a list of updates with download URLs. In the case of firmware updates, typically just one artifact, just download. So when we get firmware to install, then the device has received already download URLs, well, the one URL, and will contact either Hockpit, or we could also use another artifact repository. Anyway, repository where the firmware is and will download the updates, it will then process the update, the firmware, and it will report back to Hockpit if it's been successful or not. So that's a typical update workflow. OK, let's see the components that we integrated into there. We have SWU update. At that time, we couldn't find neither open-source over nor commercial alternatives to SWU update back then. So at that point, we had to make a decision, make versus buy open-source software. So we evaluated SWU update. It's open-source software. It's developed by a well-established player in the open-source software community, like the company Dengs. They are the maintainers of the U-Pod, the well-known boot loader, and they have experience in industrial domains. If that wasn't convincing enough, we had really convincing feature set, like, well, SWU update uses the running Linux system, the Linux user space to do the updates. So you have the full power of a Linux user space against boot loader updates. It's extensible, and it has a good integration with the mostly used boot loader, at least what we find the mostly used boot loader U-Pod. But it also supports or offers possibilities to extend to other boot loaders. It's butterproof software. Stefano has been already using the software for their projects. So in the end, it was for us an easy decision for buy the open-source software project. The back-end component, Hockbit, we also look for possible solutions, and we couldn't find any convincing alternatives to it, neither open-source software nor commercial. So we faced, again, a make versus buy decision. This software was developed originally by Bosch Software Innovation and was released as open-source software under the umbrella of the IoT Working Group of the Eclipse Foundation. It's written in Java using the Spring framework. So it's kind of, for us, embedded kind of a strange domain, so to say. For us, it was also convincing that Bosch is also in the industrial domain as Siemens. It has also a convincing set of features, not only present features, but also their working feature features that are really interesting for us. For us, it was important to be able to shift from device managed provisioning to remote managed provisioning so the back-end decides when can get the devices the updates. In the end, the devices can decide by themselves if they want to update or not, but first is the back-end deciding if it provides the updates or not. It's easy to integrate with other back-end services due to REST APIs, well-documented REST APIs. It offers the possibility of connecting directly to HotBit or indirectly using message queues. It also supports external artifacts repository, so these both, too, are important if you have a lot of devices and you don't want to put too much load into HotBit. And well, this project is kind of a young project and it's still working on some stabilization and adding some new features. But nevertheless, as I said, we couldn't find any convincing alternatives, so we also decided to buy this open-source software component. So these are the two components that we integrated. I have to say that thanks to both projects, because we were happy with what we got there, and let me get into the details. So let me try to define what I mean with firmware updates. So I'm talking about the base system that makes your system runnable without bootloader and with the kernel included into the firmware and with possibly other partitions with applications, configurations, whatever. So we see two extremes in the possible update strategies that there are in an IoT domain. And between them, there is a trade-off between time, so application downtime, and storage space that you need for your images. So one of the extremes is one firmware partition and the other one is two firmware partitions. If you have one firmware partition, you have the worst application downtime. But the best storage space, so you require less. But if you have two firmware partitions, then you have the best application downtime, but the worst storage space. It's also possible to kind of get hybrid solutions by reducing the firmware. But if you reduce the firmware enough, you would get almost to the situation what you're having almost the same storage footprint in both cases. So then you would decide for the two firmware partitions due to application downtime. Let's see the two firmware partitions. That's, in our opinion, the preferred strategy if you can afford it. So with this strategy, you can avoid breaking your device in a sense of losing connection from the back end to the device. And it offers a rollback possibility, one version backwards. Well, the colors are kind of strange looking here. But I prefer to use instead of the typical firmware 1, 2, or A, B. There is the green-blue deployment pattern. And this should be blue and this should be green. So the idea is that both firmware partitions are equivalent. You can start any of them. You will have different versions, but they are equivalent. There is no difference between them. If you would install the same version, then it would be the same number of which one you start. And in this strategy, you have short application downtime just during the reboot of the system. Another advantage of this strategy is that you can cancel an update due to manipulation in the firmware image or an error in the download of the image or for whatever reasons without losing a working version. So the previous, you would get a rollback to the previous working version. Let's have a look at the update itself, the process. So SW update would be pulling the back end, asking for updates. And then Hockpit would reply, OK, here you can get your firmware update. We are running this firmware version. Put loader configured to start it. And then SW update will start the download from Hockpit, from the firmware image, and will flash it. You could also download it to another partition, but I will show you just the use case where you are just directly flashing it. And then SW update will check the firmware, check some and signature if it has been signed. And if everything is fine, it will switch the boot loader to the green firmware partition. It will be a temporary switch and will reboot the system. At this point, we're starting to get an application downtime. When the system, well, a watchdog will get activated. And if everything runs fine, then when the application, when all the whole update has been successful and the application was fine, then the watchdog will be deactivated. And the switch to the green firmware, to the new firmware, will be made persistent. Supposing that we got here an error in the download and we flashed an erroneous firmware, then SW update would realize, OK, this version is wrong and would report back to Hockpit, OK, error in the update. So Hockpit would show the error and the operator would have the opportunity to retry. Or if there is an error in the firmware and the watchdog don't get deactivated, then the system would switch back to the firmware version one again. So I think that it's a scenario that you have seen in some other presentations regarding firmware updates in this event. And you will probably see it again. It's the preferred scenario. Let's see another one. If you don't have enough storage to have two firmware partitions, then you can get a lightweight Linux with SW update to do the updates. In this case, you will also avoid breaking your device. So your device should be always reachable from the back end. But there is no rollback possibility. The storage footprint is just one firmware, plus this partition that we didn't have in the previous case. And the downtime is relatively long. We will see later when we see the whole process. And in this case, the only rescue when you flash an erroneous image is that you stay in a recovery model from this system. As I said previously, if you reduce your firmware, so typically, this partition will be smaller than this one. But if you reduce your firmware, that match moving applications and as much as possible into these partitions, then in the end it could happen that they are almost equal and big. And then you could just take the two partitions strategy. OK, so in this case, Hockpit. So in this firmware, we would need at least some functionality contacting Hockpit to get notified when some updates are available. And the boot loader would be switched to this update partition. And our boot would be triggered. Then, well, at this point, our application downtime started. At this, OK, now we started already in our update partition. And in this partition, SW update will start downloading the firmware image from Hockpit and flashed over the previous existing firmware. In the end, SW update will check the checksum, the signature of the image, will switch temporarily to the firmware, and we will reboot. If everything runs fine, the firmware will deactivate the watchdog and will make this switch to the new firmware persistent. At this point, we have our application already up and running. Again, so as you can see, the downtime that we have in this case is much longer. Not only that, at this point, if you are flashing nonsense, you don't have SW update would notice, OK, something is wrong with the image that I downloaded and flashed and wouldn't restart here. But nevertheless, you don't have any application running here, so your system is just in recovery mode waiting to get a new working update. So these are the two extremes, in our opinion, in the IoT domain. There is, in SW update, there is another extreme to get less storage footprint, but in that case, it's a kind of SW update is kind of ragging the carpet from under its feet. So just within it already and flashing its own partition. And in that case, you wouldn't get any recovery models contacting the backend. So that's why, in our opinion, that's at least for us in these IoT domains, it's too extreme. So kind of getting to the end faster than expected. As a conclusion, we got a 100% open source software solution up and running. We got into the communities. Both communities welcome us. We got contributions into SW update. We are working on getting contributions on Hockpit. And they are both projects. The communities are happy to get involvement. And that was for us something important, not only getting open source software that you can modify and fix, but also getting into a community and active community. And that was, in our opinion, a success. Currently, our focus is on firmware updates. But with these components, you can also get in general just software provisioning applications, containers, many different scenarios. Although we use both of them, this is a modular architecture. You can take Hockpit, you can take SW update. And in both cases, you have different possibilities to connect them to other clients or other backends. So we were also happy to be into communities that can, although, that are active by themselves and that have their own lives and external other contributions. In the future, it could happen that we integrate other different backends or, I don't know. So the future features that we are working on or hoping to see is the possibility of splitting the preparation of the update and the realization so that first the update gets, in the case of two partitions, for example, gets downloaded, flashed, but not activated in order to fit into maintenance windows. So devices that cannot be updated when they get the firmware. That's one feature we are working on. And the other one is being able to synchronize the device and the back end so that you can, for example, update a device just with a USB stick, for example, because right now, Hockpit is expecting to be the source of truth. So when Hockpit says, OK, you device, you have version two, there is no way to convince Hockpit that that device got version three already. So that's another feature we are working on. And we hope to get in the future also asymmetric key signatures. And that's it from my side. Any questions? We have plenty of time for questions. Yeah, you activated. You activate a watchdog that you have to keep alive. If you want, I mean, you have to reactivate the watchdog until you are sure, OK, everything is fine. And then you can deactivate it. You can have a short watchdog. And then during the bootloader activated, when starting, activated the application. So refresh it, so restart the time out. And in the end, you would deactivate the watchdog. If in some, so the watchdog, you say, OK, watchdog for, I don't know, 30 seconds. If your whole system can start in 30 seconds, then you perhaps you don't need those refreshes in between. But if, let's suppose you say 10 seconds, and your system needs half a minute to restart. Kind of long bet. And then you would be refreshing the watchdog regularly so that it's not get triggered, because the watchdog would reboot the system. So in UBOT, you say that's something perhaps like hardware, the hardware, a hardware for a second. But you are talking about a kernel watchdog, or a hardware watchdog. A hardware watchdog? A hardware watchdog? OK. But in the moment that you reboot that, if you have only one, you don't need that watchdog for the application, because you are rebooting the application. So you can take that watchdog and use it for the whole update process. And in the end, you would reuse it for your application. The moment your application is already up and running, would deactivate that watchdog, so to say, would deactivate the watchdog, and make this switch to the partition persistent. And then you can use that watchdog. You already deactivated it. You can use it in your application. Your application, up to that point, your application is not running. You don't need that watchdog. Do you have some interfaces in UBOT to make that persistent? So we have some steady. So we say, OK, I don't remember the status that we have. But based on steady, you know, OK, UBOT knows, OK. It has been, are we in between of updating the system and something went wrong? And I have to do a rollback? Or it's already persistent status, and I can just boot normally? I'm safe, keep the new version, and rollback will also be again. But if you, for example, if you lose the update, you may have a running system, but it's not on the updateable system, and you also lost it. So this is something to decide about application data. When you actually say, OK, everything is green, it's over, and it's stable. It's good enough to stay at night. Yeah. Yeah, that's the point. Exactly, yeah. Exactly. Sorry, and how do you, sorry? We just took a UBOT. Just UBOT is a flexible boot loader that we have in a lot of our products, and we just, and UBOT integrates pretty well with SWU updates. So we just took UBOT. I'm not sure I understood the question. So typically, you don't want to touch your boot loader. So you try not to update your boot loader. That's a tricky part to update. So if you can't avoid it, you just get your UBOT, and that's booting your system and switching between the partitions, but that's it. It doesn't play any role. So HotBit doesn't contact your boot loader directly or something like that. It's just SWU update is using some existing libraries from UBOT to interact with UBOT to make those switches. So it's a already existing interface being offered by UBOT to interact with it, and SWU update, I mean, they have been developed by the same company, so they play well with each other. So you just use SWU update to make the switch in the boot loader. You could also update your boot loader with SWU update if you want. I think it's sort of OK. Yeah, yeah. Can you define this rules in the server? And the second question is about the security. Do you have any security mechanisms to change the data between the two? OK. First question regarding the rules, sorry. It's something that I didn't show here. HotBit has a specific single task. It's just making sure that a device gets notified that it should be getting, but it doesn't offer those kind of filtering mechanisms. So HotBit relies on a higher instance to decide, OK, this device should get this firmware or this artifact. So in HotBit, you should use another, let's say, a device management system or another system who knows, OK, this device can get this firmware. HotBit doesn't know anything about different devices, different firmware. So HotBit, if you say to HotBit, flash this image for Bigelbomb Black into a Raspberry Pi, we'll try to do it. There is no way. You can use tags, but for really small deployments, you can use the web UI and use tags, but it's not supposed to be used that way. There is that HotBit. You just say to HotBit, OK, HotBit, I want to get this firmware into those devices. You can also, HotBit has a mass rollout, so you can say, OK, try to start with five. If it works, then gets up to 10. If everything is fine, then gets up to 1,000 and everything, the rest of it. So that's the focus of HotBit. And making sure, OK, this device should be getting this version. Had this device asked for any version in the later time, so kind of a HotBit, which version is this device reporting to have installed, and all this stuff is the focus from HotBit, but no filtering possibility. And the second question was about the security in this communication, I imagine. So you can secure HTTP in our case, but you could also use, in HotBit, you have the possibility to connect adapters. So you use message queue where you connect your adapter. So Bosch, for example, it's using co-app adapters for that. And then you would need, on this side, to extend Suricata, probably, to support those protocols. They are not supported right now, but it has been written in such a way that you can easily extend it to support other protocols. And then you would take care. You can also use another communication channel where you take care of securing the communication. And there were three. Oh, OK. So if you use HotBit to provide different series of artifacts, and each one of them is a firmware update, then SW update would do an update, restart. I'm not really sure right now. Perhaps, yeah? On HotBit side, I mean, you have in HotBit, you can pack many artifacts. But on SW update, you also have the possibility to pack different updates together into one artifact. So depending on your scenario, I would say, try to pack your, so supposing you want to update a device with, I don't know, an FPGA and a firmware update and something else. And all the devices of this kind have the same updates. Then I would pack everything into a single SW update binary. So update, CPIO image. And then all the devices would be getting the same update. If you do it with HotBit artifacts, I'm not sure right now how we implemented it. If they are running in parallel or if it's sequentially, I cannot answer it right now. HotBit does some, I think that it depends on the implementation on this side. And check if there is an error on the firmware, or if there is something that went wrong with the flash. So my question is, how do you assess, how do you detect a bad flash? Check some. Check some of the whole partition or the files? Of the whole partition. OK. And that is built-in on Suricata and Office? In SW update. How can I do that? So it was already in SW update before Suricata. OK. Do you agree with human management? Like you said, you never mentioned it before. So we have to see if he's comfortable with it. I didn't say we'd never update bootloader. We tried to avoid it. We are trained in this project. We are trying to provide for Siemens framework, a stack, based on open-source software when possible. And so we don't have right now concrete scenarios where I can say, OK, we're updating the bootloader. But I would say, in general, you should try to avoid it if you can. If you can't, then you can use SW update. It's really flexible. It's powerful. You can update your bootloader if you want to. If you have to. It's possible. But I would recommend you not to do it if you don't need it, not just to get fancy features. Dual bootloader? Yeah? Yeah. That would be exactly. The right strategy, if you need to update your bootloader, if you foresee that you need it. Sorry. Sorry. So it depends on how you define where you do what you mentioned as commit the update. So you have to decide what is for me a successful update. And at that point, you commit that update. So for you, a right update, correct update, has been finished when your application starts and runs some tests and says, OK, everything is now fine. Then that's the point to commit that update. It's a decision that you have to take. Self-check? So right now, we didn't implement it. But you can do it. You can add into SLU update. You can extend it to add this test, this kind of check. OK. Let's check that I can contact the back end. And that's what you define as one of the criteria for a correct update. So you have to define it. So as far as I know, SLU update doesn't check that right now. And anyway, if it would, then we would have to implement what should happen in that case. Refresh of the watchdog or what should happen in that case. Or communicate to another component. So it depends on the use case. More questions? So I will repeat the answer. So that was Stefano Bavic, the maintainer and main contributor of SLU update, that you can use post-inscript to do that. OK. That's it. Thank you. Thank you.