 OK. I think I can start. So hi, everybody. Thanks for coming to my talk about Updater and evolution of Updater in embedded Linux and Internet of Things world. Just a couple of words about me. I'm a freelance embedded developer. I'm a maintainer for NXP. I make a processor for the U-boot loader. And I am an outdoor maintainer for the SWUpdater, who is one of the open source projects as updated agent. So first of all, if you have checked the title, I put the OTAs over the air in parentheses. And I want to explain why. Because really, everybody talk if I speak about over the air Updater, and I want to explain who we should really thank for that. Because when someone, a customer, call me and want to know about something about SWUpdater, he ask me, but it's over the air. Because over the air is like the keyboard. But really, this is not a feature of an Updater. Because most of the feature are part of an agent. But over the air is not what the Updater do. If we have to think someone, we have to think who provide the hardware, so the chip manufacturers, all people who write drivers in the kernel, and they all infrastructure to make this possible. So because at the end, this makes possible to have an IP address in our system. And what SWUpdater does, and what all Updater agent do, is just one simple thing, so very open socket. So using and naming as over the air Updater or just Updater is not a feature of an Updater itself. So it's more or less a marketing name. So if this is an marketing name, I have found a new one for the next one. So SWUpdater, I can say, is also a UTWUpdater, because it is an underwater Updater. It's put on this kind of device. This is an underwater drone. It's not to play. It's to measure the quality of water in the seas. I confess, I have no idea if they try to update the system when the drone is underwater. But in any case, it's also something to say that over the air is not really feature for the Updater. Feature for Updater is something else. For example, if we are talking about a limited bandwidth, very important feature is if the Updater is able to resume a broken connection. I have had a project where the product was sold in countries where there's just the GSM network and also unstable network. So the Updater can take a very long time, but it's very important that a broken connection does not make that the update is starting from the beginning, as the Updater has no chance to come to the end. Or, for example, delta updates to reduce the size of the downloaded software. As WUpdater has recently a diff handle. This is based on the LibR sync. Of course, it's not a solution for every case, because with the LibR sync, you can get a small delta if the difference are not a lot. If you, for example, want to make a delta with a two different version of Yotto, you get a delta that is much bigger as the original one. Or to get a better compression. Or, of course, in feature for an Updater agent are a lot of security feature, like sign image, encryption, many others, no downgrading. Also, it's important if the Updater has survived the audit from some security companies. It's difficult to get, because really, the security companies make a audit on the product, not on the single piece of software. But in any case, when I get the result, I get the report from these companies, I could at least fix some security leaks that was found. An important thing, so something was changed in the last time, companies now are aware that they have to update their system. So they don't ask anymore why we have to update, and they discover that they have a lot of software on their board. So it's not anymore to say, OK, I have just kernel, utefes, in so on. Of course, they have also, but there is a lot of software that needs to be updated to. First of all, of course, the main software, kernel, utefes, put on an embedded device, like an end, UB volume, MSU, whatever. But we can have even a different storage. For example, a small SPI storage, whereas the bootloader, there can be a rescue system, there could be some other software, and the updater must be able to update this device as well. Or some other storage, like EEPROM, to store data, calibration data, whatever. Very important thing is how to update an FPGA. Really, my preferred way is to use an FPGA manager, because the FPGA manager. From both alter and silings, I found their way into kernel mainline. But this means that really the bitstream is part of the utefes and is downloaded into the FPGA when the application starts over in the system boot. But there are cases where this is not possible. For example, if we need the VFPGA at boot time before Linux starts. In this case, the FPGA is connected to a small device, in many cases, an SPI device, if you know the silings as a golden image and can boot data from an SPI flash. But it's also an important thing, an important part of software, and the update should be able in some way to connect to the SPI flash and to update the bitstream as well. Another possible software in the system is if there is a microcontroller on the board. For example, to some real time part of the whole software. Microcontroller are often connected to the main processor via UART or via SPI, and the update must update the firmware as well. In SW update, there is currently just one handler to update a microcontroller. There are some other cases, they are not published, they are not in the repo, because many customers are using Lua, as the predictor is a Lua interpreter, and they use Lua to provide a handler and to update their microcontroller. And not only it's a thing of just an update, but also other projects are found that they need someone to update their stuff. The Civil Instructure Platform, CIP, is a project under the umbrella of the Linux Foundation, and has the goal to provide a very, very long term support platform. For example, for Kernel, they plan to have 10 years support. And of course, if they upgrade their stuff, they need some way to make an upgrade. So they have chosen an update agent, they made a comparison, and they choose SW update as update agent. And I don't want to say which are the good points, why they choose SW update, but just the bad points, because the bad points is something that can be solved in the future. This is a slide, I copied as it is from the CIP project, you find at the bottom there's the link, and there are also some slides regarding the comparison with other update. They say about SW update, how to manage updates from version 1 to version 5. It's quite a critical, really what they meant is they wanted to have a delta update that works with any kind of version. As I said, SW update is a Libre sync, but with Libre sync is possible or realize a small delta just if the differences are small. The second bad point is related to the U-boot library that is linked together with SW update, and really this is not an issue in SW update, but it's an issue in any project where there is the U-boot bootloader and you want to change the environment. The thing is, most nowadays, the initial environment of U-boot is linked together with U-boot because many find very simple, they install just the bootloader and system boots without setting an environment. This has the drawback that when the system is up, you are in user space, any system or any program that wants to change the environment has no idea which is the initial environment. Should search inside the flash where U-boot is, but of course is a hack because any different device has different starter des. The way it was done until now is to try to synchronize the U-boot and U-boot library. The U-boot library in Yotto is the U-boot FW utils and they must be compiled with the same commit, the same version, and for the same machine to be consistent. You understand it's very simple that they get out of sync. It's enough, you have your BSP in another layer with just U-boot, you take the U-boot utilities from Pocky, and you have a mismatch and when you use the FW September in user space, this can break your environment and your board maybe does not boot anymore. But this was solved with another library. This library is independent from hardware. It takes in configuration file as most application on Linux distribution and with it I can say that the second issue is solved. So we are seen in many cases, in many presentations about the update of a device. What about update of a system, of a whole system? I show you a real case, so I call an update in the internet of big things because this is very huge, you have to imagine this one is an engine, an engine for ships, ships, cargo, so something very huge. And each device, there are more as what I put on this slide, each device controller just a part of engine. There are many of them. There is a token ring. At the end there is an API address, so in this private network, I drop also middle controller that are using the project. The communication with the reward is done with gateway. Two gateway to have a redundancy and these are connected to the network. The outside, so the operator just sees one device. So this is just one engine and just one device. So when an update is coming, he wants to update this whole device in one shot. So there's many requirement from the customer because they say, okay, we have the same images for all devices. The gateway has a different hardware, but there's a hardware detection to understand which service will be run or not. When the whole network is updated, the software to be downloaded should be not, let's say, much bigger as the original pushed in the single device. Software must be streamed, so device controller has no place to make some temporary storage. And of course, this implies that all devices must be updated in parallel, and they think that the updater is just a building block. So they say, I need an updater, this must do the whole work. So there should be no dependencies with the application. So the application does not inform the updater that how many devices and the network, the updater must discover the device connected to the network. And most important thing, it must update all the device and just in case all devices were successfully updated, it can set that the network updater was successful and initiate a network restart. So all devices are restarted at the same time with a new version. And mainly for development, they want also to have a single updater, so it should be still possible to go to the single device to just to one of these device controller and to make a single updater. This was not enough because think about the ship is over the seas. Of course, nobody wanted to start an update, but let's see something gets broken. So a device controller gets broken, now is coming a mechanic, so not an engineer. He can replace the broken device with a new one, but you know what happened with spare parts. So which is the release on spare parts. I will say it's a random device, a random release. So in the best case is the current release when the device was manufactured. But surely it's not the same release on the rest of the network. And as a requirement, there's just one version because there can be compatibility problem. So the device must run the same version. So in some way, the updater must understand this and must install the same version that is running on the rest of the controllers. This, of course, is not possible if the software is not stored somewhere. We have no network, nothing. On the device controller was not possible, but fortunately there's space enough on the gateway. So the gateway could, and this plan, to store the incoming software that should be then installed on the single devices. So I started to split because you see the updater should work in different way, in very different way. I split in the service because when this map into system day services, I have just SW update service. This is the user update. So it's the updater of a single device. When I have a system update, this is running just on the gateway and this is responsible to forward and to install all software on all devices. For the replacement, hardware replacement, I added a pull update. This is responsible to check the version, to check the network version and if necessary, to install the correct release. So starting with update of a single device. This is what is in SW update. SW update has a meta description. This is called SW description and with it is possible to describe for each artifact, how to install it on the device. So this is a simplified version that you see for the production. It uses two copies, so there's the running copy, the standby copy on the project. There is also a rescue, something is going wrong. In any case, this description can tell to a single updater how the software must be installed on the single device. SW update then runs on the single device with integrated web server and in this way is possible to push a new software to the device. Now away is from the single device how to push on all devices and I find this way, so practically on the left you see this is the update package, so SW in SW update for the single device. On the right it's a compound image. This has payload just the SW of a single device but it has a new description because we want to install this payload in a different way. We have to forward to all devices and the description is something like this. So at the beginning there's the name, we get the controller, so this means because it was another requirement, the image for the compound cannot be installed on a single device, so if this is tried, then there's a hardware compatibility check and the update is rejected. Then on the images you see this is the name is the SW, so the update package for the single device and there's a handler, handler is the way in SW update to install some software. It's a new type, a type is forwarder. This handler makes two things, so takes a list of URL in these properties and for each of them make a connection to the web server running on the device. The web server has two interfaces as a rest API, the rest API is possible to push an image and as a web socket interface. With the web socket interface it's possible to retrieve data from the update running on the remote device. So it's possible to know how many steps are running, which percentage and the most important thing is the update that was successful or not. This was not enough because then we need a detection of topology because when we build in Yotto this is empty because we don't know how many devices are in the network, there's an engine with different number of controllers, there's an hook in the description when the hook is hit, when a lower script is running this does nothing more as try to ping all devices in the network and then fill this properties field. So when this handler is running it has a list with all device belonging to the network and which device must be updated. So this is running because there is also a chicken egg problem because the gateway must install all devices in the network but must install also the software on itself. And this was solved using allowing it to have a two instance of a step update on the same devices. The first one is the gateway manager, the compound image the other one is exactly as the other one on the single device and the only thing is the possibility to provide and to output of incoming stream on something, some place on the file system. This is used later for the pull update. So at this moment so when an update is coming they are updated here so the SW for water make the two connection and with a web socket connection it can retrieve if the update on all devices was successful. Now we need a decision and decision is done with an external process a restart controller the SW update the gateway instance forward all information regarding update to the restart controller the restart controller gets the list of the devices and get if the update the network update was successful. If this is the case it starts to make initiate a connection to all of the devices and the rest of the API send a reboot commander. What? So we have solved the network data of the system of data. What happens when we replace hardware? When we replace hardware the update should work in another way. So first of all should check if there is a gateway should check if there is a SW so there is an update package download the first kilobytes because in the first kilobytes there is a meta description you can check the meta description and can extract the version if the version is the same as the running version then skip the rest if not then first download everything and run SW update in driver mode so that means without install anything this is necessary because a previous update was interrupted you have on the gateway a corrupted file so running in driver mode you know if works that an update will be successful and they face when start and update from the gateway customer was happy and they say it's okay but I can use also to send the configuration a bunch of files a lot of files and of course this can be done exact in the same way the only thing is that at this point the update SW update should know already at the beginning if we are to store the file we should not overwrite the file used for software so we have a file for software and one for configuration this is required to add a pipeline to the whole process because the first part of the stream is then passed is verified with a public key and a output parameter is checked and this give the output file where a file must be stored you see this is quite different because now we have a tab is an archive, a compass and so for the device is different for the compound is exactly the same and this way we can also install a configuration and not only the software as we see there is a different way to where an update can work another use case is during a factory so let's say to production and first install of software on devices this is also a real case so let's say we have a manufacturer and they decide to make a hardware assembly not himself but in outsourcing another company another country where is cheaper but the manufacturer doesn't want to provide everything so it doesn't want to provide for example private keys or some other important stuff they want to make a functional test themselves but the assembly of the device so device is assembly and just a rescue system is installed on the device then device are sent back to the manufacturer and the manufacturer just put all device in its own factory network the update so as the update starts and knows can check I am now in factory mode in look for a fixed IP address search for the software install the software after installation switch the device in production mode when in production mode it works in a very different way for example the update work and is a retriever from a backend server so I can say the update is in a different way regarding the life the different time of life of your device the device is born and then they are just a rescue the device grown up it becomes a product and then the updater is also put in grown mode and will work and updater in a very different way I just put as reference is not a feature of the updater in many case the devices are not connected to the network but there is a gateway in the between is not a feature for the updater because at least in my experience the software for the device is just put on the root FS and the device itself get from the gateway maybe in another way like with M2TT or some proprietary protocol another thing was changed in the updater is like the updater is unaware about which server is used so for example I mean, which back end server is used at the beginning it was compatible just with the hope bit I get another project where there is a proprietary server they use a very different interface really they misuse the HTTP error code to provide if an updater is available or not in any case the code in STI Predator was changed to make an API for the single connector for the server and it's possible to have in future more servers so more connector connected to different server this on the left is the client to the standard server and they communicate with the main tasks of the updater via NIPC so this make very generic and can be used also to provide privileged separation because this process runs with different rights as the main process with root rights because it has to install into the hardware it's not enough I had also many keys where a custom protocol must be used so custom says okay it's nice the protocol is HTTPS it's also nice but there's already a well defined standard we have to apply this standard we cannot change or for example the whole project has already provided a download method and the project is finished or it's old project and they don't want to change just because there's another solution also for compatibility reason they have previous generation of the device working in such a way they don't want to change the way the customer are using to update the software this is solved in SW update with a library there is a client library this is now less GPL version 2.1 so it's possible to link proprietary code the interface in SW update is exactly the same as in case of the open server so there's the IPC in this way is possible to link directly to a proprietary application and to push the software to the installer before I fly to here I got this request from a customer I have no idea if it will be then realized in future is just another interesting way to provide an update they say this come from a little vehicle charging business so they say we have already a protocol the protocol is OCPP and it cannot be changed and of course many other vendors have many variants so many devices each device has different way to work for each of this way of work they need a different software but there is a cross vendor server and on this server the software for each vendor is stored but the administration of this backend can manage update file on vendor basis not 25 because they have 20 devices not just one file this belong to one vendor so the request here is to have a single file with all possible software this is was already supported by SW update but the whole software must be downloaded so SW update is able to discard the parts that are not interesting so two different device they download the same thing but one install some artifacts and the other one some other of course in this case is very bad for the device because the whole software must be downloaded and it could be very huge so the request is that the update should be able to skip the parts that are not necessary for itself and just download what is necessary it's something I have to think about what is also coming in the last time is connected to the continuous integrity is that the software must be installed as soon as possible in the development so it's important that all developers run with the same software if you we have here 10 developers I think we find 10 different ways to install software the device to test someone with U-boot with TFTP someone with kernel or whatever the result in many cases is I get an issue the other developer no cannot see the same things and at the end we check and we are running a different version something is not exactly the same so if the update is just part at the beginning of the of the software of the development phase then it's possible to install the same software on all devices and this is also a case a test that is closer to what happens in the field so we can say that an update process is very very well tested before going to production is that the same thing so many customer using Jenkins to build the software personally I like to use Directed BitBake T-Boat is a test framework developed by my friend in Denk's Hikershocher and it's a Python framework so it is possible to write test cases I have a set of test cases to install new software and to perform some basic test to see if the new release is working or not in the past there is just one or two way to build together with SW update so personally I use in most projects Yocto but there are also some projects we build with there are many projects even with Debian so VAP data should be buildable in all of these different build systems Debian is quite recent I provided a Debian package and now not understood at the beginning how Debian works I found a mentor to push our directory into Debian it was not in time for Buster was taken in experimental for the next release so as summary just short in my presentation VAP data does not work always in the same way depending on the hardware depending on the project there is a different way and also in different time of the life of the product VAP data must work in a different way and VAP data itself should be also nowadays quite independent from build system from backend server from other part should be also able to change some of this part so I think I don't know if too late I'm open now for some questions no questions thank you very much