 We're going to talk a bit about over-the-air updates, so this is typically how people develop connected devices or IoT devices. So you start off by prototyping, production design, and then you get into some kind of release deadline panic towards the end before you get into mass production. And this is also where you typically start thinking about over-the-air updates. And because you know that during this critical phase where you're almost there, you make some mistakes, and now you need a way to fix them. Of course, this is where you should be thinking about over-the-air updates and how to resolve them earlier in the process. So when you have a prototyping stage, you might want to test it out a little bit, or especially when production design, what kind of update strategy should you use for your application? It's really hard to retrofit these things. So when we see a lot at Mender, obviously, we'll make open source over-the-air updater. But what we see a lot is that people build their own, and I've divided them into three categories. By the way, how many of you have built your own over-the-air updater? Just raise your hand. It's kind of like one-third of you, and how many would like to do it again if you had a chance? One. Okay. So this is generally also what we see. People make it because they have to, and it's a panic, and things go not as expected always. It seems like an easy problem. But the first version is what I call the bricker, which is, it's very short in development time, and then you start to very, and see the effects of it afterwards. So you see that when you update devices, they lose power, they never come back online. They get shipped back to you and you have to resolve this as a support case, for example. So you start out with some bash scripts, maybe, and you work your way from there. Then you have the Honeypot, which is more security-related. So in this case, you might not be thinking about authentication when you download the update, for example. So if somebody is nearby, they're able to inject an update that was not supposed to go there. Maybe there's plain text protocols, and there's no signatures. Actually, the Tesla had partly this problem until 2016, when they got hacked by some Chinese research company. They did not have signature validation of their updates. So they were able to take over a Tesla. But it sounds like a basic problem, but even Tesla has this problem. And then you have the Needy Updater, which is the one that needs a lot of attention, because you need to do it one by one. So you have a device, and then the only way to get software to it is through a USB stick, and then maybe run some rescue mode and a shell script there. So obviously, this doesn't work that well. It doesn't scale that well. So it gets done less frequently than it should. And then why are these so hard problems? Well, it's basically because we work in a very rough environment in embedded space, as you're probably familiar with already. Devices are remote. They are expected to live for a very long time in the field. Maybe five, 10 years. For cars, it's even longer. Maybe 30 years. And then you have these factors that make it hard. So it has to work every time because it's remote and long lifetime. But then the power is unreliable. Maybe it's running on battery. Maybe there's a stupid user that unplugged it suddenly while you're doing the update. Why would he ever do that? And then what happens when the device boots again after that? And you also have unreliable network, a public network, often wireless networks that are insecure as well. So it's a pretty harsh environment, as you know. And based on this, we found some criteria that you should have as part of the update process. So it needs to be robust and secure. Integrate well with your existing environment because you probably have some way of building your software and developing your software already. And you don't want to replace all that just to get over-the-air updates to it. Easy to getting started again because we're often in that right bottom box there where you're in panic mode and you want to enable updates somehow. And you don't want to spend too much time on this. And then, of course, you have bandwidth consumptions, especially for 3G networks. Downtime during the update, this varies also by application, how long are you allowed to take your devices offline? If it's part of a telecom or a router network, this is very important, of course, to have it very short, but it varies a bit. And then lastly is the update server. So being able to manage updates across a lot of devices is also important in order to make it cheap to deploy a new update. So this is over the years after we've worked on this problem. We released the first version of Mender last year, so it's just one year old, the first version of 1.0. But we've developed this workflow that you should follow if you're doing over-the-air updates. So firstly, you do some kind of detection that there is an update available through a secure channel. You do compatibility checks to make sure the update is for this device. It works on this device, write architecture and so on. You download it, do integrity check, make sure nothing is corrupted. Check the signature, make sure it's authentic. You might also decrypt and extract it depending on if you need confidentiality in your updates. And then usually you have some pre-install and post-install actions, but the most important one and the one that you only think about when you're in panic mode is the install, because you can see it's actually quite a small thing compared to all the other things you have to do. But after you're installed it, you need to restart it. You need to do sanity check. Does the device work after you did the update? So one thing is if you get Linux to boot but do your applications work as well. Because otherwise it will be pretty useless to the end user. And then lastly is maybe most importantly what do you do if it doesn't work? So there will always be some way to resolve it. Maybe ship the device back, but that's a quite expensive way. So what we advocate at least is to have some way to do automated rollbacks. But that depends again how you do the installation. So some ways of installing, for example, RPM packages and things like this, you cannot rollback in general. So quick intro, how many have used Mender, although you, oh it's quite a lot, four, five, six, maybe one third. Sorry, you looked at it. How many are aware of how it works? Maybe half of you. Cool. Welcome back and welcome to the rest of you. I'll give you a short intro, what it is. So it has a client on a server. So you have this management server at the top here. And you have the devices might be a bit small for you, but it's on the right answer. I guess for you as well, yeah. So it's on the left hand side, the devices, and they will regularly pull for updates to the deployment server. And they will, if there is an update that's ready, they will get the answer, get this update. And then they will install it, as you can see at the bottom here. We have several partitions. So this is to enable the rollback that I mentioned in the previous slide. So we have one partition that's active and running. That's where your system lives currently. And then we update the other one. And why do we do this? So this is because if you lose power, for example, in the middle of the update process, you will still get back to the running partition. You wouldn't have altered anything of your existing system. And then you can try again to install the update. But if you have only one partition, then when you install the update, it might get half done, and then you cannot boot anymore. So this is to enable robust updates. And yeah, you can update anything you put on the root file system, including the kernel device tree applications. So maybe we should do a quick demo. Now I have not started it, so it will take a little bit. But you can also try it yourself if you follow the getting started. We say it will take less than one hour to test it out. And so far it's held through. So you can create an account. Okay, so it's starting a virtual device here. You can see there's another client there as well. So right now it should be running on my local machine. Right, it worked. So this is the user interface that we have. It's running currently in demo mode, which means that you cannot use it in production this way because it uses a shared key among other things. So it's very insecure. I hope you cannot connect to mine right now. But we do also have guidelines how to set it up properly. But there's a couple, there's three main areas to look at. So one is the devices, then you have the artifacts, which are the software basically that you want to deploy. And then you have the deployments where you kind of combine the devices and the artifacts. Let me see if I can upload one. So how many of you use Yachto project for, okay, a lot of you, maybe 70%. So we do have a meta layer for Yachto. So if you add that to your Yachto build system it will output a couple of extra files. So one file has the suffix.mender and this is what I just uploaded. So you can see it here. But this is basically a read file system and it contains some additional metadata, like your checksum. The checksum, we need that in order to be sure that we got all the bits, the build date, the size and if it's signed or not. So it's basically a raw read file system with some metadata around it. And this gets automatically built. If you use Yachto, we're looking to expand that. I'll talk a little bit about this later. But there you go. Okay. So there's one device. This is started as part of the demo setup. It's asking you to approve it because I haven't added any trust about this device before. So the device has the key of the server, but the server doesn't know anything about the device. So there are other ways to approve this device. You can preauthenticate it also, but if you haven't it will just ask you. So we provide some inventory information, but most importantly the current software that is running. So it's got a name, the software that you have on the device already. You can think of it like a version, but you can put any string there or revision. And then we also collect some additional inventory about the device. And this is completely configurable. So now I have a device and I have an artifact. So the last thing, you can see this pop-up side, just ignore them, but they will try to help you. What to do next? So we can create a deployment. We can use the artifact that we just uploaded, the root file system. And we do support grouping. So you can either do it to all devices or you can create a group. But it doesn't matter much right now since I just have one. But this is the basic workflow and you can see the reports as well afterwards and the status while it's ongoing, what is happening. So we'll tell you how long it takes. So this will probably take a couple of minutes. So we can resume. I guess it's not that interesting to watch a progress bar. But I'll switch back to the... We can resume here. Do you have any questions by the way so far? So you're asking about tiering the updates. So you have an internet connection down to a building, let's say, and then you have fast connection within the building. Yeah. We don't have anything specifically to support that, but we use HTTP. So it should be quite easy to set up a proxy, HTTP proxy there. I haven't experienced or experimented with it myself, but it should be quite straightforward. Okay. So the question is how much additional overhead is required on the client side? So part of this is shown. I can maybe submit in. So I would say the biggest overhead is that you have to have two root file systems. So there is some storage overhead. It depends how big your root file system is, obviously. We also have a data partition. So your root file system could be 50 megabytes and your data partition like a gigabyte or something. So it depends on what you want to update. But I would say that's the main thing. And then the client itself I think is about eight megabytes or something like that right now. Right. So if you already have an updater, is there an easy way to transition into Mender? So it is not trivial because we use this partition layout and that's the hardest part. And if you have a live system running and it doesn't have the partition layout, it is a bit tricky. I've heard about, I've not tried it myself, but there is a tool for this where you can actually change the partition in a running system, but it's risky business. So unfortunately, it's a bit tricky at that stage. Then you're good. Okay. Right. So if you only want to do updates, say at night or something like that, how can we support that? So we have something called state scripts, which is partly covered, I guess. It's a bit simpler or more generic version of the pre and post install actions. But we have the ability to insert scripts between each stage here. So you can have a script before detecting the update. And then you can implement any logic there to say if time is in this interval, skip this or just retry later. Did I make any sense or? Yeah. Okay. Three months ago, it should have been there. I think it was in 1.2, so maybe around September. Yeah. So that's the first state script. So there is a documentation section about it. But if you're... Yeah. So there on the client side, there is some docs about it under artifacts here. But yeah. So there's a lot of states and then you can add scripts before and after the states. And one use case is to control when the update happens using the scripts. So that's one way to do it right now. We do want to look at also schedules of updates on the server side, which I think would be a better solution for you. But it's possible using the state scripts as well. And then you can also use the local time zone of the devices to know when to deploy the updates. So there's one like C specifically. You probably should do one before artifact or before download. I have to think a little bit about where exactly to place your check, but it's possible and it's documented here. Yes. Yeah. So there's a return code that you can... You can do that too. So you can... There's two ways or three outcomes. So you can say continue or you can say fail and then it will abort and roll back. Or you can say retry later and then you can configure how long until next attempt basically. So it's possible. Yep. We have plans to support them, but we don't do it yet. So the only way to do it currently is with this script. So you can couple like update to the Linux device with updates to other devices that are close by. But we are planning to implement better support for that probably next year, I would say. Okay. They are requested. So there's no open port on the devices on the clients themselves. And they go over HTTPS and they ask the server. They hit the specific endpoint and they check and they provide their identity and then the server will respond whether or not there is an update. A partial license. Yeah. So a partial license is for everything. So it's fully open source. Yeah. So it depends on how you structure the partitions. So we recommend... Let's see if the picture is here. Yeah. So you can update anything inside this root file system partition. How to do it? No. So just the one here. Yeah. And then what we typically do is to place the kernel there as well. So you can update that. We've seen some interest for updating a bootloader, but it's maybe like 5% wanted because yeah. They don't want to break it. Any other questions? Yeah. So how can we enable factory reset? So at least until now we've considered that more of a system integration item. So we don't cover that like you say in our docs or anywhere. So what you probably ended up doing is added another partition here that had the oldest or the factory image. And we don't currently have any plans to support factory resets directly in Mender, but we can talk afterwards and maybe there's a good path for doing that. Makes sense. Yeah. So you can do layering if you're a file system. So we try to be very generic and careful not to require any specific technology, but that's also a good point. And this is partly also why it's why we currently at least consider it more of a system integration thing that you know you're using over the FS and therefore you can use that. Yeah. Right. Yeah. Yeah. So it's still planned. So Delta up this right now. I'll talk a little bit about what we're planning in the short term, but yeah, we will get there. It's just a matter of priorities at the moment. Okay. Any other questions at this point? Oh, all right. Let's see if we have a failed update. So in this case, you will get a past deployment here, obviously. And then you can see what happened. I just had one device here and it failed, but the device came back up. So now it has rebooted twice. It wrote the update and it rebooted into it or it might have rebooted into it. And then it rolled back. So it's still in a working condition here and it reported to the server what happened. But this is quite detailed. You can see the log. Let me see if I can open this. Yeah. So this is the log from the vendor client. It's quite a lot of data, but let me see what happened. All right. Yes. There's the magic message. Yeah. So I built root file system that was too big. So I have to build another one. But yeah, this is how it can fail sometimes. And this is how you would figure it out. So each device would create a log and then upload it to the server if it fails. We don't upload logs if it's successful because they're maybe not that interesting. Let me see. Yep. Yep. So there are groups. Let me see. Yeah. So I put my only device in the test group. But you can obviously add more here and then have a production group as well. So now once you do it the correct way and deploy it to test first you should be able to catch this. And we only allow, this has been a little bit of a debate, but we only allow a device to be in one group at a time because if you happen to add it to both test and production it might have some interesting side effects. So some people would like that, but currently we've been managing to keep it like this. So a little bit on what we're working on right now. How many of you know YouBoot or have done some work in YouBoot? Okay. You are YouBoot experts. Almost all of you. So we also work with some people that don't know YouBoot that well. And we do require some integration with YouBoot in order to handle the partition switching. I think, Mercer, you will give a talk about this later. And it is difficult like if you're coming from a more application level background and you're suddenly in this YouBoot world. So we're working on simplifying this work. So we have some automated patching already. And then how many of you have integrated a board with Mender? Okay. And one thing we were thinking is that there are a lot of people doing this and there's nobody sharing it. So do you think it would make sense for you to share it back to the community or do you think it's more of a proprietary thing that you... So you don't think you would be allowed to share? Yeah. It's easier to maintain also. What about you? Do you think you could share your integration you did with Mender or... No? Yeah? Yes. Okay, great. And do you think other people would be interested in seeing what you did with Mender there or how you integrated that into your board? And do you think that would make sense to share that part? Okay. That's great. Okay. That's great. Yeah. But you are again the experts. So there's a lot of people. So okay, cool. Yeah. So that's one area we're looking to just create a separate set of documentation and readme where we would basically index the board, the Mender integrations that people do. And it's somewhat similar to the Yachto in a meta layer index. We're thinking something similar. If you put it in your Github, we would index it. And then do you think that would make sense or... Run? Yeah. Yeah. Make sense. We'll get something going there pretty soon. And another thing is that we're also thinking about CIs and maintaining these integrations over time. So having a way for the community to hook up to our CI server, so we would run all the tests on your integration. And then you would get an output afterwards. If it's failed, you'd get an email, you'd get a log, and you can maintain it over time. So yeah, that's a great idea. Cool. So you'll see something about this pretty soon. And then we're looking at other ways we can integrate with boards as well. We're experimenting a little bit with UEFI on how you can actually use that to do the partition selection. So you don't need to do any patching at all. It's quite technical. But you can... Uboot actually has UEFI emulator. So you can use that. And on X86, UEFI is supported pretty much everywhere. So you can sort of get one level below Uboot in that sense. And then if... But this is pretty early stage. So we don't know if it's possible. But if it doesn't work, we were thinking also about the POC mode versus production integration, where you wouldn't have all the robustness criteria, like atomic updates. So in this case, if you're just testing Mender out on some board, you can easily do that. But if you want to bring it to production, you have to do the integration. And then finally, we want to have some way to do binary post-processing also in order to support Mender. So if you have an image already, have some tool that can repartition it for you, insert... Make sure that you can do the partition switching. So if all this is binary and there is no need to do source patching, we can also build a tool that just takes your existing image and builds it into a SD image that includes Mender and the partitions that you need. So these are some of the things we're working on right now. Does it make sense? Or do you have any other ideas how we can do the integration easier? Yeah, that makes sense. This is actually a fairly common problem. You saw it in the demo here as well. So it's too big. Makes sense. Okay. And then in terms of platforms, we're looking at X86. So this is not that well supported by UBoot. It's supposed to work, but it doesn't always. So maybe we need to look a bit into Grubb. And then there's also other systems than Yachto, of course, that are in use on Linux. There'll be an UBoot to Raspbian are very similar. And then you have BuildRoot as well that we will take a look at. Yeah. So that's basically it. Is there anything? Does it make sense? Or do you have any comments? I have not. Right. Do you have any ideas on how we could support it better? Or? Yeah. I don't have any information on that. I'm sorry. I haven't worked directly on that, but maybe, yeah. Problem solved. Yeah. Oh, but thanks for the input. Okay. Any other thoughts on, yeah. I had a little bit hard time hearing, but yeah. Right. So what we were thinking concretely is that when you first would have to create an integration with Mender on your board. So there's some integration with UBoot in particular. And then we have a checklist that you can run through manually to make sure that that works. So then you're in, let's say, on your production device. You did this integration with Mender, and then you know it works because you did it once. So the idea with the CI is that then at that point you can take one board, one of your production boards, and you can put it somewhere. It has internet connectivity, but it's isolated from your secret networks. And then you can hook it up to our CI server. So then our CI server would control that board, and it will run the latest version of Mender and build that image for your board with your integration. Yeah. So this idea was based on Yachto. So you will provide the sources for the build to this. Yeah. Yeah. So we would build the entire image. This is what we do already for the reference devices. We have Raspberry Pi 3 and Bigelbon. We use the sources from Yachto. We start from scratch, build everything, and then there's some trickery to deploying an image on a running system, like re-flashing the entire device. So it's a little bit tricky, but yeah. Yeah. So right now we have zero people that are signed up, so we'll see if it scales up. Yeah. So yeah, I'm glad you're interested and definitely validates that we should continue to work on this. But probably once we, if we are at the luxury where we have scaling problems in the CI infrastructure, we'll handle that like some caps or maybe some paid version with more resources and so. Yeah. You'd rather us to provide the entire build. Yeah. So it would be the board that you probe. Yeah. So that we can do the integration on those. Yeah. Yeah. So we have a commercial offering there that you can hire, for example, to do it. We, so that's basically what we have time to do. Otherwise it's a bit more opportunistic. So yeah, you could, we haven't, I haven't had that request before, but it's something we'll, yeah. It's a good idea. And the reason we have, yeah. When are we out of time, by the way? Like is it 11? Okay, in three minutes. Maybe one more question. No, we only use groups for that. What were you looking for in particular? Okay. So you want to have more, so this is something we have on our roadmap that you can create more dynamic groups as we call them. So you can base them on attributes on the device itself, for example. And you don't have to put them specifically into a group. They would just show up in that group because they have these characteristics. So that's something we will add. Yeah, exactly. Yes. That will be part of it. So typically the term used for it is campaign management. Yeah. But that's basically you start with 1% and then 5% or these devices than that devices. And maybe one last question. Yeah. So we do full image updates and we do compress them. So they would be reduced. But this is what the gentleman over there asked about delta updates would be a much more efficient approach where you can reduce it maybe 90% or 80%. But right now with compression, you get maybe 50% reduction or something like that. So, but this is something we will work on. Yeah. And we're streaming it right now as well. So you don't need an extra space to store any temporary files or anything like that. It goes directly to your partition. But I think we have to end. But thanks so much for joining. And we have a booth as well and a couple of other events coming up. So feel free to join those and reach out to me if you have any more feedback or questions. So thanks a lot for joining.