 Microphone is on. If no, then I will sing something because you can't hear it. The laughter tells me you can hear me. Okay. People will keep trickling in, so we'll start with introductory stuff and the entertaining things before we get to the technicals. So welcome to the first post lunch, which means most narcotic session of the whole day. I know that everybody will doze off in 10 minutes because you are all digesting. I will do my best to prevent that, but it requires a little bit of interaction on your end. I will tell you in a second. So we are talking about community fueled over the air for Sephir and other MCUs today. My name is Joseph Holtzmar. I'm the head of developer relations for Mender, slash Mender IO by Nordentech. We are a provider for over the air solutions and device life cycle management. And I have a very strong background in the Yachto project and open embedded ecosystems, which actually led me to be the new Yachto project community manager since about two months. And I also run most, if not all, of social media for Yachto and open embedded. I used to say that I'm completely antisocial, which means I am not on any social network that has kind of like changed, so I'm really, really easy to reach. When I'm around here, you just step up to me and say hi or duck or whatever it is in your local language. If it is a language that I do not speak, you can also insult me and I will say yes, that's cool man. Other than that, shoot me a mail, find me on Twitter, find me on master dawn, find me on LinkedIn. It's super easy. Just say hello Joseph. Okay, we have talked a little bit about me. Let's talk a little bit about you. Self assessment. Who of you considers himself a hardware developer? That's okay. Who considers themselves a software developer? Okay, most. Whom of the software developer considers themselves a firmware developer? Nice intersection. Who of you has also some background in embedded Linux? Yachto? For Jan Pete the ex-test? Okay, you lose man, sorry. Okay, we've got to know each other a little bit. Final question. Who has ever seen me live before? A bunch. Okay, I'll tell you how this works. This presentation is interactive. I will reward any form of interaction with chocolate until I run out of chocolate and I mean literally any kind of interaction. I have prepared a couple of examples. You can tell me what you like. That is good, that is nice. It gives me warm fuzzy feelings. That is okay. You can tell me what you don't like. That is even better because then I can improve. The best that you can do is tell me where I am wrong because then I can also learn something. You can be helpful for others. If I explain something that maybe needs some clarification, then I am convinced that others can also learn something or you can be just very, very practical and say, hey, Joseph, I want chocolate. Which counts as an interaction and I will give you. Chocolate delivery is via airmail. I have acquired some skills in throwing by now. It usually works. Sometimes it's a little bit off. Who volunteers to serve as a nap meter because if people are dozing off, then we will do some exercises to get your blood flowing again. Last bit of the organizational stuff. More details on modern tech. We have been around for quite a while. Actually for 15 years. Founded in Oslo slash bay area, San Francisco. And yeah, let's say we have been doing this for quite a while. So we think we have a little bit of a clue about it. And our mission is to secure the world's connected devices. And updates are one crucial part of this. And this is why I am standing here in front of you today. This is the organizational final running order. And as you can see on point two, let me introduce somebody to you. I've got a friend whose name is Joel. Joel is a man who is a very artisan side. He likes paper craft. So in the evenings, he sits down and he does scrapbooking. He prints photos, selects flowers sometimes, papers, cuts out shapes, assembles everything. And because his home is already full of those fancy pages, he has started giving those pages out to all of his friends, including me. It's super cool. I have a very special book that exists only one time on this planet. And whenever Joel gives me a page, I put it in there. And it's dear to my heart. How does this work in practice? Joel actually does not meet me very often. So whenever he has finished the page, he puts it in an envelope, seals it up, writes my address in front and puts it in the post box. Then I get it. And no more action is involved. This is pretty handy because Joel, as you can guess from the name, actually lives in France and I'm in Germany. So the only thing he does in order to provide me with a page is prepare it, drop it. Then the postal company takes care of all the rest. The postal company collects all the post boxes, sorts it, analyzes the addresses and then puts only the envelopes that are specifically for me into my post box. And for me, I don't have to care about anything. Just once a day or so, I walk out to my post box, open it up. See, oh yeah, that is for me. I process it. That's an invoice. I can get rid of it. That's just marketing. Other side getting, oh, nice page from Joel. I like that because Joel, of course, these are valuables. He always pays for proper tracking. So when the page is delivered to me, he gets a notification. Ah, has arrived at Joseph's place. Everything is fine. I'm relieved. However, with all the care Joel and me put into all of this process, occasionally something goes wrong. Maybe me, when I get out the page, something tears, or it gets damaged in transport or whatever. Then it's set and I have to notify the postal service. They return like a knock to him and say, ah, there has been a problem with your delivery. We're super sorry. Can you just please recent or look into it? The good part about the pages is that Joel can just like figure out how to proceed. Should he repair the page? If it was returned, should he create a new one altogether? And for me, I can still enjoy the book. So the book is usable all of the time, no matter if a new page from Joel is in transit or not. You might have guessed that this is actually a story about over-the-air updates. Surprise. Because in the end, it is really just the same. You have a device somewhere out there and you have an update somewhere here. And you prepare the new firmware for shipping. You upload it to somewhere, the postal company, which takes care of all the management and figuring out which device should receive which update when, what are the conditions, what are the dependencies and everything. And then the device only receives the one update that is really meant for it. If everything works out, the acknowledgement that the deployment of the new update has been successfully applied is passed back. If something goes wrong, dwell or you get a knock, a failure, an error, a log, whatever, then you can react. The key part is that just as in the book and the pages, the over-the-air solution specifically the client on the device needs to take care of fallback and of error handling to make sure if something goes wrong during the process, the device is still functional. You can still enjoy it. As opposed to postal services, we are in software and we are open source. A postal service in most regions or even countries is a monopoly or if it's business to business or parcel service an oligopoly, but you can just say, I want my own small postal service just for this city. It's just not practical. In software, I think we should be able to do this. Let's look at the chain again. The letter, you can build software. The book, we can build devices. We had a few hardware developers in here. That's why I asked. The next thing in the chain is the service site. You can be your own postal service. This is what I'm representing here. Of course, in my presentation, I will use Mender. It is an Apache public licensed piece of infrastructure. You can just take it, use it as is, and it has a few outstanding features that make it very, very, very suitable for this kind of application because we are completely payload agnostic. If you can put it into a binary file, then we can transport it. That is the deal. You can make a blob for sefer, you can transport a docker container, you can transport a root file system, you can transport an AI model. We literally do not care. You know how to make stuff. We know how to put it onto a device. In order to facilitate all of this, the API to manage, upload, deploy is fully documented, ready for you to use, and as already pointed out, you can be your own postal service. This is not tied into a cloud somewhere. It is open source. It is on GitHub. You can just grab it and run it on your home box if you think this is a good idea, which it sometimes is. Already pointed out, the key in order to enable people to be their own postal services is documentation. For the API, because we do not lock you into a specific client, we do not lock you into a specific pipeline, we do not lock you into a specific dashboard, there is no magic hidden in there. Our dashboard does not do anything that you cannot get rid of, replace, rewrite, whatever. The documentation states each endpoint, each parameter, each response. And payload agnostic, some people say, oh, this is super special. I need to reinvent the wheel. Yet another binary thing that I can take from here to there. And we were like, no. All of the world understands tar bowls. We understand tar bowls. So let's just use tar bowls. You have to take a few small considerations into account, like the order, so you can actually stream without having to unpack everything first. But in the end, it is just using something that has been widely understood for probably like 30 years. We do hopefully not suffer not invented tears in Rome. So the last thing that is missing in my story here is the client, the mailbox that I just mentioned. This mailbox needs to be able to understand or accept the incoming artifacts. Understand means because you can't have different kinds of artifacts and you can't have different kinds of artifact receivers in the same device. With tiny ML or tenser or whatever, prominent getting to suffer also, you could basically have an update for the firmware and one for the model. There is no magic in there. The mailbox needs to be able to apply this to device in a meaningful way. And in the case of a real firmware update, in order to keep the device working, if something goes wrong, we use the AB mechanism. So apply to B, flip over. If the reboot fails, if something goes wrong, flip back. If everything works as expected, commit and make permanent. This is something that the client has to do and report if everything went fine or not. So again, you might have guessed it. Joel is a good friend of mine, but he actually has probably no idea about papercraft because he is a software developer. And all of the story was made up. I apologize, but you get the idea. Joel, end of last year, sent me a mail and he was like, your API is so easy to understand, I bet I can make it work on a microcontroller. And I said, that's a great idea. What do you need from me? He said, no, actually nothing. I just wanted to tell you. So two weeks later, I get a mail and he said, Joseph, now I got it working. Here's an ESP32 demo. You can connect it directly to Mender. I said, Joel, that's awesome. Great work. Thanks. I'm showing it to the world. You did one hell of a job. Time passes by. I make a demo. I talk to people, all of that. Out of thin air, another mail arrives and it essentially said that I have skipped a slide. Don't care. The mail essentially said, you know, Joseph, I was kind of bored. I bet I could make it work on Sefer 2. And here it is. So he decided to just grab a Sefer board, which is that one. He went for Ethernet because he felt like, okay, I've done Wi-Fi already. So this time, let's do Ethernet. And he cut it down to only one dependency, which is C-Chasen, and he uses the HDP slash embed TLS stack that is already there in Sefer. He did it originally on Sefer 3.3. Last week, he ported it to 3.4 forward. And one thing that I want to mention exactly, it's MCUboot. So he also, again, did not reinvent the wheel. He just said MCUboot can already do all of this AP flip handling. I'm going to use that. How could he do that? He actually, when he did the ESP incarnation, he had portability already in mind. So anything that is even remotely platform or hardware specific, he had carved it out. So he just needed a handler or wrapper for the communication, for writing to storage, for the thread slash process mechanisms of the specific platform, and the partition flipping. And we are super, super fast because none of you is asking for chocolate. No, no. First question, and then I'll tell you why you are wrong. You can host it for yourself. But if you do not want to do the operations, then this is what we do. So I said you can be your own postal service. I could drive around and distribute envelopes. But in most cases, I actually don't want to do that. So thank you. This is a very, very good question. And because we get that asked quite a bit, but that is actually the explanation. Even if I could do it, it's in many cases cheaper and more convenient to just pay somebody for it. And that's our offer plus some minor. And I have to tell you why you are not correct. I got the idea to throw chocolates from a C++ conference. On YouTube, actually. 2013, 2014? Seriously. Are you from Portland, Oregon? See, then it can't be you. But we'll look it up. We'll look it up. Okay. Ah, okay. No problem. It's always great to have a discussion. Even I told you, tell me if I'm wrong because that's the best. I have a demo here, which I wanted to update right now. I even brought my own ethernet, but something is wrong with the check mobile. So I cannot do a live demo right now. I'm super sorry. I will be at the tech showcase tomorrow evening. And hopefully we have wired ethernet there. And then I will show you how to update a safer board down there along with a Linux board along with an ESP32. So to just display the heterogeneous features. But I did my very best. I've been spending 40 minutes up here to get the connection working. I'm really, really sorry. We've got the most of the boring stuff out of the way right now because it seems to be working. We can update things. How to go from here? Small side track. We as Mender ourselves are currently also rewriting the client. Why? Because we have also understood the need for portability and other platforms than just Linux, where the go client has served us really, really well. And luckily, just right this morning, the first incarnation of the standalone client has been released to the world. It is very, very alpha. It serves a quite different set of requirements than the one of Dwell. It is just Linux at the moment too. We intended to go beyond Linux. Sefer is also on the roadmap. If this is something that you would be interested in discussing, testing, contributing, or maybe even working on it in a like porting or partnering situation, feel free to step up to me. You don't have to do it on the big stage. No. I just deliver one more chocolate and then I will tell you. Sorry. Pass it on. Okay. The key is yes, I know that people who think or are convinced that Rust is an excellent programming language will tell you that you can literally program with for everything in Rust and they are technically correct. However, in reality, if you get some board and some random hardware slash firmware developer, what will they have? Or a C-toolchain. So from a real practical point of view, it's a tough sell if you want to tell people, yes, this is super cool. It is written in Rust. You just have to revisit all of your infrastructure so far and you have to ramp up all of your engineers because, I mean, like after five years or so, the invest will pay off and then you will have better software. From an academic point of view and from the enthusiast point of view that is perfectly accurate. From the real life and industrial approach, it's just not going to fly. This is the same question as if I would ask you, why are you still writing a separate not in Rust? You will also say, because it's already there. Just rewriting it for fun because the new fad has come along, sorry, Rust, might be interesting but not realistic and that is the choice here again. So thank you for the question. You had something else? Yes. Can you repeat the question? I will repeat the question. The question is that here is a solution, a device that does not consist of a singular build or single blob of software but of a composite. Let me put on my Yachto hat for one second. In Yachto, we do this with multiconfix and then end up with a single image. Yachto hat off. Mander hat on again. You could already have done this with different packages and dependencies because Mander has a very, very advanced versioning scheme with dependencies. Provide s, version, numbers, needs, this and this and everything. Right now, you have to actually talk to the gentleman here in the first row. We are in the progress of providing a so-called orchestrator solution which does exactly that. So this was the advertising part, he does the technical part. But yes, this is a common requirement and it will be available right now or soon. Fabio tells me soon. Thank you. There was another question here. Sorry. I'll try to do better. Sorry, Harold. Closer. Closer. Yeah, you are planning to rewrite it in C++ with Sefir being on the roadmap. I have big interest in C++ and in Sefir, but I have not managed to combine them yet, like running C++ code on Sefir because it's very ingrown, so to speak. When you did this roadmap, did you think about Sefir explicitly or is it just, it looks like it supports C++? We have checked that it supports C++. The key problem with Sefir is in other areas. Let's put it like that. We are aware of the fact that the C++ that Sefir gives us is not like the C++21 that you can expect on a big POSIX or whatever. Exactly. So that's why the slide says, if you think this might be something that is of interest to you in a possibly commercial aspect, you might want to talk to us. So is that good enough as an answer? Okay. Question from the live audience and then I'll pass the mic. Supports for non-HTTP transport such as co-app, for example. Is that something that you're considering? Sorry, me sober. Clear no. At the moment and in the foreseeable future, we are strictly HTTP. Do you want chocolate or are you just a messenger? Yeah, for the live audience, for sure. I'm left handed though. There was one here. Okay. Hey, sorry if you didn't answer, if you answered this before, I came in a bit late. I have a very specific use case. So I'm deploying oil to low power devices with LoreWan. Of course, they were very limited in terms of actual throughput for the downlink, so anything coming from the server to the end device. So my question is, is there the possibility for differential updates? So you have the cloud is aware of the device having updates on so and then the update for that is sent down as a differential image instead of the full image for the AB partitions. Yes. The official client supports this in, let's say, the higher levels because this is actually a paid feature. We don't give everything away. The community client by Joel does not do it yet, but nothing is keeping you from it. So if you invest a substantial amount of like brain power to A, teach the client and B, construct the artifacts accordingly, nothing is keeping you from it. For Linux, it is there and we offer it as a paid service. More questions or shall we go on? I have one, it's at the Yachto Chester. At the Yachto Chester, I know that's a stupid title. For updating bare metal and better things with Sapphire, can you run an update without having a second copy with the flash space is very small that you have a bootloader, which completely flushes and erases everything without having two partitions. That's what's the easy part. I would say yes, because that's why I noodled a little bit on the tar file. The tar file as artifact was constructed specifically to be able to do that. So again, Joel's client does not do it because he didn't have a need for it yet. The C client does exactly that. When a new root file system update is arriving, then it says, okay, unused partition and stream directly through to the partition. So there is no technical reason why you could not do this on a bare metal thing. It is not there yet. Again, invest either brain power or money or in some cases even both. But from a technical perspective and the API and the artifact format, this is definitely possible. I need to drink more beers. My aiming is totally off today. As the demo didn't work now, is a demo available at YouTube or so? YouTube? Not yet, but I could do one. And of course, all of the code is on GitHub. So the ESP demo is on GitHub, the Zephyr demo is on GitHub, all of our stuff is on GitHub, but I definitely can make a video of it. That's a very good idea. And again, tomorrow evening at the showcase, hopefully everything will be running. First one that I actually hit. Yes. Well, shameless plug for Zephyr webinars that I'm about to launch. I would be happy to have you and Joel maybe. I do a live demo and then we record it. Okay, but you don't get the second chocolate. Sorry. And the guy behind you just wanted to ask something. From what I heard, now the solution sounds comparable to Eclipse Hawkpit. Is that right? It is in some cases the same. So, but Eclipse Hawkpit is just the back end. You can combine Hawkpit with whatever front comes along with it or the other way around. You can combine whatever front end you love most with Hawkpit. It is a somewhat comparable solution, but again, it's just back to this question. Then you have to self-host because I'm not aware of anybody who will sell you Hawkpit with an SLA and all of this stuff. If it suits your requirements, it is a very comparable solution. If you want somebody to actually do the heavy lifting, then you might want to look at us. I'm getting better. Anymore? Not yet? Okay. Then because one of the additional things that Mender can do beyond just updates in software logistics is the so-called troubleshoot feature that we have. The troubleshoot feature gives you a port forwarding or simple file transfer, mostly end. It gives you a remote terminal. Obviously, this is something that would be very cool for Sefer 2. So, it used to be on the roadmap literally until today, 8.32. And I want to read this out aloud because it literally made my morning. Hello, Joseph. I know you are at the Sefer Summit actually. Hope this is a great moment. I can confirm it. Would like to share with you a picture attached to this email to tell you that the WebSocket troubleshoot protocol is clear after reverse engineering? Sorry. Yeah. We could do sometimes better on ducks. And I got communication between the device and the GUI of Mender. Will permit display of the Sefer terminal, my original goal, need some cleaning, et cetera, et cetera, but no more technical questions to move forward. So, I have to point this out. Joel is not on our payroll. Joel is not doing this for money. Joel is doing this after hours because he's an enthusiast. And that's what I meant with community fueled. This is a random dude who just said, this is something that is cool and I want to do it. And then I get those emails. And that's the coolest part about working in community. That people just are enthusiasts and you can empower them. So, thank you, Joel. I know this is on video and this one is for you. I already pointed out, I will bend it down at the ELC Tech Showcase. I will be having the Sefer demo. I will be having ESP32 and I will be having Beagle Play. The Beagle Play I do not want to take home with me. Can't just pass by. I will have a raffle chart. You drop in a business card. At the end of the day, I will find somebody who is pretty to draw out one of the business cards and the name on the business card gets to keep the demo, including a pre-flashed image built on Yachto with the support of TI. Thanks, Nishant. And Mandarin enabled. Raffle at 630. Wrapping up. Oh, I have a question. Sorry. I'm Abhishek from ARM. I am Abhishek. I'm from ARM. Okay. A couple of questions about delta updates. Yes. I wanted to check, basically, it's very difficult to do when the code is executing in place without virtual memory. Either you want to explain, but I would like to talk about that later to people who are discussing if this is sort of executing place environment where code is executing out of physical address space directly. Because I think that's a difficult problem and if there's a solution, I would like to understand that. If you're doing executed place, then it will be definitely problematic. So the deltas for Linux are obviously binary based on root file systems. I would guess that a delta mechanism for a flash-based microcontroller would need to be different. And that's again why it's how it circles back to artifact. We don't care how the delta looks like. You just need to have a clear identifier. So finding and constructing the technical means is one thing, but we don't say it has to be done that way. So if you need a delta B mechanism or whatever that applies to a flash microcontroller, then we can just use that as is. You can insert literally everything. I know that this is not a clear answer, but I'd say whatever somebody wants to insert, we will also happily take. I'm just going to wrap up this here, because the one thing that you should take away from this is if you are building a connected device, you need over-the-air updates. Not just because of regulation, but because I hope you take pride in doing a proper job. If you are shipping devices that you cannot fix for securities flaws or anything like this, then I think you are doing a bad job as a developer. You can avoid platform and vendor lock-in. You can pick a solution that is not locked into only Sefer or only Nix or only into this hosted solution or only into this provider. Like I said, everybody needs to make money on something. We provide that hosting, but with the APIs and everything open, it at least avoids vendor lock-in to the greatest extent possible. Open source guarantees that there is nothing hidden in there. There is no code that we can sneak in. You can revisit any piece of code that goes out to your device, and Mender is here to help you with that. We have covered a couple of questions already, and I think we are mostly out of time, right, Ben? Yes. I would love to throw more chocolates. I'm not allowed to do this on stage. I will just go out there and discuss and throw and talk to as many of you lovely people as humanly possible. Thank you very much for having me.