 Cool. Hey, welcome everyone. I'm Arvid. I'm not John. In case you haven't seen that, we swapped the talks. So I'm the CTO of SuperScale Networks. That's an IoT company doing Wi-Fi routers. And I'm also the founder of CoreHAL. CoreHAL is an open source project, a completely open source. This is why I talk about it here today. It's dedicated to liberating the internet of things. So I'm gonna introduce two of our projects today two for things that I can demonstrate. Which I just got working last week, so the demos are broken here with me. Yeah, let's get started on that. So I'll talk about IoT today. And there's probably lots of talks about IoT here. So I just want to get something out of the way what I mean by IoT. So we have a common understanding of that. It's, when you talk about unique challenges of IoT, it's important to get the perspective of what everyone means about IoT or whatever things of IoT. It's certainly interesting to get like deployment and this kind of stuff with Kubernetes and all the things that we've seen working on like small computers, like something like Raspberry Pis. But that's not what I'm gonna talk about today. I'm gonna talk about edge gateways and constraint devices. These are much, much lower end where you barely can run Linux on it. And still not system D and still not Docker. So these are my part of where I'm interested in when we're run companies doing them. Let's go grab my water, sorry. So if you've seen, if you work with these embedded devices, you know that some of these even don't have any MMU. So you can't run Linux on them ever. And this is really what's really interesting for me because there's barely any open source in that direction. Or if we do have open source, it's usually constrained to being kind of having a hook from the vendor who offers you that like something like embed where you're supposed to use their cloud service which isn't open source. So I'm really interested in this spectrum and I'm gonna talk about that. So let's do a test in a room. There's not a lot of people here. So let's see how that works out. Raise your hands if you know that WPA2, the Wi-Fi encryption is broken. All right, that's only half the room. I think the other half of the room, especially if I'm looking over there, it just didn't raise their hands because they're looking on their phone. So I'm gonna assume everyone knows that WPA2 is broken. If you didn't know that, do update your Wi-Fi devices because we now have an encryption problem there. Now, let's do it something harder. Raise your hands if all of your devices that you have, including your toaster, your Wi-Fi light bulb, your router, your phone is updated. Absolutely no one. And that's, I've been doing this talk for all week and this is standard. No one updates their stuff. And one of the reasons for that is that updates are just terrible. It's a vendor issue. It's not a user issue. It's something that we have in the whole industry. Basically, we have an IoT update problem. And the problem is so large that if you think of 2016 and the Mirai worm, that thing crashed a million telecom routers and almost took down GitHub. These runs on IP cameras where you don't get any updates anymore. So we pretty much have a problem there. And I'm not even gonna talk about critical infrastructure like nuclear power plants and that kind of stuff because there's a saying in Germany, I'm tired of these antwoord with the Bevölkerung vor unsichern. So we're not gonna talk about that. At all systems go, we talk a lot about orchestration. Specifically because Linux is very well positioned there because Linux is like the de facto standard operating system for service. And it makes a lot of sense to talk in that context about orchestration. What I believe is one of the possible solutions for an IoT update problem is orchestration all the way. Which means that not just on servers but also on edge gateways and down to very constrained devices. This is gonna happen anyway. In servers we already don't talk anymore about SSHing into a machine and updating packages. And that's a good thing that we're moving away from that but we also need to move away from that in IoT. Because in IoT we still do these things if you have SSH. Now let me try to explain what a specific solution, one of the specific things that I do with CoreHal Core is like a larger project. But specifically what I'm doing that there is essentially a solution to like what I've been seeing the last 15 years working with internet connected devices. And if you work with embedded devices like with very low level devices you know how just how ridiculous things are. Our tool chains are just terrible. If you do web development, if you do web service development you know that we have fancy stuff like Docker. We have orchestration and deployment tools like Kubernetes which is really, really good. And things like CoreHal is popping up. Things are looking good but in embedded we're still stuck with printf and screwdrivers and stuff like that. It's not that great. Now that is all really frustrating but there's something new coming up that everyone should know about. It's the Rust programming language by Mozilla because they're building a language and an ecosystem that actually works on embedded. It ports to all of these devices we see is usually the main driver right now but with much better tool sets. So what I do with CoreHal is kind of contribute to the ecosystem of Rust and trade to build our tool chains for IoT in a much smarter and better way. I'm only going to talk about two things because of time constraint. I'm only going to talk about Arcana and Bolter. There's a couple of other stuff that we do. One of them is a crypto incentivized edge network. Not going to talk about that today. So if you came for that, sorry. Arcana and Bolter are use cases that are very similar to Docker. This is why I think they're very interesting. Also they really work like everything else we're doing so far. Not in production yet because I do run a company that has 10,000 devices out there but we don't run a production yet because I just got to work in last week so we're not that crazy. So Arcana is specifically very similar to Docker or to something like CAsync. The only difference to, or similar to Docker but the only difference with words as Docker is it doesn't use layers. Kind of like, you know, CAsync maybe works very similar. It uses a content addressable storage instead of using layers, like what Docker does. But layers are, I mean, content addressable storage are very inefficient for IoT and I'm going to talk about that in a sec. I'm just going to quickly lose a word on what content addressable storage is just in case you don't know. I hope everyone knows by now because look at CAsync, it's actually really cool for cloud services, much better than what Docker does. So let me just quickly explain what a content addressable storage is. As the name implies, it stores and indexes content not by its name. So like a file name or something like that but by the content of the actual file. So imagine something like Docker, sorry, something like, what's its name? I would have written down here especially because I keep forgetting it. Sorry, Dropbox. Something like Dropbox or a similar service where you upload files and you can't share it without other people or downloaded. Now let's imagine that we both have like a music collection in Dropbox, like right is mine, left is yours, doesn't matter which one. We both have a file called Heavy MP3. Now what you think is heavy and what I think is heavy might be very different. So those files are not identical although they have the same name. Why we can both agree that Taylor Swift is kind of cool so we both store the same MP3. And what Dropbox or similar service can do is they just hash the content of that file. And instead of marking down that I have a file called Taylor Swift MP3, they're marking down that I have a file called this hash. And everyone who has that file gets a reference to the same data. So they only have to store the data once. I mean, hopefully they do more time for redundancy but in theory they only have to store three files here while there's four files on the slides. This is what content-resistant storage does. It's very efficient for things like when you have multiple system images, multiple operating system images and they contain sort of the same data, almost the same data, which is very common because most GNU Linux operating systems are actually very similar with some minor differences and you only want to store the difference, not really the whole thing over and over again. However, this doesn't really work out so well in IoT where we don't store file system trees. We also don't even store file names because we don't have space for that. I mean, these are just a couple of bits but we need those bits so we don't do that. Leading me to update safety. Update safety on embedded really means, it really means this one thing for me at least. It's if you fuck up something, you better be able to unroll it because if you're not able to unroll something on a device that is out there millions of times without user interaction screen, the only user interaction you can do is people will return the device and you will send them a new one. This might sound fine for when you have to return a fridge, but if you have to turn a million of a fridge, a million of a fridge is your bankrupt. So if you think of something like the diesel gate, it would have been a lot less costly if Volkswagen would be just like Tesla and they had system updates but then they would be kind of cool in the first place. So no. One typical thing that people do when with embedded systems and update safety is to just store two of the same systems. This is something that you might not be familiar with if you don't know, if you haven't worked with embedded systems is what I'm saying. What we typically do there is we store system version one and we mean the whole system version, not just like a specific state of a package tree or something like that, but the whole image. In one region of memory, and then we store a second version, like the new one we wanna have in a different memory region. And if one of them fails, we just boot the other one. A boot loader is usually the very tiny thing on a bed. It's not like a complex thing, like we do on x86 bios and everything. It's very tiny and usually only does things like if one of the version fails, boot the other one. It doesn't really do much more than that. Unlike with things like Android, for example, they have a fat boot loader. What they do is not fat the file system. It's fat in terms of really big and really complex. What they do is they put the capability of updating your device inside the boot loader. So they're putting the updater in the region that you don't update. It gets really complicated from that point on because Android can update its boot loader, but it's kind of a retrofit. It's not really like a well thought of system. And there's also the complexity of dealing with that. Plus, you still have user interaction because once your main system fails, you will not be able to interact with the user in the same way that you would with your UI or whatever you have on your device. Kind of like on Android, you have this fail-safe mode, which people can use, but it's usually used by service engineers, not by actual consumers because it's still complex. What I'm proposing to do on IoTs, actually, it's not that new, but if you fit together everything I just said about control-design storage, what we can do is we can store both of those systems, but only the parts that are actually different. So what we can, it's more network efficient, obviously, but there's one specific key element that's new, which is you can also store three system images and do things like A-B testing and these kind of things. Now, why this is possible is because most of the system image between Word and is actually shared is the same code. The only reason we don't restore it and we don't de-duplicate it is because our tools are so terrible, unembedded. We do know how to share application code between two applications. It's pretty simple, that's what busybox does. If you've heard of it, it just has multiple executables in one executables and then the shared part is shared. It's pretty easy, but you can't do it retrofit when you already have something deployed and you wanna update just one part of busybox, for example, it's not possible. We also know how to do control-design storage. It's a fairly common thing. If you wanna look at something new, there's IPFS and system DCA sync. That's pretty great. These are new things in that space, but they don't work for binaries because the reason they don't work for binaries really is that binary executable code is not like a story. It's not like a linear list of things and if you insert a piece of code in the middle, then you will just amend the story. That's not how it works. It's more like binary executable code is a list of addresses and these addresses change, the whole of the addresses change if you just change a tiny part of it. So doing something like content versus storage with it is gonna be currently pretty much impossible because content versus storage relies on sharing blocks rather than sharing executable streams. And these blocks are gonna be different. Every time you link something, every time you just swap two lines from a printf to printf something else, I'm gonna show you in a demo, it changes. This is why we have Bolter. So I talked about Archon before. Archon is basically just like, if you're in desktop operating systems, it's not that interesting because we've got the CA sync now. It's pretty much the same thing just for embedded. Now I'm gonna talk quickly, you lose a word on Bolter because Bolter solves the problem of executables being not a linear stream. Instead of linking it the way that GNULD does which is GNULD links by priority or by how you, depending on the order of the input files that you give it, it will change its linking output. And what Bolter does, it guarantees that two binaries are the same no matter where and who links it, at which point in time with which order of input. It will always put the same, let's say libz printfo in the same region and it will not touch it. It will not relocate it so that you can later share those between two different applications. And they're also linked in a global space. That means that even if you have two different application codes, both of them are, both of the printf.o are gonna be addressable the same way. This is very similar to dynamic binaries to dynamic linking. The only difference really is that dynamic linking doesn't link by content, it links by name. And this has some disadvantages in systems where you try to, I mean, if you did ever, desktop operating system, you're working desktop operating system, you know, DLL hell, it's basically what happens when you link by name and you deploy your application and there's another application on the system but that demands a newer version of the same library as you. So the user might upgrade that library but then your application is also gonna get dynamically linked against that new library, which you never tested. So it might just not work. With Bolterby, avoid that because the linking is done by content and the content is still inside the binary. So if two binaries are getting updated on one of them, Rekhaj Nualipsi, it's just not gonna get de-duplicated and both of them will get the libc that we're originally linked with or the parts that we're originally linked with and that's important, not the full libc. Just the part that's different between those two lipsies will not be de-duplicated. All right, one more time. I hope I can make this work. Oh, I forgot to change my terminal font size. That's gonna look terrible. I'm gonna need a second to change that. Very smart of me. Does that work here? I'm doing it by screen because I can't look up there. So what I'm gonna show you as soon as I get it working, there we go. Can you see this? Can you probably read it? Is the font size big enough? Okay, cool. What I'm gonna show you is Bolter with two test binaries. So as I said before, in C and you all know C, this is just print of Hello World. It's not that fancy. But if we change that to something more like, we just change a tiny part of it, which is we're just gonna print the name of the upbringing system. These two files are very similar and you would expect that the resulting binary will be similar because they share most of the Lipsy. Like the binary is gonna be mostly Lipsy if we statically link. I only did that. We'll just redo it. So now we do what I did there. Is I linked it, I linked that application statically with Lipsy, with MUSL Lipsy. One time with GNU LD, this is the dash LLD binaries or with LLD from LLVM, doesn't matter, same thing. And one of them is linked with Bolter. Now you can see that the Bolter binary is already slightly bigger, but I'm gonna tell you why that is the case in a second. Now, if we have Hello One, I showed you the code for that. And we just stored in a contract visible, dammit, in a contract visible storage. Sorry for that, badly prepared. Oh no, I used that before. If you just stored in a contract visible storage, we get, like if the contract visible storage is not prepared for this specific use case, it'll just try to split it into blocks depending on the content. In this case, we've got, oh, I just stored the BLD, I wanted to sort of show you the LLD. Okay, that broke in screen. Good to know. Okay, so we've seen that it's storing 70 kilobytes and the plus sign means that it did store these blocks as new, so this is what it's actually gonna write the disk. So it's writing 70.43 kilobytes. If we do the other binary, which is hello2.LLD linked with GNU LD, just get a different name. You can see that it's almost storing the identical content. They're both the same size, so both have 17 kilobytes, as you can see up there in the broken progress bar, but it's storing 15 kilobytes, so it can only share a tiny part of it because all the addresses have changed. Now, if we store a binary linked with Bolter, there we go, we're linking, oh yeah, I already, okay. I did already accidentally store the hello1, so here's the demo effect on hello2, which is the one where it just changed hello world to hello you name. We only have to store five kilobytes, and those five kilobytes are mostly relocation addresses and the elf header. This is significantly more efficient than anything we've ever done before. It's also significantly more efficient than compression, any compression algorithm, because no compression algorithm will be able to identify the addresses and re-relocate the addresses in a way that makes them compressible. So this allows us to do something that I'm really excited about, which is it'll allow us to do safe updates in a very, very efficient way on IoT devices. Just gonna switch back to, yeah, 20 minutes, great. We're not gonna talk about that. So I just wanna, so after the demo, just wanna work, lose a quick work on REST because I've written everything in REST and I'm really inspired by it. I think really that everyone should look at it at least, even if you're not gonna use it in production. Just because I believe it's the first language since C, or the first popular language since C, that really tries to get down to the very, very low level and tries to compete with C on devices that we currently don't have anything else for. Golang is really not an alternative on very small devices because Golang doesn't even work without a floating point unit. So that's not an option, but REST really tries to get there and we'll hope that one day we might actually have an environment with that ecosystem that's usable and that's as good as what we have in cloud deployment. Yeah, that just means that Arkon is currently built for these embedded devices. We've got things like, in startups, what we usually see, I advise startups, what I usually see is people have used things like Android for deployment. So to get a device and there's Android on it, it's just some device from China and they try to use that as a system, which is pretty bad, but Arkon will allow them to run a real Linux distribution on that and have a deployment story for it because it runs there. So yeah, I think I'm not gonna talk about the other thing because of time constraints, so I'm pretty much through with that. I will, I will do that. Just wanna tell you where I'm going with Korhal at the last slide. So I'm trying to break IoT down into engineering problems rather than problems, rather than political problems because there are political problems, things like not having updates, not having enforced updates for things that are deprecated, this is a political problem, but what I'm really trying to do is break it down engineering problems and solve it from the perspective of, okay, what if we could solve problems like end of life, end of life clouds and problems with the update crisis that we have, the service disruption problems we have. I mean, recently there was a company that did smart knocks and their cloud failed or their update failed or something like that and people got locked out of their home. Things like that are pretty terrible and I think we can solve it from an engineering perspective if you just make your tools a lot better because if we have good tools, people are just automatically gonna use them and maybe be less stupid about IoT updates. That's really my plan of where I wanna get with this. And if you're into embedded devices, if you're into especially distributed embedded devices and you wanna solve the IoT update problem, then please start me on GitHub, come to me after the talk, talk to me about your approach to it and how we can proceed with that because I'm building an open source project, this is not a startup and I'm really looking for any help in that fixing IoT, cool, that's it, thank you. Yeah, can you microphone? Awesome. Thank you. So two questions, the first one on the storage on the node itself, you're trying to kind of like optimizing instead of having two partitions and switching between them like splitting in chunks, but it looks like in the worst case, you still need the full space for two partitions. So it's like, if you need to update every single chunk because it changed for some reason, it looks like in the worst case, you're still in the same case as I need two partition and exactly double the space for storing them. Yes, but it's very unlikely because it's very unlikely that you're deploying a system that is completely different from the other one. Yeah, but your system should be still designed with that worst case in mind, so it's like you still need two times the full space. Not necessarily, you know beforehand before deploying, right, before you deploy, you already know how much space you're gonna need for that update. So you can just stop the deployment and think about your bad choices in life before you do that. The other question was related to the linker. How does it play with address space randomization? That's a great question. So that actually came up multiple times during the talk, multiple times I had the talk address space randomization just in case someone doesn't know that what it does is it loads the executable binary at a random position in memory rather than at a fixed address. The reason for that is that it's a security thing. If you find a loophole or a buffer overflow in one of the, in your code, you won't be able to jump at a known address and execute code there. So it makes attacks less reliable. And it does not conflict with that. The reason for that is that specifically on X86 there's a trick we can use. You can lay out the address space relative. All of it can be, so what the linker can do because it's a static linker, there is no dynamic libraries loaded here. All of the addresses inside the entire binary can be relative. And what we do on non-X86 on platforms that don't have PC relative addressing, what we do there's a small bootstrap section in the beginning of the executable which relocates the binary onto the correct address. How precisely do you mark in your linked binaries where the cuts shall be placed? Like do you have a header in front or do you have some special marker that can be recognized or? I tried sections, I tried elf sections but they're really, really large. So what I do instead is I emit an elf, an additional elf section which is an extension. It's an extension type. And that elf section contains a list of exact binary positions where you just split. And what's the size of your block sizes? That's depending on the input. I'm just wondering like, I mean, if we can agree on something, how suggested to cut marks are marked. I think it's fairly non-intrusive to do this into a binary because it's really just a section with positions in it. It's not that fancy. Like it's just probably like 10 lines of code in C to read those. Yeah, well, I mean, I'm wondering like, you know, see, I think we don't actually ever parse what we're processing there. But it might be nice to be able to take advantage of manually placed markers for our cuts. And then it would be kind of cool if we could teach CACing somehow to identify the markers that your linker placed there. I think that would be amazing. I'm just thinking of a way to, I think we're gonna have to think about that because the markers are currently written inside the binary, so we'd have to parse it. But maybe there's a way to... Well, I mean, one way to do it would be just place some weird identifier in there that is recognizable. Yeah, something like at the end of the file or something like that. I mean, just actually add the cuts to something, right? Like as a comment or something, maybe that works. I don't know, just an idea. Let's think about that. I think that would be a really good idea. I mean, it doesn't have to be reliable, right? Like, I mean, if there are some false positives, it doesn't really matter. Yeah, so what we can't do is we can't put content inside the linked section, like specific markers or something like that, because that's executable code. So it would be... Well, you could jump over it, right? Like, I mean, that way. Yeah, yeah. But you have, like, the same overhead as elf sections, so... You'd be really, really careful and really space efficient here. But it's actually a good idea. Maybe it's more space efficient than elf sections. Yeah, I mean, what I'm thinking really is not just executables, it's about everything, right? People could, whatever kind of resource they have, also place these markers and give us a hint where a good cut would be. And then your tool and my job would both take benefit of that. And people just have to make sure that they place these things. Let me think about doing it that way, because I think that's probably the best case for you. It would be great. Everyone asleep? Okay, you can also ask questions after the talk, if you're not. We're all in talks all the whole time, that's okay. One more question is, do you do anything else than just make the blocks better placed? Like, for example, the Corset thing from Google, have you ever looked into that? They do binary deltas and they try to deal with the offsetting stuff. And to be honest, I find it really horrible because it's so much bound to, I mean, what they ultimately do basically is they disassemble the, they try, like under the assumption that what they're processing there is assembly, like this is actual machine code, they disassemble everything and then generate the delta on the disassembled stuff instead of the original stuff. And if it compresses better or if the delta is smaller, then the binary delta would be, they use that. I personally find this, this is so transcending all the layers of the stack, I don't want to do anything like that in CSync, but what they found out what was really important was dealing with the offsets and being smarter about them, right? Like, so they basically try to take into consideration that modifying or replying some kind of filter on top of the actual data makes the data more like and they try to figure out how that works. There's two answers to that. One of them is, I still think that ZSED, the standard compression thing by Facebook is still more efficient than what they're doing because ZSED is able to build a lot of what's it called, a content thingy. Like basically, it can learn from that binary a lot better than whatever you're trying to do with compressing it based on the assembly. And the second thing is I can't do that because this has to be architecture independent. I have to guarantee that there's the same compression on any architecture. The third thing is it's actually more efficient already than GnWildy, although it's not optimized. And the main reason for that is that we don't have to do dynamic linking, which means that you can assume that your entire binary is one linear stream. And linking is a lot more efficient if you can do that because you don't have to do things like PLT, like jump tables and stuff like that, which actually makes it more efficient than GnWildy already. I guess you're all looking forward to the next talk. So if you have any more questions, come after the talk to me. Cool, thanks.