 We're talking about Loft as in robots and actually all of the stuff comes from a very stupid idea. And we come to this in a second. This one you should not see, but this is about myself. I'm Max Kerbecher. I'm the founder of Liquid Replay. Also CNCF ambassador. I run the KCD Munich and organize the KCD Munich meetups and a few other stuff. And what you see here like two different Replay companies written is like my company takes care about Kubernetes. Cloud Native, Containers, Wasm, Software Development, FinOps. Our colleagues on the Robotware side, they take care about this guy. So this is what they do. Not only this one, there's a couple of other robots running around in our offices. Sometimes a little bit annoying when there's like three of them like walking down the floor and you know like... Difficult to concentrate and you're always difficult to concentrate when this guy is running around. But this is what our colleagues are doing. Unluckily, Kevin can't take part today. He wishes you all to have a good time. But we have this lovely colleague here with us who is controlling the bot for today so that the robot is not taking over the world and do us all. So I give already a short introduction. This thing is called Spot. You most likely know about it. This is the RobotDoc version. You maybe also know this little bit 2 meter high other thing which is running around from Boston Dynamics. It has here on the back like rails where you can add up load up to 14 kilogram. That's quite a lot for this thing. And you can put up different kind of sensors and a compute base or basically server on back of his back. It can walk around autonomously means that you can tell the spot. Oh, you have to go here. You have to go there. You can teach this pass to the robot and then it will do it by itself. So it's very cool when you have like factory sites where you maybe do not want to walk around all day long or in the middle of the night in a hurricane, then this guy can also do it. So you do not get wet shoes. And in the end, it's also very customizable. So what you ever can put on the back and what fits there can be also somehow used from the bot itself. And then Boston Dynamics brings you an SDK so you can probably integrate with the system and the solution. You maybe see on the side and on the front like this kind of lids. It's like with lenses, cameras, so it has a all around awareness. And typically when it tries to walk into some directions, it also would avoid that it runs against me. So no worries when we run through the floor, except there's too many people, then you maybe it's like start playing rugby or football. And this guy is tackling you deep, so be aware. This is a short video which we have produced with one of our, or actually the customer of Roboverse to give you a little bit more impression about what actually is going to happen with spot. And in which cases it really makes sense to use it at the moment. So I said you can train it to run around anywhere typically runs a predefined pass, but you can also send it on discovery. Depending on which sensors it has, it can check for example the fire stuff. It can watch as this clocks, which you have on like water pipes and whatsoever makes picture out of it, send this information. Hey, all right, all good. Maybe too much water is running through it whatsoever. It didn't take stairs. So it's also possible to go anywhere else where a normal human should be able to go. Obviously it cannot fly. So it's a little bit difficult if there's a little bit too high stairs. Then you get maybe some problem. He also doesn't like the moving stairs. It's also not his favorite because suddenly the ground is also moving with him on top. Suspicious, but maybe in the future we will get there. So when I saw this thing, I was like, hmm, containers can run anywhere, right? We have seen it on cruise ships, on cars, on fighter jets. So why not on this thing? Should be possible. And can I put there all the Kubernetes on top? Is it maybe too much, maybe too less? So this is very stupid idea. One day I approached Kevin and Ben and was like, hey guys, do you want to sit down with me some evening? And just play around a little bit because I have a very, very stupid idea. And this is what this talk is today about. Our stupid idea, which we have, the things which we have tried out, what we have done so far, what we see also for the team because it has, and it's not just for fun. At the moment it's for fun, but we see real benefits also for the developers. Have a short look on the spec while I'm drinking. Spot comes with two different configurations. Spotcore, it's like the server which you could put on the back. It's this little gray box and Spotcore I.O., which is like the newest release version of it. When you see it's like more or less a standard CPU, some Giga from some SSDs, and surprisingly there's running Ubuntu. So already a very good foundation to make some stupid things with it. That was my hope. And then the guys told me like, listen Max, that's very crazy what you think about. And I think it will also help us because the newest version has suddenly a totally different architecture in a CPU. And we also look now on security because Boston Dynamics realized, hey, to have their server which is fully open on the back of a robot is maybe not the best idea. So let's do some kind of security and they put a lot of security measures into it. That means a few things which I will explain you were from the initial ideas and the thoughts what we can do on the Spotcore. And some parts are about Spotcore I.O. So that's why I will always try to differentiate a little bit between these both things. And because we got asked already a couple of times about, hey, is there a Kubernetes running on this green thing? No, or we don't know. This is a black box. What is running in the Boston Dynamics Spot? I have no idea. It could be, I don't know, Android with a Samsung phone, maybe, I don't know. But everything else what you can put on back for build your custom use cases for build and run your AI machine learning stuff, do camera or visual detection or audio detection, which is very, very cool. And you have a sensor and it can tells you, oh, there's some weird sound coming out of the wall. This is all running on the back of this machine and it interacts with the spot. After playing around a little bit, I found out, hey, there's already containers running. Awesome. My job is done. And you are all here. Thank you for being here. See you later. No, it's not that easy because running the Partainer on Core is actually quite nice. It makes fun. It's easy to use. The colleagues can just drop in their containers when they've built a software and they're up and running. Awesome. Oh, it's getting a little bit more complicated because Partainer isn't there anymore. It's a custom user interface where you have manually to upload your container. With every release, everything what you want to test, everything which you have ever developed needs to be packed into a tar ball with a Docker compose file, all configuration, everything else and your container manually uploaded. Sounds like a lot of pain, right, Ben? He says yes. So this was actually like, cool. We have two use cases. And so my initial idea to just throw some Kubernetes on top of it got even more stupid because it's like, hey, that sounds like a good case for playing around with Wasm and WebAssembly because we have different architectures. And this is one example, right? There's two versions of Spot which we have running around and the guys have like 10 other robots running around in the office. So the same software which is written or once written needs to run on all of these different platforms. And that's why we are thinking maybe Wasm could be here also an option. Also some downside with Partainer and this custom user interface is you have absolutely no idea what's going on on the side. The only thing that you recognize that the spot has maybe a little bit too much workload is when you don't work anymore. Also you cannot check any internal security stuff. Why you should check any security things? Well, it's very easy. This thing cannot fight so much. So if I would step now with my over 100 kilo on it, it would be not that able to stay up. And then I can plug into his backside. I can plug into the cable and upload everything what I want to upload a container which would like to run en route with some privileged escalations. It's maybe impropriate. Do you have any children here now? Yeah, but you see the back door is always like a dangerous thing. Also with the setup we do have problem with health checks obviously because it's just running container runtime. There are no probes and you write lifeless probes, nothing which try to keep my container running except the container runtime. And imagine this thing is really running around somewhere in the field and like doing this for hours coming back is charging is going back into the field to the next station. There's no human interaction. That's why we actually build it right. There should not any time all the time a human sitting beside and like look if it's really walking around, then you can walk by yourself. If it has this high level of autonomy, it would be actually good that it never fails. But how to ensure it. Well, life is probes and so on and so forth. Yeah, beside that. No, no awareness. You cannot define any custom roles. So everything runs in the same namespace. If you would run Kubernetes on it. It runs with the same privileges. It could communicate all this each other. And I said it's super easy than to just plug in a cable and upload whatever you want to upload. So security level is practically around zero for an industrial robot. Difficult. So it's a good starting point. You see it's experimental. What's done here, but we can do it better. I guess. Looking into the developer experience from from Ben and colleagues. I mentioned already everything what they have to develop. They need to test. They need to set up target, put a cable on top of the car on the spot, upload everything activated. Wait that it's unpacked, wait that it started and see if it maybe worked or maybe not. There's no way to emulate it locally. So you can just test locally if the software itself is running or not. But to see if the sensors are giving you any feedback and seeing if it can interact with spot itself with the SDK of spot. You don't know until you don't have deployed for all of your cloud native people. That sounds maybe a little bit horrible. And like the early 2000s, right? I see you liked uploading manually things. So our very dirty road of changes, which we have went and which we looked into. And this is very specific. This is not used for anything. What is anywhere in production? It's just like fun and playing around. But we have found out this has some really valuable reason to make the life of the colleagues, maybe easier. And also being in the future very valuable implementation for different spot scenarios for different robots, which can be mixed. And this is like the very far look in the future that you have different kind of robots interacting with each other, but always have like similar software running for scanning, for example, different scales and whatsoever. So our idea was, well, first of all, we would like to know what's going on in the spot. Because if I develop something and it's running somewhere, I do not want to have all the time the need and demand to log in and search for somewhere in the lockdowns. What's going on? I do not have any idea about metrics. And yeah, visibility is here. Big, big thing. The development cycle, as said, with tiring things up and getting onto the bot and getting up and running can take you one to ours. If you're in a software development project and you spend every time for everything that you want to test so much time, it's just pure waste for nowadays. We want to improve the availability of the containers, because this is some fun fact also with the container and the upload of it. If, for example, a sensor is not correctly connected and you upload the container stack with a Docker compose and the Docker container cannot connect anymore to the sensor, the whole stack crashes. That's problematic, a little bit annoying. So actually you would like to have something which takes action of it, right? It doesn't need to fail entirely. It would be just helpful if this container get maybe shut down and the rest can run. And obviously, security is the thing. So for the first step, replacing Portainer, and here we are speaking about Spotcore, the old version. I need to say it was very disappointed in replacing it because it was like just a fast thing of getting a snap with microcades, having the robot connected to the internet. It pulled microcades as installed, Kubernetes running, boom. Cloud native world. It makes things so easy, but it was a bit disappointing because I expected like, oh, we need to sit now here the whole weekend hackathon, or it's going to be crazy and like, damn it. Why do people make so good work? Don't do it. Also want to have some fun. So then we learned about, hey, this was the kindergarten version for testing around because this is really like just a server running around on some legs. Here's the Spotcore IO version with the hardened approach of running the container and you do not have any access anymore. You do not have any very fancy user interface where you can set some environment variables and whatsoever. This is all gone. You can just like run in some way a container. So we run Kubernetes in this container, makes it life easier. I have my Kubernetes, I'm happy. And as I said, I got the very stupid idea of like putting there also WebAssembly because we need to move around all the software to different parts and whatsoever. Now, some of my colleagues have built something called K-wasm and this is a very, very simple tool. You can test it out. It runs on kind, on micro-cade, on EKS, on AKS. It's just like an Helm chart on deployment and show that the runtime class is C run and you're good to go. That's it. Then you can run any kind of WebAssembly container was in your cluster. For me, again, very disappointingly because it was like five commands to have this thing up and running. However, business approach, we were like in a very short time on this experimental stage, getting at least the runtime environment as we want to have it. Why we really want to use WebAssembly? Well, if we have some modules, some core modules running in a WebAssembly module, we can ensure that if someone can access the spot, they have difficulties to manipulate the software. Before, in the old version, you have an unprotected container running. So getting into this container is the easiest thing. Manipulate what this container is doing is also easy. You will say like, ah, command max, take away the security perspective, not the problem. Yeah, more or less, what the things are built for is to make images, make videos. They store data and they send this data somewhere. Most likely in your kind of unprotected environment where you collect all the data and then process it somewhere. So you for sure can also find a way to upload stuff which you do not want to send somewhere else. And this is very easy because typically we have just a token for the connection, something which you can copy paste and then you can send whatever else you want to send there. Yeah, we were very fast on the goal. Here's just the commands which we run a little bit hand chart, make a little annotation in the next steps, apply the runtime class and then you can just throw in your containers, your pods if you want to have it. You're good to go. Sadly, again, why all these people in Kubernetes do a good, fantastic work and it's so easy to do the stuff. Horrible. We thought so. Because one thing which we didn't calculate with is that the colleagues, I'm sorry, they're developing mostly off the time in Python, which is good for this cases. But it's not so good for WebAssembly. There are a couple of approaches to turn your very easy Python program into something which can run in WebAssembly. Great. You can put something under it as Cpython whatsoever. All not a problem. But when you think about a very complicated piece of software which is popping out in the end from a machine learning, including some machine learning algorithms and do a couple of things and detection on the images which are taken from the spot or from the camera which is mounted in addition. There's a lot of stuff which you have to pull in. And then suddenly from this very easy piece of software, it's getting quite complicated. And this is what we currently can't handle. So we have to take a look or convince Ben and colleagues to use another programming language to make this running in a way that we can say like, oh, now we have a situation that this piece of software which was running before in a container can really run as a full functioning WebAssembly model. We don't have to, but this is where we would like to go. And because this was not enough for us, we were thinking about getting also some GitOps onto the spot. Why not? All the fancy topics at one. Do you have anything else which we should bring there? I'm excited for some more ideas. But then it's getting maybe really long nights. So putting them on kind GitOps operator like Aqua CD, also very easy thing. Just we need to ensure that it's connected with Wi-Fi with 5G to get some kind of container, in this case some public containers, pull it down, awesome works. For sure we have to do some interface configurations for the Aqua CD. It's not just like can pull it like this way. But this is like one config file for it. And what this mainly does and what it means for the colleagues and the development experience is that suddenly I do not have to tarball this whole stuff and plug again a cable and throw stuff and upload. I just built my container, throw it in my registry. Whenever this guy has a connection and like checking it, or when we say like in a frequent phase he needs to check it, it can just download and update this. All done. Boom. Easy. The only thing is that we did this very, very dirty. So you can do whatever you want to do with it because there is no good user management in it. It has also just the default settings. So, you know, admin and so passwords. It's not that there where we want to have it. And we also didn't evaluate if it's the right solution. It was just like, oh, we want to have some GitOps. We take Argo. Let's go. Fine. Maybe Flux is good. Maybe Harvester is good. I don't know. We need to evaluate it because it was just like, we need to throw some more stuff on it to make it a little bit more cooler and to get this experiment to a point where we say like, oh, look, we really bring some improvement to it. And we need to think about a full process around it. At the moment this is like, yeah, still throwing things around in the end. But we need to think about how to make a full development life cycle. And then we can also think a few steps more further, right? Because we don't use this for ourselves. Would be funny if you can bring in the office me coffee or something like this would be very expensive coffee machine. This is really for the end customers and they will have in the future a couple of these things running around. And maybe we will have a couple of end customers again, where we take care of the maintenance of the software we bring updates and so on so forth. So then all this development cycle and make very easy updates come hand in hand. However, long story short, what we gained on newly capable capabilities through this approach. I said the development cycle is now way more pleasant. We ensure that this end to end really integrated and that it also fits with the platform with which the guys have already built. So there's a little bit more on management console and stuff around like this, which all needs to be integrated. However, this means in the end that we are way faster in delivering software, which is awesome. Security, yes, again, I know. But this is a very, very vital point. Everything what is somehow running around out there and is counting into the direction of IOT and more or less the one or the other way is very poor in security when it's about hardware level. Whenever you can see somewhere a device and you can plug in the cable, it's almost never protected at all. The same thing for a robot. Catch it. I said it's not so difficult to catch it. You just plug in the cable and you're good to go. For sure, you need to know what needs to be done. But here's the thing. If you see this robot running around on a manufacturing place, it will run there also around maybe tomorrow and the day after so you can prepare for it. And many stuff of it you can find out how it works. The autonomy is now improved. We can see or any other GitOps operator will help to keep the things alive. But also having Kubernetes, for example, running on the platform will help us to define lifeness probes and readiness probes and startup probes and whatsoever. We can take a look with different additional controller that it really function very well, that it receives the data from the camera, that it pushes the data somewhere else away. And also if some specification changes happen, we can control it now and we are more aware of what's going on in the bot. And yeah, in the end, you all know what you can do when you're running full-fledged Kubernetes. You can think about putting their policy management on top. You can just deploy images which are signed with a specific certificate. Boom. The security problem is solved and whenever else someone tried to put their cable into and upload something, most likely this person will not have success to start it then anymore. We can apply very good user management. If you have full access to the node under it, we can also be there good in the clarification of the users and what they are allowed to do and what they are not allowed to do. And yeah, we have tons of other ideas, but we can do on it. And if you have a robot, you get stupid ideas. So maybe you can start small. There's also a mini version of it where you can work with. Fun fact, the same software which is running on this guy can run on that guy. It's no magic at all, especially when you put it into a wasn't runtime. It runs on this little thing, runs on that thing. You get the same output. That's very nice and awesome. So what you have learned spot is really a server running around with some sensors on it. So if you have a data data center full of it could be also quite funny. It's almost disappointing. As I said earlier, how easy it is to set the things up. Meanwhile, I really wish there would be more problems. For sure, there's like five more kilometers to run until it's in a fine-grained state where we say, like, this is something which we can really use in production. I don't need to discuss about it. But just like the initial setup is like too easy to go. And machine learning is nothing which is made for the setup yet. At least not with a wasm container. It still can run in the normal containers. That's a nice thing in our setup that you can run still a normal container and a wasm container beside. So if you have some very critical components, you can put them already as a wasm. And if you have something more uncritical, it can still run in a normal container. Yeah, keep in mind, this is just like big guys playing around with a robot toy. Right? That's nothing else than that. And yeah, I just can say in the end like that our cloud native ecosystem is really majoring because it's so easy so fast to do such things and to implement any kind of stupid idea to such a robot. And if you get yourself one, maybe not this one, you need to throw some money together for it. Then you can do a lot of fun implementations for it. So thank you very much and enjoy your day.