 Welcome everyone, hope you're enjoying KubeCon so far. So yeah, welcome to the talk on zero downtime migration of stateful VMs and Kubernetes. It's quite a lot of work, so let's get into it. Yeah, so first of all a little bit of me, I'm Felistas, I work at Loupolaps currently on virtualization stuff. So I regularly post about it on social media, if you guys are interested. But yeah, let's get right into it. So first of all, we gotta think about what VMs actually are or how they're being seen in the container age, you might say. Because a lot of people's first thought when they hear VMs isn't actually necessarily something positive. Most people would probably, especially if they're used to containers and Kubernetes, probably used to having pretty high overheads, them being fairly resource intensive in comparison. Also them being fairly slow to start. Usually VMs, especially cold starts, take quite a while because it's a bootable kernel. So, and there's also a lot of projects that provide these kinds of things. For example, KVM is basically what everything is built on at this point, right? It's the kernel API to the virtualization. But there are also layers of interaction on top of that. For example, we've got Libert, which is fairly common. And if you've worked, for example, with the virtual machine manager on the next desktop, you probably used to that tech already. And of course, there's other solutions, such as, for example, Kibu that is able to also serve for most virtualization needs. But all of these are kind of like an old solution. They've been around for quite a while. And while some of them are really useful, like, for example, KVM, some of them are slowly starting to show their age. And if containers are so great, and VMs are still around, then the question, of course, is why are VMs still around? Why are people still building things on virtual machines? Because the chances, even if you're running a container, if that container is gonna be running in a public cloud, probably doesn't run directly metal, runs on a virtual machine for one reason or another. So you may be wondering, why is that? Why are people still using virtual machines? And in our minds, the reason for that is, is that, well, for CATSOL type of workloads, like something that's stateless, mostly, containers, airports, and the Kubernetes context are really, really useful. But a lot of systems are Pets, and a lot of systems cannot really not be Pets, like GameService, for example. And for those, it's just not the right layer of abstraction. So what we were asking ourselves, how would a container of stateful applications look like? What would be the equivalent of a container for a virtual machine, essentially? And of course, such a thing would have to be pretty fast, right? You wanna be able to have very small image sizes, kind of like how all Docker images aren't necessarily small all of the time, but you can cache layers and so on. You wanna have those features. You wanna have a really low overhead. Preferably, you don't want those to be comparable to a virtual machine. You want them to be more like containers where it's like milliseconds, or weather modules, or it's microseconds to start them. And of course, we also want them, yeah, as we just said, a faster start. They really need to be performant. They really need to be able to, like, resume really quickly. Also, networking support is definitely an issue. If anyone has worked with, for example, like Kimo, trying to just get one port open into the virtual machine from the outside, especially if you're in Kubernetes or something, it's gonna be a hassle. Whereas with Docker, well, or Kubernetes in this case, you've got ports, you've got all of these abstraction layers that are super useful, but they aren't really in virtual machines today. So good networking support would be important. And they should also be easy to build and distribute. I wanna have a Docker image of virtual machines. I wanna have a good developer primitive that you can use for it. And also, of course, we want them to be easy to scale. The biggest benefit of a status application is if you need more capacity, you just spin a parallel instances for a stateful application. That approach to straight up doesn't work, because, well, if you have state, you can't just spin a parallel thing so it's gonna be out of sync. So this is something that a container of stateful applications in our minds must be able to do. So now we're wondering, what if this thing existed? What could we do? So we have a few hypotheticals. For example, we could do some really cool stuff around deployment on debugging. Deploying an application in a stateful context here would simply mean, well, you just move the running application to a server. That's your entire deployment. You wanna debug it locally. All right, you just move the server back to your local system. That's how that thing should work. That's how you'll be able to do it with containers. What about, what if you wanna move between cloud providers? Well, with such a hypothetical construct, you'd be able to move from on-prem to like GCP. And then it turns out, okay, next day, GZ, to AWS, sorry. And it turns out next day, GCP is cheaper. So you just move the container over to GCP with no downtime. That would be great. Then Azure offers a simple rate. That's also cool. And eventually you find out, okay, energy prices have dropped. You're able to resume it on your local system. There you go, you've just moved it back. That would be a really, really cool feature to have. And similarly, so since you're talking about stateful applications here, if you were able to do stuff like, for example, and I know everyone here is talking about AI, so I thought I might as well. But for example, if you've got a very expensive, a very computationally expensive workload, let's say, for example, an LLM or really any batch process running and you wanna scale it up to a larger system. Well, with a stateful container system, you can just move that to a different thing without downtime. And this really works for any kind of application. As long as there's not some network transparency involved, for example, I have a phone here that runs Mainland Linux on it, and this also does it. So you would be able, and both of them use the same display protocol and so on, you would be able to just move a application from your phone to your laptop. Maybe you guys know from Avatar, there's this scene where they move an application from your phone to a bigger computer. You need to take the app with you on your phone and resume it anywhere. Well, that would be impossible if we had such an abstraction layer, right? And this would work across different devices within some constraints, of course, but for example, if you had Linux and all of them, that would work. And of course, probably the most useful use case here in a cloud context would be to move services closer to the users. So like, while you have a CDN for status applications or edge functions or something, that doesn't work for stateful applications. You can't just move them closer to a user and if you can, would be extremely expensive. But if such a system, you would be able to do it. So you're probably wondering like, this is like science fiction movie stuff, right? This would be kind of cool if it existed, but it doesn't and it's impossible, right? So the thing is, we actually gonna have a demo later. I want you guys to download Minecraft if you can. And this is gonna sound super random, but if you have a computer available, if you don't mind just download it from Minecraft.net version 1.12.2, there's gonna be a demo with it later. So yeah, exactly. So, but before we do that, let's get started and ask ourselves the question, why does it doesn't exist yet? Like it seems like one of those things, like it would be super useful. Why doesn't it exist already? And from our understanding of it at least, a lot of it has got to do with like data type of a system, these things have been around for decades and as such they have accumulated a craft and it's kind of hard to use. Live migration in particular, the thing that allows you to move from A to B is extremely complicated. It's a whole process into setup and a process you need to optimize for or you're gonna risk, it's also very risky. There's a risk involved in moving it. If it breaks, you might not be able to resume it again. Also, the fast startup times. Well, VMs need to actually stream and you need to have the entire virtual machine locally available before you're able to resume it. So like you can't stream it in. Like resin module, for example, right? You're able to just like use instantiate streaming. That doesn't work for a virtual machine today. So this is definitely a bigger problem. Also, if you move from A to B, like that example where I showed, like moving from for example, like Frankfurt to New York. Well, you could do that, yeah, but your connections would break. You'd have a different IP address and even if you say like any cast, you'd still have that issue. You still have connections breaking during the migration. So it's not server downtime. Also packaging is a bit of a bar. It's just a mess. Like trying, there's no Docker file of a virtual machines at the moment, at least not in any modern reincarnation. And it's also like, generally speaking, it doesn't integrate that well with modern orchestrators. We're a coupon here, right? So if you want to have this kind of stuff instead of Kubernetes, there are a few solutions like Kubert, for example, but most of them they take existing technology and port it over to Kubernetes. It's not like something that was developed well before after Kubernetes actually existed and kind of fixed the orchestration issues for us. So you might be wondering if one of these things why did, how did you actually still do it? So before I start with that, I'm gonna give you a very short overview of existing live migration tech. So this is the thing that actually, or techniques in this case, that actually make it possible. There's two separate ones. The biggest or the first one is gonna be pre-copy migration. So in this case, what you do is you, well, the VM is running on the source node, the one you're migrating from. You just, you do it in rounds. You try to like amortize the cost of moving it by continuously checking for changes and just copying it over to the remote host until the difference is not enough. Then you resume it. That's one way of doing it. The other way is post copy migration. And what you do here is you just stop the VM on the source node. That's gonna lead to downtime, obviously. Same as in the case of pre-copy migration. And you're then gonna just page in the changes over the network from the remote. And as you can probably imagine, that's an extremely expensive operation. Maybe less so in your own data center. But doing this between continents is gonna lead to a lot of problems. So, but if you take a look at these two things and we kind of combine them, we get like something that more or less resembles a remote memory solution. And that's something we have ideas on how you can solve this kind of issues. But there are, even if there are these options, they are typically very, very application specific. Like if you've got a live migration protocol, that's probably gonna be key moves, live migration protocol, or it's gonna be VMware's live migration protocol, all of which are very useful, but they aren't really adaptable to other use cases. They are also extremely data sensitive. Most of these things are designed, you move from one data center with like maybe a millisecond maximum of latency. They don't work over the public internet and they definitely don't work securely over the public internet. And also they depend on specific transports, like are they may or something, you can't just swap out the transport layer if you want to. And if you just want one extra feature in such a system, like for example, signal the application, that it should like do something after resuming or before suspending, it's really, really hard to do that. These protocols are not designed to be extensible by design. So one of these more universal approaches to remote memory, if you look at this as a remote memory issue, is user fault FD. User fault FD is a kernel API that allows you to essentially stream in memory as you're accessing it, kind of allowing you to do the post copy migration. And we actually even tried this out, we built an open source library for Golang, it's called the user fault FD Go, very simple bindings. And if you can probably imagine, the interface you expose such a resource as, it's just an IOTA reader ads, which is probably what everyone's ever wrote, you just go, it's really hard to work with. And what that does allow you to do is you can just register a resource, you map it essentially, you register it on a byte slice. And then when you access an individual byte and that byte slice, you will fetch it in from the remote. So this is like one API that allows you to do it. But as you can see from this interface, this is only for reading, it's only for streaming and changes, right? So that memory region, you cannot track. This is where we first of all try to use file based synchronization. Cause you know, you can represent memory as a file, for example, and you're then able to continuously pull that file for changes. You're gonna be able to visit an R sync type of protocol. And so what you tried, for example, was to use what many of us developed, the highway hash, for example, very, very fast hashing algorithm. So you just like hash every single chunk, you send them to the remote, you ask for data checks and then you only synchronize what actually happens. This would be a way of implementing, of implementing pre copy migration, which would be kind of useful. Now, if we look at, and on the same time, Shiv, who's our founder is right in the front, he was working on something completely different. He was actually working with CRFS, which is a way to access, it's a way to stream in containers using a fuse. And this is a kind of useful problem, but you might be wondering like, how does this relate to using memory migration? So Shiv came up with this thing called magic block devices. They are essentially a, essentially a abstraction layer on top of NBD in this use case, which do a similar thing to CRFS, except they're able to kind of like combine both of the pre and post copy migration thing. We're doing background push and pull and background synchronization, which is kind of useful. And while of course we can't directly use this with energy, there's one component here that's quite interesting, which is the NBD protocol. And the NBD protocol is conceptually very simple. But yeah, so we thought like, okay, this might be our like interface that we're able to do, that we're able to track changes with. So we went ahead and just build an open source NBD library. This is also open source, of course. And that essentially allows you to write a Medinevrock block device in Golang without having to use any Seagull, which is quite useful. And the biggest benefit though, if I compare this to the, to use the fault of the earlier, is well, here we can actually track the writes. And if we're able to track the writes, we're able to do both pre and post copy migration with this, with this analogy. And yeah, the biggest problem here is that, NBD is also designed for LAN, at least in most of the cases. You won't be seeing or using like NBD over the public. I mean, you can, there is still a support, but you probably aren't doing it. So the biggest question here is, well, this would allow us to basically access any kind of remote resource we can represent using the network. But it's not necessarily optimized to use, to work in high RGT scenarios, like for example, moving between continents. So this is where remap comes in, which is the thing I mentioned in the description of the talk. This is like a universal resource mount and migration solution. So it tries to fix that like fragmentation in between different current ways of doing this. It's called just remote MAP. It does more than that, but that's basically it. And actually this is part of a research paper I will be linking earlier. So this is all completely open. Now what remap does for you is actually a lot of things. First of all, it's completely language independent. You can migrate and or mount any remote memory region with this using Rust go lang or JavaScript. It really doesn't matter. It's completely independent of language. And it also handles that like background pull issue I mentioned earlier. So it's able to pull it in the background. And this also works for example, doing like an initialization phase. So you don't need to worry about initialization overhead of such a technology. It also contains hooks you can use to extend the entire protocol with. So you're able to take that thing I mentioned. You just need to do one additional thing before you resume or before you respond. So before you suspend it, you can do this using this library. And we also have support for pull priority heuristics. I mentioned that shift worked on the NBD protocol. For example, if you had like a remote X4 file system, there are super blocks on there and you kind of know where those are. By providing a pull priority heuristic to remap, you're able to essentially pull those super blocks first and you're able to access it much, much more quickly than it would be by just fetching it directly on demand. We also can do workload analysis. So we're able to find out when changes happen to the memory region because we can track both reads and writes to it and then optimize for the perfect point to minimize the latency of the migration, which is really quite useful. And similarly, so this supports both mounts and migrations. So what this allows us to do is we're able to use a single unified API to both mount a remote resource and to migrate it around. It's one library, it's a completely unified system. And also of course this defines network protocol, although it doesn't depend on any specific network protocol, it's able to use different ones. But yeah, it defines some semantics for the network protocol S. And so also it allows you to do is caching. And that's quite useful. So for example, you're able to like, if you do writes you're able to just cache them locally before you write them back and things like that, which is quite useful. And also the migration protocol, as I mentioned, it's independent of the transfer layer, but it does provide you with some semantics. For example, there are two phases during the migration there, we were able to recover from one of them. We are able to basically split the pulling the chunks to the remote machine. And all of that stuff, like for you. And then we have a very, very short amount where you would usually, I'm gonna talk why it's not a case, would usually have downtime during the migration. It's a critical phase. And this protocol is very, very simple to use. Those who have a work with Torrents will know of this terminology, but for example, we just call it the seeder. This is the thing if you call that seed on a resource that you use a remover, you're able to basically expose this to be migrated from. Then you can use a leecher to then connect to this remote resource. And finally you call finalize when it has reached a level of local availability that you're comfortable with. But this is like an extremely generic protocol that works really for every kind of migration scenario for any kind of resource. So we may be wondering, well, this is using NBD. This is a network block device. It's nothing to do with memory. This must be slow. Well, it turns out that's actually not the case at all. We compared this hats on to use a fall FD, both of them are implemented in goals. I think it's fair in this case. And as you can see, there's some more spread as you could probably imagine. And this is running on a second's RTT. But it's fairly comparable in terms of latency. Now where it really gets interesting is if you migrate over RTT, because we are able to do the background push and pull that we wouldn't be able to do with, for example, use a fall FD, because you couldn't track the rights to the memory regions, we wouldn't be able to do it. And here we are able to essentially, by using current background pulls, to just like completely reduce on care about the RTT, which is super useful for migration. This is the latency story, but the throughput story is really where it shines. So this is much, much faster than use a fall FD, even though it's in user space. Like we're able, like use a fall FD from our testing was able to get like a half a gigabyte per second, which is quite fast. You need a very, very fast that running to saturate that capacity. But when it comes to migrating with, well, this remap solution, we're able to do like three gigabytes per second, which is really a lot faster, even though it's completely in user space and doesn't require anything in the kernel. So this was quite helpful. And of course, the same thing applies over RTT, because you've got a caching layer, because you've got locking and everything supported as part of the protocol. You're able to do really, really interesting things and you're able to much, much more gradually decrease the throughput of a migration when it's happening over a larger RTT. And of course, the rights very similar story. You don't need to do the rights to the remote region. You can look at local one and sync it back safely and without having to worry about concurrency issues. So instead of your rights were dropping immediately down to basic zero, you're able to just keep it locally and you don't have to worry about that kind of these kind of problems. So now you'd be wondering, okay, this is a very generic migration solution. Now, how do we use this for a hypervisor? And what we were trying to do, actually was use firecracker here. But I must stress, this is not like a firecracker integration thing. This is very generic. This can work with KeyMood. This can work with U2Duelist. It doesn't really matter. You can make it any kind of resource. We just use firecracker because it's very simple. And actually we have an open source fork of firecracker over here that you can use that actually as support for this. And the changes we needed to do to use this technology instead of firecracker because it's language independent is very minimal. Essentially it boils down to this. You just resume a snapshot and pass the map shared flag. The entire change to firecracker is 184 lines and it can suddenly live migrate, which was quite interesting. Now, this is the firecracker integration. Sorry, this is quite interesting. But when it comes to actually using this, you probably don't want to work with firecracker directly. Rather you want to use a, you want to use a, an orchestration framework or something. And this is where architect comes in. Architect uses these two tools and essentially orchestrates them with remote firecracker. And it contains different tooling. It contains, for example, well, this is the academic Rosidory, of course, and contains different tooling. For example, it contains a package. This is solving the Docker file equivalent I mentioned earlier. This allows you to package a virtual machine into something that is equivalent of an OCI image. You also have a registry. This allows you to serve these images just like a Docker registry or an OCI registry would allow you to do. You also have a control plane. And this control plane is able to do some, some optimizations around RTT that will, that integrates deeply with this like migration thing that we're able to essentially, we are only limited by light speed for these kinds of solutions. So we don't need to worry about, you don't need to worry about, we don't need to worry about integration or calling REST APIs or something. It's all very, very tight integrated. And actually the API we provided this is extremely simple. Like this is the entire, this is the entire example for the entire API. You've got one get request. You've got one post endpoint and you've got one delete endpoint. And that's all you need to launch any applications of a machine to list the ones you are finding and to migrate them from A to B. So this is much, much easier than to have to do the manual migration thing. We need to like open up ports on the firewall most of the cases where you need to like initiate a process of migrating it. And of course we're at KubeCon here, right? So we've got an operator for all of this, the Kubernetes operator. And what this essentially does is it connects the, it connects this migration technology with Kubernetes and the way it does this is actually quite simple. So we have a CRD. It's called an instance. That you can just specify a package. That's the thing that the registry serves and a node that you would like the thing to run on. And then you can just KubeCon to apply and that's running a virtual machine. You wanna move it from A to B. You just change that one string and the VM is migrated without any kind of downtime. And similarly, so if you wanna delete the virtual machine or if you wanna watch for events and so on, it's all very closely integrated with the way that the operators work. The actual networking is a very interesting story. So for the VM network interfaces we've built a custom, a custom net-based solution that allows you to get from DVM's perspective, make sure that when it's been migrated from A to B, there's not going to be any kind of like visible difference to it. We'll have the same MAC address, gonna have the same, the same thing and use network namespaces to do this. And this solves one part of the puzzle. This solves the part of the puzzle where we move the VM and the VM needs to have the same worldview essentially. But if we wanna have the other problem, which is while we move from A to B, well, that's gonna mean that our server is gonna have physically separate IP address. And this is where another loop-or-lapse project we've written working on called link comes into play. And this is essentially a new networking primitive. And what this allows you to do is, well, a lot of things, but among other, but probably the most important thing is that it fixes this problem where you get different IP addresses. And it's not just a proxy, so, yeah. So if you take a look at what it actually does, well, first of all, it's completely independent of the network topology. This works by re-running it, by gridding it from within LAN, whether you're depending on your workstation or you have this inside of a data center. It's completely independent. It doesn't require any kind of public IP on the nodes. And it also, of course, it works behind firewalls. So this fixes a lot of the issues we have around complexities with migration. The nodes also no longer require a public IP address, so you can just save the IP addresses. You don't need them anymore. That's quite useful. And it also handles TLS encryption. Now, of course, it doesn't work only with TLS, but in case you're using it, that's one of the things you can do with it. And we're also able, this is probably the most important part since this is about live migration, we're able to keep the connection of the live. So even if you're migrating during the migration, there will not be any visible downtime to you as the user. If you're sending data, if you're copying a file or whatever into a server, and you migrate it from A to B, that copy process continues while DVM is being migrated. So there's literally zero downtime from an end user perspective and from DVM perspective as well. So that's quite useful. And of course, similarly to the control plane, this integrates very tightly with architects. So we are able to just save a lot of RTTs that you would usually have with such a deployment, which keeps migration times down. And I won't be able to talk too much about link today, but if you wanna know more about this, just go to loopallapps.io and there's a field use entry email in and we'll be sure to contact you when it's ready. But of course, now I've explained, at least in a very short fashion, what this is actually about. But that's the question like, does this actually work? In order to show you this, I've prepared a little demo. So we're gonna take with us, or actually we're gonna use an application to measure Vettis latency. And we're gonna have a Vettis server running instead of Frankfurt, which is well across the ocean. In this case, and what we're going to be doing is just gonna be moving this Vettis server all the way over to New York City. And then, I mean, we're in Chicago, right? So we're just gonna be moving it over to Chicago and we can move it back if we want to. So this is a very, very long distance. Like we're talking like 10,000 kilometers here. This is not where you would usually be able to live make virtual machines. So yeah, let's get right into it. Actually, I put this mic up here. It still works. It's a little bit hard to hold the mic while I'm doing this. Okay, so essentially what I'm gonna do here is we have this CRD, and this is actually for my graph. Let me switch over to Vettis. And this is currently running inside of Frankfurt. So if you take a look over here, you'll be able to see that this is currently running on Kubernetes. Let me see, so this is currently running. And as you can see, you can use the normal control watch command you're already used to. And this is running on Frankfurt, on the Node-Frankfork. And these worker nodes, they are just a Kubernetes node name. If you run a coop control get node, you would get the same result. So if I run a little latency test over here, don't take a lot of time. I'm connected to a mobile hotspot here, by the way. So if you see this is the latency you need to remove like 50 milliseconds at least from this to get the actual latency. This is currently the latency we get. And again, this is using a persistent connection. We're not creating a new connection for every get and set request. We're actually just reusing the connection. So now what I'm gonna do is I'm just gonna go in here and I'm gonna change this node name. And for example, now we're gonna be moving into New York City. And usually this again is a very complicated process. But here we just coop control apply it. We'll take a second. And as you can see, we're currently migrating. It's in a migrating state, right? And we can jump over here. I'm not completely sure what's happening. Did I apply it? It's unchanged. Yeah, this is the thing when you do live demos, right? Let me just try and move it over to a different instance one second. I'll try and move it to Chicago. All right, let's see whether this works. Otherwise it's not a problem. I'll just delete the VM and start it again one second. Okay, here we go. Now we don't have to VM anymore. I'm not completely sure what's going on. I'm just going to be migrating. Again, this is over a mobile hotspot. So I'll just, I recorded this on a video right before we started to talk. So I think I'm just gonna try it over the video. Where is it? Right over here. This is when you always have a fallback plan in case things go wrong. Okay, so for example, over here, we have a web approach machine and we just applied this, this is changed. And what you will see is there will be a short latency spike. And now we actually, this migrates from New York City to Frankfurt. But as you can see, there's a short latency spike but that's it. The connection stay is completely connected during the entire demo, which is quite useful. And now if we go back and we change our node name to for example, Frankfurt over here, which again is moving in the other direction. And we switch back to our latency graph. It's migrating. You can see the chain. You can see the pull process all the way down there in the bottom. And as you can see, as the migration is happening, that latency is not changing whatsoever. So this is gonna take a little bit of time. Again, we're jumping over the Atlantic Ocean here. So you're gonna have some problems with network connectivity. So as you can see here, it's gonna resume. And there we go. Now that VM has moved physically from Chicago in this case, all the way over to Frankfurt. And as you can see, there's of course an increase in latency. Usually, of course, as I want to show you in the demo, you would do this the other way around. You would move this virtual machine from for example, Frankfurt closer to us right now. And there's like, again, there's no downtime here. So this is the most important part about this. And actually here we are now in Frankfurt. If I move it back, which you should see in just a second, you can see that there is some pull progress. And again, this is gonna take 10 seconds or something, but it's without any relevance because there's no downtime to you at this point. Just take about four seconds. I guess it's currently in flight and landing soon. So here we go. And now you can see the very visible latency drop that you get. So if you move this virtual machine, I'm just gonna, oh, it's already passed. And so if you do this, you're able to suddenly move your applications closer to your users. You could, for example, like if, for example, the, what is it called, the time zone, for example, changes in our people, like going to sleep in Europe and not over here in the US, then you would have the, you would be able to move the virtual machine closer. And actually I'm hoping that a second demo works, but so, okay. One second. Okay, so this is the protocol that we control technically speaking. We could affect this, right? This is a registered server. We're running get and set commands over it. What if we use something that we do not control? What if we use, for example, a Minecraft server? Well, we're gonna do the same thing. Actually, I'm probably gonna skip the Frankfurt server, but other than that, we could run this virtual machine and we're able to move this thing, even though we don't know the protocol, even though we don't control the protocol, we can't change it. Gonna move it over to New York City and then we're just gonna move it over to Chicago. So this is gonna allow us for some very, very interesting use cases. And again, this is where I mentioned earlier. If you installed Minecraft, that would be the perfect time to join. So this is actually publicly accessible under this domain. We've got 20 users maximum, so we don't accidentally de-doss our infrastructure over here. So first come, first served. But yeah, and it's important. We don't need to pick this version because we have the server running for the specific version. Of course, this would work with any newer version as well, but in case those are interested, I'm just feel free to join. And I'll try it on my end and we'll see. So again, let's take a look at the actual instance. So over here you can see we've got the, we've got the Minecraft instance, the Minecraft CRD, and we're gonna spin this up in New York City. So I'm just gonna run kubectl apply. All right, and I'm hoping we're gonna see the, over here, actually I think what's happening over here, I just, I lost the internet connection on this computer. Like if I press enter here, it's not doing anything. So Trezy, would you mind checking your hotspot please? It's something seems to be wrong. All right, may have connected. Yeah, I connected to the micromic Wi-Fi. So what happened here was I set up these terminals earlier. They're running T-Max, thanks. And so we're just gonna have to reconnect to that hotspot, one second please. It is a thing, when you run a public demo over a conference Wi-Fi, there's gonna be some issues here. Is it loading? Perfect, perfect. So we can see some people are able to join. I'll be hopefully ending those people in just a second. Trezy, did you start it? It's not showing up. I'll just toggle it for a second. Okay, here we go, dot slash phone. Okay, now I'm gonna have to kill these connections one second. All right, here we go. So sorry about that. Okay, here we go. So currently we've got the virtual machine running. And now as you can see, this is only New York City server. And if people have joined, I will also join in just a second, right over here. And let's see. Okay, here we go. And currently we have a few people connect that, which is pretty cool. So this is working. Now just feel free to play around with the server. And while this is happening, while you guys are connected to the server, I will just be moving this to New York City. Or actually to Chicago, we're already in New York City. All right, so here we go. We started it. And now you should see the full progress. There we go. It's not pulling. And again, if I'm just playing here, I can destroy blocks or anything like while the immigration is running. And you will actually, I can't do them here because I'm in the response area. But here we go. So the virtual machine just migrated. People were connected to the virtual machine. We changed things. We changed everything. We're able to connect to it back. So now we can do the same thing. And I'll jump back over to the, sorry, over here. And I'm just gonna move it back to New York City. So we're just gonna move it back. And what you will be able to see in, yeah, there we go. Something's wrong with my network. It takes a little bit longer than your city. But here we go. So you can see that virtual machine is being migrated right now while I'm playing the server. And again, this is with an application that we do not control. So like we can just remove blocks here. And you can see there's like no latency drops even if you were like poking in a block, for example, to remove it during that single break. Thanks. Right, thanks a lot. And I see, by the way, the numbers you see down here with the resumption times is actually faster than this in reality. This is including an RTT to the server and back. So it's a little bit faster than this in reality. So yeah, I'll be sure to do the remos but it's there more later. If you guys are interested, hopefully this time before working hotspot, but yeah. Okay, so again, now that we have this compute unit we can move it around at least assuming we have an internet connection on our workstation. I guess we're able to do all of the things we mentioned earlier. We're able to just deploy an application by just literally physically moving it into the clouds. Again, these were different servers. They're also by Aquinix. This is like a public cloud service we can just use. And we can do the entire thing we mentioned earlier. The thing about moving between cloud providers actually we were moving against, we were moving between cloud providers already. We were moving across the ocean with these servers. So yeah, that's a solved problem. If you see the rate for an AWS server, for example, change, you can just move the GCP with no issues. This I should mention also works with nested virtualization. Recent space should be using this maybe, but it's possible. You could even move this between different like public cloud providers if they have nested virtualization support. So that's quite useful. That thing, well, we just ran a Minecraft server. I guess that's fixed. So we can move a running application while it's computing between things. That also works. And the desktop demo, the same thing. As long as it's network transparent, you're able to migrate it from A to B without issues. That avatar thing where you move an application from your phone to your computer is suddenly possible. And of course, moving a stateful service closer to the user. Well, that's what we just did with that Minecraft demo, for example. And again, if anyone's interested, I can demo the latest thing later. It's now that we have an internet connection that actually does this. So yeah, if you guys are more interested in some more information on this, there's an entire research paper on this. It's 130 pages that just goes into every single detail you could possibly imagine about this. And it's all over GitHub. It's been peer reviewed, I think a month ago at this point. And so if you're interested in this, just feel free to check it out. And all of the benchmarks, by the way I mentioned, with the like latency and so on, all open source, so yeah. And not just the paper itself, it's open source, the library remap. Well, that's open source. Of course there's as well. The NBD server is also open source. So if you just wanna use an NBD server or build parts of this also possible. Architect, the actual migration, the orchestrator is also open source on the Loupolab's GitHub account. And the firecracker fork that we mentioned, although it's again, it's very minimal, is also open source. So yeah. And of course, again, just come talk to us. We're gonna be moving all around the conference. Actually there's three of you right here in front that in case you guys are interested in, you can just talk to you about it. We were hats, so you'll probably be able to find us pretty quickly. So yeah. And again, if you wanna find out about social media, that's my information. And also if you want more information, feel free to join the Loupolab's Discord at loupolabs.io-discord. And with that, I thank you. Thank you.