 Today, I'm going to talk to you about VM to VM communication mechanism for embedded. And you, if you're familiar with virtualization or virtualization technologies, well, actually we have been discussing IO virtualization and VM to VM communication for a very long time, more than 10 years, maybe more than 20 years, you know, XNPP drivers, VRTIO, and now have been around for a very long time. So what's actually new about it? Why we're even here discussing it? The purpose of this talk is to talk to you about how virtualization in embedded is different from virtualization in the data center, and in fact, we tend to call it static partitioning rather than virtualization. And how these requirements are not different also for communication and IO virtualization. And to show you how they affect existing communication protocols and whether we, you know, asking the question of whether we need new ones. And if so, how do the new protocol look like? What are the requirements? So let me start from what is already out there that probably you are already familiar with or at least you have already heard of. There are XNPP drivers, there is VRTIO, both of them are discoverable and dynamic. So the purpose of them is so that at runtime you can detect and use new, parallelize IO, you can hot plug new, parallelize IO devices, it's also meant to be able to create connection at runtime. So it's not like you define statically who can communicate with what. And it's made for IO virtualization rather than VM to VM communication. So think about it, the purpose of these protocols was so that a VM fully abstracted running in the cloud can have access to the network, access to the disk without knowing where it's running. Whether it's one server or another server, one cloud provider or a different cloud provider. And ideally so abstracted that you can even live migrate one VM from one server to the other. This is extremely different from what we do in embedded. In embedded we know exactly what's running where. We are not building a generic VM, we are building a purposefully built and designed environment for one specific partition we know beforehand. And also we are not trying to communicate with who knows what. We are communicating with a neighbor with another partition on the same SOC. We are not likely even going to go outside the SOC. What this means is a lot of the optimization and configuration parameters and design choices made for these protocols, they are sometimes even counterproductive for embedded. But let's go into how they are designed. So I'm going to try to use common terminology so that it applies to both. There is a front end driver in the unprivileged VM. In the Xen world we call it DOMU, U for unprivileged. That is just your regular Linux driver creating a virtual network card, a virtual block device. It's connecting to a special driver that we call backend. In VertiOS that's typically called VMM. It's really a proxy. What this guy is doing is receiving connection from the front end and then connecting to the real device. So proxying access to the real device underneath. So a couple of interesting details about this is these protocols are typically based on memory sharing rather than memory copying. This is an important detail because if you have any safety requirement, sometimes memory sharing is not allowed. Finally, VertiOS expects today privileged back ends. What does it mean? It means that in order to share memory the back ends needs to be able to map any page the guest passes to the back end, which needs full privilege and which is another thing that you might not have the ability to grant to the back end if you have any safety requirement. Okay, so now before we go into what are the specific requirement that we have in static partitioning scenarios and I'm going to go through five different ways to do VMM communication today. So please bear with me. But before we go into that, I was going to spend a couple of words on what is static partitioning and what are the requirements of static partitioning so that at least we start from common ground. So static partitioning is similar to virtualization and many of the underlying hardware capabilities are the same such as on ARM is the yield to execution level is going to be used in both. Probably we call hypervisor the piece of software that does both things, but the implementation is usually very different. So in one case is about dynamic VM packing as many as you can on a single host. In the other case is about dividing up your board into few partitions that you define statically beforehand. So in static partitioning, the focus is not much about running many, many VMs. The focus is running few VMs which typically are called also partitions and directly assigning hardware to these partitions. So direct assignment is usually key. Configuration is defined at build time. So when I say build time, I don't necessarily mean Yachto, so Yachto could be the place to do it. I mean before you turn on your target, right? So before you even power on your target board, you already know how many partitions you're going to get. You're going to get three, you're going to get four, you're going to get five, but you definitely know in the case of on a server can you manage, can you imagine there are going to be hundreds, you don't even know. You have no clue what's going to be running there because it's all dynamically deployed. Finally, last but not least, there are three requirements that often come either all of them together or at least you're going to get one of them in embedded, static partitioning that certainly data center don't have to meet at all. One of which is real time. They are very, very common in embedded. You have some sort of real time, if not proper real time, at least deterministic arcule latency. Another requirement is short boot times. So, you know, I usually joke that when you turn on the radio on your car, you don't want to wait three minutes for your BIOS to boot, right, on your server, right? It has to be instantaneous, right? And safety is often also a requirement. When I say safety, that means a lot of things, right? It can mean safety certifications of proper ISO 26262 or some time it's just safety in the sense of you are designing it so that if one component fails, that's the only, the failure is limited. That's the only component that's going to fail. It's not going to bring down everything. So, I still call it safety, also maybe it's not a proper safety certification that you're looking at. You're just designing to be able to tolerate failure. And I'm going to give as an example Xandom zero less. So, typically the software that you're going to use for server virtualization is very different from the software that you're going to use for static partitioning, because simply the characteristic are two different, but Xand is an exception. So Xand can be configured for data center or can be configured for static partitioning and depending on how you configure it, you get either one. And the configuration for static partitioning is DOM zero less. We call it DOM zero less because DOM zero is a privilege environment that is the first one to start and then the one where you do privilege operation, you manage VMs, start, stop VMs. Well, with DOM zero less, DOM zero is not needed anymore. You can even stop it entirely from running. So, don't think that with DOM zero less you necessarily don't have a DOM zero, DOM zero become optional. And that's why it's called DOM zero less because it's not necessary, it's not required anymore to get things up and running. So how does it work? You load all the kernel, RAM disk, everything you need from Uboot, and then you let Xand start everything in parallel. So you parallelize the boot. And typically if yellow or purple are Burm metal VMs or tiny RTOSs, yellow and purple will finish the boot way, way before DOM zero or any Linux environment. What it means is you can get your motor controller application on your RTOS up and running in far less than a second. And when I say this, it might be possible to achieve even with Linux or be demo in the past, but here I mean by with a pretty much default configuration. Without having to push any limits, you're not going to get a second of boot time to your motor controller application. So how does it work? The static partitions here, the VMs, yellow, purple, DOM zero are defined at build time. And again, when I say build time, I don't actually mean when you're calling GCC. I mean before you turn on your system. You get fast boot time, real time support with a null scheduler, which basically is a scheduler that does nothing. That's why it's called null scheduler. So you get real time and no scheduling. It's easier to safety certify because if you take away DOM zero, then you only need to safety certify the hypervisor component, which is a lot, a lot smaller. DOM zero is not required. And here is the thing. There is no out of the box communication mechanism. And this is where the interesting part of the talk begins. So why? Because DOM zero less is, after all, a pure static partitioning solution. So what it does is you draw partition on your board. And basically each environment is one of those partition. And that's it. It's statically divided. There is no shared anything, no communication, nothing. So how do we go about adding communication mechanism to DOM zero less? Or what is the best communication mechanism among the many technologies I'm going to talk to you about? So the first things to recognize, and maybe this is the most important message of this talk, is VM to VM communication is actually different from device virtualization. So protocol like VertioNet or VertioBlock or XMPB drivers were meant to share a block device, a network card, the console, graphic devices across multiple VMs. While here, at least in my experience working in embedded, most of the time, we don't want to share anything at all. Because after all, you are dividing out the board on purpose so that each partition get entirely different devices. But what you only need is a way for this partition to exchange data, exchange data for logging so that you can send them up to the cloud so that you can do further processing. But usually it's just about data exchange rather than IO virtualization. Now this is actually a different goal, right? And it can be, I mean, it is optimized differently. So you don't need a full network virtualization solution to just get two partition to exchange data, right? On the other hand, there are cases where you might need IO virtualization. And I can make a very simple example, which is even if you have two or three partitions, which are completely independent, and each device is assigned uniquely, there is often the need to share an SD card among them. Because typically you only have one SD card on your board, you want to set up different partition on your SD card, one for each static partition, one for each VM, and use them one partition or a VM volume in each VM. That requires block virtualization, a block device virtualization. So in other words, if you only need VM to VM communication, the requirement is a lot simpler, and in my experience, that is the most common requirement. IO virtualization is not as needed in embedded, but still if you need it, it's best addressed with an IO virtualization solution rather than to a VM to VM communication channel. The other interesting bit is that in embedded for safety, you often need to be able to define this VM to VM communication channel statically. So having them dynamic actually is counterproductive because when you go and do safety certification, you need to go and explain how one can communicate with two, but three cannot communicate with four, and it's a lot easier to do it if the channel is actually static, right, rather than created dynamically. And finally, no privileged backends, for the same reason, for safety, is often totally impossible to run privileged backends because if your backends were privileged, then you will need to safety certify them as well, which it basically usually is a no-go. There is one more thing that is in the data center when you have compatibility for Linux, Windows, BSDs, you are done. In embedded, this far from the truth, right, there is Zephyr and Friartus, and so many Arthuses, I cannot remember half of them, and Bear Metal, and the Wind River, QNX, and yeah, the Department List of Xaling Salon is more than a dozen companies, I can't remember them all. Okay, so the point is portability, so I think it's a given your driver are not gonna be available for all the systems that you would like, right? So portability of this code is essential so that you can get them up and running yourself on your esoteric Arthus, popular at the moment, and without a huge amount of effort, right? So no privileged backends, how do they look like? You just run the backend in another normal VM by first assigning the hardware device to the normal VM, and then running the proxy to the backend there, and simply from zero, you take it out of the picture just for your reference. Okay, now the background is done, and I'm gonna move into a list of four or five technologies and try to explain to you in the most objective way possible the pros and cons, and I'm telling you straight away, there is no single winner here. I'm not trying to say just use this and you're gonna be good because it's not true, it's simply not true, right? There are too many dimensions, too many configurations, there is no one that wins them all. So depending on your requirement, one might be better than others. I'm gonna start with the grand daddy of the IO virtualization world that is at least in open source the oldest technology of this, which is MPV drivers. They've been around for a very long time and they've been battle-hardened in production for many years from AWS to many other products, so they certainly work well. They are made for, again, IO virtualization. So you might use them for VM to VM communication, but it was never the original goal, the goal was IO virtualization, and they come with support for network, block, console, graphics, 2D graphics, sound, and others. What are the pros of the MPV drivers? Well, among the solution you're gonna see here, the MPV drivers are the fastest with unprivileged backends. Why? Because MPV drivers were designed for unprivileged backends. So you know that all the optimization and the performance that you get in Tom Zero, the number are gonna be identical if you move them to an unprivileged VM. The other pros is available for Linux, VSD, and Windows out of the box. The facts that are available as VSD, net VSD and free VSD means you can port them to free RTOS or whatever environment that you like to port them to without a huge amount of effort. So it's not like you have to start from scratch, you can start from the VSD implementation. Cons, might not be available, well, that's the same thing, right? You might not be available for RTOSes. I think this is generally true for all that we're gonna discuss here, but, and Tom Zero's support is available in the sense that I'm gonna demo it today for the first time at the end of the talk, but it's not up-streamed yet. So it's working progress and it's a brand new development that I'm gonna show you today. How does it work? PV driver support for Tom Zero less. So, you start your VMs in parallel, as usual, with Tom Zero less. Then you use a little tool in Tom Zero once become available to start the communication channel between your DOMUs. So it creates a connection between yellow and purple so that they can start the PV protocols for network block or whatever it is that you need. So that means, in a way, you can think about the boot sequence as a two-stage boot sequence. So your yellow or purple environment, they're gonna still come up incredibly quickly, but without PV network or PV block support. They're gonna come up immediately and you can immediately start your motor controller application. However, PV block or network or whatever it is that you need is gonna come up a little bit later when the DOM Zero tools are available, when purple is available, when everything else is available, the connection will gonna come up and the total time to get your connection up is gonna still be lower than if you use DOM Zero because you parallelize the boot. So you don't have to wait for Linux to boot twice, in a sense, but it's not gonna be as fast as less than a second, right? But you're gonna get to your critical environment, your critical routine still in less than a second. So Unimore developed these and the demo by them is gonna show you today a PV network coming up, being used for communication between two VMs in a DOM Zero less environment. Vertio, so Vertio, it's similar in many ways to XMPV drivers in the sense that it was designed for IO virtualization. Frontend drivers are available in Linux, PSD, Windows. They're probably a little bit more widespread compared to XMPV drivers. Also, there are so many RTOSs and proprietary little OSs in embedded that still it might be the case as there is no Vertio support at all in the environment you're looking for. The backend is typically called VMM, as I mentioned, because in a KVM setup, the backend, because it's privileged, doesn't just do the backend work. It also does virtual machine management tasks, and that's why it's called VMM. Which brings me to the point of uniqueness is that backends need to be privileged. So there is work in progress and I'm gonna talk to you about it in a second about making them on privilege, but as of today, it is still the case, which are safety implications, security implications, and so on. XEM support is there, so if you want to use Vertio with XEM, it's possible. You need to use a couple of non-upstream patches, but it's not gonna be very difficult to set it up. But there is no DOM Zero less support. I'm gonna also go into a little bit more details on why. So the Vertio architecture works a bit like this. So Vertio, from the point of view of your DOM use, of your regular VM, looks a lot more like a regular device, like an emulated device. So when the frontend tries to access the device, it traps into the hypervisor that then sends an IO request, which means an IO emulation request to the backend. So something like, please emulate for me this right to address blah, you know, take care of it. Then the backend receives IO request and proceed with the emulation. But to perform the emulation, what it needs to do is map the page in memory of the frontend. And this is where the privilege requirement comes from because the backend needs to have the ability to map the page of the frontend. And there can be any page of the frontend and therefore, in its privileges. So what is upstreaming then, what's missing? The IO rack infrastructure, which is this component here, right, is upstream in the hypervisor. So you can use it to hook up QMU, KVM tools, really any emulator you can write your own, as long as you send it an IO request and the emulator will take care of it. It's just as simple as that. That is fully upstream already. What's missing is tool support. So the convenience tool that lets you say, please create a VM for me with vertio block, that's missing. So the patches to say, create me a vertio block, that's missing. But it has been sent to the list. So I'm actually have a link in this presentation. I added many links in this PowerPoint presentation so I recommend you to go online, you're gonna find all of the links of all the missing and working progress activities right there and you can fetch a patch and apply to yourself and try it out. So the guys that developed this is IPAM and they have a vertio block based demo running with them. There are two interesting working progress activities. One is by Linaro Project Stratos and that is about making vertio backend not privileges. So how does it work? So if you go back to here, so the idea is to pre-share a bunch of memory between your DOM zero and your purple VM and then have all the request, all the emulation request from purple always be from that shared memory and because it's already shared, you don't need to explicitly map it any longer. Now, this is gonna work. I mean, it's a simple approach. It's gonna work. What is unknown is the performance implication and I should say the performance implication to exist in vertio protocols. Now, if you were to design a new vertio protocol with this in mind, you could make it probably as fast as you would like but an existing vertio protocol like VertioNet has been optimized for years and years by Michael and the other members of the vertio community to be as fast as it can with memory sharing, right? And a fast as it can with memory sharing is not the optimizations that necessarily works in our favor when you have a memory copying interface in there. When I say memory copy is, because the region is pre-shared, what you're gonna end up is copying to and from the shared region so that you always fetch from it. So it's gonna be more copies involved. So I'm not saying this is necessarily gonna be a bad performance. I just, I really mean is unknown and it's interesting because with these protocols and if you ask me at the end what is the performance it's really hard to tell because depending on the configuration, so another key detail for instances, are you gonna use vertioNet to talk outside like Google.com or to talk to your neighbor VM? This has a very different performance profile. So all these key detail have a very big impact and you can get to two X, five X, even 10 X performance difference just depending on the configuration. So it's really hard to tell in advance. The other interesting development is what's gonna happen with DOM0 less. So the dumb way to enable it for DOM0 less would be to just wait until the backend here in DOM0 is up and running and then you can use it but basically that effectively synchronize a boot, serializes a boot and then you have no real advantage. A better way would be to hot plug the virtual device into your guest when is actually ready the backend. So I think we need some sort of hot plug mechanism to implement this properly but this at the moment is just exist on a PowerPoint slide and nowhere else. Okay, so I think you have heard about this so far even without me I mean to tell you about it this is the one I talk to you already are existing solution based on device virtualization based on IO virtualization. What I'm getting into now are two alternatives which are pure VM to VM communication mechanism that if VM to VM communication is really your goal and you don't have any IO virtualization requirement it's likely that these are gonna be a better fit. The first one is Argo. So Argo was actually born for highly secure environment for VM to VM communication highly secure environments. So the way it works is the sender ask the hypervisor please send this buffer to the receiver and the hypervisor data copies the buffer into the receiver buffer. Why is this highly secure? Because there is a pretty large class of attacks that are all based on the sender trying to change the data under the receiver feed while the receiver is reading it. So when you have a share memory the sender plays some memory in the share memory and then he says go go and read it and then the receiver go and reads it and the sender ha ha now is gonna try to change the bits and see what happens, right? And you see with pattern meltdown even very subtle changes as just as the right time can get arbitrarily code execution. So with Argo this doesn't happen because the hypervisor first it freezes the sender buffer and then it copies the data. So good luck changing the bits while the hypervisor has frozen the buffer, right? You cannot change it any longer. Argo, so what are the pros and cons? So pros is made for VM to VM communication as great performance is data is memory copy but has been designed for memory copying. So it's actually pretty fast. What else? It's lightweight. So because it's a simple sender receiver architecture it's a lot more lightweight compared to the other IO virtualization solutions. It still requires two drivers which is also the main cons of using Argo. So it requires two drivers. One for event, to receive and send event that is the event channel driver one of the PV driver blocks components. There are BSD implementation available and it's pretty widespread, Linux, Windows, BSD is available already upstream for event channels. And then there is Argo driver. So the Argo driver what it does it manages the receiver buffer. So it manages the receiver buffer and gives you API to fetch the data from the receiver buffer. They are overall a lot smaller than I think a full IO virtualization solution but still is some code. It's not zero code there that will need to be ported to an RTOS. Let's say if you wanted to. There are BSD implementation available for both. The zero less enablement is really straightforward here because there is no back end. There is no front end. There's just a sender and the receiver. They start basically at the same time. So the enablement was trivial. So I have to thank Alec Kwapis from Donnerwerk and what he did is he wrote a patch for Linux to enable event channel for Linux and that was all it took 10 lines or 20 lines all it took to enable Argo in a zero less environment. So it's a pretty good match. And now I go to the maybe the simplest possible or the two that's the last two are the simplest possible mechanism that you can use. If you only need VM to VM communication maybe the easiest thing you can actually do is to set up a share memory between your VMs and just set up a ring buffer. So that's gonna be as easy as it gets. It's gonna be portable. It's gonna work with any OS. It's gonna be there's not gonna be any complex driver that you need to port to QNX. This is gonna work basically with virtually any environment that you can imagine. The other advantage is if you need to statically define the VM to VM communication channel this is statically defined. This is actually the only one that is statically defined among all of the ones that I mentioned so far because Argo I should say the pros and cons of Argo is this is dynamic. Now you can add restriction on who talks to who with a there is a framework in Xencode XSM that's basically like a Selenux. So you can add restriction but still is fundamentally a dynamic channel while here if you have a requirement on static this is static, right? It doesn't get any more static than that. So the only thing that's not upstream is I added a patch which is on the list which you can use is quite simple and that is a hypercall to inject interrupts in another VM. And this is just for simplicity. You could use other mechanisms, you could use physical interrupts if you can. You could use event channels by why bother when you can just call an hypercall and inject an interrupt. So that makes the enablement of this even easier everywhere. So you just need an interrupt controller driver pretty much, performance. So this can be incredibly fast if designed right. I cannot stress this more if designed right, right? If you design it badly, now it's up to you. It's up to you. We are only giving you just share memory. So if you use a wrong library, if you set up a wrong ring buffer, if you miss an update on your ring buffer pointer this is not gonna go right. It can be slow, right? But as a person that designed one of these protocols I can tell you that if you design it right it can be even the fastest among all of them because you can design it for your own data knowing the data pattern, the sending and receiving pattern on your specific application, none of the other can claim that, right? So of course theoretically this can be the fastest I think. Last is an honorable mention because you cannot use it yet but axonics we are working on a programmable logic way to solve this problem. We are a programmable logic company after all. So we think that PL is obviously the right solution for this problem, right? And so you could today actually deploy in programmable logic two simple network device with a simple switch in the middle, assign one network device here, a network device there and you have your full network infrastructure in programmable logic ready to go. This works but it's not very performant. So what we are working on is a specialized data mover made on purpose for VM to VM communication that can be as fast as possible and easy to use. When I say easy to use I mean, with no complex driver, no device virtualization, no stuff that you need to port to OSS, a very, very simple enablement as small as possible directly in user space. The kernel or I provide outside is gonna be either zero or tiny because the sender and the receiver buffer are gonna be disposable directly in user space and it's gonna be fast and ideally tiny in terms of code size. That's the goal is not ready yet. So maybe next year I hope to be able to show you a demo of this. And of course the main cons is that you need programmable logic. Okay, so I have a summary slide and really the conclusion here is not to tell you that any of these is the best is to tell you that there is no one size fit all solution among them. And we can draw some, I can help you draw some conclusions but there is certainly no winner, right? First of all, it's good that there are so many options. So just a year ago there wouldn't be as many options and if you look at the upstream status wisdom zero less column, three out of four, they are, it's possible to use them today if you go home and try them out. Well, I should say except for the XMPV drivers that I'm gonna demo today but the patches are not public but we aim on making them public by the end of the year. So by the end of the year, you're gonna have three among these four options available for them zero less and in a traditional XAN set up all four are already available. So I think that's pretty good in terms of choice. Now, the other important thing to recognize is what is your requirement? Are you just looking for a VM to VM communication mechanism? Because if you're looking for a VM to VM communication mechanism, you are likely best off with a VM to VM communication solution. And among these, Argo and Playshare Memoried Interups are certainly the two that fit the bill. On the other end, if you need an IO virtualization solution, then it might be possible. I mean, both ways. You can use a VM to VM communication frame or to do IO virtualization. If you roll your own, right? You have to basically build on top of a simple communication channel, a full virtualized IO device. You could do that. But alternatively, you can use a device virtualization solution like XMPV network for communication. Also it wasn't really designed for that. So both, you know, you can do but I think obviously if you have a IO virtualization requirement, you're likely better off with an IO virtualization solution. If you have VM to VM communication requirement, you're likely better off with a VM to VM communication solution. The compatibility here that you see on the compatibility column. So it's pretty wide. I think the one that you have to watch, I think it's a little bit more restrictive is Argo because it's ported at the moment to fewer OSs but at the same time, the code size and the number of drivers is a lot smaller because it's a smaller and a more lightweight solution. So at the end of the day, it might be that it's still easier to enable than the other. Performance, this is a dreaded question. It's really hard to tell as I was mentioning. If deployed right, they can be all very performant. If deployed badly, they can all be not great in terms of numbers. So there is a XMPV protocol that is very fast for VM to VM to communication, which is called PV calls. And that's based on memory copy. It's actually maybe the only one that's based on memory copying. It's a big green. It only supports TCP and IPv4 but that is incredibly fast. Verti-onet is known to be very fast if you can have privileged backends, of course. Instead, if you cannot have privileged backends is a question mark. Argo is good. Argo is definitely good. And I don't have recent numbers in comparison with something like PV calls but probably is close. And that also is based on memory copying. Plain memory and plain share memory, as I told you, it really depends on how you develop it. How many processors are you gonna use to write to that memory? Are you gonna use just one process? Two processes? Four? Eight? How many cores are you gonna employ to do memory copies on the share memory, right? So it's difficult to tell in advance but it can be the fastest or the slowest depending on how you implement it. And in regards to privileged backends, so again, the only one that has privileged backends requirement today and is a work in progress, the solution by Progestratos from Linaro is Verti-onet. All the others don't require privileged backends. Well, Argo and plain share memory don't have backends at all, right? So there is no privileged backends. So yeah, so not one size fit all. So XenPV drivers, if you're looking for IO virtualization and you don't want to have or you cannot have privileged backends, it's probably the best option. Argo, if you just want to do VM to VM communication, dynamic channel are okay. Argo is, give you higher API. So Argo give you a user space Linux API to do send buffer, receive buffer. It's a lot easier and it's already done compared to do your pre-share memory and that way to set up your own library. Share memory, of course, for iOS compatibility that is gonna work everywhere, right? So if you're worried about getting it to run on your strange OS that somebody else chose for you, that's gonna work. And the last, I think Verti-onet is probably the one with the largest amount of virtual devices, classes of virtual devices available such as, I think there is a 3D graphic Verti-onet front and the back end. So if you need 3D graphic virtualization, then maybe it's easiest to start from there that is already available. Demo, so the demo has been developed by Unimor University in Italy by Luca Micho and Marco Solieri. So the demo, it works like this, it's just a traditional DOM zero less setup. In this case, for simplicity is not recommended but it's also the very first demo of this technology but we only have a DOM zero and a DOM U so we don't have a true DOM zero less setup without DOM zero or with DOM zero that the only thing it does is set up something and then quit. So the front and the back end here are gonna set up with the back end in DOM zero. And but it's, after all, it's just a proof of concept so you can easily do this the same way as I showed earlier with three VMs and DOM zero basically just active at the beginning and then maybe even quitting all together if you don't need it. And let me switch to the demo right now. So we are gonna see two VMs started in parallel with DOM zero less. First being loaded from Uboot. This is a Uboot prompt by the way on Axialink's zinc and P board. So it's TFTPing all the required binaries and then you're gonna see a lot of messages on the console. First then and then a lot of messages on the console because both domains are printing out at the same time on the same console. So it's a bit messy. And we're gonna finish out the DOM zero boot. This is the other domain saying hello as you see they are both printing out on the console at the same time. By default you get access to the DOM zero serial. You can see that there are two domains running. This is DOM zero so you have Excel, the privileged command interface where you can create VMs. You can switch to domain one and then this is domain one and you have no Excel. There is no privileged utilities in there. It's just your regular and privileged environment. It's also Linux. You do Excel list and nowhere, you know, it's not there, no wonder. We switch back to DOM zero. It's control triple A by the way. That's the key, the sequence of keys to switch. And on DOM zero we initialize PVNet. Now, obviously in production you're not gonna do this by hand, right? You're gonna do this automatically. This is done by hand to show you that it's a separate additional step and this is a demo, right? To show you what's happening. Now normally instead of running by hand as script you would just run it from your init system automatically as soon as possible. Now the network now is available in both your DOM zero and DOM U as it has been initialized. So there is a bridge that is required in order to be able to connect to the outside network and has also been created. As you can see XNBR zero is present among the list of bridges with which to domain one. And now we can just use SSH to get into DOM zero, right? Or you could even ping Google or do whatever you like because the PVNetwork connection is available. And you see there is a S zero interface and that's from PVNetwork being hotplugged into your DOM zero less DOM U. We have an IP address and we'll SSH into DOM zero just as a test that to showcase a possible connection between DOM U and DOM zero, so. And that's it. We are logging into DOM zero and now we are in DOM zero. That's the end and XN list works and we have the privilege interface. Okay, I think this is the end. So do you guys have any questions? We still have five minutes, so please ask. So in a normal environment, in a regular setup, let me go back to at the beginning, right, in a regular setup, what happens is you would then start XN, XN start DOM zero, then yellow and purple are not there yet, not existing. And then in DOM zero, you do Excel create and a new VM is created with a network available straight off the bat. But that's because you are from DOM zero, you have all the tools, the backend is already up and running, so of course it's gonna work. While now we are a parallelized boot, that means the backend is not necessarily there. You don't know when the backend is gonna be there. They're starting all in parallel. Now, thankfully, by happy coincidence, the MPV driver were actually designed with a rendezvous protocol. So they can start in parallel, they can find each other, but still you need somebody to create the initial connection, like the equivalent of advertising the device on a PCI bus to say, hey, there is something there. So in order to create that connection, we need DOM zero or something like DOM zero, you could probably do it from maybe Xen directly or you could do it from DOM zero and then kill DOM zero because it's really not necessarily any longer or even maybe from the backend. But yeah, so what we are doing is we are starting yellow and purple in parallel with DOM zero and when DOM zero is up and running, create the connection and then the connection is there and DOM zero is no more in the picture. Now, for the demo, DOM zero is in the picture just because we placed the backend in there for convenience, right? But it doesn't have to be there. So the backend in this case is here, should not be here, could be in another VM. It's definitely not an architectural requirement. We have a proof of concept. No, we are more than, it's more than an idea. We have a proof of concept, we have even numbers, okay? It's just that the interface is not as good as we would like yet. So the idea is there is a component in Programmable Logic with multiple interfaces, multiple interfaces to send and multiple interfaces to receive and then you give a sender interface to one VM and the receiver interface to the other VM. Then the sender VM uses this physical, this hardware, Q to end Q request saying copy this stuff and then the receiver interface can be used to say extract this data and then the data mover does a copy and is as fast as you can imagine because it's done straight in hardware, right? And the reason why Ireland showed you more data here is that the interfaces are still working progress. So at the moment, we still require quite a bit of code changing then, quite a bit of code change Linux. There are still some active weights here and there so it's not quite ready for prime time but it will be, I mean I'm pretty sure. So it's not just a crazy idea that maybe somebody somewhere one day will implement. So probably even just in six months we'll have something more concrete, yeah. Any more questions? Okay, in that case, thank you all for attending and I hope you enjoyed the talk and have a good rest of the conference. Thank you.