 Well, thank you all for staying until the very last session, right? I don't know if I wear you up with the state for my session, so thank you. My name is Stefano Stabilini, I'm one of the XenonArm maintainers. And I would like to speak to you about XenonArm for embedded platforms. So the first question that I want to ask is, why do we even need Xen on an embedded platform, right? Why do we need an hypervisor? I mean, do we? Right? And to answer this question, I was going to make this example, except that I have attended three days of this conference and I've, or this example, be made four times. How many of you did manage to get through the conference without hearing this example? Three. All right, I was well done. So I'll be quick about it, but basically this is a Cherokee Jeep car that was hacked remotely. Okay. And the way that was done is they, they connected to the infotainment system based on UConnect. They broke into UConnect. From UConnect, they managed to revise the firmware on a nearby chip and from there they could send commands over the canvas, basically taking over the vehicle, so brakes, steering, anything. Okay. Now, the point I want to make, though, is that this happened because there were components of different level of criticality, like the Brake system or even just the rear view cameras, not properly separated from components of a very different level of criticality, like the infotainment system. So there are, there were a presentation today about how to make sure this connection over the internet to your infotainment system on all the parts of the car can be secure. Now, my point is, if somebody makes a mistake and they do manage to break into your infotainment, your internet-connected infotainment system, you shouldn't worry because it should be completely isolated from everything else. Your car should be entirely safe, still, at least the Brake system, right? And this is where Xen helps you. Xen helps you componentize your system. Like componentize is just a buzzword. But what I mean is, you design your functions separately and you deploy them separately in different VMs so that they are completely isolated from one another. Now, the second example I was going to make, and that's where I did it before them, is the drone example that the jailhouse people actually made as well. So this actually reinforces the message. So this is a very good use case for an hypervisor. This is a demo, I'm quoting a demo made by Galois, I think, actually based here in Portland. In 2014, so it's by Nomi Niu, but it proves the point, what they did is they took a pilot quadcopter and they used Xen on arm to run free RTOS for the autopilot system that they were using as MacPilot, which is a research project, I think, for doing flying out autopilot for flying drones and Linux alongside and Linux for things like getting the camera input, compressing it and streaming it to the user. Now, why would you want to do this? Because this is exactly another one of those examples. You have a very critical system component that is autopilot software and the last critical component that is a streaming, compressing camera component. So one of them is mostly surely going to be some kind of RTOS variation, could be even Linux, in fact. And the other one is most surely a more complex system with Linux, a lot of libraries, compression libraries and so on. But the key message here is, what if by any, for any reason, maybe just system load, your Linux compression, video streaming component doesn't get scheduled or doesn't actually require a run for a couple of seconds. Well, the user is going to see a blip in the camera feed and that's pretty much it. What if your autopilot system doesn't get scheduled for a couple of seconds? The drone is going to eat the tree or a baby or a dog, right? You don't want that. So the point is Xen helps you separate again components that have a different trust profile, a different security profile. And you end up, you know, when you are in your office and you drone the board, those square boxes that are supposed to be separate component, then you go to deploy them and they end up all in the same Linux system. That's not good, right? So Xen helps you actually make sure that those square you design on your draw, on your whiteboard become actually separate components on your system. So this is why Xen, for efficiency and consolidation, because things that previously could only run on a completely separate system now can run on the same system still securely separated, but the real primary goal is isolation and partitioning. So you use it to partition your system securely into different components, into different units of execution. Once you have Xen in place, then you can do a lot of other things like, for example, resilience. You can monitor your VM. If you detect any problem, you can reboot it, stop it, restart it, scaling. So if your loads require it, you can start more VMs or less VMs and portability. So let's say that you're working at the bare metal up level. So once you port your bare metal up to Xen as Xen VM, then you don't have to port it again and again and again every time a new board comes out. You can use Xen to provide that abstraction. So, well, I'll only spend a couple of words about it because you probably all know Xen is a type one hypervisor. That means it runs directly on the hardware. Everything else runs on top of Xen as a virtual machine, including DOM zero. That is the first virtual machine that boots and does all the drivers for your platform because by default Xen assigns all devices to DOM zero. So DOM zero just loads the drivers as if it was running on native. DOM zero also has a PV backend. So this is a way to share a device across multiple guests. So I'm privileged guests here only have PV frontends that are a bit like virtual your frontend in the KVM world. So there are drivers that create a virtual device of one class like a virtual network card. They connect to the backends in DOM zero and this backend can be in kernel space, user space, and they're used to share this resource, this device, at least the network card in this case. Now Xen, it's pretty popular on the cloud. So there are a lot of very large public cloud that are based on Xen including Amazon, AWS, including Tencent, the guy behind WeChat, Alibaba Cloud, Softlayers, a bunch. But Xen is not just used on cloud and servers. In fact, there are several products based on Xen for laptops and desktops. For example, QoS is a pretty famous Linux and Xen based secure environment for laptops and desktops based on Linux. Bromium that is a project to have a secure browser experience on Windows even. And there are also some ARM based projects on Xen and ARM based. I mean, like Donalworks, they provide services on top of Xen and ARM and Xen embedded. So Xen is an open source project. It's GPL v2, same license as the Linux kernel, and it works pretty much the same way. So we have a mailing list. We have a diverse community, different companies contributing it to it. And we are growing both in terms of contributors, in terms of employers of these contributors, and in terms of commits. Now, the stats for 2016 last year are a bit weird because actually we haven't counted all repositories yet. All right. So the point I want to make is the following. So Xen is popular on the cloud and on servers, but does it actually fit the bill for embedded platforms? Right? These are very different environments. There are different requirements. So here I'm going to try to list some of the different requirements that are unique to embedded. And then in the later part of the talk, I'm going to show how Xen meets these requirements. So the first two that are short boot times, a small footprint, well, obviously they're needed on embedded, but actually they don't hurt on servers as well. Like, so everybody likes short boot times and small overhead. But the last, the following are certainly unique to embedded. An example is small code base. So small code base is very important because in many embedded environments, you need a certification in order to be able to deploy your software, like Avionics, for example. This certification, not only are very expensive, but they're also very difficult to acquire. So a small code base is very important to actually be even be able to try the certification. Non-PCI device assignment, so on servers most, basically everything is PCI, right? At least in the X86 world. Unembedded is really important to be able to assign to virtual machine any device, like platform devices, AMBA devices, everything. Driver domains, we'll see later what they are, but what it's basically once you assign a device like an MMC controller, I don't know, a graphic card, anything, a coprocessor, to a VM, you also still need to be able to share it possibly with other virtual machine as well, and this is what is the driver domain. Low and deterministic RQ latency and real time schedulers, obviously a lot of embedded environments have real time requirements, and an hypervisor should be able to meet them, right? And coprocessor virtualization. Many embedded platforms come with accelerators or coprocessors or GPU that need, at the very least, to be assigned to a VM, but also shared with many VMs often. Now, I'm gonna start to show how Xenonarm meets these requirements by speaking about what Xenonarm is. So Xenonarm is a port of the Xenon hypervisor to arm and is upstream, is like arm support in Linux. However, Xenonarm is not exactly the same as Xenon X86. So when we started the port, back in late 2011, we already had several years of experience with Xenon X86, and well, we liked the project, but we recognized that actually Xenon X86 has several limitations. So when we started the project, we took the opportunity to start afresh and remove those limitations. So there are a lot of dark corners of Xenon X86, and I'm proud to say that I think Xenonarm has none of them. One example is QMU. So QMU is a great project, but when used with Xen, it's not that great. It's low, it doesn't scale well, and it increases significantly the surface of attack. So there is no QMU with Xenonarm, not at all. You cannot even run QMU for emulation. Even if you want it very hard, you cannot run it with Xenonarm. Another problem with the X86 in my Istanbul upon is that you have two kind of guests. You have PV guests that have some pros, but they don't use hardware virtualization, for example. Windows doesn't run as PV guests. And then you have HVM guests. Windows run as HVM guests. Most Amazon AWS VMs run as HVM guests, but they require QMU, which is not that great. So Xenonarm, I think, has one type of guest only, and I think it's the best of both worlds. Because it doesn't need QMU. It exploits virtualization extension in hardware. It relies on them, in fact. And it relies on para-virtualized drivers for IEO. So there is no emulation, no QMU, no slowness, no increases of surface of attack. So this is what I'm saying. Xenonarm is better than Xenonarm on X86. It's not a one-to-one port. As I said, one of the initial goals was to exploit the hardware as much as possible. And as a matter of fact, the arm virtualization extension are a really great match for the Xen architecture. And in fact, we managed to complete the port with feature-party or close to feature-party with X86, with a very small amount of lines of code. Considering that Xenonarm is such a powerful hyperfiler, and you'll see later the feature it has, the arm 32 N64-bit specific coding, then, all together, only amount to 30,000 lines of code. So this is just a quick picture to show that it's a good match for arm virtualization extensions. So what are these virtualization extensions? There is a Neo2 that is hypervisor mode, EL1 is kernel mode, EL0 is for applications, HVC is a new instruction to switch between kernel mode and hypervisor mode. And then you have virtualization support in the timer, the generic timer, in the GIC that is an interrupt controller, and second stage translation in the MMU. So Xen runs entirely and only in EL2. The guest kernel entirely and only in EL1, and the user space in EL0. HVC is used to make hypercodes. And second stage translation in the MMU is used to give memory or to assign memory to guests. The GIC is used to inject interrupt to guests. And the generic timer, virtualization is used to give timers to guests. So these are really a one-to-one match. OK. So this is the interesting part. So I'm going to try to go over to a list of features we have that I think some of them are unique to XenOnArm, like the first one, and to show how are relevant in an embedded environment. The first one, we can assign any device to a guest. Any. It doesn't have to be PCI. It can be AMDA, it can be embedded. Actually, it can be any MMI or region, any interrupt. In fact, the way we do it is there is an ILMEM parameter and an RQ parameter in the VM config file. You give a memory range, a number of interrupts, and they get assigned. That's it. We have no guest firmware by default. So if you listen to the previous talk, UFI and so on, sometimes providers insist on having UFI as a guest firmware. So by default, XenOnArm can have UFI as a guest firmware. But by default, it does not. By default, it just boots a guest kernel directly. And in fact, usually I've seen Android booting on XenOnArm under three seconds. Now, there are certification efforts ongoing. So certification, as I mentioned, are complex. So you often require a combination of, first of all, not just the open source software, but also the binary, the produced binary, as well as the hardware combined to be certified. So we, as XenProject, as an open source community under the Linux Foundation umbrella, we cannot really pursue certification. But I can tell you there are several organizations that are doing certifications for XenOnArm in different environments. RQ latency. So actually, I just measure this. And not only we have low and deterministic RQ latency, but it's actually much lower than I expected. So the warm-max means the maximum RQ latency if you remove the first couple of interrupts that are used to warm the caches. So if you take away the first two interrupts that are usually have a much higher latency because the caches are still cold, then that's the maximum you get. And our warm-max is under 2,000 nanoseconds. How did we get there? Two ways. The first way is we support EOI in hardware. So one of the features of our virtualization extensions is that it allows a virtual interrupt to EOI actually the correspondent physical interrupt directly. These ways there are no traps in the hypervisor to say, OK, I want to EOI the virtual interrupt. And then from here, you trap in there. And there you do the EOI. Or another way to do it is you can request another special interrupt in which you do the EOI yourself in the hypervisor. Obviously, there are a lot of back and forth additional overhead. So there is nothing like that on Xenonarn because we use the hardware support for that. More interestingly, and far harder to code correctly, we have a feature there we call physical follow interrupt. So let's suppose you have physical CPU 0 on top of which is running virtual CPU 0 of a given guest. Now interrupt 109, so 109, arrives in physical CPU 0 and is injected as the corresponding virtual interrupt 109 into the virtual CPU 0. This is the default, let's suppose. What happens when a guest decided to change the target of that interrupt so that it's going to be received in virtual CPU 1? It can do it for IRQ balancing reason, for example. Or this is not in the picture. But another possibility if you're running a scheduler is that your scheduler moves the CPU 0 from here to here. So what happened in these cases? Not much because we immediately moved the corresponding physical interrupt. So the physical interrupt always follows the virtual interrupt as a shadow. This way, the physical CPU that gets interrupted is always the one that is supposed to be interrupted. So you don't want the case when you get this one if you get interrupted and you have to send an SGI to the other one and then the virtual interrupt gets injected. This is what happened on many other IPervisors and it's very slow, it kills your latency. For real-time environment, it's not really possible to use. Right, send schedulers. So I know that being embedded folks here, I say scheduler and you say, ah, OK. No worries. So the send scheduler architecture is extremely flexible and allow a lot of very different complex configuration, including having no scheduler. So this is just an example I'm making. Let's suppose you have a platform with eight cores. With that, you can partition the system this way. You can take four cores and run the regular default scheduler that is called credit on that there. Then you take two of the remaining cores and you run one of the real-time scheduler we have by default in the Xeniai provider and you select at any time. And the remaining two CPUs, you completely dedicate each of them to a different virtual CPU. This is possible by default with Xeniai provider. So in most cases, if you really care about lower Q latency, obviously what you want is this. So you assign directly, entirely, a physical core to a virtual CPU of your guest. But this is also pretty reasonable. This is a real-time scheduler. It gives you software time guarantees. It can be well-suited for some environments. But alongside, it might still be OK to run the regular scheduler for things that are not really real-time sensitive. You can have your regular VM scheduling for services that are not real-time critical. This is one of the lesser-known and newer features of Xeniai. It's called memory introspection. So obviously, you cannot do that in a system with real-time constraints because it introduces latencies. What it does is you can receive notifications for events happening in your VM in a monitor process in DOM0. So suddenly, in your DOM0 process, you get all a bunch of notifications, say, oh, this VM executed this strange instruction. This VM accesses this memory range. This VM tried to execute this code over there. This VM changes page table. They are very fine-grained. And from DOM0, then you can even act. Once you receive this notification, you say, oh, no, no, this is not allowed. And so on. So this is an extremely powerful mechanism that basically allow you to do pretty much anything. So one of the classic use cases is to detect if a VM has been compromised. Because even if you just don't do anything most of the time, just receiving these events allow you to do statistical calculation to see if the behavior of your VM actually changes from average. And if it changes too much, maybe it's malfunctioning. Maybe it has been actually compromised. And then you shut it down. OK. It's worth spending some time speaking about PV protocols. So PV protocols are the main way to share a device across multiple VMs. There are a number of them already existing, and upstream in Linux and FreeBSD. The ones that are already existing and upstream are the following, a net for networking, block, console, keyboard, mouse, frame buffer. I think these are all pretty self-explanatory. XNGT. XNGT is by Intel, and it is not upstream, but is open source and available. And this to share an Intel graphic card among multiple VMs. And these also share 3D capabilities and OpenCL kind of possibilities across multiple guests. But this is actually one area of the project that is the most active at the moment. There are a lot of new PV protocols in-flight as we speak. And these are some of them. 9PFS is a PV5 system protocol. PVCos is a strange protocol that allow you to implement networking in your VMs using a different model. Now, networking is usually done by creating a virtual network card in your guest. This does not do that. What it does is it virtualizes the socket positive calls directly so that a socket or a receive message, a send message, and a set source function get sent to the host and implemented there. The host is done zero and implemented there. It's actually very fast. Multi-touch, more interesting for embedded scenarios, is to send multi-touch. It's an extension for the mouse and keyboard protocol and allow multi-touch event like Android style to be sent back and forward. Sound is to share the sound. And display is a much more advanced version of the frame buffer protocol. So this display allow you to create a different graphic object and share them. DRM style, if you are aware of the DRM framework in Linux. And is a good step forward toward a generic mechanism to be able to share a GPU across multiple guests. Yeah, absolutely. So, no, it's single host. It's for between VMs. So 90FS is usually like a pretty old actually specification to do a sort of networking file system that is more meant to be on a single host rather than a multi-host. Yes, you can do that. Yeah, you can do that. So basically what it allows you to do is to set up somewhere a file system root and saying, I'm gonna share in a VM like DOM zero and say, this is shared with VM one, two, and three. And they access this file system, not at the block level, but just as a file system level. Does it answer your question? And of course you can have multiple because you see later with the driver domains, but there is no reason to always have the backend in DOM zero. You can have backend in other domains as well. So you can have, actually you can do the other way around like having the backend for the file system in one of the DOM use and the front of the DOM zero if you like. But I want to spend, given that many of you probably, I mean, if you'll end up using Xenonarm for an embedded use case, you probably are gonna have some kind of weird device or accelerator that you wanna share across multiple VMs. And maybe you need an ad hoc protocol. Maybe not up streamable. Maybe not even open source. I don't know. How do you do that? Well, all PV protocols are based on the same building blocks, which are hypercos to share memory, to set up a shared ring. Usually it's used to set up a ring. You can share memory and do something else, but you have hypercos and function to immediately and automatically set up a ring with a shared page with a ring on it. And notifications. So you have hypercos to say, I want to send a software notification to this other VM and the other VM and I have hypercos to send that notification back and forth. So obviously when you have a ring buffer and you have a way to send notification, basically you have everything you need, right? So you can do pretty much anything you like. And in fact, all the protocols are based upon these constructs. Additionally, this is important, many of these implementation are not even GPL, they're BSD. So you can take the code and integrate it or use it as a base for any project of any license almost that you have. Okay, now this is the interesting part. This is a null slide because we always had driver domains. Nobody really used them on servers, but unembedded, they're really, really important. So the idea is, if you remember the first slide, all the devices were assigned to Domzero and all the backends were here, but there is actually no reason to do that because Xen can simply be told to assign, for example, the network card to an order of VM. We'll call it network driver domain. So this VM has the card assigned to it. So it runs the driver, the network stack, the network backend, and all the Domuse, these and all the others will connect to this backend here. So why would you wanna do that? Well, let's suppose that instead of a NIC, you have a wireless card, and your wireless card has a driver which is a bit flaky, right? We know that it happened in the past in Linux. So if your wireless driver, for any reason, panics or crushes, and you do that, the only thing that brings down is the natural driver domain because it's fully isolated from everything else. That means, okay, network connectivity goes down, obviously, but everything else stays up. And this is really the key to your partitioning, security, and isolation that I said at the beginning because if the things that are unsecure or you're not really counting on the stability of the code, you move them out on unprivileged guests, the damage they can do is much more limited. And I'm gonna make more concrete examples. So this is an architecture proposed by GlobalLogic for automotive. Actually, we are going back to 2014. The idea is you have two VMs, so you're using Xenonarm to start exactly and only two guests. One is Domzero, and the other one is your infotainment system based on Android. The idea is all the critical systems run in Domzero and only the infotainment low privilege system, low criticality system are in the infotainment VM. So the Android VM was assumed to be flaky and they had a monitoring process in Domzero to check if it was running well or not and if it wasn't, it was just being rebooted. This was the case, they were timing it and making sure that the reboot was happening in less than three seconds. To do this, write the assigned all devices to Domzero except for the graphics, the audio maybe the input to the Android VM. This is an early architecture, things have advanced. So I think one or two years later, GlobalLogic proposed these architectures. I think it's called Nautilus. So I'm seeing these as examples, obviously I'm not an expert on GlobalLogic solutions to be honest with you, but the idea here is you have actually three VMs. One is Domzero with the critical services. One is your infotainment system with the graphic card assigned to it. And the other one is a QNX neutrino VM. Because QNX in fact runs on Xenonarm and does front ends for the PD protocols. But things are never that easy in real life because this is how things got in 2016. There is a brand new video on YouTube by IPaM and it's only two minutes long, I recommend you to view it. There is a link in the slides, I'll show you later. So sorry you can't quite make it out, but I'm gonna explain it anyway. You have four VMs. One is Domzero. As usual, the most critical services and devices are here. There is, this is a driver D. Basically it is in charge of rendering. So it has a graphic card, the audio and the input. It uses Wayland for rendering and does appear parallelized backend for audio, graphics and input. And then you have two other VMs. One is your classic Android based infotainment system. This time is not having access to the GPU directly but is accessing the GPUs through a parallelized GPU protocol. Same for audio and input. And then you have this new Fusion VM which is also based on Linux and downloads software components from the cloud, run them here as containers for providing additional functionalities that can be easily upgraded or add-ons, like for example, I think in the YouTube video, they do telemetry monitoring. So they check the steering, the position of the car and so on. The idea of course is that if this crashes, who cares? If this crashes, I'll just get rebooted, right? It doesn't matter, as long as these two stays online and these two really, I mean, they are the most trusted components and they don't have internet access and so on. So yeah, okay, so if you check the slide and go here, I'm not gonna show it here but you can have a better understanding of it. Now I want to step away from the automotive world for a second because I don't want to give the impression that it's only useful in cars and drones. So this is an example of a use case for excelling zinc and PSOC. So this is a board that has a bunch of programmable logic. It has four A53 cores. It has two 32-bit real-time cores and a load of devices such as a NIC, an MMC controller, such GPU and others. So an example use case here is the following. Your client has four bare metal applications. Non-Linux, just straight bare metal apps, they use FPGA directly and you want to run them alongside on the same board without interference. How do you do that? This is a simple partitioning case. So use then, you create four VMs. Each VM has one VCPU. You completely dedicate a physical CPU to each virtual CPU so there is no scheduler over it and then you assign a bunch of FPGA space to each of these bare metal apps. And that's it. And the overhead is minimal because, well, the only thing it does basically at this point is receiving interrupts and injecting them and we have seen the overhead is 2,000 nanoseconds. Tom Zero is used to initialize the platform at boot and basically there. I've seen it being paused, in fact, because it doesn't do anything else anymore. All right, so this is the end of the interesting slides, but before I finish the talk, I want to spend a few words on the security process because it's easy, I mean, when you're choosing a new open source software project to integrate in your product, it's easy to overlook the security implication because different open source project make very different guarantees over security. And I'm proud to say that actually XM project makes one of the strongest guarantees is the open source industry. So the process is very transparent, it's all online, so many open source projects don't even have a specified process. So you don't know what's actually going on when somebody reports a security issue. Secondly, some open source projects don't even publish properly all the security advisories. So one of the reasons XM is often in the news in a bad way because there is a new security issue is because we publish every single advisory. So even for kids that are only theoretical, theoretical vulnerabilities. So this is why it makes the news, but well, the other reason is we make, we publish every other advisories also for QM, which has a lot of advisories, but as you know, Xenonarm doesn't use QM, so it doesn't affect you. So Xen has a lot of modes of execution, so there are PVH, QM on X86, Xenonarm, a lot of different stuff. And Xen, the Xen project published advisories for all these possible use cases, so overall they aggregate. The other nice thing about the security project, property of the Xen project is that we support responsible disclosure. That means you can request access to a pre-disclosure list and will notify you in advance before a vulnerability is made public so that you can prepare for the fix and reduce exposure time even further. And finally, we guarantee that we support all the stable trees for security fixes for three years, and that includes Xenonarm. Okay, so we make two releases a year. This is actually a new, we introduced it in 2016. We make a release in December and one in June. The last one was for eight and the next one will be for nine in June. So there are a few features that I'm interested in seeing in Xenonarm just to give you an example in the direction we are taking for Embedded. One of the things we wanna do is we want to create the guest directly from Xen. I'll explain. So if you see this architecture, right, so there is no need to fully boot them zero, then start the tools, use the tools and create these other VMs. I mean, come on, we are only creating two VMs. I mean, this is a bit overkill, right? So what we wanna do is to be able to create two VMs directly from device tree. So you add in, on device tree's information, you pass it to Xen and send from the start, it boots two VMs, absolutely. Well, if you, exactly, with device assignment, with no problems. Obviously, if you're using backends and the backends are in them zero, well, you have to wait. Dynamic memory map, although this is interesting for this use case. So when you are porting a bare metal app to Xen, so theoretically, you will have to support device tree because that's the way Xen tells you where the memory is. In practice, you probably are coded. But we want to support a way to specify exactly where the memory ranges are gonna be so that the porting effort of your bare metal app to Xen as a VM is even lower. So basically, you have to do nothing to port it to Xen. And another thing we want to do is we want to simplify setting up PV protocol. So, as I said, we have already all the tools to set up the shared ring as an hypercode, to set up the seven channels for communication as an other hypercode. But maybe if you're working as a very low level, as bare metal app, you don't even want to do hypercodes. So maybe you don't want to know that even you're running Xen. You don't want to pull any of this infrastructure at all. So what we want to do is to have a config file option to say, just share this page between these two VMs. And that's it. So Xen does it for you at creation time. And then the bare metal apps can talk to each other without any Xen knowledge at all. Yes? Okay, so that's it. I want to spend a couple of minutes to tell you there are a lot of resources online. So, well, we have information on how to port Xen to a new SoC. It's basically, the effort is really minimal. Usually only requires writing a new UR driver for Xen in case Xen doesn't have one for the board. How to port an operating system or a bare metal app to Xen. Our overview of Xen as an architecture. Xen on a wiki page with a lot of resources. Presentation to tell you how to do device pass through in details and all the possible problem you can run into. We have actually open embedded support in Xen. So there is a recipe online upstream in the meta virtualization package and is actively maintained. In fact, there is an open XT, there is an open source group that is based on, it's X86 today, but it's based on Xen and open embedded and maintains this open embedded build for Xen. And we have bi-weekly on community calls. So if you have any questions or you would like to see a feature or anything, you are free to join the call and ask them. Well, and one last word is, we are an open source project, well like Linux, so we need user engagement. So please engage on the mailing list, ask a question, say it when we do a good job, say it when you do a bad job even more importantly, so we know what to fix. Our ask question on IRC, please engage. Okay, that's it. Do we have any questions? Questions? Okay. So the only requirement is virtualization extension, obviously, right? We require a Geek P2 or Geek P3. So this is the reason why we cannot run on the Raspberry Pi because the Raspberry Pi has a weird interrupt controller which is not Geek compatible. So yeah, aside from the Raspberry Pi and everything that has a weird interrupt controller, we run on any basically ARM platform. So I recommend you, so if you are looking for a board, the QB track is a pretty reasonable board to work on. Yes, in kernel space, yes. Yes, there can be both in user space and kernel space, but I'll make an example. So the frontends are likely, you probably want the frontend in kernel space because when your application, let's suppose that you want to use PV calls. Yeah, for an RTOS, yes, it's perfectly fine. So you can have both frontend and backend in user space, for example, and both in non-dom zero. Everything is upstream, we have libraries to do the in user space connection. So many of the user space backends as a reference is useful, are in QMU. So I know that I mentioned QMU before. I don't want to confuse you, but QMU has various modes of execution. Usually, most people use QMU to emulate hardware. With zero-narm, it's only used to provide a few backends because somebody, for strange historical reason, the backends were upstream in QMU. But still, you cannot use QMU to do the backends and emulation on ARM, you only get the backends. But they're useful, they're GPLV too, though, but they're useful as a reference if you want to see how they can be done in user space. Yeah, we support it. So whatever you want. And we support, for example, let's suppose you are assigning a device to a DOMU that requires a snippet of device tree. We support adding the snippet directly from the VN config file. You copy and paste your device tree node and we assign it to the guest automatically. Yes, and not only depends on the GPU preemption, depends on the GPU. So, yeah, so this is a difficult topic. So GPU are difficult to virtualize. It's a fact. And I've seen several approaches to it, but the direction, so I'm not working directly, but the direction, if I have to say, the industry's taking, is to have GPU-specific drivers so that you can exploit the GPU feature at the maximum levels and get the latency requirements that you have. So with generic OpenGL drivers, so far nobody managed to have the performance level lower than low latency requirements that they need, right? So they usually, I think in this case, they had something like CEN-GT, basically. The CEN-GT model, that's the Intel model. So you have front and then back end that are specifically to your graphic card. And then they have GPU-specific knowledge to be able to share it in the best possible way. And this is how it was done here. I think actually the global logic guys, initially, tried to do generically with OpenGL and then they went back into doing it on a per-GPU basis. But hopefully, I mean, many of the concepts are gonna be portable. Like, obviously you're gonna have to have GPU-specific drivers, but probably you can reuse a lot of code. Other questions? So I'm not familiar with certification for automotive, but I can tell you a bit more about the certification for Avionics and for the DoD. So they're really, really difficult and very expensive and is like a multi-year effort, okay? That said, so usually, as there was actually a talk about certification on Xenonarm a couple of years back, so if you Google it, you'll find it. And it was a guy working in Avionics and was going to certify Xen for actual passenger planes. And before choosing Xenonarm for certifications, he did a bunch of research and he came up with two requirements to actually being able to succeed in certifying a software code base. And this is according to him, of course, right? One is it has to be small, okay? And then Xen meets their requirement. And finally enough, the other takeaway was that the original authors need to be still around. But usually when the original authors of a project are not around anymore, it becomes much harder to change something or to find exactly the knowledge that you need to certify the project. And that's also the case for Xenonarm. It wouldn't be the case for Xenon X86, for example. The original authors are doing something else now. So according to him, at least, it was certifiable for planes. But more than that, I cannot say. So the way it's done is you need the whole thing. So you need is a hand-to-hand certification process, right? So you need the source first. You need to compile the source and you need the whole binary set. And then sometimes you also need the hardware to show that some requirements end-to-end are met, like in the real-time case. So it's not simple. Yes. All right, so it's late and I'll let you finally go home. And I hope you enjoy this presentation.