 Thank you for coming. Can you hear me clearly up there? Looks like it so I'm gonna talk a little bit about Virtualization and in in embedded There'll be a bit about well quite a bit about it said as well I'm gonna look at other open source hypervisors as well But you know first actually we have to ask ourselves, why do you want to embed virtualized? virtual systems and to a large degree it's actually Quite similar to you know server virtualization. It's on to one degree about consolidation Which primarily is about cost size weight and interestingly in some cases also about heat and so on as well in In embedded context too. I think there's a few examples for example like in some car environments You know like when you have the little camera which helps you park and stuff like that That can come think of some problems in some areas with heat because you have a you know few CPUs on very in very close Proximity, which has me somewhere in a car. So there's additional Aspects to this whole thing as well another interesting thing which is particularly interesting in the embedded context. It's also about To some degree reducing the development cost. I am being able to port legacy applications or something You know which have written to a different environment without having to do all the base porting work because you're working towards in a virtual environment It's also around security and safety Can be big concerns in in that space So one thing which you typically want to do in an embedded environment is to support This is called mixed critically criticality compositions, which means you want to be able to run applications in different containers if you want so with different safety security and real-time requirements and You know one of the key things in some markets is you have to have a safety certified hypervisor if you use a hypervisor technology for Isolation and then you have a whole set of additional embedded requirements, you know such as low interrupt latency Low as you're scheduling overhead Particularly if you do something with rich I I owe Which is quite often the case for? Say automotive applications, you need to have a lot of Drivers for specialized IO, you know touch screens all these kind of things which you normally have a phone and to support All this you need to have a fairly flexible architecture To build you know Vibrary varying systems if you want so So let's just really do a quick reminder of hypervisor architectures. I'm going to focus on arm only Because that's sort of the main player this space So if you look at arm privates layer at levels you have el zero which is user mode el one kernel mode And then you have that el two space which is Where a hypervisor would run in and an underneath you would have something like trust sound as well And there's some interactions, you know between trust sounds and hypervisors to run that's going to get a little bit complicated So I'm not going to go into this. So how does a hypervisor How would they typically map into this model? So your classical type one would run in el two You would have your drivers run as part of the device the hypervisor Then you have all your virtual machines sitting on top of it, and you have this type two kind of model where you have Fundamentally the host kernel and the hypervisor kind of run Together, you know, like if you look at KVM the hypervisors kind of KVM. I own kernel model module Then you do have this whole user space You know box which comes with the host and then you have your you know VMs there And what's interesting if you look at it from a control plane perspective in a model on the right He kind of can just use the user space of the host for control Where as you know typically what you will do in a model on the left is you either have some kind of interface We externally control the the whole system where you have some bit of code as a module in there or you use One dedicated VM to do that So let's just look at a few examples of Open of hypervisors today in that space So this one which is kind of really interesting. It's called L4E. It kind of comes from a White heritage of hypervisors as a whole family called the L4 family Kind of originally were developed in the whole for the whole mobile space This one keeps on coming up in discussions about automotive It's written specifically for mixed criticality compositions of typical microkernel design in the kernel space really that just to address brace implementation threats and IPC kind of functionality everything else is in user space It's got a solid feature set I think it is a little bit weak on hardware support. It's also kind of arguably arguable Particularly at this audience at Foster when it's really open source, you know, they It is GPL, but it's gonna do a license and you have to sign, you know Do copyright assignment and all those kind of stuff if you want to contribute Also, there seems to be no real community around it But you know this could go somewhere Another one which occasionally comes up here is jailhouse It's relatively lightweight less than 10,000 lines of lines of code It's a petitioner petitioning hypervisor has come from Siemens Community-wise Relatively small I think 80% of all the code seems to come from one developer They use Linux to kind of bootstrap the system and then after the bootstrap It's kind of you know still partly used for control plane The focus there is really on petitioning but not so much on virtualization So they don't have things like schedule, you know IO emulator and stuff like that It seems to have a really good basic feature set But there's some areas which are, you know still under development. I think particularly because we're talking about Arm virtualization here, you know, that's 60 the arm 64 bit support is fairly new There's also really interesting talk from the AGR, which was given Earlier this year, which you can get to the previous slide has a has a similar more in-depth Discussion here and then we have Xen Fairly, you know general purpose hypervisor most people know it from from cloud computing and basic server virtualization But as we will show it's actually used in a lot of other market segments today, too Again, you know, we have Linux and NetBSD to bootstrap You know Got quite an expensive feature set. It's highly customizable and we have a fairly strong and diverse open source community around it And the following picture kind of gives you a sense of this Well, this just shows, you know, like different vendors having good products and services around it If you look at the top right that's sort of traditional server virt Then we have cloud computing then a top left bottom that's kind of security focused products And we kind of go a little bit into this because security is a big aspect of the whole Embedded story as well, and we do have quite a few embedded product products as well and actually I I Started working with skin around 2011 and really what happened is that whole security that in second started to pick up around that time And it's grown since there's been a push which doubled the number of players pretty much in the last two or three years Which was around that time when all this embedded stuff just started happening. We didn't plan for it It just happened, you know, basically because there seems to be a good match for it So Let's draw this back on we talked about hypervisor architectures before the foolish back to what is actually said because you know It's kind of the type one, but it's kind of not a traditional architecture. So One of the differences between your traditional type one and Cernus that the device drivers don't run in EL to and not in the hypervisor. We basically reuse the drivers from Dom zero and then basically because we have to specialized, you know, Dom zero virtual machine That's then also our control plane There's strong isolations of example drivers are running in the other one as I said in the alphory They're running in user space. So that's obviously a little better. And we have these Grand tables which kind of provide additional isolation between different VMs From a control panel in perspective, you know, insert on on on our server system You know, the sysad member basically, you know used on zero or some graphical tool You know, which connects to it to control your system in embedded. You need a lot less It's really, you know, primarily use the control plane for a config and setup and then basically primarily for system health monitoring, you know, applying possibly software updates and Those kind of things so you need them you need much less functionality there. So, you know, having a big distro running in Dom zero isn't really Necessary so the people in the space tend to do now today is just use a use a basic Linux code There's a root file system or something like that But this could be made smaller and there's some some work going on in this area and I showed you that control plane So let's talk about so much. So we talked about some of the specific requirements in embedded and and one of them was around PV drivers, which is IO and so on and so forth So there's been quite a lot of momentum in our community for a while So we have all the standard stuff around this and so on has been around forever There's also some GPU sharing is all traditionally a preview protocol in the way how it kind of works. It's Society different model, but it has the same kind of effect There's also work going on and this is being driven by some automotive vendors around General co-processor sharing so that taking some of the ideas we used for GPU sharing and Extending them to FPGA is and you know anything which might kind of Look like a pro co-processor because it's the same kind of principle Some prototypes around there right now that made some design changes recently and will probably have a Fully baked framework come come this year Then there's been quite a few new ones It's a 9p 9pfs PV cores that's sort of forwards a posix forward a Call for reading where you have your you know your posix API in one VM And then you basically just have a shim and it roots course through it's actually quite interesting There's been benchmarks as well in this things like multi-touch sound display DRM and What's really interesting and I find there's a lot so a lot of the embedded Companies they just tend to embedded companies that just take stuff to hack it and then I push it out And so we found a lot of them, you know a driver of writing their own drivers And a couple of them I managed to get them to upstream it and standardize Which is kind of good, but it's Relatively easy to drive to write those drivers particularly if you don't care about Standardizing the API which you can do if you you know do this for your for your own product so Let's look at some security properties Of Xen security is kind of important So Apart from the typical hypervisor functionality you have functionality which which enables you to be further system partitioning so you know we already touched on this you can Basically Disaggregate your system in Xen so you can write have run device drivers, you know in a specific specific VM No network drivers are typically a main root of attack for a system And you can do this for other drivers as well you can put other system components into separate VMs as well And then kind of plummet all up To make a distributed system. There's also the capability of controlling The capabilities of a specific VM With the equivalent mr. Xen of equivalent of se Linux and That kind of enables you to build a multi-layered security approach and that's actually being used in some of those products Which Some of the security products which started appearing a few years ago There's other security features like trusted execution environment and stuff like that virtual machine and introspection Altupm life-patching all this kind of stuff, but that it's not quite clear whether that's interesting for that For that how it bag it's embedded segment I mean I could imagine that Virtual machine introspection could be something interesting for health monitoring of a system, but it's it's not quite clear how this would fit in as We're talking about security. I think we do also have to talk about meltdown inspector quite quickly I don't know whether we have lots of talks about this yet today, but Obviously meltdown is the most dramatic one From a Xen perspective it can't really be exploited in fully virtualized VMs, so fundamentally on arm we don't use Software virtualization which just use hardware extensions. So from that viewpoint we're actually fine and You know on x86 with HVM and PBH that's true as well We do have mitigations, but you know it I think for that embedded and automotive use case We don't have to deal with the performance overhead. Some of those fixes will give us Spectre is a lot more interesting We started going through the code base at this stage manually Hopefully at some point tools like covarity and stuff will help us find some of the kind of gadgets and code patterns which Which could be used to exploit those vulnerabilities? But it's a lot harder to kind of go down this route an Xen based system because you know To to exploit spectrum you need something called a gadget. So in a Linux kernel there's this ebpf engine Which was you know by Google project zero used to on KVM to demonstrate how you get information out we Have gone through the code and I mean not line by line But through the architecture and we don't think there's anything similar to that in the code base But you know at the end of the day, we won't really be sure unless there's really tools which allow us to verify that So hopefully that should be fine Let's look at one of the other techniques. So this is whole sandboxing this aggregation thing where you can run Drivers in different VMS, so how does this work? Well We have a regular, you know your dom zero on the right a regular application running in a VM and then you know We have to sync with a storage domain, which would have a disk drive in it or network domain Which has a network driver in it and then basically Dom zero kind of connects to top topology and You know your application will talk to the back end which will talk at top to the front end Which would talk to the back end in a storage domain Which will talk to the real disk driver and I would then talk to the controller and the same can be done for network and Fundamentally the same can be done for any IO pretty much Then there's this thing called X is M or flask, which is like a sensor in the same units So here you have your VM You can't write the policy Which controls, you know what you can do with various interfaces You would typically you know you find some classes of VMs and then assign those Classes to specific workloads in the ends when they come up that gives extra protection because then you can only do certain things certain operations from within this VM everything else gets blocked it's Disabled by default. So you have you do have to enable it by k-config And It's very similar to the Linux security modules. It uses the same policy syntax. You can use all the same tools It's just different new objects. We should used on top of it and fundamentally The guys who you do that use XSM together with SELinux in their systems If you want to know more, there's a little bit of documentation. The slides will be online via the foster inside and also, you know on my slideshare channel So that's some of the term in terminology around security. So I kind of want to look a little bit into How this actually works So we have a number of security based products in sense So there's Qubes OS Qubes OS to have a boost here at foster and go find them if you're interested There's open XT which is in some sense similar to Qubes OS is just an open source project Which allows you to build Qubes OS type of applications and then there's a product called crucible defense from a company Cut called star lab and a that they're targeting military type applications. It's kind of certified product and so on and so forth So I wanted to briefly show how some of the stuff works in practice. So Qubes OS Edward Snowden likes it Which you know, it's probably a good thing and How does this work? So you have your typical fedora UI here on the top right You see kind of all the VMs are running and then on the left You kind of it's like an application starter like on Windows where you can basically define which application runs in which VM Under the behind the hood you have Xen the dom zero kind of do does all this UI stuff There's a network domain. They have a firewall VM which controls all the firewall policies There's something called a USB service domain when you put a USB device in that gets Started in a VM and then you have you know those defined different VMs where you run different applications in and if for example you then start, you know An application then, you know like the network traffic would be routed to those VMs And then your via traffic will be routed through that so you kind of get this extra isolation Which makes it harder for exploits to to infect your entire system because you have all these little sandboxes in place And they're going down some interesting routes of having being able to host some of those in clouds It's what well as well. So go talk to them if you're interested in this So embed it an automotive one of the interesting things is Being able to partition the system so Xen we have a lot of different schedulers with different properties and you can assign these schedulers to You know different groups of you know CPUs on the system This shows that we have an average schedule is a hard real-time standard as a soft real-time scheduler You have a regular VM scheduler and then there's the capability to pin Virtual machines to CPUs Without overhead Using the entire management stack. So that kind of doesn't give you a lot You know just give you still this virtualization boundaries and all the tools, but you know It's very attractive for position Here's the same thing again in terms of overviews I'm not gonna go through this in a lot of detail. Have a look at this later on if you're interested or talk to me afterwards Another interesting Property this is around interrupt licensee. So we have this concept of interrupts Where physical always follow in physical and follow virtual interrupts. So when an interrupt gets injected It's always injected on the CPU Which is running a virtual CPU if For whatever reason that is moved by the scheduler Then you know at one point that gets rooted back to the other virtual CPU in the next time round It comes in, you know the normal way and what's interesting is that actually that whole approach really gives you Fairly low Interrupt latency. So this was around 2,000 nanosecond seconds with the nose nose scheduler And that's the sort of maximum Over a specific run and there's a whole blog post about that too. I see one of the things we do need to talk about Particularly if you talk about automotive is safety certification because that's a big difficult issue in open source So there's seven levels to this so one is around code quality But typically you would have to run special static analysis tools which are quite expensive which you know where where You have to comply to the third miss for a coding standards We've been running a project with a with a tools vendor who's been doing this for us for free And we're starting to work through the issues. So that's kind of good first step another thing around safety certification is You have to prepare a lot of artifacts all the ways for the development process from you know requirements all the way How it gets implemented in the code? This is kind of tough in an open source project because it's a lot about reverse engineering We have had two companies who have done this So we know it's doable. So it's at the end of the day is just a costing But then then those guys of course have a commercial project and they pass it on you know to their pasta cost on to their to their customers and You kind of lose some of the sharing stuff around open source Development process is Sony open source development process does not comply with safety standards There is a way to do that, but that means backfitting a lot of stuff. So that's quite expensive That brings us then to the potential about you know How could you do this better in open source? So one idea is that you could have some sort of official You know code line or some you know project supported code line somewhere and that you get the interested parties to kind of do You know maybe create a consortium or some you know some sort of structure to do this together and share some of the cost There's a lot of discussion around this going on for various projects and so on in Leonardo and in the AGL, you know, we're very Linux and so on the biggest thing. So is you need to have somebody to blame if something goes wrong And I mean that's one of the main purposes of certification really So So this is interesting. This is evolving. I don't know whether open source can play a role in there, but we're gonna try So let's just look at some examples of vendors who are using set and embedded as well as an automotive and a quick demo and enclosed so One of the vendors we have Is Turner works and they started out as a small consulting Company in the US They're now maintaining sweeps and distros. They used to have different names, but they've kind of a brand of them now together What does like virtuosity does virtuosity for nxp and for silence? their core virtuosity product is actually Certified safety certified to various different standards. It's quite a long list down there There is a company called star lab. They're also going through that whole certification a Process right now, and then we have a few, you know That means that we actually know this stuff can be certified, right? It's doable at some reasonable cost those companies aren't huge so you can't do it and then there's a number of others Who play that space too? Automotive it gets even more interesting. So there is a company called global logic. They're tier two automotive OEM They have a product based on Xen I Done some contributions over the years Right now they're very focused on bringing some of the stuff to market They have a competitor called EPAM. It's quite similar what they're doing I like EPAM more because they're a lot more actively engaged in the community And they have some interesting features where they look at the sort of cloud automotive interface I'm gonna show you a demo quickly and then I've renown at LG electronics Bosch and a couple of others are playing with this whole stuff, too So demo so those who are not going to be there We're not here. They're gonna see that demo by going through the link I'm just gonna try and play this now and see what happens The internet of transportation is becoming a reality and soon Individual vehicles will become a part of this internet Vehicles need to keep up with the latest safety standards and regulations But combining internet-enabled software with mission-critical functions on the same platform Creates quite a challenge for the industry EPAM's cloud fusion platform is a unique solution that solves this challenge It includes hardware isolation enabled by virtualization rapid services deployment using containers and end-to-end security for full cloud integration with the EPAM cloud fusion platform Traditional in-car software stacks run on independent hardware using a special virtual machine Dedicated to cloud-managed dynamically deployed services the instrument cluster running a custom Linux instance implement some basic vehicle functions in Vehicle infotainment based on HEL IV I Provides user applications and media features the fusion virtual machine is sandboxed from hardware and can interact With the rest of the system using an OEM managed set of restricted API's each cloud service is deployed to the virtual machine in a container Deployed services cannot interfere with other processes in this demo The cloud service collects telematics data from the vehicle using a micro service-based architecture Part of the cloud application is deployed inside of the fusion container Importantly deployment of the app does not require any prior integration with the car OEM or tier one to demonstrate The simplicity of the deployment will enable some new features in the cloud application fuel consumption Engine RPM and gear settings in this scenario only some of the desired features are available from the in-car applications via a web application interface the vehicle part of the application is updated separately and new In-vehicle functions are enabled notice that the rest of the system operation is not interrupted during the service update Even if the update fails or crashes or is otherwise unable to run Operation of the critical system is not affected due to the isolation of the fusion virtual machine after the vehicle service finishes Updating we see that new functions have been enabled in the vehicle fusion is based on open-source software Technologies with fusion new automotive So looking at this diagram that was the diagram we kind of saw earlier But you know what's interesting again is that they're actually using some of these driver Rather than having specific drivers in separate virtual machines. They kind of split them up They have the drivers for one application, you know in a different VM and there's sort of cutting some corners there But it's basically the same concept So Just to summarize You know why is send really interesting in this space As I said earlier, we're not really pushing that happen really starting to push this very hard Because at the end of the day this just happened But you know after Really two years of watching this, you know, it's really something there which is worse building on So, you know, it's quite a flexible platform Used in a lot of different market environments. It's Apparently relatively easy to port to new environments. Otherwise, we wouldn't see an explosion of so many different products It's highly customizable and part of that is also relatively easy to develop new drivers There's quite a lot of security and resilience functionality there Obviously if we ever want automotive to become interesting, we're gonna have to do it as the functional safety aspect And we're kind of looking at ways now how this could potentially be done. It's It's gonna be a hard nut to crack. It's not gonna be quick But you know, everybody a lot of other projects are currently looking at the same thing You know a Linux has to solve this somehow For automotive great Linux is a lot of other projects which have to kind of help solve this and we're starting to work together There's a couple of challenges really which we're looking at so we have a lot of Different IO drivers some of them not open source and we've been working with some of the vendors to standardize on some of the protocols The same is happening around GPU virtualization. We're looking at Maybe a more minimal done done zero, you know, like maybe we can use some sort of our toss for that that would be interesting and Testing has been a surprising challenge for embedded as well so Again, there we've been working for example from working with renaissance an hour to get some of the archer our car gen boards Into a server chassis With some management functions such that we can actually deal with this in the same way as without a hardware and hopefully Some of the hardware specs for that will go open source and then we can do the same for some other similar kind of, you know boards in the similar form factors and That's really it for me today. If you have questions We do have some time do we You just rest me. I could have talked for a little bit longer. No, I'm right. There was a question up there Was it you? Well, I think at the end of the day it comes down to you know, like I'm not telling you Oh, so the question was when when do not use Xen? I Think it ultimately depends on your use case and preferences, right? So if you look at Xen today It's a lot more. It's a lot more complicated in many ways Then for example KBM That's we've been hurting from that a little bit in the traditional server a virtualization space because KBM is a lot easier to use So I think you know really if you look at if you want to have simplicity and Kind of make maybe more ease of use I think maybe there's competitors if you're looking at some more complex stuff where you want to plant things together and you want to You're actually okay with taking the overhead of the extra complexity that is probably for you, right? Okay, so the question was around Intel support and You know particularly maybe in some industrial areas in Industrial and I guess you're you're there's a lot of these How they're called a plug computers kind of this Intel in use so there is There is some momentum there. I have one Two startups who are working on a product in a space and they're probably going to use Intel Architecture first to push into that market. It's probably going to be a container element in it as well It is a lot more challenging from From I think anything which will require safety certification It's a no-go area at least for Xen Because the code base is a lot bigger the arm port is around Depending on how much you use or not between 20 or 30,000 lines of code the x86 stuff is a lot more and Quite often you need to use Q mu as well, right? So that's that's obviously a barrier But I think I think in some market segments It's it's definitely a possibility So the question was the companies who did the safety certification for Well, but who succinctly I did it for actually I think so Durner works, I believe did it both for arm and x86, but the x86 configuration would have to be very cut down and constrained right because otherwise it would be too cost cost-intensive and Particularly in that, you know today we have like PD PD. You have to forget, right? I think it might become you know HVM then you have to deal with the Q mu aspect I don't know, you know how you deal with this you will probably have to Use a Q mu and pretty much take everything out which Xen does need It's actually a lot. There's only very little Xen actually needs, but then you have to maintain that fork, right? Maybe PVH is actually when this comes more widely use that sort of probably the interesting route to take Yeah Yeah, so power management is a really interesting challenge in this space right now So this is something out. So the question was if you have drivers in different VMs There are you know properties like you know power management which could cause problems and you need to kind of address this so In fact, there's a really interesting story around this So, you know, I met some of the exciting skies you have there, you know this throw that's what that whichever Xen based distro Actually, we stopped we talked to them. I think in 2015. I Then invited me to an event wanted request to the meeting asked a lot of questions about Xen Two years later. I didn't hear anything. There's launched a product And it was apparently quite successful, you know, a lot, you know a lot of their users use it But one of the things they have come across is exactly this whole issue with power management and stuff like it but not only in the context of Not only in the context of you know Disaggregating drivers, but also higher up when you look at workloads. So on a driver side, you could you know You could you could you know focus on a number of operating systems which you allow and you can plumb some Functionality in there such that that kind of gets coordinated But you know like what happens even further up the stack, right if you have so if I look at this Kailin's Xen based product they allow they allow you to have to have a There's something called bare metal VMs which are almost like traditional ROM images There's a support library and you just can bang, you know one of these old images into a VM and they offer that and also people use it And then they have support for different artists as well But then you start unless you have order power management and all those sort of heterogeneous kind of coordination It's not just about power management other properties as well. Then you're going to start to get into trouble and I see some momentum Building in this area to start resolving some of these problems. We're out of time apparently All right, and then my colleague here Will follow up after so fight me afterwards or at the Xen booth if you have questions