 So my name is Lars Kurt. I'm the community guy at XSEM Project. I also happen to chair the XSEM Project advisory board which funds the XSEM Project with the Vision and Linux Foundation. And Russ asked me to give you a little bit of an update what's going on in the hypervisor. Maybe a little bit of introductory stuff and try and give the whole thing relevant to unicornals as well. Which is actually quite hard to do in 30 minutes. So if it doesn't quite flow and there's some gaps in between then that's because you know I had to make choices. So I think it kind of sort of works. So as I'm a community guy I sort of just wanted to sort of share some stuff you know with you. Which I find quite interesting. So you know what this graph shows you is you know the recent hypervisor releases history. Oh I spelled history right that's good. And this kind of shows you how over the last two releases we actually managed to deliver more features you know per month. And you know that trend seems to be continuing. So that's good because you know more features actually means more good stuff for all you guys to use. If you start looking at hypervisor git commits oh. You sort of see that similar well actually it was really interesting that for a long time we kind of you know we've been fairly stable and this year or last year things really jumped you know significantly you know almost 40% more commits than in a year before. And I see that trend continuing. Now it's really interesting about this. This is really showing there's more innovation. But what's happening at the same time is that the demands on the project are also changing right. But actually everyone in the ecosystem requires that the whole thing is of better quality and we also deal a lot better with potential security issues. And you know despite all that you know for the effect of that has actually been that it's harder to get code into the project particularly for newcomers you know where it can sometimes be a bit disheartening because you know the number of review cycles in those cases keep on going up. So that's sort of just setting the scene a little bit more innovation more stuff happening and you know more goodness which hopefully you guys can also then benefit from. So now to the that's sort of just setting the scene now to the introductory bit. So actually first who's familiar with the hypervisor and how it works? All right probably about third so I'm going to go a little bit through some of the architectural stuff. And then I'm trying to interleave it with some stuff which is relevant to Unicornals and throw some new things in it as well. So since the type one hypervisor is a type one with a twist because it's kind of not quite like a type one. And what does this mean? So at the bottom you have the hardware. There's memory, CPUs, ILOs. Then we have the hypervisor itself and that's you know the things like setup configuration it has schedulers and it deals with memory management, timers, interrupts. And that's kind of where the type one thing comes from. It sort of abstracts the hardware. If you look at a classical type two you would have your host operating system in there and it interacts and threads the VMs in some way shape or form. So this is kind of now where the twist comes from. So the first virtual machine we have in a system, DOM0, it's called DOM0 because it's the first one which gets started. And that runs the host kernel which is Linux so it can be one of the BSDs. And the main purpose of that of the kernel is really to provide you access to the low level hardware. To the device drivers, ION and so on and so forth. It is also the main entry point to interact with the outside world. So we have these things called two stacks which then interact with your cloud orchestration stack or command line tool or whatever. That's how you control everything in the system. Then we have your typical guest VM so I'm just showing one. You might have a guest OS and your applications in there or it might be a unique kernel. And these typically talk via the PV interface to the device drivers to the ION. So there you have a pair of PV front end drivers and back end drivers which then talk to the hardware drivers which talk to the hardware. PV? Oh, Parabritualized. Yeah. Never. And then you just replicate this across the board. Now that has some architectural advantages. It's thin, gives you big density and it makes it excellent for supporting a lot of small workloads such as unique kernels. It's highly scalable. It can run a huge number of VMs. Again, very good for a lot of small unique kernels. It's also good for something called disaggregation. Actually the same idea behind unique kernels is used within CEN as well. You can run individual device drivers or QMU and stuff like that within different VMs as well. And in some sense those things are actually like unique kernels. We have these things called stop domains which is basically QMU running on top of MiniOS which is kind of a unique kernel. Then of course this whole architecture has some security implications. Host OS is isolated within the VM. But let's not get distracted about it. It's well known that we have a lot of advantages in this area. One other real interesting thing and I'll get into this in a little bit later is that you can use, you can plug and play with schedulers. And that could potentially be quite interesting for different unique kernel workloads. But I'll revisit this a little bit later as one of the things which is interesting and where it also has been quite a bit of development in the last two releases. And then there's the whole idea of PV which is fundamentally the basis for a parallelization, fundamentally the basis for unique kernels today. And it's really easy to implement a unique kernel base against PV drivers. And it's one of the reasons like Senus the dominant platform for unique kernels today. It also gives you really fast three times which are necessary for a lot of unique kernel workloads. If you have any questions at any time just interrupt me. So that's kind of some of the basic stuff and I wanted to look at some of the more interesting things and make this a bit more relevant. So I wanted to look a bit at the interstates between Sen and unique kernels. Mainly from the viewpoint of virtualization modes, where they're going and what the implications for unique kernel development might be there. And then there's going to be a slight call to action for some of you guys to maybe think about that, engage with the project and make sure that this is going in the right direction. Because of the type of developers who do the very low level stuff and your needs might be slightly, there's a different set of people and our core developers tend to anticipate what you need, but they don't always get it right. So let's look at the different virtualization modes. So we have PV here, this is how it all started, then hardware extensions came along. So the HVM mode was introduced and then we started to optimize this along the time and make it better and faster. And what's really interesting is that this is actually today the fastest virtualization mode on Sen if you look at benchmarks. This one is the one with the lowest latency and the simplest one in terms of implementation. So we kind of wanted to get to a point where you have both, the fastest and the simplicity somehow and this is how PDA was formed. I'll get to that a little bit later. Why is this not showing anything? Then it shows you when each version was introduced in the Sen version and actually today this is really where all the or the majority of all the Unikernels are interface with Sen. So both Rump Run and Minerals interface with this mode today. There are other Unikernels like USB which use a different approach but I think the majority use TV today. So let's look a little bit at the various trade-offs and where this might be going. So this kind of table shows you how some of these things are implemented and kind of explain I'll just ignore those first two. Those virtual columns show you what implemented in which way and if you look at this column, this is where you get most of the runtime performance from through hardware virtualization of normal operations and that's where a lot of effort by chip manufacturers, Intel and so on. That's where all the effort goes and that's why that's why to a large extent PDA is the fastest virtualization mode today. And then of course for IL you just want to use the PB driver because in hardware that's kind of not so optimal. Now there in a traditional parallelization mode we kind of do this in software and that's why in a lot of workloads PD is a little bit slower than PDH or PDHVM. This one here boot and motherboard that's kind of where you get the fast boot time from. So there this is obviously done in software together with some hardware support and you have the whole setup process and that's why it takes a little bit longer. So what we ultimately want to get to is get the benefits of both of these and marry them. So I wanted to explore a little bit why we went to that mode and what we did and how we where this is going because there's some changes coming in the next two releases. So the motivation behind PBH was fundamentally we want to basically be able to run DOM0 as an HVM guest to get all this extra performance. Now DOM0 always has to be a PD guest. And the reason for that is if you look at this architectural model we had HVM today requires 2MU to run. Now if you want to boot up DOM0 and you want to run 2MU in it you have a certain dependency there so that's why you can't do that. And that's why today DOM0 almost must be a PD guest. The real motivation behind that is performance. If DOM0 is faster than all the other VMs become faster. So that's one of the reasons that gets us to the second point. We want PDHV as fast as HVM. So the initial approach which was taken there was to take an HVM container which gives you all the hardware access and run PV in it. The problem with this is that there's a lot of restrictions which come from the PD in architecture. So for example everything's implemented in the software so you can't really play with, for example a guest can only run in protected mode. You can't change modes after the guest has been created and all sorts of other things. But actually those limitations aren't really strictly necessary for that mode. They're just there. The whole concept was also designed before all these extra quality security requirements came in. We implemented PDHV as it is today and then we found if you wanted to get all the features in place you would need to add a whole lot of extra code which is new code and isn't shared with anything else which already exists in the system. So last year there was a big discussion in the community to see how can we actually fix this. The way, so if you look back at it we took an HVM container and we put PD in it. So what basically we sort of thought about was actually what if we just took HVM and caught it out and removed the dependency on QMU. That would mean it's actually exactly like all the other HVM modes. It has no QMU so it doesn't have this sort of circular bootstrap problem. It can be run as Dom0 and it should give you all the other benefits. From an end user's perspective it behaves exactly like what we did beforehand with PDH. It just implemented in a slightly different way. Now this is already in 47 and it's really interesting. Actually we ran benchmarks and so on. It's very fast. It boots up relatively quickly but it's not completely finished. So basically what we've done is we renamed it because it's actually not a PD thing anymore. It's really HVM with stuff taking out and you can already try that in Zen today. You just use an HVM guest and you choose device model version is none. Normally it's QMU so it doesn't use any device emulation and here you are. However, where are we with this? Well, it's in Zen today as DomU support. It will be in the next Zen release. We will have a few decisions still to make about naming. How do we name it? How do we make it accessible to users? It does create a little bit of confusion and we sort of need to manage this. Then in the next release there's already a Dom0 prototype for free BSD. It works. There's no Linux implementation yet so that's coming. That needs to be implemented before it's going to be fully available. There's some cleanup tasks required and the interfaces aren't yet clear to stable but we're almost there so there's discussion going on around validating all the interfaces and the benchmark is already very impressive. So it's basically very fast. So here's what I kind of want you guys to do the one to who might be using this in the future. Currently this stuff isn't used as unicolonial based but it seems that this is the way forward in the future. So really to make sure that that really works is we need to make sure that the architecture actually works for you before we declare that it's stable. So give it a run, try it, play with it and see what it does. MiniOS and Rump Run haven't been ported to that yet so we need to make sure that this actually can be done and is relatively easy. So in a lot of ways that creates an opportunity, a voice application so in many ways if I remember back in the early days of the unicolonial we had actually quite a few different MiniOS clones which were different branches of MiniOS and it took quite some time to resolve this and get it all back into one version. And by starting this process right now we're sort of avoiding those type of issues. So my call to action there will be work with us, engage with the project and see how it works. You know give the developers who are working on that feedback. So that's kind of the, that didn't really work it well there that whole part. So I wanted to look at some other things, performance. That ties a little bit back to the mode I was talking about earlier. So in the last two releases we had an awful lot of performance improvements in CEN. I'm just listing some of them here. So these were in, I'm not going to go into detail, these were in 4.5. This is stuff which came in 4.6. And actually that really made a big impact on a number of workloads. So if I look at the last published benchmark, this one, this was a, you can get the detailed benchmark in here. This was sent a 4.2 Linux kernel with CEN 4.5 and compared against QNU. And it's really interesting that most of the benchmarks either send one and clearly. So if you look at this, this is like a 2.3, 1.3 split. And this is even before a lot of the improvements which came in 4.6. So on a whole performance front I expect that there will be a lot more activity and a lot more competition and benefits for CEN going forward. But a lot of this is mainly HVM. So all of these benefits are on HVM, PVH and so on. Another area, schedulers. So I mentioned earlier that CEN has the capability to plug schedulers. And how does this really look, right? So a number of different schedulers which have different properties. And in the hypervisor layer, per host, you can choose which scheduler you want. But it's actually even better than that. You could create a CPU pool and attach a different scheduler to it. Why is this relevant? Well, you could, for example, have a number of very latency-sensitive unikernels and then choose a scheduler which works better for those workloads. And kind of optimize your system in that way. So how does it look? Say here you have your R2DS scheduler, here you have a credit scheduler and fundamentally these VMs will use one scheduler and the other VMs will use another one. Then you can change the scheduler parameters per host. You could change them per CPU pool or per individual VM. And that gives you a whole lot of ranges for optimization of a complex system where you have lots of different parts which may be collaborating with each other. So this just shows you a table of the different schedulers which are available today. For the ones which are worth looking at, possibly in the context of unikernels, our credit too, because it's a very low latency and a low latency is important for unikernels. RTDS, which is actually really a scheduler for embedded and automotive use cases may not be so interesting, but I don't know if this is interesting, come to me and I can try and figure something out and get you together with some of the other scheduler people. Scheduler use cases. So the RTDS scheduler that's one which is aimed at automotive use cases, latency sensitives, and guaranteed quality of service. So you know that in a specific time slice, you will get a specific time slice for a specific VM and you can control this at a very small granular level. Incidentally, this is also interesting for some cloud based use cases such as gaming, video decoding, and so on. Or in our areas where you might see another interesting area where we started seeing this being deployed is by vendors such as Verizon who have guaranteed quality of service. So they're doing this in hardware right now by choosing different hardware buckets, but you could also do this in software using scheduler technology. Let's look at another part on schedulers. So I just wanted to sort of give you a quick rundown on some other stuff that's going on in the community and rough up with the conclusions. I'm going to cover a little bit various areas a little bit superficially. We've seen an awful lot of contributions around x86 hardware support in the last year and a half. Predominantly from Intel. A lot of this is really specific, send supporting specific hardware features for specific use cases around NFV, optimizing your system, and so on. Rather than going into detail with that, it's basically in this area and what we are fundamentally seeing is this is actually really interesting, just in the past we saw Intel contributions hovering around maybe 1 or 2% of the entire community and it's shot up to about 7-8% recently. The same is happening in the ARM space. It's tracked very closely. The project also runs a test lab and we'll be adding ARM hardware, 64-bit hardware to it as it becomes available and there will be a lot of activity in this year. Also from the ARM, I don't know, I guess for Unikernels, your guys are mainly interested in 32-bit and smaller devices today so that's also going to provide future event use for you. Another area from activity is graphics. I'm not going to show you this demo because we won't have the time but there's been a lot of activity around graphics virtualization. It used to be called GBG but it's a technology now called Intel GBG and what this basically does is allows you to share a graphics GPU amongst different VMs and basically each VM has access to a partition of the graphics VM. Now this is already but right now part of this is still out of free but most of it is in send today. It's in use by send client and the upcoming send server 7 versions. Most of the send patches are already part of the tree but there's some refactoring going on around Linux and QMU which might affect the stuff which is already in send. A similar approach is also being developed for hardware devices by Globalogic. They're reusing some of the code which Intel has developed for X86 for ARM based architecture. There was just a blog post showing some demos that focused around embedded architecture. There's another interesting set of technologies called virtual machine introspection so what that does is basically send has this API which allows you from within a dedicated VM to monitor what's going on in another VM and these interfaces are called VMI interfaces. They started appearing in the last two releases and that has some really interesting implications for how security can be handled in the cloud. Again there's a link to a demo which you can watch once you look at the presentation later. So how does this work? The cloud security today, you have your typical system with dom zero and then you might have a number of agents running in each VM. Let's just say these agents are antivirus software or some disk and memory scanner or network more than or something like that and that kind of A it introduces a management overhead but also it means that all of those VMs might start doing virus checking at the same time and that has an impact on your system. It typically calls antivirus storm. How does this VMI approach can assault this? Actually what you have there is you have a new VM, basically a security appliance that uses that VMI interface and it monitors a number of VMs and you could have several of these security appliances which monitor different sets of VMs and that's relevant of course if you have a multi-tenancy environment. There's also a hybrid approach possible where you have several agents leaner agents which run in those VMs which then collaborate with that introspection engine. Now again, there are some products being worked on by a number of security companies like Bitdefender, Noveta and a few others and some of these things will be coming to market this year. Well the thing is actually you could run this in DOM zero but then you're going to have loading more stuff into DOM zero which you don't really want and also of course this is optimized for cloud deployment so actually what you really want is you want to have a VM image which can be easily installed and managed rather than having to normally put something in a VM into DOM zero and then deal with it. And then there's also this little lock thing here which is also quite important which is X of M is the equivalent of SE Linux for X and it allows you to control what that engine actually can do and what it can't. But that's a whole set of other issues and interesting conversations around the topics to be had and I can only just cover this superficially. A lot of other security features are also currently being developed so one of the things we've been last year you may remember there was the Venom vulnerability following that with a whole lot of other QMU bugs and this has been rather annoying for people within the community so basically right now QMU runs in unprivileged mode within DOM zero so any QMU vulnerability is basically a risk towards Xen. So what we decided to do is basically sandbox and be privileged QMU but also the other emulation technologies within Xen to basically protect ourselves against QMU vulnerabilities and to make the system more secure. So that's a direct response to eliminate a whole class of issues where we don't have a lot of control over. Another thing which, and it's coming in 4.7, another thing which is being developed but this will take several iterations to be complete is hot patching based on the code work that is exploit. So what that means is there's a security patch it can basically be wrapped up in a payload and deployed on a running Xen instance, hypervisor instance we can do this without rebooting the host and that means that things like cloud reboots will be a thing of the past. In fact some cloud providers have that capability already and are collaborating with the community on an extensible framework and common technology to make this all happen. So we're starting with some simpler use cases and successively add less common ones. Then another thing which is also being there's some work going on right now is better configurability. So again if you look at security, the more code you have or even if you don't use it, the bigger your attack surface is. So not everybody in Xen community uses the same type of feature. So ideally what we want to end up with is have the capability to switch and remove code both at runtime and at compile time which aren't being used to reduce the risk. So there's some work going into 4.7 with K conflict but actually we're going to look at the whole architecture to come up with a common approach for the future and expectable also to go over several reliefs like this. And we're nearly at the end so actually I think I hope I could show that a project has a history of proactively innovating. The rate of innovation is increasing, that means more features more quickly. The demands on a project are actually shifting quite a lot so it's a lot more focused on quality and security and what I've been dealing with particularly in the last years that has actually created tensions within the community because we have a lot of newcomers which want to get their stuff in quickly but then a lot of the more established players want to ensure quality and security are at the level as it was or even that level is sort of increasing every single year. And that sort of led to conflicts to some people being more verbal about this within the community because there are all these well let's say as a community we haven't made conscious decisions about trade-offs, what's more important than the other one and we're kind of going through this process right now. To decide when is a new feature more important than quality, can we have mechanisms to facilitate the whole process. And it leads me that the project does have a track record of adapting to challenges and criticism. Like some of the security things we've been working on I've been showing recently is a direct consequence of some of those challenges. And I think it's also the best platform for Unicurl in the cloud because A, it's basically running most clouds today and it has a lot of unique features and there's a lot of innovation going forward and that's it. Any questions please? Go ahead. The other thing is like as an open source community this doesn't work right? And particularly in the world as it is today if you don't innovate and you don't do new stuff then you don't exist you don't get any press releases. People think you're not relevant anymore and then there's a new kid on a block who does more features and how do you solve that? This kind of actually comes back to this whole finding the right trade off between different things and I think this is generally a challenge which we're facing as an industry today because I think not all projects today or all successful projects will eventually face that challenge. If you add new features then some people you're adding new stuff, you're growing a code base if you do stuff too quickly it might affect quality there's all these different trade offs you have to constantly look at and one of the lessons I learned is that a lot of the times this stuff happens implicitly without having a discussion and a conscious decision about it and that's when you start your problems. So you have to bring some of these things out in the open and then make some conscious decisions and then things buy in and people are actually happy with the direction you're going in and I guess at some point all these unicronal communities will have to face similar issues it's all quite nice when you start and you can do all this cool stuff if you want to have users, particularly when a big corporate user then things will become different. Any more questions? Well there are a lot of tools out there which you can do so you can't formally prove it, particularly in a language like C you might be able to prove some of it in some more modern languages you can use passing techniques to find issues and we do know that some of the big vendors which use SIN, they do use browsing to try and identify issues. This is also, what actually typically tends to happen is that some of them, they start a little project where they're focused on some components and then suddenly we get a whole bunch of vulnerabilities in which were discovered by some of those browsing tools and the same happens for other open source projects and then you kind of get these whole batch of security vulnerabilities which come at once and then you get the whole if you're a popular project then a media might pick up on it and then somebody found seven vulnerabilities in this area it's actually bad, but is it really? It actually means somebody is looking for issues and they talk on Friday which looks at this whole security sorry, it's Friday today. On Sunday which looks at this whole topic, comparing how different open source projects deal with this, it's actually shockingly bad in many ways, however as an industry we deal with this issue. It's a whole range of different approaches, they're often not really compatible but that comes to my focus on that. So on the individuals it's kind of, so the overall number of contributors is increasing what I have, so what keeps on happening is we keep on getting a spike of people who look like individuals and then later on it turns out that it maybe was a start-up in stealth mode but there's a number of other interesting trends that did start which have happened particularly in the last year or in the last three years, so traditional extent has been a very European and US-focused project, a predominantly European there's a lot more contributions now coming from Asia, in particular China, so the number of Chinese developers has gone up over the last three years from 2% to almost 12%, right and actually I'm regularly going out to China to give training because there's also interesting cultural challenges working with which sometimes lead to problems. Another thing I have started seeing particularly in the second half last year, a lot more security companies are starting to build, to contribute to them and it's a reflection, in some cases we know that they're building products and in some cases we know that they're doing something but we don't exactly know what they do so it will be interesting to watch that space as well and then suddenly some new company names start appearing and like one which started to become quite active recently and they're quite they're probably also looking at some unique stuff, potentially they're called Star Labs, there's a couple of other ones as well which I know exist but they haven't come out publicly yet so I'm going to share some information about those I mean that's sort of some of the inside of the constitution, there's some more about any more questions? Let's give Lars a big hand and just for closing I just want to say thank you all for coming out spending your Friday with us. Just show of hands was this worth your time? There we go, that's what I wanted to see that's cool. Thank you again I do have talk three o'clock tomorrow, Lars has another quick talk on Sunday please feel free to drop by, we thank you very much and have a very good scale conference.