 Hi. So welcome to our multi-hyper open stack session. We're just going to do a quick intro, then we'll get into the main content of the topic. OK, so my name's James Page. I'm a technical architect to the open stack engineering team at Canonical. I've been working in open source for about the last 15 years, primarily as an end user for the first 10 of those and then the last six at Canonical working on Ubuntu and OpenStack. Hi. My name is Gabriel Sanfira, and I'm a senior cloud engineer with Cloudbase, a company focusing primarily on interoperability between Windows, Microsoft Windows, and the open stack world, and various other open source projects like OpenVswitch. I've been involved in open source ever since 2006, and I've been with Cloudbase for the past three years now. OK, so hypervisor choice. Increasingly, OpenStack and KVM, which is the most well-supported option in terms of hypervisor choice for OpenStack, is it's not the only option. There are numerous other choices out there, all which have solid drivers, VMware, Hyper-V from the Cloudbase guys, and the new system container driver, LexD from Canonical, are all choices that end users have. And depending on what workloads you're putting on your cloud, you may drive what choice of hypervisor you want to use to host those workloads. But you don't really want to go for a binary option in terms of your hypervisor choice. So it may be that some things are best on KVM, some things are best on Hyper-V, and some things are best in containers. So having the ability to mix all these things together in a single OpenStack cloud is really important. So don't take a single choice. This obviously requires a consistent approach to some of the kind of core cloud concepts. So things like SDM networking, we have to get right across all of the hypervisor choices that we make and put into a single region. And we're going to dig into that in a little bit more detail about how we achieve that with Hyper-V and how LexD just plugs into the existing solution for KVM. So I'm going to hand over to Gabriel now. He's going to talk a bit about what our cloud base has been doing with Hyper-V. All right, so how many of you use Windows in OpenStack? As a guest or as a hypervisor? OK, how many of you are thinking of using Hyper-V in OpenStack? Not that many. We got used to this. That's why we do these polls every time we have sessions out of curiosity. So Hyper-V in OpenStack, it is a fully functional hypervisor from Microsoft. It's a great fit for OpenStack. And we as a company have focused on, like I said, interoperability between Microsoft Windows and OpenStack and OpenSource since the beginning. Our main focus is to allow you, if you so wish, to just get a Windows VM. I'm sorry, a Windows host, install a Nova on it and just put it in your OpenStack. Have it available and usable without any modifications to your existing infrastructure. That means that if you already have an OpenStack, be it deployed via Juju, be it a fuel deployment or anything else, even manual, it's very easy to just go to our website, get an installer, put it on your Windows box, and have Hyper-V available in your cloud. Sorry about that. We have an installer available on our website that will allow you to just point that particular box to your Keystone, to your Glance, to your Reviton Q server. And it will become available as a hypervisor inside your OpenStack. It's a standard installer that Windows sysadmin would be used to. It also has an unattended mode that is available for automation, if you so wish. And this is actually what we use. The command line parameters is what we use to automate deployments of Hyper-V inside OpenStack clouds. SDN solutions. We've ported OpenV switch to Windows, to Hyper-V, to be more specific, together with VMware. Now, you can have OVS on top of Windows, given the same functionality as you normally would have on any Linux box. So feature parities, one to one almost. Actually, I think it's one to one, period. You get VXLAN, GRE, NVGRE, SDT, and with OVS 2.5, you also get LACP bonding. So the CLI is the same. The same CLI you're used to on Linux is available on Windows. You get OVS VS CTL show at Port, Dell Port, whatever you want. Bridges become TAP devices inside Windows, so you can configure them normally. All you have to do to get them for connectivity on those is add a physical port to that bridge, and you have a medium onto which to send packets. Compatible with Open Daylight, of course. Anything, basically, that speaks OVS DB should be fine. So there's no limitations in that respect. In OpenStack, it's being integrated as a simple neutron agent. So you get the same neutron agent that runs on Linux, running on Windows as well. So there's no difference. It's all upstream. You just get the code. If you want to install it yourself, you can. But if you haven't installed it, why not use that? Now, we get a lot of questions about reference architectures when deploying Windows as a hypervisor inside OpenStack. In December, we did an article revolving around TP4, Windows 2016 Technical Preview 4. And we presented it as a hyperconverged design. Now, many people do this now. This is no different. It's basically compute, storage, and clustering, all on the same node. So in essence, you get a bunch of nodes that participate in the same cluster, sharing one or many disks, depending on how many are free to be used as storage species direct, across all nodes. Those disks basically mirror the data across all of them. It depends. It has three resiliency settings. Basically, the equivalent of rate 0, 1, and 5. This is a basic layout of the hyperconverged design. It depicts basically every service running on top of each individual Hyper-V node. You even get cinder volume on top of Windows. It exports two back ends, an Iskazi back end, and an SMB back end. We've integrated the SMB3 support in LibVirt as well. So you're free to use either Hyper-V or LibVirt with a cinder volume back end running on top of Windows storage server. The hyperconverged design also allows you to failover in case of disaster or drain a system of VMs in case you need to migrate it, decommission it, services, whatever you want. It's seamless. And starting with Mitaka, we have a clustered driver, cluster Hyper-V driver that allows you to do this. Windows as a guest. We've, for Windows as a guest, we have the equivalent of Cloud Init for Windows. It's called Cloud-Based Init. It's been around for a while. And if you've ever wanted to use Windows as a guest inside OpenStack, this is probably what you've used. We provide Windows images as well. So we have evaluation images on our website if you wish to deploy, easily deploy Windows 2012 R2 in your OpenStack Cloud. You can also build your own if you want. We have tools available on GitHub to build your own images. This means you will need an ISO, of course. And you can package any drivers that you need inside your Windows VM before you deploy it to your cloud. If you want to add, for example, vertio drivers, this makes it easy. Just go to GitHub slash cloudbase, github.com slash cloudbase. You'll find plenty of tools that will help you out. Now, our preferred way to deploy this is, of course, Juju. It's not limited to that. But it's what we've been working for for quite some time. We have an excellent partnership with Canonical. And we've integrated Windows support into Juju and Mac. We've created many charms for Windows services. And this allows you, if you so wish, to deploy Hyper-V with the hyper-convert scenario. That means clustering active directory, storage spaces direct, which is a brand new technology from Microsoft. That allows you to use commodity hardware as a storage array, in essence. So that basically means that you no longer have to have access to very expensive sands to get a decent storage array. You can share disks amongst all the nodes in the cluster. So basically, once you form the cluster and enable storage spaces direct, you'll be able to save those instances on the cluster storage itself. And furthermore, it allows you to enable scale out file server that also makes it available to any other node that wants to consume it via SNB or Samba. Like I said, we have an array of Juju charms that allows you to deploy Windows workloads and Hyper-V as a hypervisor, sender volume. The Hyper-V charm, for example, also deploys OVS. So you can have one-to-one parity between KVM and OVS. The whole point of this is to make it a seamless and experience as possible. You'll never be able to tell the difference between Hyper-V or KVM unless you try to SSH into a Hyper-V node. So when you see it in your hypervisor list, that's when you know it's a Hyper-V. Otherwise, deploying to it, using it, is just the same as using KVM. Okay, so let's spend a little bit of time talking about LexD and the driver we've written for NOVA to integrate it into OpenStack. So I'm gonna start with a bit of an overview of LexD itself. So LexD is a project that we started about 12 months ago. I think we announced it at the Paris Summit and we've been working on it ever since. It's a bit of a ground-up rethink on how to do system containers, which I'll talk about in a minute, what system containers mean compared to application containers. So a ground-up rethink in terms of the UX on how to manage system containers on Linux. So it provides a daemon that you run on your host that has a REST API, which we think is a nice, simple API for managing images, the containers booting off those images, the underlying storage that's on the host and the networking of those containers. Has a nice command line interface and it's very much a kind of retake on the original LXC work that we've done over the last sort of five, six years in the Linux kernel itself. As I said, it's designed to be really simple, really simple to use, which has made the driver integration into Nova relatively easy. It's meant to be fast. One of the objectives we had was that spinning up a container should be very, very rapid, ideally sub-second and that's pretty much what we've got to. So by combining a kind of optimized path for creating containers and by using very efficient storage backends such as ZFS to do copy and write cloning of images, and we can achieve very, very short boot times to get running containers. So the time to go from launching a container to being able to log into it is about 0.6 seconds on your average laptop. Designed to be secure. So a LXD managed container is by default an unprivileged container. The difference between a privileged container and an unprivileged container is that the processes in unprivileged container are not running as root on the host. So if there was a compromise of the containment of the container, then that compromise only has the equivalent access of the unprivileged user on the host Linux install as well. So that already helps. We wrapped that in our Palmer. So if that does happen, then there's some additional protection for processes in that process group and they can't attack the rest of the host. The rest API is network accessible. We use TLS for encryption and authentication is done using client certificates and we have a bootstrap mechanism to get clients logged into hosts to allow that to happen really neatly. So I mentioned the word system containers. Containers is a word that's overloaded all the time. It really just means containment. Actually a KVM instance is really a container, but it has them to have the kernel and firmware and all the other bits and pieces that are real machine and simulating a real machine. So if we look at things like Rocket and Docker and they're about application containment. So typically a single process. There's nothing else running in the container apart from that one thing. We have the hypervisors we all know and love, VMware, Hyper-V, KVM. They're managing a group of processes in one way or another that may be through a KVM container with QMU, maybe Hyper-V and that's where Lex defits. It's equivalent to what KVM is doing with QMU. We're managing full system containers. You log on to a Lex decad managed container. It has SSH to start off with. It has Syslog, it has Chrome. It has all the things you would expect to have on a full system and you can manage it in exactly the same way. So all your tooling that currently works with your physical server infrastructure and your KVM instances works identically in a Lex decad container. We have other hypervisory type features in Lex decad so we can snapshot containers. We can roll back in time to snapshots we've made. We can create images from those snapshots. All things that other hypers have done well, we aim to do one as well. We can do migration between Lex decad hypervisors. So moving those containers depending on for maintenance, for balancing a workload, whatever it might be, we're able to do that as well. So that makes Lex decad a really great fit for the semantics of the end user API for NOVA. So we're not aiming to be in Magnum because that's very much about application containment and management of those containers. We're aiming for NOVA as our primary use case and managing containers as systems. And as a result, a NOVA Lex decad container is managing exactly the same way as a KVM instance. You boot, reboot, resize, add floating IPs to it. As I said, we integrate into Neutron in exactly the same way as KVM does. So it's open V switch on the host or whichever SDN you might be using. And we have some nice semantics on how Lex decad can then plug in to the open V switch instance in a very similar way to how Libvert manages. We can apply constraints to containers as well. So if you've ever booted a Lexi container with no constraints, you see all the resources of the host system by default. So we're able to apply flavors as we're booting containers. So an M1 small is gonna, only containers ever gonna be able to use one processor and a couple of gig of RAM and they can't ever use any more of that. And we can use that for capacity management and reporting the capacity on a Lexi host back into the Nova scheduler for scheduling decisions as well. So we can do migration, so we can move instances around between hosts, allowing for maintenance or workload balancing, that sort of stuff. And the other thing that is really noticeable about Lexi is performance. So in effect, all the processes that are running in the container are running on the host OS. There's no additional level of kernel, there's no emulated firmware, BIOS, that sort of thing. So the KVM stats great, it's very optimized, there's been a lot of really good work gone into the virtual drivers for instances and optimizing all the copars to make sure everything is really, really efficient. When we look across the board at metrics, the one that really jumps it out is when you start doing lots and lots of high concurrency small IO and Lexi deals with that particular workload very, very well. So Cassandra Stress is one of the kind of stock benchmarks for assessing the performance of a Cassandra cluster and it generates lots and lots of really noisy writers with very small IO, both network and disk. And when we're on Lexi D, we see a much, much lower latency when compared to the same test on the same hardware with KVM, which results in a much higher throughput in terms of what a given set of hardware can do for this particular benchmark. So Lexi, as a hypervisor, is very, very good for getting your processes as close to the bare metal as possible. So that HPC applications, that sort of stuff, it's ideal. So it's very good for density, that there's very, very minimal overhead in the container. We were booting some up earlier for the demo that I'm hopefully about to do. And the overhead of each container is about eight megabytes. That's an idle run for a container. So if you've got lots and lots of small workloads that are particularly busy, then packing them into Lexi D containers, you're gonna get much better density out of your available memory and processor resources than you would do with KVM. Okay, so how do we put that all together and make a true multi-hypervisor open-stack cloud? So the first trick is how do we ensure that things get scheduled in the right place? We know we have a workload that we know runs best on Hyper-V. How do we make sure it gets onto a Hyper-V hypervisor and not onto a KVM host or an XD host? The way we approach that is to use the hypervisor type attribute that you can place on any glance image. So by tagging the images, the image for each of these hypervisors is actually in a different format, which is helpful in this case. We can ensure that the Nova scheduler knows exactly where to make scheduling decisions when you want to boot one of those images. So the semantics are you pick your image, you pick your flavor, and you ask Nova to boot it. And Nova will then use, I think it's the image properties filter to ensure that the scheduling request is then matched to an inappropriate underlying hypervisor and your instances end up in exactly the right place. And as Gabriel mentioned, he's mentioned Judeo and Maz a couple of times. So had you deployed this wonderful beast of KVM, LexD and Hyper-V all in one cloud? Well, the approach we've taken is to use Juju, which is a service modeling tool from Canonical and Maz, which is the tool, metal as a service, used for managing physical machine resources in your data center. And by modeling all the various different components of OpenStack, both on Linux and on Windows, we're able to have a single deployment tool to lay down an entire cloud, which both provides Hyper-V and provides LexD and provides KVM to give a consistent deployment experience for putting down an OpenStack cloud. I'm going to look at Gabriel. I hope he might have a monitor going by now. Let's give it a shot. Unfortunately, we've had some trouble with the video out on this thing. And no joy, unfortunately. Okay, so while Gabriel's desperately trying to get some video out going. And if we don't get it on screen, if anybody wants to come and have a look, then please come up at the end and we'll run people through on the laptop screen. He's got a, on his laptop, he's got a virtual environment set up using VMware Fusion. And that includes a full Maz deployment and a number of virtual machines that are assimilating physical hardware. They've all been registered into Maz in exactly the same ways you would do in data center. Maz is able to then power control those via the VMware Fusion API. Gives you a nice sandboxed way of testing out deployment topologies. We've then used juju to then deploy the cloud onto that infrastructure. Yes. And we're not gonna get a video out, no. Okay, so like I said, if people wanna come see the demo, please come up and we'll run you through. We've got 10 minutes left total. So if anybody's got any questions, please feel free to ask now. There's mics on either side. And if you can speak into those, then they'll be available on the recording afterwards. All right, one thing I wanted to know about was you said that it uses the image metadata. Yeah. Now, what about booting from a volume? Would that not be possible in the situation to have it pick the right hypervisor? It's an interesting question. I don't know whether that metadata is lost at that point in time, quite possibly. So that, yeah, it wouldn't be easy to deal with it. So, yeah. So you mentioned that you can do live migration on LXT, right? What are the performance of? Sorry, can you repeat? The performance about live migration of LXT. So, as of today, and the 16.04 release of Ubuntu contained the first product, well, first GA of the NovaLex driver, LXD driver, gonna do cold migration. The live migration work is still in progress. There's a preview of it in 16.04, but we didn't feel it was good enough to put into the driver. So we'll be working on that for our next release in six months' time. Performance is very much dependent on how much stuff is running in the container. So the amount of IO it's generating, the amount of memory and processor it's consuming is proportional to the amount of time it then takes to transfer that over. So, yeah, you get a blip in much the same ways you would do with a KVM migration or a Hyper-V migration. It's as minimal as we can make it. Must be. Thank you. I have two questions, one following on the other. First, you mentioned that the image type is kind of tied to the hypervisor type. Yeah. Could you explain that a little more and then also to follow up, is it possible to use the same image type for your image on multiple hypervisors? And I'm thinking here LXD and KVM because Hyper-V probably. Well, yeah, Hyper-V is the, what's the format for Hyper-V? VHDX. Yeah, so VHDX is a very specific hyper-V. The image formats for LXD and KVM are different as well. You cannot boot a QCOW 2 image on a LXD hypervisor. The underlying semantics of how an image is managed into LXD are very much different. So the image format for LXD is basically a root table which then is LXD then will unpack into a volume that it creates and then presents that to the container. So there is currently for KVM Hyper-V and LXD there isn't any crossover in the image formats which allows us to use this approach for scheduling. You showed storage latency benchmark comparing LXD versus KVM. Two questions, is there a specific reason why KVM performance was so bad compared to LXD? And second question is, did you also compare Hyper-V? I haven't done that benchmark. So I won't answer that one. I'm not a virtualization expert but we have a lot of those at Canonical who looked at our data and thought about why that might be the case. My understanding is especially when there's contention on between virtual machines on the same set of hardware resources that the cost of switching in the full KVM instance on and off processor is what then generates that drop in increasing latency. So because the LXD processes are running directly as individual processes on the host, we get all the advantages of the native process scheduling in the Linux kernel and it's just much more optimized. There's the cost of putting a process rather than the entire machine on and off processor is much, much smaller. So that's why we think we see that difference in performance. I don't know why I haven't benchmarked Hyper-V against LXD. We haven't either. So LXD is quite new, we haven't had a chance but Hyper-V is on par with KVM in terms of performance given that enlightenment drivers inside VMs running on top of KVM for Windows are enabled, for on top of Hyper-V, sorry, are enabled. So, and vice versa. So if you enable the drivers, the prior virtualized drivers in both cases you get similar performance. The added value is in other areas like licensing support for various workloads on top of Windows and so on. So to get LXD in your existing OpenStack cluster all I have to do is install, I mean we are running CentOS Red Hat, all we have to do is install LXD and add it to my nobleconf and that's it. So, CentOS is not something we've been testing on. In theory, the drivers are in in Python, the Nova driver, LXD is a Go binary. There's a number of other things you need to get lined up like kernel versions, LXD versions for libraries. You could possibly make that work on CentOS. We haven't done that particular piece in work. If you want to try it and in a place we know works really well, 16.04, which came out last Thursday is an ideal platform for it. You can do it on 14.04, you just have to jump from your hoops to get all the right things back for it and install, so. Any other questions? Cool, so if people want to come see the demo, I'm sure Gabriel will be one of the players to have people hooding around him. In the last five minutes we've got for the next session. So, you're not Tyker Anderson. Sorry? I've got your name wrong on the last page. Thank you for listening. I look like a Tyker. I don't copy and paste slides, honestly. There's a little bit more information about LXD and Linux containers there and if you grep Google for Nova LXD, you'll find a lot more detail about our actual driver as well. So, thank you very much. Thank you.