 Hi, everybody. All right. OK. Hi. Hey, everyone. I'm Stefan Khaber. I work at Canonical. I'm the LexD and LexE project leader. And I'm James Page. I also work at Canonical. And I'm the technical architect for OpenStack at Canonical. So we're going to be talking to you about deploying OpenStack with machine containers. And Stefan is going to kick off with a bit of an overview of LexD, which is the underlying container technology that we're using. And then we'll go in a bit more on the Nova driver we've implemented as well. So I'll hand you over to Stefan now. All right. Cool. Come on. Switch. All right. OK. So what's LexD really briefly? It's basically a new LexE experience. We've been working on that for a couple of years now. It's meant it doesn't have to be really simple. It comes with a completely redone, you know, command interface. It's got a REST API that you can interact with. You can script it really easily. It's got much clearer terminology than we ever did with LexE. It's really clear what you're doing at any time. It is really extremely fast because it's not the transition we're doing containers. As you know, Nova transition overhead whatsoever basically brought performance with the exception of if you set resource limits, you might have a tiny bit of overhead from the kernel to do the resource limitation. It's very secure. We use every single security feature that exists in the kernel effectively to make those containers as safe as you can possibly make them without turning them into full-fledged machine. And it's curable. It works perfectly fine for your day-to-day development thing, which is what the development of your laptop, which is what I'll show you afterwards. But it also scales to thousands of containers, which is what James will be talking about afterwards if you use it through OpenStack. What does it look like? So it's kind of usually in typical LexE deployment, you would have multiple hosts. They run on the external. You might run the same one on each of them or not. We don't really care. You run... LexE is the library that we use to actually talk to the kernel and that we've developed, which is then used by LexE, which then exports the rest API that we have clients for. So we've come in nine tools, so what I'll be showing. People have been writing their own tools, talking to rest API directly. And we've got Nova LexE, which is another client for LexE itself. Now, briefly, what it isn't. So, as I said earlier, it's not another visualization technology. We only focus on containers. That does restrict if you can run inside them. But it also gets you a lot more performance, especially for idle workloads and that kind of stuff. It's not a fork of LexE. It's by the same team that makes LexE. It was a nice way for us to reset the LexE user experience, expose LexE to a multi-host kind of thing. And we do a bunch of things. It's image-based. It's not a fork of LexE. It's not a fork of LexE. It is using live LexE underneath it. It's the least of what it isn't. And it's not another application container manager. It's not another docker. It's not another rocket. We don't care about running a single process. We don't care about what's running inside your container in any way. We take a full Linux distribution. We exec. It's in its system. And that's the extent of the interaction we've got with the inside of the container. So long as we've got any process that we can exec, we are perfectly happy with it. We don't care what you're doing with your container. So it's really the same way that we would do for virtual machine in that regard. All right. Because we're going to try and do a quick demo of what we've got right now before James goes on and talk about the output stack side of it. So, yeah, we've got terminal. All right. So, first thing on that machine, I'm just going to actually install LexE. I'm using the snap package from Ubuntu. We've got a bunch of different ways of installing it on different distros. Anyways, it's installed. It's polygenerating keys right now in the background. Well, it's done already. And so we can just do the first configuration step, which is configuring everything. First, I'm going to use the FS. I'm going to create a new pool for me. Call the pool LexE. Use it on loop storage right now. Default size. I do want LexE to be available over the network. Use default address support. Use a password. And the rest, I don't care so much. I just want default networking and everything. And, okay. So LexE is configured and ready to roll. First thing we'll do is just launch an Ubuntu 16.04 container. So it's going to physically grab the latest Ubuntu 16.04 image for the architecture of that system, which is 64-bit Intel. Then back it, read the metadata from it, create a ZFS file system, because I'm using ZFS as the back end for that particular LexE. We'll also do barfs, LVM, and a bunch of other back ends. But ZFS in that case. So it's creating a new ZFS file system, which can then be cloned for all your containers after that, making them really, really quick to create. All right. So that's an Ubuntu container created. Now let's do an Arch Linux container, just for good measure. We've got community maintained images, and produced daily for a lot of Linux distributions. We've got Alpine Linux, Arch Linux, Debian, Gentoo, Fedora, CentOS, Oracle Enterprise Linux, Polyfog, I've got a few. So there's a good chance your distro, in particular, version of that distro is already available. All right. So for those two containers created now, that suddenly doesn't really quite fit in the screen, but if I make it slightly smaller, it might. Well, all right. But we... So we see two containers running. They've got the IPv4 address, IPv6 address. Fine. Now, we can exact a shell inside the Ubuntu container, which we've done. We can look at the process list inside there. We basically see a net process and the usual system like SMD, Suspects, and Chrome and SSHD and kind of stuff. But it looks like a full system. Now we can go look at things like CPU info. If I grab... Oops. I cannot. Prox. Info. Grab. All right. So we see we've got eight processors on there, because that's what the physical machine has. Disk space. We've got the old system right now. So it's the 42 gigs we saw earlier. And memory, we've got 12 gigs. That's what the host has. But we can change that live as we want. So set... We want two CPUs. We want one gig of RAM. And we want a disk limit of five gigs. Go back inside the container. Disk space is down to five gigs. Memory is now down to one gig. And if I look at CPU info, we're down to two CPUs. So that's resource management for FlexD. We can also do resource block IO limits for IOPS to particular devices. We can do bandwidth limit for network, all that kind of stuff. All of that stuff can be applied live to our containers. All right. Now we've had that second container. Actually, I can create a snapshot of its current state with a snapshot. Then we can get the shell inside there. And let's install something like Apache. So, okay. Apache is installed. We can query that to make sure that it is. And now let's just restore the snapshot to its container. There we go. And if we query, it's gone. So you can snapshot and revert extremely quickly. It takes less than a second to snapshot, less than a second to revert anything. So that's like your normal snapshot story. Now, let's see. Our command line client tool can also deal with multiple hosts. So I've got another server that I can install. Gitflex. I do lxtremote add to other remote server. That adds it to the list of the remotes I've got in my client. And I can now do things like list what's running on that remote server, which is nothing. And say, I want to start in a point container on there. That's done. Can get a shell exactly the same way. That's another network this time. And let's create some random file in the file system and part of that container. There we go. I can now move that container to my other host. So I'm just transferred over to the network. It did the cfs and receive optimized file transfer. And we see it's now on the same machine that's running the arch, Linux, and Ubuntu container. And I can just start it again here. Get my shell in there. And the file is there. Any questions? Maybe. I can try. Let's break my back. What does that do? All right. Last thing. There was one other question. Which I assume wasn't changed contrast. So the question is, can I live my great containers between hosts? You can live my great containers between hosts with FlexD. Yes, we do integrate with Crewe. We've been doing some work on it. The fact is we've not been actively maintaining Crewe. It's just like a beta container state feature. Where if we do have interested customers, we'd be happy to introduce your particular use case works. But live migration in Linux means you need to be able to sterilize all the container state of your workload, which is rather tricky. And so we can probably make it work for your particular use case if you really care about it. But making it work for everyone is something we just don't have enough people to get done, unfortunately. But yes, we do integrate with Crewe. If Crewe works, then you can live my great containers as they're running. If the container is running and you LXC move, it's going to sterilize states, move it, restore it. All right. I'm going to try and do the last one really, really quickly before it's James Dunn. On my local laptop, one thing that we can do is also device pass through. So I've got another container called Android. Let's see. In which I can do ADB devices. It's going to show me that I've got nothing. There we go. I can add a USB device to it. It will still show me that I've got nothing, because my phone is not plugged in. So now if I plug my phone in, I unlock my phone, and I run ADB devices again. Now my USB device has been passed into the container. And I can get an ADB shell. And we can see that the cell phone has been passed into the container. So we also do that for GPUs. You can pass the GPU into the container if you want. And the write-dev, NVIDIA, or dev DRI nodes will be automatically passed for you. All right. Back to the slides. So really briefly, before we go to Nova Lexity, just a quick recap. Lexity operates on system containers. So we run full Linux disk shows exactly like a virtual machine, but using containers on Linux. We have a really simple rest API to drive all of that. You could drive what I did through curl really easily. We do daily images of most Linux distributions. And Lexity itself is available on a whole bunch of disk shows. Obviously, main focus is Ubuntu. But it's available on Debian through a Snap. It's available in Arch Linux in show user repository. And we've got native packages in Gen2 and Alpine Linux, as well as someone's maintaining some packages in user repository for Fedora. That's it on my side. Thank you. All right. And for the demo, you just have to do this. OK. So Stefan's gone into machine containers and how we manage them with Lexity. So I'm going to talk about how we've integrated Lexity into OpenStack with a Nova driver. OK. So if we look at KVM, we've got a number of guests running here. We can pretty much simulate exactly the same thing with Lexity. Those guests are much lighter weight. There's no simulated firmware BIOS. It's running on the host kernel. And it's just resource controlled via Lexity and the various kernel mechanisms available to it. And we can slide OpenStack between those things to provide a management layer. So because Lexity container is a machine container, it has very similar semantics to a liver KVM virtual machine. So all the things you'd expect to do on an OpenStack cloud to a KVM instance, you can do pretty much all of them to a Lexity container running on a cloud as well. So the kind of normal lifecycle management for managing machines and also resizing, snapshotting, plugging into virtualized networking. So whatever SDN you're using on OpenStack, we can pretty much plug into most things these days. And we plug in using the same semantics as a liver KVM driver, but without the Verti OLA and the overhead that brings as well. We can also apply resource constraints to containers. So Stefan demonstrated that on the fly by reducing the number of processor and memories down. But we basically can take the flavor mechanism in OpenStack and we can apply that to containers that they're created. So memory, CPU, we can apply quads to IO operations. And in the same way, we can apply resource controls to the resources on the machine that containers are going to consume. We can also secure the machines with the standard security group mechanisms available in OpenStack. So either IP tables, firewalls, or native open V-switch firewalling work just fine with a Lexity container managed by our OpenStack as a liver container. And we can migrate containers between hosts. So Stefan demonstrated this at the Lexity level, but we can plug into the migrate API in OpenStack as well. And we can also do a kind of neat trick of exposing the entire hypervisor as what we call an exclusive machine container. So it gives you the whole machine, but with the container abstraction layer around the processes that the user is going to be consuming on the cloud. So they're getting the bare met performance, they're getting the benefit of being able to plug into whatever SDN is used on the cloud, but without the operational overhead of having to give someone a physical server and the complexity that brings in terms of compromising firmware, compromising BIOS, and the hygiene you'd have to do as a cloud operator to be able to manage hardware that you provide to tenants. Okay, so I'm not doing a live demo, unless they're at Windows 10. Go for it. And I prerecorded mine this morning because I was spinning chicken. Okay, so we've got a Nova Lexity deployment in one of our labs. So we're just going to SSH into that and take a look at it. Right, so the normal travel about getting access to your cloud. Let's take a look at the services we've got running. This is a three compute node cloud. We've got three hypervisors, and our control plane is actually in Lexity containers on top of those as well. I'm not going to talk about that today. So let's have a look at one of the hypervisors. You can see about halfway down on the second column, this is a Lexity hypervisor type rather than a CUMU hypervisor type, which is what you typically see with Libvert. We'll also have a look at the neutron agents running across the cloud, and you'll see that, although that's wrapped a little bit, we can see the open V-switch agents and the normal L3 and DHCP agents that we run on network nodes. So from a kind of topology perspective, the cloud looks very familiar to anybody who's ever deployed a Libvert KVM-based cloud. So I haven't got any instances running at the moment. So we'll boot a couple of instances, a small one and a large one. I'm too slow, obviously. Here we go. And they should be running pretty quickly, but I can't type fast enough, obviously. Why doesn't ASCII seminar trim the gaps? So those instances are booting and they should be both active from running. They're just plugged into a private network, which in this case is a GRE overlay network on the back end. So we're using the MR2 open V-switch plug-in for the networking on the cloud. So we need to add some floating IPs to be able to access these instances. So last uptake there is 161 for the small and should be 140 when we get there for the large instance. And now we should be able to access them. Oh, no, we won't. I haven't done the security group yet. So right now we can't ping the instance because it's completely far-walled off from the outside world. So we'll just have a look at the security group. So we've got the standard egress security group roles that all profiles come with in OpenStack. So we'll add ping and SSH ingress from the outside world into the default group and then we should be able to access the instances just fine. Okay, so we're getting a ping from it. So just check the other one as well. And let's log into one and have a look. So have a look at the CPU info. Just see we've got a... I can't see how many cores we've got there. It's just a small, it's got one. And it's got two gig of memory, which is off the top. Yeah, I got this one earlier. So no, it's still off the top. So it's got two gig of memory. And you can see the process listing there. It's much trimmed down as there's no kernel processes running, but all the normal things you'd expect in a full system like SSH and Syslog and Cron and all the other things you need to operate an instance are there. Can I ask you a question in the end? Because this won't wait for me, unfortunately. So we've got an IP address that's been presented by the Lexity layer and we can see the private IP address there that Neutron has allocated for the instance. And we'll have a similar check on the large instance as well. We might get to see more CPU information here, but let's see. No, because the number's off the top. But again, this is an 8 gig machine with four cores rather than the two gig machine with one core that the other one had. And similar process listing as well. So hopefully that gives you a flavor that, like using a Lexity machine container on an OpenStack cloud is a very similar experience to what you would expect with Livebert KVM. There are still some gaps. Storage is still not a strong point. We have basic support for the Cinderice SCSI driver and we're working on support for a SAF driver this cycle as well. So that should be available in the Eucarta release towards the end of this month. So we'll quickly talk about benchmarks. So Stefan alluded to the fact that you're getting very much bare metal performance. So we've got a few benchmarks to go through, just three. This is the Hadoop Terrasault benchmark. And this is using the exclusive machine stuff I was talking about earlier where we give a full machine to a container or a KVM instance. So we look to the total run time and bare metal and Novo Lexity come very, very close. I think the diff there is mainly due to the use of overlay networking and the overhead that brings in the networking layer. KVM took quite a bit longer. A number of failed jobs that had to be retried so that they did push things out. And I've got a few latency things as well. So Cassandra Stress has lots of small writers. So we can see right latency with Lexity compared to KVM is much, much lower. And again, that results in much, much higher throughput for Cassandra Stress benchmark. Okay. So I think we've got about three and a half minutes left now. So if anybody's got any questions, please put your hand up and I'll repeat them. So the question is, can you use Lexity to pin a container to a particular set of processes, sorry, cores on the host so it's doing CPU infinity. So Steph, do you want to answer that at the next level? Yes, we absolutely can. The CPU limits we set. If you just give us a street number of CPUs, we will do blood balancing for you. But if you give us a range, say 1 through 4 or 1, 3 to 4 or something, that way we will pin you on those particular cores. Yes. So the second part of that is that can you do it through OpenStack yet? Not yet, but we will support the full set of flavor properties that allow you to do that probably by the end of the pike cycle. So another six months yet. First question is, do you support container resize? Yes. So obviously Lexity, as I was showing, you can set everything live and that means that you can do resize pretty easily. Yes, and you can hot plug as well. So if you want to add additional ports to an instance, you can use the standard semantics via the OpenStack apis to be able to add additional neutral ports to a container. And that's done as a hot plug event and is picked up in the container appropriately. And the second question was about whether we support the OCI initiative. We don't because they mostly focus on application containers. They don't care about running a full distro. So it's not been a great fit in general. We've been for Lexity specifically. We've been looking at some part like the OCI SDN and some of those aspects that could be useful for us. But the image side of it, it's not a good fit because we care about a full pristine distro image coming from the vendor as opposed to having image source and people doing layers and stuff. Okay, there's one at the back and then you're next. Okay, so the question is the device passed through was that using IOMMU? No, it's not because we're using the same kernel as the host. So we're not passing... You can't pass a PCI device into a container just like you can't load a kernel module inside a container. What you do is the kernel module itself runs on the host. In this case, it was giving me death-burst USB devices and those were being passed into the container. Same thing, when you pass a GPU, the NVIDIA driver will be installed on the host, would get to the dev NVIDIA devices, then Lexity would pass whichever of those dev NVIDIA devices you need for the container to use the driver runs on the kernel. The same would apply to something like an SRIOV virtual function in that it presents as a host device and that can be mapped into the container. So the question is about the risk of DMA attacks against the host. If you do pass a device that gets you access to the entire host memory, yes, you can absolutely read dual memory. You have to be careful as to what you pass into the container. All right, thanks a lot. If anyone wants, we've got a bunch of stickers here that you can come and collect if you're collecting stickers. And thank you very much. Yeah, thank you.