 Hi. So welcome. I'm going to be talking about multi-unit OpenStack deployment using LexD containers on your laptop. So hopefully, you're all in the right place. Looks like a good turnout, so thank you for coming. OK, so who am I? My name's James Page. I work for Canonical, and I've been technical architect on the OpenStack team for the last 18 months or so. My background in OpenStack in Ubuntu is over the last six years. I remember it right from its infancy to where it's got today. I've been working in open source technology for about the last 15 years. Debbie and developer, Ubuntu Core Dev, an OpenStack contributor. And the OpenStack engineering team at Canonical are responsible for the packaging you all use on Ubuntu, the deployment tooling we have around that, which I will touch on today, and the testing and putting that all together as part of our release every six months. OK, so the first thing I'm going to talk about is OpenStack is hard. So such a plethora of different services, performing different functions to orchestrate an infrastructure-as-a-service cloud, it becomes very difficult to put together. Some of you may recognize this picture. It's been around for a while. It's the wiring diagram from hell. It's really, really hard to put together an OpenStack cloud. It's very easy to follow the upstream documentation if you've got the willpower to get through the many hours that may take you and do it by hand. But you only really want to do that once just to learn how it all plugs together. And then it becomes, you have the requirement to have a repeatable process to then start doing that. So it means that when you start deploying a cloud for the first time, you have a question as to how are we going to do this? How are we going to make sure that what we put down today, if we need to expand it, that works. If we need to redeploy it in a different location, that works. How do we make sure that all plugs together? So I'm going to take a little bit of time to talk about the service modeling tool we use at Canonical for putting down clouds. And I'm going to talk about containers, and then I'll talk about putting that all together. So you are in the right place still, don't worry. Gigi is the service modeling tool we've been working on at Canonical for probably about the last five years. It allows us to deploy services, scale them out, integrate them together. Relations between services are very easy to model. And we can use all those concepts and then put them down on different providers. So on bare metal, on top of clouds, such as OpenStack for things you want to deploy on top of it. Or in containers, which is the topic of this talk. So for the bare metal, this is how we put down OpenStack itself. We use a tool called MAS, Metal as a Service, and we can deploy using Gigi and the Charms, which encapsulate each component of OpenStack, an entire OpenStack cloud on hardware. We also combine containers into that to make sure your control plane is nicely segregated from the underlying physical resources and from each other. So Neutron and Nova and Cinder are all in different containers within that deployment. But I'm not going to talk about that too much today because I want to get on to the stuff I've been working on to make this all happen on your laptop. So this is what an OpenStack model looks like in Gigi. All the various different components, dashboard, database, messaging. I've got some Seth in there as well for block storage. We can apply things like NTP on top of that to make sure we have consistent time, that sort of thing. And all the relations between them, we can encapsulate this in a model and then we can take that and deploy it onto hardware. But how do we condense a cloud? How do we take all those things that we find great and use every day to deploy on our server hardware? How do we take that and do it on a laptop? How do we take all those modeling concepts that we like and test them on a regular basis? So that's what I'm going to talk about next. So I'm going to dive in and talk about LexD for a little bit, which is the underlying container technology that I've used to put this together. So LexD is a pure container hypervisor. It gives us the ability to do very dense and very performant full system containers for Linux guests. So don't try and use Windows in a LexD container. It doesn't work. You're getting direct access to the Linux kernel that the host is running and it's a containerized set of processes on top of that. Dig into that a little bit of how that's different to an application container in a second. So LexD is a project that's been around for about 12 months, maybe a little bit over that. I think we named it in Paris and it's been in development ever since then. LexD is a kind of ground-up redesign around the user experience for managing Linux system containers. So you may have heard of the word LexD before and that was the group of technologies to manage Linux system containers on Ubuntu and Red Hat and all of the Linux disk drives have really adopted that. But LexD was an opportunity to kind of reflect on the work that had been done in LexD and think about how to design the command line UI, how to put a REST API on top of this to make it very consumable both for people and for other tools. So things like Juju, OpenStack, that sort of stuff. So it's got a lovely simple REST API. It's designed from the ground up to be very, very simple to use. It's designed to be fast. We all expect that a container spin-up is fast. So LexD is fast. We haven't reinvented the wheel in terms of storage backends and stuff like that. So we're leveraging a lot of other mature technologies. So a lot of the overhead of creating container is not starting the process. This is actually creating the file system image that you're gonna run underneath the container. So we've leveraged fast storage backends with copy and write features such as ZFS, the LVM and ButterFS to make that spin-up of a container very, very fast. So we essentially goal copy the image and then each container then uses that image. Starts up pretty quick and I'll demo that shortly. LexD designed to be secure. So we took that approach. LexD containers have developed the required security features over the last six years to make it a real story. So a LexD managed container is by default an unprivileged container. So that means the processes in it are not running as root on the host OS. So they're running as a sub UID range of an unprivileged user. That means if you did have a container breakout, all that breakout exposes on the host is an unprivileged user. So yeah, it's still a problem and we don't want that to happen. So we also wrap our containers in app armor which means that the processes within the container only have access to certain paths on the host OS. They can only perform certain operations that we allow them to configure. So network interfaces, parts of the file system, that sort of thing are all restricted in that way. LexD self, in terms of its API, we use TLS encryption through and through on the network connectivity to the rest API with client certificates for authentication. So again, we just used a whole load of an existing technologies to secure the product well. LexD, if you're interested, is written in Go. We've been heavy users of Go since we took the decision to make, right, juju in Go about three years ago and it's worked very well for this project as well. So what is LexD not? LexD is not about application containers. We're not trying to compete here with Docker or Rocket which are very much about containerizing a single process using those same base Linux technologies and managing those at scale. On the other end of that, we've got all of the hypervisors we know well. So Hyper-V KVM and this is where LexD plays. It's about full system containers, machines you can log into. They've got SSH, they've got Syslog, they've got CRON. They've got all the things you would expect from a server install or a KVM instance. So you can use all your existing tooling to manage them but you've got all the benefits of container. So you're next to the metal, you're getting bare metal level performance. You've got high density on that as well because the overhead of each container is very, very minimal. Okay, so I'll just give you a quick demo of LexD itself. Let's see how I get the right monitor going. One second, I'm having a multi-display disaster now. We should get there in a second. Okay, let me just increase the font on that so you can actually see it a bit. So this is the LexD command line API. So this is interacting with the demon via local unit socket using the rest API but you can equally manage remote LexD demon in the same way using the TLS transport. So I'm gonna show you that I've got a few images loaded in here so I've got a trusty and a zennial image. So LexD provides a number of ways to get images in. LexD image is typically a root table. It's not a Q-Cal too. So it's a different image format to what you'd be familiar with with KVM. On this system, and this is all on my laptop, I've got ZFS configured on half the SSD. So ZFS is a feature of our 16.04 release because you very fast perform and copy on right clone. So it makes a perfect partner to LexD in terms of how we manage the underlying storage for containers. You can see I've got a load running. I pre-deployed my open stack. It was not enough network in the room to do it but we'll get into that in a minute. So if I wanna launch a container, it's stable. So ButterFS had a lot of the same features but I think a lot of people have been burnt by ButterFS and its stability and production grade redness seems to be variable in people's opinions. ZFS has a slightly different semantics. So it's more of a combined LVM and storage fast system. Can I actually just part the question till the end and then so we can get through the content first. Thank you. So I'm gonna launch a Zennial container. It should take about two and a half seconds and I'm just gonna hop into that so you can see. The exact command is allows the someone on the local or remote system to get a shell directly into that container. Again, it's secured via the authentication that the client's providing. So you can drop in and you can see instantly that we've got in it. This is actually a system-based container so we can do the normal operations you'd expect so we can see SSH is running. We can see the operate within the container in a normal view. So that's like Steve very quickly. It's very easy to install if you wanna try it. 16.04 works out the box. Okay, wire project is never easy. Okay, so what I'm gonna talk about next is how we've plugged Juju together with LexD in our 2.0 release of Juju to allow you to deploy Juju services on containers on your laptop. So this is a feature we've actually had for a while. It used to use LexD, the base primitive rather than the demon on top and it allows you to deploy those Juju models and charms on into LexD containers on your laptop. So as I said before, this is all completely functional in 16.04. You can run it on 14.04 but there's a lot more hoops to jump through in terms of kernel versions and back ports and stuff like that. So if you wanna try it, I tried on 16.04. As I said, I'm using ZFS as my storage backend. You also have LVM and ButterFS and if you don't have a dedicated block device to do that on, you can do it using a sparse file directly managed by ZFS on top of your 64 file system if that's on your route. That works fine. It's pretty performant and gives people an instant bootstrap. I'm running some of my laptop. I can run a whole cloud on an i5 with 16 gig of RAM and about 200 gig SSD entitled. It doesn't need much more than about 50 for the actual cloud. So, we did dive on a couple of little features that we've used to make this. So as I said that LexD containers by default are secure, they're unprivileged. They have limited access to devices on the host but mostly also has the capability to expose more things to a particular container. It does that for a thing called a profile. So that can map devices from the host directly into the container. It can give it additional security permissions like being able to do nested network namespaces, that sort of thing and that's then controlled by our app armor in terms of the permissions it has. We can give it extra network interfaces so we can map a container into multiple bridges on the host so that we can have multiple network ports in our underlying containers. And that's hotplugable as well. So you can apply that to a running container and get additional network interfaces. And we'll drop into that. I said I don't wanna risk the monitor again. Once we get into the demo, I'll show you a profile. And we've got a particular bundle. So a bundle is a group of charms, the configuration of them and the relations between them. It's up on GitHub and the URL at the end of the presentation if you wanna go and take a look and some good documentation there as well. But a bundle is, we have an open stack on XD bundle which has got all the right config knobs tuned to allow you to do this. Okay, so let's have a look at the cloud I've got running. Okay, so just start with Juju. So this is the command line interface to Juju. Juju has a GUI which is, as of 2.0, is baked into the core product. So as you bootstrap an environment, you can access the GUI. It also has a command line interface. So for those of you who like to live on the CLI, rather than for a web UI, this is perfect. Or in fact, it exposes more of the features of Juju than the GUI does. The GUI is designed to be an abstraction on top of that with an ICN. I will show you that in a minute. But this is the open stack deployment we've got. Now I've trimmed down the number of services purely because I wanted to squeeze it in on this laptop. So we've got the core infrastructure services, Cinder, Glance, Keystone, Neutron, and Nova, and the dashboard as well. I haven't put Swift in. Requires quite a number of containers and was just a little bit more than my laptop could manage, but it is possible to do that. So we're using a Ceph storage backend, which is managed purely through the file system, so no requirement for block devices. And we're using the Radius Gateway just to provide some basic object storage services on top of that instead. We've also got MySQL and RabbitMQ deploy to provide all the core infrastructure services for all those components. So if we look at the actual cloud itself, typing and looking is hard, isn't it? So we've actually just got a single compute node so we can query the kind of status of the agents and stuff like that. And you can see the Neutron agent. So we're running a network node as well to provide the North-South traffic routing. So we're able to fully use overlay networks as part of the container story as well. So this is actually running GRE, but VxLan works fine. And then we have what we call our Neutron gateway node, which is a network node, which then provides a North-South traffic routing and floating IP access to instances. So I have to admit I've already preloaded a image and configured a couple of networks. So we've got the underlying external network which maps onto the IP configuration of the bridge on the host. And that's shared with the DHCP range for the lexi containers itself, although that could be a separate bridge if you wanted to do it that way. And I've also got an internal GRE network setup as well. So let me try and find that one as well. So what I said is that there's a GUI view of juju as well. This is the same view of the model that I got running on my laptop, but viewed through the GUI instead. So you can, again, you can see the various different components here, RabbitMQ, we've got the Sephrados gateway and Sep itself and then all of the Cinder components. So these are running in its own container and juju provides the relation semantics for those things to communicate with each other during configuration, set up things like user names, passwords, that sort of stuff. So I've been able to use exactly what we do for physical hardware to deploy that onto my laptop. So just to prove this isn't all neat trick and it doesn't actually work, we'll log on to the dashboard. So if we have a quick look at the hypervisors first, we'll see that there is an acutely hypervisor. So this is going to run a KVM instance and we'll hop in and launch an instance. Let's get rid of that. So I'm going to boot it from the Xenial image that we saw on the glance image output a minute ago, just using the FAML storage root. Give it, make it an M1 small, add it an internal network connection. We don't worry about network groups or security groups in moments. I've got to keep our loader as well just so we can access it afterwards. And that should be enough to give us a running instance in a container on my laptop. I think it's crossed. It worked 30 minutes ago, it should work now. So the way we've been able to do this is we have Livevert running inside the container. We've exposed the host devices that Livevert and OpenV switch need to be able to communicate with the kernel via a profile into the container. So, slashdevkvm slash devtunnet and slashdevmem are all things that those particular things need to use to be able to manage either the networking or the state of the instance directly. And we've got a running instance. Just check the console log, make sure we've got some networking. I don't know whether you can even see that. There we go. The instance spun up, it's got an IP address from the internal network range bar from the DHCP services running on the network node, which is on a different unit container to the one running the load compute instance and it's fully booted. So, I will just drop onto the command line again and we're just gonna have a look at exactly what that looks like under the hood. So, if you've ever logged on to an OpenStack hypervisor, you probably run this command. So you can see here on the internal bridge we've got the connection to the KVM instance and we've got on the internal bridge, we've got the tunnel link out to the network node to provide north-south traffic routing and DHCP services, that's metadata services as well. And you will see exactly the same view, so I can, same familiar tools, so if you wanna look at the actual instance, I can see there's a running instance from inside the container. And we'll now try and access that. So, I'll flip back to dashboard for a second. I'm gonna give it a floating IP because it's just on a private network at the moment. Okay, so we now have a floating IP associated with it, so that's 10, 20, 100.1. Ah, I remember, I didn't add any security rules to actually access that. So, let's just go and do that now. To show you that all the features you get on a normal hypervisor, you do get in containers as well. So, we'll add some SSH connectivity. I've already used this IP before, one second. Okay, so this is the KVM instance that we got running on the cloud. So, demonstrated we were able to deploy it in containers, we can spin up instances, we can network them effectively, we can do overlays, we can do north-south traffic routing, we can apply security group rules, all the things you'd expect to be able to do on hardware. I've been able to model accurately in containers on my laptop. You can also create block devices in Cinder, we can present those to the instances and simulate storage in exactly the same way. Okay, so if you're deploying an OpenStack Cloud, you'll probably have testing and production environments at a server-based, because you wanna be able to validate changes you're making into the environment, changes you're making into deployment tools, OpenStack upgrades, the number of things that you need to be able to verify. So, where does this type of concept fit in nicely? Well, it starts to allow you to make the laptop your first touch point on a change into your environment. So, you can take that same bundle, you can redux it from your laptop, you can try a different configuration option, you can try a different topology, you can add some new components, you can verify that on your laptop. It takes about 15 minutes to stack a cloud this way. So, the teardown cycle's pretty fast, if you need to change that, iterate that. And then you can then take that forward into testing on servers where your deployment time's probably gonna be longer because servers take a long time to boot for a container, there's no post, nothing, no kernel to even boot. So, the container bring up time is very, very fast. So, allows you to bring your laptop in on the front of the changes you're making into your environment. So, for people who are first touching on OpenStack, means they don't have to find five servers to try this stuff on to start off with. They can try this on their laptop today, just using the tools that Ubuntu 16.04 is shipping with. So, I demonstrated KVM cloud, we also in 16.04 have a driver for managing instances on your cloud using the LexD hypervisor as well, Nova LexD, that's not entry in Nova yet, that's a project we've been running under the LexD projects as part of the activity around LexD. But it does allow you to use a container in your cloud as well. So, alongside KVM you can have LexD containers with consistent networking and all the things you need to be able to mix those two things together. You can also run LexD in LexD. So, we can then nest that and put that all together and try it on your laptop as well. So, in the Git repo we've got, we've got a bundle that does LexD rather than KVM. So, if you wanna try that out, you can try that out as well. Another project we've been working on is a tool called ConjureUp. Used to be called the OpenStack installer which was really boring and very specific. It actually kinda turned out to be a bit more generic as a tool for taking deployment bundles, using Juju to then deploy those both in containers, in a single machine or using Mars onto multiple machines. So, in 1604 there's a tool called ConjureUp and you can ConjureUp OpenStack and that will allow you to try this. It will also allow you if you have Mars set up and some spare servers to deploy an OpenStack cloud a little bit more easily onto that as well. I think you can also ConjureUp big data if you like that sort of thing. So, it's designed as a generic framework for taking bundles and then driving placement of those across either containers for testing or for onto physical server infrastructure as you take that into more rigorous testing and into production. Okay, so we've got about 10 minutes left. So, if anyone's got any questions, please use the microphones and then they get on the recording as well. Hi, talked about the pros. I wanna know what's the cons of running OpenStack components in LXD, is there any? Sorry? Is there any con for that? Like, what's the disadvantages of that? Well, you probably wouldn't run necessarily this in everything under containers in your production environment. So, compute nodes are typically gonna go onto hardware rather than into containers on hardware. So, this is a great way of not trashing your laptop by installing OpenStack on it. And, as a result, it gives you a nice encapsulation of the thing you're trying to test rather than you having to do that all on the thing you're actually sitting on. And you can throw this away and you've got no more state left of what you just tested. So, that's the pro. Cons, I don't know, for the context, I think it's pretty cool, so. I have two questions. One, you showed kind of putting your laptop in front of your dev process and making out the first stop for changes. Can you, I assume, you're deployed everything from move on to packages? Is it possible to incorporate this into a developer workflow where you're testing patches from source? Yeah, so, this is deployed from packages. So, it's the result of development rather than part of development. The charms for deploying OpenStack on Juju do support deployment from source as well. So, we have the ability to take OpenStack projects and instead of using our binary packages, pull those in and deploy from Git repositories with appropriate requirements and pip install dependencies and stuff like that. So, we're kind of thinking along those lines yet. There's some features that I'd really like to see land in an XD before we can make that reality. I'd really love to be able to bind mount the Git repos I'm working on locally directly into my containers for my branches and install from that and then have a nice iteration cycle where I make a change, rebase, whatever it might be, reload my deployed containers and retest everything. So, the answer is, yeah, and that's kind of where we'd like to go but we're not quite there yet, so. And then second question, you mentioned the profiles. Yeah. So, you have a special profile from over at Compute so it can talk to the kernel. Is that described in the charm metadata somehow or is that some manual thing that you do after you've deployed the containers? For this demo, I pre-created the profile that Juju was gonna use because I know what its name's gonna be because it's Juju default. It's the standard model that Juju 2.0 creates. We're talking with the Juju team at the moment as to how you might express a profile as part of the charm to allow you to do that in a more automated way. And then that can then be applied on a service by service level rather than at the entire model level which is where we are today, so. So, going back to the LUXD within LUXD, there's been talk about the Nova LUXD driver, essentially. Has that been implemented with all the Juju and kind of an integrated environment? Yeah, yeah. So, the Nova Compute charm, which is a principle charm for managing the hypervisor, supports LexD as of this release of the charms which came out last Thursday. We have another charm which is called LexD which does all the underlying management of storage configuration and the actual underlying LexD daemon as part of that tool. So, it's basically you add two charms to the bundle and you can do both in one. So, kind of leading into this is where I'm really actually interested is using LUXD as a test bed. I would like to be able to test basically a multi-node open stack environment that is mixed mode KBM and LUXD hypervisors. And how it works. It's entirely possible, yeah, so. So, is that basically us using hypervisor types and delegating some computes as KBM and other. Exactly right. So, by tagging up the images with appropriate hypervisor type metadata, the image properties filter will then schedule those onto the right underlying hypervisor. So, determining the hypervisors based on the image type and not by some other. Yeah, I mean we're fortunate that LexD and KBM use quite different image types. So, it's a very clear boundary there. I think it's also possible to do using different flavors with additional extra specs as well. So, there's kind of two roots there. But the combination of just picking an image and picking a flavor means that the flavors are consistent across any underlying hypervisor which makes that a quite a neat story. I actually talked about this yesterday. So, check that one out as well. Okay, looks like there's no more questions. If anybody wants to talk to me afterwards, I'll be around for the next 10 minutes or so and thank you for listening. Thank you.