 So this session in Cloud Open is going to be about containers in the cloud and why do you need another virtual environment? I have to say this talk was put together at very short notice. I did theoretically get four days' notice, but when I heard on Sunday, I already planned to go on a hiking trip on Monday that you probably haven't heard about where we tried to kill Linus. And then I had to stay on the west of Scotland on Monday night and Tuesday just because one of our cars broke down and had to be towed that we'd hired. So I actually really got, and I left my laptop here so I couldn't do it then. So I really only had today to write this in. So here it is. So let's begin looking at containers from the history of virtualization. So virtualization has always been about hypervisors from the old days of VMware bringing it to the masses to almost everybody else's implementation. Virtualization was so much hyped by the year 2005, the competing virtualization implementations, which were basically Zen and, well, really only Zen, it was Zen and the Linux kernel, forced the distributions actually to fork the kernel itself in order to accommodate Zen. Unfortunately, KVM was, I think it was a gleam in Abby's eye, the guy you just been listening to. But it wasn't fully formed and it wasn't submitted upstream by that stage. The consequences of this fork have remained with us till today. Trying to get the Zen fork out of the distro's kernel has been almost a decade-long effort for the distributions. And it's convinced them that putting stuff in that's not upstream really, really is the wrong thing to do. And from my point of view, trying to preach the wonders of containers to a very hypervisor religious audience, it means that I'm sort of equivalent of this tiny little church down here and hypervisors are these sort of dominant big mainstream religion over there. So it makes my job much harder because hypervisor-based virtualization technologies really blinkered everybody to the possibility that you could get what you need from a different system that did virtualization in a different way. And basically, the statement today is still that if it's not a hypervisor, the enterprise is not interested. To be honest, if it's not VMware, the enterprise is not interested that much. But if you think about exploring other virtualization choices, the hosting market, which is the one my company Parallel serves, turned to containers very early on. And the reason was simple. It's cost. If you're a hoster and you're selling virtual environments for, say, $10 per month, and you try and use a hypervisor to do that, and your equipment costs rack up, and your hypervisor costs rack up, and you can get, say, 10 instances of your VPS that you've sold, virtual private server, on this one system. So you're selling it to 10 customers, you're raking in $100 a month, and it costs you $150 just for all of your software and hardware costs. You go out of business very rapidly. Whereas if you can do the same thing with containers and pack 30 instead of 10, so that you're three times as dense, which is roughly how dense containers are, you're raking in $300, you're spending $150, you're in profit, you like it. That's almost the sole reason why containers were adopted by service providers, it was all about density. Containers could get virtual private servers to a level that was dense enough that the price point you're offering to your customers made sense from an economics of what is your margin. That meant that companies like Parallels, we evolved specifically to serve the service provider market. We didn't have any interest in the enterprise or any interest in everything else. But as the hosting market evolved and wanted to offer more services, container technology became more sophisticated along with it and more dense, because remember, they care about the density. If I can offer them 310 virtual machines on one instead, sorry, 32 virtual machines instead of 30 on one, that's another $20 extra profit that they pocket. So they've been pushing us, it's all been all about the density for us for at least as long as I can remember. But the other thing that actually happened a long time ago in Linux is the container technology actually split. So the Parallels container technology called OpenVZ has been open source for a long time, but it was based on things called bean counters. And while we were trying to work out how to get bean counters upstream, a guy called Paul Minaj came along and proposed this thing called C Groups. C Groups are actually a much more sophisticated version of bean counters because the bean counter was basically just the control that sits inside a C Group. So C Groups, namespaces and the other technologies that came along actually added more sophistication to the kernel. And Google very, very early on saw the advantage of containers. They actually hired almost everybody who worked in C Groups. They went about completely containerizing their main data center with C Groups. So anything you do on Google today is done in the container, whether it's search, Gmail, Google Docs, you get your own individual container that actually serves up the web experience to you. And historically, these two types of containers, the C Groups containers and the OpenVZ containers, evolved on different tracks and they actually evolved out of tree. And as you can see from this, we're setting ourselves up for another Zen KVM war just because we have these two separate technologies. However, since we had the benefit of hindsight and we saw what happened in 2005 with forking the kernel and the massive argument over which hypervisor, a long time ago, well actually not so long, a long time ago in terms of the lifetime of this, but realistically only three years ago now at the kernel summit in Prague, we all sat down together and agreed exactly what we were going to be doing with the containers technology. And we basically, we did a show and tell swap. So we said, we did abandon bean counters and we used C Groups and Google wanted to use pieces of our technology. So we did a show and tell swap of technology and the result is that the OpenVZ and the Google kernel technologies at core are almost identical nowadays. And so we agreed to standardize on one identical set of technology in the kernel, push it all upstream. As of 3.9, it's pretty much all upstream. So if you can read that, I'm afraid for some reason this projector didn't work at 1024 by 768. You can only see the bottom of my slides if I downshift it to 800 by 600. So this diagram is somewhat greeked, but this shows you what the difference between hypervisors on the right and containers on the left are. So if you look at the hypervisor diagram, deliberately it's just more busy just because there are more moving parts in a hypervisor than there are in a container. In a hypervisor, you've effectively got one operating system, one kernel running at the base, then you have a virtual hardware emulator, then you have all your different kernels running in your virtual environments, that's the guest OS, and then finally you have whatever application you want to run above it. With containers, you're actually virtualizing at the level of the kernel. That means a containerized system, everything that runs within a container shares the same kernel. This can be an advantage, it's certainly an advantage in terms of density. It's the more natural level to virtualize at. The whole reason containers are more dense than hypervisors is mainly because a single kernel runs everything, a single kernel manages the resources, and that means it knows exactly what's going on with those resources. If you think of the tricks, a hypervisor has to pull to try and work out what the guest operating system, what the kernel sitting up there is doing with its resources. It's actually astonishing that they work at all. All communication with a hypervisor really has to be done at the machine level. Anything that a PC can do, a hypervisor can do because it emulates the hardware. So any communication that goes between a kernel and the hardware, it's easy for a hypervisor to virtualize. But any communication a hypervisor needs to know, which is not communicated at the hardware level, things like how much memory you're actually using, you have to evolve all these really weird out of band communication interfaces to actually tell you. So the scheduler in the system, which is actually the hypervisor, makes the correct decision for running your virtual machine. And obviously with the container, since it's a standard kernel, the kernel manages all the memory, it's used to managing the memory, it sees all the memory that all of the guest containers are using. It's much easier for it to make resource scheduling decisions. So the key point here is that hypervisors are an abstraction at the hardware level. Anything you can do with hardware, you do with a hypervisor. Some things are very easy to do with hardware. Some things are incredibly hard. If you remember sort of for the last five years, we've had this argument about memory hot plug. Hot adding memory to a system is very easy. Almost every sort of sophisticated server hardware on the market can do that. So we can hot add memory to a hypervisor, simple. Hot removing memory again is incredibly complicated. The reason for this is that most banks of memory operate in banks of say gigabyte or a multiple gigabyte or a multiple terabyte. And the Linux kernel has a very hard time sort of shifting a gigabyte so that you can actually have this range of memory free in physical terms as far as the guest thinks of it. So you can then hot unplug that memory and free it back up to the hardware or in this case to the hypervisor. This is why instead of actually doing memory hot unplug, hypervisors do this weird thing called ballooning where you actually have a cooperation of the operating system to inflate a balloon inside the guest operating system. Once that memory balloon is inflated, all memory inside the balloon, you can quietly free up and pass back to the hypervisor. This is an out of band communication mechanism for doing that. Containers of course just don't need to do that. It's one kernel that sees it all. We can free, once we free a page in one container, that page is free to be reused by another container because a single kernel is managing the memory in the system. So the point is that hypervisors are an abstraction at the hardware level, containers are an abstraction at the operating system kernel level. That means that hypervisors can do anything hardware can do, containers can do anything an operating system can do and an operating system has much more power and flexibility. The key thing that containers cannot do that hypervisors can do is run different kernels because there has to be a single kernel. So there's no way in a containerized system that you can boot up Linux in one container and Windows in another. That's never, theoretically, it could be done. By the time we've done it, we'll already have got ourselves to being effectively another hypervisor so there is no point doing it. And the other key difference of course is that containers run a single kernel, hypervisors always run two because there's one kernel running in the host, one kernel running in the guest. And it's actually most of the friction over these two kernels fighting each other over resources is why hypervisors run into so much difficulty. So I told you that density was the key as far as the hosting providers were concerned. So why are containers denser? And partly it's just the simple mathematics of resource saving because a single kernel manages all of the resources. It sees everything that's going on. Any page that's freed from one container is immediately available for reuse in another. I don't have to pull any of these ballooning tricks. It means that memory can be much more evenly shared out in the system without actually bothering to try and run something like KSM, the same page merging or trying to get the hardware to work out if these pages should have the same contents and so you can merge them and so on. The page cache itself, which is the engine Linux users to share pages sits inside the kernel and that means it sits below the container abstraction. It's actually very easy for us to share even the same pages belong to the same file between different containers. But it's also partly to do with better management. Single host sees everything that goes, single kernel sees everything that goes on. It's very hard for the hardware to work out just looking up from the hardware level what is actually going on inside the operating system. That's why hypervisors invent all these weird ballooning paradigms and KSM paradigms and non-existent performance counters to communicate exactly what's going on with the operating system. So mathematically, containers are denser just because the abstraction level is higher and that just makes it easier to do better resource management of that abstraction level. So here's an actual proof of what I'm telling you. This is a fairly weak Xeon system and all we've done is we're booting up virtual environments running a web server and we hammer that web server with requests and we measure the time it takes, the response of that web server to go above four seconds. So response time is going up here. Number of actual web server instances are running along the X axis. As you can see that with Xeon, it gets to about 29 before it starts topping out. KVM does a little better, it gets to 37. The reason for the difference between Xeon and KVM is that the KVM paradigm is that the hypervisor is the kernel, the Xeon paradigm is that DOM zero which technically runs as something separate as the kernel. So just Xeon itself has more difficulty communicating than KVM does. But if you look at even the open source open VZ containers, they're already up to nearly 60. That's double what Xeon is and nearly double what KVM is. And if you look at the sort of state of the art which is Parallels Cloud Server where we've tweaked it and tuned it and beaten it into submission, we can actually get 121 virtual environments serving pages before this thing starts tanking and going above four seconds response time. That there, the difference between sort of 37 and 121 is three times as much. That's three times the density we can get. So that's proof that if you're a service provider serving up virtual environments charging a 10 or a go for them, you're profitable on containers and you're just not profitable on hypervisors. So other container advantages. Originally, this was something we didn't care about in Parallels at all until the magic word cloud came along. So it was always accidental to us that containers would boot up much faster. I mean, it helped that service providers could migrate virtual environments faster with containers but by and large, they saw this as just something that they liked rather than something they pay good money for. So we didn't really care about it. It just turns out that containers are far more elastic than hypervisors. And the reason it's very simple. I mean, if you just look at hypervisors back on this slide over here, there's much more to get started on a hypervisor because you've got to get the hypervisor is booted. You can probably take it for red that was booted but by the time you started a virtual hardware level, you have to boot a kernel on that hardware. Booting a kernel actually takes a long time nowadays even on virtual hardware. Then you had to start the operating system from a net and then you were ready to run your applications. In the container case, to start a third container, the kernel is already booted and you can actually start either with a net to boot or if you're really clever, you've got most of your net stuff already done. You can just start with the application. So this means that applications inside containers can start just like that. I can pull up a new container if I'm starting just a single application sometimes in sub-second times. Hypervisors, you've got to boot the kernel, get the net system done, get the operating system up, then get your application running. You're lucky if you can do that in a minute. So containers just for orchestration end up being a lot faster than hypervisors. Migration is also a lot faster with hypervisors just because there's a lot less of the image to migrate. The container node, the kernel running the container knows exactly how much memory this container is using because that's what kernels know. I know every page that container is using. I can look at every page and just transfer exactly the amount of memory I'm using. I don't have to transfer all of the operating system image because it's a kernel and it's standard, it's shared between everything else. So the incremental image that I have to transfer to get that container to go from node A to node B is a lot smaller with containers than it actually is with hypervisors. The other thing that makes them really interesting for clouds is thing called vertical scaling. Vertical scaling is where somebody pays you for a certain amount of resources to use in the cloud and you might let them use more resources and of course charge them for the privilege of using more resources but if somebody else wants to use all the resources in your cloud as well, you would actually scale them back to what their guaranteed resource limits are. This is this thing called over commit. It means that I have a machine and I split it up between effectively 10 users and I promise I guarantee to each of those users one tenth of the machine. But because I'm a cloud and I'm charging elastically, if one of those users wants to use a fifth of the machine and it just so happens that none of the other users is doing anything, I'm perfectly happy to give you that fifth of the machine, charge you twice as much as I was charging for a tenth of the machine and oh by the way, I'm still charging each of these other people who just don't happen to be using their resources for a tenth of the machine as well. And so effectively I'm charging for 11 machines on a 10 machine system, this is called over commit. That in the cloud is the way most people make a lot of money. Of course it relies on the fact that people pay you for resources and don't use them but by and large users do this for no readily apparent reason. Every user will buy a certain set of resources on the cloud but most of the time those resources are idle and so over commit is all about charging for the same resources, multiple people, twice, three times, four times. The more people I can charge for the same resource, the more money I make in the cloud. Containers can do this very well because of vertical scaling. The trick to vertical scaling is you can't afford to get caught selling your resources multiple times. If you do, your customers leave you. If you don't get caught, they don't leave you. So what has to happen is that if I'm reselling the resources in this cloud, what I'm reselling you is not guaranteed. So I say, okay, I might give you half the CPU resources on the system. Your guaranteed minimum is a 10th but you might be able to scale up to a half if you're lucky. If you scale up that way, I charge you for it. If you don't scale up that way, I say I'm terribly sorry but I give you your guarantees anyway. And this means that if I've got some job hog using half the system and then suddenly by happenstance all these other nine containers happen to fire up and want to use their 10th of the resources, I obviously only have one sort of set of computing resources to give them. I have to very quickly scale down the guy who's using a half back to his 10th and I need to be able to do that in sub second timings. Otherwise, the guys who are trying to scale up notice that I've been reusing their resources. If I can do it very fast and they don't notice, I get away with the over commit. If I have to inflate a balloon or do something else in a hypervisor that shows them that I've resold their resources they get annoyed about it. So containers are just much more naturally the engine you use in the cloud for cheating your customers effectively. I didn't tell you that by the way. Like I said, the correct, it's what? They're recording your talk. Oh, they're recording my talk. Okay, so over commit is all about making more money out of the cloud than you could with hypervisors. But over commit is something that containers are very good at just because they can scale up and scale down far faster than hypervisors can. So the chances of you being caught out overselling resources is a lot smaller with containers. Some reason this chart shows up in the preview and then it goes wrong in anyway. Believe me, the fact that we did this test and it doesn't look like it on this chart but containers in a test where we're booting up from a knit not booting up from the application. So this is trying to treat a container as a virtual environment with its own operating system and it sort of looks like two different versions of Linux. Booting from a knit, we can get containers to respond seven times faster than hypervisors. And obviously we picked the hypervisor with the crappiest virtual hardware implementation so it took the kernel longest to boot in it but seven times is a good rule of thumb. If I was talking about application containers, I'd probably be saying somewhere more like a hundred times faster. All these figures take with a pinch of salt because it depends critically on how long it takes to get through the kernel and everything. But just as a rule of thumb, if hypervisors can do it, containers can do it at least 10 times faster. Application containers can do it almost 100 times faster. And all of this is tailor made for the cloud where elasticity used to be the byword for the cloud. Now we started to run out of space in data centers. Even density is starting to match the environment. Remember, density is where we started making containers from. Density or why containers exist. Elasticity happens to be a side effect but both of these properties are suddenly big business for the cloud. And the major disadvantage of containers which is you can't run different kernels doesn't seem to matter in the cloud. Most clouds are homogenous. So the big disadvantage of containers is gone and the great advantages are still there and this makes containers the ideal technology for the cloud. But I mean obviously there's more. The main cloud problem with pass and sass is the thing called tenancy which is how do I run an application so I can have multiple users using this application in the cloud and yet both be all being able to administer this application. One of the classic uses of hypervisors was Microsoft Exchange. Microsoft Exchange, so try and explain this to UNIX people. Microsoft Exchange just would not run two copies of Exchange on the same physical hardware. It was not possible before we had hypervisor based technology just because of the way Exchange worked. And so if your business model was selling Exchange to companies, you required one physical system per Exchange instance you resold. This meant that your Exchange ended up being a much more expensive mail service than PostFix because you could resell PostFix mailboxes a lot more easily on UNIX and you didn't have to have this one computer per PostFix set of customers. Although if you sold the same PostFix system on the same physical hardware to multiple users, you did have an administration problem because PostFix itself tends to only think in terms of one administrator. But you got away, you worked around that just by not selling the administration interface. But the point here was that virtualization got us out of that because now you could have multiple copies of Exchange running on the same physical system. That was the true advantage of hypervisors for that business model. What this is doing is in effect giving you a multi-tenant system because each of those virtual machines running Exchange can be administered separately. However, if you look at the way Gmail works, it runs the single Gmail application on the physical system. And then if you buy a business, if you buy an individual user copy of Gmail, you just get that Gmail instance like the old PostFix one, Google administers it for you, you don't have a separate administration capability. If you buy business Gmail, you actually get the postmaster address and you get the administrator address and business Gmail. And effectively what happens is that all of your users are corralled inside a single container. It's still running the single Gmail application at base, but all your users are containerized. And this means that if company A has its secrets on Gmail server and company B has its secrets on the Gmail server, because they're usually mounted to different file systems, they can't get access to each other's secrets, which is what you want. But it also means that if company A starts performing lots of data mining on this database that they're using, and so does company B, company C doesn't get hammered because all the CPU has suddenly sucked on this resource to both these Gmail instances, because they actually have a CPU C group ensuring fairness of processing resources among this. So even without doing what we did for Exchange, which was adding a virtual machine at base to ensure fairness, you can get fairness among these tenants of the Gmail system. This effectively is what containers have been doing for years. That's why Google uses them. We can use the same kernel, we use the same operating system base, we even use the same application, but we use containers to ensure that the users of that application get the SLA guarantees. This is effectively what it looks like. These are application containers. So if you look, we have the kernel, we have a user space, this is all the init subsystem, we even share all the libraries. The only thing we put into the containers, okay, so this is a web server example, we've got Apache and MySQL sitting inside a container, then I could run multiple copies of those in different containers right on the top. Containers can virtualize any group of processors. They don't have to be a group of processors that happens to correspond to a single IS operating, infrastructure as a service operating system. It can be as simple as a single process, single application. If you want Apache and MySQL, it can be a set of processors that actually do this. But the point here is that the containerization of a multi-tenant system looks like this. If I have to do the same thing with a hypervisor, it looks like this. I have to get everything inside the hypervisor. And I mean, it's just that a common sense that this is much more agile than that. And that's what makes containers much more versatile for pass and sass environments. Okay, so I preached the container gospel from my tiny little church, looking up at the massive edifice, the 20-story one, so in Salt Lake City. But the question is, I mean, if my container God is so wonderful, why aren't you all worshiping in his altar? Why aren't you using them already? The reason for this is the current cloud market, for all the hype about platform as a service, software as a service, it's mostly about infrastructure. Infrastructure, it doesn't matter whether it's a container or a hypervisor running your virtual hardware because you want to install your own operating system and boot it up. The other problem is that historically, only one out-of-tree container technologies could actually do infrastructure as a service with containers. So it was really only OpenVz and the Googleplex that could do it. LXC, which is the old Linux container technology that was begun by Pullmanage based on C groups, never achieved the level of sophistication, the level of isolation where it could actually do an operating system booting up properly from in it. The test of that was you could never actually log on to an LXC container and type reboot without having it reboot the host operating system. As of 3.9, this is all fixed. When we got the merged container technology upstream, most of the hooks that you require to actually reboot at the container level instead of at the host level are now in tree. We have sufficient isolation, such that LXC can actually be as isolating as OpenVz, which can be as isolating as the Google containers. What this meant was that because 3.9, remember, was only April this year, and that meant that because it was impossible to do infrastructure in containers pretty much before then, nobody bothered. Everybody had to use hypervisors. So the key message here is that now that we've actually improved container technology in Linux, and I've got a whole section and sort of not quite an appendix, but there's a whole second part of this talk about what we've done to Linux and how it's now all upstream and where everything is wonderful and good to go. Because it wasn't really upstream before 3.9, you had no choice if you wanted to do infrastructure, you had to be hypervisor based. But now you can actually install your infrastructure in containers, and if you do this, you'll be ready for the next level of the revolution when you actually just want to containerize those individual applications. So now we're actually enabling future proofing. So the conclusion of this half of the talk is that containers not only provide up to three times the density of hypervisors, they can be orchestrated orders of magnitude faster. They come with their own tool set for solving the tenancy problem, which is much more difficult in hypervisors. So the idea is that when it comes to the cloud and what you actually want to do with the cloud, both today with the infrastructure and tomorrow with all of these mythical services that you're serving up to end users on their cell phones or whatever it is they have, their watch or their ring or their Google Lasses, containers is a much more ideal technology than hypervisors for doing this. And you don't even have to take my word for it. All of the big boys who are serving up what you consider to be cloud services today, that's Google, Facebook, Twitter, they're all containerized. The question you should be asking yourselves if you're planning a container service is if Google does it or if you're planning a cloud service is if Google does it with containers, why am I not using containers? And even if you wanted further proof, the distributions themselves are jumping onto the bandwagon. Every distribution today, Red Hat, Suisse, Ubuntu, Debian, they all have container plans somewhere in the little road map. So Canonical is the best one. They've had the guys who run LXC working for them for a while. Suisse has had the ability to build images via studio which is its image building tool originally designed for hypervisors for a while. And Red Hat recently adopted OpenShift as part of its cloud strategy. OpenShift contains a different container technology from Makara. It's actually a curious one, it's sort of out of tree in some ways because the problem with containers before 3.9 is they weren't sufficiently isolating. So the Makara containers pull a trick with SE Linux where you use different security contexts in your containers to actually cause them to be isolating. But the point about this is that everybody has a container story nowadays. So let's look at where containers are on Linux. As I told you before, they came from three separate sources. They were really four sources. IBM did make an effort in about 2002 to get Deeper, which was their container technology on the Z and I series upstream, but nothing really came of it. There's almost no remnants of Deeper left in the kernel today. So the three sources of container technology were Virtuoso from Parallels, C Groups effectively from Google, although it didn't begin there, and Namespaces from Eric Biederman. Namespaces are just what allow you to separate, say, different network interfaces to different containers. Of those, like I said, only three are being pushed today. The main problem is we have two competing control planes, plus the open shift one, because it uses SE Linux as different. I suspect now that we've solved the isolation problem, the SE Linux one is a bit more... It's clever but complex. I suspect it won't survive much longer. And actually, this slide is now a lie again, because Google just recently open sourced all of their container toolkits. So now we have three competing control planes instead of two, but at some point we will try and unify them. So the question was, what about the liquid toolkit? And you could also ask that same question about the docker toolkit, which also has a slightly different way of orchestrating containers. I'll sidestep that question by saying that really what we need to agree on is the library for it, which is sort of the operating system control plane, how you do the orchestration, which would be liquid, docker, it would probably be the Parallels cloud, it would be open stack as well, should be up to it. But what we really need, what we should think about is starting to unify the control plane. I mean, we don't have to do this at base as long as they're all using the same kernel and they use the same ABI in the kernel because it's single technology sitting in the kernel. It doesn't matter if you have somebody wants to use LS and somebody wants to use DIR. If you use the GNU tools, they both work. Does it, do we really need to unify? I mean, and the usual thing that people who use DIR and people who use LS argue about is what are the dash command line options, although LS has every letter in the alphabet now plus a few more. Do we really need to unify them? Once we've got the kernel unified and we no longer have under threat from this KVM Zen split, chances are that we could actually survive using the same tool, it's using different toolkits. So we're thinking about unifying the toolkits, but we don't even know if that's the direction we'll go in. We discussed it, actually I've got a slide on this, I think it's my next slide. Yeah, we discussed this in the containers microconference of plumbers in New Orleans in September, but we didn't, we reached the conclusion that actually most of our tools did mostly the same things. We didn't reach the conclusion that we would unify them. We thought about it, but we haven't actually pulled the trigger on that yet. So the chances are there will be lots of toolkits floating around in the wild for a while yet, because Google came to that conference and surprised us with the news that they were actually open sourcing their toolkit. At some point, somebody will look and see if they can produce a single tool set from this, but that point might not be today. So, so that's a hard question to answer. There are things that some tools do that others can't. For instance, LXC has never really been aimed at bringing up individual Linux operating systems. So with the virtuoso containers, the OpenVz containers, we can boot up Ubuntu and Debian and Red and Fedora side-by-side all on the same kernel effectively, all in different containers. LXC never really aimed at that use case because that's the infrastructure use case and they couldn't do it. So if you want to do that use case, the OpenVz toolkit makes it much more convenient. However, if you want to do the use case where we're containerizing at the application level, so you just want to put Apache and MySQL inside a single container, it's actually, I didn't, I suppose as the parallels guy, I shouldn't really be telling you this, but it's actually much easier to do with LXC tools. So the unified toolkit would have to be an amalgam of all the tools. It's not a question of saying one toolkit is the best toolkit. It's the fact that each toolkit was designed for a different use case. Now we've allowed all of our users to do those use cases. We could do with the unified toolkit that enabled all those use cases as well. Nobody has that one true toolkit today. So we'll still be discussing it. What I think will happen is that each of the tools will move towards the other. So they'll each accommodate the other's use cases. And at the end, they'll look like LS and DIR. They'll have the same capabilities, but different options. And at that point, we might start discussing merger with more seriousness. So does that answer your question? Well, so Libvert is a different animal entirely. I mean, Libvert is a library designed to make infrastructure work. Libvert is not really a very good tool for containers because that's not what it was designed for. To be frank, Red Hat designed Libvert so you could migrate from Zen to KVM. That was its use case. That works for IS containers, infrastructure containers. So it's perfectly possible to have a Libvert interface to open VZ because it was designed as infrastructure. With the changes that are coming along for LXC, more and more of the Libvert API can be done by LXC. So it's actually possible to get Libvert to do LXC as well. But Libvert doesn't do a lot of the stuff that containers can do. So Libvert has this weird ballooning thing that we just don't care about because we want an API that just says add memory, take away memory, which we can do in containers, but hypervisors can't. So the problem with Libvert is that it's a hypervisor straight jacket that we could force containers into. But if we exclusively orchestrated containers via Libvert, we'd lose a lot of the stuff that containers can do, but hypervisors cannot do. If you want to orchestrate infrastructure containers, you can use Libvert and it works just as well. If you want to orchestrate application containers, not really, because that's not the... Libvert will try and scramble towards that use case because we're going to get a point in the market where hypervisors with hardware help are going to try and convince you they can do these application cases that containers were tailor-made for. And so at some point, Libvert will struggle towards that. The APIs they add for Libvert to do that might actually help us. They might actually be the APIs we could use in containers as well. But right at the moment, Libvert is aimed at infrastructure. If you use cases applications, you really don't want to be using it. You want to use something like the Docker API or the Virtuoso API or even the LXC API. LXC is good at orchestrating applications. Okay. But the end result of all this arguing is that Linux is going to be the best container technology that any operating system has ever had. Most operating systems have already today missed the boat. Windows has no real container technology unless you want to contradict me, KY. You can say that Azure does have naughty containers, but Windows Server, as delivered to you today, does not have containers. Mac OS, technically to say it doesn't, is wrong. BSD is also struggled grappling with how do we add containers to the operating system, but it's not really there. The only legacy operating systems that actually had containers was Solaris and AIX. Solaris is getting some sort of new lease of life with this smart OS for the cloud thing, which is actually Solaris containers. But by and large, if you look at the technology, by the time you reach the sort of the later kernels, three, nine and beyond, Linux has technology that far surpasses most of what's going on in the other operating systems. So the end result is that Linux will be the best operating system for orchestrating containers on the planet, I guarantee it. Just because I'm working on it doesn't mean I'm telling you the wrong thing. So the true conclusions from this talk, and I think I'll allow, I'm getting to the point where you'll have five minutes for questions, is that containers can be used in every place now you could have used hypervisors. That means that there's no real reason why you shouldn't be using containers today. Containers will be the new virtualization technology for the cloud, just because they can do things much faster and more efficiently than hypervisors can do things. Obviously, if you have cloud service plans, you should be looking at containers. If you're not looking at containers and you're looking at something based on a hypervisor, just ask yourself the question, will Google do it this way? And I'll leave you to come to your own conclusion on that one, but it's pretty foregone since Google is all about containers. Linux, I promise you, because of all the brain power that we've got working on the single internal containers technology solution, will be the best operating system for containers on the planet. So obviously, Linux is going to be the best platform now and in the future for cloud services. And the question I'd like to leave you with is if you want to beat the rush, why don't you use containers today instead of waiting until tomorrow for your application services, because you can use containers for infrastructure today? So with that question, I'll ask you if you have any more questions. Yeah, the back. The question is if I have any good examples for tutorials for use of application containers. The answer to that question is, I don't really from parallels because we're still about infrastructure. But I do know that Brandon Phillips, who's, he uses a type of docker container, he does have tutorials about how to bring up this type of stuff. And I think if you feed his name and containers or probably feed his name and docker into Google, it will come up with a few for you. Okay, let's go from the back of the room to the front. Okay, so the question is effectively, are containers more secure than hypervisors or vice versa? The answer to that is neither system is secure, the base level. Whether it's Zen or KVM or containers, at some point you're always relying on the Linux kernel to provide the base level of the hypervisor. Somebody knows what you're using as your base layer, they can always work out how to compromise it. So the attack on a container based system will be different from the attack on the hypervisor system. But at some point, if you're clever enough and you exploit bugs in the kernel, you can penetrate from one container to another in the same way as you can penetrate from the guest into the hypervisor and back up to another guest again. There are companies like Nebula, whose business model is trying to make this transition more difficult to do and these things much more secure. But at the end of the day, what a hacker does to get from one virtual environment to another is exploit bugs in the system. The day we have a bug-free system is the day that both hypervisors and containers will be fully isolating in that sense. And that day will come shortly after the sun turns blue, I think, just because we're software programmers who always make mistakes. An operating system is complex. There is no formal proof at the moment the operating system is free of bugs. So does that sort of, I mean, it's not a helpful answer, but it's the true answer. Okay, next question. People creating these cloud platforms for IAAS to adopt them. Is there, have you kind of the same thing? Is there something you recommend doing to try to use that? Yeah, use our technology, we can do IAS. So the problem you face is that LXC could not do infrastructure really before the 3.9 kernel. There wasn't sufficient isolation. Your only choice was either to pull tricks with SE Linux, which is what Makara did and Red Hat took over, or to use a hypervisor. You had no other choice. From kernel 3.9, you can, I mean, we've demonstrated bringing up Fedora on Fedora, say, in containers with 3.9. Fedora 19 has, actually Fedora 18 has 3.9 kernel, which is what we used. You could do the same thing to bring up Ubuntu on Fedora and so on. So the technology to do that is upstream now. Depending on what you're basing your cloud service on, you won't have it available to you. Because if your cloud service is facing the enterprise, they want enterprise distributions. The kernel and most enterprise distributions are still 2632, which is far too old to actually orchestrate containers. If you use the... I showed you Parallels Cloud Server with its 121 container density, that is based on 2632. It's a heavily modified 2632 with all of this surgically back ported from all of the later versions. Technically, we've surgically altered the 2632 kernel and then tried to forward port it upstream. So we've pushed upstream what we have in Parallels Cloud Server. If you want to use that as a commercial product, you can and it will do infrastructure containers for you. If you want to do infrastructure containers with the open source technologies, you have to wait until the world is caught up with the 3.9, 3.10 kernels. Then it will be possible to do what you want to do. So when you talk about bringing up Ubuntu on Fedora, I suspect you're referring to just Ubuntu user space. Yeah, but... So the question was, when I talk about bringing up Ubuntu on Fedora, am I not just talking about the user space? The answer to that question is yes. So I think what KY is leading to is Parallels also does containers for Windows because we actually hacked the Windows Operate, we reverse engineered Windows and sliced containers straight into it. And the reason we can bring up Fedora on Ubuntu is because the Linux kernel ABI is very solid and very stable. Ubuntu doesn't care that it's effectively running on top of an ancient 2632 Red Hat kernel that we've modified. It will just happily come up on top of it because the Linux ABI is so fixed and Linux enforces that fixedness so much. Because that ABI is fixed, user space doesn't really care what the kernel is running beneath it. If you look at our Windows technology, it's physically impossible to bring up Windows 2012 on top of, say, Windows 2002, just because Windows does not have that rigid kernel ABI. I think that's the thrust of where your question is going. Well, in fact of bringing up, would you get into a situation where your kernel happens to be a newer kernel, rather an older kernel, and you need to bring up a new user space on top of that, would that be a problem? So the question is, would newer user space on an older kernel work? And the answer is actually yes, not quite for the reason you think. It's because if you look at the way we bring up a user space in Linux, we all use this thing called glibc, even if we hate it. Glibc has many ways of doing different things. So if you look at what glibc does for standard IO, for instance, it has about three or four different fallbacks inside the library. So what will actually happen if you try and bring up what you think is your shining brand spanking new Fedora 19 kernel? And let's say Linux had a syscall for doing, let's say it was printf, this is a fictitious example. And that syscall does not exist in 2632, nor does it exist in the containerized version. And you're like still boot up this shiny new Fedora 19 operating system. The reason is that glibc quietly probed for the syscall and when it found it wasn't there, it used something else. So the library itself hides from us the fact that the kernel is different. And the reason it does this is because glibc also wants to run older systems. So it's a sort of conspiracy between the library writers and the kernel people that this thing accidentally works. Okay, let's go with you. Do you think it's a good idea to use containers for applications unboxing on mobile platforms? So you're asking me if the premise of my talk is a good idea. Yes, of course, but I'm a container advocate. Mobile platforms already do this. If you look at Android phones, they already use C groups for the surface flinger to make sure that all of the resources are concentrated on one application that you're actually seeing. All of the rest of them get starved. This is already a container use case. There is a company called Celerox who is trying to solve a different problem with containers which is basically the problem of if I have my super paranoid business environment and I have my reasonably relaxed home environment, how could I run both environments on the same phone? And that's actually your phone taking on a business personality as soon as you walk through the door and making all of the files that are in your home personality inaccessible. Even to the extent that you can't photograph the top secret plans, put them in your home environment and walk out the door with them. So there are definitely use cases for containers on mobile phones. And at base of it, the reason why you use containers on mobile phones and not hypervisors is that hypervisors don't run very well without hardware acceleration. The ARM processor which is used in most mobile phones does not have that hardware acceleration yet. Therefore everybody use containers on mobile phones. They will on, I think it's ARM 9 that I can't remember which ARM version that has it. Most of our phones don't have that ARM processor. Okay. Limitations. I don't know if they are solved with the latest kernel but I was running the latest table and I think it's a 7.2 and OVZ kernel, also the latest version. But this isn't possible as an NFC kernel in very NFC sort of container. And another thing which makes really problem is Fedora 1819 with SystemD, which doesn't really work good. Is it solved now or are there? Okay, so you asked two questions. One was about NFS, right? Which server was it, NFS? NFS, NFS. Okay, so we've had a guy working on... It's fine, yes, but not a server. NFS is a real problem. And we have had that one of the parallels guys at this conference, Stask and Spursky, it's been his job for the past two years to get this bloody thing to work. It actually now does work. It actually works better upstream than it does in OpenVZ simply because we haven't bothered to do the back port and Stask is a very enthusiastic guy so he was very interested in getting it to work. So the answer to the... For the latest kernel? For the latest kernel, it should just work. He's got most of the virtualizations upstream. Your other question was about SystemD. So we have a slight disagreement with certain people who run SystemD about who should control the container environment because SystemD, containers are based on C groups, SystemD also wants to use C groups. So with Fedora 18, we can light SystemD enough that it does, it should work. Fedora 19, I know we have problems running but then realistically, Fedora 19 has problems on physical hardware as well. It's not exclusively our fault, right? In future versions, so we just had containers get together today with Tajen to sort out how we're going to solve this. SystemD wants to own the world and we can't let it if we want to containerize it as the problem. We have had a discussion today. I only saw half of it so I can't tell you what the outcome is. I caught the Google guy, Paul Turner, in the corridor just before I came into this talk and all he said to me is he thinks we've got it sorted out. So I'm not going to promise you anything on the basis of a hallway conversation but we might actually get that problem sorted out. It is our aim to get it sorted out. Okay, one last question at the back of the room and then we're going to have to before they kick me off the stage. While you answered your question in the formulation of it, kernel developers don't care about user space. User space is a test case for the kernel. So we have no real interest in toolkits whatsoever. All we really care about as kernel developers, this is with my kernel developer hat on, all we really care about as kernel developers is that the ABI works, is consistent, does what it's supposed to do and is there for anybody to make use of, which is all of the different toolkits. With my Parallel CTO hat on, we do care about user space because multiple copies of user space doing the same thing tend to lead to confused users. So we, from a Parallels point of view and from a Google point of view at the user space level will continue to push for unification up there. But once we've got the kernel done, we've broken the back of the sort of bifurcation problem. If the kernel patch is supporting these two different toolkits were different, there is no hope of unification. Now the kernel patch set is the same. We can hope to unify the toolkit, but at the end of the day, it will be a choice. It's not up to the kernel developers to dictate, unless they have system D, it's not up to the kernel developers to dictate the one true way of doing things. So we'll let the user space people work it out amongst themselves. Okay, and I think with that I used up my, okay, one last, how am I doing on time? Guy who's the timekeeper? One last question, okay. My opinion would be more informed if I'd actually seen the GRSEC patches. You're talking about GRE tunnels, right? Security and GRE tunnels. As far as I'm concerned, it just looks like a namespace problem, but as I said, I haven't looked at the patches so I can't give you an informed opinion. So you're muttering behind, what have I said wrong? Okay. So I still haven't seen the patches. If you tell me what they do, which you're not gonna be able to do in your question in the time we have available, does what? I have heard of it, but no, it doesn't mean that much. Oh, virtual server. Yeah, I mean, we have use cases for it. At the end of the day, if people want to get the, I mean, containers are all about isolation and sort of resource control. This is just another set of resources to control. There's no reason why we wouldn't merge it. It's just that I haven't seen the patches. I can't really comment on them. So I'm not giving you a very useful answer to your question. I can probably introduce you to the person who does care. I can find him, but that person isn't me currently. So it's not an answer to your question, but it's the best I can do. With that, I'll just say, thank you very much, everybody. It's a pleasure talking to you. Thank you.