 Alright. How's everyone doing? Good. I'm standing between you and coffee, so I understand the urgency that we have in finishing this talk. So what I wanted to do today is I wanted to talk about virtual machines and containers. There's been a lot of talk about why these technologies might be brought together, a lot of technical demos of bringing these technologies together, but I wanted to provide a concrete use case and what we're doing at CoreOS around one of our products to do that. So if you haven't heard of CoreOS, who's heard of CoreOS? Can you raise your hand? Alright. That's good. So why I'm on stage right now is that CoreOS over the last three years has been building a ton of open source projects. Rocket, etc, CoreOS, Linux are some of those things and we've been deeply involved in things like Kubernetes as well. And then we've been building a number of products, enterprise products that help people adopt and essentially get involved in this whole container movement that's happened over the last couple of years. Those things are Tectonic, which is a full container platform that we've built that includes Kubernetes and our Quay registry and build system. So just to get everybody on the same page on how containers are built, the basic life cycle of a container build looks something like this. You have your source code, wherever that comes from, GitHub or wherever, you take that source code and you generate a container image. This is generally done through compilation or inserting a bunch of packages required like your language runtimes. What comes out at the end of that whole process is you get essentially a tarball. And container images really aren't that much magic. It's just a bunch of files that end up in a compressed archive. And it looks whatever is required for that particular application. And then that gets a name and it gets put on top of some type of registry so that hopefully at the end of it, you have your servers that you actually care about or you want to run this container. You tell Kubernetes or whatever orchestration system you're using. I want to have three of these containers running on top of my servers. So that's the life cycle from source code into production of a container. Pretty straightforward. But what I'm going to be talking about here is the build system. So that thing that transforms the source code into that container image that you can eventually pull down. So we run a service called Quay. Quay.io is the hosted service. You're going to have to just bear with me. I know that Quay is not the way you pronounce that word. That's how we pronounce it at CoreOS. So you're just going to have to bucket it with aluminum and vitamins. It's going to be fine. We're going to make it through. So the overview of Quay is it's a software as a service. You can sign up freely. Use your GitHub account. It's also an enterprise product that's trusted by a bunch of companies. JPL, hotels.com are some of those folks. You can see others on the list. And then it does a number of interesting things. We try to create a system that people implicitly have trust in. And we do things like audit logs and security scanning so that you can have confidence that the people who should have access to the container do have access and you get an audit log of who pushed and pulled that image. And then finally we're able to dig into the container image and tell you and give you notifications of if maybe there's vulnerabilities in that image and that sort of thing. I won't go into details. I'll touch on that at the end of the talk. But we do our best to create kind of the world's best container registry. But what I'll be talking about in particular is how we've improved the SaaS product, the Quay.io product. So the requirements for these build systems. It needs to be multi-tended. You're going to have lots of people source code coming from all over the internet and you need to ensure that one person is unable to interrupt or get access to somebody else's source code or their container images. It needs to scale. Containers continue to take off. There's a huge growth in the entire market. And so ideally we build a system that is able to horizontally scale so our engineers don't have to rebuild the whole thing again in 12 months. And we ideally are able to build a build system that can be ran behind the firewall as well. So the multi-tenancy concerns around these things are quite daunting. So you're going to be downloading arbitrary things over the internet. You're going to be running arbitrary code that the user has provided or maybe some code that you downloaded over the internet in the first step. You're going to store arbitrary files on disk. And then finally you want to give these processes that are doing the build root access to the host. So show of hands, who wants to run the site of service in their environment? So there's a lot of things here that seem a little bit scary and make for a challenging thing to build multi-tended. This root access thing is kind of important because people have this laptop experience with building containers and they want to be able to do the exact same things that work on their laptop, just jumping down to root when they actually go and build on their host in their hosted build system. And so we just want to provide a really seamless experience. So how we did it. On EC2 what we do is every time somebody submits a build, we spin up a new instance. And that instance is used just for that time period, just for that build and then thrown away. And what we do is we provide credentials to that individual instance that say you're able to do these two actions, which is push status, push logs up to us, and then push the final container image and nothing more. So we essentially protect through virtual machines. The overall architecture looks like this. You check in some source code to GitHub. It goes to Quay through some webhook and then ends up in a build queue. And then in that queue we have all these individual EC2 instances that chug through, compile the container image, and then push it to the image store. So this system's worked great. It's the system that if you use Quay, you've experienced for a long time. But we add some motivators for change. So the first is that EC2 instances take a little bit of time to start up. So it's essentially on the order of a couple minutes and this meant that once we get a webhook in, if we're adding capacity, it takes a couple minutes for those machines to start the build. We also had excess expenditure. So we weren't really spending the money on our infrastructure as efficiently as possible because these instances are built on an hour basis, which we'll talk about in a second. And then finally, enterprise customers behind the firewall can't run EC2. So it's a little challenging for them to use the same experience. So this is the timeline of kind of a build. So the first couple of minutes are spent bringing up the EC2 instance. The next couple of minutes are spent actually doing the build or actually even less in a lot of cases. It's just a few seconds. And then we get build all the way through the hour. So not an extremely efficient use of resources. So let's talk about VMs and containers and the motivators here for using VMs and containers together. So these diagrams that I'm going to be showing you are extremely technically simplified. One of the beautiful things about giving a keynote versus a regular session is that there's no Q&A. So if you'd like to discuss the details of this, there's a lot of folks at the Cora's booth, including myself, who'd be happily able to talk through it. But just to get the high-level ideas down on paper, let's talk through it. So a container process interacts with the Linux kernel in a lot of different ways. And there's device nodes, there's syscalls, there's various other IPC mechanisms. And each of these sort of mechanisms has their own way of authenticating and isolating the process. So the reason that we've always said that containers are less secure or there's always this conversation around containers being less secure is because you actually have a really complicated API surface area to lock down. We've done, as an industry, a really good job. And I would argue, in a lot of cases, containers are more secure than the classic way that we've been deploying software where we just kind of line up the various pieces of infrastructure under our NIT systems via some configuration management. We're able to more nicely isolate with containers than we were in the past. But this is the reason that there's like the constant question mark around containers being secure. It's just a big API surface area. Now what VMs have had and the reason we trust them in multi-tenant environments is because there's a smaller surface area. The APIs aren't as big. And some of them, through the KBM side of the kernel, are actually enforced through hardware on most platforms. And so we have an implicit amount of trust in the system. And then we have this constrained IO path through VertIO that kind of constrains the amount of API surface area to write files to disk and interact with the network. So that's why we implicitly trust these things. And then the next thing is we can use kind of containers and VMs together by essentially putting the resource isolation side. I only want this much CPU bandwidth and this much memory bandwidth and this much network bandwidth around the virtual machine. And that's how we kind of use containers and virtual machines together. We use the isolation mechanism of the VMs and the resource isolation of the container. So the tenant isolation, what we're trying to achieve here is we want builds to be isolated from each other. Nobody should be able to interrupt somebody else's build. We don't want users having access to other users' source code or their container images. We need to protect the network so the individual build can't attack back towards Quay. It needs to be isolated, only have access to the resources it should have access to. And then finally Quay protects itself with these one-time use tokens from nefarious users who might actually break out of the build system and get access to the wider VM. So our new approach, instead of using EC2, is we use virtual machines, containers, and Kubernetes. And it's very much the same as our previous approach. Only we've replaced a single EC2 instance with a single KVM instance running inside of a container. And the result is pretty good for users. It's essentially faster builds and we have more efficient use of the capital so we can buy better and bigger and faster machines in order to do the builds instead of spending time just kind of intally at the end of the build. So the new system looks like this. Everything comes in through Quay, gets into the build queue. Quay then talks to a Kubernetes cluster that we have running on bare metal servers on this provider called Packet. Those bare metal servers are members of this Kubernetes cluster. And then what happens is a very classic thing inside of Kubernetes, which is a container job, like a one-time run-through container, it's called a job, gets scheduled to one of these machines and executed. So if we zoom in a little bit, what happens is we have all these containers that are actually running KVM running underneath Kubernetes on top of these bare metal hosts. And then those bare metal hosts, those KVM instances running in containers pull things off the queue and start to build the container image. And so if you dive down into the actual instance, what we have is the agent that's pulling logs off, giving users status updates on their build. We have the Docker engine that's actually running through and building the Docker file and then we're pushing the container image off of disk into Quay.io. So the result is quite nice. So previously we had these EC2 instances and we had this long startup time. Now we're able to bring that down to just a few seconds, about 15 seconds, which is an 80% improvement on the initial start of the build on top of Kubernetes and boot this VM and get the agent downloading the context for the build. And so the comparison looks really good for the user is they've gone from this three minutes wait time all the way down like cut the build time in half. The cool thing is that it's available today making Quay an even better build system and service for people interested in containers. And I put the asterisks there for a very important reason. So before getting this all launched, we wanted to ensure everything works. So we rolled it out about three days ago silently to our users and people started to notice. So immediately on Twitter we started to get people saying, wow, the builds are really fast on Quay. What do you guys change? And we wanted to hold the news until here today at Linux.com. So we didn't really reply to these and it's always nice that people notice when products just kind of naturally get better. So some considerations while building this entire system. First is why did we choose Kubernetes? Well, we were used to having this API driven system so that we could have internal dashboards and metrics and monitoring in order to see what does the build queue look like today in case we got paged or whatever. So having this API driven system is really important to us in order to operate the system effectively. Containers allow us to kind of splay out these virtual machines and upgrade the processes and everything over time. And it's a really easy way of then resource isolating, perhaps creating limits and things around individual users so they can only consume X amount of CPU time, Y amount of memory to protect ourselves from denial of service. And then scaling is really easy. Kubernetes is easy to plug in and horizontally scale the cluster over time. So we're in a good position as Quay and containers continue to expand to serve our users well. And then packet and bare metal. So we chose bare metal servers because they, one, allow us to run VMs. Two, there's a really good price to performance ratio when we're thinking about expanding this service over time. They have AWS peering, a lot of Quays backed by S3 and AWS services. And so we're able to push all those assets out of our build system back into AWS, which is really important. And then finally, we have API provisioning. So with packet, we're able to in a couple of minutes add additional bare metal capacity without having to rack new servers. So with all this stuff, whether some other use cases for virtual machines and containers. This is a really compelling use case. We're able to make users really happy combining these two technologies. What are a few other use cases? One is that inside of the rocket container engine, we've combined virtual machines along with the help of Intel. So classically, you would execute a container process, and they'd be just isolated by regular Linux namespaces. But we now are able to wrap just kind of transparently with a single flag containers inside of virtual machines. Inside of OpenSack, we have worked on a project that we really cleverly called Stackinetease. This is where you're running OpenSack as a process on top of Kubernetes, and you're able to manage and upgrade OpenSack using those Kubernetes APIs. And they run inside of virtual machines. So the actual IaaS virtual machines that OpenSack is scheduling run under containers under Kubernetes. And really what this is about is we're trying to bring OpenSack over to Kubernetes with the influence of a lot of Google's infrastructure. So Google's Cloud Platform runs virtual machines to this API that they call Google Cloud Platform, and it runs on their exact same infrastructure, Borg, that schedules all their other applications. And so we have this hashtag that we like to use called Giffy, which is Google's infrastructure for everybody else. And we want to paint this picture of you're able to run your IaaS applications just alongside any other application that you might want to do on top of containers. So some future use cases, you can see virtual machines and containers being used for batch processing of video encoding and that sort of thing. Non-Linux workloads like running under emulation or Windows, or running these VM appliances that are really popular in enterprises, the GitHub and Oracle and own cloud sort of appliances that you can buy and run on your own gear. So some conclusions. Initially, we had these requirements of multi-tenancy auto-scaling and being able to deliver a system for behind-the-firewall use. We were able to do that through the use of virtual machines, containers, Kubernetes, and architectonic Kubernetes solution built on. And then some actions. I want to leave you with some takeaways. First is that Quay is a free hosted service that we let anybody building open-source projects use for free. So please try it out for your next open-source container project. The other thing is that we have this open-source project called Clare that actually builds and scans through these container images finding any known vulnerabilities. So you get this dashboard of, well, perhaps there's some packages that you can update in your container or there's some other things that you can address. It's really great. We've seen open-source projects use this software, find out that the containers have known vulnerabilities, and update them. This is a Kubernetes project that uses Quay. They saw that there were some things in the dashboard that could be addressed. They looked through the actions list that Clare gives them, and they addressed the issue, rolled out a new container image, and had it all fixed up in a few days. I'd like to invite you all to our conference if you want to learn more about all this stuff in December in New York City. And with that, I thank you for your attention and enjoy the rest of the conference.