 Okay, my name is Dan Walsh, I'm going to be doing a talk called containers and production. This is a little background, I run the container team, I'm the lead engineer on the container team at Red Hat, I've been working on Docker and containers for the last couple of years and I put out a paper, it's an internal paper, but eventually we'll be putting it into the blog talking about running containers in production, basically thinking about containers going forward. One of the things, containers talk about DevOps all the time, so developer and operators and how this new DevOps model where we're constantly putting out packages and the whole release cycles and things like that, but to me at this point Docker is concentrated almost exclusively on the developer side. Especially when you look at, I'm ignoring orchestration at this point, but just looking at what the tools have done, they've been really concentrated on the developer. When I think about containers and production I think, I'm trying to think, we're trying to look at from the operator's point of view, what are operators going to do when they get containers and production? What do they look for? So it's really sort of rethinking about containers. Most container work up to this point is concentrated on the developer, Docker build, Docker commit. It's all about building these containers, building your applications, and the standardization of OCI format, new tools can arise, so up till now everybody's been building containers with Docker, as I said, Docker build and Docker commit, but really I see there's a standardization of the format, image format, and I'm starting to see other groups starting to build containers or container images using other tools, and I think that's good. I think it's good to have innovation and stuff looking at how you build containers. So I've seen the Ansible right now has playbooks. They're allowing you to build containers with playbooks. OpenShift has a thing called Source Day Image, and they're building container images that early today. We used to talk by Fedora talking about how they're doing layered container builds in there, and then there's another project that's been going on for a while called Docker. But the real goal here is to allow people to build these applications, and as long as the output of these applications is the same, I think that's good. It also means though that we don't have to ship a Docker engine and a Docker fully instrumented for building containers in production. But really why do I care about that? What we're looking at here is the copy on write file system. So we look at containers, container, the whole Docker infrastructure has all been looking at adding copy on write, because when I'm doing Docker builds, and when I'm doing Docker commits, what is that? I'm basically taking what I had before, I put a copy on write file system on it, and then I continue developing the applications, and I'll eventually save that application to a new image for me, and that's all built around copy on write. When I'm running containers in production though, I don't want my application to be able to modify itself. I really don't want applications to be able to modify slash user. So in production, I would argue that we want to move towards a state where we're not necessarily using copy on write file systems in general for doing that, and that opens up some opportunities. So if we look at the process of copy on write systems, things like device mapper and butter FS, both break shared memory. So right now, if you run device mapper containers, which is the fault for Fedora and all Red Hat based operating systems, we run with device mapper. The second default is AOS, we're going to talk about Oval AFS, and then you have butter FS. Well, butter FS and device mapper break security, sharing memory. What happens is these are block based devices, and there's a problem that the file system and the block device don't communicate well together, and the bottom line is the kernel figures out that a shared memory is the same. So if I load glibc and another process loads glibc, the kernel looks at the file system and makes sure that the file system image, I'm simplifying this a lot, and the block device is the same. So it can say, oh, that's already loaded into memory, so I can use it. As soon as I start using device mapper or butter FS, the kernel gets confused. So we can no longer figure out that even though it's using both systems are using the same low level image, the kernel can't figure it out, so it ends up loading shared memory. The problem with that is if you run in 10 containers in a system that are loading the JRE into the system, if I run them on standard file systems, the kernel will figure that out, and it will load one JRE into memory. I'm talking about copy on write file system, right now you load 10 of them. So we end up instantaneously using tons more memory when we're using copy on write file systems. Another copy on write file system is Oval AFS. Oval AFS actually doesn't have the shared memory problem, but it has other problems. Nice thing is if you've been following the kernel patches, we've been working very hard to fix a fundamental problem with Oval AFS, and that's SC Linux. SC Linux support wasn't there, so we lost a lot of security. If you went to an Oval AFS for your back end, you've had to turn off SC Linux support for separation of kernels. My team has been working heavily on that. We have patches both going into the Linux kernel for Oval AFS and to the Linux security module, and Paul Moore is in the back somewhere here. He's the one. We're merging into the SC Linux part of it, and those should come probably by the end of the year. I'm hoping to have a raw high kernel out in the next few weeks that people can start to play with Oval AFS and SC Linux support. But a real problem with Oval AFS is it's not a politics-compliant file system. So there's some fundamental problems that people run into when they use Oval AFS based on assumptions on politics compliance. It really was, so the one we hit originally was a program that opens up a file descriptor and then reopens the file descriptor or opens the second file descriptor on the same file. They'd open up the first one for read-only and then opens the same file descriptor, second time for read-write, and it actually ends up with file descriptors to two different files. Because when I open up a file for write, it actually does a copy. Yeah, it copies the file into a new place. So RPM did that. So we ended up with RPM having two, you know, thinking that it opened the same file twice, but it actually opened up two different files. So there's other problems, things like she had sockets that basically there's problems with Oval AFS. But I think in a lot of cases people are going to start moving to Oval AFS once it gets a little more at the corner. So another thing, copy and write file systems sometimes suffer from write. They don't perform as well as XFS. You know, if you're using a standard file system, you're going to get less performance on a copy of write, especially when you go to larger scales. And as I said earlier, in production, most images should never be modified. I don't expect, when I ship an Apache version, I don't want Apache being able to rewrite Apache. Just fundamentally, you don't want applications able to write to the slash user when they're running the production. So read only container security is much better for security. If I can only write to certain directories, I can write to my executable path, so it's always considered good. Lastly, copy and write does not support network storage. So you can't run a copy and write file system on top of NFS, it's top of the list of steps. So you lose the ability to share your images from a centralized side. You're going to see some cool stuff coming up with that. Each consists. So a fundamental problem with Docker or a container model right now is, imagine I have a Kubernetes cluster that has 10 nodes, and also there's a huge spike. I always like to use the Sports Illustrator's swimsuit edition. So the Sports Illustrator gets so many hits per week, per week, per week. And also the swimsuit edition comes out, and they go like 100,000 times more hits on their website at that point. Well, that sites out in two weeks they go roll back. So you need to have the sudden expansion of hundreds of service. If you're running a container environment, what you want Kubernetes to do is suddenly launch, say, 100 new nodes. Well, each one of those nodes, in order to run your application, has to now go out to Docker or some registry and start downloading the images. So it has to install. Or you have to pre-provision all this stuff. So just think about that. As you expand out, you have all these images. Just image data moving back and forth before the application actually starts to run. If I had a shared file system, and I ran the same image on these multiple sites, as soon as the site connected to NFS on the server, your application's there and you ready to run. So just think about that from a scalability. Scaling out, if I could use my images stored on a network storage, it would be awesome. Another thing about shared storage for my images is all of a sudden I have a vulnerability. I have 100,000 installed base. Shell shock happens. Now I have to go out and do 100,000 image updates. If a lot of those machines are all sharing off the same NFS, I just update that image on NFS and it's in fantasy here, but it gets updates. I'm like, so do we want to get rid of our cloud file systems? Some ways, in production we want to, and I have Daniel Ruiz, who's one of our leads, always comes screaming into me about peck containers. So he calls it peck container, it's actually a container that you start up and you go into and you start doing DNF updates. And you run that container forever, okay? So we can't fully get rid of copyright file systems, but in production, in a lot of cases, I don't think you need to run any applications on top of copyright files. So we've been working with this, the atomic registry, atomic or open ship, the name goes back and forth all the time, whether we're calling it atomic registry or the open ship registry, it's basically taking the Docker registry service and adding additional features onto it. So what we've done, what my team's been doing is working on a service which is watching container images to arrive. So when someone does a Docker push to the registry, we have a service that's watching, sitting out there watching for messages coming in and telling us that an image has been pushed. At which point this application will explode the images onto an OS tree. Why are we using OS tree? So we don't have to use up as much space. So you do a Docker push from your server, the application goes out to the registry, this application will watch for that and then we'll explode it onto an OS tree file system. The first feature we wanna add is the ability to do an atomic scan. So what we have is hundreds of developers out there pushing applications to a service. We have an application watching it. That image comes out and we explode it out. We do an atomic scan, figure out whether or not this application has vulnerabilities. If it has vulnerabilities, we can basically deny the push. All right, so basically in order to do that, we have to explode the image onto this and then run the scanning on the exploded image. But what happens if we share that exploded image after it's been exploded via NFS? Steps. Blaster. Let's look at it. Where is my notes? I'm going to solve for engineering to turn it red hat and I'm working. We wanted to demonstrate a little bit of the software that we've been working on, which is called the Openshift Explorer, which is an image explorer. Basically what it can do is it can watch the Openshift registry or the atomic registry for changes to images and explore those images onto this so you can inspect them by making a building over the scanners. So to start with this demo, I've just got a standard Openshift deployment of your virtual machine and a default Docker registry point. So nothing really special going on with that, other than that I've configured it for persistent storage so that I can use that storage inside the Explorer. It saves us a trip. So it might be, I've called this and I've gone into the directory. It's just a go program, single binary. So it does it really quick and it builds a binary and a Docker image that we're going to use as the base of the Kubernetes pipeline. So we'll just build that image and we're going to deploy it using Openshift so you might deploy a different service than Openshift or any kind of Kubernetes service. So after we build the image using a Docker, we have to give this program a few privileges. Namely, we have to create a service demo. We have to create a deployment config for this pattern so that Openshift will create a replication which already keeps it alive. And it also adds privileges and most expeditions like for us to do that just for this demo is to give that service account a faster admin. So it's going to overkill, obviously, production using something more restricted for the purposes of the demo. It's the easiest way and we can give it a big upgrade. So going back to our Openshift console, you can see that Docker Explorer pod has been created and it's got one replica. So it's running now and if we take a look at some of the images we've got, basically this pod is watching the image streams and is going to explode them onto disk in specified locations. So it's actually pre-pushed and in which this busybox image. And we want to see if our Explorer pod has exploded. So I'm going to the pod now, looking at the logs and you can see it has a message exploded that it exploded default busybox. At least if we go back to our favorite VM, we can take a look at the explode directory and see the contents of that image have been unpacked into this specified directory for us. So we have a few folders here, digest images and repo, I'll explain the repo in just a little bit. Digests represent a checksum of an entire image and images are sort of a human real name. So something like busybox might refer to a checksum of an image which would be located in digest. So digest contains the actual file system and images just contains this file link which points into that digest folder. So it's a way for us to update the tags but also keep images around. So you can see this link contains this checksum which points to a specific location under the digest folder. This log checksum in this real fast folder contains the actual contents of the busybox image. So this is a pretty first-level system. You can do quite a lot once you have access to the real fast. You can run scanners on it, basically do whatever you want. One of the more interesting things we can do is share it. So I've got an export configure to just share this with world read only. And I'm gonna activate my NFS server so that I can browse this on my host machine and we can see what kinds of things we can do with this system once it's been shared between two machines. So back on my host here, I'm gonna mount this head slash mount and you can make sure I have the address right. Doesn't look like it's connected. Let me check the IP. That's changed. Fix that really quick. And that looks like it's now in write-up. So now I have access to these real fast folders and slash mount, you can look at it and play and do a tree on them. So you can see I have the busybox tag and it's really fast to exploit it here as well. So I'm gonna configure it for a watch on this tree so we can take a look at it later. And I'm gonna go back to the favorite VM and I'm gonna push an image that I've tagged to be on the OpenShift repo. Specifically, I'm gonna push an Alpine Linux image. It's pretty small and I should push very quickly. And so once we push this image up, we should see it's immediately exploited on the host machine, not just in the guest VM. We also now have access to this image anywhere where this shared NFS server is located. So I have actually also prepared a script which will create a container using over the FS of any image in slash mount. It's just kind of a hack that I put together. And so if I use the container of default busybox and latest it will just drop me right into a busybox shell even though I don't have the busybox image on this host. So I can look at the release file. I'm not sure if busybox has one, but you see the point that it's a standard busybox shell and I can do anything in that shell that I can do it in the busybox container. It's a run-c container running with OCI utilities. So the hack does really clean up the folder so I have to clean it up manually, but I can just create containers instantaneously out of any image that I approach to this registry without having to pull them and I don't necessarily have to use Docker. So there's really no Docker involved in this except in the push. So on any of the clients I don't need Docker. I just need run-c. So I'm gonna pull Fedora and then tag Fedora and go on to this registry. And I have to put it in a default namespace so that it has a project that it lives in. And then I'm going to push it and it'll take a second to push. So while it's pushing, I'm gonna explain some of the technology that's going on here. I mentioned earlier that we go forward. Basically what we're doing is we're pushing these images and exporting them into an OS tree repository. So OS tree is specifically designed to store a number of file system trees. So it's kind of a natural fit for images that you would never want to change. You just want to create containers out of them. And the benefit to OS tree is that we can store many different images in the OS tree repo and it will be able to duplicate common files. So two of them just have the same sort of large JRE or whatever library it can detect. Those are the same object and it manages them with hard links. So they'll actually get linked against the same file system or resources that won't consume extra space. So if I created a small modification to an image and pushed it, it would not take that many extra space. So it looks like the push worked. And we're gonna go over to the host machine and see if we can start it. And it doesn't look like it's, it has pushed but it does take a little bit of time to explode. So let's check up on it by looking at the images directory and looking at the log here in the OpenShift console and we can see that it seems to have exploded but it can't take a little bit of time to create the tag that it uses to resolve names into image checksums. So I can look at the link file and I can make sure that the link has been created correctly by just inspecting its contents. And despite the fact that my terminal gets a little messed up here, it does seem like the link has been created and if I run this script again, it'll drop me right into a Fedora shell. And I didn't have to do a Docker pull Fedora or anything. So I'm gonna run the version of Docker and I can look at the release to see if this is effective with the Dora 24. Okay, obviously we gotta do a little cutting on the length of this but you get the basic idea. He sent me this last night but we basically have the ability at this point to have a Docker registry sitting out there that instantaneously as soon as any image gets pushed to it, the image would be available via NFS. And it's not, right now we haven't implemented it yet but there's nothing to prevent you from using Cess or Cluster or anything else. We'd actually like to get to the point where, we'd like to get to the point where we can share these things even using OX3. So we wanna be able to support all different protocols, network protocols for accessing your container content and not have to do a Docker pull on every single service and whenever I wanna get a container. So that's first point, system containers. Giuseppe in that corner here has been working on the concept of system containers. And what system containers are on, first of all on an atomic host. How many people are here playing with atomic host? Okay, a whole bunch of people play with atomic host. What's the most aggravating thing about atomic host? It can't have been. Yum, doesn't work, right? Yeah, yeah, you can't install Vim, you can't install Emacs. So it's real aggravating when you get on there to do it and really what we're trying to do with atomic host is we're trying to implement an operating system where all the software has to be installed in front of container images, okay? So what we wanna do is have software must be installed as container image but we have a catch 22. Kubernetes requires SCD and FlannelD to be running before Docker comes up. So we have to have a couple applications installed in the system in order to use Kubernetes and those applications are actually gonna be used by Docker. So how do I do that? They need to stop before the Docker so it can set up its network and these containers must, well, those containers can be run on when you only file images and Docker has problems with container priority. It doesn't have any form of container priority. Now Docker's talking about putting it in but up to this point they haven't had it but the dirty word that Docker doesn't like, system D, has priority, right? You can set up that says Docker relies on FlannelD running and FlannelD relies on SCD being up and running. So I can set up a priority list using unit files to set up proper starting of my applications but I can install SCD and FlannelD because DNF doesn't work. So I have to install them in the form of containers but the only thing I get to run containers is Docker I got catch 22. So we came up with a console we were calling system containers. We're using the atomic command. So the atomic command is sort of the atomic CLA command is a tool we're using to wrap all this functionality to build the atomic command is how you would install and now you can do an atomic pull and an atomic install of what we call the system container. There's a new tool called Scopio that's got showed up in Fedora over the last few times it's actually inside of rel and sent us at this point but you can use Scopio now it's been enhanced to pull images we're going to talk a little more about Scopio as we get through the presentation and we use OS tree since we want to have images stored but we don't want to use up time at this space we're going to again use OS tree and since atomic host is based on OS tree it kind of works well together and we want to create a system to unify because we want to set up this precedent so that we can set up a precedent a little bit of containers out and out and I'm going to use Run C everybody here Run C okay Run C is a new way if you were using Docker 1.11 Run C is a standardized pod so what's called the open container initiative it's a standardized default way that you run containers and Docker actually in 1.11 you're using Run C underneath the covers to run containers but we're going to use Run C directly inside of our system to unify us so atomic command has been implemented to handle this so you can do an atomic pull if the item is raw okay anyways you can do an atomic pull dash dash system at CD and atomic install at CD system at CD we put out at CD and slendled the containers that can handle this they're both available on Docker Hub right now so you can actually install these on top of atomic codes you can install them on REL system or Fedora system as well it's nothing preventing that yep wouldn't that atomic but doesn't atomic install do a pull automatically if it doesn't have it on the system do you really need to do the separate pull I don't think you need to you can just do an atomic to do an atomic install yeah I think you can do an atomic install dash dash system at CD so this is based on the document that I wrote six months ago so that probably should be that's why you guys are experimental so with system containers now we're using Run C we're using system D and we're using OS tree for the back of the back of the store but the container you can't update the container right when a new version of the CD comes out you have to use a tool like Docker some other tools to build a new image push it to a registry and then this tool would pull it down right so again it's in production think about that you're not developing it you're using it in production another thing that we're adding is what we're calling simple signing we've been wanting to have the concept of simple signing what we mean by simple signing signing that is similar to what we use with RPM where a user is able to go out and sign a document and say hey I trust this image Dan Walsh put this image out and Dan Walsh trusts it okay but now you get the image in your company and you're working for Acne.inc and you come out and say well we're not necessarily going to trust all images that are signed by Dan Walsh but we trust that image that was signed by Dan Walsh so that Acne.inc can come in and sign that image so we can actually end up with lots and lots of signatures based on the same image and the signature can be separate from the image all it has to be is you add it cryptographically prove that this image was signed by Dan Walsh's public key or by Red Hat's public key or anybody else's public key but we're adding the concepts to atomic and these tools for simple signing it allows you to use a PGP type signature very simple signature to sign it sign basically the checks on that is appropriate for the OCI or Docker image multiple people can sign the same image signatures can be stored on this new atomic open shift registry but also we want a lot of signatures to be stored anywhere if you want to put signatures onto a website and say you know there's 40,000 Docker images out here I trust these four okay how do you do that so what I would want to do is take those four images get their checksums sign their checksums and put it on a website somewhere so now I can do an atomic pull of that application set up a tonic so it goes out to the registry to my website and says okay I just pulled this checksum Dan does it matter what you sign yep Dan will always trust it so we can trust and install on a system or ACME trust that we can install on our system really simple stuff not building up a hierarchy but just basically allowing people to play with with signing based on this concept then we want to eventually build policy rules engines that say you only allow your applications to run certain applications so we can build a Docker and some of this is not quite built yet but we can build an authorization plugin that basically sets up a policy that says I'm only gonna allow my Docker engine to run applications that are signed by ACME.acme but now if your random user goes and pulls down an application when that application goes to run it'll check and see if it's properly signed if it's not properly signed then you get thrown out we want to build this tooling all the way up into Kubernetes so Kubernetes can start to have a policy around this type of thing, yes what about revocation, make our keys compromise yeah I mean you could we have that problem you could remove the signature file but yeah we always have that problem we've had that problem with RPMs for 30 years now so right now we're just trying to get the simple signatures we can talk about revocation in a later time but most ways that people have done revocation in the past we could build that into it but let's not get bogged down on that we've been waiting for some kind of simple signing for a long time okay I don't know how much time really good plenty left okay so we talked a little bit about signing and we talked a little bit about using the atomic command and the containers and being able to run containers on shit copy and write file systems on shit storage we want to look at a new tool that we're building now okay it's called OCID and basically it's it's components needed by OpenShift and Kubernetes in order to run containers in production but we want to look at a simplified demon but the four components that we're going to use in an OCID environment the first one being container image transport so we need a way to get that image to the local system especially if I want to run a copy on a write file system then I need a way to store it so once I've pulled an image to my machine I need a way to store that image on disk the third component we're going to need is we're going to need a way to run the container okay I need a way to actually start a container and then I need an API a wrapper around all this application so what we're looking at building is this OCID spec now let's go look a little bit deeper into each one of these so the first one the way we're going to pull the application container image transport we've already mentioned scopio in terms of the system decontainment so scopio has gone out and implemented basically the equivalent of docker pull and docker push okay it also has full capabilities to do the simple signing scopio was originally written as a tool for remote viewing so scopio is Greek for remote viewing just to give you a little background around we've been using scopio for the last few releases for viewing image containers on a registry so right now if you want to look at a particular application a particular image is JSON using just standard Docker you have to pull the entire application dot in your system and then look at the JSON what we wanted to do is basically look at the data at the registry and without having to pull it so that's why we originally built scopio but after we built the ability to look at the registry we actually figured out it was easy to implement the process of pulling and pushing using the Docker protocol we've added the ability to push and pull images from the registry and we can pull and verify simple signatures as well and we've worked with CoreOS to split this go library into a separate thing and we've decided to put it out inside of the containers so basically it's said to put into scopio which was a CLI we've broken scopio into two parts so now we have a vended go language library that we're calling container slash image and this gives you the ability to pull and push it the idea is to allow other applications to potentially use this library inside of their applications not just hire scopio but allow other applications to use and there's since CoreOS is working with us on it there's potential that they can use inside of Rocket so we basically wanted to standardize and rip out one part of pulling and pushing and put it into a separate application that could be used and just have people work on just that part enhancements so the second part, now we have the ability to pull and push images from the Docker registry or inside of registry now I need to store it so after I pull the image I need to be able to store it on the desk so anybody here play with the atomic CLI okay when you play with Docker images you always have this problem I want to look at the content of the image but I don't want to actually run the container on it so back several versions of the atomic ago we added the ability to mount an image so I can take a Docker image and store it in the content and actually mount it at a mount point where you can do atomic mount fedora slash mount and then go to cd to slash mount and you'll see the root of S of the fedora image that's installed in the system the really cool thing is if you're running containers you're running a container on the system you can do an atomic mount that's live fedora slash mount you can go into the containers mount point create a content then go into the container and actually see the content show up but really we're playing around underneath the covers and allowing you to actually work with the copy and write file system while it's running inside the container so we did that functionality a while ago it works with device mapper and overlay fs but we wanted to make that more standard the problem is that the Docker daemon has all its blocks all its handling of this copy and write file system all of the graph drive code is built into Docker and Docker is actually using a locking system and locking system is all I remember so there's no way for other applications to interact with the file system while Docker is working so we originally started to look at could we take the graph driver all the copy and write file systems moving into a separate tool that Docker and the separate tool could use simultaneously and we actually found that it was very very difficult when we started to play with that and we decided to stop doing that and just to pull the tools out and get two of our tools working together using locks on the local file system and once we have it perfected we want to go back to Docker at that point and say hey we've got all the storage drivers now working independently so different applications can use the same storage at the same time as long as they interact properly in the locking system so let's backport this into Docker so we decided rather than start with trying to manipulate Docker in it we wanted to prove that we could use it and it turned out that it's kind of neat that we can be able to use all these copy and write file systems so we call this tool a different integration it's graph C, graph tool, cowlman and store tool so I think right now it's called store tool so we want to create a Docker graph drive code that's dependent on library CLI and we've again worked with Core West and we're creating a thing called container storage and all this is doing is setting up the storage to run images, set up those root FS's on the system either using, right now we're just implementing all the copy and write file systems that are currently in Docker, independent of Docker but I want to add support for OS 3 network storage so I want that tool to learn if I have an image stored on an NFS scale if I have an image stored on OS 3 deal with an image stored on a copy and write file system then I deal with it and we can start to look at how we'll be able to launch containers on different file systems different back ends based on this tool and again, it's simple all this tool does is handle file systems totally independent, the library is totally independent it can be, people can work on this tool at a separate pace they don't have to be working with other tools and we can take patches independent of other tools and CoreOS potentially could use this as a storage back end right now CoreOS Rocket just supports overlay FS's in the back ends so if they switch to this they'd be able to get advantage of using all these different file systems back ends open container initiative runtime this is the further salon of all these tools right so we've got something in pushing and pulling we have something that actually stores the image on desk the next thing we won't need is ability to run a container so the open container initiative was started about a year and a half or two years ago now a year ago, I don't remember exactly when but basically what's happened is RunC got developed based on the open container initiative RunC in a specification for how you specify which container you wanna run what the applications are, what the labels things like that have all been standardized you mean used by Docker they're being adopted by Rocket at this point how the framework, how the tools is starting to build using the specification RunC is just an initial implementation of the specification and we're starting to see other people Clare Linux is about to release a mechanism for running containers on virtual machines and they're following the specifications so if they're ready to swap out RunC with Clare containers you'll be able to run these containers but they follow the open container open container initiative specification and as they go forward I think we'll see run other systems using it as with Docker 1.11 it uses RunC as a default back end so all the software's shared between Docker or anybody else that wants to use it and the OCI tool so that's all about RunC one of the problems with using RunC is you have to actually go edit the JSON files if you wanna configure your if you wanna configure a container to run you don't have a simple Docker tree command line or a Docker run command line to configure your JSON you have to actually go all VI really complex JSON files to create a specification that's what RunC uses to actually run a container so we implemented a tool called OCI tool which is actually sucked into the open containers onto OCI tools and what that allows you to do is it basically falls pretty closely to the Docker run command line but you say OCI tools Docker tree command line OCI tools generate and then dash V whatever you want in mountain volumes dash dash PIT equals host all those kind of commands and then it just goes down and generates the JSON file that you can use for your container so it makes using RunC a lot easier because you can specify on a CLI what you wanna run or you can continue to edit the JSON file OCI tool generates a specification and you can launch RunC if you look I believe that the system containers that we were using earlier we used OCI tools to generate that he was smart enough to generate it and so if you wanna play with RunC I would heavily advise you to use OCI tools to generate the specification so the last part of this is the OCI this is sort of a wrapper around this application and we're actually wrapping this all these tools together underneath an atomic command but we wanna be able to do it both in atomic command and as a demon so OCI-D is sort of the tool that wraps up all this stuff it's the open container initiative demon I prefer to call it OCI-D but I think it's funny, but anyways this is the least developed part of this but we decided to develop it and what this does is Kubernetes I guess one of those things Kubernetes has specified ways that Kubernetes can talk to container runtime environments so the rocket is now using it I think they still talk to Docker in a traditional way but Kubernetes has basically said if you create a demon that will follow this protocol and you can set up a kubla to talk directly to that demon and have it launch containers on it, we have so OCI-D is an implementation of that API as specified by Google site Kubernetes and it's a service for launching containers that Kubernetes can talk to so when Kubernetes wants to execute a container it's gonna send OCI-D a message that says I wanna run this container OCI-D then it's gonna go and pull an image using Scopio so it'll go out and somehow get the image down if Scopio slash the image tool knows that the thing's available by NFS it comes right back and says I already got that image or if it's pulled down to its image store but after if it has to pull the image or pull it then OCI-D stores the image on this using the storage tool that we talked about and then finally it will launch the application we want to see okay so that's OCI-D and it's a standards-based alternative to Docker and Rocking okay so it's a standards-based but the goal is to make this a fully open source project anybody can contribute to it and we want to event right now it's underneath containers we want to eventually get it into potentially get it to OCI as being a standards-based thing but again it's for running containers in production we're not trying to replace Docker okay we don't want to that Docker has a whole bunch of functionality the goal of this tool is not to implement Docker build, Docker commit in some of the other features of Docker all we want is a standard tool that is basically able to do what you want to do when you container is in production which is I wanna run this application on this machine so it needs to be able to pull the application from the service install it on some kind of file system and then run the container and that's it and we want to base it on right now it's based on Kubernetes but again we're implementing the same functionality inside of Atomic so Atomic will eventually be able to not only run system containers but you'll be able to do Atomic pull and do dash dash instead of OCI backend you could say I'm gonna use the device mapper backend and it'll pull it into your local storage device mapper not inside of Docker and you'll be able to launch containers based on that and eventually we want to get once we have this up and running we want to move open to OCI-D by default and then have a standardize another nice thing about this is where Docker is constantly changing you know this will give us a little more stable underpinning for running containers but we don't have to constantly swap out based on all the innovations of building containers and different things that Docker is trying to do and we'll get a little more stable back in so we've got questions at a second but containers in production will focus on system containers and system services and basically Kubernetes running clustered applications that's our goal and at this point questions yes so what is there anything conceptually preventing you from running application containers under the same system like what's the well we need it in this system we need to by application containers I guess you mean impact container no I'm saying so you're emphasizing the use of this just for the system containers you need to run as EDF allow Kubernetes right but let's say I just throw my WordPress container at it well I would say that's a container in production WordPress is neat right so yeah when you're out defining as a non-production container there's one that's under development so if you're building a container if you're experimenting with a container you're going to probably need a different tool than this this is more about a container image already exists at a registry and I am putting it into production okay anybody else yes so all that I mean probably it looks like that OCI-V has similar features with container-V container-V doesn't implement anything to do with storage if pulling and pushing all container-V does is execute Run-C so yeah it would be similar functionality there would be nothing to prevent eventually something like Swarm or take advantage of this tool as well all we're saying is we believe fundamentally from putting a container into production you need to be able to get the container installed onto a machine load it and then run it so we're adding those three components and then having a demon again the demon can talk to Kubernetes and we want to allow all three of those packages to develop all four of the components to be able to develop independently so we don't have to lock everything into one big monolithic application yes so beginning you said that Docker is focused on developers and it seems to me that it is focused on developers so much that it's actually not suitable to run it in production I'm not saying that don't put words in my mouth it seems to me that is what I'm saying you said it I didn't I'm not getting into trouble now I think in the future I want to support Docker I just want to look at what in my opinion is better ways to run containers in production than covering Docker workflows again I think Docker is great for building applications I think it's great for running containers I just need alternatives so if you envision the foreseeable future of workflow where you continue to do Docker build and Docker run during development and at some point you do a Docker push to promote it to some production indication at which point it goes into the containers and production framework that you described here correct I think that now the Docker becomes one way of building containers I think other all I really want is I want different components I want to get back to Legos of pandering the situation and not have everything be this one monolithic where we're constantly begging people to accept patches because I want some different workflow separate these out into different components I think the development will get easier at least for production anybody else? yes maybe a slightly provocative question we have people like Courtney and Landon working on minimizing our base image is it true that one of the things we're telling things about using remote storage is that any extraneous bits that aren't actually used during running don't have to get pulled whereas as you described you can so isn't it true that if we get to the platonic ideal of no extraneous bits in the image that you're running then there's not that advantage of shared storage where containers goes away you got back to the point and I think you want to have it configurable where the admin can make that decision if every time I boot my machine I have to pull down the 50 gig executable then it makes more sense to install it locally if every time I install my application I'm just launching a patchy and it loads in some CGI scripts I can take that overhead constantly rather than dragging down the big image and I get the advantages but we all make that stuff one of the nice things about using NFS is I believe in production almost everybody's going to have the data because if you want to have a large cluster you have to use shared storage so you already have NFS set up or Gluster Assets set up so why are we picking out the executables have to be pre-installed or forcing it to always be pre-installed so we need to take advantage of the tools that people have been using NFS for 30 years for a good reason and I showed NetApp with this patch are you going to show that app? one minute anybody else? okay any comments about the presentation I'd like to hear about them as I said we're probably expanding this we go forward and we're going to sit down on the amount of time that's shown hopefully the next time I do this I'll be able to show the entire OCR ID so thank you for coming