 OK, but the normal stop time is 10, 15. OK. Yeah, that's cool. I'll try to keep it on track. Yeah, all right, cool. All right, sweet. So falling Dan's is going to be hard, because he was awesome today. He was on fire. But I'm going to dig a little deeper into a lot of the things he talked about, but just go a hair deeper into the standards and how they work and why. And essentially, if you think about it, backing up and thinking about it as a whole, the problem with the industry is if we don't have a standard, you can't consume, for example, a perfect example is say you want to use a registry server to Amazon. Say you want to build locally, build an image, push out to Amazon, push to Docker Hub, push to your own built-in registry. You have to have a standard that everybody can comply to so that you can have true portability. Like the idea of the cloud, right, is I can move things back and forth and make it easy. And so it's a mixture of products provided by companies, community projects created by individuals and communities, and services, essentially, that are also by products, or by companies. And so if you don't have that, you have bags, box, barrels, and crates, as I used to say. They're all different sizes, and trying to put them on a ship, you have to stack them all. And so really the idea, literally the good analogy is basically shipping containers. And people don't think, this is a conversation I've had a lot. In fact, this week I had it with the SyLabs guys, the guys at De-Singularity, because they have a different format, and we were discussing where should they plug into the ecosystem, right? Like this is a conversation that if you put your head down for five minutes to start working, and you lift your head, you're like, oh man, where do I fit into this ecosystem? Like do I build to the CRI spec? Do I build to the OCI spec? Like where do I fit in, and where don't I fit in? Should I use the Docker CLI? Some people have integrated with the Docker socket, things like that. And you end up in dangerous situations where you're like the world moves and tectonic plates shift, and then now your stuff doesn't work right, because something broke with a version change. And so really what I want to dig into in this talk is kind of like, where should you integrate? Where do these standards protect you? Where don't they protect you? What do you really get from all these? And I mean, Dan did a great job, because we've built an ecosystem of tools around, would build a scopio, podman, that actually work using a lot of these standards. So we have a lot of expertise in figuring out where these things should fit and where you can invest long term. And so to have this healthy ecosystem of things where people can go off and innovate and do things, but then lift their head back up, and their tool still works in the ecosystem, you pretty much have to have well-defined interfaces and well-defined standards. And so I'm going to dig into about five of them, actually. So the solution, like I mentioned, is open standards. You have to have well-defined interfaces and open standards. And so you need to be able to have this happen, right? I need to be able to swap any one of these out in certain places, right? Like the world started with Docker, they invented the concept of a container engine, at which yesterday I talked about is really basically a giant proof of concept, right? That word Docker means so many things. It's a company, it's an API, it's a demon, it was a format for images. It was also an ecosystem of images out on Docker Hub. And the only way to kind of sell the world on how to do containers in this new way was to kind of build this giant proof of concept. And that was the genius. But the downside is now four years later, you've got everything munged into this giant proof of concept. And that is bad, because it's really hard when you want to make a small change, like the damn brought up example of Rocket wants to integrate in Kubernetes, but Kubernetes is hardwired to Docker. That's a bad scenario, right? Or you want to have a standard spec on what images they pull, obviously. So we've chipped away at different pieces of this to make it so that these things are plugable. And now today, 2018, we're actually in a really good spot where you can plug these different things in. And essentially, every one of these does the same job. And I'll dig in deeper and where they all fit in and why they work. So a long time ago, several years back, I went back and kind of revisited the original, like four years ago, four and a half years ago, whatever that was, when Docker first came out, a lot of people made the shipping analogy, right? And I actually went back and dug into some of the ice up standards around actual shipping containers, because they're apropos to the software containers. And one of the ones that were actually really interesting to me was ISO 1161, hook dimensions and strength. So I just pictured, right, there was some point in history where some guy was operating a crane, trying to lift the container and the top tore off. And who knows, the container fell out, and all the stuff fell out everywhere. Because nobody had ever standardized, like, what is the strength, the potential strength of the corners and the hooks, and things like that? And really the same, we went through that same process in containers in that now we have an actual standard for what the images look like. And this also highlights not just the image, but the tooling, so the cranes. I want to be able to buy one set of cranes that can move these containers around. I want to invest in those cranes, because I'm going to buy them for 30 years. And so the business problems that are associated with that become a lot more obvious when you're building physical stuff, because software feels like you can change it really fast, but in reality, nobody wants to once they put it in production. So you want to adopt a standard, because I still want to get value out of whatever I've built in five years or three years. I have some upstream stuff that I've built, and I'm lazy. I don't want to go change it just because something changed. I don't want to have to constantly rewire it. I just want to build to the spec and say, OK, it should work. So that's what I'm trying to highlight by making this analogy with the actual real shipping container. And so these are the five specs that I just want to dig into. And they're actually similar. I would say these two, on the end, are really well defined interfaces. They're not actually, or I'd call them, community-driven kind of standards. And then these are actually standards, body standards, these three. So there's a spec for images, like the actual image format on disk, and when you're moving around. Then there's a standard now, which is recent, for distribution. So that's like moving it between registry and cache or between registry servers, which is what Scopio works on. And then there is the runtime specification, which actually explains, and I'll dig deeper into it, but it kind of explains how you take all the stuff that's in the container image and turn it into a running process and all the translation that happens in between talking to the kernel. CRI does the same thing on the network side in a very similar way, except that there are pluggable binaries that can do the work instead of run C as kind of the standard that most people do, although it's pluggable as well. And I'll dig into that a little bit. And then CNI is what I'm talking about. And then CRI is really an interface between the Kublet and the container engine. And actually, that one came up the other day. I mentioned the anecdote of where should the SyLabs guys from Singularity, they have a really cool container engine, essentially is what they have. But they never were quite sure where they fit in and where they wanted to play in the space. And they play in HPC. And in HPC, they built their own image format that's not OCI compliant. So now they're kind of trapped in that world. But that's good and bad. It allows them to go innovate, and they can do things that maybe HPC users want. Maybe HPC users do want to ship the data around with the container image, which is what they do, essentially. OCI images do not do that. OCI images are just a code. The data lives locally. But the cool part is with CRI, two different groups can build a CRI compliant container engine. And then Kubernetes can just talk to it, or CRI CTL, which I'll dig into, to just talk to it. And you can list the images, see them there. Even if they're different formats, doesn't really matter. So the beauty of understanding these is that you can understand where your project should fit in, where you should build, where you can rely on things that you know that your stuff will work. And then as an architect, somebody building an environment knowing where these work, you can kind of know which projects you want to investigate. If I'm looking at a project, personally, I would be a little bit apprehensive if Singularity didn't comply to the CR interface, because there's probably places where I'd want to be able to call it through the standard interface. So I'm going to focus on CRI, CNI, and OCI. Those three main sets of standards, if you will. So I showed this slide yesterday, and it's the only one I'm going to show from my other one. But this is kind of the full money of what a container engine does. And I walk through it kind of, if you think about it, it took me a long time to boil down what is actually happening in a container engine. And I'm going to dig deep, deep into this in some future slides. But it's really a conglomeration of options that are specified in the container image as metadata, options that are specified as defaults in the container engine, user overridden options that they pass as command line options, or either that, or through YAML and Kubernetes, which then gets passed to the kubelet, which then gets passed to the container. But in a nutshell, it's image-defined, engine-defined, and human-defined options all get combined into something that then gets passed to Run-C, which is an OCI runtime standard, and then the CNI plug-ins, which are also binaries that take a config blob that's very similar to what gets passed to Run-C, but only for the network side. And so if you think about it, this is what a container engine does. And then on the far right side there, I show all the image layers come together, get smashed together, and in a nutshell, you can think of them. They're overlay. They're actually a copy-on-write layer added to the very end, which I won't dig deep into. But at the end of the day, just think of them as container image gets smashed down into a single layer that then gets mounted as a root FS. And again, then all these options come together, and that's kind of how a container gets ran. So now I want to, I talk about it like I kind of dig in a little bit deeper into where the standards play. So I showed you CNI, OCI runtime, OCI image, and actually distribution is really up there, too. I have an update of this drawing. And then here's where CRI works, right? So CRI is, Kubernetes is talking through the CRI interface to CRIO. So CRIO is a demon that it understands the CRI protocol, essentially. And you could plug in anything there instead of CRIO. In fact, there's commonly something called a Docker shim where it speaks CRI and then speaks Docker on the other side and then talks to the Docker engine. Container D has a new branch or whatever called container D-CRI that speaks this natively. As I mentioned, the Singularity guys are thinking about plugging in Singularity there as a CRI interface. So the cool part here is you could kind of see, since you're protected by, you're kind of, notice how CRIO is boxed in here. It's boxed in by standards on every side. So it's boxed in by CNI, the OCI runtime specification, the OCI image and distribution specifications at CRI. So you can plug CRIO in and out. There's nothing stopping you from moving some other CRI compliant slash OCI compliant demon in and out of there. And that's the beauty of these essentially standards. And then this was the one I came up with for Dan. He requested that I come up with the one that shows where CRI CTL and Podman come in. So this is a little bit deeper version that kind of shows, now there are some other things going on below the covers. So we have, at this layer, we have CRI interface. And that's kind of what defines the interface to the CRI compliant container engine. So whether it's a SHIM or it's natively supporting CRI, you can plug in or out that container engine. And one of the sides that you care about is the robots, Kubernetes. That's a scheduler coming in and talking to the container engine. But on the other side, we have something called CRI CTL. Think of this as the human interface to the container engine. And this took a while for it to click for me. But if you want to have a pluggable container engine, you need a standard API that robots can talk to. But then on the other side, you really need a utility that humans can become familiar with. So if I learn how to use CRI CTL, if I just learn how to use CRI CTL images and CRI CTL PS, you'll notice that the interface is quite similar to what Docker looks like, except for there's no run. Because it's assumed that with CRI interface, the KubeLit will be basically sending the runs to the engine. But in a nutshell, it's a pretty familiar command line interface. But you can now plug out that container engine pretty transparently to both the robots and the humans. And so that's the beauty of CRI, CRI CTL, and then KubeLit talking CRI to a CRI compliant container engine. And then one level down, many or most CRI compliant engines will be pulling OCI compliant images from OCI compliant registries through the distribution and image spec. The one caveat there, as I mentioned, for example, the singularity guys, they could plug in a non-OCI compliant where it goes and pulls its own container images that are a completely different format, which is fine. If that's what you want, as long as you understand that's what you want because you want some other value from that container engine. But then all rules below this might not apply anymore. But in the cryo world, this is what it looks like. So you can pull OCI compliant images. Then one more layer down, you've got container images and container storage that kind of mandate in a well-defined interface how things get laid out on disk and how the images get stored. And to look at that, you can actually do it like a CRI CTL images and see what images are cached locally. But if you want to tag images and play around with what's in the cache, that's where something like Podman or Builder or any of the ones that can talk to the local container cache. You can now go muck around with that stuff at a lower level if you want to. And then at the end of the day, though, as I showed in the last slide, everything gets handed to RunC. When it gets fired up, you're just passing it a root FS in that config.json to RunC. And then RunC talks to the kernel on a standard interface. And as long as the kernel features are there, those things get turned on, whether it's C groups, SeLinux, S-Fert, which I tracked down the other day and I have a whole deep article on that. And whatever clone arguments you want to pass, so whatever namespaces you want the container to run in, all that's defined in that RunC interface. All right, so what is a container anyway, though? I mean, I think Naolin mentioned he's going to go into this deeper. It's not rocket science. It's a black box, and so it's a little bit intimidating when you first start. And I have to admit tracking down a lot of this in the code and asking Naolin and Dan and a bunch of the guys on the Runtimes team different questions. I realized there was a class of people that understood what was going on under the covers. And it was like five, 10, 15 people in the world. And I was like, we need more people to understand this. Because as architects, when you're building these environments, I mean, I was a long time sysadmin, way too long, I have PTSD from it. But sysadmins, architects, people like that need to understand this so that they realize, oh, OK, some of this stuff is not that hard. I can feel a little bit more warm and fuzzy about where I can swap things in and out and feel good and actually get the value out of things I want. So there's places where you might want to swap out the Runtimer or the CRI engine, et cetera, et cetera. But they're not magic, right? At the end of the day, it's just a bunch of metadata and a bunch of files. I mean, that's all it is. That's all container images. And the metadata is really, if you think about it this way, it's a way for the human that is building the container images to express to the person that is consuming the container images, hey, here's how I think you should consume this. Here are some sane defaults. Here's the basic command that should run. Like this is a memcached container image, so the command is going to be memcached. Like that makes sense, right? That's pretty obvious. But as container images get more complex, there's other things that the container builder might want to express to the container consumer. And so you really think about this. This is a human interface. This is for humans to communicate with each other using code. And really, if you think about what these containers are, they're a format for collaboration. And the analogy that I talk about is, if you think about hardware, hardware was physical letters, right? I used to get servers from HP or IBM or Dell, and they had to ship them to me, just like a letter. And that was a very slow process of collaboration. Like if I got the server and it had the wrong OS on it or it was not updated to the right level, there was all this stuff where I'd have to call and update the drivers, blah, blah, blah, all these nasty things. And it was very slow and tedious. It was still decent collaboration. It was still more open than we had in mainframe and even Unix days, but it was still slow collaboration. Then kind of the next level I joke is email, right? That was VMs. Email and VMs, they're better. It's not awesome, though, because still you have to go, if I want to collaborate with another human being with a VM image, like the virtual appliances that VM were tried and failed, I have to go pull a 10 gig image from an FTP site or from SCP from somewhere. And whenever I want to collaborate with another human being, I have to mark up changes in that user space, in that VM, change them, save them, and then they have to upload them back to them. That's terrible. That's like email. It's like emailing back and forth LibreOffice documents and using the markup. And if you send it out to five people and each of them make changes, you're now screwed because I can't actually get all five changes. I don't know who made changes first and there might be collisions and all this. But then when you get to containers and you have a format where you can actually define the layers and you can actually see the diffs between the layers, the container image layers. Now we get to a point where the files portion of this and the inputs portion of this are really collaborative. And so we can really collaborate on what the container builder and the container consumer want to do. And so in a nutshell, that's all this is doing. And so there's two basic things, right? There's the config.json, which I mentioned here. And then there is the image layers. And so the image layers get all combined together through a graph driver that basically builds this root file system in that config.json. Then that gets handed off to RunC. And then I show a little bit more complex version of it. So this is the OCI image specification. It defines that you put these things in that container image. The container engine explodes that stuff, does all the work to actually make it into a root file system, builds the full config.json, hands that config.json and root file system off to an OCI compliant runtime, which could be RunC, Catech containers, rail car, G there's a whole bunch of tools that are OCI compliant runtimes. But as was mentioned, RunC is probably 98% of the world is using that. And then that calls the clone syscall, which then talks to the Linux. Typically, Linux operating system could be Windows. Windows has their own syscall layer thing. And then here's the full money shot, as I mentioned. So what are we doing here? There's a container builder. There's a container user. So this is the consumer. This is the builder. And what this person is doing is they're saving a bunch of inputs to kind of communicate, as I mentioned, to the end user. What do I think you should do with this container? And it could be, like I mentioned, a memcache d. And maybe I'm going to set the entry point to memcache d so that this thing always works. And I'm going to set a default username and password for MySQL server, whatever. There's all kinds of things that you can communicate to the end user here. Or maybe you're going to communicate to them that they shouldn't use this without a password. So it should fail out unless you pass an environment variable. Or with Microsoft SQL server, their images won't run unless you agree to the EULA. So you have to put EULA equal Y. So this is a way for the container image builder to basically pass information on to the container consumer in a really collaborative, cool way, which is way better than VMs. And I give some examples. They can use to build a Docker build, Umochi. There's all kinds of tools that are now building OCI compliant images. You can do it with Podman and a Docker file. And then at the end of the day, what gets happened, what happens is you end up with essentially the equivalent of the config on the image side, JSON file, shoved in or registry with a bunch of image layers. And they're bunch of tar balls, is all they are. But I show the daemon or the engine, and it really is what I should change this to, because it used to be only daemon, and now it's no more big fat daemon. So I need to change that to really the engine. The engine now has a job. It has to interpolate what's going on in some of these environment variables. Which ones do I want to obey? Which ones don't I want to obey? Which things do I want to add to myself? Like a perfect example is if you don't specify a sec-comp file, the engine has some defaults. If you don't specify a dash dash privilege, the engine has some defaults of what it does with security. In a Red Hat world, S-Fert is used by default. So it will automatically generate an SE Linux context that's dynamic and unique to that container that fires up, and then it will start that process in that context. Those are all defaults that are specified by the engine itself. And then the engine will build the config.json, I mentioned, and essentially the user at that point, though, when they go to interact with the engine, they can override some things. So I give some examples here. kubectl run environment A will be. So even if you're interacting with it at the orchestration layer, you can override some of these environment variables that might be buried in the container image somewhere. But again, it's kind of a way for the person that built it to communicate to the user, hey, here's some defaults, same defaults, but maybe you want to override them because it makes sense. And then again, I already mentioned, it kind of gets handed off to run one of the OCI compliant runtimes off to the kernel and then run. So I'm going to show just a quick example of this, which actually I already have it running. So watch this. So the beauty here is we can docker run bash. We all understand how to do that. cat slash etsy red hat release. It makes perfect sense. We've probably all seen that. But then let's do this with, here it is. Actually, let's just do it by hand. So podman run dash it rel7 bash. Look at that. Same thing, right? Completely command line compatible. Pretty familiar, right? I think most of you would know how to run that. How many of you feel good with that? Pretty easy, right? Now, something I didn't talk about is that now this CLI interface is not covered by a standard. Neither is the Docker API. So there is one gap there, right? Like if you start, I'll tell you, the one place I would recommend not integrating is directly against the Docker socket or against the demon, where you're actually going back and forth in native Docker API. I wouldn't do that because they're completely not protected then. Then that can change and break. We can protect you with a tool like podman for the CLI. And then behind that, we can protect you with the image format, the distribution spec, the CRI interface, the CNI interface. These are, you're pretty safe once you get into that. But there may be a few places where people have integrated directly, natively with the API. That's the one place I'd say steer clear from, doesn't make sense, don't do it. Does that make sense to everyone? So let me show you what I did here. So this is basically what I did, right? It's that simple. I swapped out Docker engine for podman. Pretty much exactly the same, CLI compatible. You won't notice a difference. You're getting the same output. You're getting the same things happening, config.json, root file system getting generated, RunC is running it. Probably 90% the same code, except no demon. And here, I'll give you one other piece that I want to show you is, watch this. So we'll go into another terminal over here, and I'll show you what's going on. So ps3-ac. So here's what's happening. You see, it's actually really easy to track down with podman. SSHDman, my original bash that I was in. I ran podman. Here's that bash. Boom, some things are getting fired off underneath podman. That's so much easier than trying to do with Docker. If you've ever tried to figure this out with Docker, you could strace this and see what syscalls are happening inside the process. You have a lot better understanding of what's going on in the sub-process. With Docker, it's a pain, because you have to strace the running demon, basically mess with the CLI, send some commands to the demon, try to figure out what the demon is firing off. And I've actually ran into this problem where it's actually kind of a nightmare, especially remotely, to figure out what's going on with it. This is a much more elegant interface. This is better for things like another use case that came up last week was the container executor. So they didn't want to use Kubernetes. They wanted to use Yarn. It was a Hadoop environment. They have their own scheduler. They've already built their own jobs. They probably have a ton of investment in that. But they were using the Docker demon to fire off containers. That's an inelegant solution, really, if you think about it. And what was happening is that the Yarn container executor was firing up containers and shutting them down so fast that it was corrupting the Docker demon. Because it's a demon, and it's trying to keep track of all this nasty stuff. And so they were firing up 1,000 containers, shutting them down real quick, and that can end up corrupting the demon. In this scenario, it doesn't happen. This is just cloning and running, essentially, the coolant of a fork and running another process. So there's no client-server interaction. There's no race conditions that can happen where you send a command to the demon, and oh, it thinks it's already shut down, but it's not shut down. And then race conditions happen where it tries to send another command to send it shut down. They were running into all kinds of nasty stuff where it would get corrupted. And so we had them try Podman. And so far, they love it. And so this is a perfect example of where they were protected because they used the CLI. They weren't tightly integrated against the API. And everything else below that, all the container images and everything else were protected by the OCI image. So they could fire up the exact same images with the same command line interface, but in a very different scenario with much better results. And those are the kinds of scenarios. People don't care until they do, because usually what happens is you end up in some kind of nasty problem where something breaks like that. And then you're in a deep pickle. And that's where you start to care. So why would you want to do this, right? Well, because I mean, this has already went through pretty well by Dan. But fully community-driven, open-source set of tools, small, nimble core utilities that are standalone, can be used to build, can just be used to run, can be integrated with other schedulers in random ways, like the yarn one. Better security can run as non-route. FIPS is something that we're working on now that we've enabled the possibility of, because our underlying Go Wang is linked against open SSL dynamically so that it can be put into FIPS mode. All kinds of innovations like that. You start to be able to break all this stuff apart. And you can really start to mess with some of the underlying things. And again, you don't realize what you need until you actually start to dig into the use case and actually start to build out real production stuff. And that's when you get burned by having everything buried into a giant POC, where it's hard to make changes to this big ecosystem of things that are all glued together very tightly coupled. And so I would just call it action. Here's some good places to go check some things out. Go check out Cryo, Scopio, Buildup, Podman. I didn't put Podman on here yet, because this is an old slide. And then go check out the upstream work in these libraries, too, because you can kind of see what we're doing and how they're leveraged in these tools. And so I'm actually ahead for once. I guess I flew through that. But I guess I'll break and let you guys have any questions. No questions? I'm surprised. I don't know what you tell me. Question is, can you run Podman as non-rude? The answer is yes. So that's another thing. Actually, that brings up a good point. I'll rant about that for two seconds. So I had a guy beat me up on a mailing list one time about, well, I can run Docker as non-rude. I'm like, no, you can't. Because you're running a CLI as non-rude, but it's still talking to a demon. And essentially, you have complete root access. So you can run a dash-dash privilege container and have a complete root mouth of a local file system, do dash dash v, your dash v space slash colon slash mouth local file system completely corrupted to hell. You are not preventing anything by doing that. It is an illusion of, it is a difference between the model in your mind and reality. And your threat model is off if you think that you're running Docker as non-rude. It is always rude. But I saw a question over here. Oh, sorry. You had a question? And then he had one too. Yes. Thursday's dojo, George Burke has presented Kubernetes in 15 minutes on Santos. He said install Docker in his instructions to get that done. Would it be possible, from speaking as an office, to do that without using Docker? Yeah, you can do it with Cryo very easily. I used to do a demo during this talk where I would actually swap out Cryo live and show how Kubernetes doesn't even care. And actually, we'll keep track of the containers, see that they're not running and restart them in Cryo. Because to it, it's just like pulling that CRI interface looking for are the containers running? They're not up. It'll try and start them again. So it's actually very, very easy. It's one config line change, basically, is what I demoed. And just installing a package, essentially. So let's suppose in our companies or any company, you have a private-owned infrastructure, cloud, hybrid cloud, and whatever you have. So you have deployed these flows in this environment where you have a mix of solution because I feel this more close to private-owned and not to hybrid or cloud models. So this works in every environment or in a combined environment. I think that's the beauty. Is that a question? Yes. I don't quite understand the question. But the idea is, yes. Say Amazon has a registry, perfect example. They have their own registry, right? And if there's no standard, how do I know that I can push my container image into the Amazon registry and then pull it within Amazon? Sometimes you don't want to set up your own registry server. But you want one that's local on that network so that it can fire up quickly. So you have 1,000 container hosts out in Amazon. And you don't want to set up your own registry. You just want to push into theirs. These standards are what protect you that make sure that will work. And that's the beauty. Maybe I just want to run a registry for a few minutes to go cache locally, pull an image, and then blow it out to all these other container hosts locally. That's really nice to be able to do because I can speed up distribution real quick for, say, a batch job that I just want to run. So that's kind of, this is what protects you to do that. Otherwise, you would have to set up your own registry and make sure that it's compatible with everything. And it would be a pain in the butt. Any other questions? I think we'll drop. So, all right, well, thanks. Thank you so much.