 Welcome, everyone. My name is Antonio. I'm a senior software engineer at Red Hat, where I work for the containers team where I also maintain the Cryo project. Probably you've heard about it. And I'm also maintaining other container-related tools within Red Hat. Today we're going to talk about Cryo, which is a container runtime for Kubernetes. We'll go through why Kubernetes introduced a new API called the CRI. What were the issue it was facing? How it solved it with the CRI and how we've built Cryo from little tools and libraries. We've been building over the past few years, I guess. So I'll briefly talk about some of the container tools we've built as part of the whole Cryo project, especially the project atomic project. But this will be at the end. So roughly four years ago, the only supported runtime in Q was just Docker. And that means that at the time the interaction between the Kubelet and the container runtime was built into the Kubelet itself, like in the codebase of the Kubelet. There was a sort of maintenance burden in doing that because every new feature being added to Q required the maintainers to actually change also the interaction between the Kubelet and Docker itself. Then Rocket came and again the interaction was wired into the Kubelet itself. So the Kubernetes maintainer thought that if another runtime would come, then that requires another built-in interaction in the Kubelet. There wasn't really scalable. And on top of that, Docker, the runtime, was actually breaking Q, but every release. And there was also another issue which was extensibility. So Qube was extensible, and it's meant to be extensible, but there was no way to actually plug a new runtime at will. And so it was pretty much restricted to Docker and Rocket. So the community gathered together and they came up with this idea of removing the interaction between the Kubelet and a given runtime outside of Qube itself. And that was the time when the CRI or the container runtime interface has been created. The CRI interface is a plug-and-play API, and it means that as long as your container runtime implements the server side of the CRI, everything is going to work just fine. Kubelet is going to be able to create pods, containers, and do all the work you need to do. It's a GRPC API. And it's available from Qube 1.5 as an alpha feature. Nowadays, it's the default and even the Docker runtime is running through the CRI itself. It's a client-server architecture, and again, the container runtime implements the server side of this API and the Kubelet implement the client side. The CRI itself consists of two main pieces. One is the runtime service, which is the one in charge of the pods and the container life cycles, which includes actions like running a pod, stopping it, creating a container in a given pod, getting the least of all running containers in Kubernetes, as well as being in charge of the interaction between the containers for things like rubbing the logs for a container, exacting into a running container, attaching and attaching. For the runtime service, it's the first piece, and it's in charge of all of this. And then there is the image service, which, as you can imagine, is in charge of the image life cycle. And again, it includes actions like pulling an image, getting the status of an image to gather information like the image file size, and it's also responsible for reporting file system information to the Kubelet for stuff like the image garbage collection, like the Kubelet can request the runtime, how much space there is left, so that you can actually evict some pods if there is disk pressure. This is an eye-overview of the CRI in action. You can see on the left side, we have the Kubelet, which is implementing the GRPC client, and it talks to, in this case, there is a CRI shim. This is the default when running with the Docker runtime. The CRI shim is a demon implementing the GRPC server, and then on the right side, we do have the container runtime and all the containers. So, at that point, we've been working with the Kubernetes community and maintainers, and we thought that the status of the container runtime within Kube wasn't that stable at the time, so we came up with this idea of implementing a new container runtime, which was meant just for Kube, and which was specifically an open governance project, and nowadays, CRI is under the Kubernetes 6 organization, so we adhere to the Kube rules, and again, that's open governance for us. CRI is also, of course, an open source project. We're on GitHub. It's really tiny as far as the code base goes, so it's really lean and it's really stable because it's small. It's also secure because since it's small, we can also audit the full source code and fix bugs instantly, and probably the most important thing is that CRI is boring because we don't do anything else that just what Kubernetes needs. In CRI, we don't have any such features that Kubernetes don't need, for instance. We don't build images. Kubernetes doesn't need to build images. We don't build images. And that's really important because when you have a huge project like a container runtime, if you implement features that can actually mine stability on other parts of the runtime, that's bad, so we didn't want to do anything like that. So CRI just implements what it needs for Kube, and that's it. And again, that's boring. That's also boring because we choose to use the open container initiative specification. Specifically, we are using the runtime specification and the image specification. This makes sure that from release to release, there won't be any breakage in the way we talk to the runtimes, and that's pretty much what Kube wants as well. Kube just wants for every release to run the containers as it was doing in the previous release. So by adhering to such standards, we make sure that hopefully we don't break from time to time. And again, CRI is made for Kube and just for Kube. There is no other intended user other than Kubernetes. So there is no CRIO PS or CRIO logs or whatever. It's just meant for Kube. So to actually narrow down the scope, the scope of CRIO itself is tied to the CRI. So from release to release, if the CRI needs a new action to be implemented, we'll do that, and only that. We won't do anything other than that. CRIO itself is shaped around Kube, and that means that we know what Kube needs as far as the container runtime is. So we know that there are certain actions like getting the image status or the container status, which are recurring actions. And so we are able to speed up the code path in CRI itself to make it faster because Kube needs that. Again, the only supported user is Kube. And we will never add any features that combine stability and performance. I'll talk about that in a moment as well. But we make sure that we never break Kube in any way. And we make sure for every release to not regress in performance. Versioning is also tied to upstream Kube. So right now, 1.12 is going out of there. And guess what? CRIO is also going with the CRIO 1.12 so that it's a no-brainer to understand which version goes with the version of Kubernetes supported. Support is also tied to Kube, and that means CRIO is backporting patches as long as Kube needs it. So right now, the old supported release is probably 1.7, and we do still support 1.7 for security fixes. This is, again, an eye-lever view of the architecture of CRIO within Kube. You can see the Kube button on the left side calling through the GRPC API into CRIO. And you can see it's made of the image service and the runtime service I've talked before. For the image service, we're using a library called Container's Image, which is the one in charge of pulling images, listing images only when you give a node and stuff like that. And the runtime service is the one which is actually running the containers through the OCI Container runtime. We make use of library slice, the OCI Generate, which is a library to create configurations for OCI-compatible runtime. So we make sure that we can generate one configuration for every OCI-compatible runtime. So you can come up with your own container runtime, and it should just work. For networking, we're using the CNI. Probably you're already familiar with this if you use Kube. And at the lowest level, we use a library called Container Storage, which is the one in charge of, for instance, exploding the image into layers on a copy and write file system like OverlayFS and then run your container. Above you can see, those are two examples of two pods running with cryo. So there is an infra container because neither cryo nor Docker, for that matter, is a pod concept. So we did the same as Docker did, and we implemented the pod concept as an infra container, holding namespaces and cgroup, and then we spawned the actual application containers inside the namespaces and cgroups for the infra container so that we achieved the pod concept in that way. And for every container running, we also have a little tool called common, which stands for container monitoring. I will dive into that in a moment. So the heart of cryo and Kube itself, it's running containers. So we do make sure that we can run almost every container out there if it's OCI compatible. Right now, our default is Renzi, which is a project and a runtime inside the open container initiative repository. But we also support new runtimes like other containers, which was previously known as clear containers. And for every runtime, we also make sure that you won't break Kube in any way. And, for instance, the Cata containers, maintainers reports to us for every request if something goes wrong so that we won't merge any features that can actually break the other container runtime. Container storage is the library I've talked to you before, and it's the one in charge of managing layers on Kube on right field systems. Cryo, by default, uses overlay FS, which is pretty stable, and it's our default. Container storage is also, you can view it as the storage drivers for Docker, like device mapper overlay. Then we do, again, the containers image library, which is a library, again, for pulling images. This library is probably where everything started because we extracted it from a tool I'm going to talk about later, and we embedded it into our downstream Docker as well, so it's really battle tested. You can pull images without any privilege as well, like today, if you want to, with other tools, if you want to pull an image, it actually needs to be root. That wasn't what we wanted, so as part of developing containers image, we make sure that we can pull images as a privileged user, as it should be. And containers image is also, for us, has been the playground for adding and experimenting with new features. Cryo itself, today, can run with what we call simple signing, which is signature verification for containers image, so you can enable that in Cryo if you want it, and that has been possible because we actually implemented all of this in container image, and we're using it in Cryo. The other library that we use, it's a library extracted from a tool, which is the OCI runtime tool, and as I've told you before, it's just a library that generates OCI-compatible configuration so that every runtime can read it and create containers in pretty much the same way as other OCI-compatible runtime. The CNI is the container network interface, and you can come up with your own network stuff if you want, as long as it implements this interface. Cryo is able to run with most of the plugins out there, supported in Kube, like Flannel, Wave, you name them, and also the OpenShift SDN. And then we do have this little binary, which is called common, which is the container monitor for every container. And as the word suggests, it's in charge of monitoring the whole lifecycle of a container. It's basically in charge of starting the container itself. So when a request from the Kubelet comes, we do spawn a common process that in turn spawn a run C for our default runtime, a run C process, and take care of it for the whole lifecycle of the container itself. It's responsible for logging, for handling the TTY for the container, serving a touch client, detecting and reporting out of memory condition, and probably most importantly, it's also common is the parent process for the container itself. That means that Cryo can go down entirely, but your containers will still keep running and serving requests. That hasn't been the case before with other runtime as well, whether if you updated the runtime in question or Cryo, it will go, it will bring down all your containers. So since we do have this little binary in between Cryo and the container itself, we are able to upgrade Cryo without any downtime for the containers running. This is a zoom on the pod architecture. We're running with run C. You can see this whole box is the pod itself, which shares the IPC net and PNA spaces and C group. And these are the application containers that joins the namespaces of the infrastructure containers, all the namespaces. And for each container, we do have a common process. This is the architecture, which is a bit different with container and Cata containers. Since this is a virtual machine, the Cata containers developer implemented a shim in between the common process and the application containers and the pod itself. So this is pretty different than this one. This is how it is in Cata containers. And now I hope everything is going to go well. I'll show you how transparent was to actually implement, well, use Cryo with Qube, since you'll have the same experience as with other container runtime you're already used to. So I do have a one node Qube cluster running on my laptop and draining all my battery power. You can see here we have the node. It's a 1.11 node, so it's the lightest. I do have the Qube NNS, which is in charge of the networking for the whole cluster, and the dashboard running. And we're going to run an HTTP container with the image HTTP alpine. And we should be seeing it running. You can see it's running. I'll show you in a moment that we're actually running with Cryo and nothing else. I don't know. I'm using this terminal for the first time because I thought it was better. Can you read it? Not at all? Are you able to see it now? I'm going to run this, you know, okay. So we're going to run this again with the nodes, all the pods running. We can see that the HTTP pod is running. And I'm starting a new one here. I do have two now. So the last one, you can see it has been started successfully. We can test it by curling the IP address for that pod. We can see it's working. We can curl a not found page for the sake of grabbing the logs and see that it's working over there. You can see the slash as gconf 2018. We can also Qube CTL exec as you would normally do with other container runtime. And you can see HTTP is running. You can also run interactively. And it's working. And now just check that we're running with Cryo itself. You can see that the output of Qube CTL describe is showing that we're running with the cryo slash slash, which means that we're running indeed with Cryo. And there is no Docker running at all. Rather, all our containers for every pod that you've seen in the output of Qube CTL get pods is actually running through RunC. And this is our container, the HTTP container running. We can check that is the one by looking at the PSOX output. You can see it's still HTTPD. And for each container we're running, we're also running a system, the scope for it. So for the sake of blogging and monitoring it, so if you prepare Cryo dash to the container with the scope, you can actually, you can see that it's integrated with system D itself. So we can see all the processes running for it, the logs streaming to the system, this scope as well. And we do also run a scope for common as well. You can see it here. As you would normally do with other container run time, of course, everything is going to work normally. So here's the dashboard, which is working just fine. You can see we do have the two pods running. And since we do care about performance, in Cryo itself with implemented Prometheus, I know this is really small with implemented Prometheus metrics so that if something goes wrong, or if something isn't looking that good, someone can go here and just try to understand what's going on. So back here. So the status as of today is that the CRI interface is implemented at any time. And that means for every release, we make sure all the CRIs implemented. So if any new actions are is required by the CRI itself, we implement it before releasing it. We do have maintainers from and contributors, more than 90 maintainers, more than 90 contributors from many companies like Red Hat, Intel, ABM, you name it. You can run Cryo with CUBE with CUBE ADM or Minicube. There's support for mixed workloads. Nowadays, these are called runtime classes in 1, in Kubernetes 1.12. That means that you can have certain pods running with RunC, which is runtime just not virtualized, whether you can have other pods that can run with run times like other containers so that you have a virtual machine for a pod. Cryo itself is deployed to our open shift online cluster, so we do have a pretty good user base that it's testing on it. And the package for Cryo is available in Fedora, Ubuntu, and it's really easy to get started. We do make sure that we don't, as I told you before, we make sure that we don't actually break CUBE at any time. We do run more than 1,000 end-to-end tests for every request that allow us not to break Kubernetes in any way. We do also run the open shift tests, not as part of every request, but we do run them, which is a superset of the CUBE end-to-end test. We do run a conformance testing, which is the CRI test. We do have our own integration test, and before every release, we also run a performance run to make sure that we don't regress. If you want to get started with Cryo and CUBE, you can use MiniCube with this command line. I'll upload this slide so that everyone can actually take this. Or if you're running from a local setup, you can pass some environment variables to local op clusters and get going, as I'm doing here on my laptop. To debug what's going on with Cryo, and especially with the whole CRI interface, the community, the upstream community in CUBE came up with a new tool called CRI CTL, which is a small binary that just talks to the CRI end point of any container runtime, so that in case something goes wrong on a node, you usually SSH into it and Docker PS, Docker images, Docker exec, you can pretty much do the same with CRI CTL. So this is a debug tool. And we're going to move this project into the Kubernetes organization itself, since it's crucial for the CUBE project. So everything in Cryo has been made possible by some libraries, like containers, storage, and over time, even before starting Cryo itself, we created some tools that are working with containers in general, so that we created tools like Scopio, which is a tool to run, to play with containers image, like you can pull images, as I told you before, by using containers image, which is the one scope you're using, so you can actually pull an image without being root, as it's desirable. There is no demo running with pulling an image, so it's just straight command, and you'll have the images on your roast, or you can sync up images between registers. Another tool that we've built by leveraging those little libraries that we're using in Cryo itself is Builda, which is a tool to build container images. And again, we don't have any demo running Builda uses a shell-like syntax to build images, so there is still support for Docker files, but of course, we know that people are still using Docker files, so it's still supported. And then we created a tool called Podman, which is just running containers as Docker does, like today you can Podman run something. And all of these three tools, maybe a Scopio, are going to be demoed by Dan Walsh later today, so make sure to come to the session. So overall, the Kubernetes community came up with this idea of the CRR interface, and we leveraged that by creating a, probably, I'd say the best runtime since it's the only runtime made for CUBE, specifically for CUBE. Any other runtime as other supported user, whether Cryo is just for CUBE. And so by also leveraging all the libraries, we use it into Cryo. We also built many tools for the whole container's ecosystem, building up from libraries, like containers, storage, all the open containers, initiative libraries. So the roadmap, very briefly, is to, well, not the first one, but we aim to be the default in CUBE upstream, since we think that we are specifically for CUBE. We will keep pace with upstream CUBE, and that means tracking and supporting all the Kubernetes version out there. We moved out the Kubernetes incubator organization to something called the Kubernetes 6, which is a repository for six specific tools. So Cryo is in there now. It's not the default container runtime for OpenShift, but that's on our roadmap. But it's already the default container runtime in OpenSUSE and microS. So if you install and use microS, right now you get the kubelet and the Cryo runtime as default. And of course, we are at having more adoption overall. If you want to get involved, make sure to go to these links, and we do have a free node channel, Cryo. And we usually hang around in the signaled channel and Cryo channel in the Kubernetes Slack, with a website if you want to see it as well. With that, thank you. If you want stickers, Vincent has stickers right there. I ran out of time, so I guess I'll take questions outside.