 Hi, welcome everyone. So today I'm going to talk about Cryo. I'm Rinal Patel. I'm the lead developer of Cryo. And I'm also a maintainer of RunC. I'm the OCI runtime specification. I work as a software engineer at Red Hat on containers. So let's go back and ask ourselves why Cryo was created. So if you look at the history of Kubernetes, in the beginning Docker was the only supported runtime. And then CoreOS came along. They announced Rocket. And they wanted to integrate Rocket into Kubernetes. So the Rocket code path was another code path in Kubernetes. And then adding and maintaining runtimes was becoming a burden. So CRF, which is the short form of container runtime interface, was introduced in Kubernetes. So Kubernetes can decouple from the container runtimes. It's basically a GRPC-based API. So Cubelet implements the client side of the API. And anyone who's interested to plug in a new runtime can come in and implement the server side for the use cases. So the GRPC has different services. And the CRI consists of two services. The image service and the runtime service. So we'll go into some detail about what that means. So why Cryo? So when the CRI came out, we asked ourselves, is it possible to create a minimal runtime using just standard, pluggable components to satisfy the CRI so that it doesn't break as much and it's stable. And you just implement the CRI in Kubernetes is happy and your workloads are stable. So what's the scope of the project? Scope is tied tightly to the CRI. Kubernetes is the only supported client. So there are no features that aren't required by CRI. Whenever CRI adds any new features, any new APIs, we implement them. That's it. And we maintain support for password versions of Kubernetes. So we have branches corresponding to each version of Kubernetes that is supported. So let's take a look at the components that make up Cryo. So the first question we need to answer is what exactly makes up a container? What do you need to run a container? Whenever people think of containers, they think of pulling an image using docker pool. I'm running it somewhere. And what is an image? Image is the bits or the bits that make up your container. So you could have a busybox image, you could have an HTTP image, you can have an nginx image. And these images are basically binaries that are stored in some registry made up of different layers. And you need a way to manage these images to pull them down onto your system so you could run the binaries in the image. So we created this library called GitHub Containers Image. So it originally came from a tool called Scopeo. Scopeo's original scope was reading the metadata about images from a remote registry. And then the scope increased. And we thought, why not just use it for pulling images and supporting various transports? So you can pull an image from one registry and copy it to another. You can support the OCI image format. You can copy image into an OS tree, local store, and so on. So it also supports image signature verification. It supports the OCI image specification V1 that came out earlier this year. And most recent project that uses it is the Google Container Dev tool. So once you get the image, you need a root file system for your container. So image is a bunch of layers. And the next bit is, how do you create a root file system from it? So typically, we use copy on write file systems for exploring an image into a root file system. And we need a library for that. So Pradhat contributed a bunch of drivers to Docker. And we thought that that was a good library to take out and make it reusable for everyone. So we took out that code and added support for new features such as NFS. And that's a library that we use for pulling down an image and converting it into a root file system used for the container. And the most important bit, how do you actually run the container? So OCI came out with version one of the runtime specification earlier this year with RunC as a default implementation. And RunC has been used in Docker for quite some time. And before that, it was lip container. So it's proven code base. Works, it's stable. So we decided to support any OCI-compatible runtime in cryo. So what does a runtime need? It needs a config.json. The config.json has the settings for the container. For example, what process is going to run on it? What are the environment variables? What capabilities you give it? What are your second settings and so on? Basically, the settings that are translated to system calls in the Linux kernel to eventually launch a container. And runtimes such as RunC implement, understand and implement the spec to execute your container. And so Intel came and they implemented clear containers, which is also OCI compliant. And with minimal effort, we were able to plug that in because we support OCI-compatible runtimes. And so Cata containers was announced yesterday. So it will be supported soon in cryo. So we also support mixed mode, where on a single node, you can run some containers in RunC and others in clear containers to the use of pod annotations. So we have something called as a trusted runtime and untrusted workload runtime. So if you want more security for your workload, then you would say that clear container is the runtime for untrusted workloads. And then cryo would read that annotation and say, OK, this is untrusted, so I should be using clear containers instead of RunC for this workload. And then finally, there's another small library that we use for generating the config.json. It's under OCI runtime tools. It's an open containers project. For networking, CNI is used for setting up whatever plugin is configured. CNI has become the standard for container networking. And most companies that are in that space have plugins that support CNI. So we have tested a bunch of them and pretty confident that any CNI plugin can be plugged into cryo, and it should just work. And then finally, we have this monitoring utility called conmon. So conmon is responsible for monitoring the container process and detecting when it exits and storing its exit code. It's also responsible for collecting the logs from the container process and storing them in the CRI format. So CRI defines the format for how the runtime should store the logs so that Kubelet can understand them and interpret them. And also, when you want interactive terminals, conmon is responsible for holding the master PTY and copying data back and forth from the container process. So it supports attached clients from Kubernetes. And then finally, it also supports detection of out-of-memory conditions and reporting them where kubectl described. And because conmon is the parent of your container, cryo can be restarted without actually affecting any end of the running containers. Because conmon is a parent. So this is what a Kubernetes pod looks like with RunC in cryo. So you have the pod, which is the holder of the IPC net and, optionally, the prid namespaces. And the pod also has a C group assigned to it. All the containers running inside the pod are under that C group and share these namespaces. And you can see that, traditionally, the Kubernetes had an infra container, which is the roll-off, which is expanded now to also being a reaper for prid namespaces. So you can use a default Kubernetes POS container, or you can plug in your own. You can choose in its system of choice. It's configurable in cryo. And then corresponding to each container, we have conmon. And you may think conmon might behave, but it's lightweight monitoring agent written in C. And it uses shared libraries, so the memory over it is pretty low. And this is what the architecture looks like with clear containers. I'll let Sam answer more questions about it later. So this is what the node looks like with cryo in the picture. So on the left, you have your kubelet talking GRPC to cryo. Cryo implements the image service and the runtime service. The image service is responsible for making sure that the image that is requested in a pod is available on the node. So kubelet basically queries cryo, uses the image service to see if an image is present. If it is not present, it issues a pull on the image. And ultimately, it ends up calling into the containers image library. And for the runtime service, we use the OCI Generate Library to create the config.json. So config.json is basically converted from to the kubelet sends a bunch of settings as part of the Create Container API. And those are translated into a config.json. And then Container Store Library is used to create a root file system. And finally, CNI plugins are used to set up the networking. So CNI is very orthogonal. So it just takes a network namespace. And before the container is started, it goes into the network namespace and does what it needs to and sets up an IP address that is returned to the kubelet. And then on top, you can see a couple of pods running similar to the diagram that we saw earlier. So what's the status of the project? So we pass all end-to-end tests, the CRI tools tests, and some 100-odd tests, integration tests that we have added. So we're passing more than 300 tests on each pull request. We are dedicated to Kubernetes. No pull requests are merged without passing all of these tests. We released 1.0 a few months back. And the supported version of kube was 1.7x. After that, we decided to make it easy. Since our only user is kube, we just match kube versions. So 1.8 of cryo supports kube 1.8. And right now, we just released 1.9 beta last week. And we are tracking kube 1.9 releases. So as soon as kube 1.9 is out, I think we should be able to release cryo 1.9 within a week. We are passing all the tests for 1.9. And we have maintainers and contributors from Red Hat, Intel, Suzy, and many others. So now let's jump into some demos to see this thing in action. So I've set up a local node. It's open running, and it's using cryo as a runtime. So you can see cryo is running as a system deservice. Now let's see if any pods are running. You can see nothing is running. We'll try to run a simple HTTPD pod. So I'm basically starting a deployment called myweb. And I'm saying the image I want is HTTPD version 2.4. And let's run it. Let's see the status of the pod. So OK. So since I pulled the image earlier, it didn't take any time at all to start. So you can see the status. It was pulled, created, and started. And then you can see it running here. Let's try to curl it and see the response. So let's see if our logging integration is working fine. So you're able to see the tree gets corresponding to the curls that I just did earlier. So this basically shows the create and the start API calls in CRI being exercised. And also the logs are in the right format because Cubelet is able to read them. Now let's try exec. We're able to exec into the container. Let's see what's running inside. So HTTPD as expected. Let's try running an interactive container. So we are able to run a busybox container. So this is basically demonstrating the attach API. We are able to launch a container, give it a TTY, and Cubelet is able to connect to conmon, which is proxying the TTY back and forth from the container process. And it all works. Resize and action, basically. Let's take a closer look at what's happening behind the scenes for the MyWeb pod. So this is cryo CTL itself. They could debug CLI for cryo. And the output is very robust. We are moving to another upstream CRI tools project, which has a better CLI. But right now, this can do for debugging. So I got the ID of the container for the MyWeb pod. And let's see what's actually happening. So this is just a RunC container underneath. So you can see that it's visible to RunC. In fact, you can even RunC exec into it. You can see the same output that you saw from the Cubectl exec. And then for C groups, we have two options. You can either use a C group FS or you can use system DC groups. So on this node, I have setup cryo to use system DC groups. So I can run some system decommands and actually see what's happening inside the container. So we can query system control status. And we give it the name cryo-id. So you can see HTTPD running inside the container. And we have another slice for conmon, which is conveniently named cryo-conmon-id. It has some debug information about bytes being written in and out, which can be disabled optionally. So conmon is the parent of the HTTPD container. And it's responsible for the terminal copying back and forth. As you can see, the total bytes written over there. And then let's try another example. Let's try running a job. This is a pi calculation job. It's tumbated. It uses Perl. The image is huge, so I'd pulled it ahead of time. It calculates 2,000 digits of pi after the decimal. This also shows the logging in action, the multiple chunks, and they are all being read correctly back. So basically, through this demo, we can see that the CRI is completely implemented and supported as a Kubernetes user. You don't even need to know the details of what is actually happening behind the scenes. You can use all the Kubernetes APIs and commands that you use today, and it all just works with cryo. So besides that, we have a couple of other demos that you're welcome to check out from the slides. So one is for QBADM support and mixed workloads, where we demonstrate running RunC and clear containers side by side on the same node. The second one is supporting different nodes and with different distros running in them, like one is in Ubuntu and one is a Fedora node, and everything just works together nicely. So what are the next steps for the project? So at this point, we're just tracking and supporting Kubernetes versions. We'll be releasing 1.9 soon, I think mid-December, when Q1.9 is released. And besides that, we keep an eye on issues and bugs that go into Qube. And we pick them into our stable branches. So we are supporting our 1.0x branch for Q1.7 and our 1.8 branch for Q1.8. We have our blog here on Medium and is a GitHub. And we are available on IRC hash cryo. Also one more thing, I think I forgot the slide. So we are working on another tool called Kpod. And we'll soon be renamed. So what is Kpod? So Kpod is basically a serverless debugging utility for cryo. So you can use cryo with Kubernetes. But what if you want to go and do things under the hood and see what's happening? So you can use Kpod to pull images, tag images, push images. You can do Kpod, PS to list all the running containers, and so on. The CLI is very similar to Docker. So it's easier to use it. The big difference being that Kpod shares storage with cryo. So whatever you pull or do with Kpod is available to cryo instantly. And also it doesn't have a daemon. So OK, any questions? Yeah, so we're planning for OpenShift online to use cryo. And that is in progress. We are targeting using Kubernetes 1.8 and OpenShift 3.8, which is based on Kube 1.8. Why does the logo look like a snowflake? So actually, it comes from the name. So cryogenics, and so we went with a snowflake and we thought, why not make it special, like a snowflake? Hi. Do you consider this production ready? There's a feature parity between, as a Kubernetes user, there's a full feature parity between Docker and... Yes, absolutely. I mean, if you want any confidence in that, you can see that each pull request and cryo master passes all the end-to-end tests and Kubernetes. So you can run all the tests and you can see our results in, yeah. Just addition to the previous question, and the log formats are also kind of the same, right? So we just switch from Docker to cryo and we do custom parsing of the logs. We will not have to rewrite anything there. Do you want to know what to do with the logs? No, no. You mentioned that there is a certain format for the logs, which is understood by Kubernetes. Is that mechanism the same like with Docker? No, the log format is different from the Docker JSON format. And so I know that Kubernetes folks have made changes to the FluentD plugin to understand the CRI log format. And it's a fairly simple format to understand, like it's probably 50 lines to write code to parse that. Okay, so anyone who wants to accomplish successful transition from Docker to cryo will have to escape the payload of the logs and wrap it into the new format, right? Right, I mean, anything that consumes logs from containers directly will have to understand the CRI format. And it's not just, I mean, any CRI compatible runtime has to output the logs in this new format. Yeah, so if you plan to switch from Docker to anything that's CRI based, you will have to transition to the CRI format. Hey, sorry if I mis-ed it. My question is about CRI. We're seeing CRIs everywhere. How is cryo different from container G, for example? Like, how do I pick cryo over the others, Docker or? Great question. So I think, like, so with CRI, right, Kubernetes got out of the business of picking the runtime for you. And everyone who has a use case for how they want their runtime to work now has the ability to either implement a runtime or evaluate all the runtimes that are available and pick and choose what they want. So cryo is very tight to Kubernetes. So if you want something that's closely tight to Kubernetes, you might choose it. Container D has an API, so it can be used with Kubernetes as well as for other workloads. So best would be for you to evaluate them and see what works. I mean, you can take a look at the overall architecture. So one thing we have is like one, we have one fewer daemon than container D. So, sorry, so does this simplify, let's say the upgrade capabilities of OpenShift toward the release of the different cryo's virgins rather than relying on Docker where every time Docker is updated, OpenShift breaks out, you know, all of those upgrade cycle challenges that we've seen with OpenShift. So I think that's one of the challenges that we've had with Kubernetes and OpenShift. So we wanted something that just supports whatever Kubernetes needs and we hope that the disruption is none too minimal. As long as your GRPC maintains backwards compatibility, the upgrade should be better. Yeah, because like in one line, there was barely anything new added. Only thing that changed was the couple of debug APIs and the CRI logging format had an issue which was fixed, but the rest of the runtime and the image API was unchanged. So nothing should have broken moving from, moving cube from one eight to one nine. You mentioned we should evaluate the differences between different runtimes. How trivial or not is it to plug cryo into your existing nodes or do you have to change your whole cluster over? How does that work? So can you just repeat it? I'm getting the entire question. If I wanna give cryo a test run, how would I do so? So yeah, so it's very easy to start up cryo. So all you need to do is when you start Kubernetes, you need to point it to the cryo endpoint after you started your cryo server. We have RPMs and Debian's available everywhere. So you can install those easily. You can point your Kubernetes to the cryo socket and it'll start talking to cryo. And besides that, we also have mini cube integration available so you should be able to download a new version of mini cube and play with cryo to see how it all works. And besides our performance team is doing work testing cryo to see what the overhead is and we'll soon be able to publish numbers comparing it with Docker. As of OpenShift 3.7, we're shipping cryo with OpenShift. So it's just setting an environmental variable if you wanna run it on top of a rail system. Yeah, yeah, for OpenShift, it's just an Ansible variable. You say OpenShift use cryo equal to true and Ansible will set up OpenShift with cryo. It's tech preview in 3.7, Docker's still at fault. And the goal is for 3.9 to make it fully supported underneath OpenShift. Yep, any other questions? All right, thanks everyone for joining.