 Hello everybody and welcome again to another OpenShip Commons briefing. This time we have N'Ral Patel and Dan Walsh from Red Hat and they're going to talk to us about and probably pronouncing the acronym. I say Creo, an OCI based Kubernetes runtime that has been really hot and very topical thing around containers and that world. And rather than blather on myself, I'm going to let N'Ral introduce himself and Dan and just kick this off. The way the session works is you can ask questions in the chat. We'll try and answer them. I will probably turn everybody's video off during the demos so that we don't have any faces in the middle of our terminal screens. But other than that, it's pretty simplistic and then we'll have live Q&A at the end. So without further ado, N'Ral, take it away. Sure. Hi, I'm N'Ral Patel. So I work on the containers team in Red Hat. So I'm a maintainer of the OCI runtime specification, the RunC and LIP container projects and lately I've been busy with the CRIO project. Yep. And my name is Dan Walsh. I run the container team at Red Hat. That's the base of the team that does all the containers underneath the Kubernetes level. So we're basically in the operating system level. So my team manages, you know, fixes things like Docker, underlying storage, pretty much everything that happens at the host that, you know, something like Kubernetes or regular container run times need to do. We do in this team. So starting out, so we pronounce a CRIO. So CRIO really stands for, if you go back in history, we wanted to build a runtime, a container runtime that followed the OCI specification. So and if you go back a couple, about a year ago, Kubernetes was based totally on top of Docker and the Docker daemon. And CoreOS had come to open up a bunch of pull requests against Kubernetes in order to support Rocket underneath the Kubernetes to basically to replace the Docker daemon with Rocket. And what Kubernetes didn't want to have to do was to have a whole bunch of, you know, sort of if and else statements that says if I'm using Rocket, then go down this tool chain. If I'm using Docker, go down this chain. So they decided to do is rather than support a whole bunch of different run times that they would specify a runtime interface. And that's what the CRI stands for is container runtime interface. So Kubernetes defined basically interfaces that it needs to, when it goes to run a container, it defines those interfaces and it will call into any container runtime that implements those interfaces. So since that time, so after that happened, we, Red Hat, we kicked off a little project, a little side project to see if we could implement a really simplified container runtime whose main goal was totally tied to Kubernetes. So we looked at what Docker is doing and what Rocket is doing. And they always had sort of conflicting goals. So, you know, there's this CLI support, there's other support, but they weren't really, you know, totally dedicated to Kubernetes. So what we really wanted to say is that we're going to build a runtime whose only job in the world is to satisfy Kubernetes request. So if Kubernetes changes the CRI, we implement it on top of our tool. If other orchestration tools come in and want to use the CRIO daemon, they have to talk to us at VSCRI and we won't add any interfaces to our daemon that aren't specifically specified by Kubernetes. Lastly, every pull request that comes in from Kubernetes, I mean, every pull request that comes into CRIO will not get merged unless we can fully pass the entire Kubernetes test suite. So really CRIO is totally dedicated and being optimized for the Kubernetes workflow. And next slide. So we're going to talk a little bit about the components that make up CRIO. So one of the things, I actually wrote a long blog that talks about the evolution of containers, opensource.com, you can go look it up if you Google it, you find it. We talked about breaking a patch. So if you look at the tradition of what Docker has done, they basically set up one big daemon that sort of built all the technology into that one daemon. So if you go out and you want to run a container, you talk to the daemon, ask the daemon to run the container far here, that daemon goes out, pulls the container from somewhere, stores it on disk, and then does some management of the storage and then launches a process to run your container underneath it. But everything has to go through that daemon and the daemon becomes the central point of all control. And what we wanted to do is break apart that daemon into sort of core components. And so we broke out a few different core components. The original one that most people have heard of is the OCI specification, which allowed us to basically get down to a core container runtime low level. And that, you know, a couple of examples that are run C and clear containers. And really what they are is tools for just running containers. So you give me a rudifest and a JSON and I'll run a container on top of it. And that's what OCI specification did. But we also needed two other things in order to run containers for Kubernetes. We needed storage. So we needed a way to take an image and actually put it onto disk. And so we really originally did is we started with sort of what we had done for Docker and building up the storage for Docker and pulled that out into a separate package. And that's stored on the kithub.com slash container slash storage. And we started developing technologies under there. And what we wanted to do is actually move all the locking and all the controls out of the centralized daemon and onto disk. And so that's what GitHub's container storage is. And then the secondary part of running containers is how do I pull an image? How do I pull it back and forth between container registries and my host? And so we broke out that into another separate go library called GitHub containers image. And so we've done there is we're basically, well, we have engineers and people contributing different ways of moving images around. So images can move back and forth between container registries and Docker daemons or other types of storage can move to our container storage, even to directories. So we basically built these two different libraries along with the OCI specification. And we have all the building blocks we needed to basically wrap them with the daemon. So cryo basically takes full advantage of using container storage for storing its data, contains image for pulling images from different registries in different locations. And then we use RunC and we support using RunC to run the containers underneath us, which is exactly the same thing that the Docker daemon does. So if you look to the Docker daemon now, you'd see that all containers are actually being launched under RunC. But we've also worked with in our OCI effort, Clear Containers, which is an Intel sponsored effort. Clear Containers is running containers inside of virtual machines. Instead, it's died of lightweight VMs. And so we've been working hand in hand with Intel to make sure that cryo works well with multiple different container runtimes. So right now we do RunC and collect containers. And others have looked into potentially using it. So next slide. Yep. So the remaining components are the OCI runtime tools. So Dan mentioned that we need a spec file, a spec config.json file for running a RunC container under root of for the generate library in the OCI runtime tool is used to generate these configurations. And the advantage is it's used by all other projects and it has all the latest work fixes. So whenever there are any changes to the spec, we get the fixes in the generate library. And then for networking, we went with CNI, which is kind of become the container networking standard. And we use CNI to hook up networking for the pods and cryo. And it tests with pretty much all the different plugins. We have tested with Flannel, we've OpenShift SDN, and a few others. And the final, but like one of the most important bits there is Konmon. So the way the OCI runtime is set up, there's a separate create and a start step. And you need some kind of monitoring process to keep a hold of the container, container TTYs, and also like get its exit status and so on. So Konmon is used for like monitoring, handling logs, handling TTY, serving attach. Attach is like when you can do a keep attach to the TTY of the container and also detecting and reporting out of memory conditions. So we'll see that in action shortly in the demo. So this is what a pod looks like when using cryo. So you have an infra container. The infra container is optional. And it is what holds the IPC net and prid namespaces that are shared by the other containers in the pod. And we have made the infra container pluggable. So you can select whatever you want. You can take the default Kubernetes pause container or potentially replace it with a system decontainer. So system D can take care of reaping zombies inside of the containers in the pod. And running on top of each container is Konmon. So Konmon, I mean, even though we launch many instances of Konmon, Konmon is written in C for like efficiency and it's very efficient in terms of CPU and memory usage. It has just the minimum bits required to satisfy requirements for logging and monitoring. So this is what it looks like with clear containers. They have an additional stream process and an agent. So clear containers are actually based on VMs. So instead of launching pure Linux containers, they are using VM as the pod container. And inside that, they then spawn other containers. And for the latest version, 3.0 that they are working on, they are also switching to using lip container, which is the same library as RunC for starting the containers inside the VM. So this is what the overall architecture looks like. On the left, you have the Kubelet. And Kubelet is talking to the cryo daemon using GRPC. Now, the Kubelet CRI defines two interfaces. One is the image service and the runtime service. So anyone that wants to implement the CRI needs to implement both of these services. And cryo implements them both. And for the image service, we use our containers image library that Dan mentioned earlier. And for the runtime service, we use the other components like the OCI Generate Library, TNI for networking, and storage for setting up the root of S. So what storage really is doing is when you say, hey, I want a Redis-based container, it takes the container from the image library and explodes it onto disk, setting up copy and write, and gives you the part to the root of S in your config.json, which is then passed on to the OCI runtime RunC in our case, which is a default. And on top, you see two pods running with containers, AB, the infracontainer, and conmon running on top of each of the containers. So what is the status of the project? So all the Kubernetes node conformance tests are passing, and they are run on each pull request to cryo. We merge the PR only if the tests pass, so there are no regressions. Additionally, we also pass all the end-to-end tests in Kubernetes. All the CRI APIs have been implemented. If you want to try out cryo, like I would encourage you to go to kubernetesbyexample.com website and try out all the examples, and they should work. We released the beta version a couple of weeks back, the 1.0 beta, and we are working on some bug fixes, and after that, we'll be ready to release the final 1.0 version. We have maintainers and contributors from Red Hat, Intel, Suzy, and many other companies. I think at this point, we are at over 50 contributors on GitHub. Then for easy setup of cryo, we have integration with QVADM, and so we have a few repos under GitHub cryo org, where you can check out one of the repos, and I'll help you set up QVADM and cryo. Minicube integration is in progress, and we also support mixed workloads. Mixed workloads means that under the same Kubernetes deployment, you can have some pods that are running under Run-C and some others that are running under clear containers, and we do that using annotations today. You have something called as a trusted workload, and if you have an untrusted workload, then you would want to run it under a VM-based container and the clear containers implement for additional security. Now let's move on to some demos. Can you see my screen? Yes, it looks great, and the font is wonderful. Can I ask a quick question while you're in here just around OpenShift? This mixed workload thing, does this mean that eventually OpenShift will be able to do mixed workloads as well? Yeah, I would say this is, so from an OpenShift origin point of view, one of the reasons, I mean Intel is working very hard because they really want clear containers to run underneath OpenShift. So from an OpenShift origin point of view, I see this definitely happening when we move to cryo as a backend. So whether or not Red Hat will end up supporting it on top of REL is always a question. So that's something we're talking about a lot internally, but right now we can't make any statement about support for clear containers underneath REL, but I definitely see this as being something that will work very well in an OpenShift origin type environment. Awesome, good news. Yeah, and there are some additional challenges in the cube world on how to manage two different kinds of runtimes. So those need to be worked out whether they should be allowed on the same node or not, how should they be scheduled, how are the resources calculated. So this is like a proving ground. So if people want to try out those ideas, they can use cryo and hopefully we'll make it to a cube when it is at some point. One thing to think about when you're looking at clear containers also is you really have to control, you have to run them on physical hardware because they are usual virtualization and most of the cloud vendors block virtualization inside of virtualization. So clear containers is a great, I think it's the best solution for running containers inside of VMs from a security point of view is awesome for giving you isolation. But the big hindrance again is that you have, they have to run on physical machines, they can't run on top of virtual machines at this point. Yeah, but we have a lot of people running OpenShift on bare metal too, so. That case it works real well. All right, demo time. All right, so I have the cryo demon running, so I'll just show it here. And then I started Kubernetes local cluster, pointing it to the cryo demon. And now let's get playing with it. I'm going to start busybox container and you can see that this works. So what this proves is that the attach API in the CRI is implemented correctly. So we spawn a busybox container and then we are able to attach to the TTY of the container. So your shell is working just fine. Now let's try an HTTP container. So it is still pulling the container and it started. So you can see that this is all running under RunC. You see that the first two are stopped, which is a previous container that we had started. And now we see two running containers. One is for the pause and other is for HTTPD. Let's do a get pods again and let's see if we can curl it. So this shows that CNI integration is working because we can reach the pod. Let's try to get the logs of this container. So you can see the three entries for the three times that we called the pod. And now let's actually take a look at what's happening underneath the covers. We got two RunC containers running, one for the pause container and the other for the HTTPD container. And we also have Conmon running. And you can take a look at this by taking, we use the system DC groups here. So let's see what's happening in those files. So here you see the C group that is set up for the container. And we also have a C group that is set up for the Conmon that is running on top of the container. And you can notice that both of these are running under the same slice. So they are getting charged to the pod slice here, which is Q pods best effort, pod something something dot slice. And we can also take a look at the ps3 of Conmon to see how the processes are set up. Conmon is a parent of the HTTPD inside the container. And it's monitoring and supporting logging and attach and all those features. Let's try to exec into this container. So you've got your exact working. We can exec. That's one more CRI API. So you can go in there and debug your container. So that covers like all the features in CRI. Now let's take a look at how whether this works with OpenShift as well. Now OpenShift shifted to the CRI in version 3.6. So we can actually replace the runtime under OpenShift to make it talk to CRIO. So what I set up here is an OpenShift local cluster with CRIO as the runtime. And let's try some OpenShift features. So let's do OC get pods. Only the router is running right now. And we can try and create a pod. We just started the Hello OpenShift pod. Let's create a service for that pod now. Hello service. Let's try to reach it. So the router is working fine. We can reach this pod using the router and the host header. Also, this one more example to show the integration working. Yeah, the last minute restart been told. All right. Yeah, the demo goes on into this one. It wouldn't be a live demo if we didn't have something. Yeah. Sorry about that. If you want to take a look at more demos, we have links to a couple of more ASCII gamers where we have a demo of showing the CUBE ADM and mixed workloads and also a multi-node multi-OS cluster. So we show it running on Debian's and Ubuntu's and on different OS's. And now I'll hand it over to Dan. Okay. So one of the problems with us, you know, everybody that's using Kubernetes right now on top of DarkerDemon, a lot of people like to go in and sort of debug around or figure out what's going on. So unless you have the knowledge that RUNL has right now going in and executing those runC commands and trying to match what Kubernetes is doing to what actually is going on in the system, or if the Demon for some or the container runtime for some reason gets hung up, how do you debug situations? So most people right now are actually using the Docker CLI to do that type of activity. So show me what images are running, allow me to kill containers, different types of activity. So we decided that we wanted to build sort of a simple interface on the back end of basically behind cryo. And now since cryo is using container storage and containers image and actually storing all its data on disk and putting its locking and stuff on disk, we could actually build up a command line interface to actually interact with basically the storage and the images and be able to look at everything that's going on. So we started building a tool called Kpod. So it's a management tool for containers and images and it's demonless. So you don't even have to run the cryo Demon to be able to use this. So if the cryo Demon is hung or shut down or whatever, you can still go in and look at storage and look at what the containers and pods are running in the environment. So the other thing we wanted to do is we wanted to make sure that it was easy for users to transfer the knowledge. So if you're going, if you go from a Docker backed Kubernetes and you go to cryo based one, we wanted to make it simple for you to transfer the knowledge. So we're really basing the K pods, at least the initial version of K pods is matching the Docker CLI. So if we go to the next slide, we'll show you where we are instead in development of the K pod CLI. So these are some of the commands that we've implemented so far. This was actually up on if you go to the cryo Kubernetes incubator cryo project, you'll find on the read me that we list out all of the K pod commands that are currently implemented. And this picture was taken as of yesterday. So you see that we have, you know, sort of a lot of the image stuff based stuff that's available in Docker. So if you know how to do it, Docker images, you'll be able to just, you know, switch out the Docker command for K pod images. And pretty much all of the options and things like that remain the same. So we've implemented probably about half to three quarters of all the interfaces and we're for K pod at this point to match the entire Docker CLI suite. Obviously, we're not implementing things like swarm and other other parts of it, but pretty much everything everybody's familiar with, we're implementing in K pod. And again, it's not demon based. So when you do a K pod run, the container will be a child of you, you're not going out and do an client server operation to talk to a centralized demon to do it. So you'll be able to do K pod exact to get into a container and you'll be able to do K pod PS to show you all the containers that are running. So the first phase is to implement most of the Docker CLI. But we're also implementing some other interesting tools in here. So if you look down about halfway, you'll see a K pod mount. And what K pod mount can actually do is mount up the container. So you can actually get the mount point of where a container is. And so you can actually start to fool around with the container, you can actually just go to the directory and look at what's inside of the container. You can actually copy stuff into the container, you can copy stuff out of the container. And you can use any tool you want to interact with the container. So you can use external tools on the machine to manipulate data. So we're looking to enhance sort of the sort of the Docker CLI experience by taking advantage of some of the other tools. Eventually, right now, K pod is really about managing containers. But the next phase after we've completed the Docker CLI, we want to actually get into management of pods. So how do I launch a pod? How do I add a container to a pod? How do I remove a pod? I'm also looking at things like sort of the management. We want to experiment on how management of pods happens. So can I set up pods in such a way that if I have two containers in a pod, what happens when one container dies? Does the entire pod shut down? Or does the container management tool restart? So look, we're investigating and thinking about things more at the pod level than the containers. But at least in the beginning, we want to make it easy to transfer knowledge across from the darker CLI to the K pod CLI. Next slide. So at this point, we're going to just conclude and why don't you take over and Ronald? Yeah, so the next step is releasing one.to just making sure that everything works fine. There are no bugs. We are doing some testing. So we will have the one dot go out in two to three weeks. And then we want to graduate out of the Kubernetes incubator, meet whatever the requirements are for that for graduation. And post one.to will be will be changing a technique, a technique around how to be versioned cryo. So we'll be versioning it to match the Kubernetes version. So it's very simple for someone to understand what version of cryo works with what version of Kubernetes to allow cryo one seven, working with cube one seven cryo one eight, working with cube one eight and so on. And also we are working on integrating cryo and targeting it for open shift three dot seven and getting it on to open shift online. So we have a blog on medium.com slash cryo. We have been blogging quite frequently about the new features that we add to cryo. And you can follow us on get a talk to us on IRC and take a look at the website. The contributions are welcome. Feedback is welcome. Any help with testing is super, super welcome. Awesome. Well, there's a couple of questions in the chat. So let's get those answered if we can. The one regarding K pods, how does something like K pods PS know what containers are running if there are no central, if there is no central demon? Basically, what's happening is at any time cryo stats a container that it basically puts down the information on to disk. So we're using, so the central store for information about what containers are running or being stored on disk, as well as you can also query front C. So we can figure out run C also can keep track of which containers are running on the system. So if you look at one of the issues I always had with the Docker demon is that all this information is being stored and hidden away inside of the Docker demon. And therefore, if the Docker demon crashes or whatever, there's a chance that you could lose information. But we've moved all the locking and the status information is actually being stored on disk. So we can interact with the with the data the same way that cryo did. We're actually making a new library call. We're calling it live pod. We're working on that now. By the way, K pod and live pod are not blocking cryo. So cryo is probably going to get released to 1.0 before we've completed all of K pod. So but basically, we're looking at building out your pure library so that others can interact with the data stores and stuff that cryo is using. So someone earlier, Seraj asked earlier, and it might be a topic for another OpenShift Commons briefing, but how do you set up an OpenShift cluster with cryo as the back end with K, every time I see K, with Kubernetes, are there flags you can change when you do that? Yeah. So right now, like just a couple of weeks back, a pull request was merged into OpenShift Ansible. And basically, you just pass set an environment variable to tell it to use cryo, and it'll do the right thing. I can quickly show you what we do. And the other thing, Munral, is there's a typo in your URL for the word incubator. Someone has just pointed that out. We'll fix that. So if you want to answer the question about setting up cryo, you just go into your mode config and pass these arguments to the Kubelet, the container runtime, the runtime endpoint socket. CRI, there's like four or five things you need to pass. So this is the stanza that you need to pass, and then OpenShift starts talking to cryo. It's really that simple. Wow. So we don't need a whole OpenShift Commons briefing on it. That's great. Is there documentation for that yet? So there isn't. So you can use Ansible, OpenShift Ansible to start an OpenShift cluster with cryo. And I have another Ansible script that builds everything from source and does this. But it'll be good to get a blog out. So maybe one of us can write out a quick blog. Well, I'm also teasing out the question around documentation for all of this stuff, too. So because the documentation by blogging is wonderful to get things started, but real documentation on stuff like this. And so I've been pushing people to write stuff for the OpenShift Dock set. So we'll see if we can get that in there somewhere. So it's searchable. Brad is asking one more question. What are some of the types of workloads when you say mixed workloads? Okay. So basically, if you think about what is the riskiest thing to run in your environment, you potentially, if you've seen any of my talks in the past, I've always talked about your virtual machine separation is better than container separation. And there's several reasons for that. But the main one is that the kernel is a single point of failure between that there's an issue in the kernel, there's potential breakout. So if you're going to run sort of really dangerous workloads, workloads that really require heavily privileged operations to happen inside your containers, then those workloads, you're probably better off putting them inside of a virtual machine. So using some likely containers for that would be helpful. One thing we've thought about is sort of the, would we be able to do something like OpenShift builds? So if you think about people pulling down random internet packages and building and running random code, that might be something that we want to put inside of a virtual, add the security of a virtualization wrapper around it. But I mean, it goes back and forth. One of the problems with some likely containers is that you start to lose, you start to get better separation, but you lose access to certain things on the host. So there's pluses and minus to using VM isolation as well as using standard containers. So I think it really, it's up to you, but I'd like to have the ability to have choice. Let's see. I think that was the last question. If anyone else was going to, yes, Michael Hasenblas, that was directed at you around getting the docs. If you're a pet peeve of mine, we tend to blog, but a document by blogging and trying to get it into the doc set is one thing that I'm trying to move us forward on. And Mike says, yes, they'll do both. Awesome. Seeing is the other thing this, you said this was targeted, but for 3.7 of OpenShift and so that by the time we get to the OpenShift commons gathering in December, hopefully we'll be able to show it off a little bit more and we'll have Dan and Munral there at the gathering, which is the day before OpenShift commons. And we may have, if there's new things that come up in between now and then, we may have you guys back. Here's one more question that's just come in from Ivan. What plans are there to have Pryo adopted in public cloud providers container services such as GKE? Yeah. Right now with Pryo, we tend to have a lot of people sniffing around the edges, so there's all the big players are taking a look at Pryo. They're investigating it, but we have no, nothing we could talk about or nothing is actually solid as far as potentially adoption by large cloud providers or anything like that. But again, our main goal is to satisfy Kubernetes. We're basically saying Pryo is a slimmed down, purpose built demon for running Kubernetes workloads. And so in a perfect world, I'd love to see it get adopted in GKE, but right now, everybody's just looking at it. But it's one of those things where we can perhaps have the t-shirt that says Kubernetes, not just for containers anymore. We all thought VMs would go away and they're back. So this is wonderful news for lots of us, people with security minded. So I really appreciate all the work you guys are doing and all the collaboration that's going on across all of the different communities to make this happen. So one of the things I'd like to do on Twitter is add a trending of no big fat demons. All right, well that's the t-shirt that we should help you set. Another place that we're actually looking to integrate with, we actually have another part of this project, which is called Builda, which is a replacement for, so obviously, Pryo doesn't do anything about building containers. It's just for running Kubernetes and Kubernetes doesn't build container images. So we've actually broken apart new similar tools to build a thing called Builda. So that's Builda, like I would say it, B-U-I-L-D-A-H. And maybe we'll talk at the OpenShift Commons about Builda and potential integration between OpenShift and Builda as a back end for building container images and potential integration with source damage. I'm going to give you guys a half an hour slot there and make a talk. So I'm looking forward to that because there'll be lots of new stuff and interesting use cases as well, I'm sure. And if you're on this session and you are running on bare metal or you're thinking about using Pryo, let me know and maybe we can get you there too and to share your use cases. So that would be great. So that's going to be December 5th in Austin, Texas, along with KubeCon, which will be the following day. So it's going to be lots of fun and we can listen to Dan's wonderful Boston accent talking to more about Builda. So thanks everybody for joining us today. This should be up with the slides if Mineral sends them to me and the links by tomorrow and we will be distributing that on blog.OpenShift.com and you'll see that go out on the Twitter. So thanks again.