 Yeah, so in this talk, I'm gonna talk about rocket. What's the state of rocket? Where are we? What's being new in rocket and what's the future of rocket? And we'll also talk about rocket lit, which is the part that integrates rocket with communities So first something's about me. So hi, I'm Yago. I live in Berlin and I'm one of the Kimphal co-founders You probably know Kimphal by now because we are sort of organizing this conference So but for those of you who don't know us We are basically a for hire Linux engineering team that focus on core cloud infrastructure technologies So yeah, one of those cloud infrastructure technologies is rocket and we've been working a lot on that in the past and Yeah, we've also retaken this work recently And yeah, I'm currently a maintainer of rocket and the rocket lit and yeah There's my github and my Twitter account if you want to find out more about me So the plan for today is I'm gonna give a brief explanation of rocket So what is rocket and then I will explain how it works. So some of the rocket internals I'll do the same for the rocket lit Yeah, what it is, how does it work? I'll explain what's new and yeah What are the new features we've been working on recently and then I will explain what's missing. So yeah We have a vision for the future So rocket very briefly is a modern secure composable run container runtime Modern as in we try to take advantage in the latest technologies and Linux specifically like user namespaces or overlay effects or whatnot And come secure as is as in we try to be secure by default. So all the security options are enabled by default if you want to Yeah, disable it. You have to do it explicitly So you're yeah conscious of what you're doing and composable because we try to play nice with the rest of the system We have a good integration with system D It's not required. But yeah, we have a very good integration with system D and yeah Composable And ultimately it's just an implementation of the absi spec Specifically the parts of image format and execution environment Yeah, so absi is What we started when we started rocket to specify the image format and and the security environment And now it's kind of not maintained but rocket still uses it heavily. So I will give a brief Yeah, look at what absi is. So, yeah, it's a standard application container specification It's an open specification and it Defines some associated tooling. Yeah, there's some repos under the absi organization Like a C server or Docker to ACI C&I was part of the absi spec But now it's moved to container networking since it's being used used by a broader audience But for our purposes, we're gonna talk a bit about the spec So it defines four things image format, which is what the what an application consists of So it's basically a tarble with some manifests associated that says the results limits you want the executable You want to execute? Then there's the image discovery part, which is how we translate An image name to an actual URL where the artifact of this image is and for that We take advantage of the DNS system. So we have a translation from the name to a DNS namespace Then it defines spots, which is how applications are grouped and run and Then it defines the executor, which is what the application is gonna see when it's executed So Yeah, rocket is a simple CLI tool. So we don't have any demons that are mandatory It's written in Golang and it's yeah, it's basically Linux only and It is self-contained as in We don't require any weird dependencies and just your typical Dynamic libraries that you have installing all the systems and it's in its system and district agnostic Although as I said, we optimize heavily for a system D So yeah being a simple CLI tool it means that the applications run directly under the spawning process So there's some diagram there You can see that there's a yeah the spawning process which can be bash or system D or our unit you name it Then under that there's rocket and then under that does the pod and the two apps in this example So you can see these some stage zero stage one stage two. So the execution of rocket is divided in three stages So basically stage zero is the the CLI tool. So it takes our discovering fetching images and rendering them on the on the disk Stage one is basically the actual container So that sets up the exact environment for pods and it manages the processes life cycle and applies resource constraints to And the stage two is the app itself. So whatever you're running engine X or yeah any app So one cool thing about stage one is that's swappable So that means there's different implementations of stage one, which is pretty cool So to the full one it uses Linux namespaces and secrets for isolation. So what do you call a Linux container? And it's based on system D and spawn and system D So we basically run a system D and spawn container and inside we run a system D instance that manages all the processes in the pod There is also a KVM implementation, which uses hardware virtualization for isolation instead of containers So that's useful when you want a bit more security And it's based on QMU nowadays and also system D inside the pod So the this the part that manages processes is shared between those stages Then there's also fly which is basically no isolation stage one It's just a CH route and its purpose is to take advantage of Rockets image handling and and and signing and all that stuff There's some more but these are the main ones So I'm gonna stop talking about rocket and I'm gonna introduce Kubernetes for those who don't know it Yeah, so basically Kubernetes is a Container orchestration system, so it gives a developer an API endpoint where it can Define which containers the developer wants to run and it doesn't have the developer doesn't have to care about where they run or How this is done? It just says I want to run five instances of this container And then Kubernetes takes care of putting those containers in actual hosts so you can see there that there's a this funny icon on the top that's at CD and That's basically a distributed key value system that offers the cluster like as Like a you know a full view of a cluster that's consistent between the among different hosts so that's Yeah distributed primitive that's needed to implement Kubernetes and those worker cubelet Boxes are basically hosts So on each host a cubelet is running and the cubelet is the one that is responsible for creating creating the actual containers inside the host so yeah then the cubelet basically Nowadays implements an interface called the CRI well not implements, but defines an interface called CRI and CRI defines an interface Where there are different methods Where you can that express the things that you want to do with containers So for example the cubelet can say rampart sandbox So a new sandbox will be started in a host and the cubelet can say okay Now create this container in that sandbox and then start it or stop it or get the status or yeah all those stuff So this is implemented via gRPC, which is a remote procedure called by Google So cubelet will call one of those methods and there will be something on the other side listening and actually doing the actual operation so these things are called shims and Yeah, there are several of them. There's CRI continuity, which is recently Being developed a lot by Google. There's CRIO, which I think it reached version 1.0 in the last week or two weeks And then there's rocketlit, which is the rocket implementation of the CRI and yeah, it's bold because that's what I'm gonna talk about So how does it work? Yeah, so you have the cubelet that connects through gRPC as I said before to the rocketlit and Basically what the rocketlit does when the cubelet requests a new sandbox is called system Iran And it just starts a system the unit That runs rocket the rocket sandbox and then the rocketlet communicates with rocket by Yeah, just running rocket commands and it can ask rocket to add containers or remove containers or whatnot So in order to implement this we had to change the design of rocket a bit because it was basically immutable Design so if you start a pod it will be this way and that's it until you stop it and you start a new one So we had to do some changes These changes are what we call the app experiment So yeah, as I said before the CRI operates in the sandbox and container terms not just in pods So rocket had to be redesigned and we had a new sub command for that Which is the sub command app So you will have the rocket sub command that matches each one of the methods of this CRI interface But I was talking about before So with this system It was pretty helpful for us because this maps pretty neat neatly with what system D does So for example rocket app ad we'll just add a new unit file to the pod And load it and rocket app start. We'll just call system CTL start basically And yeah This is the experimental because it's a pretty big change to rocket So for now it's it's under an environment variable called rocket experiment up. So you have to specify that before calling rocket to Use these features because if if not will they will be hidden Another thing that we had to deal with is the logs So being that we run system D inside the pod The natural way to handle logs is to just use the journal and that's what we were doing Rocket was logging through to the turn to journal D. The problem is that the CRI interface Wants a locks in a custom text format. It doesn't want the journal format So the initial solution kind of hackish way to solve this was to have a Basically another container on each pot that Translates the journal logs to the CRI format So that had some problems because it was yeah, first of all very resource consuming So you don't want to have a sidecord container. You just want to have the containers you want to run And it was kind of unreliable. So sometimes the logs were not there and you will get weird error messages, which yeah, nobody likes Moreover we wanted to implement things like attach to to a to the standard input a standard output of a container and Journal D doesn't have anything to so we can do that so The proper solution is what we call the attach experiment. So it's another experiment And that uses a component called IOI OTT Y max and that allows us to write directly to the to the journal to the CRI file format So this is Simply a simplified version of how it works So this is the rocket system at the rocket process tree So you can see system the unspawned and then the actual container which runs system the inside So basically every application will connect its standard input standard output and error to this I OTT Y max demon and then that will pass through The the standard input output. Yeah to system knee to an external process or Into a log file. So in this case only standard output and error So this is the the simplified version of the CRI format. It's just very simple just a date What a stream it is and then the message? so yet this allows us to attach and to write logs to the in the format directly Without having a sidecar like we had before So basically those were the two things that were the most important things that we had to change for Rocket to work well with Kubernetes Yeah, I'm gonna talk about about the recent developments like in the last two three months in rocket so first we added a feature that a Client requested to share the host IPC namespace because they wanted to do some Yeah operations on the IPC namespace and and we you need to Yeah, share it with the host so other apps can actually do that So Michael Lee Alban implemented that and then we need to do a lot of fixes to make rocket possible Yeah, a lot of buck fixes, especially with the experiments. Yeah experimental. So they didn't weren't really working well and We need to do some integration work between the attach Experiment and Kubernetes because yeah, it needs some to put the locks in a specific path And we need to add some configuration options to do that And another thing that we did is we switched the default KVM flavor to from LKVM Which was not very reliable for us to QM. So yeah, now it works much better So, yeah, what comes next in rocket one thing that I think Casey started but didn't finished is Uploading CNI to version 06 Because we're using o3 and there's a lot of new features in in newer CNI versions that we need to update And We need to stabilize the rocket experiments and remove all these environment variables So we need to do a lot more tests and remove those so they can be used out of a box Yeah, we Experimented a bit with using run C to set up the stage to runtime So what we were doing until now was using system the unit files and those provide a lot of primitives to do a lot of Yeah resource limiting or Yeah, mainly resource limiting so Run C gives a bit more flexibility though and we started looking at using run C instead of Yeah, I think Casey over there was starting to work Yeah, so that we that would be cool and We did some experiments with cacing which for those of you who don't know that it's a new project by Leonard portraying and Yeah, I'll talk a bit more about that in the next slide and also general bug fixing because there's always bugs, right? So, yeah, the cacing tests are explained here in this graph, so basically cacing is a mixture between the rsync algorithm and a content addressable store and that allows Allows you to distribute images for example in a much more efficient way because it will only distribute the the parts that are missing And so we wanted to test that with container images and see what the improvements were So you can see that the blue the blue line is basically just downloading everything and The red line is what cacing transfers on the network So you can see that there are some spikes and that's when a lot of things change in the image So in that case you have to Resend everything basically, but Overall you can see an improvement and so that's very cool There's this the repo there where we did the experiments where you can find scripts to reproduce them So, yeah, we'll probably look a bit more into that in the future Yeah, so now about rocket lit itself what comes next so we need to implement missing CRI Features for example the attached thing that iotdy mox makes possible is not really working as of now We think is it was not that important because it's usually used as a way to debug your container But yeah, that needs to be implemented We need to also implement this CRI container stats API So Kubernetes uses a thing called see advisor to get stats from the containers That's not a really nice code and it doesn't work as well as we hoped So there's a new definition in the CRI for container stats. So individual runtimes can Can yeah figure out a way to get the stats and then send it back to communities Performance is something we haven't really been looking at But yeah, we we think it's not such great performance. We see some a lot of CPU usage So we need to improve that and as of now rocket lit only supports Kubernetes version one seven and We want to support one eight because that's the latest table release We need to improve error propagation. So when a command a rocket command fails We don't give very nice error messages. Sometimes it's just an exit status so that's something that yeah, it's very high priority in our list and We've been starting to run the Conformance test for Kubernetes and CRI and there's some tests failing. So we need to investigate those Okay, so Yeah, I want to do a demo. So at Kimfield, we developed this tool called cube spawn and What that does is it lets you spawn multi node clusters in your machine by using containers So each container would be a node basically so instead of virtual machines Like it's usually done like with the mini cube We use containers and yeah, that allows us to do tests More quickly and with less resource consumption So I have a rocket lit enable Kubernetes cluster running in my machine has some troubles Yeah, some minutes ago. I was trying to make it run and it didn't work, but I think now it's working So yeah, I'll share my screen You go Yeah So is it is this size? Okay. Yeah, okay So yeah, let's see so I have an application called Microservices demo, which is something that we've worse developed. It's just a very Nice micro service example application. So there's a lot of containers that work together to to bring you a Suck shop. So it's yeah, that's actually So we can see That there's a lot of pots running for carts the database for the carts a catalog Yeah, payment a lot of stuff And yeah, I can show cluster info dump Grab container runtime to to yeah convince you that this is not running docker I Can go to the machines because they're just containers I can use machines in TL shell and For example this one and yeah, there's no docker whatsoever There is the rocket Yeah, this oops, sorry List these are the images running And yeah, so this is basically working and I can get a list of the services of the namespace of the sock shop and Yeah, I can see that the front end has No, it's an output service. So that means that if I access any IP of the cluster on this port, I will get to the front-end service. So let's just try that So, yeah, I can use directly the name of the container, which is pretty cool and Yeah, this is the this the shop doesn't look so nice here in this small screen, but Yeah, so you can see a catalog and yeah, so this is fully running in in rocket with rocket lit It's a pretty complicated application. So I think it's pretty cool Yep That is basically it So yeah, thanks, and if you have any questions, please ask I'll try to reply Are there any plans to directly support the OCI? Format the open container image format in rocket, right? So right now with we support the OCI format by being able to run docker images And we do a conversion there to to the ACI format that's rocket internal and We would like to have OCI support but I think nobody has had time yet to look at it and People that are using rocket lit at the moment. They use ACI images So it's not really a priority for us now, but yeah, that will be definitely very nice to have In the west of the question over there, but I don't know Same good So Without the attach are there also so about the things that are not implemented that integrate with cube Yeah, will there is there a currently support for HPAs and things like that and when the custom metrics come out in 1.8 for HPAs will it support that as well? Yep, I don't know HPAs. Can you elaborate a bit? It's the auto scaling so like the the auto scaling for Memory and number not just replicas, but like the memory limits and all of that stuff native to kube in 1.8 There's going to be custom metrics involved So, you know, we see request per minute increase and we can increase the number of processors or the amount of The upper limits and the initial start containers right is rocket going to support those as well Yeah, I guess so I mean when we try to implement 1.8 We will look at those issues and we'll try to support them It should we should be technically possible. So Okay, thanks very much