 We've, I hope you're here because you're interested in Container D, but we've actually called it the Container D and Friends update because we have quite a few sub-projects now that I know many are interested in. Who was able to be in the keynotes and see Derek McGowan give our video update? That was pretty cool to have Derek involved. He's got a new baby at home, so he couldn't be here this year. A lot of times he and I have been tag teaming this talk, but I'm always amazed that people still come to hear about Container D. It's like inviting a bunch of friends over and having a plumber come talk about the plumbing in your house and how interesting the pipes are. But you know, we're a pretty core foundational runtime, but maybe not super exciting in the sense of like tons of new features, but we're going to try and cover a little bit about where the project is going. You heard from Derek if he saw that video, kind of our release cycle updates. So there's four of us here. It's going to move a little bit quick. I'm just going to give the intro, some of the community and release updates. And then you can see we've got Maxim, Michael Zappa, and Mike Brown. We're all maintainers of the project as well as many others who are even some here in the room. And so hopefully if we don't cover the thing you're interested in, you can find us after the talk. You can find us around this week. Grab us and take us to the sponsor showcase and find a seat. And we'd love to chat about anything you're interested in. So as I said, I'm going to kind of rapid fire go through a few hopefully interesting updates. That is that we currently have, we're about to have three release lines under development. One five would be end of life, but we've just extended that into January. So a few more months and we just made a release to catch a few back ports of bug fixes. So folks that are still using one five, there's still a few months of support for that. One six is kind of the biggest news. We just made a read me update and I have a slide about that in a minute. We just made our latest release just a few days ago of one dot six. That's probably kind of the most stable and used version of container D at the moment. That's what a lot of projects are importing. And then of course there's our main branch where we've been working towards our one seven release. And we just released the beta zero for that yesterday. Derek mentioned that in his video. And so that's where a lot of the new functionality and kind of our stepping stone towards 2.0 that Derek mentioned. So a bit more about the LTS release, our long term support. So we know a lot of folks again like container D because of stable and kind of slow moving in a sense. And so we wanted to take the release one six branch again because one seven is kind of a stepping stone towards 2.0 and some deprecations. And we want to make sure people have a well supported container D branch they can use for many years. And so we've extended that out through 2025. And I will read this paragraph. You can go to the releases.md in our repo and read what we're trying to define as what does it mean that we're going to have long term support? What does it mean for back porting bug fixes? What does it mean for CRI changes and Kubernetes releases? And if you see something there that doesn't sound right or sounds like we haven't really defined it well enough, we're happy to take input, open an issue and let's talk about how we can properly define what it means that we're going to have this LTS release. And again, one six is going to be that for now. A couple of community updates. One, obviously Docker is still a huge consumer of container D. And they now have been working towards for quite a while, but have announced this integration, this deeper integration with container D using snapshotters and image store. And so if you're a Docker desktop user, this is a feature you can access today in preview. But I think one of the main benefits is that a lot of the innovations in container D around snapshotters will now be able to be exposed up through Docker as well instead of having their own push and pull and image management and graph drivers. So that's something we've been looking forward to for a long time. Also, many of you know that the Docker shim deprecation has finally come to pass. And so there's been a lot of talk about migration and using container D. And we've definitely seen an uptick in contributions and interest in users, which has, again, been great for our community. We've got a lot more contributions. I put some stats there on growth. It's always a little bit hard to find out very specific data on user growth. We don't have telemetry. We don't know about downloads. But definitely there's some different reports that come out every year that have shown pretty significant growth for container D as a Kubernetes runtime as well. So I'm going to touch on a few more sub-projects before I turn it over to Maxim. Nerd CTL, a very popular client. Akihiro is here. The creator of that has grown into an amazing community. They just released their V1.0, released just a few days ago. This gives you a Docker-compatible UX for essentially having a Docker client-like capability built on container D. And Lima is kind of a co-project that allows you to then expose that on your Mac by having a lightweight VM where Nerd CTL and container D run. And so that project has, again, grown by leaps and bounds. And there's, again, a lot of amazing updates that Akihiro provided about support for lazy loading Snapshotters, new logging drivers, network drivers. And just also big news that Lima joined the CNCF Sandbox just a few weeks ago, which was a huge milestone for that project. I just mentioned lazy loading Snapshotters. This has been a really cool area of innovation around being able to pull just the pieces of images you need just in time to allow for faster start times. StarGZ is probably the most well-known that was created initially as an external project and then in 2020 came in as a non-core sub-project of container D. Based on that, a lot of others have been coming up with new innovations and ideas around lazy loading. Our team at AWS came up with Sochi Seekable OCI format. And so we've open sourced that Snapshotter this summer and looking for interested parties who would love to collaborate around Sochi. It uses some of the new features around OCI and the references working group to allow you to not have to modify the core image that you built, but to have this index that's referenced and allows you to know how to seek into the GZIP tar files of your layers. And then NIDA Snapshotter also has been around a little while and just this year they contributed as a non-core sub-project that's a block based lazy loading Snapshotter that relies on a few other projects like the overlay block device driver that they've written. So again, a lot of innovation there, a lot of interest around that, and it's cool to see these innovations basically coming that aren't tied to container D's life cycle. These can be developed separately, have their own communities, and integrate with container D. So at this point I'll turn it over to Maxim who will share more about Rust and some other projects. Thank you. So a couple years ago we got Rust version of TTRPC, our lightweight communication protocol that container D DEMON uses to talk to its streams. And this year we have added a few more additions to our Rust family. We have open source Rust crates. These are crates that let you write plugins and extension for container D DEMON, obviously in Rust language. And currently we've got seven different crates. We have JRPC clients so you can interact with container D DEMON from your Rust program. There is also Snapshotter crates. So container D has the concept of proxy Snapshotters. These are Snapshotters that can be communicated via JRPC protocol. So basically these crates let you write custom Snapshotters by just implementing a single interface. And everything else is taken care of by the library. And also we offer a full set of crates to implement custom streams if you want to benefit from Rust speed and safety features. So this stream library aims to provide same seamless experience as Go version. So basically you just implement a task service interface that manages container lifecycle. And everything else is hidden from the user. So the library will take care of parsing command line arguments, evaluating an environment, running into the RPC server behind, and so on. And also it provides both sync and async versions of the code. In addition to stream library, there is a Rust version of RunCshim. This work mainly serves two goals. So first one, we wanted to provide a reference implementation. So people have an example how to write custom streams end to end and how to work with the library. Second one, it's a fully featured production ready stream implementation. We run some integration tests on OSCI against the stream. And originally it was adopted by people from Huawei Cloud. They run it in production on more than 5,000 nodes. And they've got quite decent resource utilization. So on decent environments, you can expect more than twice better memory consumption. Now what's coming in 1.7 release? We're introducing two new APIs. The first one, we continue to work on sandbox API to quickly recap its new set of APIs that add a notion of group of containers. And by group, we mean really anything here. So it's a certain entity that allocates resources which makes sense for a particular group of containers. For instance, in Kubernetes case, it can be a pod sandbox that reserves a network namespace for your pod and runs a post containers that serves as a parent process for these containers. But with sandbox API, we can substitute this with something else, like hypervisor. Sandbox implementations will be provided by container dshims. So shim expected to implement this new controller interface that will manage sandbox lifecycle in addition to containers lifecycle. And this interface is going to be optional. So when implemented, container dshim know that the shim supports sandboxes. Otherwise, we'll default to current implementation with containers. The demon itself will remain implementation agnostic. So we'll have no knowledge of what kind of sandboxes we are running. In future, once fully migrated to sandbox API, we want to explore more whether we could get rid of post containers at all as a concept. So it's not enough to for shim to just implement sandbox interface. Someone has to call it. And this is where our CRI plugin comes into picture. In current implementation, when we schedule the code our CRI layer manages pod sandbox directly. And instead, the goal for 1.7 is to add extra level in direction. So instead of managing pod sandboxes, CRI will be calling our sandbox API instead. And in this case, pod sandbox will become one of controller implementations. The other one that we plan to add is remote controller that will call in shims via TTRPC protocol. So while working on this to keep things from falling apart and to prevent breaking existing customers of container D we decided to fork CRI server and do all refactoring work there. The new implementation can be enabled with enabled CRI sandboxes environment variable if you want to give it a try. And we plan to have it a default implementation in 2.0. Another new addition that we plan to get in in 1.7 is transfer service. So high level, it's a new API to transfer artifacts from source to destination. And in other case, source and destination will be plugins. So we can request a particular plugin when we initiate new transfer session. So this table on the slide it demonstrates current functionality supported by container D but all of this currently happening on client side bypassing container D demand. So every user of this client defines how they want to use it. There is no central place to configure it. For instance, today, container D demand does not participate in pull operation. It's all happening on client side. With transfer API, this will be moved to demand side and will be implemented as a source and destination plugins. So let's take as an example this pull operation. So we will be using registry as a source plugin. So we will be downloading container image from OCI compatible registry. And we use image store as destination. So container D will be saving this image locally. However, now with transfer API instead of registry we can request a different plugin. For instance, it could be S3 storage. And this way we can pull directly from S3 to and download images locally. These are some new use cases that we want to cover in future. We plan to implement new plugins, plugin types in container D core for image signing and validation. We also exploring better support of credentials, image encryption and so on. Some further work that we plan in sandbox API is, in further work that we plan is in box API integration. So container D shims will be able to provide its own plugin implementations and this unlocks a lot new use cases. Such as, for instance, confidential computing. So you can pull image directly to sandbox bypassing host machine and this way achieving higher level of security. Or support cross platform image pooling and unpacking. So Michael is gonna give you update on CNI. Michael, over to you. Sweet deal. All right. Yeah, I'm gonna be talking about the Kubernetes interfaces updates which will just include the CNI. Mike Brown will do the CRI and NRI changes. Sweet. So let's discuss the CNI. This should be a come at no surprise that the Docker shim is no more. And a few people asked me to clarify where the CNI is being invoked. And we'll get that clarified. For those that don't know, the CNI is what is responsible for the setup and cleanup of network resources in Kubernetes. And with that, we have our updates. So right now I've been working on the CNI and container D workflow diagrams to help clarify where the CNI is actually invoked. And we'll be posting that to the CNI container D GitHub repos. There is a big change around the run pod sandbox CRI method which is pretty much the glue magic of the pod setup. We had to do some reordering which was to fix a bug that surfaced itself as IP address exhaustion if the container exited with an error. We captured some future work to improve this cleanup in the event of failure. So look for that. The CNI maintainers actually intend on deprecating the loopback plugin. It's actually not a really true CNI plugin. And funny enough that there's actually a surprise that regardless of your CNI of choice, you still need the loopback plugin. So we intend on making that transition rather seamless. You should even know it was there and you probably won't even notice the change after it happens. If you're curious of the why's and how's just come find me. With the removal of the Docker shim, your plugin and network configurations are configurable in the config Toml and the defaults are the same ops CNI bin for binaries Etsy CNI net D for your network configurations. And just dovetailing off of that, please test your updates and not in production. There's a few hiccups when that happened. One thing that we'll be focusing on was myself, one of the CNI maintainers also contained a demaintainer is how we support the CNI 2.0 moving forward, especially around the multi network cap that is currently in play. So just to give a quick overview of where the CNI comes in, I wanted to clarify a few items. If you have any specific questions, please come find me. But just for where everything starts is everything starts once the pod is bind it via the CUBE scheduler. And the CUBE, Kubelet will use is the sync pod method to reach out to the API server to determine if it needs to schedule pod on the specific node. Once this happens, the Kubelet uses the run pod sandbox CRI method to call container D. In the order of operations from a high level R, the sandbox container is created, the network namespace is created, and then setup is called. And setup is what actually calls CNI add, but that's actually a, we use a go, our own wrapper called go CNI, that wraps lib CNI. And that calls add network list, execute plugin with result, and then exact plugin. And that's actually what calls CNI add. And then at that point, the sandbox container is now started. And what actually starts the regular containers is the create container CRI method. And that's called by the Kubelet as well, right after the run pod sandbox. And that's what starts the ephemeral container, starts the init containers, and then starts the normal containers, now many you have. And something that's often missed is status. So if you don't actually have your CNI setup correctly, AKA you don't have a network configuration, your binaries are missing, status will actually call back up to the Kubelet and say you're not ready. So you won't be able to actually schedule pods on that node until you do so. So that's often missed. When a pod is actually terminated, the kill pod function is called, was responsible in the Kubelet. And this is calls the stop pod sandbox CRI method. And from a high level, this stops the containers and then stops the sandbox containers and then actually calls the remove function in go CNI. And that actually calls the Dell network list, which executes your CNI plugins in reverse. And then you call the exact plugin without result, exact plugin, and then the network namespace is deleted. Eventually, once the garbage collection kicks in and that's via the run method and garbage collection is run on a timer, that's actually what calls the remove pod sandbox. And if the pod sandbox is not stopped, we're actually going to go ahead and call the stop pod sandbox. And then that's what removes the containers, then removes the sandbox container state. And then the pod is officially gone. Mike, you're up. So we're an open source community, right? We work together with the Kubernetes teams. We work together with the CNI groups. We are always looking for new reviewers, new maintainers. Mr. Zappup is a new maintainer for container D and reviewer. He's been doing a great job for us around CNI. He called me up about a year ago, I think it was. And we were chatting and slag, saying, what's going on? How does this thing work? And the next thing you know, he's doing pull requests and doing designs and make it all work better and helping with the CNI group as well. And I know now he's working with Microsoft on the same topic, right? We see a lot of our maintainers move to different companies and everything, but it's great that they never seem to leave. We're keeping our maintainers. We enjoy the feedback that we're getting from you guys. We like working with, I'm a, say, node member, try to get into all of the calls if I can. We work with the CRIO team and other container run times, other providers of the APIs down in the lower level. And the CRI, this Container Runtime Interface, is that API that sits between Kubelet and our container run times that we call Container Runtime Integration, proof points, we make sure that your pods, your containers run the same on all of your nodes and all of your clouds, on all of your bare metal systems. And that's what our job is to make sure that everything's working out well. The current API is at the V1 beta stage, which means it's not done yet. We're continually adding new features, new functions, new fields, if you will, to the CRI structures and APIs. And right now we're at the O25, which is a mirror of what the current version of Kubernetes is at. We bring in these, we've entered the stuff in and we make sure that the APIs work. And we're always looking for help and adding new features. For example, there's been some issues over the last few years around isolation and security. Well, we'd like this user namespace support to be one way that you can isolate better your pods in the containers with a user namespace isolation as done in Linux and not have to wrap it in a virtual machine so that you can protect yourself from a user exit of a container getting access to root. Today, if you wanna have that extra layer of protection outside of the existing namespaces that Linux supports for containers, you would have to have it at a VM, okay? And with user namespace support, we'll be able to give you an extra layer of isolation. We have an open issue and we're still working on PR, details to be worked out on the container D side, but we know what we need to do. Checkpoint restore, this is an interesting effort that's been going on for a long time by Adrian and Reaver and a bunch of people down in Linux, if you know about CryU, but basically the idea is you freeze your container, make a copy of everything, including memory, store it on disk, and then you should be able to debug the container and resume that container and restore it. We don't have it working with pods yet, that work is underway, and we currently don't support freezing and restoring containers, you know, by default, there's a feature branch, you have to turn it on, and it's mostly focused on the forensic effort right now, we're gonna do things like secure the where we're storing your checkpoint, so stuff going on there, registry configuration. So, apologies here, it's hard to configure container D, admittedly, we do support configuring to point to your registries, but I think it's important to point out that we don't own the job of actually doing that, we share it with Kubernetes, Kubernetes has, as you may know, image pull secrets, and they own the mission, Kubelet does specifically, of pulling your images through the cry API, so when a new pod is coming up, what they'll do is they'll ask the container runtime, what is the status of the images for all these containers that I need to run in this pod, and then based on that point, they'll start calling down into the container runtime in an imperative way, asking us to pull those images, and of course we have to do that resolve, and you may have seen some earlier APIs that we're gonna do to make this better, but come ask questions if you need to know more details about how this works. So, NRI, Michael put together an interesting project called NRI, it's been one of our sub-projects, and we're improving it to support actual running programs that could use the TTRPC API to request for changes to your specs, OCI specs. This is to be able to support additional advice types, the Intel team has been doing a lot of work here, they would like to be able to improve the way we configure OCI specs, and Kirsten Licky is doing some work here, come to the Friday call and we'll have more detail on that. OCI, of course OCI is the base spec for containers, we're working continuously with the OCI team, we're actually maintainers as well with OCI, there's been a new work group that was successfully bringing forth artifact support into the spec, and we're working out some details, but basically we would like to be able to not just pull container images, but also artifacts from your registries that are associated with those container images, things like gas bombs and image scanning results, and other types of projects that are underway. So please come talk to us about how we can enhance support for container images with these new artifact types, and how they work and what we're gonna do in container D around that, okay? I think we have time for questions, and Phil? Yeah, go on, amazingly we are on time, which not to throw my kind of the bus, but we thought we'd talk longer than that, so we're great, we're in good shape. So the CNCF asked us to put up this QR code, only scan that if you think we did a great job and you wanna submit feedback. No, you can provide any feedback on the talk there. I don't know how, I don't see Mike set up in the room, but we do have five minutes, I think, before the official end of the session, so any questions in the room? And again, if not, we're happy to stay after. The question was, was that diagram about the CNI flow post the fix about the IP exhaustion issues we had? The answer was yes, that is in our main branch, and I think it actually made it in 169 release just a few days ago. Yeah, so question is, is there also a diagram of before we made the ordering fix? Yes, so if anyone else is interested, Zappa can help you out with that. Go ahead, yeah. Yeah, yeah, good point. Yeah, so in prior talks, there's the CERN folks that have the CVMFS Snapshotter, there may be others. Apologies, there are a lot of interesting innovations in Snapshotters, yeah, in this talk, we only mentioned lazy loading, but like Maxim said, the ability to have a proxy Snapshotter means, as long as you handle the simple Snapshotter interface, you can write that on top of a distributed file system or whatever interface you'd like. Yep. Yeah, what was your name? Okay, yeah, so Jerry's here in the room, distributed file system, Snapshotter, yep. Cool, anything else? Yes, sure. Yeah, so question was docker shim deprecation. If I'm no longer running Docker on my node, what's the implication to pods running on that node that may have expected certain things about the Docker environment? Also follow on, now I have a problem and I go into that node and now there's no Docker, how do I debug? So yeah, there's a few great blog posts that have been written since the 1.24 release on some of those topics and I'm sure we can't exhaust it here, but one is that there are some nice tools that people have written to detect pods that are trying to mount like the Docker socket. So you've got containers within a pod that are actually trying to do something, interact with Docker, that's gonna be an immediate issue. Container D doesn't offer a Docker socket, not the Docker API. So that's kind of probably the biggest thing we've seen people have issue with is I had something that depended on Docker, I had something that mounted, expected the Docker interface. For debug, obviously there's a lot of improvement in tools that are using the CRI, so CRI, CTL or Cuddle or whatever you wanna call it. That can talk to any CRI runtime and do a lot of the same list containers, list pods, stop, start, list images. So some of those same command line tools are available. Again, you lose some of the muscle memory of well, I knew the Docker commands, a lot of those similar things are available. We have a simple CTR tool, there's NERD CTL, so some of these tools are filling in the gaps of I can still do poke around, see what's happening, what's the status of my tasks or processes. Yeah, so again, there's a lot more on that and I think there's some good blog posts that have been written about how do I convert what I'm gonna run into as I transition. So we are completely out of time, but thank you all so much for coming. These slides will be on the sked.org site under our talk. If you wanna follow up with any of us, we'd be happy to. So thank you so much.