 Good morning, good evening, good afternoon depending on where you are. Let's give it a couple of minutes. All right, so I think we can get started. We've got nine people on the call, so welcome everyone. So we've got two items on the agenda. Today we have Brandon, he's going to talk for I think five minutes about the security white paper. And then we have Akahiro, and he's going to talk about the current state of rootless containers. So I'll pass it on to Brandon. Awesome, thanks Ricardo. Yeah, so I was just speaking to Ricardo and I mentioned it about a couple months ago, or several months ago now in the last couple of weeks, we announced the six security white paper. This was kind of a collaborative effort. We had this really nice document. Actually, let me share my screen on the button. So a while back, we kind of released a white paper. This was like something very conceptual on a high level covering different aspects of cognitive security. And part of that, we would get a lot of good information from the various different groups. But really what this white paper was about is kind of touching on concepts, very high level here, very generic recommendations, nothing to do with anything specific. Kubernetes isn't even mentioned within this white paper, because it's supposed to be non-biased, really looking at concepts. So for example, if we go into the runtime section, which there's in the link, so I'm going to use the Markdown one, which is a certificate, which we also have this in transit and Chinese now. So you've got the runtime environment, we talk about the different phases, how to compute, what needs to be scale, the orchestration level, protecting resource, audit and things like that. But we don't have specific tooling involved with this. So what we've been recently working on to kind of augment the information here is something we call the security, cognitive security map. So the idea here is that the idea here is that it would be kind of additions to the cognitive security white paper. So you have information about the general concepts, and then you can click in and you could see the different projects that are related to the different ideas or security requirements, and then there will be examples. So kind of just to give a quick overview, we are currently curating content for this. So one example here is for security checks and development, we will have a list of projects, so this is kind of similar to landscape. And also we will have examples to kind of help illustrate for someone to go in and say, what would a security control within this area look like? And so the examples of what you would do, and these would be very specific to, okay, if you were using a specific technology, what would this look like? They are meant to kind of just give a more graphical idea on what a control may be. And what's going to happen is all this, we are currently working on kind of a UI to put all this in. So the idea is all this information would be navigable. So if you're interested in, you know, development, you could go in and click on this, go in, and then there would be lists of projects, there would be lists of examples. And, you know, one thing that we want to do in the future is also, for example, if you're talking about in the distribution phase, right, for artifacts and images, let's say you have signed trust and integrity. The idea is like, whenever you have signed trust and integrity on the, on the, the DevOps slash the distribution stage, you still want to also be able to enforce that integrity at the runtime as well. So our plan is to have linked content points. So, for example, if you look at, you know, this artifact integrity, that would be a link here that says, okay, you may also want to check out the runtime section on this. So then you cover how do I assign the images, but at the same time, you also cover within my orchestration system, how do I verify these images when they're running? Oops. So what we are looking for right now is content contributions. We have under the runtime section, we do have a couple that are still kind of semi-filled or not yet filled. So this is kind of like a plug to see if anyone in this group is interested in contributing to any parts of this. That would be super helpful. Especially, I know, like, the group here is a lot more familiar with the runtime projects. So this could be a good way to also introduce that ecosystem. Yeah, that's kind of all that I had to share. If there's any questions, if not, this is pretty much all I have. What are some of the runtime sections that are missing some part? Initially, we had our resource request limits, contrapping authentication, secrets encryption. I think this was a tree of the main ones. I think these and bootstrapping and the bootstrapping. So these are a few that I'm missing. I would say the main focus areas that we are lacking expertise is resource request and limits and the contrapping authentication and secrets encryption part of it. But I would say, you know, whatever, I think a lot of projects on the runtime side, we don't have great visibility on. So, you know, if anyone can also just go through the different projects in those sections and say, okay, maybe this additional project may be helpful here, that would also be, that will also be very good for the document. Got it. Thank you. So one more question. So do you have a sample of one of those sections? Yeah. So let's pick one here. Let's pick runtime. I haven't reviewed the one that yet. So let's see. So this is like service mesh. So just a bunch of service mesh projects. We also are kind of keeping track of a few commercial projects as well. So there's actually another motivation of this is that we also want to look for gaps within the CNC and ecosystem. So part of this exercise is kind of seeing, okay, if there are certain areas where there are open source projects, sorry, there are commercial projects, but there are no open source projects, then we would try and make a recommendation to the talk about, you know, maybe we should try and get some of these products in or, you know, try and see whether we can create new products around these areas. So I think runtime is actually at the, if you scroll down, yeah, there. Is it this one? Yeah. Yeah. So this one is, like there's only one project, right? So maybe there are some that we may not be aware here. Maybe the EBPS stuff, stuff in the area. Yeah. So it's not something that's like content heavy is pretty much, you know, yeah, there's a project here, some examples, and just to help help folks get a better idea on how to secure the aspect of the ecosystem. Got it. Thank you. Any other questions? Oh, was the link to the doc published anywhere? Oh, yeah. I'm going to put the link to the issue and to the doc in the chat. Thank you. Once I can find the chat. All right. And thanks, thanks, Ricardo, for the thing we come by and share about this. Yeah, you're welcome. Thank you for coming by. Okay, so now we have a key hero and thank you for joining. I was excited to have you talk about the state of ruthless answers. So I share my screen. Sure. Can you see my screen? So let me introduce myself. I'm a software engineer at entity, telecom company in Japan. And I'm a manager of several projects, including RunC, Continuity, and Moby, and also several subs under dw.com slash continuous. And today, I'll talk about continuous, which means running continuous runtimes and also continuous, of course, as a non-root user on the host just includes running all serial runtimes, such as RunC, and serial runtimes, such as continuity and cryo, and CNI problems, such as front now, and also keyword and keyword proxy. So this is useful for protecting the host from potential vulnerabilities and misconfigurations. So even in the rustic case, the attacker can gain the normal user on the host, but the attacker cannot gain root privileges. And the rootless containers are implemented using a kind of feature called user name spaces, which allows mapping a non-root user to something like root users, but with a limited set of privileges. And in the Kubernetes signal, there's a similar enhancement proposal about user name space, but rootless containers is not related to this user name space enhancement proposal. And do not conflict either. There, a user name space proposal is about just running continuous as a non-root user, but they run runtimes as the root user. On the other hand, rootless containers run everything as a non-root user. So the rootless containers is quite different from the user name space proposal. And of course, rootless containers is not a panacea, and there are some drawbacks. For example, the network throughput is slightly slow, just like 50 GWPS to 10 GWPS. So this is really slow, but we are seeing huge improvements in these years. And rootless containers cannot support NFS and block storages, but this is not a huge deal for Kubernetes, because when you can use managed databases and object storages such as Amazon S3, you don't need NFS and you don't need block storages. So I don't think this is a serious problem. And let me show the history of rootless containers. It started about 10 years ago by LXE folks, but this wasn't popular until a few years ago. So in 2018, Bill Teet started to support rootless nodes, and we also had Jocular Continuity, Portrayon, and Cryo, all of them supported rootless containers this year. And we also made a purchase for Kubernetes to support rootless nodes, but the purchase are not merged into the Kubernetes upstream at this moment. But T3S already supported rootless nodes in 2019. And this year, we added rootless support for kind Kubernetes in Docker. So now you can run Kubernetes in rootless Docker and also rootless Portman. And with regard to root layers, there is a lot of news. So last year, there was a new feature called SETCOMP activity. And in this year, we are seeing a kernel mode overlay with rootless containers. And what should I runtime spec is being modified to support SETCOMP activity? I will talk about this topic later. And let me show some examples of rootless containers, such as UserNetist. UserNetist is our reference distribution of Kubernetes for supporting rootless nodes. And we have a demo using Jocaconpose for showing a module of the demo of Kubernetes cluster. I don't show demo today, but I have some screenshots in this screen. So if you run Jocaconpose up, you will have a kernel-based Kubernetes cluster. And these components are running as a non-root user. And we support the both continuity and cryo as the CRY runtimes. And T3S is also supporting rootless mode using the patches of UserNetist. The patches are not matched into the Kubernetes app scene, but T3S uses a whole version of Kubernetes, so it already supports rootless mode. And T3S uses continuity as the CRY runtime. And just a rustic, kind, Kubernetes in Docker started to support rootless mode. And this way we included in kind version 0.11. And kind supports Kubernetes without patching. But for supporting Kubernetes without patching, we have several complicated hacks, such as bind-mounting dev-nur into slash-proxy, sorry, slash-proxy, slash kernel to emulate some cys-controller values. And we also need some complicated configuration for Kube proxy to avoid setting several privileged CCC values. But we have several complex hacks about how we can run Kubernetes without patching at all. And this way we included in kind version 0.11. So within a few weeks, you can try this easily. And the topic of this year is SecComp. So just two years ago, kind of version 5.0 added a new feature called SecComp user notification. This is a new way to emulate system calls in user space. This is very similar to P2S, but it's more lightweight so it doesn't need context switches compared to P2S. And this feature wasn't supported by the also runtime spec. But just two days ago, there was a progress that was merged into also runtime spec. So also runtime spec now supports this SecComp user notification. Yeah. And with this continuous, we'll be able to use this SecComp user notification for emulating sub-URGs without a file called slash-etc-sub-urg. So previously, you had to have this file. This file is something similar to slash-etc-password and slash-etc-group. And you had to configure this file for the previous users. So this is very difficult on other environments because you have a really bunch of users and you had to configure itc-sub-urg. But with SecComp user notification, you no longer need to prepare this slash-etc-sub-urg file. And also with this SecComp user notification, we can completely remove set-urg binary files such as user-slash-bin-slash-new-urg-map. So this is better in the security. And kind of 5.9 added support for SecComp IOCTL notif at fd. This is a new feature that allows injecting file descriptors from a host process into container processes. So we can replace the file descriptor of the socket on the connect-c-score. So we can remove the overhead of user-mode networking called slurp. So this is very faster compared to previous users' containers. And we have some proof-of-concept code but not ready for production at this moment. So there are several tasks that need help from the community. So we need to have more experiments in the SecComp. We also want to port-over rootless kind to MiniCube with Docker driver and Portman driver. And we have a proposal for key-minted signals but this proposal is not merged yet. So we need some help to facilitate acceptance of this proposal. And we have some questions about rootless containers on SQL workflow and Reddit and also on a similar community site. But I don't have enough bandwidth to answer all these questions. So it should be great if we can get help from the community to answer these questions. And rootless containers components such as rootless git and counter-run times such as runc and controller g and mobi, all of these components have several bugs and I don't have enough bandwidth to work on these bugs by myself. So I'd like to have some help from the community to resolve these bugs. And for further information please visit hdps.com, rootlesscontain.rs and last year in cubicle we have some presentation about rootless containers and we also have a proposal to change signals. That's all. Do you have questions? I have a question about the block storage support. Is it a go or is it something that the project owners are looking into and is it just because the mount is not available or why you cannot use that? So counter maintenance do not want to support username spaces for x4 file system and xf file system and so several file system drivers are likely to have bugs and they don't want to expose these potential bugs to non-root user. So the counter maintenance do not want to have rootless support for x4 and xdface and other personal ready file systems. So we cannot support a block storage. But we could have huge file system in user space for implementing x4 and xf in the user space. So potentially we can support hub block storage. But we are likely to have significant performance overhead because file system in user space is slow. But we can support block storage if we want to do. Okay, thank you. I have a question. I have a few actually. But one is, is there any update on a kernel supportive overlay fs from user namespaces? I know that they've talked about that for a long time, but I know there's fused overlay fs, but is there, so there you go, the kernel mode overlay fs. Oh yeah. So there's two implementations of overlay fs in the kernel mode and in the user space fused overlay fs. But kernel mode overlay fs didn't support rootless until kernel 5.11. So it is already in 5.11? Yeah, 5.11. It's released last month, I think. Ah, okay. I hadn't heard that. So that could be a good improvement in performance hopefully over fused overlay fs. Yeah. Another one is I run into lots of problems with getting the PID namespaces. Backer and Kubernetes block them by default. They put in, they mask them by mounting other masking, by mounting other file, other things on top of slash proc so that you can't mount a new slash proc in an unprivileged user namespace. Do you have any insight into that? I presume the rootless ones don't block those things like Docker and Kubernetes do. Oh, sorry. You said slash proc slash what? Slash proc for unprivileged slash proc. So if you have a new PID namespace and an unprivileged user namespace, right? So you can't, so in order to make a new PID namespace, which, you know, any Docker container has its own PID namespace and you need that for any time. So, but you can't run within Docker, you can't create another unprivileged PID namespace because they block slash proc. Unless you run with, you know, the unmasked, it's called the security. Oh, yeah. So basically, you need to bind the mount slash proc file system. And rootless port amount already supports dash slash PID with such bind amount. Which does it just say? You mean bind mounted from outside the slash proc? Yeah, yeah, bind mount slash proc to continue. Yeah, but that you can do, but if you want to have its own PID namespace inside of a container, then you need a new slash proc. You need to have an unprivileged PID namespace. So I'm talking about a separate, to be able to isolate two different containers. So in just case, if you have a mount namespace along with a PID namespace, you can just mount proc. And all of the rootless containers, including Docker and Potomac, supports PID namespace in that way. So if you have some problem, maybe you can post the coffee JSON of OCR runtime to the GTAW issues of RunCityPo, or maybe somewhere else. So maybe I can look into that to help us with the problem. So I mean, Docker is relatively easy because they have this option security system pass equals unconfined. For that, if you're just running inside Docker, you can run this and it will allow you to mount a slash proc. Kubernetes, we have a regular K8S, we haven't actually figured out how to enable the corresponding option. It's supposed to be a way to do it with proc mount unmasked, but we haven't got it to work. And I presume the K3S and stuff don't block these, don't have this by default, blocking the mount of a new slash proc. So okay, so you're not, you probably haven't run into this and you don't use Kubernetes anymore. You've got all these other variations of not K8S, the, you know, use K3S probably, you know, whatever else kind. Yeah. I guess we need to have work in the kernel to facilitate the amount of proc for a result such as security opt. Yeah. So I think we need to have some work in the kernel. Yeah. This is a beyond kernel problem because the kernel allows it, but the problem is that these container systems are blocking other things inside the container from using it. So because they're doing, they're putting, they're deliberately putting in other things on top of slash proc so you can't, so to prevent it. So by default, Docker will make those confined of slash proc and so does Kubernetes and Kubernetes makes it even harder to do that. Now, loud proc types is the option inside the pod security policy and we haven't got it to work. So Dave, if I may ask, what are you trying to do at a high level? What is the overall goal that you're trying to accomplish within the container? Like, I'd like to be able to run a completely unprivileged container within a Kubernetes pod. Sure. And in particular, we use singularity unprivileged and with the dash dash PID option so that you can create a separate PID namespace. So I put a link there to a runtime that may help you. I'm the lead developer at runtime called Sysbox that allows you to create a rootless boss inside of which you can run things like Docker and system.dn, microservices and stuff like that. And it will probably solve the problem that I think you're trying to get at. If you run Kubernetes inside of this? No. Or you can deploy the pod with Kubernetes and that. I say, so you run Docker? It's like a new run C. Okay. So that's a replacement for run C? Yes. I'll take a look. Hey, I can hear you. If I may ask, my name is Cesar and I am, as I mentioned, the developer in the runtime called Sysbox and you run C. And number one, I wanted to do this myself and thank you so much for all the work that you've done on bootless containers all the many years. It's a lot of perseverance from your part to work on very challenging technical feeds in the kernel. So we really appreciate that. The other thing is, just at a high level, when you talk about rootless containers, I almost feel like the term rootless containers is not entirely accurate to what you are actually doing. I think what you're trying to do is a rootless runtime because the rootless container itself is sort of a subset of that. A rootless runtime generates a rootless container, but the reason I mentioned it is because, for example, in the Kubernetes KEP that you mentioned right there, there's going to be the notion of a rootless pod, but Kubernetes itself is running rootful in order to do whatever it needs to do. And so I can sense already a bit of confusion there where people say rootless containers doesn't mean that Kubernetes is running rootless or doesn't mean the pods are running rootless. And they're two different things. It seems to me in your case, you're really targeting a very challenging problem, which this is a rootless runtime, which has a lot of merit, but it is not quite the same as the rootless container because the runtime can be rootful, but the container can be rootless. So what do you think about that? Yeah, it makes sense. Actually, rootless containers, it's not something I invented, so there was already rootless containers when I became interested in it. So actually, it's difficult to re-name rootless containers, but I have ways that rootless runtimes. Even the name rootless is already kind of weird because you are root inside of the container or inside the pod, but you're not root at the host, so there's also that. There's a whole thing, you know, but yeah, just so that I would mention it. In our case, for example, with the runtime that we developed, all the containers we generate are rootless, meaning they are in the Linux username space, right? And we do things like virtualizing ProcFS, virtualizing Cisco Trapping, you know, special mounts, ShiftFS, you know, we're doing all sorts of advanced things to enable these rootless containers to run things like Docker, SystemD, and microservices on the side. But it is just rootless containers. Our runtime itself needs root because it needs to do very advanced things like Cisco Trapping and things like that that are really hard to do. Rootless, you know. So we're, I think our results are rootless containers, but we're not a rootless runtime, you know. And I think a lot of what you're mentioning here is a rootless runtime, which is an even more challenging thing than what we're up to, if I may answer. Sounds like the backing of scope creep, yeah, salary rootless containers and not a rootless one. Did you say that again? Sorry, I didn't catch that. No, it's a bad job. I was saying it's probably maybe scope creep, you know, they want to do rootless containers and then they're like, why don't we make it the runtime as well? Yeah. I had a question out here actually about the sitcom stuff we talked about. The user notifications, you said that it has better performance in pictures, you know. Do you have any details about, you know, if we were looking at this last time, if not right, we looked at S-Trace. Is this supposed to be better than S-Trace as well? Do you have any experiments or any numbers that you could kind of share? Yeah. So for pictures, you have to inject hooks for every C-scores, even for C-scores, you are not interested in, but with second visualization, you can only inject hooks into the interesting system course. So the number of content switches is proportional to the number of C-scores you want to emulate in the user space. Okay, but do you still get like the overhead of having to go back between kernel space and user space, right? But I guess that's not as big as, that's not as bad. I don't have benchmark data. Sorry, I just wanted to ask you. You said you wanted to get more experiments on sitcom. Yeah. What kind of experiments are you looking for? Yeah, it would be great if we can get some experiments overhead of this C-score visualization. Yeah. Okay, so you're looking for performance data. Okay. Thanks. If I may add, are we, in the simple runtime, we use this mechanism quite a bit to intercept certain system calls. It is very useful. And because we can select the specific system calls that we want to intercept. For example, intercept the amount system call, because it's very important that whenever PROCFS or C-SFS are mounted inside of one of these wireless containers, it is our emulated PROCFS and C-SFS that get mounted, not the kernel's PROCFS, otherwise you create a security hole. So we use that second user notification mechanism. And it also has the advantage that you can just decide when to process a particular system call. And if the arguments are not the ones that you need to process, you can send it back to the kernel so that the kernel can do the regular processing of the system call. So it doesn't, like the amount, for example, system call is a very complex system call to emulate, right? So we only emulate it in certain scenarios. In the case that we don't care about it, we send it back to the kernel and then the kernel continues with the processing, right? I do think that it does add a bit of overhead, so we only use it in control path operations. We shy away from using a measure of data path operations. And how much overhead, it depends how much, what are you doing there with that particular system call in user space, you know? It's a very powerful mechanism. Yes. Yeah. For emulating sub-urges, we have to emulate a lot of system calls such as C.J. Orwell and C.G.U.I.D. and C.B.U.I.S.C.S calls. Yeah. So these are very, yeah, complicated. What are you thinking of doing as far as a set comp, like a hero for ETC for emulating sub-U.I.D.s without the ETC sub-U.I.D. file? What exactly does that mean? What sort of system calls would you be intercepting, for example, in your user second? So, without a sub-U.I.D. file, we can't emulate multiple users for root-less contracts. But with second-user notification, we can emulate system calls such as C.J.U.I.D. to adjust no operation, but modify the system calls such as C.J.U.I.D. to return fake value. And we can also inject hooks into C.H.O.I.D. to fix ownership of the file permissions. Got it. Thanks. I had one here. Actually, this was also the initial trigger list, but in your help wanted slides after here. It's kind of amazing the amount of work you've done tracking this in all these different projects. It's insane. But my question would be, what's the best way to help you out? Because I don't know if you need some help from the end-users to try this out. But it's not very clear to me how to report this. Would it help to have some place where we can coordinate all these things you're doing in different components and maybe report user experience? What do you think? Yeah. I think it'd be great if the community can have more tweets and blog articles and answering questions on Stack Overflow and Reddit and other community sites. That would be very helpful for end-users of risk containers. Okay. Yeah. Thanks. Yeah. I was wondering, I don't know, maybe Ricardo can say, I don't know where it would fit. And if there's one place where you could have a working group dedicated to this, where we can track all the points, all the work, all the components that have support, what's missing. Because I guess it's in your head. But I'm wondering if it's worth creating something somewhere for this, just dedicated to rootless support. So we have hgps.com, rootlesscontain.rs to answer your question. I will check it out. Yeah, maybe. So I think I'm interjecting here. But I think what Ricardo is talking about is, you know, maybe from the CNC of how the community can organize around it, right? So and into one way, the SIGs and the TOC have of organizing the communities by creating work groups under some of the SIGs. So I mean, there will be somebody who drives that, like when it could be you, if you're interested right here, it could be the chair or the lead of that community, and basically gather the people around so that you get more contributors, right? And then also you get the support from the CNCF. Sounds interesting. Yeah, so yeah, if you think that's a good way to go, you know, the CNCF will be happy to be there to help you out. And Ricardo is also in the TOC. Yeah, I think if you're interested, I think we can come up with something to give more visibility to this effort. Oh, thanks. Yeah. So it just continues. We have several people, such as Alex Cercera from Susie and Giuseppe Scribano from RETAT. So maybe I will ask these guys about the SIG group. Yeah, that sounds great. Yeah. Thanks. One more thing I wanted to point out is I'm submitting a change to your singularity page on rootless containers because there you're talking about the set UID mode and fake root mode of singularity. But there's actually that's not the main either of those. Well, let's say there's another, there's a third mode, which is more the default when you're running unprivileged. You can run unprivileged singularity without fake root. The default mode is to just use what's called unprivileged user namespace mode. It has no root at all inside the container. That's actually more rootless than what you guys are talking about. There's no root user at all inside the container. It's only the original user, same one inside and outside. So it can be completely unprivileged and it doesn't need any help or new UID map tool or anything like that because it just has no root. So that's not useful in some cases, but it's useful in a lot of cases too. If we're trying to isolate, we have an unprivileged user that runs other users payloads. So this one's very useful for that because we don't care if they don't have any access to root, we just want them to. So it's not nearly as complete a container system, but it's still very useful. Something interesting. Yeah, yeah. So I think I need to modify this one just especially about singularity. Yeah, I'll send you a poll request. Oh, thanks. That's great. So Kihiro, if I may ask, what is your vision sort of where rootless containers? I mean, the project that you're working on, what is your vision for it? When would you say, okay, I'm happy that we've reached the state where I accomplished everything that I wanted to accomplish for this project, right? Where do you see this like say five years from now if you've had your way? Sorry, you said, what is your vision for the project? Where do you see it going, let's say in five years, right? Yeah, in five years, I think most drug users and Potomac users will use rootless containers by default. I think it's highly likely for local containers on laptop, but for promoting rootless containers to production clusters, especially difference clusters, we will need some help from cloud providers such as GKE or AKS or AKS. Yeah, so if we can get help from these human service providers, we will be able to promote rootless containers into the production clusters within five years. And by that, you mean the whole Kubernetes run time itself running rootless, right? The user-native approach, that's what I'm talking about. So in the ideal world, so GKE and AKS and AKS will have a checkbox on their web user interface and click here to run a QBRED and the continuity as a non-router user. So that's my dream, I was in five years, yeah. But I don't work for Google, I don't work for Amazon or Microsoft. So obviously, as a team, help from the industry. And the goal there being to protect the host from vulnerabilities in the Kubelet, for example, itself, right? Or in the underlying run time, is that correct? Yeah, yeah, that's the main motivation, yeah. But aside from that HPC users, hyperbolic computing users, we want to use rootless for some kind of multi-services in a shared cluster. And just a general question for anyone that wants to answer is, I mean, in your experience, do people care more about securing the run time itself or do they care more about securing the container itself, right? Because as I was mentioning, the Kubernetes KDP that would use our namespaces would secure the pot itself, but not the Kubelet, right? The Kubelet will continue to run root, but the pot itself is now rootless, right? I'm wondering, you know, obviously, if people could have both, they'll probably have both, but I'm wondering at this stage of things, where is more of that demand? Is it more on protecting the run times or just protecting the container, the container isolation boundary itself? Yeah, so I think both points are really important. And so, and it's a rootless mode, and the user namespace, it has been proposed that do not conflict either. So we can just send them together by nesting user namespace inside rootless containers. Yeah, so actually, we run times as non-routes such as UID 10000, and also run ports with different UIDs such as 2000 and 20001, and such as, so yeah. I see, I see. So you feel that it would be first rootless pots, initially just the pots, right? The Kubelet will continue to run rootful, but then eventually, as you continue to work in enabling user NADs and running the Kubelet, let's say rootless, at some point, they could interact together, right? In the sense that you have the Kubelet rootless, and then inside of that you could even nest a username space inside for the pots themselves, is that what you're saying? Yeah. Yeah. Thank you. Yeah, I think if I may add, the production environments would like to have something less privileged at the lowest level. So I think the rootless run times hasn't been necessarily available in all places or hasn't been available at all, right? So what some people have been doing is just trying to secure the pot or secure the container, right? So more at the nested level. So yeah, if you are able to break out of the container and if you are running something that's rude, the Kubelet or the runtime, then an attacker could actually gain access to the host. So I mean, I think the largest or the biggest concern is that somebody could gain access to that host and cause some damage, right? So and that's what people have been talking about with multi-tenancy and also, you know, some of the other run times like Cata containers that run the container in the VM. So it's actually preventing that access to the host. So sorry. So what was your question? No, it wasn't necessarily a question. It was more of a comment. Yeah, yeah. So it's more like an opinion, right? Yeah. So actually, Cata containers provides more stronger isolation. Yeah, but Cata containers require CPU support for virtualization, such as Intel BGX. And this virtualization is not available on typical cloud instances. So it's available on Azure and also Google computing engine, but nested virtualization on these clouds are really slow. And AWS doesn't support nested virtualization at all. So, but AWS provides bare metal instances. So you can run Cata containers inside bare metal instances of EC2, but bare metal instances are really expensive. Yeah. So running Cata containers on on-premise is really productive, but running Cata containers on cloud is somewhat difficult. Yeah. Yeah. And I think that's one of the things that when we were developing Cisco, we wanted to create a rootless container capable of running same workloads as VMs, because we wanted to avoid the VM, basically, right? And so, but to your point, Ricardo, I mean, we're using things like second notification, what you do, that's just called trapping and advanced techniques like that. And I just don't know how I don't see us going, making the runtime rootless, because we do have to be root in order to be doing things like Cisco trapping from other processes, right? So for us, at least short term, we don't have rootless running the runtime that we created rootless in the short term, because we're just doing things that require really true root sort of permission like Cisco trapping, just one example. Having said that, if we could run rootless, that would be great, also, right? Because then we could secure ourselves too, right? But our focus has been more on the container itself, yes. But I think it's very challenging. I think at the work that Hacker Hero has been doing all of these last years is securing the runtime shows how challenging it is to secure the run. And it's getting more challenging because things like EVPF and and second of the fact that a lot of the kernel power is going to move into user space, but that requires some sort of high privilege in order to use that, right? Yeah, it's kind of like a chicken and egg type of problem, right? So yeah, you want to allow, you know, or you want to have less privilege, but then to have that less privilege, maybe you need to have some sort of privilege. That's right. Yeah, it's an interesting topic. I think it's because people have talked about over the years on how to be more secure and allow less and less and within containers. But then in the end, it's a matter of what's the most important for particular users, right? So like, there's got to be some compromises, right? So if you are, for example, you know, preventing people from using root, but then you need to allow something, you know, so. I think part of it is also like just granularity, a lot of the capabilities have kind of scaled down from a big umbrella over a couple of years, so that helps. And I think just like, you know, having parts of the code that you're using, which are privileged, be measured and attestable, I think that that's kind of like both soft, half of the problem as well. So before we run out of time like a hero, if you need any help, I mean, we're not, the work that we're doing is not exactly the same, what you're doing, right? Because you're figuring the runtime, so we're just securing the container. But there is an overlap, right? There's an overlap, certainly. So we'll be happy to help you in that area of overlap right there, right? And the overlap revolves around the rootless container itself, right? And the things that can run in the rootless container itself, right? That's the area of overlap. And so, you know, we'll be happy to help in that area, for sure. Oh, thanks. All right. So I think it's nine o'clock. Well, thank you everyone for attending. Thank you, Akihiro, for the presentation. Thank you, Brandon, for the presentation. Yeah. And if you want to keep the conversation going, I mean, we have a Slack channel so we can also have any conversation there and follow-ups and any questions or anything related to the work that Akihiro is doing or Sysbox is doing. And yeah, and also Syssecurity. Great. Thank you. Thank you.