 Good morning or evening wherever you may be. Hey, Jeffrey. Good morning. Hey, Tal. Hey, Taylor. Good morning, everybody. Give it just another minute or two, and then we can kick things off. All right, we're a couple minutes in. I think we can go ahead and kick things off. Taylor, do you want to start with the least-privileged best practice? Well, we can. I was and surrounding a bit late. I've been here joined here. Does anyone have anything else that I'd like to add to the agenda before we get gone? If you haven't written it, do you want to verbalize it? Yeah, so I guess if no one has anything else, then I'm good with starting, Jeffrey. OK, do you want to give an overview? Yeah, so this is, I guess, just to start. This is related to the least-privileged discussions that we've been having. Least-privileged, there could be principle, I guess, the least-privileged principle. We could have several best practices that could come out of this and it will seem like no root in containers would be one that should be pretty reasonable to have agreement on. And we can work on others, of course, but this would be a first. And maybe the other thing about this was if we can get get a few best practices in there, into the repo, then it may be easier for other people to read. So you're looking at we have a template and we have the only one there is not really a best practice, it's the process. So it's harder to look at that than if you're the type of person that likes to start from examples. So it's twofold, get this best practice in and have it as an example. And anyways, so it has all the sections except for the user story section. There's a lot of other content, so we're pretty close to writing out an actual user story. We'll probably link that to a new PR for that. But it has all the other sections, including a pretty pretty extensive reference section for going out to what other people talked about in this one. But yeah, the the general idea would be not having any processes in a container that are running as route so that they can they would if anything goes wrong, they're less likely to cause damage primarily to themselves since there is some isolation being in a container. And potentially if there's any type of shared resources, like containers, multiple containers in a pod, and there could be some type of damage if there's not isolation between those, but that's the main idea. And and then when you're thinking about running a process that's non-route and what you're having to do as far as dropping privileges, then it just moves you towards that whole mindset of running with the least capabilities necessary to get the job done. Some environments, like SE Linux based environments requiring dropping route privileges. So if you're already building your application to not require route, then you should be good to go in those type of environments. Yep, that would that's kind of covers it. We have the proposal section talk about when you're building a container and how that's related and what you can do. Some of the trade-offs, I guess, would be the first process by default may be running as route. So you have to think a little bit more. I guess it's a little bit more work up front. There may be upstream images that didn't think about setting the user, the UID and GID so that they're not route, which means if those are your base configurations, you would have to take that into account if you're building on an existing upstream image and then do modifications in your own Docker file, etc. And most of it's just around the, I guess, the extra work required and does the process expect things to be route-owned and the file system, like wherever it's accessing and making sure directories and stuff are owned by the user that you create. So let's see. And this would be work-leg context. And I'm talking while you're showing your screen. Jeffrey, would it be better if I shut my screen while I'm talking to different points? All right. Let me do that. Let's know when you want us to jump into. Oh, yeah. Totally. If you want to jump in. I guess the last part was just the work. I can't share my screen unless you stop. There we go. Cool. All right. So this is just looking in the, the diff display diff for you. And there's all the references, different places. Work-leg context. So this would be everywhere. This by default. And pretty straightforward on a test plan, but there's a split between doing static analysis and then runtime analysis for stuff like, did they set user, the run commands allow you to express the user and other things like that. So there's things you can look at before the CNF starts running. And then you can do various things with the runtime analysis. Baco has some stuff. It's a CNCF project. And there's other projects out there that can try to look at the environment. See if any containers are running with root. Yeah. And so I guess that's it. Folks have questions. I don't think there's, I guess there has been some comments. I guess before going through those. I apologize. When I look to my browsers this morning, I realized I never actually submitted my review. Oh, all right. If it's in draft, you can hit submit. I didn't write this as a comment, but maybe we'll just raise it now. Another issue I thought about is it's quite rare, but there are pods that use system D. Various reasons to do it. It's not so simple to set up, but you can definitely set it up. In that case, you might be running system services that are root. So it's not, your main entry point might not be root, but there might be, yes, there might be things running in root anyway. And I can think of other things, prawn jobs, it could also be through system D. It doesn't have to be system D, right? It could be any kind of a system service. Even in that case, do you need the root user or do you need to relish permissions on that kind of process? It can be either. System D can run user jobs, but usually the reasons you want it is probably because you need some sort of specific system service that is root. Yeah, that's a good call out as well. Root does not necessarily mean privileges. You can have root without privileges or you could have privileges without root. So we probably want to make sure that we have guidance for both sides. Yeah, but that might just be that you're not compliant with this best practice. You've always got that option. But for things like that, why do you need to be root? Well, they only mentioned, I don't know, there might be a million reasons and you can always find another reason. I'm only mentioning this. We talked about Docker files and if that's the kind of test methodology that looks specifically at entry points. When you're talking about, when you're talking about init systems on occasions, I don't know how system D is implemented here, but many init systems make the assumption that they're the lowest bid and if they're not the lowest bid, then they'll fail at doing what they want to do by design. So it may not be an issue in system D, but there are init systems where your bid matters. Yeah, but that's fine. Your bid would still be zero, one, whatever, rather. Right, my whole point here is just thinking about background services as well as just the entry point. Yeah, I see that. But, you know, a background service doesn't have to run as root is my point. If system D, for whatever reason, requires being root, then you could fix system D to be compliant rather than changing the user story because that would be the tail wagging the dog. Exactly. No, my point is that maybe in the best practice you can add wording about, as well as background services, including system D, something like that, just to make sure that it's covered as well. Yeah, and just to remind people to double check your thinking. And another thing, these are guidelines. So people can escape them if they need to. I was going to say, one of the things we should consider capturing, right, is we should limit one best practice per input here, right? I think that there's a bunch of best practices that could come out of this. If we agree and move forward that running as root when you don't need to is a best practice, then we should cap this one to that. I do think that when we do these live discussions, it's a good chance to talk about what other best practices would potentially fit to a user story or a use case. So that way you kind of build a portfolio against it because we talked about, you know, getting like containers from vendors. I can tell you like one of the challenges I deal with and this in right here is the whole black box. I'm just going to give you something that works versus, you know, I want to rip it apart. What if you hand me a container that is defaulting to root and I come in and say the CNCF says the best practice is not to run in root. If I go in and modify the image or I add additional layers, you'll tell me I'm outside of my service contract and that, you know, I'm breaking your stuff. Not you specifically, I'm just talking about in general like you hand this off. So like we've talked about like the clean handoffs and stuff like that before. And I think this is kind of like one of those good first case examples of I don't want to run stuff in root and give you access to these types of things. I mean, I have a, you know, a big multi-tenant environment. What if someone comes in and starts doing things they shouldn't, et cetera, right? Like what does it look like when, you know, I get something from a third party that defaults to root? Yeah. So there's two parts to that. One is this best practices is audience is the CNF developer more than it is anybody else. You might care. You might say follow the best practices during your CNF development. But the CNF developer is the one that should care about this because ultimately running as root give or take the fact that isolation between users is not quite what it could be in Kubernetes. But ultimately the use of root in a container is more about your CNF, if your CNF breaks, or if there's a security intrusion or whatever, not going completely ballistic, but it's still contained even with root privileges. It's still contained because it's only container root. It doesn't get you out as the container. But we did say at the bottom of this, this is how you can test compliance. If the first user it starts with is not root and there are no sets in files in the file system, then nothing is ever going to be able to get to root for one. And the other one is we said, well, there are monitoring systems. And I think we gave a couple of examples, which double check to make sure there aren't any local root processes running as well. So you have, it's not your job to change it. It's your job to say that I gave you a list of instructions, you know, a set of acceptance tests, and you're not passing them. And so, you know, that is a different thing from, I'm going to break my support contract by, you know, making changes to the software. Instead you're saying you haven't met your acceptance test and the deliverable is not acceptable. I'm not paying you, which gives you the leverage you're looking for. Yeah. I don't think it's quite that clean in the real world because product people like to come up with deadlines and, you know, things need to go to market and all this. I think in an ideal world, that's how it would work. And like you said, this being for developers, like we've talked about this in the past, right? Like people will probably look at these repos and bake some of this stuff into an RFP, right? So like, ideally you would solve this as part of like an onboarding process. But it doesn't work that way. I think your onboarding process should not say if I find root-y sort of signs in your containers, I will reverse them. Your onboarding process should say I will detect these things and flag them on compliance. There is no winner if I supply you a when you're wanting to be, right? There's no way for you to get B out of A. I either have to supply B or you have to modify it. And neither of those things you're saying is in the real world going to be anything. Anyone's going to be happy about absolutely. But the levers are there. Yeah. And in the, I'm just wondering that this negotiation is just going to be tricky, right? Because this is going to come down to the system operator telling the software developer how to develop. And then you're going to have someone who comes and I'm just playing devil's advocate for a second. Because I think this was a good start. This needs a lot more meat in it. Cause like you're going to have the person who's like, listen, Mr. Telco, don't tell me how to write my software. Cause they're going to come up with examples like pal just gave of, you know, I've got this or that running. And I have to have it because of this. And so I'm just curious. That's a slightly different thing. And I think something we need to write up at some point as well, which is for anything like this, where you've got best practices. And again, I'm talking about when I used to work in, well, safety critical systems, you are allowed to basically deviate from the standard as long as you write it down and the receiving side accepts what you've written. So we need to, at some point come up with how you document what you are compliant with in our best practice baselines. And if you're not compliant, you write down what you're not compliant with. And, you know, should you so choose why you are not compliant because you're making an argument that the receiver should accept what you've given them regardless. The fact that you're not compliant because, you know, because presumably it's no worse than it would otherwise be. I mean, we had ridiculous things like pretty much don't use pointers in C. You can imagine how well that goes down. So there were non-compliances. We used to have to write in our software and it would have to be judged on a case by case basis by whoever we were selling that software to. So on this point, though, and the point to my ramble is if you go to the goal section. Taylor. Yeah, there we go. I would, I would make this a little bit more like flushed out and like succinct. I just clearly, I mean, I feel like the second bullet's pretty clear-cut, but like the first one doesn't really read like a goal. So I would just call out like distinctly these are the goals we're trying to achieve with these best practices. And then if you go down to the test plan. And this gets to your exact point in. When we talk about like, when you're saying the goals, like we may, the summary pieces, the motivation goals, there's a lot of overlap between this. You feel like. Yeah, but it all kind of like read weird. Like, I don't know. I put some comments in like, like motivation, like it says both the benefit, like it's two succinct sections, but then like you're kind of like just inferring from the summary and the motivation, which depending on how you attack this, it just reads weird. So I would like make each section stand a little bit more extra. Let's look at, did you actually put the comment in? Mm-hmm. And I'll give you and Ian some more direct feedback. I read this over the weekend. And like I said, I forgot to hit submit. Hence the 16 minutes ago thing, but, um, and then like it, just real quick, I want to stay on like this original point. So clearly, very clearly defining the goals. So that way someone knows what this best practice is striving to achieve. And then in the test plan. And once again, this is one of those things where like when I first made my comment, like, if you actually look into static and runtime analysis, it gets a little like you're kind of doing some inference from like some things, but like I think instead of being like too generic, like for the actual test plan, I would like actually write out like some expectations. And then I put in a comment. Um, we've established this group isn't specifically about writing code. Um, but there is the CNF test week. I think, um, like if we put like test plans together that like validate that a, um, best practice has been achieved or whatever. Hey, this gives somebody to like grade themselves against to Ian's point. So then if they say we're just not going to do this, they can point to exactly where they're making their compromises. And then additionally, if people did want to get more, um, on the code contribution side, they theoretically could automate some of the test plan and then, you know, contribute that back to the CNF test suite. So it gives our group a way to tie into some of the other groups and for the people who want to write some code, but, um, I think if we keep the test plan too generic, then like the best practices like, because you've put somewhere down in the bottom two about like evaluating CNF's this and that. I mean, um, if I just, I'm confused here because that test plan is pretty specific. It tells you exactly what you look for. I mean, I think maybe we just have different definitions of the test plan. Like I get that tells me what I'm looking for. It doesn't give lay out a plan of what I'm testing. Some of that is in the static and runtime analysis. Like I said, there's some inheritance between the, um, different like subsections, but. Well, I would write like, that's what I'm going to hear. What would you write? I would write like, you know, expected inputs and outputs. And this is what we're expecting, right? Like, and some of that, like see user and run commands. Um, some of that, like I said, is in the static analysis, but like, I mean a test plan to me infers that there is a plan that you're executing against. And then that would have predefined inputs and outputs to validate whether or not. I agree. The test plan is effectively. I think maybe the name of this section is, um, not good by saying test plan. If, if we take like the definitions of that, people are going to use that are actually writing tests. It probably has a more specific thing, which I think is what you're saying. Um, Jeffrey. So maybe the whole section is, um, just named wrong versus, we're going much higher level, like a business level expectation. I agree with that. I'll add that. Um, I read this originally as this is. This is not a best practice. This is a principle, right? We intend to derive best practices from this. So talking about test plans here, a little bit jumped the gun for me because a best practice in my tone is different. It's, if you want to do this, then you should probably do this. And here where we're taking this principle and saying, well, this is the rule right now. And there's a compliance week. This is maybe as far here. I'm a little confused by what you mean by that. I mean, the best practice is literally don't do this thing. And the test plan says, I will look to make sure you are not doing this thing. Um, and the test plan is written in such a way that he's testable by, you know, code. Um, I mean, I wouldn't care if it was testable. I wouldn't be happy, but I would at least be acceptable if it was had to be tested by a human being that is testable by code in this instance. Um, the test plan talks in generic terms because it isn't a test. It's a plan. Um, you know, I'm not writing code to do the test and embedding it in the document. I'm telling you how the code goes about what it does and what it's looking for. So when you say it's principle, not a best practice. The principle is least privilege, which isn't written in here. The best practices do not run anything as route. Um, which is a yes, no answer. Are you running things as route? Yes. No. Can I test whether you're running things as you route? Yes. No. If these tests pass, you are not running things as route. There is no way you can get to route. So I think I'm arguing with both of you, but I, I mean, starting with towel, why is this a principle and not a best practice? I, well, if it is a best practice, I'm a little worried then too with the tone. Um, I didn't read it that way. Why? Because sometimes you do need to use root and then the question is, okay, what guidance can we give to people? For example, uh, pods. I'll give you one example, right? A pod does not need to have one kind of container. It could have multiple containers, but the way Kubernetes works, privileges are granted per pod and not per container. So one best practice could be if you do have a container that needs root, you want to isolate it maybe into a pod rather than mix it with, uh, uh, with other containers that do not need root, right? So to, to reduce the attack surface, if we will, if you want to talk security. Um, if you need to use system D, then use system D with a user rather than root, right? There are all these things that you can. Why do I need root in a container? I don't know. There, there is a zillion reasons I can think about in terms of, uh, specifically network functions, interactions with a host, the operating system, at some point you'll probably need to do that somewhere. And the idea is to isolate it. So, so the principle is least, right? It doesn't mean zero, uh, somewhere there, there will need to be some sort of privilege. So then the questions are, do you want it to be even a workload on Kubernetes? Should it be something like a CNI plugin that runs on the host rather than a workload, a containerized workload? There's a lot of questions here that I think are very interesting and that we could provide guidance. Uh, the idea of jumping to compliance about containers seems to me going to some very specific use case here. To me, it looks like a step too far. We haven't even established compliance and what that could mean and why a compliant application has to have all containers avoid the use of root 100%. Um, Ian. Hey, Ian, maybe this word right here. We don't have enough context. I think what you're remaining, when we put this in, inside a compliant application, I think that's compliant to this best practice. Correct. Compliant to all things. Yeah. So the suggestion then, and maybe we could write it to be more explicit like that. So then we're saying, if this is a best practice, then, um, it just because you don't follow this best practice doesn't mean that the entire CNF or application is not cloud native or anything else. We're saying try to avoid using root, not when you can't. So if you can have a best practice that you decide is, doesn't work for you. And that's fine. Just saying in general, in general we're saying try to avoid. And then help people. Well, how do you avoid to use root? I think that's what, I look for guidance here. I'm not looking for somebody to tell me what not to do. I'm trying, well, I, this is how I work, you know, I need root at some places. Well, if not, what can you suggest? I disagree a little bit. Well, I want to disagree on that point. Like I think we're supposed to be providing guidance. I think Taylor hit the nail on the head though. And I don't necessarily think this needs to stop like a PR from being accepted, but like, I think that there is not enough context. Taylor used the right word is like, you need to like, like I said, I think the goals need to be very, very clearly and succinctly stated independently. Like you get this, this, and this, this is what we're, you know, we're seeking out to achieve with this best practice. I mean, I don't know, like, I don't, I slightly disagree with you, Tal. And like, I do think we should be somewhat prescriptive because you can just choose not to follow this best practice and be like, these are my reasons why I just don't think there's enough context in this for someone to cleanly articulate their reasons. Why? Like, you know, I intended for a second and see what you read in because it might not have come across in the document. The thing that container roots lets you do literally the only thing it lets you do because most of the things that root can do really aren't allowed in containers. Is it lets you change? It lets you read, write and change permissions, any file in the file system, which is probably not what you want. You can't stop anything that's got root from doing that. So every file system protection you put in place is thrown out the window. And so if you wish your file system protections to be in anything, then you should not run anything as root. But what if, what if you do want to change a file system installed file for, for whatever reason? You just don't use this best practice though. You use it, you give ownership, you give ownership of it to a different user that can change the file. So I think, I think part of the way to look at it is I think there is a context issue here. What I would recommend doing is separating this up into first from a test perspective. We need to be able to say that this thing is an audit mode versus this thing is in force mode. And get, and that gives the, the various groups the capability to pick and choose the things, the best practices or future compliance things that they exist, the ability to set their, their policies and baseline so they can then discover and optionally enforce things that they care about. The other thing as well is you look at it from a vendor and consumer perspective, the vendor will say, I have verified or ransom verification program that, that has given me information about whether this thing meets certain requirements. And the test bed can help there. What it doesn't do is it doesn't say whether this thing is fit for purpose by the consumer, whether it's valid or whether it's a validated system. That's something that the consumer in this case, Jeffrey and his team would have to work out is this thing fit for purpose. And this is where the, we want to have some of the best guidance or best practices to say like, is it fit for purpose? Is it fit for purpose from a security perspective? Just use that as an example. In which case you could say we want, we want no root. We want to have privilege escalation disabled. There's actually a privilege escalation data in Kubernetes that doesn't allow you to, that turns off the suit or ignores the suits, the suit bit. We want to drop all capabilities except for the ones that are absolutely minimum, that you've compartmentalized. And some of it is through the mixture of the verification and validation combined. Ideally you should get to somewhere that is, that is, consumable, but at the same time you provide that information out. So it ends up becoming a conversation between the two, and not an enforcement, which I don't think we'll have the ability to do from here. I want to point out one thing too. I don't think that it's a zero sum game. Like I think both Ian and Cal can have what they want. I think Ian, if you do a better job of describing in this document, what you just described in words, then we say, okay, this is what we're trying to achieve with this best practice. Then if Cal has an exception, we shouldn't write the best practices with the mindset of all these exceptions, but we should just tell you to be a say like, look, this is what this best practice speaking out, but I have to do this. Therefore I'm going to run this container as root. And then you say, okay, accept that risk because Cal needs to accomplish X, you know, but like we shouldn't water down the best practice in my opinion, Cal, because we know that the exceptions are coming. We should just make sure that the best practices clearly state what they're trying to do. So that way when you do want to make the exception, it's easier for you to articulate that. Yeah. I just agree with that. I think my point is, uh, is that there's upset too that there's, there's room for more context here and more guidance, I would say, and more, uh, I think, you know, Jeffery said it reads a little bit weird. I think the tone is just, uh, the word compliance, I find a little bit scary, right? I could easily see a telco taking this and saying, okay, you need to do this. There's a test suite. And it does, it does. We can read. We can read. We, we don't have to do a compliance as far as these words. It's not, it's not the intended thing for what Ian was saying. So I would, it is literally, it is literally compliant to this best practice. And the word compliance is a perfectly acceptable one to use there, but I think you're saying that it might come with connotations. We don't want to go with towel. You said several times without listing them, there are reasons why you might want to use root. And I think what we're saying here is. If there are reasons why you might want to root, if we can list those out, or at least as many of them as we can think of out, then we can give people reasons either, you know, you can explain to both parties, well, this is a reason why you would want to deviation. So you should write your deviation up and explain these words. Dear application consumer, a deviation of this nature is normal and expected. You might want to just give them the green light on that. Or alternatively, have you thought of this alternative way of doing it that doesn't require root? That would also be another thing you could do. There are. Your example of system D to take an example is basically saying, there is software in the world that people want to use, which shouldn't need root, but insists upon it anyway, perhaps. So we've got to. Non-compliances are perfectly fine. They are an expected part of this process, frankly, nothing is going to be 100% compliant at least to begin with. So. We just need to help people through the process of using what we write, not necessarily changing this because a best practice. That says, well, you might not care to follow the best practice. Is basically not a best practice, whereas a best practice that says this is the. Platonic ideal of a container in an application is, you know, fine, you know, it doesn't always come up to. It doesn't always work in the face of reality, but it is nevertheless the ideal that we should all strive towards. The framework that is often used in this, in this particular path, when you start looking at overall governance is you end up with policy and standards at the top policies or things like all data at rest must be encrypted. Standard to be like, we use AES and then at the lower end, you end up with procedures and guidance. And then the other thing is that procedures are, we use BitLocker to encrypt window systems with AES in this way versus guidelines, which is like, don't leave your laptop in the car. The difference between the two is that procedures are always mandatory. Guidelines are optional, but very tend to be very good advice. In this scenario, we're focusing primarily on the guidelines, but it is possible that in the operational side, we're focusing primarily on these things into procedures. And so we want to make sure that they're, that they are testable when they can be, which gives the various groups, both on the vendor and consumer side, the ability to audit and enforce when they want to, but ultimately those knobs become tunable. And we focus only on guidance. We don't tell people when should you move up to procedures because that's something that the internal organizations need to work out based upon their policies and standards. Everything you said is perfectly reasonable. I'll just add, look, the user story here is TBD, right? We didn't really start with the user story. So we don't, we kind of jumped a little bit ahead here. I think that the document as it stands right now just feels a little lopsided to me. It kind of jumped to the end of, okay, no root. And there's a big story along the way starting from the user stories, stories, right? And going towards, well, you know, the user thinks they need to root, they probably don't. And if they think they do, they don't realize what the problems could be and requiring root. So we kind of jumped right to the end rather than building something along the way. It might be obvious to all of us reading this, but I'm not sure it'll be a hundred percent obvious to people who said, well, why not? This was never intended to be released without, this was never intended to be released without the user story to back it up. It's marked as a work in progress. I couldn't, for some reason on this repo, I couldn't make this literally a draft pull request or I would have done that. So there's definitely no suggestion to accept it. We're wanting feedback and the user story would go in there too. I do, Ian, before you respond on anything, I would do want to point out that this section here in the trade-offs caveats notes is specifically for adding things like here are why someone would be doing something else and an exception. And then you can talk about how to deal with that. I mean, that's what this section is. Yeah. Right here is very similar to what you're saying with system day and other stuff. So you have a container image that already uses root and already has set ID binaries. So that's just an example. We could, you could do a comment with the suggest edit to add some other examples if you want to a section. Go ahead. I would suggest that if we're going to come up with advice on how to consume this, like for instance, those things where again, Tal's saying, you know, I've got a laundry list of reasons why I might want to use root, that we don't, we try not to build it into the best practice. We try and keep the best practice short and sweet. We come up with a guidance note that goes along with the best practice to kind of say, well, you know, these are things where you will probably find deviations. I agree with that. I think this is one example, but we don't want to listen. And we didn't, you didn't go into like extensive here's how to, how to handle that. I don't think we should. I agree that we shouldn't have all the examples of the, all the exceptions listed and how to manage them. As well as how to manage. We want to change that. That's awesome with good, right? Sorry. Go ahead, Jeffrey. And I thought, like, that was really weird. You, your thing cut out and I thought you would stop talking. I didn't mean to stomp all over you there. That was weird stuff with my headset, I think. I was just going to say, we don't want to write things to exceptions. We want to write them to the 90%, right? And I think one other thing that would help when we talk about this context and things that we should add in is like, as you're reading these, you know that there's going to be complimentary best practices that fit underneath one of these principles, right? So if least privileges are principle, no root is one best practice. No root is another best practice, you know, and we should like capture this and, you know, key tabs on it is, you know, create users with the appropriate amount of permissions, right? This gets into Ian's point of, you know, are there going to be times when we might need to use root? Sure. But the best practice is don't run root. Another best practice is to create the correct user with the appropriate permissions to execute the task as needed. Right. So I think when we get into some of these, like, you know, what up discussions to, um, if we just track what other best practices would compliment this one and how you kind of build like the suite of best practices underneath the least privileges umbrella, some of that context is also handled. And then to your point, Taylor, we don't need to list the 5,000 exceptions of when we're going to violate this. Well, you know that I'm, I'm really interested in those guidance is more than this document. I have to say, you know, there are a lot of questions all the time. For example, people wanting to run SSH, SSHD, right? In a container for various reasons. If it's accepting net cons, if it's something else. Now, how do you do that with it without access to root? How do you update certificates? How do you do other things? Where do you put it in the build process? Um, nobody wants to run root. I think, I think this, any decent programmer reads this and this is very obvious. Right. Of course we don't want to run root. But what do you want us to do instead? What do you suggest? What is the ecosystem? What is an alternative to SSHD that can run in a more cloud native environment without these requirements? Um, that to me is the more interesting part than this, but, but, but, but fine, you know, we have to start somewhere. I'm, I'm, this is, I have no problem with it with the principle itself. Um, let's go, that's Tal. I, I, sorry, Victor. Um, I. Love the idea of recommendations on how to implement something. So whether that's dealing with exceptions or people that are saying, Oh, I, I lack the best practice. Now, how do I do it in with the software I have, because it's not set up for that. So helping people to actually implement sounds great. And I think that could be a new section that we could have within the CNF working group where we talk about that. Here's the best practice. And now we start maybe, maybe this is a folder for instance. So this would be one way to look at it. Um, just bring this up. So we don't have folders yet, but under use cases, some of these, we have folders. So if we had a best practice for not using a root within that folder, maybe we have, here's a bunch of other documents that talk about what about images that use system D. What do you do for that? And then we could talk all about that and keep it separate from the main best practice document that allow us to add as many as we want for real situations. That makes sense to me. Yeah. And I'll just remind myself and others here that we specifically picked this one because it's a low hanging fruit. That's very obvious. So yeah, in a sense, this is a very trivial best practice, right? But even here we see that there's, uh, there's some subtlety in terms of, um, um, actually, no, this is non subtle. I think that's what's interesting about it. This is not a non subtle best practice, right? Don't use root. It's quite blunt. It's deliberately quite blunt. Um, and, and again, you're, you're shopping list of reasons by actually, well, I might want to use root anyway. The ones that I would categorize as pragmatic are ones where I'm using something like system D, which thinks it's going to be root because it's been written in a circumstance where it will always be root and it's going to be difficult to run otherwise. And I don't spend intent to spend all the time changing it in a circumstance that it isn't going to be root in all circumstances. Yeah. Are there use cases that fall on the other side of the spectrum where root is absolutely essential to what I'm doing and nothing else will do because those would be interesting ones. The thing I like about the guidance here specifically is that I can have this thing as a test and I'm not saying you don't do it, but in an audit mode, if when I, when I'm in the process of doing an audit, sorry, two different audits, then I can say, well, here are the things that use root that we need to spend extra time focusing on because they do dangerous things. And it allows me to have teams focus on those explicit things because I know that they're high risk items. So I think part of it is being very blunt here is good. Having something that's testable is good. And again, how you, how you adjust the knobs matters a lot. And I think part of what we need to do is make sure that we're very clear that we're developing guidance and that the guidance can be consumed in these other ways other than just enforced mode on. I might add, you know, I said this is a very blunt best practice. There is a tiny little subtlety that I want to point out. And that is that there's been an evolution from the early days of Docker to truly what's cloud native and Kubernetes, right? There's something very cloud native about this because early on Docker was exactly about the point of isolation. Sure, use root. Do whatever you want inside the container because it's completely isolated. It's not the real root. It's not the host root. All you're doing is changing stuff inside the container. We're thinking about this differently. We've evolved our thinking over time, right? Because we're thinking about these different pods, different containers working together to do something. And the kind of reliability that we want there is not, you know, do whatever you want inside the container. It's remember that you're playing as part of a bigger system here. So you're thinking about attack surfaces as being not isolated in one place, but you're saying, well, this is one dot inside a larger topology that I need to think of. So if I have everybody using root, I'm really, it's very hard to tame these caps. So there's something a little bit surprising here for some people who might be new to Kubernetes and, you know, not 100% obvious to every developer, I guess. That's an excellent point. And we ran into that same problem when Docker was new. So 2013, a large part of the messaging that we did was look at 12 factor acts. We know that they're not perfect, that they don't align to everything. But if you understand the guidance there, then it'll help you achieve what you want within Docker. And I think that there's similar that this, this can help in that same path where we're going to get people who have never used Kubernetes. They don't know what a pot is. And we're asking them to go build CNF. So we're going to run to the same problem. Why do I not need root? Why do I not need privileges? Why do I need immutable infrastructure? And providing that, that guidance as to why it's important and how it all fits together is crucial towards helping them be successful. Yeah. And it's, it's actually a huge developer problem. A lot of people work with Docker. And then they think the next step is then putting it on the cloud. But the whole idea of cloud native, right? We're, what we're trying to say here is you have to think cloud natively from the very beginning, from your architecture. But you don't write, write a single line of code, right? Of a network function before you understand that this is going to be a cloud native network function. Well, the other thing that I would like to discuss is, well, when I was reading these particular PR, I noticed these, that Docker has the username space remap feature that you have to enable in a demon to, especially if it takes to, to Docker to, to modify the, the, the user, use when, when a new container is user, uses. But I noticed that this feature is not supported by, by Kubernetes or at least I tried to, to, to run a CubeSpray cluster using that feature. And that didn't work because API services requires that, that specific mapping. So I don't know if we should also, I don't know if, if that, once that this feature, this username space remap feature is supported in Kubernetes, it's going to make sense to still enforcing users to not use no root containers or it's going to be an extra addition, I think to consider or something like that. That's a very good point. That's another one of those, you know, when I said, there are so many reasons why you might want to use root, you know, I don't know even where to start. Here's one potentially, right? So again, there are other things you can do instead. You know, there's, there's never only one way to do something. I see the part that you keep coming to this point and the part that I'm a little hung up with is even if there are instances where we would want to use root, does that change the fact that it's still a best practice to avoid using root? No, no, no, I'm not disagreeing at all with this best practice. I'm, I'm, I'm trying to make this document helpful. So I think, but what we can add, we can put it as, you know, supplements to it, you know, it's, it's the point of this particular document is just to state the principle. I'm happy with that. I think it kind of bleeds into other thing when it talks about test plans, I think that's beyond just stating the principle. That's how to enforce it. That's a little bit different. But I think I made my point. So whether people agree with me or not, I have nothing more to add. I would say edit for the test plan to remove compliant. The, the ideas, do we, do we have to do that? All best practices that anyone puts forward, we would like to be able to somehow test it. It doesn't mean that anything about enforcement. Frederick keeps saying, I've said that, are you a difference between enforcement and just notifying whether expectation of the test is one way or another. That's all we're putting forward. When you have a best practice, whether you're using root or not, whether you're using root or not, yes, it's possible to use root or not use root. Can you test that whether you're using root or not? Those are the, the, the main things that we want for any best practice. Bit forward. Sure. Yeah. I might have a history here with, with the words, and I'm sure others do too. And CNTT and other organizations. The word compliance is a very, very big word. It's based around compliance and software, et cetera. It's, it's a difficult word. It's also an over-noted word because compliance. How about this right here? Yeah. An application which follows this best practice will not have any containers with processes running as root. Remove compliance completely. That's fine. I mean, it reads a little bit tautological, but again, this is a very blunt best practice. It's very clear what it says. It's there in the title to an extent. So, so yeah. Yeah. So in that sense, have you are considering side cards as well? Oh, absolutely. I mean, side car is kind of, it's just another pod in the container. They're also the issue of init containers. There are various ways you can put containers inside pod, a pod. The thing is that the pod concept itself is very interesting, right? It's a, it's the pod creates a context that is shared by all the containers that are running in there, but the containers are still containers. They're still isolated. This is a thing. By the way. A lot of people, right? Side cars end up with the same security as the, as the main container. So there's no difference. You give root to your side car. Your main application has root. Exactly. I'll give a very popular example, which, you know, when people hear a side card, they always think of Istio, right? That's the, the one operator and product that uses probably side cars more than anything. The side car is changing the IP tables for the pod. The IP tables are used by all containers in the pod. So the side car is able here to use privileges, right? To actually redirect all the network traffic coming inside and outside of the pod. It's a very powerful thing to do. And that is all based on privileges, right? You're giving your side car the privileges to do that because you trust, you know what it's going to do. And the features is one that you want. So, yeah, my point is that there's, there's a much bigger story here than, than just this best practice as it is. But it's still, you know, I agree with the best practice, right? You do want to not use root when you don't have to. And if you think you have to, you probably don't. So there's a narrative. It's, but, but I'm okay with making it, you know, short and sweet is the word that people seem to like here. Let's make it succinct. And then add supplemental documents that, that, that dive deeper. Yeah. If you want to hear something really scary. All pods by default get the network capability, which means that they can read and write things directly onto the, onto the network. And they need this for pain because I CMT respond, responding to ICP requires the network capability. So yeah, in short, this is, this is why layering and other things like sec comp or other policy engines like Falco and similar become very, become very important in, in that aspect. So there's other topics we can jump into there as well, but we want to keep this, we want to keep this. Right. Just a quick point. Sidecars actually, we're at the top of the hour. So let's keep adding or hold it, either add right into the notes or the ticket or bring it next week to respect everyone's time. Cal, please add some stuff into the poll request. And check out anything that's there as well. Add some comments. See you next week.