 Good morning. It is Friday. I'm quite happy. Well morning for you. Wait morning for me afternoon for you guys probably beer soon for most people But for the Americans So I am going to dig into a Question that I get a lot and like this is kind of the style of talks that I do where I Where I like dig super deep into something so people can like understand it So that then they can use their own architecture brain to like build things the way they want and actually get the results they want so like Like most people that were sys admins for a long time Like I have to understand how something works so that I can build it the right way and and I always get these questions of like like How does root work in the container? Why does user name spaces work like how does how does? Users work and I basically came with a system I was actually chatting with Vincent bats at devconf us like maybe 18 months ago And we brainstormed this and I imagined in my brain a chart that would explain basically this Two people like I was like oh this will be the easiest way to describe it to people like I could show them Root outside the container root inside the container, you know root outside user inside user outside and user inside And then just to cut to the chase We really want the bottom right box like in a perfect world We want defense and depth and as many layers of security as we can So we want to use a user in the container and a user outside to run the actual container engine But I'm gonna demo this and dig a little bit deeper But then I realized this actually isn't enough because then people ask you well, what about dash dash privilege? So I jokingly have I have this talk got longer and longer and longer But actually with this I am going to jump right into the first demo So I want you guys to kind of dig into this deeply and understand So I figured it was easier to script so I would not screw this up and forget these but first I'm gonna show you root Outside of the container with a who am I so like the whom I command if you're not familiar that it just shows you What's your user you're logged in as you'll see when I run the who am I it prompted me for see-do and And now it now I'm actually executing these commands as root But um in a nutshell you'll see here. I'm root outside the container What this means is I'm running pod man as root So I'm running the pod man command as root and then when I run pod man I run this who am I command inside the container and you'll see it's also root So like it believes it's root even inside the container Um, and I have a few drawings that I can show you how that works Um, then I'll show you later But now let's do a second one where we do like root outside the container So again, we're executing pod man as root But then this time we're going to execute the who am I command inside the container as a sync user You'll see this dash you sync. That's like a really critical thing. This is something even docker had forever Um, you could just execute. It's kind of like the the user Directive in a docker file where you tell it to execute the following commands after that as a certain user And what this does this is drops privileges, right? Like it drops down to a regular user inside of the container instead of being root And you'll see if we execute this it shows it as the sync user You may wonder why did I choose sync? I just choose to sync I chose sync randomly because the sync user is one that's available inside of all container images Like usually if you look at the password file, there's a sync user. So it's just an easy demo I could have done it with a father linux, you know added a user or something like that But I didn't want to create any kind of special container mix So now let's look at a user. So I'm running as father linux My user account that's logged into the system and then I'm going to execute the who am I as As a regular as without specifying what user actually I think I lied on this one. Yeah, that's that's a lie. That's not an itu Um, I actually copied and pasted that and so screwed that up But you'll see I'm actually not executing it with this itu This is gone imagine that but it is executing as root inside if you don't specify this and that's actually what's happening because that's That's what's happening here. But um, and then notice I will execute it with the itu The last time I'm I'm father linux outside the container and then I'm Sync inside and you'll see now again I'm at the who am I is right So now this this last option is what we want to do as much as possible and to be honest with you this is This is not easily possible in open shift, for example, because cryo runs as root But then you want to run the user inside as like sync um And even the scc is the security context controls In obit shift force you to run as a user inside by default Those are like the restrictions and people get really tripped up by that because they're like Why am I being forced to run this as a user because so many container images don't specify what user run? and so then they just run as root by default and so Um, this is like a holdover from unix world where like, you know A patch you would start as root and then drop privs and then run as a patchy like, you know The the sub processes will run as a patchy um It what people don't realize the reason why we did this is because port 80 was a controlled port right and only root could connect to it But in a container it doesn't matter like you can put you can connect a port 8080 And then there's always a load balancer in front So it's actually not that difficult to do you know to basically do this second version Where you know where you run the container engine as root, which would be cryo And then and then run, you know The stuff inside is like say a patchy and then a patchy could fire up the you know on port 80 And it doesn't have to be root when it start you're on port 8080 I'm sorry instead of port 80 and then it doesn't have to actually be root but but a lot of people don't do this so Um, I wanted to at least dig in that's the root of the entire talk But now i'm going to go deeper and deeper into like each of these pieces So everything in unix is a file right but to understand root You have to really understand like what a user is And I think people forget that there's what's you know Everything in unix is a file But that's because we're exposing contents of memory as files to make it easy to poke around with Like these data structures that are essentially in the kernel memory And so one of the data structures is like the process id table I'm actually that's probably one of the main data structures that matters for for unix and linux So so in this we're going to look at like we're going to I'm going to use some new nomenclature Like very technical nomenclature to get your brains to work in the right way to think about this So, you know, we all know the ps command that's pretty common Like it just shows you the processes that are running That's like the business way of saying it. Well, it's the processes are running But like the technical way of saying it is like it's dumping the contents of the process id table And then if you think of like top top is really like continually monitoring the process id table And then if you think of like digging around in slash proc That's like in manual inspecting the process id table, but you're using a file to do it, right? Like you're looking at a file that actually maps to memory and then you're gonna, you know, you're gonna take a look and This may sound super easy and simple, but I'll bet I still show you something that you didn't know about before so Let's do psdf. Everybody knows how this works. This is pretty common. This is like a regular work Again, we're dumping the contents of process id table my giant amount of google chrome tabs have like you annoyingly in numbers of Of processes running And then let's do top to like continually monitor it, right? Like every two seconds or whatever this updates or five seconds And you see what it's doing is it's basically just constantly Dumping the process id table and then showing me in a different type of screen And then if you get out of there and you just look in ls proc This is actually a file system representation, you know representation of it Um, but then let's do something that maybe you haven't done in a long time or maybe haven't done ever Let's let's actually like inspect for example in this one. Let's look at uh system d. It's always process id one And then let's dump its status Um in this status file is a bunch of stuff. This file is basically made to be read by humans Um, and this is a way to see all that information that's shown in ps and top But in a way that's a lot more like human readable and it feels more like a data structure, right? You're like, oh, there's the u id. Oh, and then you go, okay u id and g id I know what those are but like why are there four of them and you're like, oh I don't remember why there are four of them. Does anyone remember why there are four of them? I'll I'll I'll cheat because I already know these but like this is the real id This is the effective id. This is the saved id and I'll bet like 90 of people don't remember what this one is And then file system set id which be honest to you I did not even remember what the heck this one was and the good news is it's not used anymore This is like a holdover from old school unix stuff. Um, apparently the way the linux kernel Works nowadays it doesn't actually need this one. And so really you only need these three But but the fourth one's there just for system calls for software to work. That's old basically Um, this is going to become important as we run these processes in containers I'm going to show you some hairy things that you may not have seen in a long time or may not even remember at all Okay, so let's talk through what the real id the effective id The saved id and then the file system id are so the real id never changes. Um, that's who you actually are So like that's father linux like when i'm logged in as user 1001, which happens to be my uid on on my hat on my laptop Um, that's that never changes like a user does not have permission to change that you can never change your real id Um, but the effective id can be changed in a couple different ways root can actually change it itself So like it can drop prives it can drop to a different effective id So like it could start as root and then drop to apache Um effective id can also be set by like the set uid bit in a program So like people forget like things like new uid map and password and things like that actually are are set uid root So when we say set uid root what happens is the the the effective id when the program started is root But the real id remains father linux So a good example of this is password like the password file or the password program is pretty cool And and that that was actually one of the best ones. Um, that actually was like a two hour Rat hole for me to get my demo to work with password, which i won't bore you with but Running password in the background and not letting it fail and capturing all the output of it Is actually much harder in shell script than you would think to to script a demo like this but um the saved user id now gets into um say you are Like apache say say that apache process, you know Needs to drop prives and then run the actual web servers as a lower You know the the threads that will handle the the responses to like port 80, but then say it needs to like uh fire up a port 81 it can actually like in unix you can actually There's essentially this ability to like change your effective id But you can only set it to your saved id or your real id So um essentially if if uh, you know if you were once root You have to put it in saved id because you can't go back It'll like won't let you go back unless you need to save it in saved id long story short It's complex, but this we don't see very often This is not something you necessarily need to worry about for containers But it's important to know that this stuff's still happening inside of a container even with username spaces and i'll get We're going to go deeper into that So let me demo the effective user id just so you understand this pretty crisply Um, let me get out of here. Uh, let me go into this one. So who am I i'm father linux. I'm run password um Ooh, that's not supposed to happen I don't know what happened. Let's see if this still works Okay, looks like it might have worked let's and then let's oh didn't work son of a gun Okay, we have to do this one live Uh So let's run actually we will do this one live we we can recover from this I don't let one demo failure hurt me that badly So let me run password as father linux So i'm gonna i'm gonna let it hang in this terminal because this is actually the easiest way to do it And then i'm going to do p id of password so oops p id of Password will get me the user id or the the process id's let's let's just use this one And now let's inspect this cat slash proc slash this process id and then let's look at the status of this process id So again, I showed you what a status looks like Oops, that's the wrong one. We do not want to show that one. Let's use the other one. I don't want to get you too far ahead um Let me grab this guy which I think is actually the right one so This is the right one. This is the one you want to see so notice the u id here It the real id is 1001 which is father linux, but the effective user id and saved state one are root So so basically what's happening is this password program has the permissions now to go at you know modify etsy password And then the password program itself is responsible for not letting My u id 1001 it has to go look at my real id and go Oh, i'm only going to let him change this one line in etsy password here. Let me cat etsy password You know the password program basically is smart enough to only let me modify Modify this one was my u id at All right, let's do this Grab 1001 So it's smart enough to only let me modify this line that is mine, right? And I think a lot of people don't realize the same thing with etsy shadow. So Essentially we're deferring the security to this password program to like not break and if somebody programmed password wrong And there's a way to hack through password then then i'll be able to do nasty things as root with the password program Which is why people don't really like set u id programs This all gets even more fun when you start running these things in containers. So, uh I will stop there for a second and see if anybody has any questions Is everything super clear up to this point? I don't hear jen breaking in So i'm going to now you're good. We were maybe uh suggesting that you did not do the appropriate homage payment to the demo gods That's about it Yeah, exactly. No, what I did. Well, I know exactly what I did I didn't run the demo to the end where it has this cleanup and so it didn't get rid of the other password So I noticed that there's a container running with the word with the password program in it And that's why it it borked up, but oh well Um, so Thanks god. Okay continue. Okay no problem, so So, okay, so we go I I was thinking about putting a big like x through this screen and saying this is not actually what you need to care about um You need to actually think through Uh, this is not actually what happens in a container, but it kind of does it's kind of a lie that it doesn't But it's not the main way that a container is constrained is what I should say So this is actually the way a container works. So you'll notice without username spaces like those first containers. I ran Um, you know when you have uid zero in the container It's uid zero outside if you're uid outside and you run uid zero inside the container It's still uid zero. So if you break out of the se linux context or the cgroup or um, you know The sec comp rules or if you have those disabled because you ran Which will dig into a little bit deeper That is definitely going to this process that you run inside of this container is definitely going to Break out and be able to do nasty stuff as root. Um, and this is a danger with something like docker because The actual demon always runs as root like in this scenario I'm showing you that podman will run as root and then this will happen But then if you run it if you run, you know, if I if I did that dash itu that I specified father linux It will at least run You know the uid inside the container. So it protects us a little bit, right? But obviously if you give somebody access to podman as root they have root on the box already So like it is what it is. It's not it's very well understood with podman The problem with docker is you don't necessarily realize you're giving everyone that has access to the docker socket root Because the demon is always running as root. Um, and so it's a little bit more clear The threat model, um, you know the model in your brain and the reality are a lot more aligned with podman It's very easy to understand when you're giving somebody root Now with username spaces and running as a regular user, um, which I I show here You know the the podman program itself is running as you know, uid zero zero, you know one zero zero one One thousand one and then if I run a container as root it will run as one thousand one And then if I run it as some other id like uid zero, you know one zero zero one inside the container It will actually map to some crazy high number. Um, this crazy high number We call this etsy sub uid and sub gid. We did the same thing with group IDs. Um, and this is a an ability within the The shadow utils the kernel everything else basically all the utils within linux to basically Start a process and then and then basically have these New processes run in this sub uid range as opposed to as the actual user This is user namespaces in a nutshell like this is we we essentially allocate what people don't understand is like Namespaces are just hiding reality and it makes it look different But in reality, it's still mapping back to a real uid on the host So you still need a real uid and so outside on the host. It's going to be this high port inside It's going to look like this this um, you know this low number The the security controls have nothing to really do with user namespaces The user namespace are basically more like hiding it from you so that so that you know It looks effectively like you know one thousand zero one for the processes that you run in this container Um, but it's not really totally isolated You know, it's it's still running as a user id and hopefully these high port user id or high number user id Don't have very much access on the system, right? Like that in and of itself limits a little bit What they can do, you know, what kind of nasty things they can do, but let's let's demo that as well. Let's demo user namespaces These should not break. I believe Let us see All right, so we're starting a container as user outside Then we're going to run the password program inside So we're running password now in a container as opposed to like on the system And then we're going to use this really cool program call or the sub command called top in uh In podman and podman top dash l basically shows you the last Container that was ran and it's going to show me The user the the host user the p id The command um, you can do hp id you can show you can show like the p id in the container outside the container You can do all kinds of cool things with these args But uh, I just wanted to show basically notice if I just run a container without specifying the user as a regular user It will be root inside, but the host user will always be my id So i'm basically running these this bash process and this Password process and they're on the system you can like ps and grep for these see them But they just run they look like root inside the container, but they're really you know regular user outside um Now let's like go look inside the container at the status of this password You'll see that inside the container if we cat that file This thing really thinks it's root inside the container right like the effective the real user id the effective user id Everything is root which makes sense right because it's root inside the container and it's in a namespace So it feels like it's a separate linux kernel. It's like a virtualized Set, you know, this is a virtualized data structure in the kernel basically you can think of and you're like Oh, okay, this makes sense now. Um This is really password running is root, but now let's do something trickier Let's run the same bash what let's run a container as the sync user Then let's run the password command as a sync user And then let's look at it and this one's a little bit more interesting. So you'll notice the the Bash that we started was started as the sync user outs You know in the container and as this high port, you know high or upside not high port, but a high UID number outside this this maps to my etsy sub uid And then notice this is root in the container like wait what? Um, and then this is uid, you know zero, you know, one zero zero one outside Well, remember my my I I don't want to cut ahead because I love this demo But let's let's now look at the status of this thing so we can see what's going on here Well, the effective uid you'll see is zero So it thinks it's able to operate on the password file inside of the container You know image that's bind mounted or that's mounted Um, and then you but it's actually the real uid is sync. It would sync is always five And so this looks a lot more like what we saw right, but but it's running as the user and then it's it's hiding it But in reality, it's not actually five. It's actually like one thousand six, you know, whatever that was sixteen thousand five hundred something Whatever the high high high number was So when you use user namespaces, you can do multiple uids in the container You can basically enact all of that normal infrastructure that would happen all the effective uid and real uid And all that jazz still happens in the container when you're doing things like, you know Password changing a password or new new gid you can actually do a nested, you know user namespaces and things like that So you can use like new uid map and things like that inside um, all right, let's go back now, uh I should have been on this one, but there's more so the next question that always comes up is well What about dash dash privileged like doesn't that give you root and you're like, well, not exactly I would think of it more as um What gives you root is is the user id that you specify what I just showed you there's root inside and outside the container That's what gives you root, but se linux our friend se linux that people sometimes, you know Sometimes this revolt is happening against se linux that i'm showing in this picture But the the point of this picture is actually to show that like I joked that like se linux is who you can talk to Set comp is what you can say so if you're trying to stop a revolt and you're trying to control people and you have rules in place that You know basically limit who they can talk to and what they can say It's going to be really hard to organize a revolt against your castle. So um, the same is true with like a kernel, right? Like if you can limit what um, what Essentially what files I can open what processes I can communicate with what ports I can communicate with and then I can also limit What they say which system calls can be executed you have a it's a lot harder for a hacker to get into your system These are on by default Whether you know whether you technically se linux is on in the red hat world We don't always have set comp on but these if you're using set comp these are on all the time whether you're running root or rootless um, and so The dash dash privilege is really disables these it disables the c groups it disables se linux that disables set comp This is true whether it is running as root or not Um, and now you may say okay, that's weird. How can a user disable se linux? Well We can disable We can disable the svert like where we automatically generate a uid or a or a label A context label for the container and then that will change everything that happens So let me let me demo let me demo that because this is this is actually much harder to talk about without a demo So let's run again as root first And then let's ls roots home directory So you'll see that like it Unconfined you don't get too wrapped around the axle on these things like I don't actually I only let dan dan walsh know about these I I basically just know enough to look for s0 and unconfined and then I know a few types long story Sure, it's pretty easy for me to pattern match and go okay. Remember this one like uh, and then I go Okay, let's run a container as root, but let's bind mount roots home directory into here Let me give you a really important warning Never ever ever set the colon z when you're gonna bind mount roots home directory. You will screw up your system So there's two ways you can like now now let's see if we can ls This this root directory that we've basically bind mounted in let's exec in up. We can't open it. I wonder why let's ps it so if you look The bash command that is running at basically anything that runs in this container Whether it's bash or not all run with this context. You'll see this mcs label It's not just s0. It has the c15 c 121. It's also running as container t So it's a it's a constrained type that it's running as instead of an on you know, it's it's a system user It's it's like these labels do not match So basically what's happening is these processes in the container cannot access this right because I didn't run with dash dash privileged I'm still rude in the container. I'm still rude outside the container I still can't talk to root like I still can't read roots home directory because se linux is blocking it um Now there's two options here I could do the colon z that I mentioned that many of you may be familiar with the colon capital z Do not do that because that's terrible that will relabel your entire home directory with this label right here And then you won't be able to log in your system anymore. Um, I've done that before I did it to myself before by mistake and I tried resetting it was a giant pain in the butt Don't do that the other thing that you can do is the dash dash privileged flag the dash dash privilege flag says start this container again But this time let's see if we can ls the home directory. Oh, we can we can ls it. Why can we do that? We'll do a ps s e f notice that the new Processes that are running in this container are not constrained anymore. They're basically run is unconfined you they have this s zero There's no mcs label. So se linux is like cool. No problem. Go talk to those files and read them No problem And so this now starts to make a lot more sense. Um, you're like, okay I get that now I'm rude outside the container rude inside the container and I've disabled Um the se linux the extra se linux constraints on top of whatever already enabled on the host So i'm able to now read these files now what happens when we do it as a user So let's do this again as user as father linux obviously outside the container I should be able to ls my home directory. I can so you can see my my my files They're also unconfined you they also are home t user home t type, you know type and then there's no mcs label All right, cool. Now, let's run a container without disabled, you know, no privileged And then let's do an exec. Let's try to let's try to up. I can't read my home directory No, mind you. I mounted it as root just to make the demos the same But but i'm basically mounting my home directory into the container as root I can't ls it and you go. Well, why not same reason as with root because if you look this one is running as a system user Container t and then I have this mcs label So this mcs label is not going to let this process this bash or any other thing that runs in this container Talk to these files and read them. You're just not going to be allowed. Um, and so You could think of dash dash privileges like giving me my full user My full user capabilities inside of the container But what makes it a container is that it has these extra constraints on it If I do dash dash privilege You're just running a process that got from a from a container image You're not really running a container per se like you're just running like you're you're basically just running a regular process and so like In this scenario, I will be able, you know, like I when I run it as privileged obviously I can ls the home directory No problem. This is like running a regular regular process It's running as unconfined you and then there's no mcs label So this would be like if I ran this ls as regular as myself basically just like I showed you at the beginning here Like if I run, you know this command, it's basically exactly the same So, um, you'll see here again the processes are what you'd expect and now let's go back Okay So, um Let's get through here All right, so hopefully that makes sense So what privilege that I think we now understand just disables security controls, but it doesn't change the users All right. Now we have like three more slides to go through some cool stuff. Um Here's I want to run through three things in our roadmap that are coming up like things that you should pay attention to And I think that will now make a lot more sense if you understand users and privileged um In in kubernetes, for example, uh, we have this problem with cryo So challenges users want to run rude in a container with less risk Does not but it does not completely mitigate like the like people always say that user namespaces are a security tool But they're not actually they're a management tool that allows you to Run a little bit less secure, but do it in a In a less bad way. So like it's like kind of like the best bad idea We have for letting people run rude in container like still best like I showed you at the very beginning Run the sync user inside the container run, you know, some other user outside the container. That would be the best Um second best would be run rude outside the container like cryo You know, it's a demon so it has to be rude and then run as a regular user inside the container That would be second best third best is like, okay, we're gonna let people run rude in the container But we're gonna map it till like we're gonna use user namespaces to like map this to some regular user, right? Um, and so that's like kind of the best bad idea we have to let people run rude in the container um, we also, uh, we want user namespaces because especially for builds like image builds is probably the biggest place So I think dan walsh answered a question yesterday in our in our like sort of birds of a feather that we did yesterday about 10 10 10 30 to 11 30 something like that. Um our time, uh Somebody asked about this exact question. They said, um, will you ever be able to run builds? In obit shift with the default security context where it's limited to a regular user the short answer is probably not Like it's gonna be really hard to get that to happen But if we can run a new user namespace, we can at least allow Some of the things that root needs, um, you know and do that a little bit better that The short the longer answer is it's it's we're still on not completely sure like things like build it if we enable The kernel based overlay or you know, we'll we'll be able to do that as a regular user And there there is some idea that like a regular user will be able to you know There's only certain kinds of mounts that regular users can do And if we can hack through and figure out a way to let a regular user do some of these mounts and do some of these things We could probably execute build a in a container as a regular user But but running podman as a regular user It's probably almost impossible Uh because it needs to do all the things that need to be root basically like root only root are allowed to do it So so i'm long story short. This is a really interesting piece of work Keep your eye out the way it's going to be implemented is you'll use a runtime class and open shift So if you're not familiar with kubernetes, there's this concept of runtime classes Typically, this would be used to like run something in cryo or in cata containers So if you had two different container engines running side by side on a node You could use a runtime class to invoke one or the other you can also do it to do things like this Where it would change the security context So what's going to happen is we're going to have a different security context and then we're going to enable cryo We're going to basically pass a flag to cryo to start the container with user namespaces kind of like I showed you with podman And then you will be able to do interesting things inside of the container then you'll at least be able to use user namespaces We'll do it by default for the builds in open shift But then at some point in the future like the early work is going to be just to enable builds ourselves And this won't this this runtime class won't necessarily be made for end users to go invoke themselves But we will eventually expose a version of this that that we document and we show people how to use The next one that I want to talk about that will make a lot more sense is nfs and containers so So Giuseppe has been working on some really cool work Giuseppe Scrivano is one of the engineers on our team that's like just awesome And he does like all kinds of crazy hacky like low level stuff like he'll come back like the next weekend And he he wrote c run in like a weekend. He's that guy. So um He had this idea like well enough like in linux in like the in the version of linux kernel from like It's pretty recent. I think dan said it was september or maybe november october last year like it's pretty recent The the latest versions of the nfs server and clients in linux do know how to do Extended attributes, but they don't know how to do user namespaces. In fact, I don't think nfs would ever be able to understand user namespaces because as I showed You know it it would have to well I guess it could if it read ed see sub uid and like could understand all that But then it wouldn't understand on the server side like you would need to map though somehow to the server So this is pretty hairy Problem so what Giuseppe came up with is like well some nfs servers like the linux one Support extended attributes so if we can tuck The the uids if we can basically do a chone back to the original uid and tuck the high Number uid in an extended attribute we can read them back out and push them in every time we can do a chone So we can do some interesting hilarious things to hack and make this basically work with nfs And then this is this is nice because if you want to have your home directories on nfs or in hpc This is a common use case Or or you want to actually do image bind mounts where like say you have a standard set of images that you share out to every node in the cluster Um, and you don't want to have those come from a registry server There's all kinds of interesting places where you want to be able to use nfs to do this kind of thing We're hoping to be able to support this around the rel 8 5 time frame So it should be pretty interesting keep your eye on this But basically what we're doing is we're taking that high number that I showed you And we're tucking in an extended attribute and then we're basically reading it back out We're going to chone on the actual file system So the next place that I think is pretty interesting is and this is the last one Is around free ipa so in the context of that nfs those nfs mounts that I mentioned you can imagine now Okay, cool. I can now put my home directories on nfs But if I don't have these etsy sub uids and gids managed across an entire cluster of either desktops or hpc environment Um, how would I actually you know if I tuck these extended attributes in an extent? You know if I tuck these high number uids in an extended attribute then save them in the nfs server in a file Then read them back out on another node and etsy sub uids are not the same That would be a serious problem, right? It'd be the standard kind of nfs if you don't have the user's available problem It's the same kind of problem where like if you have a user crate on one node, but not on another That's why we use free ipa those kinds of things. Um So we're actually again targeting for uh rel 8 5 we're looking at being able to manage those etsy sub uid and and sub gid files across the cluster using free ipa So that'll be cool So then if you have like an nfs cluster a free ipa server and a bunch of rel nodes or fedora nodes or whatever You would be able to access, you know, basically have your home directories for podman On there and rootless podman would work fine across, you know, a whole bunch of different nodes And so this is pretty cool This gets us to a place where we're doing some really interesting stuff that we just don't ever really see like a Docker to be able to do um because it's just architected differently Like since podman executes has a local program in your home You know and it saves all of its storage in your home directory like a normal program you'd expect to do Almost like a desktop program It will be able to use, you know, the nfs server and the free ipa to basically do this kind of stuff. So it's pretty cool Um, and that is really it. I uh, I did compile some source material It's just like I did obviously refresh a lot of my memory on certain of these things So I want to share some of those so those pieces also I want to call out like bryan smith does some awesome videos So like I wrote an original article about running root inside and outside that was inspired by a talk with vincen bats But but then um bryan really went above and beyond and wrote this to or did this two-part Video and so it's really awesome And so I think if you need if like you want to get this crisp in your brain He he does some really good work explaining the same things I explain So with that, um, I will say if there are any questions. I think we're pretty close to the time Yeah, we are we have about four minutes left and I do have one question for you scott um from Dan in chat when using docker. You can't just run a container as a regular user. You need to be in the docker group which docker Which docker docks? Well, I think dan made this a uh, he did this on deliberately by the way Which docker docks warns you about uh with the docker group grants privileges equivalent to the root user For details on how this impacts security in your system What do you mean by running the container as a regular user or is it just a docker? Is it just a docker things compared to the podman? Well, so so this is this is a really good question. And yeah, I kind of assumed people understood this I didn't explain all the way so The docker so docker is a client server program right like it has a A client that runs either as root or as a user and then it has a server that always runs as root And so for you know root obviously can run the client and talk to the demon all the time But then like you're saying if you want a regular user to be able to access that demon What I'll have is that regular user will try to talk to that socket and the docker demon will say Oh, this user is not in the docker group. I'm not letting him talk to the socket So if you add your user to that group and then you do talk to that socket you have root access because as I showed Since the demon the container engine is running as root It's always root outside the container. And so when you run a program with dash dash privileged I have a one-liner where I show as a regular user you can execute like, you know docker run You know dash it dash dash privileged and then just run, you know ubi8 bash And if you run as bash And and then mount, you know, or i'm sorry dash v Slash cohen slash or you know or slash cohen, you know mnt or whatever You can mount the underlying hard drive from the underlying container host into the container Run this process with dash dash privileged and as I showed you if you're root outside the container and you're root inside the container And you run dash dash privileged you're root on the box Like there's there's you can go in and hack anything you want on the file system If you bind mount the file system into into the container So like you could cd into what I usually show is I would shrewd into like slash mnt And show look i'm root on the system right now and the worst part about becoming root with a docker demon is it doesn't even preserve your There's a login id that gets saved in linux in slash proc somewhere And for that for that i for that process and for that process the login id is not set because docker gets started by system D which gets started by the kernel and so there's no login id like so you have no idea who ran these commands as root So yeah, it's it's a really weird nasty thing that is basically um You know, it has it's the way the architecture of of docker works. There's just no way around it There is a way to run the docker demon in your home directory as a regular user and then run the docker cli to connect to the demon running as your uid But that's obviously not very elegant like and so that's why podman is just this think of it as bash for containers Like you just run it and disappears It's just like a bash for containers. It just runs as a program. It doesn't have a demon So it's just a single program. So it's just a lot more It's a lot easier for the threat model in your mind to like make sense with podman than with docker