 Hello everyone, how are you? Excellent, I'm going to ask you some questions to start. So how many of you are experts in containers? He is all right, I have one guy that can back me up if I have questions How many of you have ever had a hex put on you a? curse Have you ever had a curse put on you? You know what a curse is like when a witch like like somebody you do something bad to someone and then the witch Put some kind of curse on you, and then you have bad luck for a long time I'm hoping that during this talk since it is made out of a lot of demo It will not have bad luck because last week I had a lot of bad luck in Greece And I think somebody put a curse on me So if anybody knows how to remove a curse after this I need to talk to you All right, so On a serious note though, how many of you have used docker before? Good, how many of you have used podman? That's very good. How many of you have used cryo? Okay, last but how many about how many have used container D? All right, so less how about just kubernetes in general? Okay, good, so we have a pretty good. This is good now. How many of you feel like? There's magical black box stuff happening in the container engine I Will dispel this after this you will realize how not complicated this is or how maybe complicated and not complicated is at the same time It's actually quite complicated in certain ways, but then not in other ways, so I'm going to start with this talk by doing three things We're going to have some drawings to set up the demo that I'm going to run the demo and then I'm going to ask You some trick questions and see if you can answer the question So hopefully everyone is ready for participation raise your hand if you are ready for participation Okay, good. We have enough all right, so I Remember this this question comes up to me all the time right what does a container engine do? And it comes up through all kinds of weird questions that people ask me that show that they don't that it's black box to them They don't understand what's happening. They'll ask me things like Ah Cryo hasn't been around that long. How's the performance on it? And that's a weird question because even even that question doesn't completely make sense in that context It could if they asked it right, but just a general question like that doesn't actually make sense So I'm going to ask you a bunch of weird questions around these things but at a high level all a container engine does is Provide a user interface whether that's a command line or an API like cryo or container D That is used by another robot like the cooblet or a human interface like like podman or docker where it's just a CLI The other thing it does is mount storage a lot of people don't realize it kind of handles the meta operations of getting the storage set up The the kernel does all the actual storage work, but it's the thing that invokes it in an easy way Thirdly it creates a config file for run C. How many of you know what run C is? Not that many. I'm surprised. Okay, so it is a black box. It's definitely a black box So right you will learn what run C does in this talk So let's get right into the first demo and this is the one where I trick you first Does everyone understand basic processes in Linux, I guess raise your hand if you understand processes Okay, that's good. I think everybody feels pretty comfortable with that So in a nutshell you could think of it this way There's a user interface, which is just a CLI the user types of command in bash bash Does a fork or exec depending on what command you're running and then it talks to the Linux kernel fire off another process This is very simple basic Unix Linux 101 If you look at the way a container engine works, especially something like podman It really does something very similar, right? So and this is a simplified version. I'm going to get much more complicated But in a nutshell the user talks to podman you can almost think of podman like bash It's really just a utility that then goes and talks to run C Which then does a clone sys call instead of a fork or exact But it's really a very much like a fork except that it is it is it has some special Options that you pass to it to basically tell it what what you what namespaces to use in the kernel And then it talks to the kernel to fire off a process. We just happen to call this process a container But it's really just a process Okay, so looking at them side by side. It's pretty similar And I'm going to now do the first demo where I kind of show you What this looks like with a couple different container engines? so This is a fedora 30 box that I have That I have not upgraded to fedora 31 because I want to show you how this works on with just run C How many of you are using fedora 31? Okay, so some of you how many of you are using containers on fedora 31 do you do know that you're using C run, right? Okay, good a different Container runtime than run C. It is written in C by Giuseppe who is here who is actually I was just in this talk Okay, so I'm using fedora 30 though because I want to kind of use standard technologies that we've been using for a long time And then kind of show how this works. So to demonstrate I'm going to fire up a Few containers with docker side by side with podman. So let's do this Bash to and then let's run bash to so what I want to explain what this is I created a special container image that I linked bash to to bash And the only reason I did this was to simplify the process table as you'll see So let me run this and then I'll show you why so I run this one This ID just means interactive, but run it as a demon basically in the background So what this does is this gives me the ability to run two or three of them whatever so now Let's do the same thing with podman. So let's run ID local host I've created the same image for for Podman as I have in the docker storage for the container images that docker has and the container image storage that the podman has So they look identical Now we'll fire up a couple of these guys are three and then we'll do a ps and We're gonna do a psef with a capital Z so that we can see the se linux contexts And then we're gonna grep for bash to you should this should become clear why I did this now with bash to I just wanted something very simple that I could grep for in the process table. So The first trick question here is Which one of these was started with docker and which one was started with podman? Actually raise your hand if you think you can tell The difference, okay, we have one anybody else That hand Let's go for it. How could you tell the difference? Okay, you're on to something. I'll say that let's do this So yeah, you're seeing the parent-child relationship between the docker demon there so you can see these three ones Whereas and we'll dig into this deeper. You'll see that these three are actually their parents is con mon So yeah, that's a good way But but from but once they're running as a process like this just looking at the process table. Can you tell how? Say that one more time. Okay. Good. How would you tell what what about that? Do you think would give that away? Well, so so I'm gonna I'm gonna say no No because they actually use the exact same library They actually both use the se linux library from containers the containers repository on github actually docker demon and podman both They use exact same code Which I have I've sifted through to understand all this So yeah, if you look at this they're actually just randomly generated context and they're generated by the exact same code They could be different versions Possibly, you know depending on when the go binary was compiled, but I don't think that codes changed much in the last five years So I don't I think it's pretty much identical The point of this is You can clearly see that these are just processes, right? There is no difference You can't actually reverse engineer without looking at the process tree With kind of looking the parent-child relationship There's no way to tell technology-wise the exact same technologies in the linux kernel have been turned on and are running with These processes and it's pretty much it's impossible to tell from that alone Okay, now I want to do another demo I want to do the same thing, but I want to do it here as a non-route user Does everyone notice here? I had the hash so I was I was root when I did this. This is a common thing It's bad, but now I'm going to do it as non-route. So let's do the same thing. So let's do local host Oops. Yeah, I gotta So let's run a couple here as this and actually let's go over here and do a and then do an RM That's a on this so we get rid of these root running ones Kill these guys too just so that we don't have these polluting are What we're looking at? All right, so now we've ran a couple as pod man. Let's run a couple as docker and we'll do three That's good. No. All right. So now let's do a ps dash EFZ grep for a bash to Now you'll notice that you might be able to tell them apart Does anyone can anyone see what the difference is? Yeah, the fact that pod man runs like bash means it has a parent child relationship And it basically runs the process as that user and so using user namespaces We're actually running essentially all of the processes in these containers as that user's privilege Whereas when I ran it with docker, but but you might ask how is this possible? I ran it as a user. I've had people say this to me. They're like I run docker as a user. So it's fine Does that become quickly clear why that's not fine? You're still running you're essentially giving the user root access when you give them if you add the user to the docker group Which gives them the ability to execute the docker CLI They now have root on the on on your system essentially and I actually do a demo where I'll run one command To become root and change some stuff and muck with the file system and actually Dan Walsh does some demos where he shows It's actually worse than just you can become root. It's an untrackable root You can't even tell who sued you to it the login ID is not preserved and you're essentially are a completely Anonymous user running things as a root. So it's a really bad scenario But the interesting thing here is really actually that you can tell the difference now because if you run them rootless You're actually running as a regular user But you'll notice the same thing is happening with the se linux context and the labels are getting generated the same So the same technology is basically being used to isolate these containers All right, so let's go to the the next one. I already asked you this I asked you this this trick question and I like your process ID tree answer. I like that one of us So Let's do let's start with like the three things I talked about So creating a container is different than starting a container Creating a container is really about actually sorry. I got to show you one more slide I'm gonna kind of use this set of drawings to explain and set up the demo as I do it And I'm gonna use podman because podman has a lot more granularity in its ability to kind of To kind of show you what pieces of technology are being used and like what is getting generated and how things are happening? Whereas docker just kind of does it all in one shot You can actually break it down as sort of separate steps with podman and a lot of these commands are probably obscure And most people don't know them, but I find them very entertaining for showing how things work So in this in this drawing what I'm showing here is that podman is Basically creating a container and by create I'm gonna run the command and show you what happens Here what we're assuming is is that there's a container image already cached locally in the data storage in the storage for Varlib containers. This is this is the set of cached container image layers that are local and then we're also Going to container storage doesn't just store the images. It actually also stores metadata. So that's what that MD is So when you first create the container what we're gonna have is a container image locally and the metadata that would describe a running container But nothing else. So here let me show you So let's go back to this user. So podman create dash I D, let's do this and then let's do UBI 8 And bash actually, let's do a more complex one. Let's do dash V Slash MNT colon slash MNT. All right, so Again, we're we're creating a container We're running it interactive so that it'll have a terminal basically, but it will be disconnected Detached so that I won't I won't have it connected to the terminal here, but it'll have a terminal in the container And then I'm going to do what's called bind mounting volume So it's going to become very crystal clear in the next three or four demos what all's happening here But let's run this command and then see what happens Okay, so we get back this long string It's it's it's nothing more than a piece of metadata that represents a user space Essentially identity for that container that we will fire up, but isn't fired up yet. So let's do this podman ps Oh, we still have those other ones. Let's redo this. Let's do podman kill Dash a podman arm dash a and then let's do this command one more time. Okay, so now podman PS there's no container running right there's just the metadata representation of a container Which means there's an image pulled locally and then there is a metadata representation just that string and some other stuff But I'm going to show you what that looks like so podman has this cool command call The L dash L I should say command line option you can just run the last the last container image that was or the last container That was run so we'll do you dash q dash dash no Truncate this is just going to give us back that string. So now let's do this. Let's do a find on Slash var lib Slash containers. This is where podman stores everything and then let's grep for Let's grep for that that ID Hopefully I'm not using too much bash food here. Oops. What did I do? Hmm? Oh That's right. I'm not root darn it. That's uh, I should do this as root. Let's do this as root Let's do this same thing as root. I like to do these Without scripting them so that they're more fun. All right, so let's run that as root now. Let's do Let's do our grep again Does everyone understand I'm I was an I was a user so I I don't have actual rights to go look at Varlib containers, it's actually all happening in in the in the user's home directory All right, so I've created a container as root the metadata representation is there You'll see that some directories have been created under varlib container storage overlay containers You'll see that there's a user data Directory and artifacts directory in the sshm directory, but there's nothing else. There's no there's no mounted storage There's no config file that gets handed to run C which I'll show you how that works There's nothing else yet But at least what happens here is there is a piece of metadata that knows when I start this I've already kind of saved that it's going to fire up that process bash, right? So now Let's look at the next Step so the story with storage right so podman has this really cool feature That we can actually construct this container little by a little we created the metadata representation now We're going to cause that copy-on-write Piece to come into existence does everyone understand look at this this container image is three layers I'm showing a container image here that is three layers and then For a point of example, I'm showing what happens when I actually mount the storage is it adds one more layer on top of that container image But that that layer is copy-on-write so it well every time I write into that writable directory It will compare with all the layers in the container image and if something's different It'll write a new piece of data on that upper layer. That's all that happens that's the difference between between a running container and not running container and this might become more clear because I'm going to build it up and Then I'm going to tear it back down and show you how I tear these pieces apart and then I think it might I saw some some puzzled looks, but I think this will make sense So let's run the command So we'll do a podman mount on this container. We just created so we'll use this tag here This command will basically take the data representation of that container Image the mount point that's in a mount namespace and expose it to the root user So what I get back here is a directory. It's essentially mounted the storage that is in that mount namespace So now let's let's do that same find Actually, no, sorry You'll see here. It's actually a different ID this mount is actually a different ID than the container itself But let's go into this directory so you can see so you can see what's there You'll notice that this looks just like a virtual machine, right? This looks like if you exact into a into a container if you were to use the exact command or if you were to run bash It's like a container Now we're gonna do some funky stuff though. You do remember that I created the bind mount, right? So you would think maybe this bind mount Mnt should have something in it, right? Like this should be the local Mnt Well, let's touch a file and you'll see this later And then let's get out of here. Let's go back to my to my root home directory Okay, so now let's move on to the next step does everyone understand now We have the metadata created that represents the container We have a copy on right layer now mounted and ready to go and and we also have a persistent volume Somewhere out there dangling that we're not quite sure what's going on with it yet This is this is where I asked a question I Asked some more trick questions. How do you think that a container engine can affect the performance of the storage? Actually, let me let me let me ask the simpler question Do you believe that the container engine can have a profound effect on the storage? Raise your hand if you think it can Exactly, that's it. That's the best response The engine itself doesn't do anything. Alright, so and I'll highlight something very important here This is the fundamental difference between like traditional software and operating systems system software and operating systems and cloud That meta operation of mounting and unmounting soft, you know essentially overlay FS layers. That's essentially what we're doing That's really more Probably more controlled by the kernel than it is by podman But podman could have some effect on the mounting and unmounting So if you asked a question could podman have an effect if he tried to start a thousand containers at the same time Yes, the answer is yes because you would be doing a ton of metadata operations mounting and unmounting You know essentially overlay layers, but some of that would come from the kernels ability to basically mount and unmount that So now we can ask that question now the runtime performance though once it's running has nothing to do with the container engine Which which we'll get into in the next thing Is that crystal clear to everyone? Only the mounting and unmounting has anything to do with the container engine. There's nothing else that it does All right, so we have a mount point. We have the metadata that represents the container In podman's basically, you know storage area and now the next thing we do is we create the config file So you'll see I have metadata plus see which means config So now we're starting to get closer to a container, but podman has this cool command where we can actually just cause this to be created so Let's go over here, and I showed you that fine command you'll notice that there is no config file in this storage yet There's only that user data, and that's it, but let's do this podman ps Podman ps-a. I forgot to show you this you'll see this is still sitting in the created Status it's not it's not running yet, but let's do this podman in it How many of you are familiar with the podman in it command raise your hand? Nobody okay. This is a cool feature that Matt he unadded What I just did was cause podman to go generate the config file that it's going to hand off to run C now Let's go look at that config file You'll see a few things got created, but this is the most important one Let's cat this guy and then JQ dot Dan mentioned in his talk how many of you are in Dan's talk the security talk he mentioned this config file I don't know if you caught that, but he mentioned that this config file. It's kind of nasty, right? It's got a bunch of sec comp rules in it. It's got Well, it's very long here, so let's go back I piped it into a command called jq just to make it pretty so we can look at it But you'll see once I get to the top of this thing. Okay now we're starting to see stuff that looks like what describes a container, right? We're seeing the om killer stuff. We're seeing some labels or annotations. We're seeing things like Different options We're seeing the bind mount. You notice the bind mount or there are some already pre-existing bind mounts like a host file the resolve file Run secrets run secrets here And then you'll see here's some options for terminal things. Here's our bind mount. Here's the one that we added the one when we did the dash V So now podman has generated a config file that says hey bind mount Mnt on Mnt bind mount slash Mnt on the host into the container And you'll see all kinds of things you'll even see things like sysfs and then you'll see At the top you'll see things like what command it ran It'll even say things like sometimes it'll say linux depending on which container image you pulled and what made of data Isn't the container image and I'm going to show you that config file is actually a very complex It's actually I don't want to walk you through all this I do this in a deeper talk But that config.json that I just showed you is actually the culmination of three things It's the culmination of things that the image builder put in the container image that you pulled That's sort of the first thing like the default command that that the image builder embeds in there Like if you put bash in there and that's the default thing that runs That is kind of the baseline set of config options that get passed in there The next thing is the user can override things you saw I did the dash V the bind mount So I added the bind mount then the engine added all those for example set comp rules You saw that long list of set comp rules the se linux expert stuff is all added by the container engine So it's a culmination of things that the builder that the image user wanted the image consumer to basically have certain set defaults The things that the actual person running the container wanted and then things that the container engine wanted or what's set up to Do by default so that's kind of how that config gets built Let's ask the nasty question. Do you think you can create this configuration by hand? You can Yes, would you that is a question you see the container engine basically makes my life our lives much easier Because it goes and it knows it is building that config.json file based on the OCI specification For what that config file should look like I don't go deep into it in this talk but that that that config file is governed by a specification and then Run C the container at runtime that that file gets passed to The that's the only option that you pass to run C is that config file so things like cat a containers and G visor and run C and C run in Fedora 31 all of these OCI compliant run times Know how to consume this OCI compliant config file that podman built This OCI compliant config file is also almost identical to what docker would build or what cryo would build or what any other Container engine on the planet would build so I'm showing kind of this is kind of what the modern day a container looks like It's that config file. That's kind of the magic All right, so now we hand off we have we have We you know like I've mentioned we have the metadata. We have the config file. We have the copy on right layer We have the over we have the bunch of layers that are mapped into the file system from the container image We have persistent data kind of Which I'm gonna delve into a little bit and now we're gonna actually hand out all this off to run C so You might think that it's actually I showed you a simplified version and I lied to you What actually happens is podman talks to something called container manager con mon con mon does what is called a double fork It this it runs two forks in a row disconnects itself from podman Allows itself to and then calls it forks are run C and then actually that calls a clone system Called to the Linux kernel to then create the container and then what we are left with is Let's show you this So now let's do a podman ps Dash EF It's podman. Sorry podman ps dash a So we have this container. It's still in the created state. There's a config file now Let's actually run it podman start and we'll tell this thing to start. Okay now. Let's do a podman ps dash a or actually I don't have to do a dash a anymore You see it was created 11 minutes ago, but it's only up for five seconds We now have you know, we metadata is there. So there's nothing magical, but the process is running So let's take a look at ps tree and then let's try to find this guy So here it is You'll notice system D is the you know first process that fires up when a when when a system boots when a rel or or fedora system Boots then you'll you'll notice that conman is has the parent of system D And then bash is the child of conman. So let's show you in a drawing So you say this is the process of how it starts, but then once it's running. This is actually what it looks like It's just the container it's parent conman and then it's parent system D now all this other stuff is still in place, right? Podman is managing the config file the metadata the copy and write layers It's making sure that that stuff's there or not there This is kind of the definition of whether containers running or not now. Let's see something Let's exec into that running container podman ps And then let's do podman exec does everyone understand an exec is going to run What I'm going to do is I'm actually going to run another process in the same namespaces as that existing container That's basically what we're doing. We're basically firing up another process inside of there So I'm going to give it the ID then I'm going to tell it to run bash and then we're going to get a shell back Now we're in the container Actually before I do that I think I may have forgotten to delete something No, good. I didn't all right. I just want to make sure Let's get back in there. Okay. Let's go into MNT. Oh I can't get in there it's because I Did not use the capital Z option And so that container has a different se linux context or label than the MNT file system You know the MNT directory in the underlying host So if I wanted to actually go to get in there and look I would actually have to do something slightly different So let's let's actually let's do this again. Let's kill this guy and Let's do a podman RM RM-a sorry Now let's run it or let's do another create Now let's let's add this dash Z option And now let's do a mount actually podman do this quickly Podman mount Go look at that directory Let's do our little trick again where I go into MNT touch test file and Then let's go back now. Let's do a podman ps Show you it's there. It's still created, but now let's do a podman run or start I Don't have to do the in it because the in it will actually happen automatically if I start it So in this case it just created the config file and fired up the process at the same time Now let's exec into this podman exec try t Oops gotta give it a command. We're here now. Let's go into MNT There's no test file So what does that mean? Is it who's brave enough to answer this? No, it's not copyright No, in fact, it's not But it is a lazily up It is lazily mounted when you run the container. So it actually isn't mounted yet. So I basically just wrote Garbage data there that test file is gone basically it's mounted over basically as soon as I start the container So if I were to there's no way to actually get to it now. It's gone pretty much forever. I Actually, I haven't tried if there's a way to hack there might be a way to hackily get back to it But it's it's not gonna be easy But that I want you to understand that that MNT is very different than that copy on right layer That's actually what I was trying to point out. It's a very different thing It has native write speed So an anecdote that I like to talk about with this one was I had a person in a conference a few years ago Come up to me and say we're building yachto linux in containers and it's super slow And I said are you using a bind mount and they said no They're like, why would we do that? We don't care about it. It's ephemeral data And I'm like, this has nothing to do with it. It's ephemeral data compiling a linux distribution Which is essentially what compiling yachto is immediately what went through my my architect brain was Compiling a linux distro sounds like a lot of file system operations, right? Like it is building a root file system setting all kinds of labels permissions user, you know read maybe changing users I don't know what all it does but it makes a ton of metadata operations If you're doing all that in a copy-on-write file system, that's going to be very very slow Whereas if you do it in a bind mount, it's going to be native speed It will be just like compiling if we were compiling it not in a container So understanding the difference between the bind mount And the overlay amount is really important when you start to think about performance Because dan mentioned it if you have maria db or or my escrow and it's bind mounted into the you know Under that bind mount the performance is going to be native speed. It's going to be whether that's an nfs share Okay, it might be as slow as an nfs share or a or a or a you know block device that's mounted that's over You know a fiber channel network or something like that, but if it's a local disc It's going to be a local disc But not you're now back to basic operating system principles with that bind mount. Does that make sense to everyone? All right, so now what we have we have this running container, right, but I think Okay, let's ask some more trick questions So could we run a windows container here? Raise your hand if you think we could Well, just with what I showed you could we do Good good. There were some people that were quiet though. It scares me. I think you might have thought it was possible But I'll let you go This is not possible because run c as you see All it does is it takes this it takes this config file that podman built for it or docker or container d Or any other container engine cryo Takes that all it does is take that as an option and run a process So unless there's I mean possibly with key mu possibly with cata containers You could do that with cata containers. Maybe there's ways that you could do it if you fired up a vm But but with just basic containers that is not possible, right? These are linux processes is all they are All right So now the fun part where I start to pick apart the cli of docker, which we basically copied with podman But it has some confusing things So let's kill the container and then see what happens So podman ps So odd man Hope i'm in the mount. Sorry. Yeah, let me get out here. So podman ps All right, so now what happens when we kill this? Let's do a podman kill Who who thinks they know what happened here? Raise your hand if you think you can describe what I just did Say that one more time the process has been killed. That's one part. What else has happened? No, the copy on right layer has not been deleted yet. That's actually the magic But yes, the process has been killed to be honest with you. I'm not sure if the bind mount's gone Completely. I don't know if dan even knows that is it gone. I think it is But but most importantly the overlay file system is still there that copy on right layer is absolutely still there It's not gone. So when we do a podman ps Uh dash a you'll see that container is still there. It's just in an exited status But what that really means is that copy on right layer is still there. I can use it. I could save it as a new image It's it looks like this so So you'll see still we have the metadata that's when I do a ps dash a I can dump the content see the metadata I could the config file is still there so I could fire it back up if I want Uh, I the copy on right layer is still there and the persistent volume is in lala land Like we can't get to it. It's probably unmounted, but it's irrelevant um Because you can't use it because there's no processes running but then Now what happens when we go to Now what's the next thing we can do we can do a podman rm and we start to We start to deconstruct it right now. What does it look like? Well now Or You'll see it's gone the metadata is gone the config file is gone. Everything in varlib storage is gone um And you start to see okay now Now I couldn't turn that old container into a new container image because that copy on right layer is gone So now it looks like this notice here. I have the copy on right layer. I have the metadata I have the config here. I have nothing var. You know container storage is empty doesn't have anything The container that copy on right layer is gone the persistent volume is basically gone But but the data is still there on disk So obviously it doesn't delete like anything I put into mnt is still there So if I would have right wrote a test file to mnt and it was bind mount on the host It would be left on the host right and that's actually the magic of bind mount now uh What happens when we do an rmi I Or you know, so so now you'll see if we do a podman images If we do like podman Just do rmi on this guy, you know actually this one is fine Now it's gone gone right like like it's it's every all the copy on right layers are gone So now it looks like this There's just a container storage is still there. It's empty that persistent volume is still there system d is still there But there's nothing Though all those copy on right layers are gone now all the copy on right layers that have been mirrored down from that Container image being stored locally are gone even the read only ones that were basically part of that container image so does that I'll close with basically the fact that container engines are both more and less complex than you think There is actually this long string of events happening where podman's calling conman and conman's calling runc runc He's talking to the linux trolla fire up containers and I do this spiel a lot of time when I talk about how that works But in reality the things that the container engine are doing are not that complex It's creating config file. It's mounting. It's telling the kernel to mount storage. It's not even doing it itself And then it is You know just basically handing that off to runc And runc does all the heavy lifting of the of the interface between the kernel and and and basically the entire config file So the question I'll leave you with a little bit of further reading, but I'll say so now we can answer questions like I I'll give an example of a question that somebody said to me On a webinar I did like a month ago. He said is cryo mature enough Like how do you feel about it? Is it mature enough? And and I'm like it doesn't yeah I'm 100 confident in it because it creates a config file it mounts some storage and then it Basically hands everything off to the linux kernel And so we've been running these exact same types of containers with the exact same technology with the exact same s e linux libraries You know setconf runc all of this stuff It's all been running exactly the same for years now five seven six seven years now so Yeah, what is there to not be confident in in its maturity does that make sense now? Like there's not much that the container engine's actually doing So I get these weird questions that don't make sense because people just don't have a worm and fuzzy feeling of In their stomach of how this works. They don't have a gut understanding of how it works So they ask weird questions that don't quite make sense So I'll leave you with Whenever you have Performance or security questions You should now be able to kind of reverse engineer From this understanding and kind of answer the question with a lot more confidence things like should I compile? Yachto linux in an ephemeral overlay mount fs probably not because it's writing tons of file system operations That should now be crystal clear things like how does the performance? How's the performance of cryo versus? You know container d I mean there's some metadata operations that there might be differences, but that's it You know the running containers are identical Um storage network same same exact thing You know these are all the exact you can answer the question with storage or with network without even knowing it I haven't even showed you how it works, but it basically works the same way Um No, it doesn't have a demon it uses the met. So let me repeat the question. So how I think the question is essentially uh docker historically had Docker is a demon and so it can control all of the storage basically in place like like everybody has to talk through that demon So basically it should basically be the bottleneck the governor for all of that metadata. Well, it's the same thing I mean it's file locks and and and podman it, you know 20 different people Well, here's a here's a couple of answers to your question How does podman manage that? Well when you're root, you know, it's basically everybody's accessing varlip, you know varlips container storage basically Um, I don't know if there's file locks dan is I assume there are file locks. Yeah, exactly Yeah, so it's that simple Does that make sense? It's file system. It's basic file system operations file systems in general are atomic in nature They know how to do atomic transactions. So you rely on the file system to do it basically I do show that in a in a slide Uh, I don't have it in this deck But like I do show a slide where people don't realize but that file system driver and the block driver are shared Between all the containers, right? If you if you if you bind mount stuff and it's on an nfs share you bind mount something And it's on a block It's on a, you know, uh fiber channel, you know ice cozy or you know something like that It uses the same ice cozy driver like everybody shares that ice cozy driver So this if it's xfs if it's you know Extension for the block and faucets and drivers are all shared and basically like dan said they know how to handle atomic transactions So there's no problem there now that said I'll go a step further and say users have their storage in their home directory So different users can't see each other's storage That's something docker can't do because everybody's storage is there So if one user creates an image and has passwords in it, which you shouldn't do But if you did by mistake all the other users can see that storage because everybody's sharing that storage Whereas in podman if you're running rootless, they all have their own storage everybody It's basic unix 101 everybody's storage is in their home directory. It's dot local slash container storage or whatever Does that make sense? any other questions Do you have any nagging performance and security questions? Am I am I supposed to talk about that? right now it is But we're working on something else that we That we basically did we publish that yet? There It did go out. I wasn't sure it went out. All right, so it went out so I could talk about it But in a nutshell, we're working on an api So there's the local podman cli api if you will that's the human interface, right? It's essentially the docker api It's just podman run podman ps podman, you know kill podman rm rmi all those sub commands That stays the same. I would call that the local api But then we had this remote api called var link and var link was a way of exposing An api that could basically control all of these same things But in a programmatic way so you could get request response basically so that you could run for example podman in a virtual machine But have a command Locally on a mac for example or local on a windows machine or even locally on another linux machine and then have podman running inside of a vm But we're actually working on and we just announced that there's actually a 2.0 version of essentially of this api And it's actually going to Implement the docker api. So we're going to have an api We're going to use system d I think right so basically socket activated or or network activated and then basically have it talk So the magic here I'm just going to cut the chase is you can run the docker command and talk to podman Like the docker command wants to talk to a remote api that is docker compliant And we're actually building an api an api a remote api that will be you know Docker composed docker command can talk to it And then we go fire off a process for podman in the background do everything I basically showed you That's kind of the answer to that So var link goes away But something very cool comes in replaces it So I can stop asking the question. Can I use docker composed? I can just say yes People ability is docker composed with podman Any other questions Any weird questions like yachto linux in a container? I love those weird ones No Going once We're close, right? I think it's in the It's in the uh, I forget what stages of yeah, there's a different stage if you're a devian user We will have podman soon in devian. Uh, it's in like a the testing ish phase of whatever they call there It's new Yeah, new right, but we had one more I think you might be right, but I don't know I haven't tried that but I think you might be right Yeah, yeah, I think you're right I just didn't have time to I just thought about it as I was giving this and it was like wait a minute How would I do that because running something like mounted inside of a container strange? Yeah, but yeah, I think you're right I think that would work All right, I think I'm gonna cut everyone go so thank you