 Hi, I'm Lucas Roberts. Hi, I'm Lucas Roberts. So Lucas, who knew me, he takes care of S.C. Lennox. So we're going to be talking today about S.C. Lennox and containers. And, you know, I'm a big believer in S.C. Lennox being the greatest thing for all containers. It's the American expression of S.C. Lennox and containers, but again, the white peanut butter and jelly, but basically is a perfect combination of two things. But we're going to go through that in a minute. S.C. Lennox is an expert at protecting the host file system from the containers, right? So we want to make sure that the container doesn't break out onto the file system and attack the file system. So that's really what S.C. Lennox sort of explains that. So the best tool for that, and my belief, is S.C. Lennox. So, have you done my presentations before? Oh, there's like even a church. Anything in red, you've got to say. Got it? S.C. Lennox. Perfect, he's working. That was our end this time. Let's see if we can get together in the next one. Ready? Every file and directory system has a label. Policy rules control the access between label processes and label objects. The kernel and process rules. That's all that is right. I preach this all time. That's when I choose a simple labeling system. Every process has a label, every object from the system has a label. And then we write rules that says how an object interacts with that. So now we're going to go into a deep technical dive. So I'm going to bring up, of course, S.C. Lennox coloring. So S.C. Lennox is based on a thing called type enforcement. Type enforcement is basically, we say a process is a type. It's an Apache process. The data is on disk, it's an Apache data. And then we write rules that says Apache processes can only read, Apache read only data, or they can only read and write Apache read by data. If a Apache process tries to do anything else on the system, it's going to get denied. Everything's been denied by default. But to make that simpler, Lennox is going to say we have two processes in the system. We have a cat process and a dog process. Everybody look me so far? Then we have a artist, food artist. We label one of them as cat child and one of them as dog child. Then we write a rule that looks exactly like this rule. This is pretty much exactly S.C. Lennox policy right here. Allow a cat process to eat, permission to eat cat child in the category of food. Then we write a rule that says allow a dog to eat dog food. Everything's denied by default. We write these rules. Then the cat is able to eat the cat food and the dog is able to eat the dog food. The cat can get more from the kernel, gets more cat food. But when the dog tries to eat the cat food, the kernel steps in and blocks it. That is what type enforcement is. In the S.C. Lennox world, we label containers. We're talking about containers here. We basically type enforcement protects the host from the container processes. We have container processes. We actually allow container processes to extremely read stuff that's under slot choosing. The reason for that is when a lot of people volume out in things from the host's operating system, from the host's user into the container. But the container processes are not allowed to write to slash a user. Just able to read, but that's by default policy. Container processes are only able to write to container files. These are the types that we use. So this is the cat and this is the cat food. So we label a container process as container T and we label a file type container as container file T. The only thing in the world that a container is allowed to write to is container file T. The only file logic slots right there. Most container run time CVs, containers break out of the file system have been blocked. This CV, blocked by S.C. Lennox. This CV, blocked by S.C. Lennox. This CV, blocked by S.C. Lennox. This CV, blocked by S.C. Lennox. This CV, blocked by S.C. Lennox. Those are the five that I found this morning. Almost every single break out of container run times has been a file system attack. They've figured out a way to break out of the file system. A couple of examples of those. Run C had an open file descriptor. I had created a container exactly to a container and I brought an open file descriptor with me. The process inside the container could attack that open file descriptor. They could get out that open file descriptor to the file on the file system and then it could start attacking the file system. Another example, Slash on the host operating system is a well-known entity. And what was happening is people, there's this call called open file ask. And UA was put in, if you guys know the file system you're on and you know the I know of the slash file system, you can actually open it up without worrying about long namespace. So people broke out of containers, my patient would get into that slash and then they would go walk the file system. But S.C. Lennox would step in and break through at that point. So we've talked so far about S.C. Lennox type enforcement but S.C. Lennox, obviously if everything in the world is a cat or everything in the world is a dog, the dog's in the cat's can attack each other. So we just go type enforcement. We have the cats, the container keys are able to attack the other container. So we want to use S.C. Lennox for separation between the containers. So type enforcement, S.C. Lennox contains them all. What about, I already covered that. So S.C. Lennox, the second part of S.C. Lennox is based on the start and analyze multi-level security. Years ago when we started doing virtualization, oh by the way, Donald Trump could be proud of it. But basically years ago we had a, we were doing virtualization and we created a thing called S.C. Lennox. And what we did is S.C. Lennox had this fourth field. If you look at the next thing that's almost always you see it as S.C. Zero. That's the system low in the MLS world. Well most people never used MLS. This is like top secret, secret stuff. So we have this field in S.C. Lennox, ladies. We decided to create a new type of policy. And we created a policy called MCS, multi-category of security. I actually have some patents. I'm not so very proud of it. I'm not so proud of it, too. So what we're able to do with MCS category, we basically said we're just going to pick random MCS categories and then we're just going to guarantee that the MCS categories match. So based on that MLS and what we do now is we're able to say, we're able to identify the dogs. So we have dog types and we can control a dog type attacking the host operating system. But if we can identify each container type differently then we can have them isolated. So we put an MCS label of Fido on this dog and on Spot on that and we're going to create food. And now we can also put the same MCS label on the food object. So it's still a container file team object or a food object but we're adding the Fido to it. So I can have Fido in Spot and then obviously Fido's able to eat his food but when Fido tries to eat Spot's food the kernel steps in. So what happens when a static container is a container that has a lot of times basically pick out a random MCS label and then they assign it to all the content and they assign it to the process that's launched the first process in the container. And then basically that is how we do it and we do the same thing with virtualization we did it with original ownership. In original ownership we had millions of users each one with separate MCS labels. So that's how MCS enforcement works. So what happens when a container breaks out onto the system? We're about to show you. So this is a script. By the way all the demos, if you're looking on the slides the demos are actually listed. You can go and play with these at your leisure. So we're going to use Podman to actually launch a container but in here we have how you can subvert your system. So we're going to do Sudo. If you run Docker in the Docker group you can do this and break in your system but we're going to do it as root. So we're doing Podman run. This is just to make out my scripts look better so I'm setting it home in by a little bit but this is a really critical thing. So while you're mounting the slash in the root host operating system launch a slash host. I'm just running Fedora container. I could run any container so we're not even going to use that. And then we're using to root to host to break out of the container. So basically I am getting out to the host operating system I'm getting out of the slide. If I ran this container you would see I was full slash in it and I happened to write a little script here to make this a little quicker. So I'm actually going to when I break out I'm going to try my breakout script to actually cause happy to have. So now we're going to do it and just to show you another one. So right here it shows that I'm running as system zero. Basically the type is here container T and up here we have the MCS label. And again Podman picked that out randomly and Podman has the whole database of containers that previously created so that it doesn't overflow it. Now he basically tried to cat into shadow as root, failed, touched file and system root. He's going to go into the home directory. He's going to try to get into the SSA directory. Now he's going to look at a pit one on the system. You get permission to die. Now he's going to stop doing system D communication. And now he's going to try to run a docker container and he's going to get permission denied and clear. So basically with SELENIX again the only thing it's allowed to do is to have a container files. It's not allowed to, even if you break out as full root on the system, that's it. Now when we run containers it's why we all containers run with a single type. Inside the container, all the processes run as container T. But we don't have isolation inside of the container but that's by choice. The microservices world we're trying to get to the point that the container does one thing. It's the Apache process. It isn't the Apache process it's in a database process. In a container world you have the Apache process and then you have a data process. And those would be two different paths two different dots. And that's how you're supposed to do stuff in the container world. So in another American expression what happens in Vegas stays in Vegas. We want to keep you in Vegas. We don't want you breaking out into rural America. So this man is going to now be our rural character. So you have still some commands here? Okay. So as I'm running those attacks I actually have some auditing to show you what's happening on the auditing sub system. So this shows that this is when I cat it out Etsy Shadow. It shows that container T try to read Etsy Shadow. And I was going to show what happens from the Docker when I tried to run the Docker command. And so here you see that I was trying to connect to Docker demon via the Docker socket. And so in your log file it would be going wilds at the point that these containers are broken out. So you'd have to watch your audit log file to say containers are doing something often evil on my system. I think I'm done now, right? Yep. You're done. Okay. And this is the next generation of the S.I.S.T. experts working on their training right now. So if they can understand it. Okay. So don't talk about the MCAs and let's look on the MCAs how it's worked with relation or how it's worked with specifically with the categories. So let's say we have two containers and as we can see both the containers have same S.I.S.T. Linux type which is container underscore T. But these containers have different categories. The first one has C1 and C2. And in the bullets here we have four objects which are labeled container underscore file underscore T which let's say it's kind of files on the system. But all these all these categories are subsets of of the following set of categories which is C1 and C2. So we can say that this category category dominance these categories and same story for container with category C2 and C3. But let's talk about the containers. So let's focus just on these two Yes, of course. So when I think about we picked two categories by default. And we usually concentrate on two categories. So you share the dominance that means that if I pick just one category other domains can start to dominate it. But the interesting thing is why two categories? Well first of all there's 1024 categories available. If I pick two categories I get 1024 times 1024 which is basically around a million. But you have to divide it two because C1 comma C2 is the exact same thing as C2 comma C1. So that actually cuts the total number of potential containers you can run on a system to basically half a million. So if we wanted to get more containers eventually we find out half a million containers on a notice is not enough we could always add a third category but the math gets a lot harder than that. Okay, so let's let's look on this picture. Again two containers with the same SLINUX type container underscore P but different categories C1 and C2 that's one set and second one is C2 and C3. So if we have three objects on the system again with the same SLINUX label which is container underscore file underscore T but there are different categories. So this container can access the let's say this first object why? Because there is an allow rule saying that container T can access container file T but also this set of categories is subset of the categories of the process. So C1 and C2 can access C1 and C2 object and also this file can use containers for the sharing because there is no more specific category which is again the subset and same story with the second container with categories C2 and C3. So this is how containers can share access to the same object on the system and this is the similar picture just showing that what is not allowed that container with label container underscore T with categories C2 and C3 cannot access object file with categories C1 and C2 and same story for container with categories C1 and C2. So container engines like let's say podman can have a reliable feature if you mounting for example we have example here and after the mount point you put semicolon with capital Z that what will happen the podman will reliable the mount point on the file system as container underscore file underscore T but with the specific categories which means that only this container can access the file on the host system next option is use the small normal Z which means that podman will also relabel the mount point on the file system but it will label as container underscore file underscore T which means that both containers executed in second command and also in last command can access the object labeled as container underscore file underscore T on the system so what's we have some problems with this of course because we are using just one policy for containers which is default the default label or default type which is container underscore T and in some situation this type is really strict for example fedora silver blue project needs containers to read write home directories and second second case is fluendi project needs to containers to be able to read log files in slash var slash log directory on the other hand we have situation when container underscore T type is to lose for certain use cases which is no as Linux network controls it means that container T can bind on any port any network port sorry and also there is no control on Linux capabilities which means that container T can use all Linux capabilities so what's the current solution for these problems the first one is to use the relabeling feature for example if you want to start fluendi inside container you can relabel slash var slash log to have label container file underscore T with the specific categories which is of course the bad idea because var log directory gets container underscore file with set of two categories and other confined tools won't be able to access slash var slash log which is not good and second solution is turn as Linux off for the container which is of course bad idea and my question is why it is bad idea anything what does it mean and there is one more important thing then will be set up so and now I propose you another solution or another solutions can be write as Linux policy for the container from the scratch which is too difficult for system administrator because it's time consuming and you need to have deep solenoogs expertise second option is to add additional solenoogs rules to container underscore T which is a custom module which is still not ideal because you need to have these solenoogs expertise and this additional rules will apply for all containers which is things what you probably don't want so there was a request of creating these custom policies for the different team for their containers and the solenoogs team and also the containers team had a lot of request with please could you write for our container because we need to access home directories same story for while and as you know this is not long term solution and in solenoogs team we realize that we always give one policy and then there will be another team ask for another policy so it's like we give some team just the fish and I like this saying that if you government fish in one hour if you teach him to catch a fish you do him a good term so in solenoogs team we created tool called Udica which is fishing route in my native language I'm Slovak so we really call it Udica and it's fishing route and right now I teach you how to use the fishing route to ride your own solenoogs policy so Udica is tool for generating solenoogs security profiles for containers so we will have one really easy example we will have example container and this container is mounting a slash home with read and write permissions mounting a slash slash full for read only and exposing FTP port TCP 21 so general solenoogs general policy container underscore T cannot write as then mentioned or read for slash home and also cannot read slash bar slash full on the other hand the container underscore T can expose all the ports so let's generate solenoogs policy for this container so we have solenoogs in enforcing just to be sure and I will start the first container mounting slash home with read write permissions and mounting slash bar slash full with read only and exposing port 21 okay so we are here on the right side we can check for the solenoogs labor of the container which is container underscore T which is the generic type for containers and two categories and as I told you the container underscore T cannot access cannot access slash home and slash bar spool I will prove it to you cd home ls permission denied as expected and slash bar slash spool ls permission denied as expected on the right side we see sc search tool I'm trying to query for allow rules when processes labels as container T and object is labeled as home underscore root underscore T or var underscore spool underscore T and there are no allow rules on the other hand important ABC important allow rule is over there that container T can bind on all port types so with podman ps we can check for container id which in future we need to use so I put it here to have complete demo and this is the most important command of this tool which is the podman inspect minus L which means the last container so the last container will be inspected and I will create a selenux policy named my underscore container so I executed it and policy is created we need to do two things first one is allow the old needed only modules and second one we need to restart the container with another parameter which is the security label and security label is my underscore container dot process that will be labeled for container for running container so policy is loaded using the s e module minus e I start the second container with the with the security label as I told you which is the my underscore container dot process again let's check for the label of the container and it's my container dot process instead of generic container underscore t so again let's check if I can access a slash home and as you can see I can access it it was mounted as read write so for example let's touch this directory it's working and let's check access for the s e module and again it's working on the right side again we see s e search s e search queries and we see that we have already existing allow rules loaded in the kernel and then please also show that my container dot process can bind only on ftp port t as you can see okay so so right now we generate s linux policy and we can read and write slash home we can read slash bar slash pool expose the ftp tcp port without writing any allow rule so how it works under the hood the concept is based on block inheritance seal language which I show you in next slide and Odisha creates the policy combines combine something what we call blocks or we can call it also templates by inspecting container json file and Odisha looking for mounts, ports and capabilities because as then mentioned what happened in Vegas it stays in Vegas so from host we see a container as a black box and we are care about just the interaction with container and the host system all these allow rules are combined with the default template we call it base underscore container and it's required for every policy based on Odisha generated policy so the first block is allowing read and exact slash user and read some configuration files then we need net container for allowing network access and last thing we need a block which we can call home for accessing home directories as I mentioned base is required for every container we need net block for allow binding on FTP port 21 and last one we are allowing read and write home deals that's these are the blocks we will use and as you can see on the right side we inherit all the allow rule from these three blocks and build my underscore container policy but there is also one more block which is spool block we don't have any default block for slash bar slash spool and that's the reason why I used this path that if there is situation when Odisha detects there is no default block Odisha will check the aslinux database what labels can be inside the path in this case it's slash bar slash spool and create all the allow rules for it and add it to my container my container block so this generated policy can be used with multiple container runtimes such as podman, docker, build and last but not least with the Kubernetes so these are the Odisha repositories the first one we have sources there and in second one there is a proof of concept how these blocks work so if you are interested feel free to read it also Odisha is already available in federal repositories so all you need to do is just download Odisha and message of the day is that you can generate custom aslinux policy for your containers with one simple command called Odisha which is phishing grid so these are all the all the slides, information and block files from then and the rest of the aslinux there is no link to the slide if you want to just get the slide and get them from there every demo that we are showing today we have created a demos repo so anybody wants to play with any of these anything we are trying to encourage anybody that goes to beta containers if you want to play with any of the tools there so those people do a demonstration that you can take home with you or take to your local host and run as demonstrations yes the aslinux break out somebody on the other track I am sure is going to be opening a local request after this to put it on the demos so you can actually see how to play with Utica if you want to interest more in Cloudman this is where if you want to look if anybody is really interested in container aslinux which is where container T is defined that is also a repo up there and then the stuff that he has done the customizations is there and those are all a lot of people talk about aslinux so at this point we open it up to questions that is a bit too close everybody thinks this is perfect what I can bring up do with a pie man perspective to show what it is yes so what is next what is next I think we are going to have people play with this if some of the stuff is so high he has templates because for instance blogs he didn't show blogs even though we talked about this so the problem is in certain directories there are lots and lots of types so that is why he has a blog that looks at content in the home directory well that is a real big this is probably 50 different types in the home directory he has a sort of hot code the types there in the home directory similar to our blog so I think what we will be looking at in three situations we have been hit with recently we are home directory biolog and etcd so people want to be able to share the systems those are free that I know of but I think in the future we will be hitting people want web services so I think as we evolve we will evolve additional types of that because I really hate the fact that right now I have to tell people to this is for container separation I never tell anyone so this is a really nice tool for that but let's bring up the department inspector so we want to show you what he is looking at so so this is basically all the information you on so here he is looking at the mouse he knows what the default mouse are but if he sees a bind mouse from the holes he sees slash home and says that is going to trigger his home block he is built in so he is not worried about those but he gets down to so he is only looking at bind mouse go down to the capabilities I would like to answer a question why are we so loose versus tight some ways we are too tight some ways we are too loose the reason we are loose on networks and capabilities is because people in the podman command or the docket command can change really easily which capabilities are available in the container and which ones are out and we can't podman the container types are sort of hot coded and we don't want to have 32 different types for each capability there is no way to generate the policy on the fly we have no idea inside of the container which ports you can be binding to so we have a lot of time to really help so we have really loose on capabilities we allow other parts of the kernel to control the capabilities set up and allow networks to be controlled by firewalls but this allows us to actually enforce some of the SMIs you want to talk to me? so basically we are using a container or just a container using a podman to design some kind of policies have you ever think about including this functionality to podman itself because it looks more logical when you are defining this with a podman a running container to define the same solutions initially? I don't think five minutes should be generating maybe we could add a generate policy flag or something like that but I think I hate to make five minutes like the tour where I really understand the sequence I'd rather leave that I think what he's done is the traditional unix way of being able to generate based on output so I hesitate to add this to in some cases in some cases I think the colon Z is a better idea than doing this so just because I volume mount in if I volume mount in slash com slash dwall slash foobar I would rather than use the colon Z in that case I want to volume in the huge file systems that I really in fact SC Linux the podman the SC Linux going libraries now will block you from relabeling slash home some people put slash as we showed in the breakout if you do a podman slash slash host and then put a colon Z in it podman is going to try to relabel the entire file system it's really easy to use to make mistakes but in the case you do a private directory you should lock it down to only the capabilities this was discussed in the past and we decided that we will have a standalone tool for it you showed that you can have a simpler label without a share content with you containers but what if I want to share content only with those two containers and not the others how do you do that I think that would be an enhancement to you because basically it's a great new type for sharing content so perhaps perhaps we could look for the colon Z in there and maybe there's a flag to you to generate a special type for lawyers or something that to me would be a decent thing because right now we have two choices either things totally separated or every container in the universe can read and write it so the thing would be to create a new type say my container private you can have rules that says his new container process can read and write next to you anything that is my share now if I'm into the containers I'd be great for any ideas we do it so please feel free to create some issue on github and it would be perfect yes but they are different if I understand your question you're asking what's the idea to just inspect the container and allow everything inside the container without the knowledge of what I'm allowing right well as you explained we're really looking at content from a host that's getting leaked into the container so finding to pause the sort of host concept by mounting in as a host concept most of the other stuff that's in the inspection really a host concept most of the stuff doesn't so if someone specifies initial IP address there's not a decimates control on that that would be interesting to look through here maybe to find additional things we could lock on I think he got the primary isn't this a case exactly for not having this by the fall down of a matter of human problems we're happy to separate it so the admin can be done when the user can try to do what everyone the definition of the policy would lock it's interesting we've built in the wiring for everybody to be able to pass types down all the content you can do it build a 5-minute Docker everybody does it and it fills in the Kubernetes so if the only problem is going to be you'd have to use some kind of operator or even set to make sure that this policy gets them solid in all your folks so so if we still we did some great content in part we hand made the policy written for the code so we can do exactly the same things in the host policy well, I always think of Fluentee, the example of Fluentee right now has to run unconfined by AC Linux now they can write their own policy fairly easily but they know what they want and without them having to be AC Linux experts they really all they say is I want to read everything about lock, generate policy funding now they did their job and I was to take their policy and tell OpenShift, I need this installed on all machines that we run in Fluentee right, in the example of the desktop Silica Blue they would probably generate this policy and install it for so there's a thing called Toolbox I don't prefer Toolbox right now it's totally unlocked by AC Linux and they might want to all they really care about is that slash home is home direct we can't do anything there but they really don't want that thing being able to execute suit and do all those things in hopes they might want to generate their own policy for the Toolbox Any other questions? Could this be usually something like flatback? Well, yes it can be Yeah flatback, flatback right not flatback, sadly this point has very little sandboxing in it and there's reasons for that the ultimate goal of flatback for those that don't really know what flatback is they're talking gibberish only one person doesn't know that's almost the language get your head out of the crew so flatback is a way to run desktop applications in a container format so you want to run firefox you want to run flip so what they've done is they've sandboxed applications now you can run the same version of firefox in ported operating systems it's different than sort of pardoning containers it's a different format but what you really want is you want firefox eventually to be isolated from home directory so what I would want is I'd want my firefox not to be able to read my data as a safety piece by default and not be able to to say attack all the nodes road processes firefox would be those data states and the goal eventually would be to have any time I do it obviously firefox has to read stuff in my home directory once in a while if I want to upload an image to the internet or if I want to download a file I want that to end up in my home directory but usually that's indicated by the user you go to say the save button and say I want to save a file and you go to the button says open a file you want to open a file and what they're going to eventually do on the desktop that loads files be a separate application and then the file loader will read the file and basically give an open file descriptors some version of an open file descriptor to the thing that's in the container now also you could have applications running on the desktop that are isolated from your home directory but right off the back it doesn't do that once we had that we could start to look at wrapping it as a Linux but right now it's basically needs everything Weyland also has that Years ago I wrote a thing called Sandbox that the X-Wing had that in fact it was closer to what Podman does nowadays for doing finding it but hopefully eventually it's live back Anybody else? Yes? Can I exec a file where's the file is it? I'm not a problem the only place I can take container t it only executes files that are on the need slash user a lot better in the container and that's just so you can volume so you can do a dash v slash user ptrace which is what I always do because ptrace exists inside of my container so if I were to use a bin ptrace into my container that's running into a running container that I can basically stop ptracing or pitting and things like that of course that's a little shaky doing that because you have to rely on a shift but I have to decide the image to work so for that use case we decided slash user is pretty much not a place where you write your sequence so we allow you to read and execute slash user it's not allowed to write slash user but everything else the only place you have to write is container files Everybody totally understand it? I have no coloring books to end on so I'm too heavy to bring alright you guys get 15 minutes back or 5 minutes back Thanks for coming