 Can everybody hear me? Yeah, great. Well, I'm actually a bit surprised to see that this many people here. Like, everybody cares about security. Woo-hoo! We've won. But yeah, hopefully you did not just come because of the buzzwords. I mean, I guess the only one is OpenShift nowadays. But yeah, so we're gonna give a talk about Excel Linux, OpenShift, and generating custom policies. Super interesting. Woo-hoo! And that's that. So first of all, who are we? We're gonna start with my colleague Lukash. So hello, everyone. My name is Lukash Rabetz. I'm a Excel Linux evangelist at Redhead. I'm also a member of Security Technologies Team at Redhead. And I'm a rail and federal contributor, mainly Excel Linux policy, ex-guest, Uditsa, and slightly also, Netlabel tools. Right. My introduction is a bit less impressive, but here it goes. Mexican living in Finland, previously OpenStack. A lot of PTSD about that. Now I still love OpenStack, really. And nowadays, I'm doing OpenShift and security and compliance, right? So I would also like to know a little bit about you, especially about what you know and what, hopefully how acquainted are you with this. So I'd like to know who of you have heard of Excel Linux before, hopefully? Couple of hands out there, yeah, plenty. And who of you are proficient with Excel Linux? Great. Oh, there's Dunwoj there, so that makes sense. Yeah. Yeah. Who of you have heard of Kubernetes and OpenShift? And who are you are proficient with it? A bit less hands, right there, cool. All right, well, now I get a little bit better notion about it, but so first of all, we're gonna talk about why are we giving this presentation at all a little bit? How this all started, I guess. Lukash is gonna give a quick introduction to Excel Linux and to a cool project that you all should be using. And then he's gonna talk about, well, that cool project that you all should be using that he will name in a little bit. And finally, I'm gonna talk about the couple of small projects that I did related to generating policies in OpenShift, right? So first of all, why? All right, so it all kind of started when I joined the OpenShift team. And like every other developer, you do your first deployment, you'll do your first cluster, right? And I wanted to check it out, what's going on? And one nice thing about it was that Excel Linux was important. That was a nice surprise for me because not everybody does that. So I feel pretty good about that. Started being a bit deeper and found out there's 189 pods. Hopefully people know what a pod is, but that's a collection of containers in short. And okay, that's pretty big. Out of those, there were 618 containers. All right, it's a big system, it makes sense, but there were 134 privileged containers in the deployment. That's a lot. And privileged containers to me are scary. I don't know about you, but at least I do get a little scared by them. So I wanted to figure out what's going on here and I wanted to come up with, first of all an explanation, why are people running so many privileged containers and figure out maybe if we can make that a little better? So what are privileged containers in the first place? Hopefully you'll get a couple of answers out of that. Why is Excel Linux important? Hopefully that's not a question anymore, but just in case. And why am I here? Let's start. Thank you. So now I make you Excel Linux experts in five minutes. So, Excel Linux is labeling system, right? Everything from Excel Linux point of view has a label on the system. Whatever it is, socket, file, directory, seemling, process, and a lot of others, classes. But in this talk, we will talk mainly about the processes and the files and directories on the system, right? So these labels are assigned to processes, to system resources, which I mean, directory, file, socket, and a lot of others. And all these labels are assigned to these objects and subjects by Excel Linux security policy. So how it looks like in reality for objects, they are stored in the, these labels are storing the extended attributes of file system. Here is the list of several of them, which are supports extended attributes. And here is the way how you can show the actual real label of some object on the file system. There are two ways. I would say everybody uses the second one. And if you want to see the label of Etsy Password, you can type LS minus capital Z and you see the label of the file is Password underscore file underscore T. For processes is the similar. We are using command PS, again with parameter capital Z. And for example, HTTPD processes have a Selenux label HTTPD underscore T. So what is a Selenux policy? We can say Selenux policy describes the interaction between the subjects and objects or processes and system resources, right? So it's a collection of the allow rules. Most of them are simply allow rules because everything what is not explicitly allowed is denied by default. And this was really, really quick introduction to Selenux policy and let's talk how Selenux helps containers security. So for containers in general, we have one generic Selenux policy called container underscore T. This policy protects the host files, host system from container processes. And generic container underscore T Selenux policy can only read and execute slash user files and write only to container file. Question is what is container file? Basically it's every object on the file system with a Selenux label container underscore file underscore T and the container itself has label container underscore T. So as you can see, this Selenux policy is really strict and we show you some issues with this generic Selenux policy. What I would like to also show is the list of recent CVEs of container runtime. And as you can see, every CVE in past was a breakout from container to host system. The really nice one is from February 2019 when there was a vulnerability in Ransi and Selenux blocked this attack completely. So yeah, Selenux contain them all. So what about, this was the situation when container tries to break out to host file system, right? But what about containers attacking each other? For this, we use multi-category security, which is a part of multi-level security. And here is the nice example. Let's say we have two containers running with container underscore T Selenux policy but there are also categories, right? Here are categories C1 and C2 and another container with different categories. And this is the way how we can separate two processes or containers with same Selenux type but different categories. And this container here can access only these two files, right? Because C1 and C2 is subset of categories C1 and C2 and here is no category, which means that this content and it's also subset of C1, C2. So containers can act, both of them can access only the file in the middle. But on the other slide, let's say C1 and C2 is not subset of C2 and C3. So this container cannot access this file even though there is existing allow rule saying that container T, domain container underscore T can read and write to container file underscore T. So this is the same slide with containers and container files but I would like to also mention that there is a way how you can, if you are starting some container and you want to buy them out, for example this path slash one slash leap slash MariaDB you can use parameter capital Z and container runtime we relabel the path on the host with container underscore file underscore T also with the same categories like running container has. Another option you can use normal Z as parameter which means the container runtime in this case podman will relabel it also but there will be no unique categories for this path. There will be only container underscore file underscore T with no categories which means both of these containers can access this directory. So this is the current situation and we found out there are several problems, right? The first one is that in certain cases container underscore T generic as a Linux policy is too strict. For example, Fedora's Silver Group project require access to user home directories for with read and write permission, right? Another example is a project called FluentD when FluentD needs to access container with FluentD required to access slash var slash log directory. So what you can do with the current situation, sorry, there is more slides, in some situation on the other hand, generic as a Linux policies to lose. For example, there is no SL Linux network controls. In other situation, there is no control for Linux capabilities which means that all capabilities are allowed to containers in SL Linux container generic policy. So current situation, you can start you can start a container with FluentD and you can use capital Z as parameter, as I mentioned, but what will happen? Putman will relabel all files in slash var slash log to container underscore file, which is bad because then other services cannot access this directory because the label instead of var underscore log underscore T will be container underscore file underscore T, right? And this breaks the policy. Another situation is to disable SL Linux and run it as privileged using this option, but everybody in this room knows it's bad. You know why? Security, because then we'll be sad of course. So that's the main reason, right? So what's the solution? The solution is you can write completely new SL Linux policy for that, but it's the best solution, but it's let's say too difficult, right? The system administrator has to be SL Linux expert. Another solution, you can put additional allow rules to generate container underscore T, SL Linux policy, but this is still not ideal because it means that these allow rules will apply to all containers on the system, not only to your specific one. So the best solution is a project called Uditsa. Uditsa in Slovak means phishing root. And Uditsa, we can say it's a tool for generating SL Linux security profiles for containers. So let's have example container where we are mounting slash home with permissions read and write. We are mounting slash var slash spool for read only and we're exposing only TCP port 21. But we know that generic SL Linux policy is not allowed to read or write to slash home directory. It's not allowed to read slash var slash spool. And on the other hand, it's expose all the ports, right? Because there is a no network security in SL Linux policy for containers. So what you can do, you can use three simple commands, right? So you start a container with the options I mentioned, right? Then you can inspect the container using following command. Minus L means the last started container and you can pass the output of inspect Uditsa and this parameter is name of your SL Linux policy. Uditsa will parse this inspection file and generate it SL Linux policy for it. When Uditsa is done, Uditsa recommends you to run following command where you basically load the SL Linux policies for your case to kernel. And the last step is to restart the container with one different option, which is here where you explicitly say that you want to start a new container with following SL Linux label, which is my container, not process, which was generated by Uditsa. And that's it, it works. You can see that SL Linux label for your container is really my underscore container dot process instead of generic one container underscore team. So there is a demo of this example. If you are interested, you can find it on this webpage, but we have better and new demos for you, so there was no time for that. So quickly, Uditsa under the hood. So the whole concept is based on the blog inheritance feature in SEAL language, which is the intermediate language for writing SL Linux policies. And Uditsa just or creates the policy by combining rules from predefined SEAL blocks or we can say templates. And Uditsa inspecting container JSON file and looking for three things. First one are mounts, ports and capabilities. And then Uditsa combines the blocks with the generic or with the base container, which is basically generic SL Linux policy for containers. So our example, we need a base block, which is required for every container. And as I mentioned, it's for reading and executing binaries in slash user and reading configuration files in slash at sea. Then we need another block, which is called net. And we need for network access, mainly for binding on port 21, which is labeled as FTP underscore port underscore T. And the last block, predefined block is home, which we need for reading and writing to user home directories, right? So Uditsa just inherit all the rules from these blocks and generate one new block called my underscore container and that's the name of the policy. But I didn't mention slash var slash spool situation. So there is no block predefined in Uditsa for slash var slash spool. So what happened there? That Uditsa detect all possible SL Linux labels, which could be in slash var slash spool and generate for this use case, the new block. Let's call it slash spool. And then again, all allow rules are inherit to our block, which is our final SL Linux policy, my underscore container. And basically that's it. This is how Uditsa works and how you can easily using free commands generate SL Linux policy. Then you can start or you can use this SL Linux policy with multiple runtimes, which is spotman, docker, build.ac, and last one, but about this we'll speak one. All right, so I did find Uditsa project and immediately I liked it quite a bit. So I did recommend it for OpenStack because they were using potman at the time. But I still noticed that a lot of people in Kubernetes or OpenShift did not use this, right? Even though as you saw, it's fairly simple to use. So I'm taking it actually, taking a type or a new type into using Kubernetes is actually fairly simple. I mean, you just have to specify the type that Uditsa generated for you very kindly in the SL Linux option in your bot spec. And that's it. I mean, it is that simple. So why aren't more people using it, right? And why so many previous containers, right? So I mean, I did not actually take the time to do proper research about why people are not doing it. I could do a survey and maybe I should. But out of informal conversations, at least I've noticed that people still think that SL Linux is complicated. People still think that even having an extra step to run Uditsa is a little bit too much, even though, come on, it's one command. And finally, awareness. I mean, that's one thing. And now actually, finally is that a lot of people actually don't develop in botman, you know, straight, right? They do stuff directly in Kubernetes or in OpenShift, especially if you're building an operator, especially if you're building like FluentD operator that's forwarding the logs out of your whole cluster, you don't do it straight in botman, just go directly to OpenShift. So in the sake of UX or usability, what I was thinking was like, how might we help developers generate their policies in an easier manner, right? How can we encourage people to have, to take the time, right? Or what if they don't need to take the time? What if it was as simple as annotating your bot, right? What if you could just tell your bot to generate an SELinux policy and something would magically do it for you, right? So for this, I came up with an operator. I mean, everything in OpenShift is operators nowadays. So SELinux policy helper operator. It's a little bit of a nameful, but that's what I came up with. And because I'm crazy enough, I'm gonna try a live demo of a simple hello world. Let's see how that goes. If I fail miserably, I have a recording. Let's see how that goes, right? So I'm right now in a pod, OC, oh, oh, the font. How do I do that? Like a savage here and I'm using this. All right, global shortcuts, profiles. What, this one? Got it, ControlShift plus. There you go, thank you. All right, config, use context. Right, so I'm now using somebody called good user, right? So they're not particularly privileged or anything, but they can generate bots in this development namespace, right? So we have a simple workload, right? I mean, I just took the idea out of FluentD because that's something that they're doing. But they mount from the host with an account name called hostMounter. The var log directory, right? And so let's try to create this. I mean, basically what they mounted for is that, well, in this case, this pod just writes some error information, as you can see, nothing too fancy, just some error info into a file into that directory. So let's see how this goes. Now this takes a little bit of time because the scheduler, well the scheduler needs to schedule this pod and they didn't need to get it from, get the image from the remote registry. So it's gonna pull the image, it's gonna do its thing, and then at some point it's gonna schedule our pod. There it is, finally. But this should be simple, but it isn't. We'll see logs, there you go. So I'm gonna save us some time debugging this thing and just blame SELinux for this, right? I mean, we're not, it's usually SELinux or DNS, rule of thumb. But yeah, so we don't have permissions to write to the host's VARLOG directory, right? I mean, of course we should mount it in the first place, but in this case for some reason we need to. So let's clean up. We'll see the lead, one, two, one, there you go. And let's try this again. I said, let's try this again. Kubernetes, please cooperate. Well that's great. Anyway, so what I'm gonna try instead now is to, like every other developer does, and this is the developer namespace, so it's not a big deal. But now, as you see, I'm gonna give it a privileged, I'm gonna create a privileged container. Now, it's depth namespace, so I feel comfortable doing this, but I'm gonna try to generate a policy for this, so at some point I can actually get this to production. Right, so that's the only difference here, this privileged flag right here. And there you go. Thank you, Kubernetes. So again, let's wait for Kubernetes to cooperate. Now, in the background, what's going on is that I do have this Isolinix policy helper operator, which I should rename at some point, because I'm getting tired of saying it. And it should listen for the creation of this bot and actually give us a policy in this same namespace. Right, so, Isolinix, up. So I'm gonna wait until the policy is created and that's taken a little bit because obviously I did not do this before and the image is not in any node in the cluster, so it's gonna pull the image. It's in my to-do list to make that image lighter, which shouldn't be too hard, but, oh, there it is. Yes, thank you for cooperating. And very similarly to what Luca showed us, using block inheritance, we actually do get the appropriate permissions for this container, right? So inherit from container and then inherit from the policy that allows you to read and write to the logs, right? So that's what it does. I still need to work on the formatting. This indentation is not great, but it's what I have right now. So that's it, right? Is that it? So unfortunately, it's not that easy, not quite, because, yes, we have a policy and Udit said it is job because I use that under the hood, but there's a few problems still, right? I mean, first of all, how do we install the module, right? And of course you can automate everything with Ansible because Ansible is awesome, but do you really wanna keep a playbook to install your policy every time that you generate a new one, right? So then you wanna figure out also what modules do I have installed in my cluster, right? Because maybe you install new hosts, maybe you delete hosts as well from your cluster, just Kubernetes is dynamic like that. And then how do I take those modules into use? Is it even installed there? Is it even available? And then, oh, and also who can use my modules, right? I mean, a lot of, if you install a module in your cluster, it means that any user in the cluster can actually use that SLU Next module so they could take any prior capabilities and any permissions into use. So that might not be exactly what you want. And finally, you probably wanna create, well, delete your modules as well. I mean, maybe you messed up something in your module, you gave it too much capabilities and now you refactored your whole thing and it's better. So as you can see, there's still a lot of stuff to do here. So what I came up with, again, a little bit less imaginative name is the SLU Next operator. And basically what it does is that it introduces the SLU Next policy type, which you saw a little bit earlier. It manages installing and removing your modules from your cluster. So every time that you create a new policy, that'll be automatically created in your cluster. So it's usable and everything. And there's also a webhook to validate that you are appropriately trying to use it. So if you create an SLU Next module in a namespace, only people in that namespace can use that module. So it gives a little bit of RBAC. It also makes sure that people without permissions cannot create the module. So if you can, Kubernetes gives you a really powerful tool, which is RBAC, and it allows you to create or delete or use even resources that are defining your policy in your RBAC policy. So now what really comes up with is that SLU Next policies are a first class citizen in Kubernetes. This is what's really happening. So the policy would look something like this. Just your policy name and what namespace do you want the policy, the actual policy, which is the one that I showed before. And there's also the chance that, okay, it generated a policy for you, but maybe you wanna audit it a little bit. Maybe you wanna check out that, does it really do what you think it does? So by default, it doesn't apply the policy. It doesn't install it in the cluster, but just setting out this flag will do it for you. Right, so this is what it does. So because I'm still crazy, I'm gonna continue a live demo. The demo gods haven't paid a toll yet, so let's see. Policy, you leave myself, zero two. And of course, it's gonna take a while. Plenty of time. But yeah, as you can see, the policy wasn't applied by default. I could just edit this policy, but now I'm gonna try to change namespace, head to the production namespace, and apply it there and take it into use. Right, so this is what I'm gonna try to do. As soon as Kubernetes lets me do what I want. There you go, thank you, Kubernetes. Okay, we'll see project, prod namespace. So now I have the policy, I inspected it, and would it study this job? I did not need to change anything. I could add more permissions if I wanted to, but I won't this time. So now I'm gonna create this object in the other namespace and apply it. There you go. So now we should have this available. We'll see, get a Linux policies. Hello. If Kubernetes cooperates, there you go. And you can inspect the policy. Now, the output is a little different. Oh, well, because it's a list. There you go. The output is a little different. One thing that you'll notice is that you do have a status. So you can't actually use the name of the policy straight into your Kubernetes YAML file or manifest. What I do is that I namespace it, right? So it would be the name of your policy appended to the broad namespace, well, the namespace that you created the policy. And this way we ensure that policies are unique in your cluster, right? Because name and namespace is the identifier for uniqueness in Kubernetes. And the suffix.process is something that Odisha adds. So I just have it there always, and that's the way that it works, right? So this should be installed in the cluster. So we'll see create C, SW. Oh yeah, and I did not even show you what I did. So now it's the same workload, right? So it still mounts VARLOG. It still writes the same thing. Nothing really changed, except that now I'm taking this new SELINX type that Odisha did for me. I'm taking it into use, right? It's the exact same string that the usage status gave me, right? So OC get bots, on, there you go. OC logs, no logs, because it didn't get any errors. OC RSH. And I was generating the info without being a privileged container, all thanks to Odisha and this operator, right? So it was that simple. So OC get config, OC config. Now I'm in another namespace, I'm a bad user. I wanna start reading. I know that somebody created a policy with that specific name, so I'm gonna try to read the logs from that other container, right? So error logger, error logger, broad namespace, right? The policy that should be installed in the cluster. Normally you would be able to take this into use. So OC get, yeah, of course I can check it out. Let's hope that I was clever and I did install the validating webhook. Else this is not gonna work. So OC create myself, set of five. I did install it. So now that I tried to take the policy into use, showing that a SELINX policies are now a first class member in Kubernetes, it did not allow you to use it, right? So you can only use policies that are in the namespace that you're allowed to use, right? So that's the thing that we did here. So next steps, I mean, yes, we have something working. At least RBAC is working as well. You can install your policies. You can even go and OC login queue admin. Come on, you can do it. policies, namespaces, right? And you can even view every policy that's installed in your cluster. As you can see, I have two. Only one of them is applied, but one in the dev namespace, one in the broad namespace, right? So you can actually audit what's installed in your system. Deleting these namespace will actually uninstall the policy as well. So there's a lot of stuff that's done there already. What I wanna add that's not there is the installation status, right? What if you wrote your policy and you edited it yourself and you missed one parentheses, right? Right now, you just won't be able to use the policy. So I wanna make it so that if you make an error, your, the CRD, I mean, the SLMX policy should show you the error as well. Like, hey, you messed up here, right? So that's missing. Shouldn't be too hard, I just haven't done it. Right, user facing error logger. If it was successful, it should also show you, right? The status is that your policy is installed. Little bit usability things. The user bird, right? So you might wanna go as far as saying, I only want this specific service account to be able to use this specific policy. So that's something that I still need to do. Again, not as hard, just need to get the time. And I want people to use it, right? So right now you have to download the repo to OEC deploy and whatnot, OEC create. So we need an operator hub. Or maybe the dream would be that it would be part of OpenShift, let's see. But yeah, that's the stuff that still needs to be done. Now I did show that it already works with volume mounts. Right now, it also works with ports, which is, again, thanks to Odizen. But I haven't done capabilities yet for this operator, right? So that's something that I need to do still and naming. I mean, that is a handful. If people have ideas, I would appreciate it. Please contact me, that would be great. So I just wanna say big thanks to Odizen that made it all possible. Thank you, Lucas. Honestly, this is a great project and the code base is not that big. So if you're interested in helping security, I mean, usability and security is a big thing. So please contribute to this. Both operators were generated with operator SDK. So if you're interested in writing your own operator to automate whatever you want, operator SDK is pretty awesome and easy to use. So if you wanna check out the code, I work for Red Hat, he works for Red Hat. We do open source, everything's out there. So check it out. The code repository for these is there. We will share this slide so you can actually check this out, right? So that would be it, easy peasy. So actually I was not aware that it was an hour slot. I thought it was 45 minutes like other talks. So we have 20 grueling minutes of questions. Anybody? Or do you wanna keep your day, get some 20 minutes back from your day? That's fine too. Yes, no? Maybe? Everything was really clear. We did a great job. Yep. Probably yes. All right, well, thank you for joining. Thank you.