 Hi everyone, thanks for joining. Today we are going to talk about how we can use FICO to monitor capabilities and all the work that we have done in order to expand FICO to monitor capability. So let's just introduce ourselves. I'm Stefano and I'm a security researcher at CISDIG and with me today is presenting Lorenzo Susini, which is an open source engineer at CISDIG. We are both from Italy and we are both FICO contributors. So let's start for today and let's start with talking about container isolation. So we all know container is a great way to deploy our application and transport our code. However, we also know that containers aren't security boundaries. So what makes container secure and what makes container isolation are all the layers that are around containers and in here are reported some of them. But we have also capabilities, but we have also Linux namespaces and C-group for instance. So all these layers come together to bring container isolation. Today, for today, the main topic will be capabilities, but let's just introduce also APARMOR and SECOMPS. So capabilities are a way to restrict the privileges that are permitted in containers. SECOMPS instead is Linux feature that can be used to restrict CISCOs and the argument that we can pass to CISCOs. And also APARMOR is a mandatory access control framework that can be used to detect on programs and we can enforce that specific programs can have specific access to objects. So for instance, we can say a specific program can have access or read access to, I don't know, ATC Pasco D but not to ATC Shadow. It's also worth mentioning that by default when we use Docker and we deploy a container, all these security layers are enabled by default in order to provide this privileged concept, and that's enabled by default. So we don't need to provide any particular flags or we set particular flags. It's all enabled by default. Today, we talk about how we can use FICO to, since FICO is able to have a visibility over what is going on inside our container, we can also, we extended also FICO to see capabilities and which capabilities are actually available. So I'll let Lorenzo talk about capabilities and how they work. Yeah, thanks a lot Stefano for introducing me and let's now jump into the main topic. And well, as we all know, traditionally we use to distinguish Linux processes into two different categories, which are privileged processes and privileged ones. So we can say that this model is binary and it can be risky from a security perspective. This is because basically privileged processes that are those running with effective user ID equal to zero can basically bypass all kernel permission checks, as we all know. And on the other hand, privileged processes instead are those subject to all kernel permission checks, and most of the time this is done using their own effective user ID or group ID. And from an attacker standpoint, this model is exploitable, of course, by attacking directly root processes. And once the root process is compromised, the whole system is at risk of being compromised. For this reason, to improve its security model, Linux has implemented capabilities. And capabilities, we are a clear example of applying and implementing the principle of this privilege. And in fact, as the model states, the power of the superuser is split into distinct units that can be independently granted or revoked. And these allow us to run processes with the minimal set of privileges that they need. And hopefully an attacker gaining control on these processes cannot compromise fully the system with the same ease as before. And the classic example about this is the one of a web server, for instance. And before the introduction of capabilities, a web server needed to run as root only because it had to open a privileged part that is a port below 1024. With capabilities, we can map these specific privilege to the capability, which is, in this case, kept net bind service. And in the meanwhile, we can let it not bypass checks, for instance, on the file system, which is translated into a completely different capabilities, which is, in this case, kept that cover right. So since you are here to implement proper monitoring of capabilities with the purpose of detecting attacks, our take on this is that not all capabilities are the same in terms of privileges they allow. In fact, some of them can be defined as full root equivalent if gained by an attacker. And this is because if an attacker manages to gain one of them, he can possibly gain all of them with other successful attacks. And as we will show later. So let's go on and to implement monitoring for capabilities, we have to better understand how they work. So basically, capabilities are usually divided into three main sets, which are the effective, the permitted and inheritable set. And we can see that the effective set in a certain sense reflects a little bit the concept of the effective user ID. And in fact, this is the set that the kernel uses to perform actual permission check. On the other hand, we have the permitted set, which is basically the super set of all capabilities a process may ever acquire. And we can move capabilities from the effective set to the permitted set to let the effective set have only the privileges needed at a specific point in time. And last but not least, the inheritable set instead is used to control which capabilities are inherited upon exactly a system call, which is a pretty relevant example. So the inheritable set is used to perform specific operations by the kernel to compute the new permitted set, as you can see in the next slide. And yeah, so this is how capabilities change is upon exact here. And what I want to stress here is that the kernel use specific set of rules to change capabilities of a process. And following the previous reasoning, we have to say that the effective set should start empty when exactly is performed. This is because we want processes to manually move with syscalls capabilities from the permitted set to the effective one. However, the kernel also offered backward compatibilities for those binaries that can be defined as capability down, meaning that they are not able to move capabilities in between sets. And for doing that, the file effective sets, the file effective flag, sorry, can be used. And this is basically used to move the new permitted set to the effective set so that the process doesn't have to move capabilities using other syscalls. So now let's go back to container security and see how capabilities apply to it. And in fact, when talking about container security, we basically have two different possibilities to let our workload run securely. So the first one is to avoid container running less route. I think everybody here knows about this, because basically containers like Stefan was saying before, are not the security in boundary like VMs. And if you're not using user namespaces, routing the container is also root on the host. But sometimes we want our containers to run privileges or privileged operations. And we may be tempted to run them as route. So what can we do? We can use capabilities. As we said, capabilities on the other hand have the possibility to selectively choose the privileges that we want. Anyway, we think that since capabilities are a lot, and sometimes it's really difficult to understand which capabilities our workload needs, we may end up in certain cases running excessively privileged containers as well. And this is for sure not good from a security standpoint. And Stefan will now show you why and how capabilities can be used in real attacks. So thanks. Let's now dive into attacks on container escaping. I just reported this slide just to say basically what we have said before. So containers aren't isolated by default. And in this case, we all know if we deploy a container as privileged, this is far to be isolated. So of course, we are still using container, we are still using Docker, but this is far to be an isolated environment. This because if we use capabilities and we see since we are talking about capabilities with privileged, all the capabilities are available and all the capabilities can be used and loaded by a possible attack. So that just to mention that when we deploy a container as privileged, we need to be careful that this is needed. And we need to be super careful on what we do. So let's start now really talk about container escaping attacks. In this case, let's start with this technique with the CIS module capability. So CIS module is used to load and unload kernel module. And for this technique, we have the only requirement that we have is having the CIS module capability. And how it works is pretty simple. So let's say that an attacker is able to get an initial foothold in the container and find out that the CIS module capability is available and can be used. So the attacker just need to drop the module inside the container. And since the module is loaded in the kernel, all the malicious code is going to be executed in the host. So in this example, once we load the malicious module, the bash reversal is going to be executed. And so the connection will be open from the containers host back to the back to attacker machine. It's also worth mentioning that the attacker might need to compile the module inside the container because just to be sure that the module is going to run. And for this, the attacker might need the Linux header installed in the host. Otherwise, it's not possible to run this attack. Let's just move on with another technique. And in this case, we use the capability ptrace. In particular, ptrace is used to control other processes. But also, with ptrace, we can also modify the process memory to inject arbitrary code. And we can also modify the pointer extraction in order to run this arbitrary code. So in other words, with ptrace, we can inject malicious code and actually execute it. So it's pretty powerful capability. So in this case, for this technique, we have three main requirements. So first one is, of course, having the sys ptrace available. The second one is we need to have a part more set as unconfined or allowing ptrace. And the third one is the container and the host needs to share the pid address space. So as I said before, let's say an attacker has access to the container and was able to get initial access into the container and find out that the ptrace capability is available. So in this case, the attacker just need to drop its own injector with the malicious shell code and find a process that he wants to inject and just execute the shell code. In this case, as we said before, since the address space is shared between the container and the host, the malicious shell code is going to be executed in the host. And that's why, as we said before, if you open a reversal, the reversal is going to start the connection from the container's host back to the attacker machine. And that's another attack, but using a different capability. I'll just close with the last example for today. So this attack is a well-known attack for container escaping. In this case, it was published on 2019 on Twitter. In this case, the first example that is reported here on the left required a privileged container run as privileged. After that, a new exploit came out. And in this case, we just need the sysadmin capability in order to be successful. For this specific attack, we have three requirements. So sysadmin is one, a parmer with unconfined or just allowing mount operation. And also, we need to have root user inside the container. So to understand how this work, we just need to understand how notify or release works. And this is pretty easy. So notify or release is a feature in the cgroup v1, version one. And basically, if it's enabled, so it's set to one, once a container, a process terminates, the kernel is going to execute the code inside the release agent file. So whatever code is inside the release agent is going to be executed by the kernel. And that's basically how the exploit works. So just going through the code, the exploit just create a directory. A directory is mounting the cgroup controller inside this directory with the option RDMA. And also, it's creating the cgroup directory inside it. Also, as we can see, the notify or release is set to one. As we said, it's just to enable the feature. And then all the other commands is done to create the release agent with the malicious command that we want to execute. In this case, it's just getting the path from the overlays and the command that we want to execute. So in this case, be in SH, so we are opening a shell in the host. And the last command is just used to trigger the exploit running an eco command. And that's it. So once the eco command is going to terminate, notify or release is enabled, it's going to trigger the workflow and executing the malicious command that is inside the release agent. So just to recap what we have said so far. So we have these three techniques. All the three techniques that we have presented are using three different capabilities. And so now we let's see what we did in FileCo in order to detect this malicious behavior and what we can do with FileCo for monitoring the capabilities. Yeah. Thanks, Stefano, for going into that with such examples. And so now that we are aware about talk of abilities work and how they can be used in real attacks, we want to show you how to monitor them using FileCo. So for those of you who doesn't know about FileCo, FileCo is an open source project for runtime security. And it became the defect of standard for Kubernetes threat detection. It was originally created by sysdig and then donated to the CNCF. And it's currently at inquisition level. And we both work for it to become better and better. And we always try to improve its detection capabilities. So another thing that it's exciting about FileCo is its vibrant community. And I think that it's really nice to work in such an environment because people are really active and contributing. And hopefully this talk that shows an extension that we made to FileCo will encourage you to do the same and improve it and make it become better and better. So before we dig into the details on how we extended FileCo to monitor capabilities, let me give everybody a little overview about it. We can say that FileCo is powered by its kernel module and it's in VPF probe to collect system calls data directly from the kernel, which is considered the source of truth for our purposes. So both of the event collection components, user ring buffer to push events that are uninflated and used in user space in our libraries. What I would like to stress most about this is that there is a part in one of our libraries that is responsible for maintaining the state of the system. And this part is of course in LibSIMS as you can see. And what we do with this part of state is making it accessible by conditions and rules and making it able to write rules and condition able to spot new kind of attacks. So with this in mind, we realize that FileCo was only able to see in black and white, meaning that it was only able to discriminate and distinguish root and root processes. And we wanted to put some light in that gray area that capabilities have created in between. So to summarize the part of the work that we have done, we can do it explaining the show main steps. So basically what we did is that when FileCo starts, it needs to be the state of the word that is the actual system state at a given point in time that is when FileCo starts. And most of the information used to do this comes from the Procfile system. And fortunately, there is a file in the Procfile system which lets us retrieve information about all capabilities of process. And we use this to populate the initial state. So once this is built, FileCo starts capturing system calls and keep updating it using system called data. So what we did is enumerating all the possible system calls that may interact with a process and modifying its own capabilities. And we started with the fork system call, which was the easiest one, because basically we only have to copy the capabilities from the parent process to the child. And almost the same goes for the clone system calls, except for those clones that creates new namespaces. And moreover, to push to use a space, the new capabilities are also in the exec file. We modified the event of exec file to push information about capabilities also in that case. And last but not least, we have also the possibility to move the capabilities in between sets, as we already know. And this is usually done with the offset system calls. And to do so, we implemented the monitoring for that system calls from scratch. So in this way, we are able to let rules extract capabilities from the state and we can write some interesting rules to spot attacks based on capabilities. And now, Stefan, we'll show you what kind of rules we came up with and hopefully it is interesting to see it together. So thanks. And thanks to the Lawrence work on FICO, we are now able to use capabilities inside FICO rules. This is a rule that we created in order to control and detect if someone deploy a new container with excessive capabilities. So as we can see here, just a word on how FICO rules work. So the FICO rules are loaded inside the FICO engine and once a new system is coming, FICO engine is able to check and merge all the rules and checking if a rule match with the CISCO. FICO is able to trigger a security alert. And that's how it works. So let's see the condition here. So the rule is going to trigger when we create a new container with excessive capabilities and the macro shows all the capabilities that we are monitoring in this case. And as you can see, there are a CIS module and CISP trace. There are the two capabilities that we have talked before. And in the bottom, we can see the output from FICO. And also, we have also the other rule that we created. This is specific for the use case that we talked before about the release agent file. So as we discussed before, the attacker needs to modify the release agent file in order to inject the code that he wants to execute. And that's mainly where we create the detection. So once the release agent file is open to write commands and also the container has the CIS admin capability and the user is root. We are going to write a security alert. And in the bottom, we see the output mentioning all the capabilities available. So we can see the CIS admin is one of them. And in this way, we are able to detect with very low false positive the potential malicious behavior. So having said that, this is it from my side. I'll let Lorenzo close the session and start the Q&A. Yeah, that was basically it. I hope you have enjoyed this talk. And if you have some questions, feel free to reach out now or in the Falco Kubernetes Slack. And that's it. I hope you have enjoyed this talk and you will use capabilities or try to improve routes and hopefully contribute to be able to improve Falco always. Thanks a lot for being here. Thank you. So if you have any, someone has any questions? Yeah, if you want to use Falco in your cluster, there are charts to do that. And you can use, they are officially provided by the Falco security organization. Yeah, when you wanted to use in Kubernetes, it can be used as containers. And, you know, the M charts will spawn pods on each node in the cluster as a demo set. And that's it, basically. You can use this to deploy Falco. Or Falco is also deployable on a node. And you can use our packages to do that. Yeah. Thank you. Maybe another question for Falco. We see all the rule configurations that go in the YAML configuration for the cluster. Does Falco look also into those type of aspects? Do we consider that any configuration that occurs is treated as an event and then Falco will see and analyze or these are two separate? No, no, this is just a different thing because Falco, it's runtime security. It only, it doesn't look about configuration in the YAML. It only looks at runtime if a process is being executed with a specific capability or not. And I don't know if this answers your question. Well, these are configurations that can happen at runtime. So it's still part of the runtime. Whether I change, I do a change in my environment, is this going to be captured or not? Yeah. If you spawn another pod with different capabilities, it will be captured at runtime by Falco. Thank you. Yeah. So if I'm able now to capture at runtime also those type of configuration, do I have an overlap or do I always need to have both of them because there are configuration types that cannot be captured by the event triggering? I'm looking if I'm allowing Falco to be able to detect all the configurations at runtime that happen through this YAML configuration. Do I have a duplication now? I have the rule engines that detect that. I have also a Falco that is looking from a logging perspective to the same thing. Do I have a duplication? Can I focus only on one of them? Okay. So there will be like, treat it as different Falco will have. Okay. Or maybe there is an overlap, but not big enough to separate. There is no overlap. Okay. Okay. Thank you. Any other questions? Yeah. Sorry. I was just looking at the rule. I see that you're actually ignoring Falco because you're running as a privilege. Is that correct? The privilege rule? Yeah. It says not Falco privileged. Why is that? Sorry. This one, right? That one. Yes. A condition. It says and not Falco privileged container. Yeah. Other macros are used to reduce false positive. I haven't reported all the macros because it's hard to fit all in the slide. But there are other macros that are reported to reduce false positive in environments sometimes. So usually macros are used to simplify the overall condition to make it more readable. And sometimes use macros to reduce false positive or whitelist some use cases that we know is going to trigger false positive. And that's one of those cases. But in general, we can use macros for those reasons. Yeah. You are saying if it is possible to deploy Falco without privileges, Falco, it's a specific unique workload because it has to inspect the system. So it has to have some privileges, of course. And we have in the album charts, we try to run our BPF probe with these privileges. And there is an option to do that. But in general, it has to have some privileges. I think that answers your question. So they are ignoring it because it is already privileged. And if they select, then it's a false positive alert and raising for the Falco itself. This was used to avoid false positive on it. There may not be an option to run without privilege mode. That's what he said. It needs some privileges to run. Because we are securing Falco, I guess. I mean, this is a security tool, right? So it needs to have visibility over what is going on. Otherwise, it's not going to work. It's a security tool and we need to have visibility over what is going on. So that's the main point of all of this, right? So of course, there are ways to, as we said, with the BPF to reduce this kind of visibility for Falco. But this is mandatory for the type of tool that we are using and to have visibility. So I guess, I mean, I think most of other tools are basically using the same way. Since I mean, it's not possible to do any way in other ways. So of course, we can reduce it. But at the end is something that we need in order to have visibility. And of course, we want to have visibility in order to provide the best detection possible. So I think that's the point. Good. Thank you. Thank you. So thanks, everyone. And I wish a nice rest of your day. Thank you.