 Hello everybody. Welcome to Valencia. Thank you for joining the stock. My name is Shea and today we're going to talk about Falco by passing Falco default rule set and perhaps even how to compromise a cluster without shipping the stock. Let's start. We'll start with the background what Falco is for and what's container security is about. And this is my view of container security. You can sort of divide it into four areas for security, cluster security, pre-deployment and post- deployment. And of course, host security is important because if host is compromised then it's really hard to reason about the workload security that runs on this host. Pre-deployment is about building the base images in an efficient and secure way, container and scanning, although some people call it a little bit scanning, but I like to be more precise. Scan for exposed secrets, static compliance, basically everything that we can do with the static image file of the container. Cluster security is all about Kubernetes config and then there's a post-deployment and that's where Falco is coming into the picture, runtime detection. There's also runtime protection, dynamic appliance and prevention is about existing, mostly about existing Linux security, kernel security mechanism that sometimes given for free in the container context. I've recently listened to the cloud security podcast and they suggested a bit different division into areas of container security. So they were talking about build infrastructure and runtime and that's sort of the way this maps on onto this view. Intra is cluster security and host security to build is pre-deployment and the runtime is actually post-deployment. So if that helps to think about it as well. And we can see that where the Falco belongs to, runtime detection through the Falco architecture. Falco is built on top of either kernel module or EBPF sensors running in the kernel space. The events, mostly syscalls and enriched syscalls are bubbling through the ring buffer up to the user space where a bunch of Falco libraries and most importantly rule engine are taking them, processing them and decide whether the certain rules that are expressed as YAML files, whether the rule should trigger given that event or shouldn't trigger. And the if rule is triggered then one can integrate that with the typical cloud stuff like CM or event in some kind of event engine or gRPC. But for our purpose we're going to use STD out today for the sake of simplicity and you'll see that in the demo. Typical Falco rule, I took in this case, this allowed the session connection. There is a concept of macro in Falco rule set. In this case it's called inbound outbound. This is really good concept because it prevents the rule file from bloating. Otherwise it would be way bigger than this because the macro can be used by different rules. In this case, this allowed the session connection uses three macros inbound outbound sysh port allowed sysh host and one of them is actually has and not condition which is the opposite of true. So if allowed sysh host is false then this rule has a chance of triggering. And then there's a priority mitre tag, et cetera, kind of metadata description. So all good. Now Falco rules in addition to behavioral detections, they can also be tailored to specific to detect specific exploitation attempts of the CVs if they are constructed properly. And this is the example to those four, but there are more right now. There's just four interesting ones that I thought to mention. Now let's talk about previous work on Falco bypasses. This is not the first talk about it. Before that NCC group did some research as well. They mostly described the bypasses around the image naming. Leonardo talked about this in one of his talks about the twin sys calls. And I think he also mentioned the similar evasion. There is an ongoing issue about the sister calls, missing sister calls, for example, open, open net, execute, execute, add, dupe, dupe, two, this kind of stuff. And then the recent work by Goa and Zen on Defcon, they were talking about talktow of file names mostly. So classic example for the, in the open net sys call the time of the check of the file name and the time of actual resolution of the file name are different. So that keeps the possibility of the evasion. And then they talked quickly about the similar invasions. All right. So now with that in mind, let's go into the interesting section. Let's start with the read sensitive file and trusted rule. We're going to use it a lot. And let's start with something simple, similar evasion. This is the typical way the Falka is used. We spin container, we do something malicious within the container, and Falka is spitting this output, warning sensitive files open because we're trying to read the it's a shadow. How can we prevent Falka from knowing that we did this? Well, let's create the same link to it's a shadow. And we do see that there's notice this Falka still saying something, but that's a different rule, right? Because the previous rule was warning level and right now we're notice. So that means that it's at least degradation of the detection. And this, by the way, this rule, the create, the monitoring of the creation of the of the simulings over the sensitive files was one of the protection mechanisms against the talk to attack. We'll see why this is not enough because even though we have degradation of detection, which is nice, but we can easier get rid of this rule as well. What can we do? We can create the simulings to the unmonitored directory. In this case, it's a security. Then we're using the cat to cat the relative path based on the security link. So one step above and then shadow. And we see the output of the shadow and Falka is silent because it's confused and the simulink is not resolved in Falka's context. What else can we do? Let's take a look at right below it's right below root files. In this case, we see the example of how they can be triggered. So we're doing, we're creating the symbolic link not to the root directory, but we call it the root link and then using it. And of course, it's that easy to bypass this rule as well just by using the sim link to the directory and then using it for the relative path. Okay, what about hard links? Yes, as well, we can use them. Hard link is a direct sort of point or to the I node. And in that case, Falka until version 0.31 where we actually committed this, the new rule for the hard links, it wasn't able to detect it as well. Now there's a special case of pseudo potential privilege escalation rule. This case is kind of reverse. So let's see what we're talking about here. There's a, the last year there was a infamous CV in pseudo, which allowed attacker to pretty easily escalate the privileges from the regular user to root. So when I was trying the typical POC by Warwick, I just downloaded it and so and try to see if that works or not. And it worked very, very easily on Ubuntu. I think that was 1804 and Falka was silent. So I got interested in what happened there. And what a stress S trace showed me was that the exactly was actually running the user bin pseudo and not the pseudo edit that the rule would expect. You can see here prokname equals pseudo edit rather than pseudo. And what happens here is that Ubuntu, in Ubuntu, actually pseudo edit is the sim link to the pseudo. So what the rule, the way it was constructed, what was expecting, the rule was expecting the sim link to the binary and it was getting the binary. And of course the exploit was still working and the rule wasn't detecting. So that was one of the fixes that we, that we committed in the version 0.31 as well. But this, this is interesting, kind of the other way around of this typical sim link invasion that you showed before. All right, enough of the sim links, let's talk about prokname. Prokname is the name of the process and many, many Falka rules rely on it a lot. So typical construct here lists, there's a list of login binaries names and management binaries names. And these lists are used in the rule called treat sensitive file entrusted, the same rule that we're looking at, we're using for our examples. And you can see here the construct and not that I was talking about, that's the, I call it accepted construct, construct where the rule doesn't trigger. So all we need to ensure here is our process that will read the sensitive file will be named as one of the user management binaries. So what we do here, we, we write the very simple C utility, the treats file, and we compile it into system D log in D and call it system D log in D and using it to dump the shadow. And of course it's fine because Falka thinks that system D log in D is the name of that one of the user management binaries and the rule doesn't trigger. What else can we do? The thing is that this approach is not scalable, we can't write utility for, for every, every utility CLI command that we want to use. So but we can't do, can't create the symbolic link. And so, and call it system D log in D, like, like in this case, and use that symbolic link to dump the shadow and again, Falka is silent in this case. So again, symbolic link, but with the proc.name perspective. Eventually, simple renaming also works. If we can rename cat into the system D log in D, there we go. We, we have a, we have a perfect invasion here. What about parent and assets trust names? Yeah, so that's also a problem because in this case, for example, when we're looking at the condition where proc.name should be something and proc.pname should be something, what we can do, we can come up with a script and call it password, one of the accepted binary names and the script, the parent name will call the, the proc.name and that's how we sort of build this condition. And once that works, we can, we can easy dump it's a shadow without being detected. But we can make this evasion even more generic. What we have in the Fulver latest image, and you'll see this image a lot. This is the publicly available image that they created with various goodies to bypass the, to bypass Falka. This image creates one of the Fulver's Fulver fake parents. So that binary, that small utility, it knows how to fork a series of processes with the depth in this case depth of three. And it calls the ancestor, the first initial process is a Google account for solve the rest and then runs this command. And what it does, it basically tailor specifically this condition because the proc.name of three, it indeed starts and equal to Google accounts. And that's why the Falka is silent in this case. We can use this utility to build the nest in a name as answered our ancestors as, as we want. Reversal detection. This is also very interesting. This is a typical reversal payload that creates initiates reversal. And indeed it has, it causes for three detections here. So of course, the typical attacker can't, can't use it. What can we use? We can try to use make note and then type it into the ANSI. This is also very, very known one liner for initiating the reversal. And however, it's still, it's still shows detection. However, this detection is where you're very easy to, to avoid because to evade, sorry, because we can use either Simlink or proc.name evasion that we already talked about, but we can, we can do something more interesting. How about we use the MSF Venom? And for people who know MSF Venom, they know that this is the de facto payload generator. It's, it's a part of metasploit framework, right? And it's very easy to, to generate any type of payload. In this case, I'm generating reverse TCP payload for Linux base 64, copying it into the container, marking it as elf file. And there we go. We see the connection on the host's name and Falco is silent. So indeed this is, this is a tough problem to detect the payload execution as an elf, where it doesn't have a hash, a known hash, but, but Falco even not a, not a antivirus, right? So it's, it's a tough problem to detect this type of behavior. Perhaps this could, this should be detected on a network level though. What else do we have? We can talk about comment arguments manipulation. Netcat remote code execution in container, you can see that it tries to detect the indication of NC or NCAT along with a bunch of flags. So for NC, for example, there are two execution flags dash E and dash C that allow you to execute bash, for example, or other program upon the connection. And that's what this rule is trying to avoid. However, we know that Linux old and bigly, big Linux utilities, they have so many flag options that it's really easy, really hard to come up with the rule that will, will block all the potential flag combination. That's indeed the case. In this case, when we, we're using NC with dash E with classic dash E bin bash, it's of course being detected by the Netcat runs inside container rule. However, if we just add dash V as in verbosity mode, then the syntactical comparison fails here because the proc arcs doesn't contain dash E at this point. And then there we go. The Netcat runs inside doesn't trigger anymore. All right. What else can we do with the proc arcs? Um, we can take a look at the grab private keys rule. So it tries to block the find when any search contains IDRSA or IDDSA. And for people who know fine, know how much, how many options does it have and what kind of reach functionality does it have. And of course, with dash regex, it's, it's very trivial to bypass. We just substituting the R with, with a dot, but we can do the same with any other letter. And there we go. And there we go. We don't have syntactical comparison of IDRSA at this point. And the last interesting rule that I'm going to show bypass is the sensitive mount bypass. So the launch container with sensitive mount rule relies on this condition. And I don't know if you see the problem here. The problem here is that this, this comparison is very specific. What the attacker can do, they can mount var run, for example, or var, and then they use subdirectory path to refer to the docker. And I will use it during the full simulation attack later. But in this case, this is very easy to show. We just run in the docker with dash regex, var run, var run. And there we go. And we do have a shell response rule, but not the mounting rule that should have been triggered. And I think for the sake of time, we, I'm going to skip the crypto mining detection bypass. You can, and the privilege container detection bypass, although this one is interesting, but you can, you can read about this in the, in the GitHub director of the project later if you want, if you're interested. But let's jump to the tool bit, our tool bit so far contained two images, for the latest. And that's the source. You can see that it can, it has the docker client, it has kubectl client, and it has a bunch of network utilities, gbash, runc, and I tweak the names of those clients and bash so that Falco won't be able to detect them with the static, with the traditional rules. And then I have the CV 20, 21, 3, 1, 5, 6 images just to, to simulate that specific rule trigger that, that actually didn't, it didn't work. And then, but now it works after our changes. In case somebody wants to play with this. And finally, let's go to the interesting part. Let's try to put together all those evasions and see if we can perform the full attack simulation. Okay, so the demo setup is I was running it within the GKE cluster with the cluster nodes and single load cluster. One word about the cluster that they used is it was presented in the KubeCon 2019, two years ago. It was really cool. Just a testing cluster that shows how to just show some attack techniques and defense technique. And highly recommended. Pretty interesting one. And that works perfectly for our case. So it has two attack scenarios. Scenario one and scenario two. And scenario one is more basic with a more basic attacker and scenario two is with more advanced attacker. So let's go and start with the detection, the detections of the scenario one. Just, just we'll see what, what we expect from Falco to detect within the scenario. So the all scenarios start with the attacker given the web shell assumes that pod is already compromised. And, and then the attacker starts all kind of reconnaissance enumeration from within the pod. It runs the end, it runs the unnamed sR and then it sees, sorry, they see that the pod, the host is part of the Kubernetes pods. And there we go, we have the Kubernetes version, et cetera. We have all the environment variables. We have some kind of service account mounted here. And then at this point attacker tries to go to the typical way of compromising the, the cluster. So the, the download the cube cattle, they try to use it to get all the resources, whatever they can, and there was a bunch of resources. So it takes some time. And there we go. We have the attackers can see the pods. And then attacker uses AuthKanai to enumerate the, the permissions that the current pod has with the amounted service account. And apparently the attacker can't even create pods. Right. And that's what the attacker does. They create the, the malicious Bitcoin error pod that of course will mine Bitcoin. And we can see that this Bitcoin is an error is created successfully. But all this time we see that Falco is successfully detecting various malicious activities right here. So this will, for the socket, will be really hard to not to notice any malicious activity in this case. And here I'm using the lint piece just to demonstrate if, for those of you who know lint piece is a very known into Linux enumeration and privileges case escalation tool. And it's very popular. So if, if typical attacker will, will run it on the host, of course nothing will happen, but if they run it on the, on the falco monitor or host or container, then, then that's a problem because Falco detects a bunch of sketchy activities here. And of course, SOC will not stay silent in this case. Okay, let's go scenario two detection scenario two is a bit more evolved. The attacker is more advanced after the, so after the scenario one is finished, the, the defense actually detected easily that caught and they killed it. And so scenario two was able to get the same web shell, but now attacker is more advanced and they want to do something sneakier. And they're using this one liner. This one liner is very interesting because it smashes the hosped name space amounts the same process name space and launch, launches the privilege privilege pod. And at this point container has the access to host resources. You can see that root is the root of host at this point. So I'll skip ahead because this right in the attacker has access to the Docker site. And what the tiger does, they deploy the Bitcoin error this time on the Docker through the direct Docker API rather than the Kubernetes API. And that's a bit sneakier because then the defense will, for the defense, it will be harder to find that Bitcoin error. And in addition, what the attacker does, they want to, they think about the future. So for the persistent, they will, they find the cubelet token mounted on the host and use this token to actually, this is the token and they use the token to mount. So that's the get secrets commands. So yes, attacker can find any get secrets. And this is the mount of the, and this is the creation of the node port service for the future access. So attacker at this point is able to open the port on a note. And in the future access this port, this port if needed. And we still see that focus very active at the detection. So again, it will be quite easy for Sock to detect that something malicious is going on, on the, on the particular cluster. So put in the attacker's head. Now we're trying to avoid Falco as hard as possible. We are doing the same scenario to actions, but with the mind that we don't want Falco rules to trigger. So what we do here, we're still copying the kubectl, however, we're using our name evasion and we renaming the kubectl as kcattle. And from now on, the Falco rules that detects the kubectl activation within the container will not work. And we can easily use the kcattle from this point. And so the one liner that I am using, it's a bit trickier here. It's not privileged container at this point. We're still smashing the hostpad if we can. And I'm running the Fiber Latest, which is the Falco ultimate bypasser image. And as you saw already, it has gbash, which is different from the bash. So that rule will not trigger and we're mounting var run to get the access to the Docker socket without triggering the mount rule. And we are running the gdocker ps command. And let's see if we have the access to the host level Docker socket. Yes, we do. We can see that there's a bunch of kubernetes control plane images running here, containers running here. And we do have access to that. And that's great. So at this point, we can run commands. I'm what I'm doing right now. I am running the gdocker. I'm actually deploying the Bitcoin error on the Docker. And you can see that Falco is a bit active so far. It just had several three detections at this point, but it's all the same rule. And this level is notice and expected connection to kubernetes API. And I'm plowing through using the same one liner. This time, I'm looking for the for the kubelet through the kubelet mounted service account token. I'm looking for the one that will be able to open that to create that node port service and to open the node port. By using this token, we can see that it finally can do whatever we want pretty much. So now we have confidence we can mount that we can create the node port service. And that's what we do. Node port service. The name is Istio management, which is a sneaky way to pretend to be one of the Istio services. But for our purpose, what's interesting here is that Falco still only detects the notice and expected connection to kubernetes API. So in this case, if the suck is geared towards detection of the warnings or errors or up, then they will not see the notice. Plus they might think that this is some kind of false positive. That's some kind of rule that triggers excessive amounts of false positive and decide to disable this rule if they do see this rule. But let's see if we can do even better than that. All right. So the difference here, this we're still talking about the second scenario. However, I'm modifying the command a bit to mount varlib as well. The mount command will still not trigger. I'm still using Fulberlatest. I'm still smashing hostpid. However, I'm performing the gbash command. So now I'm supposed to get the bashell into the container. It's not privileged container, but it has the access to the docker and the varlib and the cubelet tokens for the future. We have one detection so far. We have two detections so far, but we are root within the container. And we can pull the Fulberlatest again. And the more important that we have the access to the docker sock. So the neat trick that we're doing right here, we're tagging the Fulberlatest as a sysdick container. And from this point and on, Falcott thinks that sysdick, that Fulberlatest is actually named sysdick image. So we will be able to bypass that unexpected connection to Kubernetes API server because the sysdick is one of the containers that allow to make those calls. And as I called it, one of the accepted conditions in that rule. So let's see that we indeed do that. So at this point, I'm running an additional container from within the sysdick container, which is not really sysdick. I call it sysdick, but it's not. It's still Fulberlatest. We can see this in the last on the temp folder. It has all those utilities that we put there before. However, right now, from this point and on, Falcott thinks that we're in the sysdick container. So it will excuse this container from accessing the API, Kubernetes API. And at this point, we can do whatever we want. We can rummage through the Kubelet tokens, get in the tokens, and this command is important. I want to list my permissions, whatever I can do, from within the nested container. I'm using the insecure skip TLS course. I'm using the GKube Cattle so that Falcott will not detect that I'm using actual Kube Cattle. I'm using the server from the Kubernetes server host from the parent container. And I'm using the cache here. And cache here is important here. I want to set it as temp because otherwise the Kube Cattle will save the cache in the root slash dot cube directory, which would trigger the Falco otherwise. And we can see that we do have the permissions to do whatever we want at this point. So at this point, we can create node port and whatever we want. And still, we only have two triggers of unexpected connection. So there we go. We have only two triggers for the full cluster compromise, including the persistent access for the future, hopefully. Can we do better than that? I'll leave it as a homework. So for the fixes, if you're interested in the full project with all the details, those one-liners, you can find them in the Blackberry GitHub. For the change lock, we had a series of commits on January 25th that went into the 0.31 version. Discussion and recommendations. Well, very quickly, and we can discuss it more in depth during the Q&A section, I would say that hooking points are very important because we do want the sim links, for example, to get resolved as close as possible to the kernel C to get that kernel picture. No easy way to prevent attacker from bypassing the rules to rely on a prockname and prockname. And filename, yes, that thing needs to be minimized. Too many a rule includes construct and not. Because every exception, I understand that that's too minimized false positives, but that's also the entry for the attacker and possibility for the attacker to evade that rule. Review rule priorities in the bypass context, purely check of public exploits for the CVE-specific rules on the example of that to the privileged exploit that I showed, and then eventually encourage clients to develop their own private rules because then the attacker needs to know the ruleset source before doing any action to understand how to evade the customer-level rules before starting the app. And that's all I had for today. Thank you so much, and I'll be happy to take your questions.