 Hello everyone. Let's start. Thanks for joining my talk and on this last day of KubeCon. My name is Magno Logan and I'll talk about Kubernetes post-exploitation. Also, thank you for the organizers for having me and for everyone to stay until the last day. So let's start. Okay, so just a little bit about myself. My name is Magno Logan. I'm originally from Brazil. I've been in Canada for over five years now. I'm part of the cloud and container security research team at Trend Micro, where we focus on cloud, Kubernetes, Docker, and container research. This is the QR code for my LinkedIn, the best way to reach out to me. I'm also instructor at Go Hacking, where we provide cybersecurity trainings, secure coding, DevSecOps, pen testing, and things like that. I have a background as a developer and I've been doing AppSec for the past 12 years. So that's that. Okay. So this is the agenda that I have for this talk for today. First, I'm going to give you a little bit of introduction about the MITRE attack for containers and also the Microsoft threat matrix for Kubernetes. Then we're going to go and do an overview of a Kubernetes attack, one that we posted, we blogged about one year, two years ago, of attackers compromising Kubernetes through the Kubelet API. Then we're going to talk about the sidecar pattern and kind of a technique to compromise Kubernetes clusters through the sidecar container injection, which is not very well known. I think so. And also at the end, we're going to cover some mitigations and some tools that you can leverage to detect these scenarios or even prevent them from happening in the future or in your own environment. Okay. Before I start, the kind of I always like to share some of the projects or tools that I've or blogs that I've created. This one, for example, I started back in 2020 when I started doing some Kubernetes research and I kind of was saving all the links and reference materials in a text file. And I decided to then create this GitHub project, the awesome Kubernetes security list. You're probably seeing similar projects already for different topics. So this one has a bunch of content links for blogs, videos, PDFs and all that stuff, not just for Kubernetes security, but also from the basics, right? There's no way for you to understand Kubernetes security without understanding how Kubernetes works. So it talks about the basic stuff as well. And this last one, just the last thing here, this is the latest article that we've published around Kubernetes threat modeling as well. We do a deep dive on the components and analyze how someone would kind of evaluate the components of a threat off a Kubernetes cluster and be able to either like prevent those issues and attacks from happening. Okay. So if you haven't heard about the MITRE attack framework, if you are in cybersecurity, you probably have heard about it, but it's like a framework of techniques that attackers leverage in the wild. So how actors are compromising different environments. And we have like a series of matrices for different environments such as enterprise, cloud, and now we also have one for containers. So I'm going to give you a little bit of timeline overview of how this MITRE framework for containers was created. The MITRE attack framework is also used as a foundation for the development of specific threat models and it's open and available for any organization and it's free. So back at the end of 2020, MITRE had had reached out to the community. They posted a blog reaching out and asking for help saying that we want to create this MITRE attack for containers and we're looking for information for contribution from the community, from other organizations on how can we, what are the main tactics and techniques that attackers are leveraging in the wild, right? And at this time, we already had some data and information and blogs that we publish based on our Honepots that we deployed. So Docker and Kubernetes Honepots. So we responded to them. And then there was the first draft review in early 2021 discussing the techniques and kind of which, where would each technique fit into the matrix? And at the end, right in April 2021, the first version of the MITRE attack for containers was released. And we contributed to 25% of the first, the techniques from the first edition. And we had evidence of the attacks in container environments as early as 2018. So this is the latest matrix now, the latest version of the MITRE attack for containers. And I add the parentheses there for and Kubernetes as well, because it does involve some sort of orchestration tools as well. So some of the techniques here are related to orchestration environments such as Kubernetes. This matrix was based on the Microsoft threat matrix that was published earlier. And what they do is they focus here on the main attacks techniques that are used in the wild, that they're seeing threat actors compromising environments and leveraging those techniques in the wild. But if we look at the Microsoft threat matrix, there is one technique there that it's not present in the MITRE attack framework, which is sidecar injection right at the bottom there for execution. And we'll cover that technique very soon, but first let's analyze how a threat actor group was compromising Kubernetes clusters in the wild. This is Team TNT. It's a threat actor group based out of Europe, probably out of Germany from what we were able to identify. And they were leveraging different techniques and abusing the Kubelet API to execute commands inside the containers. And at the end, mining cryptocurrencies, right? What we've seen from our honeypots that most of the attacks in containerized environments and Kubernetes environments at the end go is usually resource hijacking, which means basically mining cryptocurrencies. And this attack was carried out after they got internal access to the victim's network. And it can be either on premises or in the cloud. So first they get access to the network, and then they start looking for vulnerable nodes that have the Kubelet API exposed. So now we're going to see part of the scripts that we were able to collect from their command and control server, and that we analyzed and kind of to understand what they were doing. So this is the script here. It's a very, it's a piece of the script here, the cube pound function that they created. I have more slides breaking it down at each step. So we're going to do that here. And but basically what are, in general, what they're looking is, they're looking for, they're scanning the network for using an open source tool called mass scan. And they are looking for anything that it's open on port 10, 2050, right? 10, 250, 250, sorry. And which is the Kubelet API by default, right? So they're looking for that. And weirdly enough is that the endpoints that they're trying to target here, we're going to see running pods endpoint and run endpoint. The Kubelet API is not very well documented. So to know these endpoints, they would probably have to go into the Kubernetes code to find out more about what it does because it's not very well documented. So let's analyze, let's break it down. First here on the first steps, you see that there is a mass scan that the command that we're using, which is this kind of scanning IP scanning to network scanning tool that it's available on GitHub. And it's, they're specifically targeting only open ports on 10, 250. And you can see the ranges there, the LAN ranges, these are private network ranges. So it means that they already have access to your network. And now they're trying to do like a lateral movement or escalate privileges inside your cluster. After that, if they found something, there is also, they send like how, how we're able to count the number of IP addresses compromised, right? So each node that was compromised, it sent the IP address of the node to their command and control server where there's a function there, there's a curl, a function that sends the IP address of that node to the command control server. Also, all these curl requests here at the end, it's the, they are finding, they're looping through each container that it's running on the node. So it's not just the pod, but every container running on the node. And they're executing these commands, which is basically updating packages, installing some tools like bash, curl, W get, and then at the end running their crypto miners. So there's two binaries that they installed, they deploy there to run their crypto miners. Here is just a previous script from, from the same file that we collected, where they are installing the mass scan tool prior to using the cube pound function. And the next two that they leverage here as well is the tool called Z grab, which is a banner grabbing tool to find out which services are running in that port. And interestingly enough, this tool is deprecated. So it's an outdated version. There is a new version called Z grab two, but your attackers were using the outdated version. So it's not just us that use outdated versions. The attackers, sometimes they do too. Okay, but this attack is very noisy, right? It's going through each node of your cluster and updating packages and installing the crypto miners. So of course, they're looking for free money through this exploitation of cryptocurrency, but they soon will be detected. You're going to notice that something's going on in your cluster because all the pods, all the containers are running this process and running crypto miners. So one of my kind of thoughts or ideas that I had earlier this year was, okay, how if I'm an attacker myself, how can I try to stay hidden inside the cluster and still, you know, compromise and leave maybe a backdoor or install a malicious payload inside the cluster and still avoid some detection? So before I talk about that, I need to kind of cover the kind of the sidecar pattern. The sidecar pattern is basically a design pattern in Kubernetes where you add a new container to the same pod to add like new features for your application. For example, for collecting logs, for service mesh, things like that. It's handy when you don't want one container running multiple processes, multiple applications and you're going to deploy a sidecar here. Here's just an example of a login agent for a sidecar as well. You have the app container and the login agent is the one collecting the logs and sending to a back end, for example, like a SIEM solution. Just to sum up here, you can see that inside the pod, I can have one container or many containers running. So I have here as an example, I can have more and it can even have more than one sidecar container running. So just in a nutshell, this sidecar pattern, it's going to operate these supplement containers next to the primary one within the same pod. They can transfer data from external source to a local storage, for example, and they maintain a common life cycle, resources and network environment. I know that in the version 128 for Kubernetes, there are some changes now going on with sidecar containers there. So they're kind of built in and things like that. So what are the pros or what are the benefits of using a sidecar? So first, you have isolation. You separate your application from other side tasks that you have, that you can use for logging, for service mesh and things like that. It's easy, like you can quickly deploy a sidecar because you just need to add a new container next to your main application container. And it's scalable. You can use, as I said, you can have more than one sidecar inside the same pod and you can do that for your whole cluster. But there are some cons as well. There are some disadvantages. First, if you have a lot of sidecars, that's going to increase the resource utilization of your cluster. It can also add a lot of operational complexity because you can double or triple the amount of containers that you now need to monitor and maintain and make sure that they are protected as well. And synchronization challenges between the main application container and the sidecar. So this is a pattern that you can use, but you need to understand the problems before you use them. Okay. So sidecar injection. As I said, this is a technique that was covered on the Microsoft threat matrix. And I thought, okay, this is a possibility here. What happens if one of the ways that usually when you compromise a Kubernetes cluster is you want to deploy a privileged pod or takeover a privileged pod. But what happens, right, and that can generate alerts, that can generate some notifications there. But what happens if I compromise the sidecar only, and I don't compromise the main application? Can I avoid or at least delay the detection of my attack from, you know, the cluster admins? So the sidecar injection, basically, I can either inject a new container inside the same pod, or I can leverage the ones that are already running to execute commands for myself. Just like what the team TNT did for the previous attack, where they executed commands on all the containers inside the node, I can use specifically the one that I know. This is a sidecar and might be like harder to detect my attack here. And, yeah, attackers can use this to hide their activity. And the problem with sidecars is that they're considered kind of second class citizens in the cluster. Because they usually come from third party tools, they're pre-made, it's not something that you develop in house in your organization, for example, if you're using like fluent bit for your logging purposes for shipping your logs. And sometimes also, you might download them from public registries such as Docker Hub. So yeah, this is just kind of the description there from the Microsoft threat matrix and some of the mitigations that we're going to cover in a second. So just to demonstrate this issue here, I'm using, in this example, I'm using a cluster that I created on AKS. So it has a bunch of pods running and you can see there, some of these pods have more than one container running, have some sidecars. I'm going to perform, I'm going to show two scripts that, of course, it really depends on the permissions that you have on the cluster. But since we're assuming here, this is a post exploitation scenario. So I'm assuming that the conditions are met to perform this attack. So on the first one, I'm using kubectl. Let's say I compromise the pod inside the cluster. And I have the permissions to install kubectl inside the pod itself. So now, just an example here, I'm using kubectl to go through the containers, all the pods, sorry, that have more than one container running. And then I'm executing this command. In this case, it's just an echo command to, okay, executing crypto miner in the last container. So the last one that I have, so if I have like three, I'm only executing the command on the third one. And this script assumes two things, right? kubectl is installed and configured correctly. So I was able to do that. And the container has to have the sh binary. Otherwise, the command won't work. And we'll see here, by the output, some of the pods, some of the pods didn't have, some of the containers inside the pods didn't have the sh binary. So yeah, every time I run that script, it's going to go through all the pods that have more than one and execute the echo commander as we can see. Okay, but what happens if I don't have this kubectl installed and I don't have this, can I still leverage this kind of approach, this technique to compromise one of the sidecars and try to stay hidden? Yes, you can. If the attacker is unable to install kubectl, right, to perform the sidecar injection, they can still leverage the API server, curl commands and other commands, as long as they have access to the necessary files. So things like you need to know the API server URL, the better token and the CI certificate, the CA certificate file path. But if you have compromised the pod, you should have that information already. So that shouldn't be a problem. So I think this is a low hanging fruit for attackers to try to stay hidden. And it's important to consider that, that scenario, especially when you're tracking modeling your Kubernetes cluster, because you need to have visibility, not just on your main application, but on your sidecar containers as well, to make sure that they're not compromised. So what are some of the kind of mitigations that you can implement here for preventing sidecar injection in your cluster? The first one is applying the list privilege principle. I know it's easier said than done. Everybody that probably works in cybersecurity, they say, oh, yeah, apply list privilege principle, right, only giving the necessary permissions for that application or for the user for the time that they require those. And we know that in Kubernetes, our back can be very tricky, right, road based access control. It's very hard to get it right. So I know that this is a recommendation, but it's hard to get it right. Other things that you can do is restrict over permissive containers like running as privilege or root, for example, you shouldn't do that. You shouldn't have those containers run, especially in production clusters. And the third thing is gating images to the Kubernetes cluster, right? This is usually done via admission controllers. Admission controllers are like the bouncers of a nightclub, right? They check who can go inside the cluster and start running your containers. So you can specify if the containers are coming from a public registry or if those containers have a known vulnerability or they haven't been scanned for vulnerability, you can prevent through the admission controller as well. And other things that you can leverage here, I always like to talk about this kind of Kubernetes security triad from my perspective of some of the main tools that you should have in your cluster. You should have image scanning, right? To scan your images for vulnerabilities before you deploy them. The admission controller, as I said, the gatekeeper, the bouncer there to prevent unknown containers for running your cluster and runtime protection as well. But the admission controller, for example, in this case, it will only prevent new containers from running the cluster, right? If you're affecting the sidecar containers that are already running, the admission controller will probably won't be able to help you here. So that's why you need some sort of tool that monitors the runtime of your containers. And these are called runtime security or runtime protection tools. And CNCF has a great tool here that it's called Falco. It's a tool that does the runtime security detection for Kubernetes. And it's a root-based engine, right? You can create your rules for Falco to detect different attacks. It also has a bunch of preset rules that you can leverage as well. So you don't need to be able to create your own rules if you don't want to. And by specifying this simple rule here, this next one, and just to detect SH binaries, right? So the way that I was using there, the sidecar injection commands, I was using the shell. So just by detecting this, but just having this simple rule, you would be able to detect this attack as well. But you need to be monitoring. Of course, Falco looks at all the containers in the cluster. So it would be able to detect this, the previous scenarios, even if the attacker only injects the process inside the sidecar without creating a new one. If you want to learn more about Falco, I'm not going to cover that in depth here. But there are some trainings and some, a lot of documentation available for Falco online. And the Linux foundation even have a training available for Falco as well. So, yeah, this is all I had to share today about sidecar injection. I hope you enjoyed the session. And there's going to be a blog post with more details. We're still working some of the some of the things for the blog. So in the next coming weeks, I'll publish a blog and stay tuned and thank you very much for listening. If you have questions, please speak on the mic there. Thank you. Hi, our sidecar injection, some of the most popular ones that you're seeing today, or is it superseded by something else? So I haven't seen that splitted in the wild. So that's an idea that I from the trap matrix that I came up. But from our honeypots, we haven't seen that using it yet in the wild. Most of the time 90 to 95% of the attacks in the end is crypto mining, right? It's it's cryptocurrency mining. Yeah. Is that through some other method other than sidecar injection? Or is it mostly sidecar injection? So the cryptocurrency mining, yeah, it can be through the Kubla API or to some sort of privilege pod. It really depends on how the cluster is configured. The way that we deploy our honeypots, for example, we have clusters that have all sorts of misconfigurations. So it's really depend on how the cluster is set up. So you spoke about sidecars and the injection and admission controller as a way to mitigate this. Have you seen attacks when the admission controller was compromised and essentially injects the sidecars? Not knowing to you? No, I haven't. No, I haven't seen that attacks yet from the admission controller. Yeah, like just in general idea, if you have access to the codes on the level you described in your presentation, you're likely to be able to compromise admission control as well and just register your own web hook and redirect the traffic. Sure, sure. Yeah, also makes sense. Thank you. How come the Kubla API isn't protected in any way, like no authentication or anything? Yeah, so in this scenario, the Kubla API didn't have any authentication. So the cluster was misconfigured, right? That's why the attackers there, the TMT and T, they were able to compromise that. So they were inside the network and then scanning for the Kubla API, which is an API that shouldn't be exposed and should have authentication there. So on that blog post, I talk about, let me go back, just real quick. On that blog post, I talk about this one. I talk about the authentication, that it's recommendation for having that, right? If you don't have it, then it could cause this problem. Hello, my name is Emmer and I'm from CERT team in Microsoft. So I used to work on this like pretty much. So but just my question, since you are mentioning if it's like the Kubla API is misconfigured, but they still can access, let's say, they can do the sidecar injection, whatever you have, one of the privilege, let's say, bots. So my question on your rule, how do you mitigate that? Sorry, can I repeat the question, please? Like you have the rule, the file code rule that you just shared, but they still can, let's say, inject it using their any of your privileged bots, actually. Yeah, yeah, it's not only about the misconfiguration of your Kubla API. No, no, it's not. It's not just the Kubla API. So the Kubla API was an example from the previous attacker that was showing here, but the sidecar injection can be done other ways as well. And regarding the file code rule, can you just maybe like explain to me what you are trying here to do with the file code rule that you just? Sure. Yeah, this file code rule is just detecting the SH binaries, right? So this one here is just a shell binaries that is running on the container there. So if you detect that, you can get an alert. So Falco will alert you for that. Okay, thank you. Thank you for the good talk. So we got the alert now, right? The Falco, it could be Falco, it could be any other cloud native tool that notifies the security analyst that, okay, this has been noticed. What kind of actions could the security analyst, the blue team, can take to kind of not only quarantine that container or that host, but kind of root you out of the cluster. Sure, sure. Yeah, that's going to depend on the incident response process of each organization. But of course you probably should maybe isolate the node of that container, of that Kubernetes cluster, right? And then investigate what's going on. If it's, it could be a legitimate alert, like a false positive, for example, but it could be a vulnerability as well or someone that it has access to your cluster. So you need to analyze that. You might need to investigate other logs and try to correlate those logs and see what's going on to take the appropriate actions. Just to add to this question, what exactly do you mean by isolating this node? You mean just cut it off from the communication with the API? Yeah, you can, you can remove it from the cluster itself, right? Yeah, but in general, concepts of Kubernetes is resilient application, meaning that it will try to restart the exactly the same thing. And if the image is compromised, then the general, in general, this won't work. Yeah, yeah, that's, that's true. Yeah, there could be other options there, but I think analyzing the logs and messing with the understanding what's going on, I think that would be a good idea. Thank you, everyone.