 Hello everyone and welcome to our session about insights into unsecured Kubernetes in the wild. We are really excited to be here virtually and hope you'll enjoy and learn from this talk. All right, so let's go over the agenda for today. First, we'll talk about attacks against misconfigured Docker, we'll talk about the notorious misconfiguration in Docker that allow executing commands on Docker Demons and then we will talk about the trends, the growing trends of attacking those demons and after that we'll go over the research we did which is split into two parts. The first one is that we thought how Kubernetes can be misconfigured and then we actively scanned the internet for such misconfigured Kubernetes clusters and the results are going to shock you, at least they shocked me. After that we'll go and talk about the honeypot we deployed. So we deployed those honeypots in order to see how frequent and what are the impact of those attacks on misconfigured Kubernetes components. So basically those honeypots are the misconfigured Kubernetes components and then we'll show you the results of the honeypots and then we'll talk about Hidegard which is a crypto jack-in malware we found using one of our honeypots. It's the first ever documented malware that is designed to attack Hublets and we show you how it takes over all of the cluster. Then right after it we'll talk about how to avoid such misconfiguration so that stuff like this won't happen to you. All right, so let's go over who we are. So Umi and Jay both are security researchers at Palo Alto Networks under unit 42 which is the division for threat intelligence in Palo Alto Networks. Our area of expertise is the cloud. We're both really enthusiastic about the cloud and all of our researchers were about anything that involves the cloud that includes vulnerabilities, malware, campaign, attack surfaces and everything that is security-wise interesting. And as you see by the pictures we are both proud pet owners. I got my dog and Jay got his cats. Hopefully someday we'll be friends. So let's start and talk about insecure Docker demons. So basically Docker demons listen on a unique socket for API requests and only root can acquire this unique socket. So Docker count on that and that's why they don't have any authentication or authorization mechanism inside. So anyone that acquired this socket can just run any commands on the Docker demon but there are also another option to configure Docker to listen on a TCP port but as I said earlier by default there isn't any authentication or authorization. So that leads to the situation where people just expose their Docker demon using the TCP socket and they don't have any authentication or authorization and anyone can just access their demon and execute commands that can create new containers, delete existing containers, pool images, execute commands on existing containers. They just got full privileges. And actually last year we found that there were over 1,400 exposed, unsecured Docker demons online and my hunch is that during the last year this number just got in increased, right? So we have this misconfiguration but is this really a problem? Does anyone really exploit those demons? So we wanted to test it and in 2019 unit 42 researchers detected a malware that exploits exploit this misconfiguration and they call it Graboid and it was a warm, a crypto jacking warm. Its purpose was to mine Monero which is a cryptocurrency and it propagated through the misconfiguration of Docker and just exploited it and scan other Docker demons and exploited them as well. So we saw that and this was two years ago and it was just the tip of the iceberg because since then we saw numerous other malware designed to attack unsecured Docker demons and we actually have a honeypot that been operating for more than one year and a half and we could see that during this time the frequency of attacks was more than doubled and the number of adversaries was more than tripled. I talked about it on this blackhead and the findings over there are just nuts. So we have this problem with Docker and everyone exploited but we wanted to ask ourselves is there any similar issue like this in Kubernetes as well? Are there any common misconfiguration in Kubernetes that can allow such attacks and if there are, are there any attack surfaces for that misconfiguration and if there are any attack surfaces, are they being exploited in the wild? So that's the question that led us in this research and now I'm going to hand it to Jay. Thank you, Arif. Let's dive into Kubernetes. So we have seen a lot of attacks against misconfigured Docker demons and we thought that, hey, this may also happen to Docker's bigger brother Kubernetes. So we want to look into the attacks against Kubernetes in the wild and what are the components, what are the attack surface of Kubernetes that we are focusing on? So we picked two components in Kubernetes, API server and Kubernetes. The reason we picked these two components is that they mirror the functionalities of Docker demons in Docker environment. API server is like a gateway to the entire Kubernetes cluster and Kubernetes is like a gateway to a single worker node. So API server can control. So API server takes HTTP request and translate into operation to control nodes, parts, and deployments in a Kubernetes cluster, could be that text usually takes HTTP request from API server and translates those HTTP requests into operations in a single worker node. So let's see how these two components can be misconfigured. API server misconfiguration. So by default, Kubernetes API server listens on two ports, a secure port and a local host port. Secured port use mutual TOS to authenticate between clients and servers. By default, it runs on port 6443. So any clients who need to communicate with the API server need to pass strict mutual authentication in order to invoke any API on the API server. And for the local host port, it is by default, it is intended for testing and bootstrap purpose. By default, it is exposed to only local host on port 8080. It is not supposed to be exposed to the entire internet. It can only listen to requests coming from the same host of the API server. And authenticate clients accessing this particular local host port are mapped to system master groups, meaning that any request sent to this port doesn't need to be authenticated. Any anonymous request sent to this port can actually act as a system domain of the Kubernetes cluster. So what can possibly go wrong with this local host port? What if the local host port is exposed to the public internet? So on the right, this is how a could be API server can be misconfigured. If someone accidentally bind this port to 0.0.0.0 instead of local host, this port is exposed to the entire internet. And anyone who knows the address of the IP address of this API server will be able to control the entire cluster. So this is one of the most common misconfiguration that we have seen in the past. And another commonly seen misconfiguration in API server is overly permissive role-based access control. So by default, Kubernetes come with a set of before role, before cluster roles, and before role-binding. For example, by default, what I highlight in the order box here, then an authenticate user or an anonymous request sent to the API server is mapped to an authenticate group. And this group is bound with the public info viewer. So the system public info viewer role doesn't have much permission, it can only view a list of APIs available in this API server or in this Kubernetes. Whatever Kubernetes complex, each cluster can be shared between tens or hundreds of developers. And more and more developers working in the same cluster, what if someone accidentally finds this system authenticated to another more privileged role such as cluster admin or admin. And this can be also, this again can be really bad. Any attackers or cyber criminals who knows the IP address of this API server will be able to gain privilege control to the cluster. Now let's look at the second component that we focus on, Kubilet. Kubilet works similar to API server, but it can control only a single worker node. It can control only the containers and parts in that worker node. By default, an authenticate request sent to Kubilet are mapped to system anonymous user or system. This user is mapped to the authenticate group. So this user in group doesn't have too much, doesn't have a lot of permission in the cluster itself. However, by default, Kubilet a set anonymous request. So anyone who can communicate with Kubilet can send request to it. And this is a problem. By default, this Kubilet allow anonymous access. What if someone accidentally exposed this Kubilet port to the entire internet? This again, a very common misconfiguration we have seen. Although the misconfigured Kubilet is not as bad as misconfigured API server. However, a misconfigured Kubilet allow an attacker to gain control to a single worker node. And from this single worker node, they gain the initial access and can start to move laterally to other worker nodes in the cluster. We will see in the Heal the Gar mailware that we found. Now let's look at how we can, how we find real-world attacks against Kubernetes. We need to find, we, at the time we started the research. We have the hypothesis that Kubernetes will be attacked in the same way as attack Dr. Demen. But we didn't have any proof. We didn't have any data. So we have to find our data or create our data. So we lay out two different research methods. We proactively, a project method and a reactive method. For the proactive method, we periodically scan the entire internet for misconfigured Kubernetes. And for the reactive method, as Abby will talk about in a few minutes, we deploy a few intentionally misconfigured Kubernetes having to track attackers and to learn what are the techniques and procedures of these attacks. So one advantage of having these two methods deployed at the same time is that they can actually cross-verify with each other. Our Kubernetes scanner should be able to find our misconfigured Kubernetes Honeypot. And on our Kubernetes Honeypot, it should be able to pick up the scanning traffic from our scanner. So this is how we actually make sure these two different methods all work as we expected. For the proactive research, as much as we would like to scan the entire IPv4 space for ourselves, it is not that easy. Actually, as soon as I start to scan the internet, either my ISP or the cloud server provider send me a warning. They typically don't like their customer to scan the internet. This scanning traffic is usually considered malicious. So I went for an easier path. Instead of scanning the internet by myself, I used the internet IoT scanner, such as SENSIS and Shodan, to help me at least identify the potential misconfigured endpoints. So I used these two services from SENSIS and Shodan to collect data for them every week and do some data aggregation and filtering to come up with a list of potential Kubernetes nodes or potential ports that run API server or Kubernetes. Then I started to query those nodes to see if they are misconfigured. And after that, I did some post-analysis. The result is what I'm going to show in the next few slides. Overall, in the period of three months, we identified more than 2,000 unsecured Kubernetes cluster that consists of more than 5,000 nodes. And within these 5,000 nodes, there are more than 31,000 CPUs that attacker can potentially abuse. And within these 5,000 nodes, there are more than 75,000 active bots running. And one of the biggest clusters that I encountered during the research had more than 500 nodes and more than 2,000 active parts. And that's a very shocking finding and also sad because if an attacker found this cluster, they can not only steal data from that Kubernetes cluster, they can also start to deploy a lot of malicious payload in this particular cluster. So there's no way that I could individually identify the owner of each misconfigured Kubernetes cluster. But for a few that I could pick into, I can see that they belong to e-commerce, finance, or even health care providers. And the figure here shows the overall findings throughout the research. We overall came across more than 37,000 containers and 47 different images. And in this cluster, they exposed more than 83,000 secretes that can be used to access application or parts or even the host. And each cluster exposed, each misconfigured Kubernetes cluster exposed different set of APIs. So because due to the misconfigured role-based access control, every misconfigured cluster can be misconfigured differently. For example, among all the misconfigured Kubernetes cluster that we encounter, 18% of them expose the part API. They can list all the parts and read all the parts in those clusters. And only 13% of the cluster exposed the secret API. And overall, China and the United States, these two countries add up to more, about 75% of the misconfigured Kubernetes cluster. This is kind of expected of these two countries where most of the network infrastructure or cloud infrastructures are hosted. And these are some common, these are some malicious activities that we have seen in those misconfigured Kubernetes. Most of the malicious activities started from deploying an official and signed Docker image, such as you want to send OS or DB. When attacker deployed this type of official and signed image, they are generally trust by the Kubernetes platform. However, after deploying this image, the attacker then deployed the malicious payload inside this container. Most of this payload that we have seen involves in crypt objecting operations. Some of the payload may also attempt to infect other nodes in the cluster, such as Hildegar that everything will spend in a few sites. Now, I will pass to Avid to talk about the reactive research, what we found in our honeypacks. All right. Thanks, Jay. I'll take it from here. So let's talk about the reactive research. So basically, the reason we did this reactive research is to see what is the frequency of those attacks and to see if there are any other attacks. And if there are, then what are the impacts? So we deployed two types of honeypots for around six months. And the first one is misconfigured API server, whereas the RBAC is misconfigured in a way that any user that will access the API server will have admin privileges. So that's really critical. And the second honeypot is misconfigured Qubelet. In this honeypot, we just deploy Qubelet with anonymous access allowed and exposed to the internet so that anyone that will access this Qubelet will have permission to do anything he wants on that specific node. So let's go over to the results. For the API server, it was really kind of disappointing because we saw zero exploit attempts. But one interesting thing that we saw is that census scanned us really in depth. And when I say in depth, I mean that they query the API server for nodes, for pods, for RBAC, for secrets, for anything that the API server had to offer. And for those of you who doesn't know what census is, so that's a search engine. They just scan the whole internet, put it in a database, and allow users to query this database. So our honeypot for sure is being in census. But other than that, nothing interesting happened over there. And so let's go to the second part, to the Qubelet honeypot, which was really exciting. We got exploited numerous times by different adversaries. Most of the attacks were for cryptojacking purposes. And one of the attacks included a really interesting malware. They did a lot of mess, so we decided to explore it. And then we found out about Hildegard. So that's the name of the malware. We decided to call it Hildegard as there was a string in the binary, which was Hildegard. And this malware is just crazy. It's the first ever malware that is designed to attack Qubelets and to propagate across the cluster. And we'll talk about how it operates. So first of all, the first thing that happens is that the attacker exploits an unsecured Qubelet. And after that, it deploy a container that includes first teammate client, which is a tool that can be used as a reverse shell. And this way, the attacker can run commands on the node. And the second thing they do is they deploy a mass scan, which is a scan tools. And this tool scans the internal network of the cluster and try to find other Qubelets. And when it does, it exploits them as well. So Hildegard just tried to propagate to all of the nodes in the cluster. And on every node, it deployed first a crypto miner, which mine Monero, and an IRC client that will connect to an IRC server owned by the attackers. And this way, the attackers could send commands to the containers and do whatever they want on the cluster. So that's how everything works, basically. But we wanted to talk a little bit about the techniques they use because we think it's really worth talking about. So let's go over and talk about through the escalation. And Hildegard deployed two open source tools. The first one is pirates. So this tool scan for cloud credentials and in the cloud metadata server, whether if it's AWS, GCP, or Azure. And it also looks for service account tokens or secrets. And when it finds one of them, they just try to escalate to get higher privileges. The second tool is PotB. I hope I pronounce it good. Well, it's break out the box. And as the name implies, it will try to break out of containers. It will try to use known vulnerabilities. It will try to find if there are any capability or syscall that are in there by mistake and try to exploit them. They'll try to access Docker daemon on the host, as we talked about in the first slides of the presentation. They also try to find some cloud and Kubernetes credentials and even look at the environment variables for credentials. And other than that, Hildegard by itself has a script that tries also to find credentials. And it will scan the cloud provider metadata server. They scan for SSH keys. They scan for Docker credentials. And of course, and foremost, they scan for Kubernetes service accounts token. So you can see the script below. That's how it does so. All right. And two, my favorite part, which is defense evasion. Well, Hildegard uses two types of techniques of defense evasion. The first one is they use LD preload. And we won't get into what is LD preload because that can take, like, 30 minutes. But we'll just say that Hildegard just hijacked some libc functions. And while hijacking them, it replaced them with other functions that basically does the same as the normal functions, except the fact that they sanitize the output. And by that, I mean that this is how Hildegard hides itself. So let's say, for example, we run PS. So we won't see Hildegard in the PS. And if they run LS, we won't see Hildegard in the LS. And if we run top to see the CPU usage, we won't see the CPU usage because the LD preload function will show us something else. And the second part is the encrypted ELF binary. And Hildegard uses a well-known RSC client. And in order for him not to get detected, they encrypt it and decrypt it only when running into the memory. So in this way, Statica has his tools, cannot be effective. All right, that's all for Hildegard's first day. Jay, can you take it from here? Thank you, Aviv. Now, let's look at how individual can protect and defend against these cyber criminals. This threat exploit is configuration. 99% of the known attacks exploit is configuration. Like we only look at the API server and the misconfigure. API server and misconfigure could be left. But as Aviv mentioned earlier, this is just the tip of the iceberg. To our biggest surprise is that we didn't see any attacks targeting our API server honeypot. We think that attackers haven't really moved from attacking back a demon to Kubernetes. Kubernetes is much more complicated than the attackers. And however, the potential computational and storage resource as well as data in the Kubernetes can be much bigger and more juicier than is configured by the demon. So we think this can be just the beginning. And we expect to see more and more attacks against Kubernetes in the near future. What are the best, what are the strategies to protect your Kubernetes infrastructure? Best practice, follow the best practice should be, should keep you secure, really secure. The layers of defense building to the Kubernetes platform provide very strong security, such as namespace isolation, network policy and second, secure your network and identity. As long as you make sure you don't accidentally expose your Kubernetes components, such as API server, Kubernetes, proxy, XCD, you should be safe from 99% of these attacks and also secure your identity, make sure you follow the principle of this privilege, minimize the number of users in your, users and groups in your Kubernetes and continuously monitors the permission provision to these users and applications. Finally, patch frequently, patch not only your application, but patch also your Kubernetes. Well, Kubernetes is complicated, involving a lot of moving parts and when you have a cluster of hundreds of worker nodes, it's not easy for a single person to oversee them. You can never be 100% sure that you don't have any insecure or accidentally introduce any misconfiguration. So if you are not so comfortable to provision and maintain the Kubernetes clusters from scratch, use a managed Kubernetes service, such as AWS EKS or Azure EKS. Call service providers can help you manage and control the control plan, and you only need to manage the application running on top of the Kubernetes clusters. And also most of the call service provider have some sort of guardrail to oversee your Kubernetes cluster to make sure that you don't have any misconfiguration. Next one, ephemeral workload design. So if you have a ephemeral workload design, meaning that all the parts and containers in your cluster can are stainless, they can be keel and restart anytime without losing anything. So if your container infrastructure or microservice applications are designed this way, then attackers don't really have any way to get the initial or get the foothold inside your application. As long as you see a container's drift from the normal or benign state, you can simply just keel and restart the container and you start from a fresh and clean state again. And finally, continuous monitoring is also important. Use tools to check configurations of your cluster and use tool to check for suspicious workloads or check. Again, although your infrastructure is here, your application may be vulnerable. Your zero day or new vulnerability may be found in the application deployed in your clusters. And if a attacker can attack those applications, there may be a chance that they can escape, break out the container and start to attack your company. So continuous monitoring to a company cluster is important. And this is the end of our presentation. Thank you for being in our session. If you have any questions, feel free to reach me, Jake, at peolato.networks.com for a VIP. This is a session. It's his email at peolato.networks and we also publish the findings in Unifor two bucks. Feel free to go to the Unifor two.peolato.networks.com to see more of our research.