 Hello everybody, welcome to our talk on hacking your misconfigured Kubernetes cluster. My name is Eric Smalling. I am a senior developer advocate at Sneak. Little bit of myself, you know where I'm coming from. I'm about 30 year developer the last decade or so consulting around the DevOps space I've been working with Docker since frickin 2013. My info is all on the slide. You can take a look at it. Hit me up on Twitter, LinkedIn, wherever you want, but let's get right into it. Okay, so today we're going to be talking about how the combination of application vulnerabilities and commit configurations can allow an attacker to spread the blast radius of their attack. This is a pattern that almost every major exploit of recent years has followed. An application vulnerability gives the attacker an initial foothold and then the application infrastructure level misconfigurations allow that attacker to spread into other parts of your system. Today, we're going to be talking about this in the context of Kubernetes. We'll walk through how we can go from exploiting an app vulnerability to basically owning your cluster. Okay, let's set the scene here real quick. So say I'm a hacker and I found a vulnerable application on the internet. I don't know much else about it other than it has a particular vulnerability. For the purposes of this demonstration, the application is a mockup. Now it has a remote command execution vulnerability, which is going to allow me to run commands directly on the server where the application is running. This is a simple flask out, but these kinds of vulnerabilities do exist in the wild, such as in Tomcat or other platforms. This kind of vulnerability allows an attacker to pass malformed or specially crafted HTTP requests to allow them to run commands directly on the server. Okay, so as we go along, as I said, the blast radius is going to get bigger and bigger as we exploit this cluster more and more. And we're going to keep track of that on this graph, which kind of goes up as things get worse throughout the hack. Okay, so here we are at my desktop and you're looking at the front page of this mocked up flask app we've written that has a remote command execution exploit in it. Now this is fake, but this kind of an exploit really can happen in app servers or other things like that. Or your own code. So let's take a look at this. Now I'm going to issue a first thing I'm going to check on is I'm going to issue an env command. This is useful because we can find out what environmental variables are set up for your process. That can give us a lot of hints about the environment we're running in, right? So we have a lot of Kubernetes stuff showing up. So we believe we're in a Kubernetes cluster because that's a lot of evidence there, which means we're in a container. So the first thing we see here is we have the Kubernetes port. That's the API server endpoint for this pod. So it could connect if it needed to. Another thing we see is a bunch of web admin service host ports. This is the service info. So this is running on port 5000. It looks like that's also kind of interesting. So let's take this. I'm going to leave that tab open for reference and open another tab. And we're going to change the command now to IP% 20a. So we ran IPA on the box, basically. And sure enough, there's our pod IP address. And we know that we're in a range 24. So we kind of know that the range our pods are going to begin. That's that's good to know as well. Okay, so what do we know so far? We know that we can on port 80 hit a Kubernetes cluster that is exposing a service that appears to be on port 5000. So we probably got an ingress there that's mapping us to port 80. We know the internal IP address of the API server and we now know the IP address of the pod as well. So let's go back. What else can we glean from this? Okay, so I've run a cat command and I'm spitting out the token, the service account token for this pod that got auto mounted in. Now what that means is that somebody has left a default here for the service. What is it? So look at my notes, the auto mount service account token, you could set that to false. And because we didn't, this automatically it's mounted and is available to the process or anyone else attacking the box. So let's add that to our list of things. So our timeline of doing is expanded. We've got a credential we've found now, which is the pod token. So we'll go back here. And okay, so two things I just found out now, curl is available in here. That's good to know because that's useful. I could use that to pull down my own scripts. All sorts of stuff you can do with curl. But I've also found that the endpoints API is available to my user. This is not uncommon. So what I've done here is I've curled the API server endpoint, which we found out from our environmental variable. And I'm hitting the endpoints endpoint of that endpoint, that API. And I fed it the token for the pod service account user, along with the CI cert that lives in the same folder. And sure enough, we get this back, we can see that this is Kubernetes. So this is the front end of the control plane, and which is expected. And the that endpoint has this IP address. Now, if this was in a, this is running in a kind cluster, if this was in a managed or like a large scale Kubernetes cluster, that would be the external IP. So when you are using a managed cloud provider, for instance, and they give you a kube config that has an IP address or a host name or something in it to connect to along with your token, this is where that IP would match, right? So this is the control plane public IP. So we've got quite a bit of information now. We now, what else do we know now, we have the internal IP internal IP and we can hit it. We know that, which we now know has permissions that I that token had the permission to access the endpoints API, which allowed us to gather some more information about the cluster. So let's see if we can get access, you know, actually get into the cluster from outside now without having to use the exploit. So first thing I'm going to do is I've got a setup script here that and I'm going to come back over here. I need to get that token. So here's the pods token, service accounts token, I should say. Let's come back over to Michelle, paste that. And like I said, this is a kind cluster. I know this is running on my local host at 6443 as opposed to what we would have found out otherwise. Now let's take that that's created a kube config for me is all it did. I'm going to say export the kube config to this file just got created called demo config. Now let's try to get some pods and see what we can do. I have a k alias to kube CTL because I don't like to type. And we got an error, but it's not, not a useless error. It's telling us, hey, that's forbidden. You this account, our account in his web admin and we are in the secure namespace. Good to know. And it cannot get pods for the default namespace, but we now have a namespace name. We know that there's a secure web namespace. And let me clarify that there is a namespace named secure namespaces are not a security boundary. They are not a it's up to you to secure your namespaces with whatever we'll talk about in a minute. But just so you know, that doesn't mean it's secure. It's just named that. But sure enough, when I pull pods with the in that namespace, I with this token can see them. And I see the web admin pod has made itself known. I wonder what else I could do with this. So let's do a QCTL auth. Can I list and we need to pass the token to it? I think it's still in the Facebooker. So in the default namespace, I can, we knew we could get at the endpoints. We figured that out. But I really don't have a lot of anything else. What can I do in this secure namespace? Ah, cool. Like expected, I pretty much can do anything in the secure namespace. So all these are available to me. Just so you there's actually a really cool plugin for QCTL called Access Matrix. It's part of the crew plugins setup. Let me show you that. It's kind of cool. Access Matrix. And so it gives me a nice list of all the resources and then my privileges on them. And in as we would expect in default, I don't have much of anything. But if I add the secure namespace to it, spell it right. Sure enough, I have a lot of access. So just another review on top of it, a little more detailed getting into what I have access to. Okay, so let's kind of review where we are. So what do we know now? So we have knowledge of namespaces. So we know that we are seeing an application that's running in a namespace named secure. There is a default namespace. And we also know that the role that's been set up is not very restrictive. It's not stopping me from figuring what I have out here. It's likely bound just as secure namespace. That's a common pattern. When you see a pod security policy and a role applied, often they apply it to the namespace. The application developers come up with it, right? They apply it to the namespace that they're running their application in. And there's a good chance that the default namespaces doesn't have permission set up unless the cluster ops have done something there. So we'll keep that in mind. Let's hold on that. So next, let's see if we can get a shell on the pod we've compromised. So let's get pods and secure again just so we can get the name of it, right? Cool. So now we're going to exec in to in the namespace secure to the web admin pod. And we're going to run bash. Yes, we can. That didn't surprise me. Let's do a who am I? I'm the web admin user. Okay, that's fair. I'm not running this route. That's a good sign for the owners. But can it become root? No. Okay, that's good for them. I wonder, can I create a file? Yes, looks like I can. Sure enough, I'm able to create files. So that means I'm not in a read only root file system. If you're not familiar with that, by the way, timeline, we know we know our role gives too many permissions. We were able to get out to the end points and stuff. But we know that our read write file system is in play. By default, your container runtime engine, whether it's Docker container to cry or whatever, it's going to put that read write layer at the top above your image layers and uses copy on right. And that's all a bunch of details you can get into elsewhere. But people wonder why would you want to do that? Why would you want to read only? Why wouldn't you want to read write file system? Well, for this reason, as a bad actor, I can now can create files. I can pull files down from the internet. I can change configurations. I can clean up after myself. I can delete, you know, evidence of my involvement. All you need to do to get away from that is set your read only root file system to true in your pod security context in your container. The problem is your application might not run if it needs read write file system. So you might be some refactoring to your app you need, or maybe you can shunt off the read write access it needs to a volume. Maybe it's FFS, who knows. But if you can, it's a great security mitigation effort to make things read write false or read only true. Sorry. It's just a good idea. So let's go back here. So we'll keep that in mind. We can write the file system. Let's see. Next. Let's try to run. Let's see if we can run another pod of our own. I've got one. Let me get out of here through my screen. I've got a simple demo YAML root pod, right? So I've got this simple pod. It just runs Alpine, which is a root user by default. And it just sleeps. So we're just going to see if we can start this up. I demo YAML's root pod. And this is in the secure namespace. API server took it, but let's take a look here. Yeah, sure. No, it's not running, though. They describe your pod root. Let's see what's going on here. Yeah, just as I would expect. So run is non root is being enforced. So there is a pod security policy here or something that's doing that. It's not letting me run as root. That's fair. Got to try. So what we know is that we have a pod security policy in place, at least in the secure namespace. So now let's come back out here. Let's deploy something that's not roots, but we have a non root privilege. Let's see if we can deploy a privileged container, which would be bad. Privileged containers are pretty frowned upon unless you're doing something really unique. So we're going to deploy that into secure namespace. And sure, that's forbidden right away. Pod security policy right there. So it is a pod security policy. It's most likely what's causing the root thing to not work. Same thing here. So that's fair. So now, how else could we get a root user? Well, I have another manifest we can try. Demo, YAMLs, non root, non-priv. And this one, same image. This is my, but Matt Jarvis is my colleague here at Sneaky. He crafted this sneaky image. This is going to run as a non root user. It's not asking to be privileged. So that seems pretty fair. It's bad enough that I can deploy this, right? But at least they stopped those. Let's see. Demo, non root, non-priv, YAML, secure namespace. And it's creating. So that's a good sign for me. Let's wait till it comes up here. Okay, it's up. I'm not going to exec into it. I'm going to do something else. Let's do a port forward. We wrote this. I know what's in this thing. So Sneaky and port forward in the secure namespace. Sneaky, port 8080. Okay, so we have a port forward going. Let's go back here now. Hit local host 8080. And I've got a little go TTY running inside that container. So I have a shell that's running in that container. And I'm the sneaky user at the moment. But now I'm the root user. So how do you think I did that? Well, what it comes down to is pod security policies. You need to be explicit if you're going to use these things. And honestly, people are moving to OPA now in Rego. So the same things apply there kind of sort of. But if you're using a pod security policy, a lot of people will do the things we saw here. They'll list. Don't let it run as root. Don't let a privilege container come up. But they'll fail to say don't allow, allow, don't allow privilege escalation. Because honestly, the documentation is a little ambiguous on this one. You would think that that's redundant. If I'm saying no root, no privilege, then I wouldn't also also would not be able to escalate my privilege in the container. But you can. And unfortunately, it is confusing and there's actually, you know, documentation and blogging out there that says it wrong. So just know that that you can you need to set allow privilege escalation false so that it doesn't happen to you. Let's go back to here. So we've got another piece of information. We have some permissions here where the PSP didn't allow disallow that. Another thing that I didn't talk about that's actually kind of interesting. I was able to get my image from public Internet image registry. So that means that our network is not limiting me to only use images from their, you know, this company's registries or there's seems to be no limit. If I can download that I can pretty much do anything. So that's that's interesting. Let's talk about network. So I'm going to let me get out of root for a second. Are you familiar with NMAP? NMAP is a tool for diagnostics and troubleshooting and networking. It's also a tool for hackers to very much spread across the network, spray across packets to find open ports on things. It's a port scanner basically. I mean, I'm sure it's more than that. I'm not a network expert, but it's you. It is useful in that use case. So first thing I'm going to do is run. I have config so I can see what's my IP. So my IP is dot seven. Okay. And now I have to copy this because I cannot remember the syntax of NMAP. I'm going to paste this in. I'm going to fix this dot seven. So we're going to say, hey, oh, I can't do this. I'm not rude. Because I'm rude and I have NMAP, it is scanning. And sure enough, we found two ports for port 5000. We saw that before. The service that's running on the web app that we started with is running on 5000. And we see two of them now. Interesting. Let's go back over to our console for the, okay. So here's the web facing the internet facing web app is at dot one dot four is IP address. And we're seeing 5000 come up because that's this one. But we also see a dot five, which is not this. It's not the my sneaky container. And it's not the one we saw. So somebody else is running something on port 5000 in this network segment in this cluster, basically. So I'd like to know what that is because it sounds like somebody might be running another copy of the app. It's not in the secure net. Not namespace because we'd see it if it was a pod running there or we'd see a scale, you know, we'd see multiple pods scaled up. So let's update our what we know. So we now know that there's another pod probably running in this cluster for 5000 outside. We don't know where it is. We know it's not insecure. Now, one of the things we know based on the fact that I was able to get those images and the fact that I was able to run in map is we probably don't. We either don't have any network controls in place or they're they're way loose and they need to be tightened up. So just take that to note. So what I'm going to do is in this sneaky container, I'm going to run so cat, which is socket cat. Whatever it uses for tunneling. Usually that's what I use for. So I'm going to use it to tunnel any traffic coming into the sneaky container on 5000 one and it's going to spit that traffic out. Let me fix this IP to the this rent in this new pod that we're finding. Right. Okay, so that's set up. And now I'm going to go back to my terminal. We're going back out of the cluster now. And we're still port forwarding here. I'm going to background that type. Okay, and I'm going to start another port forward port forward in the secure name space. And we're going to forward to sneaky again on port 5000 one. Right. Right. Okay. Start that up and let's go back to our browser open another tab. Local host 5000 one. Well look at that it is the same app. Amazed. So somebody else is running another copy of this app on another port that's not part of that secure deployment. Secure name space deployment. Let's see if we can send a command to it to see if it's exploited was exploitable. So this is cool. So now we know we have this rogue vulnerable application somewhere else. Go back to our notes and we know we can tunnel into it. We can get to it. Good to know. So what can we do with that seems to be the ongoing question. What can we do with that? And let's come back to this tab. And we're going to change this. I want to grab the service account token in that. I'm going to call it the unknown app. Okay. The unknown apps. Token. I'm going to grab that. And I'm going to go back to my terminal. And I'm going to kill this total. Don't need it right now. I'm going to edit though my cube config. It's going to do it manually. And we'll go to bottom and comment out the token from the original app. And we're going to replace it with this new token. And let's let's see if we can pull up anything with here. Oh, I could pull default. I can pull default. We see web admin that it's the same app. Obviously I can pull my pods from the default namespace. That's that's cool. So let's let's find out what else we can do off. Can I list token equals that. I have full access in the default namespace to do all these verbs. So that's golden. That's pretty cool. So let's go back. We now know it's in the default. Let's try to run something in the default namespace should be able to now. Right. So we're going to go back to my shell. And given that we have that new token, I'm just going to apply into from the demo yamls. Non root. Let's go for gold here. Okay. Let's see if it's starting. That's that's pretty cool. It's like into that pod it into the non root prove on bash. Okay. Who am I sneaky? That's good. Remember what we can do is sneaky. We can become root. Now, do you remember this this yaml we looked at earlier? This is a privileged container. I am now running a privilege container in the default namespace in this cluster. And I'm root. If you know anything about how Linux stuff works. Actually, I'm giveaway. Let's do a PS. So I can only see the processes in the container. Right. Well, if you know how it works, you can do true. Now, if you remember from that yaml, I'm mounting the root file system into a file system or a mount point in my container named true. And now if I if I change my route to that, I can I'm effectively on the host. I'm root on the hosts. So I'm now basically in control of a worker node. Yeah. It's kind of like, wow, okay, that's bad. What am I going to do from here? Well, let's see what else we can do on that node. So I'm going to I need to get I need to set up the kube config. I'm going to kuber neti. Sure. There it is. And the kube let's have to live on the nodes. Right. So kube, I can't use my aliases here. Get pods. Let's go for gusto here. Let's see if we can actually get to the control panel. The control plane. Sorry. Yes. Yes, I can't. Heck. And there's my nodes. The fact that I could do that tells me there's no restrictions in the default namespace. So we confirmed our suspicion before when we said a pod security policy probably set up by the developers for their namespace. Yeah, it looks like it. And now I can get in. I'm in I'm in default and I'm kind of the kid in the candy store now. So and we now are in the kube system namespace and what lives in kube system. It's a day. So let's see if we can get at that. So let's go back here. And the first thing I'm going to try to do, let's try to run kube CTL. See if I can run something in here that I can then use as a launch vector. Busy box. We start never and show don't can't do that. So kubelet token is not allowed to start up. It's not allowed to start up a pod in the kube system. It's just it's a security restrictions built in. It's good. But honestly, it's it's kind of okay, I can't do that, but I could do this. Kubernetes manifests. I could add a static pod, which can do basically the same thing. If I'm wanting to start a pod as the kubelet in with that kind of access, I could do it that way. Let's not do that. Let's I'm going to get out and we'll do something different. So I also have an ecd manifest that can deploy an ecd client. Let's take a look. There it is. It's creating. We are going to do a member list. That tells us we are connected to the ecd server or we can. So we're cooking with gas as they say, right? If I can get to ecd from the from the default namespace from a worker pod, what can I find out? What's in ecd? What isn't in ecd, right? But one of the most famous things people like to look for in ecd are secrets. And there's a specific secret I'm interested in here. And I want to get, let me see if I can find it. There you are. Controller, cholesterol, aggregation, controller token. There's a token that is interesting. What can you do with that token, do you think? Well, let's find out. Copy that. Now I'm going to do kauth. Can I list token equal that? I'm root. I have the admin. I have admin, basically. I can do whatever I want in the cluster. I have the cluster token, basically. So, I mean, there's not a lot more to say there. Once you get to that point, once you have cluster admin rights, it's game over. There's no further real, not much more, I have to say. So how can we prevent all of this? Well, obviously the vulnerability in that application is the most glaring issue. And scanning your application and its libraries is the first thing you'd probably want to tackle, so that you're not actually deploying applications with these kind of vulnerabilities in them in the first place. Next, scanning your container images is critical for similar reasons. Vulnerabilities brought in from a base image or operating system level packages that you might be building into your images can introduce some of the same kind of exploit possibilities. Finally, making sure your Kubernetes settings are hardened against the kinds of things we talked about today not only adds another layer of protection against vulnerabilities that slip through or may still be zero day or something, but it also serves as a great way to train your teams on these tactics and to help foster a culture of security. Now, of course, Sneak offers scanning for all of these things and more. Go sign up for your free account at sneak.io and take a look at it today. Before I wrap up, I wanted to give a shout out to the Kubernetes security community. One of the great things about working with Kubernetes is how generous the community of contributors are. And much of the content that we've put together to build this talk is gathered by research done by our friends and colleagues. In particular, we want to give special thanks to Mark Manning, Ian Coldwater, Duffy Cooley, as well as all our friends in the Kubernetes SIG security, Meetups, and Book Close. Thanks again for listening and we can't wait to see you at the next talk.