 Ah, come back here. Why does that keep doing that? All right, great. We are here and ready to do securing in path testing, great spaghetti monster, as I like to call it, or Kubernetes. This is who I am, Kat Fitzgerald. You can find me on Twitter at Rainbow Cat. I've been doing this a long time. I'm the CEO of B-Side Chicago. I was COO for Diana Initiative last year, part of B-Side's Pittsburgh. And also, my first DEFCON was DEFCON 3. How many other people remember that one? There were 16 speakers there, if I have to remember that. I have an emphasis on the blue team, however, I'm a formal purple teamer and all this stuff. And I'm in Pittsburgh now, but moving to Kirkland, Washington very, very soon. You will always find me sipping Grand Mayan extra Neho, or in this case, it is Casa Noble and Neho. So cheers, let's kick off DEFCON. Jason already did a great job. And also special thanks to the entire red team village, all the staff, all the volunteers, everybody for kicking butt and making this happen. A lot of my favorite stuff are honey pots and refrigerators. On Saturday, I'm doing a talk for IoT security, so check that out too. But this is one of my favorite ones because I just finished writing this particular presentation just about a month ago, made some changes to it and so on. Remember, please, the views and opinions here are mine and mine alone. They do not reflect anything from my employers, past or present. Anything that I show you, as always, please use with a grain of salt. Don't go breaking into things that you shouldn't be breaking into. And those of you with an overwhelming fear of the unknown, you'll be happy to know that if you, there is no hidden message in the disclaimer if you read it backwards. Yes, my jokes are that bad. All right, let's get this thing on the road. So why are we not here? Well, first of all, I'm not gonna solve all your Kubernetes security woes. I wish I could. I wish Kubernetes and containers and everything were secured by default. They are not. Neither is the person sitting next to you, although, well, that doesn't work anymore because nobody's sitting next, well, maybe somebody's sitting next to you, you never know, you never know. Okay. And common sense went out the window decades ago. Think about that. When it comes to information security, cyber security, hacking, whatever you wanna call it, I honestly think we lost common sense quite some time ago. Why really are we here? Well, Kubernetes, I don't know, I still think it's fairly new. It's fun, it's crazy, it's a lot of things. You can deploy it in clouds, you can deploy it in, I have Raspberry Pi sitting down here. I was gonna be showing that, there've been some demo problems and also some timing issues. So I won't be showing my Raspberry Pi cluster today, but you can build very extensive Raspberry Pi clusters and do all the things that I'm gonna talk about today. Containers, containers are not new, but people seem to have forgotten that they have anything to do with security. They get all hyped up on Kubernetes security and they're forgetting about containers. We're gonna talk about containers because it's important, it's in fact very important. And always remember this, security is for everyone. Common sense is required. So please disregard the previous slide, throw it out the window. Why? Because if we're not using common sense in security, then we're definitely not gonna get where we need to be. All right, just some quick security facts. We wanna know this stuff. In 2018 alone, $114 billion were spent by companies on information security, on fancy software, fancy hardware, things with blinky lights and all of that, but breaches kept on going up. We're doing something wrong here. Security appliances and software have vulnerabilities that we're not dealing with or that are need to be dealt with by the vendors. Kubernetes itself has had a few issues. We'll talk about that, but so has Docker. So don't forget your containers. Also, I've actually had people say to me, we have a very unique security architecture. And I'm like, no, you don't. We all do the same stuff. So remember, your architecture is not unique. I love this quote. If you don't have this book, Offensive Countermeasures by John Strand, go out and get it. It's a small book with a ton of information, so get it. But he said, instead of brilliance, we've standardized on mediocrity. And that seems to be what information security is about these days. And I kind of have issues with that. So let's talk about that some more. First of all, let's get a common definition of what is a breach. Or maybe we'll call it what is an incident. Let's not necessarily call it a breach. So most incidents are not caused by zero day fancy stuff that you find on the internet. Typically, they're done because people have made mistakes and so on. Most breaches are not coming from your vulnerability scanners, especially when it comes to Kubernetes. You're not gonna run a vulnerability scanner and go, oh, look, fix this one thing and we're done and ready to go. No, it doesn't happen that way. Most breaches are going to come from configuration issues. This is something I want to emphasize. In fact, I'm gonna show you some examples how configuration issues tend to cause most of the breaches. A close second is compromise credentials. When I say credentials, I still go out on GitHub. Well, maybe not every single day. And I think Jason even mentioned it too in his previous talk on bug bounties. Go out and look on GitHub and search for simple things like password. See how many times you find the word root. Yes, thousands of them are invalid. They don't make any sense. It's probably gonna find the keyword root or password and it's not gonna get you anywhere. And yet I still find where people hard code credentials into variables that say password. And right next to it, there's the key. There's the whatever the password is. Also, trailing in third place for most incidents is overprivileged users. This is incredibly important in the Kubernetes world. I'm gonna show you this in just a second or in a couple of slides here. So keep these three things in mind. Configuration issues, compromise credentials and overprivileged users. Kubernetes itself, we have to deal with all three of these compromises or these types of incidents. They're going to cause issues in Kubernetes. But first, think about this. We think about the Capital One hack that happened recently. Well, that was a combination of compromise credentials and access that should not have been granted. We see how Tesla, they had poor access credentials and so on. Equifax, we all know about that one. Go back and check if you don't. Equifax was hacked not only by a simple vulnerability that had not been patched, but also because they had very, very poor lateral movement detection on the inside. Once they got in with more compromise credentials through the vulnerability, they weren't detected for quite some time. This is going to be critical. And one other thing, I was bored this morning. I decided to throw this up here in this slide. Why? Well, I put up a new spice rack in my kitchen. I really liked how it looked. So I took a picture and I added it to the slide. There it is. Isn't it a great spice rack? Come on. All right, you all better be laughing. All right, let's talk about containers. Quick review of containers. They are not secure by default. Now, I'm really focusing more on Docker containers for now because that tends to be the norm. And one thing to think about when it comes to containers, containers are not like a VM. A VM is actually more isolated and this is something that's gonna be important coming up here. So a container is a grouping of stuff and the stuff that we like to call it are resources. Well, the resources are grouped into namespaces. If you are new to containers, remember this. Namespaces are critical. You group your processes, your networks, your users, your ports, your interprocess. All of this stuff can be controlled and locked down within namespaces. Resources are exploitable. I was sitting in another web broadcast just the other day on containers and I was surprised and because it was supposed to be a more intermediate presentation on containers and yet all of the people were asking, is there a way to prevent process abuse such as CPUs and memory and so on? And again, this was supposed to be an intermediate class on containers. So it confused me a little bit that people were asking that because that should be something that you know from the very beginning that you get into containers. Using namespaces and the next most important thing is control groups or C groups. These are the limits that you are going to be putting on containers. If you don't put limits on them, you're going to run into problems. Every time I do a pen test of a new application deployed into a container, I would say that seven out of 10 times I find that the author of the application who has put it in a container has not put any limits. I'll get into the application in whatever way possible, whether it's a SQL injection or an RCE or something like that. And once I'm in, I can take over not just the container but the entire system that it's on because there are no C groups applied to the container. The other thing to remember about containers is think of a container as an image or more correctly, think of it as a file system snapshot. So who decides what goes in that image? If you look at this particular picture, you'll see, you know, here we have namespaces, the control groups that I show, you might, it's going to possibly be routed if you've set it up correctly. You know, and here's the application. There is a Docker client that you might use to launch this. There's a Docker registry. This is critical to the security of your system. If you're new to Docker, you may not know this but Docker Hub was compromised two years ago. I believe it was. I've lost track of time during COVID. I don't know anymore. But you want to have a secure registry where you keep your images because if there are somewhere that you don't have control, you never know what's going to happen. But here's the difference. Notice that we have a host OS and hardware down here. Unlike a VM, in a VM, you would see the entire operating system within the VM here. You would not see a shared OS. Yes, there's a host OS, but then you also have the guest operating systems themselves. And that's where we get into how a VM is much more isolated and contained than a container is. So don't think the word container means that you're more secure. It just means that we have some work to do. So that's why I say right here in this slide, container isolation, is it really isolated? The simple answer to that is no, it is not. So let's talk about some container best practices. I should have also said at the very beginning of this talk, this is an intro. I can't show you everything. I'm trying to get people to start thinking about how you secure containers. Then we're going to jump into how you secure Kubernetes at the same time. So remember, we have a base image. That means that base image has to come from a secure repo. If you do not know what CIS is, definitely go to CIS security dot org. Check it out because you want to use CIS benchmarks for locking down your base image. I'm going to talk about that in a little more coming up. Don't run your images as root. By default, if you do not set a username, your container and your application in the container is going to run it as root. I have found so many who have deployed WordPress applications in a Docker container via Kubernetes and they don't set the username and WordPress is running as root. So is the database and everything else. So of course, once it's compromised, you take over everything. Also, visibility, you need monitoring within your containers. Your containers put a lot of information out and they can log it, but you have to capture that. We're going to talk about that in a minute. Also, think of it as a domino. If one container is compromised, I guarantee you the rest of them will fall over like dominoes. When you start to get into containers in Kubernetes, you're going to see this. Also, we want to set some controls. Besides setting a username that is not root, we also set privileges and we set certain attributes to those users where it says no elevated privileges or no new privileges. You don't want user accounts or IDs to be able to escalate privileges and I'll show you some examples coming up. This one, my God, this is near and dear to my heart, secrets management. I still find, I would love to say by a show of hands, but I can't do that right here, how many people put the secrets and tokens and things like that in their Docker YAML files? So when they then deploy the Docker image, if somebody gets ahold of their YAML file which describes the configuration of that container, well, the secrets are right in it. I've worked with a lot of developers and a lot of engineers and I get them, you have to start thinking about putting your secrets in another place, whether it's HashiCorp Vault, whether it's a cloud secrets management system, something, but get them out of the apps and get them out of the configuration files and stop allowing SSH into your hosts. When we get into Kubernetes, coming up in a couple of slides here, one of the key constructs of Kubernetes is you have a master node or master nodes and then you have worker nodes and I still find worker nodes with SSH enabled on them so people can SSH in and quote, fix things. No, no, you use the management tools to fix things and redeploy, stop allowing SSH into your systems. Also, I can't stress it enough times, name space limits. Go read up on how to control namespaces. That is going to be one of your best efforts and best practices for containing and controlling and securing a container before you distribute it to or deploy it via Kubernetes. This is one of my favorite tools, it's called Docker Slim. If you've never heard of it, if I were to create a container, let's say I create a container and it has a boot two in it or the base of a boot two and then maybe it has WordPress running in it for whatever got off a reason that I wanna do this, let's say that entire image ends up being three to 400 meg. You know, it's still manageable but if I were to run that through Docker Slim, not only will it add some extra configurations and namespace limits and so on, but it shrinks it down and something that might be three or 400 megabytes deployed into a container will get shrunk down to maybe 10, 15, 20 meg. This makes it even that much more manageable, it's easier to deploy, it's quicker to deploy especially if there are requirements for scaling. So think about this, take a look at this. I also mentioned in Discord that my slides will be available in my GitHub repo and I'll get them to the red team village so they can add them to the Google Docs location after my talk. And always remember, apply SE Linux or app armor. One of the things that Docker Slim does is it will do that for you. It's very easy to enable SE Linux or app armor within a container. If you can do that, you're going to make it that much more secure, it's not on by default but you can set it up so it will be. All right, and remember the infrastructure and as I put it, ding, ding, ding, ding, ding. Yeah, you can't forget that all of this stuff is being deployed on an infrastructure of some sort and that infrastructure better be secure. I mentioned SSH, people keep allowing SSH into their environment. Be very aware that your infrastructure itself should be secure and I will remind you later about that. All right, here's a quick picture. Picture is worth a thousand words, right? If you look at our first example, this is our monolithic operating system here. We see hardware and operating system in run apps. Here we see a VM, so we see hardware operating system. There's a hypervisor involved and then there's going to be virtual machines but when you have virtual machines, you have additional operating systems, you have additional libraries. They're very much isolated. I should have used a different picture where there's a big line through it that shows, yes, these are isolated and so on but that should not have happened. Nope, go back, sorry. Okay, but when we look at using Docker or a container, we go hardware operating system, there is a container runtime, there's no real dividing lines and then we see containers. The only thing that is unique between the containers is that being in the library, the things that are going to run and then the application. So there's not a lot of isolation there and people want to think that there is but there is not. So please keep that in mind. I do love this comic. It came out a couple of years ago. The great defense is, we'll inundate them with quarterly Kubernetes releases. Yes, they do update Kubernetes a lot. That's why you have to pay attention to your environment and the tools that you're gonna use so you can do upgrades to your Kubernetes cluster. So let's bring on the orchestrator. Let's talk about Kubernetes. All right, here is Kubernetes 101. In one slide, I'm gonna teach you at least the basics of how Kubernetes works. We have a master node. The master node is critical. It manages all of the Kubernetes worker nodes. It manages the clusters. It deploys pods. Think of a pod as a collection of containers that are grouped. A pod might be the way that you decide, I wanna have a pod that supports a particular client or maybe it's a WordPress instance and that pod has WordPress. It has a container, a separate container for the database and so on. So the pod is how you can group things that are various containers into certain groupings and those groupings are kind of defined by you. But stop and think about how you want to deploy pods. It's gonna be important. A worker node. The node is where all the containers are gonna run. You have other Kubernetes components that have to run there. But the important part is, this is where the containers run. If the worker node is not secure and not patched and I can get into that underneath the visibility of Kubernetes, then Kubernetes has no idea that I've compromised a node. I found numerous Kubernetes sites around the internet where there's additional applications running on worker nodes because they've been compromised, of course, of Bitcoin miners. But the master node has no idea that it's running on there because they got in via SSH and they launched the application that the miner underneath the visibility of Kubernetes. So you really want to lock down these worker nodes. A pod I already explained, it's a unit of deployment and it's also how you address things from Kubernetes. And a pod will have an IP address. So that's important to remember. But it's gonna contain one or more containers. I like to think of pods as even though it might have its own IP address but think of it like a VLAN in a way because I've isolated it in one way. Services are various things that run within Kubernetes that communicate, that balance, that do a lot of work. We're gonna look at services coming up. There are various components that are used, three that are critical, the API server. You're gonna have kublets and you're also at CD. The one concept you should take about at CD is this is the database that describes and builds everything that is your Kubernetes cluster. The at CD environment should actually be walled off and firewalled and controlled very, very securely. If you do that, you will have a much more secure Kubernetes cluster. So keep that in mind and you can see it here by looking at a picture. So we take all of those concepts that we just talked about. We see here's an API server, here's a controller. This is all part, the kubectl app. Stop doing that. Come on. kubectl is the command you use to control everything. And even though this is a disk kind of a representation of at CD, I like to say, you know what? This needs to be off on its own firewalled and separated and it can be done that way. Here's your worker node and you can see how you might have multiple pods on a worker node. This pod might be talking to this pod. It doesn't necessarily have to talk to that one or they may not talk at all. These are some of the other services that are running the proxies. This is where we are exposed to the internet not over here, okay? So the only place you're exposing Kubernetes is at the worker nodes. Now, Kubernetes is not perfect. There have been problems. There was, I need to update the slide again because there have been a couple of other Kubernetes. These are more for reference. One's a little older. This was back, there was the backdoor that involved the API server. There were some others. These are only for your reference so you can go look them up and see. And there are more. The whole point I'm trying to make here is Kubernetes is not secured by default and it's not perfect. Well, let's try to make it perfect. Let's take a common Kubernetes cluster and say, you know, if we were to threat model this and that is very, very important. First thing we wanna do is to say TLS everywhere. Yeah, I know I have too many exclamation points but I'm trying to make the point. If everything within your Kubernetes cluster is using TLS and is encrypted then it's that much safer. I still have engineers come to me and say our application is completely internal. Why do we need TLS? It's never gonna be exposed to the internet. So really? So what happens if Joe's application over here which is exposed to the internet is compromised and then they use that as a pivot point to gain access to this environment and then they compromise your system. Oh, is typically the response. So I can't say it enough, harden the bloody infrastructure. Yes, I'm saying it again but you're running this thing on your infrastructure. Make sure it's hardened. And very important, two concepts. There's RBAC which is regular role-based access control. Set that up, make sure you're doing it right. Disable ABAC. ABAC is incredibly difficult to manage. You will spend days and hours and hours trying to do ABAC access controls which is more granular but it's not worth the effort. Make sure you use RBAC and manage it correctly. And I'm gonna show you a couple of tools coming up on how to check it. Also, check out this blog entry. This is, I mentioned this previously about monitoring. This is one way to audit Kubernetes. So keep this in mind. Take a look at that blog entry from Wazoo and it's an excellent, excuse me, an excellent place to start when it comes to monitoring your environment. Also, kubectl, get roll binding and all your namespaces. You should do this periodically in your environment especially if you have engineers deploying their own applications. Make sure that all the namespaces have good controls in their RBAC accounting. Also use third-party authentication for the API server. Again, something like HashiCorp Vault. Yes, I love that tool. If you are not familiar with it, get familiar with it. It's a great tool and it integrates very, very well with your API server and Kubernetes. I already mentioned this, separate and firewall your Etsy cluster, make sure it's protected. That is the keys to the kingdom. It's the entire configuration of your cluster and rotate encryption keys. Yes, use Vault. And by default, turn on security policies in the pods. It's some simple lines of configuration in the ammo files. It's not hard to do or use Docker Slim, that tool that I mentioned earlier because that'll help you set this up. All right, let's talk about risk. Kubectl, get secrets. That is the one that I just mentioned where you can look at the secrets from pretty much everything. But here's some tools that I absolutely love by cyber art. One is called Kubernetes RBAC audit. The other is called KubiScan. The RBAC audit does exactly what you think it does. It audits all of your RBAC roles. The KubiScan will look at all the permissions that are granted from the roles. These are two great tools that you want to put into your toolbox for securing Kubernetes clusters. Because remember at the very beginning I said, it's all about configurations. It's the number one area that Kubernetes clusters get compromised. Also, we wanna take a look at GitHub. Go out and make sure that, take that away, go out and make sure that your own configurations are not being shared up to GitHub by your engineers. Accidentally, how many times does this happen? It happens. I've had it happen by engineers. They don't do it on purpose, but they upload code and in the code are secrets. And oops, oops, it's not a good thing. Also, you need a locked down secure image repo and create gold images. Don't just let your engineers go out and do, hey, Docker run, bring down Ubuntu and pull it from the internet. No, you need your own repo and registry for your Docker images. Make sure that happens and please secure the libraries. That's another big one. We also, I need an entire slide on just the applications. I'm going out on the limb. This is my personal opinion, but I wanna say that nine times out of 10, okay, maybe eight and a half. Eight and a half times out of 10, Kubernetes clusters and containers are compromised because the applications themselves were poorly written and not tested. And that's what gets me is I have environments where I see the engineers create an application. They do so much work to lock down Kubernetes in the container, but they do a poor job and there's a SQL injection vulnerability in the application directly. So then the application is compromised and then the container is compromised because oops, it was running as root and so on and so on. Remember the dominoes, everything falls over. Also, remember your pods. Another common mistake that people make when they're new to Kubernetes is they don't stop and think, how can I group things into pods and secure those pods? At least if a container falls within a pod then maybe there's another couple more containers in there, those containers that get compromised are going to be somewhat limited within the pod itself. Typically a pod is not going to fall over when another pod falls over. It's usually containers within the pods. So you wanna think about grouping things. Also, Jason mentioned this in his talk earlier, but showdown, if you have never used showdown, get familiar with it, even for Kubernetes, you can go out to showdown, you can start searching for Kubernetes, you can search for API clusters or API servers, you can search for XED servers and if you find one of yours, oops, but they're out there. I did a search this morning for some XED exposure and to be, well, there were over 6,000 that I found that were exposed. I don't know how many of those have vulnerabilities that would have allowed me to compromise them because I was just curious what the number would be. And yeah, it goes up and down on a daily basis, but you should be aware of that, so use showdown. Let's talk about the configurations. Said this before, most of the breaches come from configuration issues. Use CIS hardening, Center for Internet Security. It is an amazing place. It is done by volunteers and workers within CIS. You can get benchmarks for Docker and you can get benchmarks for Kubernetes. You can use these to configure your environment so they are indeed hardened. Stop ignoring them. I still see people that tell me they've never heard of CIS and it makes me very sad when I hear that because it is such a great resource. The whole point here is we have to harden the environment from the standpoint of the basic configuration but don't forget the application. Also, I know a lot of people have environments that are CICD when it comes to Docker and Kubernetes. That's why we have them. It's for speed, it's for development, it's for putting things out quickly. Okay, great. A lot of people use Jenkins. Is there some way to automate all of this? So at least I can look at my images, I can look at my containers all within the CICD pipeline. Yes, there is. There is a great tool called Anchor. Anchor is, as it says here, it's a container image scanner plugin. And what it does is you put it into the pipeline, it's a plugin for Jenkins and it actually does security scans against the images that you're trying to deploy. This is a great way to get testing in before you actually deploy it to production. So let's talk about some additional tools before we get into the pen testing because we're still trying to lock it down. Kubbench, we want to run Kubbench. What it does is it runs against your CIS benchmarks, it deploys or I'm sorry, it displays a configuration that shows you what has passed, what has failed. I'll show an example in a minute. There's also Kube Audit. It's a little more granular than KubeSec. And if you run Kubbench and Kube Audit, it will give you a report that shows you, oops, my system is insecure. Here's an example of a report. And when we look at it, we see allow privilege escalation is not set. There's a very first one. Oops, that needs to be turned on. Remember, that was the one that I mentioned that allows users to do privilege escalation. We don't want that. Oh crap, we have run as non-rude as not set. We don't have a read-only file system set. There are so many simple things that can be done. Again, no app armor, no security comp, it's not being turned on. These are things that are just so easy. And this is from every single container within the environment that is being deployed under this particular. It's much longer, it goes into it. This is just a quick snapshot. The whole point here is you run this against your environment before you deploy it. And you go, oops, let's go back to our configuration file and fix all these things. Here are some more tools for you in the environment. Clare, another great tool for doing static analysis of applications. Clare does, it integrates Clare and your Docker registry so it can automate that a little bit. Falco, I need to put huge stars next to this one because this is actually one of my favorite, favorite tools. It was developed years ago and then it kind of morphed off and became falco.org. It monitors behavior within containers. It can detect if a container has been compromised because suddenly the container starts acting weird or there are commands being executed inside the container like cat, Etsy password, probably not a command that should be showing up in one of your containers. So if anything of this entire talk that I'm giving today, you get out of this, go look at Falco, this is one of the best tools out there. Kubhunter is another great tool. We're gonna talk about that. It's used more so for looking at doing some pen testing. We're gonna take a look at that in just a second. All right, ready, set, attack. If we were going to pen test an environment, this is an example of all the ports and all of the things that are open in a Kubernetes cluster, now they're open, but are they open by firewalls and so on? Well, we don't know. The API server itself is, yes, it's HTTPS and better damn well be protected. At CD on 6666, that should be protected wherever it happens to be. This just gives you an idea. All of these ports can be changed. These are all the defaults in most cases. Just be aware of the ports that are out there and exposed that you need to make sure are protected because as an attacker, what am I gonna do? Well, one of the first things, even Jason was talking about in his talk, looking at ports, running in map, running mask scan and so on. This is exactly one of the things that Shodan does. It looks for all of these things that are exposed on the internet. So please make sure you're aware of all the ports within your cluster. Now, I are a hacker, see me hack. This is one of the ways that I'm going to start to attack a Kubernetes cluster. I want to first and foremost check access to the API server, make sure that is protected. I'm going to be looking at at CD, making sure that's protected. If I find it exposed, I'm going to get keys and kingdom in that case. What about the kublets that are communicating in the command channels that are there? They should be read only, but they should not be exposed. There was actually a vulnerability exposed about this. I want to say two months ago, I'm probably off in that sense, but take a look at that and it allowed, even though it's a read only port and it's supposed to limit what you can see, there was some vulnerabilities that allowed for a lot of exposure there. Also container vulnerabilities. What about, do you have files in that container that the application needs that you should be protecting? Well, that's where I would set up Falco and tell Falco, hey, there's a special file over here. This application is using it, but keep an eye on that, make sure that the only thing that is reading that file happens to be the application and Falco can monitor that. Privilege escalation, nine times out of 10, once somebody compromises an application in a container, they elevate privilege because there's some other bugs in there. Go back to the Kubotic slide, where it showed privilege escalation was not set. All you have to do is set that configuration variable. It's not that hard, okay? Vulnerable applications are critical. Can't say it enough times, especially because there are so many people using Kubernetes and Docker to deploy WordPress. And WordPress is, of course, one of our favorite things to deal with. All right, so your final threat model, when it comes to pen testing and looking at the environment that is Kubernetes, you think of your workload, you think of that security within your pods. Can I set up my pods so they're indeed secure? The container security itself, the configuration for that container, the more hooks I put in, run as root, no, privilege escalation, no. These are the things that we want to make sure that are set. Also, file storage security. Maybe there's a particular file system that has to be read right, but the rest of the file system can be read only. Those are other things. The network security, remember the infrastructure. Put firewalls in, put controls in there that detect lateral movement. If somebody does compromise one of your applications, make sure you have something monitoring. I gave you that slide way back about the blog entry from Wazoo. They talk about how to audit and monitor a Kubernetes cluster using their tool. I love the Wazoo manager and agent. Take a look at it if you haven't heard of it before. So remember, most breaches are not zero day. I can't say it enough times. Everybody's always trying to protect against this stuff. They're not fancy. Most of these things are being broken into because people make silly mistakes because remember common sense, whatever happened to it. The breaches are gonna come from configuration issues. The CUBE audit was a very good example where it was showing all the problems that would allow someone to get in. Compromise credentials, make sure your credentials are stored in a secure place somewhere and don't allow for privilege escalation. Kubernetes and Docker give you the tools in the configurations to prevent that just turn them on. And that's what we're not doing. We have to turn on the configurations. So your key takeaway is from today. Common sense, please go back and stop and think about where is the exposure? Where are my pods exposed? What about my SED configuration that is so critical to any Kubernetes and always remember containers. Containers are the linchpin of a Kubernetes environment. And if I don't secure my individual containers, then Kubernetes, I don't care about the security of Kubernetes because the container is insecure. So stop overthinking this stuff and think about security basics. Remember the CIS benchmarks. Can't say it enough, the basics. I still think as security professionals, hackers, whatever we wanna call ourselves, we have forgotten the basics of how to secure an environment. And that's why things keep getting broken into. Stop doing fancy crap. The fancy crap doesn't work. Secure the environment, please. Use the CIS benchmarks for the containers, for Kubernetes, for the OS that your stuff runs on. If you're gonna deploy this whole Kubernetes cluster on Ubuntu, then pull up the CIS benchmarks for Ubuntu or Red Hat or CentOS, whatever you're using. Remember, RBoc is your friend. ABAC is not. Your infrastructure better be secure. Your pods better be secure. Containers themselves, the network. Still a lot of people forget the network because they forget about TLS everywhere. Threat model your applications. If you don't take the beginning effort to threat model your applications, you end up within the secure applications and all the security of your Kubernetes cluster is useless because you have insecure applications within your environment. Access controls go along with RBoc, but access controls into the environment, kind of referencing SSH. I found one worker node configuration in an environment, probably nine months ago, but for some reason on the worker nodes, they had FTP enabled on all the worker nodes. And my head exploded. It allowed me to get into everything, but oh well. Yes, did I mention RBoc though? Enough times, I can't say it enough. Why? Because people tend to ignore it. Remember the tools I mentioned earlier about monitoring or auditing the RBoc roles and also the permissions. And finally, not quite finally, remember to patch everything, the infrastructure that goes along with that, logging and auditing and monitoring. I still see people set up their environments. There's that blog entry again. Please take a look at that. It is well written. It's a great tool. You'll learn a lot about Wazoo. I use it on all my systems. I deploy a lot of honey pots with Kubernetes and containers and things like that, but I monitor them all with Wazoo, okay? You can't have a secure environment if you're not monitoring that environment. That is the biggest thing. So that brings me to the end. I thank you for your attention. There's contact info, there's my Twitter account. I will get this to the Red Team Village and they can upload it to their environment, but I'll have it on my GitHub later in an hour or so. I'll update it there. Thank you all very much. Hopefully you learned something today. Go out and hack and have some fun. Thanks to Red Team Village and have an awesome DEF CON. Awesome. Thank you so much, Kat and amazing presentation. As always, thank you for supporting DEF CON. Thank you for supporting the community. Of course, the Red Team Village. For those of you that just joined or are looking at the presentation in Twitch or YouTube, please look at all the activities that we have going on. Go to the website that is in the description of the stream. Please also join the conversation in Discord. We're gonna go in a brief break and the next presentation will start in a few minutes. Thank you again and thank you, Kat.