 Hello, everyone. Fantastic to be here. Even better, to see such an awesome audience. I'm really impressed and scared. My name's Patricia. And today, we are going to explore live Kubernetes hacking together. So get ready for ride. First, a briefing production. I've been doing professional software development for over 20 years. Yep, sometimes I do feel like a vintage dinosaur. But hey, who doesn't love a classic? I've always been close to code. And that bond still thrives today. However, almost two years ago, I've joined the dark side of platform engineering at Form 3. And guess what? I absolutely loved it. Today, I'm a lead SRE engineer there. So what do we do at Form 3? In short, Form 3 is a financial cloud. We provide a payment-as-platform model for quick and easy integration with various payment schemes via REST API. We are technology-oriented, fully remote, cloud-based, with microservices architectures. We treat our infrastructure as code almost religiously, terraforming everything possible. Lately, we are building a new multi-cloud payment platform based on Kubernetes and free clouds. So obviously, Kubernetes security is super important to us. What am I going to talk about today? Just a splash of theory in the ocean of practice, the key part of today's talk features two demos to hijack your accounts, exploiting various security issues in a Kubernetes application, as well as in cluster. But before the fun begins, let's do a quick review of the Kubernetes architecture. There's two main concepts in Kubernetes, the control plane and worker nodes. The control plane basically manages the entire cluster. The heart here is an API server. And when we want to create or update any Kubernetes resource, we need to communicate with that server. It exposes a Kubernetes HTTP API. That way, we can interact with a cluster using tools like Qubectl. And who doesn't know Qubectl? Worker nodes are the place where the real work happens. The actual pods run over there. Qubelet is a main component responsible for instantiation and execution of pods. And like the API server, Qubelet also exposes a REST API. But it's only like internal API. However, sneak peek will exploit an unsecured Qubelet API to gain unauthorized access to the cluster. What are the kind of security issues can we see in our case clusters? Oops, sorry. Yeah, hopefully it won't happen again. But you never know. Fingers crossed. Adjust it always slight. Nope, it's not going to work that way. OK, so it's time to dive into OWASP top 10 security list. So here, OWASP basically stands for Open Web Application Security Project. I'm pretty sure that all of you know what OWASP is. Basically, they aim to raise and increase security awareness among developers. So they publish those top 10 security lists every couple of years for various technologies. And last year, for the very first time, they published Qubelet's top 10 security list. So it's like a hit parade, where we can see what vulnerabilities are the most important or the most popular. So let's see the chart toppers. Number one, insecure workload configuration. So basically here, watch out for root users, for privileged mode, or writable file systems. Number two, supply chain vulnerabilities. Here, you need to keep an eye on how you build, package, and distribute your Qubelet's applications. Number three, don't be too generous with your RBAC permissions, because otherwise, you will end up in a cluster where everyone can be let in. So number four, basically a good piece of advice is to centralize your policies in a way that you reject any misconfigured Kubernetes resources. Number five, if you don't watch your logs, basically you are like a security guard who fell asleep on the job. Then, number six, broken authentication mechanism. Basically, you need to configure your authentication in a way that doesn't allow anyone into the cluster only the users who have permissions. Number seven, with Kubernetes Cloud network security model, we have a problem. Because basically, each workload can communicate with each other. So it's like a house with all the doors wide open. What can we do? Basically, we need to lock them up. Number eight, secrets management failures. As simple as that, protect your secrets. By the way, a colleague of mine, Maurice, is having a talk on external secrets operator later today. So don't miss that one. It's going to be a blast. Number nine, configure and secure your Kubernetes components properly, or otherwise, you will end up with a messy Kubernetes jumble. And last but not least, keep your Kubernetes components as well as your workloads up to date. Because otherwise, you might end up with a bunch of uninvited guests in your cluster. So that's it when it comes to theory. Let's move to practice. And it's going to be like one-on-one course in Kubernetes hacking. But hopefully, you will get the feel of danger. And you will see how a series of simple misconfigurations can lead to a total compromise of the entire system. Here, I will need your help. I have this vulnerable Kubernetes application prepared. It's available on the kubecon.yonlabs.com link. The link is available in the slide as well as in my Twitter account. So please, produce the new accounts, log in, and wait to be hacked. My objective here is to hack your accounts and learn your secrets. So let the fun begin. You can see my Twitter account. You can see the link available here. If you were not able to scan it from the QR code, let's see how my application looks like. So once you log in, you can see your secret. You can update your secret. And you can see active users. So, oh, so many of you, thank you for your cooperation. So once we see that we have so many users waiting to be hacked, let's move on to my evil server. So basically, I will log in to my evil server. That's the server from which I will do all hacking. This server is in the same subnet as my Kubernetes cluster. That's important because I'm going to target OpenCubelet API. So the first thing usually hackers do is some sort of retonnaissance. Usually in the form of port scanning, using tools like Nmap, Netcat, or something like that. Here we are targeting Cubelet API. So we will use a dedicated tool for this one. Basically, we will use Cubelet CTL. Cubelet CTL is a tool implemented by CyberArch. That's a cybersecurity company. And it's similar to Cube CTL. So basically here we have a wrapper around Cubelet API. We also have the scan command. I will use the scan command to scan my subnet for OpenCubelet API. I will use quite a small size of range, only slash 24Y, because I don't want to take it too long. Let's see what we have here. OK, yes, as expected, because I did it. So yeah, we have OpenCubelet API. What a surprise. What can we do with this one? Basically, with REST API, we can just use care and see what's going on here. Let's see that one. We have for a form, not found. What happened? We don't have any endpoint on this path. Let's use paths endpoint. OK, we have something here. Let's use jq. It looks much better, but it's not very readable. So let's search for items with metadata and name. That's going to give us the names of all the containers running on a given node. OK, we got it. Nice. So we see what containers run on this node. What can we do else? We can still use Cubelet CTL and scan command. Let's see help for this one. We have two subcommands available. We have RSC and token. Those are used. The first one is used to scan nodes for pods with remote code execution. So basically, all the pods that allow to execute any commands inside. And another one is token command. And it scans for all the tokens in a given node. So we are going for RCE one. Let's do Cubelet CTL scan RCE. And we are targeting our server. Yeah, thank you. You are very vigilant. But then I am on the stage. OK, so what we have here. We have a couple of pods that have RCE available. Calico node. That's a default installation of Calico. We have demo web server. And we have demo server, like backend here. OK, so that's most likely the pods we are interested in where our demo application is running. OK, so let's try to run some command inside this pod. So the pod is here. The namespace is backend and contain another typo. Sorry for this one. I'm too stressed, it seems. OK, and server is node. So let's run. Yeah, we are root over there. That's very nice. So what can we do now? Usually when you are a hacker and you see that you have a remote command execution, then the next step is pretty simple. Establish a reverse shell. What is a reverse shell? Basically, if you want to access remote server from a client machine, usually you SSH from a client machine to a remote server. You provide your credentials. Basically, you log in and you are given a shell. Reverse shell goes the other way around. So basically, a server connects to a client machine this way by passing authentication step. So let's do reverse shell then. To do reverse shell, I need a second session. I need to log into my evil server. And I need netcard. Usually, you establish your reverse shells with netcard. And you need one point that listens. So I will open listener on my evil server. And then I will establish, execute, basically, a command inside my demo server pod to connect to my evil server. That's an IP 4444 port. And then I want to execute shell. OK. Let's give it a try. Yes, we have a connection. We are in. That was nice and quick. So we are inside a port. We have a shell. So what can we do now? Basically, a game sniff around, do some reconnaissance, and see what else can we do to do more harm. So usually, we check for who am I. We know that we are raw. That's perfect. Let's see mounts. Unfortunately, we are not running in privileged mode. We don't have access to the host file system. However, let's see if we have, like, writable file system here. Yes, we have. So basically, when we have a writable file system, we can install whatever we want if we have access to the internet. But what else we can take a look at? Let's do net start and see what's going on on the spot. We see that the spot is listening on 1990 port. That's nice. It would be good to sniff the traffic. To sniff the traffic, we need to have, like, TCP dump. Do we have it? Let's see. I'm sorry. I'm trying to do a redirect here. Why redirect? Because this reverse shell is not fully interactive shell. So it doesn't show me the error stream. So I did a redirect to see an error. We don't have TCP dump, but we are wrought. Possibly we have APK here. So let's add TCP dump. OK, so now we have TCP dump ready. So let's sniff our traffic, sorry, here on port 99. And now I will need your help. Could you please refresh your pages after logging in? Yeah, thank you. It happened even before I managed to say anything. OK, too many data. Let's stop it and let's search for something interesting, like a better token. OK, we have a better token now. So what can we do with a better token? Now it's simple. We just go to our demo application, and we replace my valid token with one of yours. Let's refresh the page. And I'm Ferriero. Congratulations, Ferriero. You've been hacked. OK, so that's it for the first demo. What problems do we see here? Obviously, OpenCubelet API. Let's take a look at the official Kubernetes documentation. What do we see here? We see that anonymous auth has the default true. Also, authorization mode has that default always allow. We could do better here. However, it's not that bad, because those are the defaults for raw, plain Kubernetes installation. Usually, we use managed Kubernetes, like in Cloud, EKS, GKE, AKS, whatever you imagine. And those have sensible defaults. Also, if you use such installation tools, like Kube, ADM, you are good. They have, again, sensible security defaults. However, if you install your Kubernetes clusters by hand, you can end up with a Cubelet API fully open. So basically, OpenCubelet API is like K9 from our OWASP top 10 list, misconfigured cluster components. Another issue we see is like writable file system in your container. In that case, you can see that basically hackers can install or modify, can install anything or modify the code of your application. So definitely use only a writ-only file systems. Another thing is running as root. As root, we have super privilege. We are super user. We can do almost anything. Another thing is that in containers, in that container, we observed quite a few tools available. So that's definitely not recommended. It's better to use distro less images. However, remember that it doesn't guarantee 100% security. There is that block post from that team lead of offensive security at form free who explains all the corner cases of distro less images. Another thing is networking. I've been doing the sub-professional software development for over 25 years, actually. And I'm not able to count how many times I had this discussion with less experienced developers about unencrypted traffic inside internal network. So it's countless. And always the arguments are the same, like, oh, come on. It's on the internal network. No one can sniff it. Oh, come on. It's all about performance. At the moment, it's all about security. If you want to be secure, basically encrypt your traffic even inside your private networks. Another thing is open egress to internet. Once you have this outbound traffic to the internet open, you are also open to many external threats. Or once the hackers get in, they can download anything what they can imagine. So basically, by keeping those very simple security measures in your deployment configuration, you can improve the overall security of your Kubernetes workloads. And that's not a rocket science. OK, now it's time for our second demo. With our second demo, I will still use this application, however, I will also use another one. Another one, which is like a simple hello application. It says hello, Patricia, in that case. However, it also has like RCE remote code execution available. So to hijack your account in our demo application, I will use this vulnerable RCE hello application, basically to access the port and the workload in the same Kubernetes cluster. So let's see how it works. OK, you see that who am I is www.data. Usually it means that's Apache HTTPD server underneath because that's the default user for Apache. And what can we do here? Basically, I'm like a one trick pony. Again, I'm going to use a reverse shell. But before I go to my reverse shell, I have one ask to you. If you could log out and log in again. Why? I will explain that later. So please log out, log in again, and I will start my hacking then. OK. I'm in my evil server. Again, opening a listener, listening with NetCAD on this 4444 port. And here I have this NC with bash available. Let me know that's not the right IP. Sorry for this one. Too many attempts in the past. I think that's the one IP. I need to provide the correct IP of my evil server. OK, we are in. We are in. Let's take a look what's going on here. Again, who am I? www.data. What about months? Oh, what a pity we are not ruled because we have this pod running in a privileged mode. However, as we are not ruled, I'm not going to go this path. I'm going to go another path. Let's just scan our Kubernetes subnet and see what's listening over there. Here, to avoid and save your time, I have some cheat sheet available. So basically, I'm going to scan only one IP and given port range. Let's do this one. And let's do our scanning. We see that we have this 6379 port open. What does it mean? Basically, this port is a default port for Redis. So let's try that one and send some Redis commands here. Again, I have the cheat sheet available. Basically, Redis server doesn't support REST API. They support REST protocol. That's Redis' realization protocol. Why? Usually because they sent a lot of data, so to save time on the realization and the realization with JSON. And they came up with this specific protocol here. So I'm sending a ping command. Yeah, we have Pong. So basically, we have access to Redis. Let's see what is stored in this Redis cluster. Oh, my god. It looks like JWT tokens. Again, what a surprise. OK. Let's grab one of the tokens. OK. Let's grab this one. But first, before we grab it, let's delete it. So let's delete it. OK. It's deleted. I need this token because I'm going to use it again in our demo application. Maybe that's going to be successful. And again, I will hijack some of your accounts. So let's replace it. Let's refresh. Kamis, you've been hacked. Kamis, I think you are Polish as me. Nice to see you here. And happy that you've been hacked. So what happened here? We had remote code execution in one Kubernetes deployment. And this deployment is running in the same cluster as our target application. So you can see how one vulnerable workload can impact another workload in the same cluster. So basically, that came 10 from the OWASP top 10 list, outdated and vulnerable components. Also, another thing we exploited is missing network segmentation. Basically from this RCE Hello application pod, we could connect to Redis cluster in another namespace dedicated for another application. So that's like missing network segmentation controls K7 from the OWASP top 10 list. Again, quite a few tools available. Reversal possible and one big no-no anonymous access to Redis. Obviously, you shouldn't have any anonymous accesses to any data store or workload in your clusters. So let's sum up what we've learned today. I can only tell you that's like the tip of the iceberg when it comes to Kubernetes security. So I could go on and on about the topic, but unfortunately, or perhaps fortunately for you, we are running out of time, so only some final remarks here. You should remember about the Swiss cheese security model. Basically, this model illustrates the importance of layered security. Like stacking Swiss cheese slices, you have those multiple layers, multiple overlapping layers of your security mechanism and controls. And basically this way you can cover the holes or weaknesses of other layers. So you are only vulnerable if all those holes are aligned in the same line. So basically, instead of relying on a single security mechanism, you need to implement defense in depth, layer security. And your ultimate goal is not to have like a perfect solution because that's impossible when it comes to security. The goal is to make the life of bad guys more difficult and to help us good guys to detect and respond to any potential security threats. Remember that a fool with a tool is only a fool. So don't be fools. Use your tools wisely. Use your Kubernetes wisely. And here in software development we have different paradigms like continuous integration, continuous deployment, some talk about continuous refactoring, but the fundamental paradigm is continuous learning. So we as DevOps should keep educating ourselves. Thank you and thank you for your cooperation.