 Thank you so much, everyone, for joining me in this awesome session at Cloud Village Defcon in a virtual edition. So today I'll be focusing on talking on Kubernetes code. It's a project meant to be trying to teach Kubernetes security by using vulnerable by design labs. So today I will be like a big disclaimer trying to be not showing a slides or mostly focusing on the demo and how the project can be set it up and used as a normal user and learn from it and go back and try out your Kubernetes skills. So without you, let me go head over to the project. This is the project. You can find it in the GitHub.com slash mother of Kubernetes code. As you see, it's pretty much trying to design for intentionally use of like learning towards Kubernetes security. It doesn't matter if you are someone beginner or trying to learn offensive side or defensive side or even some kind of tool which you are trying to learn and see how it is effective. So we kind of look at like some of the scenarios and see how it looks like. Before we dive into the code project, let me guess, you will never see making sure everything is fine. So we kind of have a ton of scenarios in this code, which is like basically trying to take this naming from similar to web code, which is a standard OVAS project for years. And we kind of showcase a different variety of scenarios on different areas, both offensive security and defensive side of the Kubernetes. So let's see how it goes. About me, myself, Madhu Akula, I kind of working towards Kubernetes and containers were quite some time, nearly past four years. Apart from that, I have been trying to present my research and share with the community around the world like Black Eyed Defcon use NAICS or really a bunch of other conferences. I'm fortunate to find some kind of vulnerabilities quite some time back and contribute to the community, especially open source ecosystem. And also part of bunch of communities like now already have apps and also written and reviewed a bunch of books. And yeah, that's pretty much about me. Regarding the project, if you are someone who are getting started with Kubernetes, I would highly recommend learning the basics before trying out this project. So it adds a ton of value than learning directly the security parts, right? I feel that if you understand the technology, the most of the concept which you learn from security can be applied here, right? Sometimes whatever you say firewalls, maybe in cloud world, they call it as like security groups or NACL, right? Something like that. In Kubernetes, maybe they call network security policies, right? So this terminology may differ a bit. I would highly recommend learning about the technology and understand these concepts so that you can make most of it than just having it. If you are someone new to Kubernetes, it's an open source container orchestration system, mainly designed for deploying your applications into the cloud so that you can scale and manage efficiently. It was originally created by a project from Google called Park. And now they don't take this as a Kubernetes to CNCF, which is the cloud media computing foundation. And I definitely highly recommend this video. I think I do a bunch of trainings and workshops. So everyone, I highly recommend to go through this video. It is kind of children's way, how you can explain Kubernetes to someone which is just getting started, right? And yeah, maybe I can just give a very quick, fast-track one-minute version of overview, which is definitely hard, though. In Kubernetes, there is a concept like master and node, like where it's like client server architecture type. So the master holds a bunch of components, which means master node has something called API server, which is like a brain of Kubernetes. It doesn't matter if you are sending a request from external world or internal components, which are like something like their own processes, which is talking to the Kubernetes, right? So it has to go through the API server. So think of this as a quite critical component in the Kubernetes like a brain, right? Once it sends, so the XED is like a database. It's kind of key value store where it stores all this data. And okay, let's say as a developer or DevOps team, I sent, like, I wanted to scale like 10 NGN exports in a Kubernetes and run it, right? So to answer this, like where I have these resources and everything. So scheduler is the job to pick up like how much resources I have in the nodes, where I should schedule and all these things, right? So controller manager is like kind of the way how it's like, talk to this XED and making sure these are running always the declared or desired state, right? So this is a very high level. I think there is a ton of things once you get into the details. Similarly, let's say once scheduler says I wanted to deploy this pod into some node, right? So Kubelet is another process which sits on the node, making sure talking to the runtime, it doesn't matter. It is Docker, RunSeqata or some other container runtime. So it is talking to the container runtime, which is in the node to schedule the pod, which is told by like API server, right? And it kind of works across like sharing this data here from to the API server as a way, right? And another thing is like a Kube proxy, which is like traditional IP tables rules, right? So which kind of connects these pro services across these nodes to serve to the users and applications and services, right? And all your pods are deployed here, pod is like a minimalist unit. It can have one or more containers, right? So this is like a super quick one-on-one view, but I would highly recommend refer to the documentation. They have ton of really detailed things. And also, if you're someone trying to get started, I would highly recommend this Kubernetes by examples. If you stuck something, maybe I want to just know what is exactly is namespaces or volumes, right? Maybe you can quickly get your hands on on top of it. So these are quite learning material if you are someone getting started, okay? So let's say that you already be comfortable with Kubernetes, like now I wanted to go and understand what kind of security things I can identify issues or like learn and secure the cluster, right? So I feel because I have been doing trends for quite some time, it's tough to people to set up these environments. So what I have done is I tried to create an online playground, which is just by clicking in a URL, you can start a browser environment where you can play with this project and try to learn as a concept rather than focusing on your setup and understanding the complexity of the things, right? So you can go ahead and start here, like a click on the launch, which means that you will start a Kubernetes cluster for you within the browser. And once you run these two, then these two kind of sets up a project for you, right? So whatever the guide you see, it is detailed in a way like everything is documented, what all things you can go back and try. Definitely, I don't have enough time to cover all of these scenarios, but I try to document each and every step by step so that you can go back and apply this learning knowledge, right? So while it is trying to set up, like you can see that we just set it up and we hit the enter, it will go ahead and set up some of the scenarios which we are going to play around today or after later as well. And we can able to get the access after this, right? So for the sake of the time, I try to set it up already. I can just type, QQed will get nodes. You can see that there are a bunch of nodes, sorry, again. So you can see three nodes running within the Kubernetes cluster and which means that three node parts of the nodes are there in the Kubernetes side. And you can also see bunch of parts, which means number of containers, one or more running, right? In some cases two, some cases one, right? So this is like kind of trying to replicate a real-world cluster with the intentionally vulnerable these to showcase how Kubernetes security we can learn, right? So if you are someone like playing this using CataCoder, definitely you can set it up your own. So you can also do like making sure the parts all are running before you hitting the last script, which is nothing but trying to expose all vulnerable apps or services within your local system so that it may not expose to the Internet, someone like a real attacker may not just exploit your cluster, right? And also I would highly recommend, maybe I didn't go through the disclaimer a bit early. Maybe I think I might have put, please do not try to put or run your Kubernetes code project on your production workloads or in your company workloads, which basically has intentionally vulnerabilities. So it doesn't come with any warranties or guarantees. So making sure try to run in your safe environment, try to run in a play environment where you can just write out things, right? Cool. Once you set it up basically the project, then you kind of return with a URL basically where you can access the Kubernetes code, right? Like which is kind of like once you run this script, you end up seeing this. So this is how it looks like. This is kind of a home for the Kubernetes code where you can find up all the scenarios which is available and how you can get started with them to learn or practice, right? So if you see here, definitely this guide is quite handy. Like this is what we are trying to go through today as well because I can't remember everything for sure. And there are a ton of other things also. So you can, if you stuck somewhere when you are trying certain scenario, you can come here and try to refer certain example or certain scenario more details, right? So yeah, let me just see making sure I'll try to showcase you how you can access this as well. So once you hit the bar script of access Kubernetes code, it all exposed. So now you can go ahead and use this project and it is making sure it's up and right. So if you are someone trying to browse certain other scenarios, like let's say SSRFNK as well, right? Rather than browsing, if you are using the CataCoda, I would highly recommend use this custom port and mention this port number and just use it, right? So that simple as to just set up the project and get started your learning of community security, right? So let me go through some of the scenarios and showcase both attackers or offensive side as well as defenders and how you can learn security of the system. And also see last part, how vendors like enterprise security or open source projects can leverage this as a tool to see how much gaps or improve their projects as well, right? Our products. To quickly showcase a simple attack factor or scenario, I think this is quite commonly you see in day-to-day CACD system, right? Docker in Docker. If you are someone using any most of the build system like Big Lab CIR, Bitbucket Pipelines, or GitHub Actions and all, you might end up seeing something called a TIN, which means Docker in Docker. So if someone building a Docker command or like a Docker container within your build system, they might want some way to access your Docker socket or Docker API so that they can build that image, right? Definitely there are some alternatives keep coming in the world. So most people still to date I see especially when testing engagements are in real world, people try to use this method to build that thing. So what I have trying to showcase is how we can exploit this or gain access to the underlying host system by escaping from the container to the host system, right? So to get started with this scenario, you can go ahead and click this URL, so which has an intentionally vulnerable lab or like application. So as for this demo, I don't want to focus on application security issues. So I'll try to go ahead and showcase you that this application is vulnerable to remote code execution, which means that it has like a C, right? Command injection vulnerability. So from here, which means that you have to identify what is this it is running in and how we can get escape out of this container by looking at different paths and aspects, right? So now one thing as I look like or maybe I can look at like my process, okay, where it is running exactly. So you can go ahead and run cat procs help C group. It kind of tells that, okay, it is running and Kubernetes pods. So it tried to showcase you the runtime sometimes and also there is tools like MI Contain by Jessica Frazell, which also kind of showcase and give you the container information. So now once I identify, definitely you need to understand, is there anything which is running in the file system or is there anything which I can think of like something interesting, right? So as I already know, this is intentionally built and trying to put some vulnerabilities. I can see that there is a Docker socket mounted here, right? It is in the custom Docker and Docker.soc, right? So which means that there is a Docker socket which is passed from the host system to the container, right? Maybe I can just showcase in a nicer way, maybe. So this is your application running, which is in a pod or container, I would say, right? So which is running in a node, which is in the Kubernetes node, right? So now the Docker is running here, like which is where run a Docker.soc, right? So now what we are trying to see is this Docker socket also passed to the container in a different part, which is like custom Docker or Docker.soc. So now we are trying to leverage this Docker socket and talk to the host system because this is mapped with the host system and see if we can escape out of this pod or container and gain access to node, right? Like that is what we are trying to exploit and see if we can gain access, right? As I said, you can definitely go through this documentation and you can see this, how you can leverage. Maybe we can use Event Curl basically to talk to the thing or you can even use the Docker, right? So there is no Docker. So the one way in real world attackers can leverage is definitely you can download the Docker because it's just a binary, you can duplicate or like curl and exploit this system. Or in another way, maybe you can even interact with the Docker by using call with the unique socket by passing, right? So for this thing, you can just download this and make it available into a local system as binary. Now you can see, I tried to already put it here. And now as a attacker, if you want to interact with the Docker socket, you need to specify the iPhone H flag so that you can talk to this specific Docker host socket, right? So now you can see here, I can just say that and I can also query images, right? So as you can see here, you already got bunch of Docker images which are in the host system, right? Which is basically now we are trying to query the node Docker container images which are here underlying in the node system. So now as a attacker, if you wanted to basically go exploit this and get a shell back or something, maybe you can just run a container or volume on everything which is in the host system with either read or read, right? Whatever the way you want. Or you can even pass a flag like privileged equal to true to get a privileged containers if you're adding extra capabilities or something, right? Or you can just run your own container like I can create my own container saying that it gets a reverse shell back to me in this container, whatever the thing, that's it. Once you run basically shell back from there you can traverse and learn more attack surface and exploit more. But for the sake of the demo, I just will try to stop here. But definitely in real world, there's a lot of little movement and attack exploration from here, right? So this is a typical example of how you can exploit Docker in Docker sockets in a Kubernetes clusters are in general, right? And there are a ton of other vulnerabilities. Definitely I try to not able to cover but you can go back and definitely explore like there is another vulnerability where if they have extra privileges, right? Maybe you can just see if they have so many privileges by running that type of thing. Now you can see bunch of other capabilities it has already given, right? Maybe you can see that there's extra file system mounted or like a phone in the whole system, right? There is a whole system. So you can definitely leverage the traditional way of like a CH route and slash host system and whatever the command you want to run on top, right? So now if you see here, like we are trying to basically leverage this and try to run commands or system within the host system. So now whatever the commands you run, which are going to run within the host system by escaping out of the container, looks like there is no containers running. I know the reason, which is most probably due to the reason they are might be using continuity because the underlying host system is changed. But you can see talking images works pretty fine, right? So this is maybe another way to escape. And from there, maybe you can use and leverage this kubectl config, like which is available in most of the container nodes, right? So which is in QTC, Bernadies, and you can see kubectl config. So this is a kubectl config, which means that it used to talk to the master node, like as you see when you describe the architecture, right? So as an attacker, you can also leverage this config like kubectl dash dash, kubectl config and pass the flag of whatever the config which you found, kubectl config and get pods, right? So now we can use this to query if any pods available within the cluster and see if we can be able to use this privileges or something to get the data, right? Looks like, yeah, cool. Now we can see, so what we just did here is, now we kind of escaped from the host system by leveraging the host system file path mount and we use this vulnerability to run here and also we leverage the kubectl config to talk to the API server. And from there, you can also leverage, see what kind of privileges it has and from there, you can have, go and pick more authorization, authentication exploits, right? So I can also query using, can I delete nodes? Sorry, can I delete nodes? So this is kind of imperative command to query to the API server, asking whatever the config I found, kubectl, using this, can I do something, whatever operation? So you can see this configuration or the permission which we have, we can't be able to delete nodes, right? Maybe we can create pods, right? So we can use this extra permissions to maybe create another pod within another node, maybe you can hop into another node. From there, you can have to do lateral movement on top of each node and node and maybe gain more and more data or container access, right? So this is how a typical, maybe offensive side, you can definitely look, and as I said, a bunch of other examples and scenarios, you can find like how we can consume resources and memories when they don't define resource limits and quotas are limit ranges, right? And if you look at like example of defense, right? Maybe I'm trying to show you as a simple use case of open source utility called like Falco, which kind of help us to do monitoring of runtime security, right? So I want to deploy this simple Falco deployment into our Kubernetes cluster and I will go ahead and update this repo in my help and I will install this Helmkart into the Kubernetes cluster, which is available, right? So now we can see this, it got deployed. Now you can see the pods running of the Falco in the live, which is basically creating. And the reason you may ask why these are running three, right? If you look at like nodes, which we have, three different nodes. So Falco currently runs with a daemon set, which means that if we have a number of nodes, so each node will have one pod running so that we wanted to collect all the data and metrics so that we can monitor the each and every node level, the runtime and all this data, right? Containers. So that's why it is kind of running with a daemon set. So if you go ahead and see the pods, you can see that they're running, up and running. So now what we can do is Falco has like a bunch of ruleset to detect certain kind of security issues within the Kubernetes or in general Linux, right? So there are something like, if someone tried to browse some sensitive files like Kati, Idisi, Shado or something, it may trigger an alert, right? So what I'm going to do is, I'll go ahead and query these logs from this Falco, right, from all these pods and also try to simulate some kind of attack or vulnerability, something which Falco triggers and see how we can detect these as a defense point of view, right? So let me go ahead and query these logs. Now you can see the logs coming. So what I'll do is I'll go ahead and create a container which kind of execs into a bash and you can see I'm trying to use a hacker container. Maybe I'll come a bit later, what exactly it is. Let me go ahead and run this container. If you see me, I just started a container and executing into the bash of that container, right? Now, okay, I'll just quickly showcase this example. I think then I think I'll try to cover the last section. Yeah, so now as you see, whenever some kind of new triggers happens within the Kubernetes cluster in terms of security point of view, you can see these logs coming up, right? So let me go ahead and do the simple example, right? Kat Edishi Shado, whenever I read the file sensitive because as you see, Falco has a rule thread for that. It automatically detects that which container it got triggered and which file it has read, which container, which node. So it kind of provides a detection or defensive mechanism to understand, identify like a security point of view, what kind of things happening around the system and all. So we try to focus on different scenarios, how we can teach this defense side as well in terms of Kubernetes security and how we can leverage these things and like network security policies and all. As I'm trying to come to the end, as I said, please give it a try. There are a ton of other scenarios which I couldn't cover. And also I try to showcase some vulnerability reports from some of the tools which are open source as well as the commercial, maybe not commercial to see how they can detect these things as part of your CACD file system like privilege escalation is allowed. So maybe you wanted to detect them. So these are also quite helpful when you're trying to learn and how we can apply in your organization and all. So with that, I think thank you so much once again for giving the opportunity. And I would highly recommend give it a try and see how it looks like and share the feedback. And also if you like the project and if you use, please share us some love by giving your handle our name so that we can maybe helpful for us to take feedback and improve this as well.