 Your support for supporting the DEF CON community and the Red Team Village specifically, we're looking forward to your presentation. So take it away. Thank you so much, Omar, for introducing and having me here. Let me just share the screen and confirm. Can you see the screen? Yes, you're all set. Awesome. Perfect. Thank you, everyone, for joining me. And first of all, to just give some context. Thank you so much, DEF CON Red Team Village, especially in this pandemic times to make it virtual and people can enjoy these talks without any physical intervention. And this talk is about the Kubernetes code. So, disclaimer, basically, today's we will not see any slides, basically. And it's mostly everything is in online. You can play along with me, or if you can go back and try yourself, you can go ahead and try as well. So there will be no slides, mostly live demos and the hands on exercises. Cool. So this talk is primarily focusing on introducing a project recently I have created called Kubernetes code, which is this. This is at ingithub.moduocla Kubernetes code. This project was mainly intentionally vulnerable application, which is to basically try and teach participants like anyone come to the new to Kubernetes to understand how you can assess this Kubernetes clusters and find vulnerabilities so that you can go back and write out in your organizations, day to day pen test, which you find are a team exercises. Right. So I have presented at some of the places and also I have created a playground. If you wanted to write now, try it in your browser and follow along. Right. So a little bit about me, myself, Madhu Akula, I work at currently Zibia as a cloud native security specialist. And I have been speaking and training at bunch of conferences around the world, like DevCon, Black Hat, Usenix, Orelia and some of them. And I'm fortunate to find some of the vulnerabilities long back using my research in some of the organizations like Microsoft, LinkedIn, Google and some of them. So apart from that, my primary interest mostly working around security of cloud containers, Kubernetes and automating the infrastructure part. So that said, a little bit about me. If you wanted to follow me, you can follow me at Madhu Akula in most of the social media stuff. Cool. Let's get into what exactly is this Kubernetes code and what we are trying to learn like within this next one hour. Right. So to give a little context about Kubernetes code. So as I said, it is intentionally vulnerable. I think as most of you have come across the different types of code like web code or Terra code or some other codes. So this is a similar project, but it is mostly focusing on Kubernetes to understand what are the known vulnerabilities. And also we have seen common vulnerabilities in the real world environments, like while performing penetration testing or like while performing assessments in Kubernetes environments. From those we have taken as a scenarios and these are the some of the scenarios. Let me zoom a bit. I think maybe easier to see. Yeah. So these are the some of the scenarios I have created and we will see some of the scenarios, exploitation and how you can bypass or gain access to Kubernetes clusters as an attacker point of view. Right. And I'm really sorry and I'm shameful. I said I'll keep update this slides at the documentation. And also I have been working on the defense part also. Hopefully this month we will see them coming. This part I'm going to skip. So before diving into Kubernetes, if you are really completely new and coming to the containers of Kubernetes world, I highly recommend understanding the technology because there is a fair amount of gap in between penetration testing or finding or assessing clusters. If you don't understand what knowledge you are doing, right? So we have seen commonly, especially in the modern tech infrastructure world, there is a huge gap for the security team to understand the technology before even testing or trying out. Because once you able to understand, you can able to give your best up, you can able to possibly find most of the out of Kubernetes or any ecosystem. This is a very funny video and it's a simple children's guide to explain what exactly Kubernetes is. So I would highly recommend if you want to go through these things. So to give you a little quick context, how Kubernetes architecture looks like at a higher level, right? So this is taken from the wiki comments. You see a server and client architecture similar to most of the server client. So one thing you have to observe here is in Kubernetes, everything which you make any operation, it may be from the within the master, which means server within the node, which means worker node. So with whatever the operations which will make like it can be from the node, it can be from the developer or any user, any service will go and touch the something called API server. So which is one of the key component in terms of Kubernetes because Kubernetes is completely driven by API first approach. So whatever you make call request or something, everything you will see going through the API server, right? And control manager is basically to make the decisions, whatever the request you make and scheduler is ensuring to place the workloads at the the the request which you made. Suppose if you wanted to serve a engine X server with 10 parts, it will ensure that how many nodes has the capacity to schedule. So the scheduler job is to figure out that and place them accordingly. So at CD is one of the key value store database, which is used by Kubernetes, which is as it names are just key value. So it stores the data which is coming to the API server and stores the key value base. And we will see a little bit in a later coming sections. So apart from that nodes, when you come to the node cubelite is the talking point to the API server and communicates within the local node itself. So whenever you wanted to make a request as a scheduler, wanted to schedule in the node, it will talk to the container runtime, which is within the node, like it can be Docker, RunSea, etc. So then it will go ahead and schedule based on the workload kind of thing. So cube proxy is one of the another key component, which is kind of like IP tables, like it will try to provide a communication across like nodes as well as a cluster. So let's go ahead and a little bit go deep, but I would highly recommend you practice the Kubernetes before you diving into the assessments or security testing. So here are the some of the references I have curated. These are pretty handsome, like if you wanted to try it, so you can go ahead and try as a like a literally playground and also specific objects. If you start while testing, you can click on understand what exactly it is. So these references I would, I think it would be definitely helpful to try out. Cool. Setting up Kubernetes code, the Kubernetes code is in design in such a way it can set up any out of the box. It can be in your existing cloud provider services like GKE, AKS, AKS or Distillation Base and also your locally provisioned mini cube or another playground to make easy off setup for the people to try out and learn. I have created something called CataCord of Playground. If you just go ahead and click this, it will open up a playground for you, which is completely free and online. So you don't need anything to be done. I think you have to register first time. So then you can get the complete setup here for you to practice and learn. Let me put it in a presentation mode so that you can see a little better. Cool. So I'll walk you through how you can set this up and practice. And if you wanted to follow along with me, you can follow this guide at madhuakula.com slash Kubernetes code. This is the URL. Cool. And if you wanted to set up your own, you can just follow this documentation. Basically, you need to have a working Kubernetes cluster setup with admin access. And also you need a Helm version too, because in one of the scenarios, we will try to focus on how you can exploit the existing vulnerability in the older version of Helm. So that's it. So if you wanted to set up your own, let's go ahead and do that as well. So if you have your own set cluster, you clone the repository, which is the Git clone of the Kubernetes code. And then you can go ahead and set up the Kubernetes code. So it checks if Kubernetes cube cut is available, Helm 2 is available in your path and it will go ahead and deploy the tiller, which is the server side component of the Helm. And a bunch of vulnerable services, which will be used as practice to exploit or try practicing a Kubernetes security system. So if you are someone following the Kata Quota live, you can use this playground, which is completely free and online. You can go ahead and do the same thing in the playground as well. So the playground is designed in such a way, you can go ahead and see here. Only the guide in the second in the specific tab, and also you can browse those boards and other stuff within the browser itself, even in your mobile also you can try this. So it takes a while to set up the entire Kubernetes code platform. I think it's almost done. So it creates a bunch of vulnerable scenarios, which I have pre-created. You can go through the code and you can able to see what exactly it is trying to do if you wanted to look at. So this is nothing but it checks if kubectl is existing or Helm to existing and it sets a bunch of emails, which is nothing but deployments in the cluster. Cool. Now we have the cluster up and running and we have Kubernetes code setup. So what's next? Right. So I think looks like here also it is done. And sure you check all the ports are running, you can go ahead and run kubectl getPorts. It is currently container creating, which means it's starting. It takes a while to set up the depends on the way you are running the Kubernetes cluster. So let me go ahead and do this as well here. kubectl getPorts. I hope you people can see the screen. I think let me zoom a little bit as well. So you can see bunch of them. Most of the ports are created. I think just only one port is waiting to get created. That's awesome. Now you can see all parts are running and I think this is intentional. This is a job and get company. So if you see the status, basically your entire Kubernetes code is set up and running. So now it's time to hack and get into the cluster. Right. Cool. That's where we go back and refer to the documentation. So if you look at the scenarios, so there are a bunch of scenarios we have created as I said. So for the time being for the entire today's presentation, we will not be able to look at all of those scenarios, but definitely to be looking at some of the primary scenarios, which might be interesting and fun to try out. And I would highly recommend go back and try out yourself and check out more other scenarios as well. And in some cases, especially, not all of them scenarios, but in some scenarios, I try to create a flag in this format. K it is good dash good. And some MD5s, basically it's a number of 32 characters. Then if you reach this kind of flag, basically it means end of the scenarios, but it may not be found in for all the scenarios, but I try to keep certain scenarios to reduce the effort of like going and beyond post-exploitation. Cool. So let's go ahead and start with the scenario one, sensitive keys in code base. So I would recommend before doing that, I think if you are setting up locally or in the same place, all parts are running. Make sure you run bash access Kubernetes code. What it does is basically put forward all the parts or services into your local system so that you can access locally without even exposing to the world, which is internet, right? So now if you go ahead and click this, basically you can see the homepage, like which is the Kubernetes code homepage, right? So this is where you can see when it is good is running. You can follow the guide here. And also if you wanted to check out each scenario, the scenarios are here. So let's get started with the bus scenario, which we will look at. One of them is called sensitive keys in code bases. So the Kubernetes code is designed in such a way. If you wanted to practice as an attacker or learner, you can go ahead and try this scenario by clicking here and starting with attacking. Or if you are someone who wanted to learn and practice yourself and follow some kind of guide, we have created the documentation. The solutions will be here in the same guide for each scenario, like you can follow them as well here if you stuck somewhere, right? And some scenarios we try to create two different methodologies. It can be attacked in different ways. So we try to document in the same way. So if you see here, let's go back and read a little bit about the scenario. Developers tends to commit sensitive information to version control systems. As we are moving towards the ACD and GitHub systems, we tend to forget identifying sensitive information in code and commits. Let's see what we can find something cool here, right? So by looking at, you can see that some kind of vulnerability or some kind of mistake has happened. Now, as you can read this, this is version control using it. So what I can do is I can try as a dotkit and you can see the default file in the git will be config. Now you can see that there is a bare repository hanging there, right? And by the way, these vulnerabilities, which you see, I was showing like in the web vulnerabilities, these are just an entry points into getting into the cluster. In real world, it may not be the same vulnerabilities you point. Maybe there is a different approach you find to get it here, right? That's the thing. So now I can see that there is a git config. Let's go ahead and see how I can clone this repository locally so that I can understand this code base better, right? So if you want to start, basically you can go back and follow this repository here. So there is a tool called git dumper. It can be any other tool as well. I just take an example. So you can try this tool by running Python 3 git dumper localhosts and save it like this. Maybe I think I have configured here. Git dumper. So what I'm trying to do is I'm cloning the localhosts1230.git folder into local system. So what it is trying is it is going to fetch all the, the repository and clone it locally, right? So now if you go here, kdisk.git, so that is the git repository which we have. So if you see here, we can see there is a golang bunch of files, min.co and a bunch of custom code, right? And also you can see git, the folder. So you can also look for what are the old commits which are happening, right? As the developers sometimes make mistakes, by the way, this scenario is not only focused towards containers or Kubernetes. In general world, we have seen these kind of mistakes in most of the modern software development, especially in cloud deployments or any internal deployments, right? So I can see some of the bunch of commits as you attacker, you can go and check manually, like git checkout to specific the log or commit. And also you can see, oh, you can see an interesting file called env, right? So there you go. So you can see there is the access key and secret key for the AWS has mistakenly committed. And you can also see there is a, the kdisk code flag is there. So basically this means that you have reached the end of this scenario, but in real world, you can use this AWS access key, secret key and get the session token and you can do post exploitation and go along beyond. So these scenarios are just we have taken from what we see in real world as a pen testing engagement and some of them are like in a common scenarios, which we see in real world, right? So this may not be possible and this may be possible in different use cases. So feel free to give it a try and understand what exactly we are trying to achieve here. So basically what will happen is in the same use case. So we can try in different method, like what I'm trying to do is most of the time developers mistakenly build a Docker images, which may be committed with the dot git files because they don't ignore these files when it is creating into Docker container, maybe Docker ignore or something. So which might end up pushing this data into the containers as well. So looking at these kinds of sensitive information is very useful in the container. Like the another methodology you can try as I said, explanation, so you can get the pod name, you can exec into the pod. This is like another method. Now we can also use tools like Trufflehawk. So Trufflehawk, I think most of you know, this is a tool, which is basically finds high entropy or sensitive secrets in a git repositories. So now I am trying to do Trufflehawk and given the current is the current directories, where git repository is there. So what it does is it will check through all the commits and finds is there any high entropy values or like sensitive secrets found based on the regular expressions and return to the studio. So you can include these kind of tools in your CICD systems to detect these kind of vulnerabilities. But it attack us point of view. If you find any of the code basis which has git or something, maybe it's a really definitely good way to check out in a, especially the container environment. You might end up finding some kind of sensitive information like you can see the same thing. Cool. So this is our first scenario, which we just tried from the Kubernetes code. And similarly, as I said, you can try out here as well. So what you can do is you can look at here and if you wanted to specifically browse specific port like one to three zero in the custom port tab, you can just open in a new browser window. So if you press this icon, then you can just say, I wanted to browse one to three zero port. So you can able to browse the same thing and apply it in online yourself for free and understand what exactly you can assess that test for Kubernetes security. Right. Cool. So that's kind of one of the simple scenario. So let's go ahead and see what could be the sum of the complex scenarios or different kind of vulnerabilities you see in Kubernetes security assessment. Right. So let's go back to the sum, maybe a little down, especially like attacking private registries or Kubernetes CS benchmarks or containers. Right. So let me go and showcase you how container escape to host system works to browse to the scenario. Basically, you navigate here. So this is basically one of the pod inside Kubernetes cluster running with some specific privileged access has. Right. So to set the context here, most of the times, especially operations teams may tend to require to run specific containers with high privileges, maybe giving taxes admin or host privileges or privilege context, which is full admin CS admin capabilities. So these are things. And also to just give an example, how we found in a real world, I have seen, especially while we're performing a penetration testing engagement. So most of the developers get an access to completely one name space with a full privilege. So if your clusters doesn't have PSPs, which is nothing but pod security policies, developers can basically has full access to that name space, whatever they can do, they can deploy a pod with specific privileges, which they can gain into the whole system access. Right. So also, this is one of the use case we have taken to simulate our showcase, how you can find these kind of container escape and gain access to the whole system. Right. Cool. So what we are trying to do here is we can try and look for some bunch of information. Most of the time look for print ENV. So I think I print cool. So most of the time you might see some kind of stuff here as well in print ENV. And also I'm checking for mount. Is there any mounted directories? Now you can see there is a mounted directory for host from host system and which is into the container, which means basically in the pod in this Kubernetes case. Right. So now what I can do is. Uh, I think, uh, let's look at what exactly the host system is doing. So if you see there is a whole system mounted, like which is basically from the whole system to just give an example. So which is nothing but you are underlying host system, which is node is mounted into your pod as a volume, right? So due to the reason we have extra capabilities, right, like this container has bunch of extra capabilities as well. So using those capabilities as a attacker, what we can do is, uh, we can like try to. Gain some kind of access into host system. What you can do is we can say it's true term, the host system because we have capability of the privilege and now we can see that it is given that the host system safe route. And now what we can do is we can perform the internal commands, which are specific to system. Right. Like now you can see Docker PS. So these are nothing but running inside the host system, which is not a container. So you went below one level from the pod, uh, to the host system from there, you are executing commands, right? So, uh, the, the possible, uh, attack surface here you can do is, uh, to do post exploitation might be basically using the, the host system configuration, which is nothing but node level, uh, Kubernetes configuration, uh, to go and, uh, try attacking other nodes or to gain access to other nodes, uh, which basically you controlling the entire cluster by passing through, passing through, right? So for example, what you can do is, uh, you can use the configuration of node, which is available in the cat, where, uh, let me do cat, where, so you can, uh, use the configuration of cat, where, live config, which has the node level configuration. So you can use that configuration and perform the, the Kubernetes commands, which is not only available for the existing namespace specific, it can also perform at, uh, the node level six, uh, whatever the node has the six, uh, the service account, you can able to do that level, uh, to do that. For example, you can do this command. Oops. Yeah. Oh, cube curtail command is not available. Let's download cube curtail. Uh, let me download this. Cube curtail here. Uh, it just takes some time. Hopefully it is faster. Yeah. Yep. There you go. So what we are trying to do is, uh, we are using now, uh, cube curtail command and passing the cube config flag, which is using the node configuration and getting all, like whatever the, all resources in namespace of cube system, right? Oops. I'm trying the same thing again. Oh, I have to do CH mod plus sex. Good. Cool. Now you can see when I run cube curtail, cube config with the specific configuration and get all in cube system, it will go ahead and execute that command and get all the resources. Now you can see here, uh, it can able to get all the parts which are not restricted to specific namespace, which is the developer or, uh, the user has, and you can able to explore more, access more resources and also services. And, uh, you can see some of the, the specific use cases, especially, I think the replication controllers and other jobs and secrets. You cannot able to access. So the node, whatever the access it has, the node configuration, you can able to perform same level. So as an attacker, you can do post exploitation, basically deploy the another pod, uh, in a different node, and, uh, you can able to pass to the another node and again get access to that. So for that, you can just look for how many nodes you have, uh, like currently two nodes, maybe you are in this node, then you can deploy another pod in the different node and get, again, access to the different. So you can basically, uh, use this kind of hopping methodology and hop on to another node and gain complete cluster access. So this, uh, is mostly, you will see if an organization, uh, the, the company, which is not running, uh, the pod security policies, you can able to perform these kind of, uh, uh, attacks or, uh, escalations of the container steps, right? So you might, uh, use a PSPs as a defense for these kind of vulnerabilities and, uh, it would be a easy and simple way to fix these kind of, uh, exploitation. Cool. So let's go ahead and see another, uh, use case. I think we have some more time to see two other scenarios. Let's take an example of Kubernetes namespace by Spas. Okay. I'm going here. Kubernetes namespace, uh, by pass. I don't know exactly. Yeah. So by default, Kubernetes uses flat networking scheme. I think this is, uh, one of the, the default feature of Kubernetes. So by default, Kubernetes uses flat networking scheme means I can able to talk to from any part to any part by default, there is no firewall rules, which will happen, allow or deny by default, anything can talk to anything within the cluster IP range, right? So which means that whenever someone, uh, sorry, our ops team or like security people says that there is a namespace we have created in the Kubernetes, which means it's just a logical boundary. There is no way you literally segregating, uh, between the different namespace unless until apply some kind of, uh, security policies like NSPs and network security policies, right? So if, if you are testing a Kubernetes clusters where there was no NSPs and there was no restrictions between the namespace as attacker, you can able to talk to other namespaces and other services within the cluster, uh, by using a standard network security tools, like which we use day to day, like NMAAP or, uh, the, like ZMAAP or bunch of tools, right? So to just get started with the scenario, I have created something, uh, an example. So what we are trying to do is we are running something called hacker container, which has, uh, again created by me, I have while performing bunch of Kubernetes security testings, I thought that there are so many tools we require, especially focused on container environment. So I have created curated list of tools, which are maybe 50 plus tools, which will help you aid in while performing Kubernetes security assessment. So this container has those all tools, like while you are performing Kubernetes security assessment, and you can use this as a, like a Swiss army knife, uh, insecurity assessment, right? And I have seen this has been very valuable while performing assessment. So definitely I'll try to create some kind of documentation out of it. Maybe hopefully you can go back and try out, uh, different use cases with the hacker. I think it's going and deploying there and cool. So we got a shell. So what we are doing is we are inside a pod, uh, in a one of the default namespace, like if you wanted to just see, uh, let's go ahead and run some command called cube. Cuttle get pause. Now you can see hacker container, which is running in a default namespace and, uh, let's go ahead and see what else we can do from the shell. Right. So as attacker, you are in one of the pod and you don't know what else you can do from there. Right. So as I said, hacker container has bunch of tools like Docker Ben security, cube Bench, Linus and a bunch of other tools. So one interesting thing, uh, you can do especially is, uh, looking for the entire cluster ring. As I said, by default, Kubernetes uses flat network and scheme. You can look at the entire cluster range, right? So you can do IP route. You can see this is using 10 dot slash eight network range. And also you can look for what is the current IP address and 10 dot zero dot zero dot 12. So you can scan the entire network and identify services. So one thing I can confirm, I can assure is, uh, most of the times, uh, internally within the cluster as a micro services or services which are running. Many of the developers or operations teams tend to forget or assume that it is only exposed to internal networks. They will never use authentications or authorizations and you might end up, uh, seeing some kind of bypasses, which is, uh, exposed internally itself, especially Mongo services, elastic search, uh, instances and also even some micro services because the API gateway terminates at the ingress level, uh, which means before exposing into the world like using load balancers, they will terminate at the ingress level. So you might end up seeing some kind of services, especially the application services also don't have authentication authorization checks within the clusters, right? So this is a really great place to explore and do as much work on you can do and you can gain as much as sensitive information or services access. Cool. So what I'm trying to do is, uh, here I'm exploring same and what I did is I'm trying to use the map, which is, uh, similar to Nmap. It's trying to do a scan for the entire cluster range, which is, uh, to do a faster way. And, uh, I'm specifically looking for a port of six, three seven, which is a redis instance. So I assume that there might be a redis running or like a nine twos double you can also try, which is elastic search or Mongo or MySQL or any other services you can scan. It's like traditional security will apply here, right? Let's go ahead and run this Zmap and see what we can see. So, oops. Okay. So we have to have enable the internal network range, uh, allow within this list. Let's go ahead and do that. Let's comment this and run again. Sorry. Oh, okay. You have to do another thing, which is specify the gateway where this Gmap is starting. Let's go ahead and do that as well. F and G capital. Okay. And what is the gateway? Yeah, this is the MAC address. Cool. So, yeah, so what it is doing, it is trying to scan the entire cluster range for the specific port 6379 and results the output. If any of the IP is running or services running or what is running with the specific port open, it will save into the results.csv. It take a while. I think it's pretty fast, but in this case, but this will take some more time to go through and scan. But as I said, in general, we have seen in real world a lot of services running, especially elastic search instances where you can look for without authentication and where you can gain access to indices and the log information, especially audit logs and sensitive information and also manga instances without any authentication or also the microservices, right? So this is a pretty gold mine if you are inside a pod and without NSPs, network security policies applied within the cluster, right? Cool. So this will take a while. I think 76 percent and all stuff. Yeah, as I said, similarly, you can try out those vulnerabilities here and follow the guide if you wanted to do as well. And I think it's a simple way to try out this entire part of the environment within the browser. You don't need to set up the entire cluster itself locally and also once you are done, you can just quit the browser. It's all done like you don't need to clean up the entire environment. And another disclaimer, I think I forgot to mention in the beginning, do not try to run or set up this Kubernetes code in your production environments or in your office because it has intentionally vulnerabilities introduced due to the application flaws or something. It is a vulnerable application which may have gain access to your production or wherever you run these environments. And at least I try to take care using exposing the ports within to the local host, but it's highly not recommended to run in production environments. Cool. We have Zmap completed. Let's go ahead and see what we got from the results. You can see that there is a the IP which has 6379 port is exposed within the cluster and let's go ahead and see what we can get out of it. As you know, it's ready. As I said this the hacker container has built-in tools to look for these kind of common attacks and all stuff. We can use ready CLI and I forgot to be honest how to access iPhone H and the IP and now you can see there is another integrated redis running within the cluster you can able to access because there is no network security policies applied which is by default in the cluster means you can able to access without any issues. I mean so they don't have any allow list or denial list within the cluster. Now you can do enumeration standard like get key. You can see nothing. Okay keys. You can see the keys what are available and you can see there is a key and get secret stuff. Cool. So you reach the flag. That means you are done with the challenge. Right. So this is just an example of try to simulate the scenarios but definitely you see these kind of vulnerable days in real world. Once you end up in a port try to look out for more services and try as much as you can and you will get most information out of it. Cool. So let's go back and see another scenario before we conclude the presentation and moving into the next sections. Let's look at one of the common oldest the vulnerabilities which we have seen exploiting the helm or tiller. So helm is like a package manager for Kubernetes. So how we deploy application install applications into suppose you are using devian based apitget install the package name. Similarly developers or operations teams use helm which is a package manager to deploy applications into Kubernetes. Right. So they will just do helm install and the nginx it will go and install the nginx within the cluster. Right. So by default if you if you are testing any cluster and which is using version two which is the older version of helm and it used to be by default most people using is helm two in a previous cases but they have reached three recently like someone's back. So by default helm version two has our back set up with the full cluster admin access. So if you end up seeing these kind of helm two configurations running within the cluster and there are no network security policies applied and it is running with the default setup basically using this misconfiguration as an attacker you can get entire cluster admin access and upon the entire cluster. Right. Let's go ahead and see how this scenario looks like. I'm going and running a simple pod and exacting into that. Right. Cool. Let's go ahead and run this command. It will go ahead and spin up another pod. It has code helm teller and give you a bash access. Right. Yep. So now we are inside a pod and we assume that this cluster has helm install and we wanted to see how we can exploit this kind of vulnerability in a real world environment. Right. So let's go back to our documentation. Right. So by default in Kubernetes there is a schema which they follow to access or basically it's like a DNS similarly how we use they use in the Kubernetes. So this is nothing but a service name. This is the name space and they use either service name and name space and cluster.local. So this is how the the structure which is followed. Right. So by default if you have installed the helm which is pillar is get deployed in version 2 which is nothing but a server side component of helm. Helm is the client side version where you do helm install or something. So the tiller is the server side component which has a full cluster admin access. Right. So now we are doing tiller deploy is the service name and it is by default applies in your cube system. So I'm trying to look for cube system and I'm checking telnet and see if this port is able to access it. So this is the default port which is in a by default configurations exposed to 0.0.0 which means if you are in any one of the port within the cluster you can able to access the tiller deployment service. Right. You can see it looks like you can able to access the tiller. So which means you can able to talk to the tiller deployment service because there were no NSPs and by default it is exposed to 0.0.0 which is within the cluster. So I have also installed a helm within this same container to talk to the helm. So in the helm version 2 you can specify the dash dash host flag which means you can specify which tiller service you want to talk to. Now we know the tiller service name as well as the DNS basically and we are specifying talk to this host and perform these operations. The command I am trying to run helm command and specifying the specific service and the port and I am just running helm version command. Right. Now you can see the client version which is helm is 2.16.7 and the server version which is tiller is 2.16.9 So I can able to talk to the helm service and I can also see what all charts deployed helm charts like you can see metadata dv is deployed as well. Right. So what we have done is as an attacker simply to showcase the complexity or what you can do as an attacker. Let's go ahead and run a simple command by default Kubernetes stores has a service account which is stored in where service secret secrets Kubernetes service account namespace because I am running in default namespace and this is the token service account which is mapped to the current pod. So what I am trying to do now here is we have created a helm chart which is on chat and this I have taken from the the bitnami repository research and I have given the reference here and go and check more research what it does is basically it's a simple helm chart what it is do is it has bunch of values which is nothing but namespace is default and the name of it is templates and cluster role so it has cluster role which is nothing but it gives all API groups in the Kubernetes for all the resources and all the verbs which means you can do get post delete list all of the resources for any of the resources in the Kubernetes cluster any of the kind of API groups you can perform right so what we are trying to do is templates cluster role binding so we are trying to bind this cluster role which is enter across the cluster to the service account which is taken if you've seen which is namespace is default because we are in the this pod is currently running in the default namespace and the service account is default because you see here it's the service account is default when you do slash run a secret service account so what we are trying to do is we are creating a new cluster role to perform anything on the all resources and binding the cluster role to the default service account so now as you can't get the secrets or any operations within the cluster using the default account once we deploy this Helm chart you can do anything as a cluster at me right let's go ahead and run this deploy this Helm chart so we are trying to do Helm and we are using the same tiller configuration and deploying the pod chart let's go ahead and run so it went ahead Helm has the capability to talk to the tiller and tiller has the full cluster admin access so you can go ahead and deploy this chart so now if you go ahead and run kubectl get secrets you can able to get the secrets of the entire communities here right so you can also get kubectl get pods you can able to get all the pods and in all namespaces as well right so basically your full cluster admin access because currently whatever the service account which is there in the region we have created a cluster role this cluster admin sorry this pod has basically full cluster admin access which can able to perform any kind of operations and all resources right so as an attacker some of the useful commands is like look for auth can I create pods you can check if the current service account which you own if you anytime get a service account access you can able to create pods or something you can also check delete pods you can able to delete pods or something you can also specify can I do in namespace of kubesystem so this is talk to the APA and give you the information so this this command is pretty useful if you got a service account and you wanted to see what kind of privilege it has you can make these calls right and also some tips I don't know we are able to conclude and let me whenever you make these commands you can also increase the verbosity you can increase the six or seven you can able to see each and every request which is making to the Kubernetes cluster and in a verbose way like if you increase this you can able to see much better even like headers and if you increase like right like you can even see like response body and everything and if you increase to nine and like little more depth it will keep going right so this is completely API driven so you can able to play around with the verbose and understand exactly each and every thing which is being done and also very useful command is explained if you want to understand port then it will give you more details about it and I want to learn more about port spec you can also do to cut a spec so this is very handy and while you are performing assessment you can quickly look at the documentation rather like explain cool so now you see due to the simple default misconfiguration of Helm tiller we can able to control the entire cluster this vulnerability I have seen in so many of the cluster in real time pen testing assessment due to the administrator doesn't configure properly and this is default security installation by Helm itself so the newer version of Helm v3 fix this vulnerability so it kind of fast cool so let me go with the last scenario I think then we kind of conclude and this you can try it's a dint docker in docker in exploitation this you commonly see in CICD systems in pipelines like in real world where they pass the socket you can go ahead and try and SSRF vulnerability especially this is very widely exploited in the cloud infrastructure but in coming to the Kubernetes context it is little much you can get more out of it because as I said within the cluster there might not be some services authenticated there might be services which are giving more health information of the nodes as well as the ports you can get more information out of it within the clusters compared to the cloud native cloud way of exploiting cool so let me just complete with the loss yeah this looks fun cool so this is the use case basically I'm trying to showcase if you end up in any of the pod and the developer or operations teams didn't set up any resource consumptions like you can only consume these many resources like so you can able to use utilities especially like I'm trying to showcase here stress and G so let me showcase simple example let me show in the terminal maybe exit cube cut I'll get parts now you see bunch of parts are running so I'm looking for specific part let's look for that cube cut top part so when top it's basically system like top command it shows how much CPU resources and memory resources it is using and if any of the deployment or manifest which doesn't have specifically resource limit set up and the cluster is deployed in like cloud infrastructure like which has auto scaling enabled if you really hit this limit it will go and scale up a new node or it deploys like some more nodes into the cluster right so that times you can exploit this in a attack as point of view I'm running with two kicks and other seconds and so now you can see if you run the same command from the specific resources of 3MB 1M 2 like increasing so much right I think it's just taking some time let's keep looking you can see it hits crazy amount of memory resource limits because there was no resource limit set for this deployment whatever the hunger check which is deployed basically consume more CPU resources and memory which basically cost the billing like cost effective for the company's perspective right billing and due to the reason the most of the modern infrastructure is set up in such a way which is auto scalable like scalable based on the certain parameters so you can able to basically DDoS or gain more resource consumption for the companies or organizations right so this is definitely one thing before moving in let me just conclude with the hacker container preview as I said hacker container let me just showcase maybe here hacker container so a hacker container is a similar another utility I created as part of the assessment so you can read more about it here the blog post so this has a bunch of utilities which you use day to day like while performing assessments in the Kubernetes cluster so I have used it extensively while performing assessment and it is very useful utility let's go ahead and run looks like it's already there let's exact into the pod hacker container so we are inside the pod like you see a bunch of other stuff like default tools so some of the things like it has is am I contained so which is again utility written by Jessica Frazil she has done quite a lot of work for the containers and security community communities and the container ecosystem so what it does is it will try to identify what container runtime you are using and do you have shared name spaces from the host system like PID user name space or is there an app armor or second profile applied to the container to restrict certain capabilities and is that what all capabilities allowed currently system calls allowed and it kind of container introspection tool not only I think specific to the container communities generally it's a pretty useful tool to try out right so you can also do the same thing with proc self C group maybe yeah cool so you can see this is basically trying to figure out QPORs and you can also do CAPSH print so it will look for the what all capabilities are there I think maybe I try to not document it here so I'm going to try to see what can I kind of create out of it and how I use in day to day while exploiting and some use cases I would say is very useful is running benchmarks if you are inside clusters and you want to just get a glance of view what kind of vulnerabilities possibly there within the Kubernetes clusters both in terms of container you can just go ahead and run this command let's go ahead and run yep so what it does is I have created something called daemon set which means each node how many of our nodes you have in the cluster will get one port running in all the nodes that is called daemon set in Kubernetes let's go ahead and get the node so you can see two nodes are available now you should be able to see two parts running in the Docker bench because there are two nodes so let's go ahead and exec into one of the the pod and see what we can do with the Docker bench security right so Docker bench security I'm exec into the pod now you can see Docker bench security now you can go ahead and run Docker bench security dot this is also part of a hacker container can go ahead and run this so do the way I have set it up the sorry the scenario manifest is if you can see here I have created the the Docker bench security in such a way it access the host BID IPC network as well as a bunch of volumes and it also had extra capability called audit control so that you can access and talk to your underlying Docker container runtime and you can perform benchmark analysis on possible security issues right so this is a pretty simple utility just go ahead and run this once you are inside cluster then you will get and use the whatever the found vulnerabilities are the checks as attack paths or you can use those while you are doing assessments or post-exploitation right once you are inside a node maybe you can use some kind of vulnerability which is detected here to do some kind of post-exploitation right so you can see it does bunch of checks like host level configuration checks and specific to like the Docker daemon or like directories and daemon configuration the files which is specific to the container runtime here in this case Docker and also container images whether container running with a root privilege or it has an extra capability set or it doesn't have any second per apartment profiles these kind of things right similarly you can go ahead and run the cube bench as well so you can go ahead and run this cube bench like this let me go ahead and run here only I think back so you can see cube bench I have run you can give it parts the cube bench is running here can get the logs iphone f and the bench so it will show you can see there are 12 checks passed 9 checks failed and 2 checks running so you can see what all the vulnerabilities possibly there in this Kubernetes cluster and what would be the remediations so you can use these vulnerabilities maybe to do further exploitation and understand the tools are very useful just to get a glance of it and there are bunch of utilities out there as well like cube hunter and cubes I call it and the the different goal and tool written by different researchers and I'll try to document and put it in the same scenarios hopefully you can go back and try out yourself right cool so there are bunch of other scenarios and you should be able to get and follow using the the playground of the Kotakoda and if you have set it up in your own environment like in our case so you can go ahead and tear down the cluster it will try to delay possibly resources which have created as part of the Kubernetes deployment but I highly recommend you delay the cluster to just clean up of the entire environment and I have also showcased and run using a tool called Chekhov to see what all possible vulnerabilities using CACD as part of pipeline so it found bunch of vulnerabilities in the deployments and other things which have used as part of Kubernetes code as well right cool so that's it pretty much about it and I would highly recommend having your feedback and understand what exactly you want to look forward and I'm really looking forward to building more and more scenarios and also adding the defense scenario I'm really building like most of the scenarios and making them in such a way that how you can practice for the CNC of certified Kubernetes security so you can use this playground to practice and understand how most of the Kubernetes vulnerabilities you can access and learn more about it so feel free to get involved and you can share your feedback and share with your colleagues, teammates or your friends I would really appreciate if you can share the feedback and I would love to improve and work on it and if you see any documentation or anything needs to be improved please let me know by GitHub issues and I'm trying to be pretty active but in the past month I was a little busy so I kind of work on this working on defense scenarios as well hopefully we will practice this attacks and also more attack scenarios as well and with that I think I'm pretty much done with the session thank you so much once again for having me and really looking forward awesome thank you so much it was an amazing presentation I learned so much and as a matter of fact in this court there are a lot of people actually talking about your presentation and once again thank you for supporting the communities thank you for supporting us we got to go in a little break but we invite everybody that will have any further questions and so on to join the discord channel which is in the description of the