 So now this is an interesting timing for a talk because I want to introduce an old friend of ours who is this is the first time we're going to have a talk but then immediately you're going to have to hop over to the next room for a workshop so this is like the prelude to your workshop isn't it without much ado JBL everyone hi hi hi hi i'm gonna try really hard to stay within range of the mic and let's see how i can do it then i'm i'm not really good at staying in one place i kind of like to walk around uh if you've seen me at many def cons in the past i like to dance so anyway uh with that said i'm gonna talk to you about attacking and defending kubernetes um i it's topic it's become incredibly important to me and dear to me and interesting to me so i have a bio slide but we all know bio slides are just big c's of text so i have a graphical bio slide which basically says this is stuff i've done or stuff i'm interested in um the uh i really i really enjoy this is stuff kind of going back way back to the back in the day writing writing its security focus and doing c is work and giving some early early def con talks and nowadays teaching a teaching a class at black hat and uh and enjoying a heck of a lot of working guardians as a cto um who likes to still be a very technical cto so anyway so let's uh let's get on with what you came for so what are we going to see today we're going to talk about uh we're going to talk about kubernetes i'm going to do i'm going to do a bunch of demos um we'll do a little bit of choose your own adventure if all my uh if the tech with the tech works the choose your own adventure part is entirely new so uh the nice thing is cutting edge fresh the downside is uh so hopefully this is uh hopefully this goes really well so i'm going to talk about what kubernetes is i'm going to talk about how you um what you attack and how you attack it and then a little bit about how you defend we're going to do some demos um let's just start so the first thing i do anytime i talk about kubernetes is talk about what the heck this thing is for and um because and i want to make sure i'm coming through on the mic might come in through on the mic and like the back of the room perfect okay so what the heck this thing is for and um and you know people are like i i know that this is hot i know that i should use it that's kind of the same way we all felt about docker so you know why why do we use this what's made it so big and the basically the the kubernetes came out of in a lot of ways it came out of what google is doing with the borg um right that's it's an open source project that initially got released by google and now has many many many contributors um but kubernetes basically came out of this need we know that google we know that google had just a has a ton of computers and a ton of workloads to be clear you know it's not exactly the most transparent company all the time but they've they have some public statements they've made one is back in 2014 um they were launching two billion containers per week if that were evenly distributed it'd be 3300 containers per second um what's a container it's like a workload it's a program that's going to run for a little while and do something we don't know how long that while is but if you're launching 3300 a second you can't have a human being decide which virtual machine or which host you know each program each workload is going to go on so um you end up needing a tool for that now what else um well they did this they were they were launching all of those workloads onto 2.5 million servers at least that was their number back in 2016 and in 2016 hard drive failure rates were just right below two percent so if every server had just one drive and that's a big you know hand waving thing but every if every server had one drive that meant they had 133 drive failures a day or one every nine minutes so now you've got now you've got a resiliency problem right like you wanted to guarantee you know you wanted to guarantee so many nines of up time and yet your physical machines are going to you know entire physical machines are going to die on some regular basis like say read notwithstanding one every nine minutes so what features would you want if you wanted to be able to stage a ton of workloads over a ton of servers figure out how to use the servers are the greatest as close to their capacity as you could so you weren't buying lots of extra machines and you wanted to make sure that as programs crashed and as and as hard drives failed and so on that you still had availability so you end up with what kubernetes is which is basically a software defined data center and we do that by container orchestration so what do you get so basically this is your feature set this is what you end up saying you end up saying well sometimes I have a lot of people checking their email on gmail or or searching right now for the thing that just happened in the news and sometimes I don't have as many so I'd like a whole lot of scaling I'd like that horizontal scaling I'd like to be able to say uh I have four copies of that workload no weight I have 400 no weight I have 50 and so on and you get automatic bin packing which is that like which workload should go on which machine so we can keep all the machines as full as we can and you get self-healing like that hard drive crash thing and you get a whole bunch of other cool features you get like automated rollouts and rollbacks where you can basically deploy multiple versions at once and see if one of them gets see if one of them ends up producing more sales or more of whatever it is you're trying to produce or see if one's buggy and then be able to basically route all the traffic to the one that isn't and so on you get you get the other kinds of things you would need if you're going to do microservices so you get service discovery and you get load balancing and then you get orchestration of your storage and you get secrets and configuration management so that's kind of that's what the purpose was I'm going to do a little bit more theory so just kind of like what's a pod what's a container so right we when we're talking about Kubernetes we're talking about containers but we say pods a lot and basically pods are the smallest unit of computing Kubernetes so all the containers in a pod are sharing an IP address the only reason you put multiple containers in a pod is if you want them to absolutely positively land on the same host like they have need for very very quick network access or what have you maybe they need the same local storage an odd thing about pods is all the containers in a pod share an IP address okay so then you've got Kubernetes nodes simply put a node is any system you're going to run Kubernetes on a node is any of the any of the systems in your cluster whether those are virtual machines or physical hardware the nodes run a well first and foremost the nodes to be a node you've got a you have to have a container run time so you have to have something like Docker run see or what have you something it's going to basically take instructions about take instructions from Kubernetes about creating containers and create them and maybe destroy them and so on you also need a kubelet and a kubelet Kubernetes kubelet a kubelet is just this thing that runs in every Kubernetes node and it ties the it ties the container run time or the node into the rest of Kubernetes it talks to the API servers and so on then you've got a kube proxy and that thing manipulates IP tables rules to create cool networking it also serves as an actual proxy so yeah we'll get into that a little bit more and then services services are big because in Kubernetes and services give me a great give me a great occasion to to kind of describe how you talk about the names of things in Kubernetes when we're doing the attack demos we're going to be talking about this pod and this pod and this pod but for the most part in Kubernetes you don't ever think about which pod names you want right it's the whole idea is that the individual pods right the containers they're very very much supposed to be cattle they're not supposed to have names they're supposed to be interchangeable i'm supposed to be able to trash them start new ones up i have to i'm going for i'm going for that so what does that mean i mean if i want to talk to a pod what i do you don't i mean you might but rarely mostly you talk to a service and a service is mapped to multiple pods think of it as an internal load balancer now now if you've got like why am i why do i say that would get me into something cool that gets me into labels and basically you have a you have a load balancer you can't say this load balancer should be pointed at these five pod names right first you're going to have a whole lot more they're going to get added second those pods are going to die at any moment third you're trying to go really really fast remember you know not 3,300 containers per second being launched right so what do you do you end up basically labeling everything so you label all the pods you give them labels and those labels are basically sometimes they're by convention sometimes they're whatever you chose but you label the pods and then you create a service and you say everything that you know this service is going to be a name an ip address a port and everything that goes to that name and port or ip address and port is going to be sent to one of the pods that have the set of labels the service specifies it does kind of open an interesting question like if you are allowed to stage pods onto a cluster could you start intercepting other people's traffic by just giving yourself the same labels that their services are looking to land track to land things on okay interesting last thing we talk about last big big you know kind of simple glossary concept we need is namespaces the namespaces are just a logical grouping for things so we'll see those they're often used they're often used to separate projects or to separate teams to separate an individual you know maybe to separate well anything that you anything you choose so there's some way of chopping things up and it's it's important to say a kubernetes namespace is not the same as a docker or linux namespace so much so that when i teach classes on this i one of the other folks in guardians jared fredis asked if we could please call kubernetes namespaces space names since docker and linux you know already have namespaces so as you know don't overload don't overload the term so here's a quick glossary slide i'm gonna say a tiny bit more on theory and then we'll have some fun so so kubernetes is basically works in this declarative model you don't generally say you're gonna see me generally like hey do this thing right now but you don't generally tell kubernetes to do a thing right now you don't say hey take this you know take this container image start a container on that note over there with it right you tend to say you know it's it's very it's very delegating and hands off right so the idea is you say kubernetes i uh i have this program i've created an image maybe i've placed that image on docker hub maybe put it on some other container registry and i'd like you to now like download that in like have some nodes download that image i'd like at least say five to be running it like if you need more scale up but you know i want five no less than three like let's you know just do me a favor and get that running and kubernetes says sure and it's got a bunch of like control loops that basically first make that the state of the world so you describe the state of the world you want it it basically spends its whole life trying to keep the state of the world where you want it where you've said it has to be and then it basically loops and it says ah a machine fell down i guess i'll have to reassign those workloads or ah looks like something's not responding i'll have to you know i'll have maybe i'll kill that off and i'll start something else so you just guard the state of the world um i think this is basically my last theory and and i hope i haven't given you too much um so the kind of target components there are a bunch of components you'll target the big ones that you hear about are especially on the first two slides here so you'll target the the api server the thing that everybody's basically doing all these all of these uh htps and god forbid htp requests against and asking to do things um and um the cubelet which we said bridged the bridged each node into the system the container runtime you can often think of docker and i'll often say docker but there are many there are multiple container runtimes um and that container runtimes the thing that pulls the container images from the registry you know like docker hub or would have you and instructs the kernel to start up a container you have you have an xcd server that maintains all the state and if you ever get access to that thing uh yeah there's not really a lot of chopped up authorization uh so if you get access to it you kind of own all the you own all of the persistent um idea of what the system is um and so that's very good for you uh a kubernetes dashboard um that was the thing that got uh as i recall that was the thing that got teslan trouble where their dashboard was available to the internet and uh uh i think was allowing anonymous access to do anything um but that's a web interface as i um i think it's being deprecated there's metrics components and different kinds of cluts cluster services it'll give you good information those those are your usual targets um just to throw in two more two more components that people that it's really helpful just you know if you're operating kubernetes defending it or attacking it or useful to know one is the controller manager and the controller manager it makes it sound like it's like the controller manager would have lots of separate controller programs and it kind of does but they're all just compiled into one big you know they're just compiled into one big program so the controller manager has all of the controllers the controllers are the things that basically do those loops and say is everything the way it was supposed to be is everything the way it was supposed to be ah some things different i'm supposed to have five copies of this i've got four i'll go talk to the next one the scheduler and say excuse me scheduler can you find somewhere to can you find somewhere to put this um can you find a node we can put things on we tend to target these components only through the api server so attacks on kubernetes um this is where we start to finally get to attack and defense attacks on kubernetes they start from they start from the perspective of compromise pod so you can attack you can attack a kubernetes cluster from across the network and it's even less effective than attacking most companies you know across the internet has been right lately it's it's we've gotten pretty good at parameters kubernetes is pretty darn good at parameters because for the most part it doesn't allow any there's nothing there's no traffic being allowed in unless it's had and unless it's had a um some kind of ingress defined for it so um so for the most part you attack a kubernetes cluster from the perspective that you are in a workload that you're in a container and that container is compromised or otherwise under your control so what are the you know what are the what are the what are the ways the attacker might end up with that kind of control one is um yeah as you'll see in some in our demos this is the this is a vector i like to i like to show it's the uh the attacker found a vulnerable thing that you were hosting kubernetes got remote code execution and now they're in a container and from that container they may have the ability to do much more far-ranging things because we're in an orchestration system a very automatic one they may be able to do far more wide-ranging things than if they just busted that you know that if they just busted a a you know some web server some or something that was in a uh that was in a cloud environment or in a data or in a or in an on-prem data center so um how else like the attacker got in the attacker might have just fished somebody um uh brad geeseman who i'll uh i'll whose uh whose name i probably just butchered and whose twitter handle i'll have at the end of these slides um had a has a great slide in one of his presentations um about how coop control apply and that a url is the new curl url into shell right if you uh uh one of the great ways to fish somebody would be if you put up a how-to article on stack overflow or would have you and the you know part of that article was oh just apply my you know apply my yaml from this url um you can get anyone to apply the yaml from the url then you've got places for your crypto mining to go um so um and our attacker could just be some internal threat actor who's looking to escalate their privileges i often you'll you'll often hear me use the word user and attacker uh interchangeably uh on the defense side that's not really because i think all my users are hostile i just don't know which ones of them might be i don't know who's being coerced i don't know who's you know getting curious i don't know who's actually got a axe to grind and i think we all know these privileges a good idea at this point so um okay so so if an attacker is going to be in a pod what are they going to do with that i'm going to give you a few i'm going to give you a few things there's even more i'm sure so this is kind of the general way so they could use just the network access that they get provided by the pod to access other services and i'll say very very very often especially in the microservice model a kubernetes cluster has a bunch of things that are all supposed to talk to each other and it's actually it can be a little difficult or at least take some work to sit down and say okay what should and what shouldn't so once you're in the cluster you have access to a whole bunch of things to a whole bunch of things many of which might not be intended to be to be uh communicated with by anyone on the internet or anyone in the anyone the that the service that you've pond is trying to uh is trying to uh um serve so what else the attacker could attack other containers that are in their same pod um multiple containers in a pod they share well i told you they share they share an ip address they share a local host so it's a cool little linux namespace thing which we won't get into unless somebody asks me later on um what else so the attacker could make requests to the api server or to a kubelet and ask those to maybe run commands um in another pod maybe even interactively um to start a new pod that has privilege and that privilege could be all kinds it could be a privileged pod which means it'll start on the host and have access to system devices and be able to my other things maybe mount the hard drive of the node but um but there's uh but there's just there's all kinds of privilege you can get out of out of starting new pods and i'll show you that they might gather secrets that kubernetes provides to pods because kubernetes has a whole thing where it's um it it needs it needs to provide it needs to provide configurations to pods because you're really supposed to make those pods come up with a static image so the idea is the pod starts with a static totally replaceable image and pulls whatever information it needs um from out from elsewhere um you might connect to the kubernetes dashboard perform actions maybe it wasn't accessible from outside the cluster god god hope or gosh hope what else uh the attacker might find a way to interact with the etcd server and change the cluster state a nice way if there's if there were network access if there are if there are network access control rules their network policies in place that are getting in your way if you can talk to the thing that stores them uh yeah you could you could potentially change them um and the other one the one that's really fun to me for anything that's either in you know it's either in a cloud provider or even in an on-prem cloud um is that the attacker might interact with the cloud service provider and uh because many workloads will one way or another and i'll show this to you too will have privilege on the on the uh on the cloud provider so i'm going to show you a bunch of demos um and these demos are real they're they're often things that they're often things that we do in classes i teach they're also i'm going to tell you later on we have an open source project and it's a intentionally vulnerable kubernetes cluster call it like damn vulnerable kubernetes cluster this one's called bust a kube um uh we've got an easter egg bust a move video that you can find if you go to the bust a kube.com website um but uh uh so some of the things i'm showing you are basically available as scenarios that you can play in this open source free uh damn vulnerable kubernetes cluster so i've done a thing where i've made my videos part of the slide and so now i'm going to see if that actually works please work and if it doesn't we'll have to use youtube one way or another or something else okay color me very embarrassed let's just see if this one's the one you should always test your demos um even when you record them so uh so i tell you what we're gonna we're gonna find another way to do it um and i am going to let's see i'm gonna unplug from the projector for a second because that means i'm going to go and look at my own file system and i'd like to not share it and here we go just embed the videos in the slides they said it'll be fun they said anybody old enough to have played uh to have played warcraft and you click on the guy and he's like what and you click on him again and he says something else and he says you know join the army he said see the world he said okay so here's our here's our demo and hopefully entering oh well that's not a demo is that a demo that's not a demo either anybody's seen my screen there we go let's just hit this on mirror and call it a day okay cool so so that's not the right don't mind me i'm a i'm a i'm clearly a noob uh i may need alcohol um let's so this is this is one of the this is one of the bust a cube uh this is one of the bust a cube scenarios so you go up and you see a wordpress site and uh the wordpress site i will tell you is hosted in is hosted in kubernetes um and uh you fire up maybe derbuster to go looking for hidden content and um and derbuster takes a little bit of time but we can run it and let's go a little faster and what we find is backdoor dot php and so when we serve to backdoor dot php this is what happens i get this thing where basically this wordpress instance like so many other wordpress instances has been pre compromised for your convenience um so i type id and it says your dub dub dub data and uh hopefully i shut down derbuster because it always choose a ton of resources something we always say in the class they're like my laptop's running slow uh have you turned off derbuster no it's 500 requests a second well that'll that'll do something so i'm going to stand up a quick little uh a quick a quick little web server in cali um because i found a uh i found that reverse shell so i'm going to stand up a little web server and my purpose the web server is to be able to you know hand a um just be able to hand a a interpreter um or a uh you know or if i wanted a you know whatever some other kind of reverse shell but i'm going to go for a interpreter and so i just put in here like hey could you please pull this interpreter binary i've created from my web server and um while you're at it make it executable and while you're at it run it and i like to fast forward my videos because who doesn't and uh go look at the you go look at the web server and mrs ben has been pulled the interpreter binary but also a copy of coop control the uh the command line utility that you uh that you use to to do damage i mean to uh administer or use a kubernetes cluster so i'm going to pull coop control and the other thing i'm going to do is just take and make sure that that it's landed here so here's mrs ben my interpreter binary here's coop control cool and now if if recorded j will type a little faster um will run the interpreter binary and i'm going to go over to my metasploit console and actually be like oh wait i have an incoming interpreter that needs to connect to the console i should probably start the console um this ordering is just great huh but that's often how you work so i'll interact with that session and i'm now in a pod i know it doesn't look like it it looks like an ordinary metasploit you know pwned metasploit instance so but i see my copy of coop control i'm happy and when i run mount i get a tremendous number of file systems so i'm not looking at one hard disk or what have you um and a bunch of them put there by docker um and by kubernetes and so the one that's really interesting to me is this one that says run secrets kubernetes service count i o service account as soon as i see that directory mounted i know that i'm going to have some level of access to the api server and i'm happy so i'm going to go and first i'm just going to go and collect my flag i've built a built this scenario as i capture the flag so let's look at our little flag and it says hey you sound the server's back door you might notice this isn't your usual Linux system you're in a cluster my friend it's time to bust a cube this container doesn't have any more flags can you find your way to another pod hint hint so um cool thing in kubernetes is like i said one of things you can do is see if you can move to another workload so you're in a container can you move to another one so i'm going to see what my service count can do i'm going to go and look in that directory and that run secrets kubernetes i o service count directory if if recorded j will type just a little bit faster and i look at that namespace and it says you're in the marketing namespace so that directory has a token it has the certificate authorities public cert and it has a name it has a description of what namespace you're in and that namespace like i said is kind of a wave organizing so we're in the marketing namespace we'll come back to that later and then i'm also going to go and look at the environment variables that get pushed into every pod and my environment variables will tell me where to find the kubernetes api server so there's this assumption in kubernetes that every workload will need to ask kubernetes to do things i i don't know if that's really uh the greatest assumption but i think let's just call it not the strongest of defaults so i've got the api servers ip address and port and i'll clear my screen and i'm going to set up a quick little kube control um where i'm saying alias kube alias kube controlled the path to kube control the token pasted in from this long path to the token the certificate authority path um the namespace and the server that i just got and so every time i get type kube control all that stuff will be passed in so let's just kind of skip and let's see what did i do so i said let's do a quick get pods and get pods is my my is honestly my go to just figure out if if my setup is working and so it says oh there's two pods you're the word press one and i ran host name and it says that's the one you're in and then the word press my sequel one and that's another one so let me see if i can move over to it because that was what the hint said so i'll do kube control exec it looks a whole lot like docker exec it huh and what program i'd like to add to that container i'm going to add a shell i get this kind of error but i'm in the i'm in the container i've just moved over and i've probably moved over to a different host like i'm probably somewhere else in the data center now um so there's another flag and we'll go get our flag and i can't clear my screen because my terminal is not set so i set it to vt 100 because that was one of my first back in the bbs days because i'm exceedingly old um and it says hey you found your way to another container actually you're on a whole new virtual machine um we're moving from one container to another in a kubernetes system often means switching nodes your end goal is to get to developers non-containerized host so it's to get not just not just to own the cluster or potentially but to but to but to get to a machine that's not in the cluster so to get there you're gonna want to see if you can hop tenants tenants are usually separated by namespaces so let's see what we can do i know i'm fast-forwarding through my stuff but so i'm in i'm in the other i mean that my sequel want to try to get pods and the problem i'm i'm running into is like hey you're not allowed to get pods so the cool thing is i'm in a different service account i was in the marketing wordpress service account it's allowed to get pods the marketing my sequel one can't um so over time i might want to like go around and pick up all these service counts and like hold on to them remember it was just tokens so if i can store all the tokens i'll have like i can end up building privilege wider than just what one thing had so i did get nodes and i found out the ip address for node um here's a name but this node has an ip address um this is uh this one is um this one's running in the cloud provider and there's a cool thing so you can talk to the kubelet and this is thank gosh finally not a default but some clusters will have it you can talk to the kubelet um on it's got a couple ports 10 250 i'm going to go and say can you give me a list of running pods and it gives me json because everything in kubernetes is json yaml so i'll take that and i'll run that again and i apologize for this big block here so i'll run that again and i'll add some jq json query really good tool to learn and i'm going to say give me all the pods kind of better format it give me their name and their namespace and what i'm looking for is to see if there's a pod that's in um that's in another namespace besides the marketing one and a quickly fine one there's a dev pod and that dev pod so kubernetes loves to tell you about anything you want and that's really really nice as an attacker especially if you can script it um so i've got dev sync i've got dev web um these are two containers that are running in the same pod i've got the image they came from so they're up on docker hub you could pull them yourself if you wanted um and as an attacker that's often what you're doing is basically saying well what workloads are running it can go look in container images and see if there are any secrets or anything that would be really helpful to me it's very very nice of the defenders to hand you descriptions of their workloads so um i'm going to look at that i'm going to run another command and this is i did running pods i'm going to i'm going to tell the kubelet hey would you mind just executing a command for me so i'm going to pass it run and then the namespace the pod and the container name and i'll do command equals id and i get an error code that says basically no either there's no shell in there or um or i'm not allowed i will tell you that there's no shell in there um so but remember there were two there were two containers in that pod so i saw dev sync and dev web and that's a really common a really common thing you see outside of containerized environments like you have a web server and you have something that's and you have something that's there to pull it content in from wherever that content's being created or stored so uh i was hoping i could get to dev sync because it'll have that it'll have the credentials for pulling content it'll have an ssh key um i couldn't so let me try dev web so dev web i can run commands in i ran id says uh yeah that thing's even running is root very nice and so let's try a different command let's look at the file system oh look another flag i like flags i love doing ctfs and i love doing ctfs actually like first doing them as an attacker and then going and basically doing them as a defender it's saying how many how much of the attack can i break with proactive stuff with stuff that you could have thought of before you knew that you had the vulnerability so this thing says hey you're not in the marketing department anymore you're in the development department's name space so can you go take over the development department's non-containerized machines and they're big fans of the matrix so it's named the mainframe named for the machine's mainframe that may or may not have been in the second and third movie depending on your religious beliefs there there are many people i know who acknowledge the truth as they believe it that there were no second and third matrix movies only quests for more money so here we go so we've got um to do a get pod we're trying to do a get pods so i am i'm in one pod asking the cooblet um on another on another host if it mines running commands in another pod and i'm asking that pod to run coop control commands i find out okay dev app isn't allowed to list pods but it's kind of cool i now know that i've got a third service account i've got i've got the dev dev app one so i'll try get secrets let's see what secrets are available oh nice there's a machine there's a mainframe login and an ssh key well uh the two pods in there uh the two containers in there are gonna likely have access to the same secret um they've been placed in the same place they don't necessarily need it in dev web but what the hell um let's go and ask for a copy that secret and i promised the uh the next demo is hackers themed and choose your own adventure so we're gonna have some fun so here's our ssh key and you're like that doesn't look like an ssh key and that's because it's a secret it's it's opaigly encoded it's base 64 encoded so we can decode it and boom there's our ssh private key and we can do the same thing with the other secret here so we can find out basically what is it used for so i'm just going to fast forward a little bit and tell you that it's the that it's a user um it's a user and a host in aws um but not part of the cluster so now we could ssh into that and we're in the developer's mainframe and it says hey you made it to the machine mainframe can you for extra credit can you get credentials to take over the whole cloud environment and and we can do that but i'm going to set the rest of that aside because we do that in another demo so um so let's see um let me go back to the slides for a second and here was our and i'm sure that i'll be because the vagaries of having the videos you know in a separate program i'm sure i'm going to be doing a little bit of going back and forth and having the whole screen do this so here's our attack path we got a interpreter in the wordpress container um we moved into mysql container with the kubectl exec we used the mysql containers totally unfettered network access inside the cluster to reach a kubelet that we wouldn't have been able to reach from outside the cluster on the master node and that kubelet let us basically take a look at a take a look at take a look at run commands in another container in another namespace and from that other container in that other namespace we found we figured out there was a there were secrets there was an ssh key so we got that ssh key we authenticated to the high value developer machine and later on we ended up getting to take control of the aws account and we're going to skip that right now because we're going to do that in a much more fun way so we're going to play hackers choose your own adventure and i'm going to drink some water for a second because whoa so i like the movie hackers it's uh it's a fun movie um it is an oldie book goodie it is thoroughly realistic we all know that whenever we're hacking we we see like strange translucent 3d models and and uh you know of buildings and we fly through a city and and it's really cool and all of that was on was clearly done with the technology that had apple twos and floppy disks um so i mean i remember 3d rendering back then i wasn't doing it i was watching people do it and uh yeah no not on that so here we go so we're going to start here and uh and again this is where i have embedded video um so we will unless i click on that and it works let's see oh hey look at that now can we full screen that is there a way no so we're going back to oldie book goodie and here we are so first this is going to look a little familiar so we've got a uh we've got what's that oh uh you're on the slide and i'm on the and i'm on the video okay boom and switch come on i'm going to windows computer soon um and that's a lot for me to say trust me so anyway here we go so we've got the uh we've got the gibson control room uh we have uh pen from pennon tower um who's apparently a techno weenie according to uh ben from short circuit who uh somehow uh was playing in indian and talking about mr johnny five um but mr johnny five here was the plague and um and we have this nice little form that's called the gibson control room and the gibson control room clearly lets you like leave a message so it's based on a it's based on a actual uh on the kubernetes documentation vulnerable app they didn't mean for it to be vulnerable um but it's only a little vulnerable so i've added the vulnerability to that so the um so what we've got is i look at the source on that page and i find that the form um is basically being provided by some java script so if i go look at that java script um what happens when you submit the form is you're you're referencing guestbook.php you're sending to guestbook php you're saying hey command set key messages value this okay really you're gonna let me just just you're gonna let me decide what uh what key i'm gonna set you're not just gonna have me set whatever key you know was meant to be set by this form so that was their bug um and what i've added to it is i have something that actually runs code out of a different redis key in this you know back in redis database so we go and take the gibson control room and if we were to type a message you know it shows up this is the intended this is the intended purpose but instead we're going to paste that same thing that happens when you submit the form except instead of the message key we're going to try a different key or we'll try a let's see how it works we can hit foo um and we'll put in just some key some random key oh cool okay so java script is probably updating that key let's try going to status php so i'm asking you to imagine that we did a derbuster or looked at a robots dot text we found a status php i'm not doing it because well i'd like to spend our time on kubernetes so i go to the status php page it says hey no command to run i'm going to set the redis key the redis command key to this so that you can reload the page and and that will get run so if i reload the page i get the output of you know i get the output of go curling that web server it says all all services are operational it here at uh what was it ellison mineral uh or ellington mineral ellison i think um so well as long as there's a command key let's go and set it and i've set my command key now to environments to run environments and like we did before to find a kubernetes cluster to find the the kubernetes api server and here we go so we know that once we're in a workload um if we reach out to 109601 we'll get to the kubernetes api server so i'm gonna go a little further kind of similar last time i'm going to curl in i'm going to tell this thing excuse me would you just pull in a interpreter binary i created and connect back to me um so i have it pull it in i set that key and now when i run status the page hangs i love when pages hang when i've just given them binaries because it generally means the binary is running successfully so if i go over to my terminal where i've got a menisploit handler waiting this time i find there's a session waiting for me and the first thing i'm going to do before i start interacting with a shell there is just i'm going to upload um i'm going to upload a copy of kubectl which i've stored on this cali system in this directory so i'll put kubectl and i'm going to upload some yaml which we'll get to later so i've uploaded some yaml which we may or may not get to later depending on your choices and i've asked for a shell hit pwd looks like my video is cut off by one character hit id it says your dub dub dub data user um i try sshing and or i well i don't know what i tried because it's because that was the end of that part and now we're going to do this part so that we're a little bit gentler i'm going to ask you so do you want to try the yaml files that i put up were were ones that would mount a volume on the we try to mount a volume on the host like they try to mount the the host some directory the old you know docker you know start a container uh mounting the host so do you want us to start the host mounting pod um and see if we can take over see if we can take over a node or do you want us to exact into another pod i heard a host i heard the host mount first so we'll try that so let's see boom and boom here we are on that front end and we'll set kubectl executable so we can run it we'll go and set up our kubectl um we'll try to run kubectl apply realize wait we've got to set all those parameters we set before sometimes you get lucky um and so i'm going to take that same directory that same run secrets kubernetes io service account that's being mounted into the pod i'll set a variable called dir to that export dir that just to make this a little bit easier to type a little more manageable so if i look at the dir got the ca cert namespace token just like before let's go and run alias kubectl is going to mean that same way we did before so servers out on 10 9601 that was what we found from that environment variable and now let's look at our alias that's that token being pasted in so kubectl get pods cool we've got a front ends we've got front end pods we've got a redis master we've got uh we've got an apache status that thing we saw before we're in the front end um so let's try launching a pod kubectl attack a plot so kubectl apply i like it as kubectl attack attack pod uh and it says sorry your service count front end isn't allowed to create resources wah wah so uh and this is where i get to my slide so we'll do that real quick i was considering for this one oh damn it let's just keep the screen mirrored so i was considering for this one doing a uh spandex quote from the movie and uh i decided that wasn't a that wasn't a sentiment i wanted to voice um so um i see that i am uh i see that i'm going to make sure that i'm moving on time so at this point we'll go back to our original we'll go back to the other thing we could do so our two options where do we want to launch a pod or exact in the back end so let's exact in the back end and or let's try exacting into another pod so we'll go like we did the other like we did in our other demo we'll try exacting into a pod keep wishing we had a better terminal kubectl get pods i see that there's a redis there's a redis that redis setup remember i was pushing redis keys um i might try to to investigate the redis and see what else now that i've got direct i've got more direct access to it but uh because redis is very often well almost always unauthenticated but uh let's go into the redis pod so i'm rooting there i'm redis master i'll grab a copy the interpreter binary i've been using just because that's a very convenient way of putting files into something and i'll background that and i'm gonna i'm gonna switch over now that i've got the i've got the redis master um that pod running my interpreter so i'm gonna switch over and catch that session and so in that session i'll go and upload the same kubectl that i did before i'll also upload that attack pod that i just used before and i'll also attacks add something called attack demon set just in case we need it so now we get another choice so you're in the back end do you want to try host mounting pods or do you want to check out the cloud apis cloud apis cool okay so here we are and i'm going to show you this in two ways and can i get a rough time check how much time we let me be late oh less than five okay so this is where i normally kick on the speed control so okay so we're in a pod we're in that redis master pod and we say hey i'd like to talk to the metadata api so uh every cloud provider gives you 169254 169254 that part's common what the calls are that you make against it they document wonderfully uh and you read the documentation so uh this is on this is on gcp um so i'm going to go and say uh can you tell me what the project id is i need that project id to do api calls but then the more interesting one is usually where you say um would you mind in addition to the project id which i'm going to set into a variable would you mind telling me what service accounts are assigned to the to the machine i'm on so the machine this pod's running on and there are two but default is mapped to that other one so i'm just going to use default because it's easier typing and i'm going to go and say okay for that service account would you do me a favor and just give me a uh bearer token that i can use to act as that service account and it says sure here's your token it expires in like a half an hour or more an hour but i've got a token so i'll set a variable to that and now what i'm going to do is say uh let's go and i've set that token to that and i need to refresh that token so most of my commands will pull that token over and over every command so i'm going to use that token type out authorization bearer token where the token is provided say i'll take json and now i'll say hey for this project i'd like a list of buckets and it says oh there's only one okay well what's in that bucket and so i basically say give me a list of objects and the way i do that is it's like slash b for a list of buckets then it's the slash b that bucket slash o for a list of objects and then for those objects i can run through them now on this one um i don't find anything that's all that useful i don't find i find i don't find uh what i find in some cloud environments i'm going to show you in this in another demo somehow um and um but i do find a kubian v so i can pull a in a metadata instance um for this cluster that was set by the installer and when i do that what i get out of it is among other things a couple service count tokens for the cluster and that ends up leading um which i won't now show you but that that ends up leading to win and so just for the end of our uh for the end of that uh that then leads to our crash and burn so uh if if ming will let me have one more minute i will show you a cooler version of that we have a tool that we're that we keep soft releasing but never kind of like putting up on twitter uh but so it's called peratties and peratties will if i uh if i actually just i'm just nervous so it's in here but somehow i'm uh let's do this so peratties is basically a remote access trojan um for for kubernetes you run this and it does all kinds of things and i'm going to start this from the beginning but it is under two minutes and then i will get the heck off the stage um and tell you that you should ask me for stickers so it's got a bunch of things one of those i had said okay 13 uh go get the im credentials from gcp the same thing we just did get the cloud service account and then um come on j type faster highlight faster so that's for that same one i just got um and now let's take that and let's pull let's go looking in gcs in in the in the storage buckets for service count tokens oh look this installer this one was cops um in left uh left service count tokens uh there for all the components including a cluster admin which would be perfectly useful and so that's the like basically that's probably roughly what happened to capital one recently um where it was talk to the metadata api um get your g get your get your cloud account use that to um go rating for information or access um so anyway so now once i've done that peratties will basically store every every count it gets and then you basically say okay i'm going to switch context and you go and you go through and you collect all of the service count tokens then you basically have this big old store that's your loot and it keeps it in memory um and uh and you run around and basically get more access and get more access we've even got a this is in this video will be up on youtube but we've even got something that basically takes over the nodes uh if it's allowed to if it's allowed to mount a host path volume it'll go and add a cron job that will send a netcat shell back to your back to the ip address of your choice so once we have this cluster admin we're allowed to we can go and do everything which included like hey now show me all the secrets and all the secrets includes all of the tokens all the service counts for every other for every component and and uh peratties will not just grab will not just list these and get them but it'll also store them let you use them so it's a fun little pen test tool and uh it's up on github we'd love some help uh we'd love some help producing it so it's an inguardians tool we do a ton of Kubernetes work and guardians and i will now get off the stage and take questions all the way out in the hallway so if you see me please ask me for a sticker you