 Hello everybody and welcome to the capture the flag Shake down we will go through what has occurred today and We'll actually replay the scenarios and look at What you could have won now notably we didn't actually have any prizes today, and that was intentional So let's explain why that was and what has happened Hopefully my screens should be sharing we shall see Okay So, yes, we we had a CTF today Hello and welcome. Thanks for your attendance And hopefully this will leave you mildly entertained. So what were we doing today? Today was a day of ambition. We aimed to educate a muse beguile and everything in between in a safe and happy environment And we had the absolute pleasure of introducing people to the joys of container hacking Scenarios are each built with a learning outcome in mind and our capable taskmasters also constructed the scenarios which gave us a little bit more closure on actually Helping and giving hints to people as they went through Notably absent from the list of goals is ranking This is because there's no official scores There's no prizes and the idea here is just to take the pressure off and to encourage people to go and hack stuff and to learn and to enjoy themselves Breaking things in a safe and importantly legal environment Although saying that if you have captured a flag or more And you'd like some some secret swag then do DM a physical address to the nice taskmaster And they will ship you something Worthy of your efforts okay, so today was hopefully an exercise in preparing for the unexpected thinking a little bit outside of the box and An encouragement that Nothing is entirely secure and we should pay attention to these things So we are in the UK time zone on this end. So please bear with us. It's been five o'clock for quite some time We had six scenarios One scenario per talk so if you did the first scenario it was concurrently running with the first talk and then consecutively throughout the day and lots of talking but nefarious small and potentially mild vulnerabilities Unless of course someone did better than we bargained for and the AWS build will be significantly higher than we thought So there are many places to look for vulnerable clusters But we believe the only way to upskill people is to give them a safe place to practice in So instead of going on cluster safari We prefer to run CTF based training But capture the flags can often be daunting big challenging filled with the kind of people who we we look up to and We want to break those barriers down and give everybody the opportunity to be involved So we hope there's been a strong first user experience there's attentive moderators in the channels and teaching assistants and Yeah, we just want to give people a boost on their cloud native journey so spoiler alert if anybody is still playing your time is up and we will Not be Validating any flags after this point So let's go through the scenarios and find all the flags now Excuse me while I shuffle my screens. Okay, so the first scenario Was called node secret breach So back to the screen share hopefully, okay So what happened here? Well, first of all We start we're starting as the root user and this is this is mixed of course Root has a special set of privileges in Linux. The root user is a specific and special user that means it has capabilities and those capabilities may include things like changing other people's files or opening a network adapter in a low-level mode so we can just either send custom packets or turn it Or set it into a different mode so we can sniff stuff on the network These are not things we'd want an average user to do and so the distinction of running in a container without user name spaces is important at this point so Let's see what else is in this container Let's have a look around. Well, this is what we expect, right? We've got a process namespace so we can only see the processes that we have available to us What else might we do we might have a look around and see what else is mounted? This is kind of noisy that there's not really much here. That's that's useful. There are a few things anytime we see a Docker Or perhaps one of these mounts. They could be interesting so Df the disk-free tool is a little bit of a quicker way to get a view on this in my opinion and Of course, we see a service account there service accounts are juicy and We are people that love messing around with service counts on this occasion though. This was not a service account based challenge and Okay, so we've had a look what's running. Sorry. We've had a look what's mounted within the container There's a lot here But some of the things notably a device from the host Other stuff that doesn't look like it's inside the container. Well, that's because containers are Wonderful, but they're not a perfect abstraction and that concretely means that Container runtimes have to put effort in to hide certain things from us and those things may include parts of the proc file system bits of sys the way that we interact with the process table And also the way that things are mounted in so okay containers are a child of evolution rather than intelligent design and Like everything else at the end to that are gaffer taped together No disrespect to people who've done an excellent job making them available to us Okay, but enough of this that there's a lot of noise here. We want to get some signal Let's think about what we can do with this service counts Right so we can see that Dev is here. So what does this mean? So Docker will mount in Etsy hosts using the Mountpoint from the host that it's on so wherever you store your container images those read write layers. That's where this is So already the abstraction has leaked with the file system here What do we do next then well I guess it's probably worth checking our privilege now. There is a canonical way of doing this Which is Jesse for cells am I contained of course? It's not doing anything magic. It's checking states and files that are available to us inside a container But it makes it very easy for us to do and gives us a unified view. So let's try and do that One thing to bear in mind is that we are roots and that means that we have the ability to run package manager commands We need to be able to write to any location on the file system. So we have to be root to do this generally and Yeah, there we are. So just install curl because without curl I'm gonna have trouble pulling in stuff from remote endpoints Although of course any programming languages and interpreter and most can be used to replicate curl in some way So let's pull am I contained from github to mod it and run it Happy days. Okay, so we've got some stuff here. What does this say? We're in Kubernetes. Does it have namespaces pid namespace generally that is always true anyway username space is generally always false because Username spaces are not enabled. I suppose by default in in run C or in Kubernetes This is interesting though We're unconfined in app armor. Now if we were running in Docker that would always have a profile Kubernetes, of course disables this and So it's just reinforcing the fact that we're in Kubernetes, but it doesn't tell us anything too drastic This however, what is this? This is a Wide bounding set of capabilities. So stuff in here. This means we can change ownerships override discretionary access control, which is basically writing on other people's files And all sorts of things in here that we recognize processes being killed raw input and outputs access to the network adapters this is Pretty likely to be a privileged container Okay, so we know what that means that means that actually what is in dev Is a reflection of what's on the host? So as a reminder the privileged flag the most dangerous flag in the history of computing as I will off to repeat disables Most namespaces it turns off app armor and set comp if they're enabled it grants all capabilities And it mounts all of the host's devices into the container This is a bad day for a system administrator. Why because we can say All right, we know which disk is mounted from the host. So let's Mount that disk into an empty mount directory Okay, so this is probably not what we were hoping to see because if we were blue team because That's what the roots of the containers File system partition looks like whereas what we've mounted in from the roots Let's get back there. We've mounted in from the host Contains extra stuff and lo and behold It contains a directory called node secrets. This may be obvious Oops, that's actually yep go in there and there we go. So there is our first flag The Disadvantages of running privileged containers are that they are equivalent to running the process on the host there is essentially no containerization going on when we run privilege and Yes run for the hills if that's the kind of thing you are trying to defend Because it's more or less indefensible. Okay, that was the first one Next we have escalate and cover secrets and Let's jump in so this alias will just take me into the into the next host Okay, let's spend a bit more time looking at what's happening here. So we're in the the attack container and Escalate and cover secret SRE has deployed a post-squares demon set and a new pod for monitoring the processes in your cluster Following a routine audit of an application It is believed that the deployment setup allows a compromised process audit pod container A mouthful to escalate its attack to other containers on the host This doesn't sound like it makes sense. There's obviously something a little bit askew here Okay, so the the question then is verify this by uncovering both the post-squares database password and the secret key We will start in a process audit pod okay so First of all, we know that it's possible to provision secrets via environment variables. This is This is something that the 12 factor app has told us historically But as we know the environment of a process leaks to other users Of the system and so we actually prefer to provision passwords with a with a file and an environment variable as a as a pointer containing the path of that file So I mean the first place to start here Then is probably just looking at the environment Let's just see what it's in here okay, well, there's actually nothing particularly juicy in here and Because we know I'll just go up and Because we know that the that the flag has a specific format at this point We can use that string To to just grep through the M. We haven't missed anything. There's nothing there okay so we're supposed to be Let's just remind ourselves this audit this process audit pod And we're looking to understand what the database Adjacent to us is doing so let's just see what we've got running Okay, this is a red flag For a cattle union festival We should not be seeing this many processes because as we saw I'll just go back up to where we were earlier as we saw in the process table Excuse me, it's further than I thought the process table here for the container which is Um isolated to itself We are only seeing a very small number of of processes and notably we see pit one Which is not system D or in a knit system. It's just a sleep command So that's what we would expect to see But actually we can see all sorts of things we can see stuff on the host so This is again a remarkably bad day The question here though is not how we We take over the host Although from from here. It's not especially difficult We still want to exfiltrate those Those crucial bits of data The secret and the key Okay, so This is probably how we traversed containers because we shouldn't be able to see the process list the things we can Let's list it in its fullest extent and have a look for postgres Okay, so we found postgres and just to be clear This is postgres running in a container on the same host What we're seeing here is a host Pids namespace share and this again is um Of questionable repute as a practice That there is some necessity for it, but we should always be careful when disabling A container security feature or a container primitive like host namespaces for networking or pids Or even mounting stuff in That we're breaking the container abstraction by adding stuff or taking stuff away Really we want to be locking it down and not sharing stuff Further with other Workloads on the system. Okay, so Right, we've got a pit for postgres And we can also see the postgres We can also see the pit so Let's do some magic proc diving So if we're going to proc and we put the pit in there Then let's start off. I'm just looking at the command Okay, so a foible of proc is that everything is uh is null terminated So in order to just get a bit of visibility into what's actually happening here, let's Replace null with uh just space this time. Okay, so we can see that it's just been invoked as postgres All right, no problem. That's uh, that sounds enough But what else is in proc? um, I'll give you a clue The environment and at this point even though that looks like a massive junk Let's see if we've got there is our flag So again using the flag ctf grep when there's a well known flag string Can help because otherwise it's kind of indeterminate mess of Of black and white that's a little bit difficult to pick things out of Okay, one down. So that is uh, that is a flag But if we go back to the beginning of this scenario, we were told There is more than one flag. So I'm covering both Database password and the secret key Okay, so we know we've got the database password, but there's a secret key We don't have any access to that adjacent postgres container except for through the The proc table. Okay, so What else can we do with proc? we can Examine the root file system mounted into that container Okay, so it's a sim link We don't mind too much about that Here we go again We have got into the file system of a container Running on the same machine as we are so we've got into the mount namespace essentially Without having access to the container itself If we go into secrets we can see that there is something it looks like a key there And if I have read line There we go Okay, that is key number two I'm conscious of time and I will try and keep on clipping Um, I so the point here is that there is a real risk with enabling username, um, sorry process namespaces and uh Yes, well, it's necessary. Obviously the feature was shipped for a reason We should be cognizant and Of course everything should be threat modeled and then we can balance the impact of that thing Actually being breached or negatively affected or or exfiltrated With the benefit that we get from using it in the way that it was intended Okay On we go and we're into a ci server vulnerability Right. So what are we doing here? We're pentesting a cluster. We found the vulnerability The pot is part of the build infrastructure as we know Uh, build infrastructure is a juicy target. Uh supply chain security is a Uh, a particular interest of uh of of SIG security and the cloud native security day of which we are a part And so build infrastructure. Yeah, that's okay. That's of interest to me. Uh, all right What's happening then? We're concerned that a compromise may lead to leaked secrets Okay So we want to extract the secret key and uh looks especially like we are in a Jenkins flavored pod Okay, so so again, we kind of just want to do a bit of recon. Let's just figure out what's what's going on process tables. Okay Uh the mount namespace So again, there's a couple of things here that jump straight out. One is a service count Um, of course, we can do the same thing again looking here But nothing really jumps out from there immediately Uh, that to me again is just a little bit of a mess Um, the thing that jumps out for me here is not the service count It is the presence of the hallowed Docker socket A socket is uh an inter process communication mechanism um amongst other things and In this case, it means that we can probably talk to the docker demon whose socket is mounted into the pod Which we're probably safe to assume belongs to the Jenkins host um Now at this point, we would hope the docker is installed Uh, we would probably check, uh, see the kernel is nice and recent. Okay. That's all good. So what release we've got Okay, it's all relatively recent. So we assume probably that we can install docker through the package manager But we can also do something by a back door because We have curl So let's just pull the official docker installer Um, maybe we'll we'll free ourselves from bugs in docker itself Uh, although those days are much further behind us Um, but it's nice to be on the latest version, isn't it? Even if we're going to break stuff Let's do it with uh, correct operator practices all right, so We're installing the docker client in the expectation that we can use it to abuse the docker socket We could just send restful commands over the docker socket, but it's a bit more long-winded And uh, right. So what we see here is the docker version command Has given us the client and the server Happy days. So let's see what's running here There is a lot and of course we see Kubernetes oops, uh, let's do color And the u.s. Spelling always yeah, and then we see kubernetes all over the place So it's probably again not a good day for somebody In this case we Probably want to look in the engine x container He says so let's um Let's have a look in here If you do a docker inspect in fact we can do it more elegantly with the um the containers sure Okay, again, there's a lot of information here We can kind of spool through it But because we're volume hunting and we uh, or flag hunting I suppose and we know the flag again Sweet there we go. There is our secret access key That looks so let's have a look at some context around that um And colorization is always useful Okay, so what's happened the environment of the container has specified This environment variable. So we're back into environment variables again Not only do they leak on the host on which they are running but also They leak from uh metadata about the thing as you see in this case It is an insecure provisioning method Because um, it's either set at runtime or set in the image itself. I ideally not of course so Really what we want here is um Is to instead point that secret to a file and this is as kubernetes will do with a secret file mount for us And that way we we need to have access not only to to the uh To the containers metadata like this, but also the container itself or as we saw previously the process table Etc Right at this point, um, we had pretty good, um I would say turnout we had we had a lot of people will uh, we'll get to those numbers at the end But also a lot of people got through these scenarios and uh, and at this point most people were still Still with us So let's persist next we have um a non user compromise Okay And out and back in again Okay, so what we'll be doing here um More uh escalation sideways. Um, so Just just moving laterally through um through kubernetes or maybe actually on the yep on multiple nodes So, okay, so secrets have been extracted from the cluster We're in a post mortem phase But it's not clear how the anonymous user managed to escalate sideways after the initial breach So let's try and replay the intrusion from inside the cluster Okay, so we're in the pod that was breached Excuse me, and we're going to Replay what the what the what we expect the attacker did um Now again, we've got our um our kind of basic recon, which is just seeing uh, how's our process table Uh, do we have anything? Spare or extraneous mounted in here? Um, we've gated them out points for the whole system Uh, we can install ammo contained and see what our bounding set is But in the interest of time Uh, this is uh, this is slightly different So at this point we are attacking things that are outside our domain as sorry outside our namespace, let's say and We have got the ip addresses of the nodes So What runs on a kubernetes worker node? Well, there's the kubelet. There's kube proxy. There may be Things for the cni as well The kubelet has some configurations that um Are less than optimal let's say uh, such as the uh the read only port so Let's just see if we can find anything excuse me for my mouse constantly doing that. Um In and around here So, uh, again, I know what I need to use in advance. So I will just install it I don't need to do an update Uh and get curl and jq So What are we actually doing here? Let's just see if we can access these host nodes So let's go Up in here And as a reminder, this is a network route from the pod On to the public interface of the host And really there's not a great deal of rationale for um For for running like this, uh, we should be using network network policy and um Yeah, frankly constraining our outbound traffic so that we can't hit anything at all start with a default deny And uh, and then upgrade Okay, that was smooth. Let's just sort that too So as you can see, I'm having to install a lot of software as I go along That kind of standard. There is uh, Generally, we don't ship curl and that kind of thing to production because why would we unless we needed it for our application but As an attacker I am able to install stuff inside the container Really just by setting a a non-route to user Maybe even removing the package manager if needs be that makes for a much Safe a day. Let's say the jq and curl if I installed those. What did I get wrong there? Nothing. Haha. Okay, and uh, let's um Skip verifying the certificates Okay, that means that we can read from the cubelets And we can see lots of things that are running here And what we care about Uh, potentially is something um compromised ish. So let's have a look Nope, there's nothing in there Uh, now that probably means that what we're looking for is on the other node So let's just remind ourselves Which nodes for which and switch over Happy days. That is the pod that we are looking for. Okay. So now we have more information about What we're looking to attack So what should we do with that? Well, let's see if we can dump the end of this Um, this pod so the pod is Um, what is it going to be the workload pod there? And then if you will just excuse my copypasta one second And this won't quite work because of the node. Ip. So let's just place That Ip is the one we've just used which is the second one Okay, so we're posting a command Oops, okay, so we're posting to the node. Ip Forced of habits On the insecure on the read only port rather And uh, we want we want to go into this pods And then run the print end of command Okay Um joy of all joys That is not coming back with anything. Have I done something wrong? um, I Guess the pod name should be pod name and not pod That was a great relief to me and as we go again There is our flag happy days Okay, let's uh steam roller on through the rest of these Um, we are on to pod breach extract and for this I will pass over to my uh worthy companion Magno Hi everyone I think you need to stop sharing Andy Okay, thank you This scenario here that we have for the for the CTF Is the the pod Pod breach tracked, right? So in this scenario, let me just Look in again here Verify your suspicion by breaking to the pod and extract the value of user cred's password, right? So and and you're starting point here You start on the virtual machine external to the cluster, right? So so how do you get access to to the cluster to the pods of the cluster itself, right? So um Basic things here in the interest of time, uh, I'll Good, uh, so first thing if if I need to access that That service or their service, right? I need to Uh, I don't I don't have the credentials, right? So I need to first do like a Your server make sure that everything's okay. I ran this before just before, uh, the example just to make sure that everything is Is uh running smoothly and we don't need to download it again so Uh, and here one of the the tools that you can use to to do that to do the network Mapping and scanning is and math, right? Thank you You will now be placed into Guys hear me again. Hi, can you hear me? You guys give me some feedback on the chat? Yeah, okay sounds good. Awesome. Uh, yeah, so as I was saying, sorry about technical difficulties here um So as I was saying we install nmap there and another uh, one of the things that we can do is to run nmap on this Uh, specific node for ip that was provided to us for this challenge, right? Okay, and I've selected a few ports here. So the scan doesn't take forever, right? So specifically for uh, just the demonstration here, uh, and we can see that there is a port open on on this higher port here Uh, free zero zero two two, right? so one of the things uh, and it's very famous on on on think testing and and like application security scenarios is Uh, brute forcing, right? So one of the attacks that you can run Uh, you can get a list of user name and password then try to just brute force the system to gas those Prudentials, right? And one of the tools that you can use here for brute forcing, uh, this This port and and access the service, right? So I don't know exactly from the nmap there I don't know exactly which server it is, but I can uh I think I can run this one. Let's see if it's going to give us to us A little bit more information there if it takes too long then You can just move on. Well, we can see that this service, right? So it was showing on the first scan was showing as unknown, right? So I run, uh, I added the flag dash a here. So to scan the services and the versions as well So we know that this port is running an open ssh server there And and then and now I can try to To brute force it right passing a list of username and passwords, right? So one of the tools that you can easily do that is called hydra And uh the way to install it just typing oops in this command here Okay, so I have hydra installed already just to show you guys here And now what I need to run this brute force, right? I need a list of username and passwords, right? One one of the the very common List of passwords is the rock your list. So you could have downloaded that and use it But this server this this server here is not very protected. So the The password is not the username and the passwords are not very hard So I created a list here already on This server called the list of users. So I have a list of user names here And I have a list of passwords that I could try so it doesn't take forever when running, right? So here's the command that I'm going to use for hydra to Try to brute force this This username and password on this server location here Okay, uh, let me just show you guys, uh, so Just moving on here. So we don't lose a lot of time. We're almost running out of time here Uh, basically if you run this command, you should get the Uh, the results I had it here before I know I I really Uh, okay sounds good almost done Uh, so yeah, basically just After I find out the username and password I can access this server oops copy and paste here and So the the user's admin and the passwords passwords very very easy for you guys You know, even if you don't have like a tool you could try gassing it and access that Okay, so I'm inside this Called jump box, right? So What I'm gonna do here is try to uh Connect to the api server and make some requests So base base The the base request here that I can make is check the version of the the kubernetes api server This is one of the ways to do that, right? So I can see that's running our version 119.4. That's like that, right? So just to wrap up here, uh, one of the things that I can do is Well, uh as as Andy mentioned before, uh, there is some Secrets and tokens inside the pods, right? So I can grab that and and use to impersonate The pod to talk to the api server, right? So basically here what I'm gonna do is just create two variables the namespace Telling that's located at var run secrets kubernetes.io service account namespace And I'll do the same thing for the token here cube token. Okay, almost done And then now I can make uh, if I have uh permissions to do that I can make a request to this namespace and uh Via the api server and ask for the secrets and since this is a misconfigured cluster. I have the permissions Uh, so You're just just making this api. Let me show that again, right? So this is the the request that I made and I can see all the secrets and here is the flag user crats password And and that's it for this challenge Uh, I'll I'll hand it over to Andrew again. So thank you guys, right? Awesome. Thank you very much Okay, let's zoom through the ends of the slides There was one more scenario, um, which people didn't quite get to so, um, if you would like to go at that then Do feel free to DM Right, let's get through to the end. So, uh, this was this was the control plane team at seven o'clock this morning We are just about through the other end of it So honorable mentions. Thank you to these individuals for, uh, being with us through most of the journey today and, um We had a great time on DMs. It's fair to say, uh, there's some really, uh People showed some real grit and persistence and and that's what it's all about. So good job those people Um, we have some honorable mentions as well um Dilshan, Matias, Michael, Matthew, Walid, Steve, Muhammad and Noel were all there for the whole journey And thank you very much for your efforts in parting the, uh, container defenses Um, various people enjoyed themselves slightly Uh, and I hope this has been a beneficial learning experience. I'm sorry. There was no cake And thank you very much to the control plane People at this end for for manning everything and to magno for all his assistants and helping us out and testing, etc This is a public service announcement We don't run administrative endpoints on the public internet. The kubernetes api server is one of them If you like what you saw today control plane do this for a living and, uh, we'd be happy to, uh Stand up a ctfu Thank you very much for your attention and thank you to everybody who contributed today We had a total of 327 clusters spun up. Uh, that is about 15 000 nodes Um, we had a peak of 73 users So thank you for everybody who played and, um, we Welcome all and any feedback Uh, have a wonderful day