 But yeah, let's get the show started. I'm here, built to hack, hacked to build. Who am I? My name's Chris. I'm a security engineer at Heroku. We're pretty much remote code execution as a service. If you don't know what I'm talking about, go look the platform and you'll see what I mean. Find me on Twitter, Brompony. But yeah, I like hacking stuff, I like building stuff. I am lazy, so I build stuff that helps me hack stuff quicker, whether it's web, Windows, Android, and recently I've been looking at containers. They're a lot of fun. So this is what we're going to look at, problem, some existing research. I'll introduce to you Bob. So it's Bob with the silent T. And we'll look at some stuff that Bob can do. And yeah, this slide's more for me just to actually remind myself what it is that I actually share with you today. So we might deviate a bit, but I'll try to bring it back in. So what are the problems? So in a modern cloud, DevOps, SDLC, agile environment, there was one big question that I had to ask myself quite often was, how do we identify and exploit container vulnerabilities? So for a pen test or if you've got a shell in the box and you want to break out, you want to point some stuff, how do you identify and exploit container vulnerabilities? But then in an engineering environment, how do we test, secure, and monitor our containers? So I come from a red teaming, consulting background, and now I do more engineering. So I'm constantly trying to break our platform. And when you've got thousands of containers and you don't always want to pop shells and stuff, how do you do security at scale? So these two questions are very much aligned, but they're pretty much asking the same thing, which is, how do we basically identify and point stuff and break out of containers? So in terms of existing research, these are just categories of types of tools. This list is by no means complete, but if you're in a terminal, you've got a shell, and you want to find out if you're in a container, and are contained, identify as the sec, com, caps, LXC, or Docker environment that you're in, and it will tell you, very important. Then there's a whole bunch of Google container tools. I highly recommend you look at their repo. Great stuff in terms of static analysis for container images, be it a Docker or an LXC image. Clay is really good. If you want to look for vulnerabilities in images, it's a really great tool. And then when it comes to benchmarking, Aqua Security's done a lot of good work there. If you actually want to benchmark your images and your security controls that are in place. Now you'll see that from this list, there's not really much there in terms of actually breaking out and poning containers. So, I came up, I had this problem that I experienced. This was the existing research that I had. So generally when I like to solve a problem, I like to just hone everything if I can. So, that's why I did Bob. You can find it on GitHub. It's written in Go. Yeah, the hipsters were doing it, so I figured let's see why the hipsters doing everything in Go. I'm gonna know it's slightly ironic by my hat in beard, but you know, I'm in denial. So basically, like I said, I'm lazy, and I also wanted something, when you're writing the same exploits over and over again and doing the same breakouts, you kind of want to automate that stuff. So, Bob will give you the ability to auto-pwn. So, skids are gonna love that. Sorry I've had to say, but it's definitely a thing. Also helps you do not only breaking out containers, but things, Linux, post-exploitation. When there's a lot of things that you want to do and we'll get into it, you want to automate it. So, Bob will help you do that. And of course, you can do all these things in a pen test, you can't always. If you want shells, you get shells. But if you also want to help your engineering environment, so set things like return codes and plug into your CR environment, Bob's also designed for that. And I'll show you what I mean by that. So that's Bob. Like I said, we're gonna be pawning stuff. When you're pawning containers, the hollow world of pawning containers is exploiting docker.sock. It's a Unix domain socket that docker uses to orchestrate basically containers. If that thing is ever exposed, you find that it is game overs. So basically, it comes from people wanting to use docker don't do that, use that. There's better ways to do it originally. But basically, this allowed us to do a local privates to root and basically break out of the container. It's a really fun way to get into container exploitation. So how do you do it? What are the manual steps? So first, you've got to find docker.sock. So it's a Unix domain socket. It's speaking HTTP. It's basically a RESTful API on a domain socket. Then you want to interact with the socket with curl or docker clients or net cats. Once you find it and you can interact with it, you then want to create a new container and mount the host's file system in your container with the dash view command. You then run that new container and then you chroot or chroot or chroot, as I found this week. There are multiple pronunciations for that. And there you go. You are then pretty much root on the host and you have access to the host file system. And now you know if it's Linux and you access the file system and everything's a file, it's game overs. So that's the manual step. Now you don't want to do that all the time. Excuse me. But I think it's a little bit tricky in containers. Because when you're living off the land, especially in containers, the commands that you need, for example, the find command, might not be in the container. So if you're using really minimal images like the docker image or the alpine image, that's the thing. You might not have the tools to actually do the stuff that you need to do, like what if there's no, I have config or IP adder. Sure, you can read from ProcFS, but this stuff can slow you down. Especially if you need to find the docker socket. It's not always that var run docker.soc. It could be at moo slash bob. If you don't have the find command, you don't have recursive searches, how are you actually gonna find the socket? So this is where syscalls come in. So on the left there, those are the syscalls that the find command makes use of. And on the right there, that's what bob does in go code to do pretty much the find command. The nice thing about go is that when you start interacting with the operating system, go makes use of syscalls. It won't actually make use of actual shell commands. Some languages do that. So the nice thing is, is that if the command isn't there, you can use the syscalls. And the syscalls will be there to allow you to perform the actions that you wanna do. So that's what we do with bob. And I'm gonna show you a demo now. These are prerecorded. I'm done doing live demos because I just ran out of sacrifices all the time. So what we're gonna do here is, at the bottom terminal, so I know it's a little bit small, but basically that's our host. And we're just gonna watch a directory on the host. So you see there's one file called boblinux. In the top, we're gonna run it in a container. So we're just saying a Docker run and then we mounting a socket. Turn off the light. Turn off the light. Yeah, can you make it bigger at all? Yeah, unfortunately that's, can we turn the lights dim? Can we, no? Yes? Can you turn the lights? Yeah. Fortunately, I can't make it bigger because the resolution will go off the board. We can just break the lights. Ha, ha, ha. Ah. Ah, there you go. Ooh. Ooh. Ooh. Ha, ha, ha. The room full of hackers and you just turn the lights and you get that reaction. Ha, ha, ha. We didn't actually do that. Ha, ha, it's magic. Someone in the audience did. Ha, ha, ha. So we're in the container now and we're gonna run bob and you know, if you're inside a container you wanna find sockets and we're just saying dash socket equals true and what bob's gonna do is gonna go and look for all the unix domain sockets. Now you found a unix domain socket what now? What do you do? So of course you wanna auto-pwn this stuff. So I'm now providing dash auto-pwn equals true and what bob's now going to do is it's gonna look for the domain sockets that you can find and it's gonna try and pop it basically. And what you'll see now, bob is now giving me a TTY. So this terminal that I'm in now is not actually a terminal on the host. You'll see that that's a OSX host there and I'm not gonna create a file inside a 10-fold and that file is now gonna appear at the bottom terminal which is on our host. And we'll see that appear in a bit. So now I'm in a TTY that bob has provided me and you'll see that file has now appeared on the underlying host within the container. Now once you're done with that you can now exit bob and now you're back into your container's terminal. So that's one way that instead of doing those steps that I mentioned earlier, you can just basically say auto-pwn and boom you're root on the host. Of course if the docker socket is not authenticated or protected often it's not. So automatically just auto-pwn all that stuff. So it's quite nice to get a TTY on the host. If you want think-pen test you wanna get a shell on the host. Also if you wanna test maybe in your CI environment if there are exploitable sockets in your pre-production or staging environment you can tell bob to do this and we'll make use of return codes. So what we'll do here is we have a CI environment and we're gonna run the same thing. So what's happening over here is at the top right I've configured my CI environment to run bob and I've provided the auto-pwn but you'll see that I've added the CRCD equals true flag. So what that means is don't drop into a shell but if you successfully exploit that vulnerability set the return code greater than zero because if you're familiar with CI environments fail tests are marked as any test that has return code greater than zero. So if your CI environment runs a command a test and the return code is greater than zero the test is failed. So bob will do that. So this is very useful for engineers and for CI environments. And if you look on the left the test is running and we'll see that bob picks up a couple of UNIX domain sockets but because they're not exploitable they're not Docker sockets you'll see bob's found them the test will actually pass. So this is quite a nice thing that you can do if it doesn't make sense to try to get a shell now like if you're doing thousands of push requests pull requests a day you're not gonna want shells in this context but from an engineering environment you wanna test are they exposed Docker sockets is this thing ponable? So you don't want your engineers focusing on understanding okay now I've got to exploit Docker sockets like just run it automatically and then you can find out if that's the case. So it's very useful in CI environments but of course there's more to prone there's more to more than Docker sockets. It's called privileged containers. Anybody raise a hand privileged container in here? Cool so thank you for putting your hand up but it's the worst when you're at a talk and someone's like can you please put your hand up if you have seen X or A or B or C. It's always the worst but privileged containers are pretty much the worst thing in the world of containers why? Cue awkward silence. The case for that is I know I just try hard behind comedy and it's always an epic fail. So privileged containers are basically a I was gonna say another word there but it's not a good thing. And it's very dangerous when it comes to your syscalls that you have access to and the Linux capabilities. And this basically means very very easy container escapes but now you're in a privileged container. How do you break out? So I'm gonna show you a new feature that I released this week. What this does is Bob makes use of C group controllers and we're inside a privileged container and what we're gonna do is we're gonna abuse the C group Linux release agent capability and what that means is that we're gonna get command execution on the underlying host. This is how C groups work because we have access to the mount syscall and also because the Linux capability sysadmin has not been dropped, we can abuse that. So basically we mount the IDMAC group controller, we then kill off a process and then basically we get command execution. So you'll see now I'm using the Pone group command, I'm saying hostname and I've got a hostname of Docker desktop. So now I'm getting command execution blah blah blah fun stuff, let's get a show. So at the bottom I'm on my jump box, standard netcat listener, the great thing, if you're using Docker on OS X, be aware that your VMs, your containers are running in a Linux kit VM, that VM by default comes with netcat enabled, of course it's got busybox, but of course it's got the dash E flag is enabled, the evil command, you do not want netcat on your boxes with that. So it makes getting shells really easy. So we're gonna sell it, Bob, Pone the C group, so we're in the container, run this netcat command, connect back to our jump box and what's gonna happen is that we're gonna get a shell on the underlying host, which happens to be the Linux kit VM and in the OS X environment that's basically game overs. So because we're running in a privileged container, this is really easy, trail of bits that are great right up on this, C group, Pone Edge, it's really interesting and you'll see now we've got a shell at the bottom and you'll see we're on a host called Docker desktops, if you see that hostname, that is the VM that Docker for OS X uses and now we're in the host file system and if you ever wanna find out where your container is living on the underlying host, look at the mount command and if you look there you'll see var lib Docker overlay two, that's where your container is living and what I'm actually gonna do now is I'm gonna create a file in the Linux kit VM and you'll see that it'll appear inside my container. So I'm in the container and you'll see the OS does not like it because there's some Docker weirdness but you can see hello is over here and we created it on our jump box. So basically, Pone Edge C group, you break out of the container and you basically root on underlying host. If you're running something on a Debian system, it's running bare metal, you'll land up actually being root on the host. So of course, if you added the CR CD flag to Bob with this, it would exploit this but then set the return code. So you wouldn't be getting shells or you can just execute a command that you want. So pretty useful, privileged containers, they are out there but when it comes to containers, there's also environment variables, very, very good place to look when you wanna Pone containers, you'd be surprised what's in environment variables. So there's good old ENV, if you type that in a terminal, that's what you'll get but there's also the PROC FS in Linux. PROC FS, it's given me a lot of this hair loss that you see over here today. But basically if you read PROC and then the PID of the process environment, you'll get access to the environment variables that have been set for that process. Now this gets interesting because ENV is not always cleared and things might be living in PROC PID and this is when ephemerality becomes a problem. So you might do like a PS orcs and see okay, there's a process with PID-99, you go read PROC-99 environment, there might be nothing there because the process is dead. So you gotta be really quick when you're doing this. Bob does this really quickly via OS environment and Bob will go ahead and check the ENV, OS.ENV so like typing ENV in your terminal and go in the PROC FS and analyze that because sometimes there's the case where the ENV in your terminal is being cleared but the PROC environment mapping has not been cleared. So sometimes you can find a whole bunch of stuff because of course environment variables, it's the next general secret storage mechanism. So that's pretty much what the go code looks like if you want to process PROC FS. It's really nice and minimal. I'm a big fan of go as you can probably hear or see but how do we actually go about doing this? So we're gonna run a container and I should probably press play. That's a good place to start. And we're gonna add in an environment variable. So it's called touch of approval and super secret password. So what Bob will do by default, if you do the recon flag, if you don't pass a word list, Bob will look for password and secret. And you can see now Bob has said, okay, we've now found these PIDs that have this key term in PROC environment and Bob will then look in the ENV for the process that it exists in. And that's by default. But what you can also do with Bob is you can also supply a word list. So you can tell Bob, you know what? Analyze all the processes that you have access to and look for these words in this word list. So we're gonna provide a word list now and inside that word list were the terms password, URL, secret, find me and Bob went ahead and looked at ENV and in all the processes inside there. Now this is where the return codes also happened. So if Bob has found a keyword in a process environment that you're looking for, the return code will be greater than zero. That's what I'm gonna show you now. So now you're gonna use a word list where there's nothing inside. And you can see Bob didn't find any of the entries inside the word list inside any of the processes. And then you'll see the return code now will be zero because Bob didn't find any of those key terms. So that's all fine and dandy. How do we also do this in a CI environment? So you definitely wanna scan your environment variables in your pre-production production environments cause you never know what's gonna be lingering there. So in this case, and these configs are all the same on the CI environments that you're using. So we're using a shell script here and we're gonna be running two Bob commands. So we're gonna be doing the socket auto-pot. So we're gonna try to see if we have any vulnerable sockets and we're gonna be scanning environment variables and Bob will go ahead and do that on every get push and every pull request or however you've configured your CI environments. So let me just fast forward this. This is in true Chris fashion, I made too much content for what I have. So I'm like, let's get to the really cool fun stuff. And of course you can't double up speed with the videos that I make. I'm just challenged in that way. So in every CI environment, Bob is gonna look for these key terms. It is regex as well. So you can say, okay, look for S3 tokens or whatever. It's regex, the world will definitely burn at some point. So we can see Bob has gone ahead and run those two commands. And the test will pass because from the previous tests, we know that there are no exploitable docker sockets. And in the container that we are running, I remember now in this CI environment, all our tests are running in containers. That's how we configured it. And now we know great stuff that test passed our containers in pre-production do not have exploitable docker sockets and the environment variables that are there do not have any of the key terms that we specified. But of course, clouds, clouds, yeah. Clouds love to host containers. And of course we've got metadata services. So the 169.2.5.4 address, it exists in GCP or AWS Azure. And basically if you can get access to this, you can do a whole bunch of stuff. It depends on the platform. So for example, that cool request in AWS, there's an IAM role associated to that instance. You get the credentials for that. It'll change, it varies on GCP. There's a whole, yeah. That is a whole world and talk right there on what you can do with metadata services. Excuse me. So you rarely want to find these things. But also if you think Kubernetes, there's also APRs out there. Metadata service is basically an APR. And you wanna look for these things. Also you wanna find out what is on your network ranges for your containers. For example, EC2 classic, still around. If you use EC2 classic, and this is from the documentation, you're all the hosts. Every client that uses EC2 classic is all on one big, flat network. Yes, I know. What year is this? So if you are running anything on EC2 classic, your host is exposed on that 10 range. So if you're in container environments or any post-exploitation environment, you wanna find what APRs or what stuff's available. But if we go back to the container quirks now, you might not have Nmap in the container. You might not have Wget or Kool. You might not have the tools that you need to do the footprinting that you wanna do. So of course, Bob has a useful feature. So you can tell Bob, go ahead, and look for metadata services if you don't have Wget or Kool and all the interesting stuff. So you just add the metadata flag. And Bob, by default, will look for two endpoints, Kubernetes control plane endpoints, and the 169.254. So you can see now in this container, those two endpoints weren't there. But now we're gonna tell Bob, use the word list and see if the container can hit Heroku.com. And we're gonna specify the endpoint list, and then Bob's just gonna process that list and see, okay, can I issue a successful HTTP request to that host? And there we go. We see, Bob says, we've got a response from Heroku.com. And if we look at the return code, it's one. So you can use this in your CIO environments as well if you wanna check what the egress is. You wanna see, okay, I know that my Kubernetes or my control planes are sitting on these RPs. Tell Bob to scan those RPs in your pre-part or in your production, and Bob will tell you, oh, by the way, I can interact with this endpoint, which I shouldn't. And of course, when you're on a host and you wanna break out or just pawn more stuff, it's pretty useful when you don't have Wget or Kool. So there's also binary hijacking. This is awesome because a lot of orchestration environments explicitly trust the commands that they're provided with. So if you run Docker exec on a running container, Docker exec doesn't go and check, am I gonna run the legit LS command on the container? It just runs LS. So if you can control that container, you can override LS command to write your command. And we can do some pretty cool stuff with that. An example is kubectl or kubectl. I heard it's another pronunciation. You learn something new in Vegas every year. So with kubectl, it executes the containers or the pods tar command. That was recently exploited and used in an interesting chain by TwistLock. I highly recommend. But because as an attacker, you had access to the tar command on the host, you could mess with Kubernetes and basically break out. So pretty interesting. Important to remember that at some points in a container's lifetime, a binary might be executed. Binary I mean command. So maybe the power off command or the RP adder command might be executed in your container. So what do we do? We hijack all the binaries. So be careful. You can really break stuff here. You're overwriting binaries. So it's not too friendly for production environments. So I've been told. So if you wanna do this, what do you do? So these are the manual steps. So you gotta find out if you have read or write permissions to burn or S-bin or user S-bin in the container. You wanna create a malicious executable that does something. So either a webhook to verify execution or you wanna find something. You can go ahead and do that. You then wanna replace every binary that's an S-bin or bin with your malicious executable. And then basically you just, you wait and you can loot some hosts. So in terms of payloads, Bob will do this by default. We use webhooks. Why? Because sometimes if you execute, if you put in a shell payload and then you set up your handler and then it could be one minute, it could be 24 hours until you get a callback. Webhooks are useful because it will confirm exploitability. In this case, we're gonna say, okay, tell me which command was called and what parameters were sent to it. That's how you would do it in Bash if you were on a Linux host or in a container. If you were to write a Bash script, that's what it would look like. So how do we actually do this and what do we get from this? So you could get a callback in one minute or five minute or next week. It really depends. What I've successfully achieved with this kind of attack is getting tokens provided to call commands. So some platforms will say, okay, this client's running a container. It's been 24 hours. Let's back up the content of the container and let's push it to an S3 bucket. And of course, it's not an open S3 bucket. They authenticate it as they should, but because they're running the call command inside my container, I've overwritten the call command and they're actually calling my command and then because they're using a token, they have to give the token to the call command that they're running in my container to back up the contents. So you can find some interesting stuff over there. You can also prevent logic. So often client environments, in your hijack binaries hijack the power off command. A lot of orchestrators say, okay, this person has had two hours of compute, turn off that instance. Okay, cool, but now you've just hijacked the power off command and that orchestrator thinks that they've executed the actual legit power off command, but you've hijacked it, so you might get more compute time. And general mortgage, you can really break stuff with this kind of stuff. So hijacking binaries, it's fun. So enough of me talking, let's actually see this in action. So on the right, we're in the container and basically I'm telling Bob, go ahead and use the hijack command. And I know that text is really small, I apologize. You can throw stuff at me later about that. But we're telling Bob, hijack all the binaries in the host, in the container and execute this command. So Bob is gonna configure the host such that any command that's been hijacked, it's gonna execute a call command and that's gonna hit a web endpoint, which is the logs that you see over here. So I executed the ID command and you'll see that I got a callback over here. I was told, so I'm gonna say ID hello and then my logs on my web server say, okay, someone executed the ID command on the container and they sent the parameters hello y'all. So what does it look like if we were to run maybe docket exec on this running container? So that's what we're gonna do now. So this could be kubectl copy because they don't execute the talk command. So we're gonna run the ID command via docket exec and then you'll see on the left, we just got a callback because we've overwritten. We've hijacked the ID command. So that gets a lot of fun, that gets pretty interesting. I enjoy doing that. You'd be surprised as to what you can get. So Bob automates that whole task and it's a lot of fun. But of course there's more. What about all the CVs and all the lovely stuff that we have in docket? Yes, you did just see shell shock and bash bug and you probably like what year is it? It's actually quite a useful thing to actually load containers because you'd be surprised how old some infrastructure is. And if you're in infosec, you know, people don't patch things. So that's a good payload to know, especially container environments, especially for LXC containers. There's a whole bunch of stuff. That's some really useful CVs. But one of my recent favorite ones is the Run C ponage that came out really cool. Basically similar to hijacking binaries but here you are hijacking the Run C binary on the underlying host. That's what the vulnerability allows you to do. So basically from your container, you overwrite the Run C binary on underlying host and then you run in the context of that and because it is owned by root, you run as root. Now this is slightly destructive. So yeah, I always call this a last resort exploit because you can actually break the underlying runtime because you're breaking the Run C binary. For Docker and LXC it gets a bit tricky. Sometimes you need to be a privileged container. Sometimes you need root. This is because LXC and Docker treat privileged containers very differently. I won't get into that but it's really interesting. And yeah, what you're gonna do in this case and Bob is pretty useful is that if you wanna verify if a host is exploitable to this because you can't actually trigger your payload being executed. The payload is only executed in this vulnerability when someone runs a command inside your container. So if someone's running a kubectl copy or if someone's running a Docker exec. So you could wait. So in this case, we're gonna give a very nice webhook so we're gonna tell Bob. Oh, I hate cotton mouth. Sorry, excuse me. And I should probably place play or to play. If only someone invented that by now. So we're in a container in the top and we're gonna tell Bob, try and exploit CVE blah, blah, blah, blah, blah, blah. And the payload that we're gonna give is a code command. So we're gonna tell if the successfully is exploited we should get a callback on our host. So that will be the terminal on the left. So that's our code command we're saying, execute this. So now we've run the payload. Now you gotta wait with this payload. So now to trigger it, I'm gonna run a Docker exec and that's at the bottom terminal on that container. So you can't trigger this. And basically I'm gonna run an exec and I'm gonna get a shell because I wanna administer this container that I'm working from. So I'm gonna run binsh and you'll see Bob will exit at the top. So it says finished and now we've got a callback in the bottom left. So now that tells us, okay this particular container was vulnerable to that CVE and we now got a callback. So now you could go ahead and get another container on that platform and then give your shell payload. So pretty useful. If you wanna do that exploit manually there is a bit of work to it but Bob will take care of that for you. Awesome, conclusion. Yes and I haven't had my time cut yet because that always happens. So Bob can be used by both pen testers and engineers. As you can see, I can use it to get shells or to verify exploits ability. You can also use it within your SDLC environment so security at scale is really, really hard. And one important point here is that we are testing for two kinds of vulnerabilities. Immediate, so like getting a shell immediately or we are also getting delayed. So remember the hijacking boundaries or the run C that is delayed. I have 10 minutes. Was that 10 minutes Jeff? Awesome, this is like, this never happens. I apologize, your ears must be, I'm going really fast as well. Caffeine, I blame that. And of course it being 10 o'clock in the morning. It's like, get everything out before everyone falls asleep. So you see, I digress. Now I need to come back. Yes, Chris, what were you saying? Feedback, yes, that thing. So you're gonna be able to get feedback on two kinds of vulnerabilities. So if you hijack all the binaries, you have a web endpoint, you can get it at some time and also with the run C stuff. There are a couple other features that I have not touched on today. So there is some stuff that like, if you're on a hosting, you wanna push stuff to S3 buckets for data X-fold, it's pretty useful. I added that recently to Bob. There's also quite a cool way that you can scrape metadata stuff. So if you've ever gotten access to a metadata services, that's a real pain to get the actual content. There's a nice trick that you can do on GCP legacy metadata services. You just basically connect on port 80 with netcat. Don't send the host header, enter, enter, and blah. All the metadata stuff is yours. So I built that into Bob to do it automatically. So yeah, there's a couple other things. It's all on the GitHub repo. So all the documentation is there. I haven't gone through all the functionality, but yeah, it's all there. You can find it. All the binary releases are there. And yes, I thank you for your time. It's early on Saturday morning. And yeah, I'm gonna end it there. Thank you so much. Thank you.