 Hey, everyone, we're going to get started. So my name's Greg. This is CJ. We're from the communities and GKE security team at Google. And today, we're talking about practical attack and defense in Kubernetes. So we're security at in Kubernetes at the moment. The community is working really hard on security controls. There's already a lot of defensive options. And there might not be all that obvious way to start or how to prioritize. So we're not going to cover all the security best practices in this talk. There's a lot to cover. And so today's focus is going to be mostly on prevention. We're not going to be talking much about detection or instant response or application security or a bunch of other kind of topics that we could be talking about. I've only got 35 minutes, and we're trying to make it entertaining by having some demos. So we're focusing on cluster administrator and developer tasks. And we're focusing on Kubernetes. There's nothing GKE specific in this presentation. We have a link to a blog post at the end that gives you a whole bunch of hardening specifics for GKE. So the documentation has the how. Today, we're trying to give you the what defenses are interesting, why you would apply them, in giving you a kind of like a real attack scenario to experience and visualize, and a sense of the relative priority of those things. So we're going to keep with the pirate theme. The Kubernetes threat model assumes application compromise. So if you consider the entire world of Kubernetes clusters that are running in production, there's lots of bugs. There's lots of vulnerabilities in that whole spectrum. So it's really kind of what happens after code execution that's actually interesting for Kubernetes security. And so our goal is always to be secure by default where we can. But often we have some sort of opt-in step for new security controls, like we did with RBAC, to make sure we can keep that backwards compatibility and give people a migration path. So we have three demos, which is a lot to fit in 35 minutes. But hopefully it keeps things interesting. And the idea is to show you an attack of landing in clusters at different stages of security evolution. So the first one is RBAC is off, or it's in an old cluster that doesn't have RBAC. And there's a fairly well-known kind of application to cluster escalation path if you have a cluster like that. So I want to make people aware of that. The next one is we kind of take that cluster and apply some hardening to it. And we try the same attack and demonstrate that it doesn't work. And then we move into like a container breakout and privilege escalation kind of problem that can get you access to the KubeLit. And then from there, we apply some more hardening to that cluster. And then we run the same attack. And once you've closed off the obvious privilege escalations and the attacker looks around a bit and there's kind of not an easy way to escalate on that node, they probably look to propagating through the cluster. Can I look for vulnerable services that are running in the cluster? And so we'll demonstrate kind of how that might work and then talk about some defenses for that scenario too. So I said we were going to keep the pirate theme and we're going to actually assume roles here. So I'm going to be Pirate Pew Pew. And so I'm going to be kind of wreaking havoc on these clusters in an adorable kind of ragamuff and rosy cheek pirate kind of way. And I will be your skipper-shipper. And behind this well-provisioned uniform and cap, I hide some slightly lax coding standards and a bit of security naivete. So to demonstrate some of these attacks and the defenses that are available in Kubernetes, we made up this fake app. This isn't real. This is not a real thing. But this will let us talk through some of the things that you might think about. So taking a look at what this app looks like, we're going to have a new member webpage that'll allow all of our new members to sign up. We will store their info in some database. In the back we'll have a payment processor that churns on the database, takes our new members and charges them, takes our winners and pays them out through some third-party payments API. And then just so our admins have a view into the payment system, we'll build them an admin portal. So yeah, let's ship this app on our first cluster. Okay, let's start some port forwards. Readable, yeah. So here we have our web front end that Skipper Shipper has built. So I've already done some reconnaissance on this web front end. And I'm pretty sure there's a shell injection in this input field here. So some pretty lax coding standards going on that Skipper Shipper's been enforcing. So looking from my reconnaissance, I've kind of realized that he's probably taking this input from the internet that's completely untrusted and shelling out and making a call with it without doing any input sanitization. So this is obviously a huge no-no. So I'm going to go ahead and exploit it. So I'm going to close off the previous command on the semicolon and then download a backdoor and run the backdoor. And over here I've got my MetasploitListener and that's a penetration testing framework that allows you to run an attack and have installed these backdoors on systems and then connect back to this server. So this is my exploitation server. So I'm going to go ahead and hit submit on that. And you can see I have a shell now on that pod that I've compromised. So I'm running in a dub dub dub directory. It's actually readable, yeah. And you can see my exploit has landed in there along with the kind of PHP code and other stuff that was already in that directory. So I mentioned that RBAC is disabled in this cluster or it's an old cluster that doesn't have RBAC enabled. And so since in the interest of time I'm going to cut kind of straight to the chase here. So I know in clusters that don't have RBAC enabled there's a service account that lives at this location on the file system that's very privileged. So let's just see if I can see that on this system. And I can, so this is the token here that's the secret for that service account. So I'm going to use Metasploit to download that token and then I'm going to on my own attack on machine just write out a cube config file that includes that token so that I can make requests to that API server. So now if I switch over to my attacker machine, so this is the cube config file that I wrote out from my attack that includes the token that I stole. And this is completely separate from the cluster now. So I've used Metasploit to extract this token out to my own machine. And now I'm going to come at the API server from completely outside the cluster using that token. I'm going to try and get pods. That works so I can see the pods that are running in the cluster. What else can I do with this token? So let's try getting secrets for all namespaces. And I'm just going to grab out the cube system stuff to kind of keep this the output down a little bit. So I have a couple of Kubernetes service account tokens here which are not super interesting. I already have one of those. But this payments API key sounds really interesting. So from skipper shippers kind of overview of the application, this is how money gets transferred and stuff gets done inside the application. So let's try getting that secret because that one sounds interesting. So we're going to get that payments API output in YAML format. And so we can see this here is the base64 encoded secret. So that's actually clear text. It's not encrypted in this view here. So I now have that payments API secret and I could make some requests to the payments API. But it actually gets worse. So this service account, as I said, is very, very privileged in the cluster when you don't have our back running. So I can actually just exec anywhere in the cluster into any pod. So I can see there's this payments pod. So I'm just going to run a shell inside there. So now I've got a shell from my attacker machine all the way into the payments pod. And you can see the payments related code here is running in this directory. So that seems kind of bad. Yeah, so let's walk through what happened there. Our attacker was able to find a somewhat obvious pretty simple shell injection into our web facing front end. From there, he knew that there was likely a service account token just kind of hanging out inside of that pod. Using that service account token, he started probing around. He proved out that it had a little bit too much access on our API and from that point, he was able to exec directly into our payments pod. Maybe he changed the code. Maybe he's now stealing credit card numbers. He was also able to just directly take our payments API key and probably cost me a lot of money. So let's talk through what we could have done to help prevent that. Obviously you want to enforce lease privilege on your workloads and the most obvious way to do that in Kubernetes is to turn our back on. That means disable the legacy A back or make sure you have some authorization on what your service accounts are allowed to do. Once you, and that'll actually make it so that service accounts have no privileges by default. You have to grant them if you want them. If you do want them, it's nice to separate out your workloads by namespace because then you're able to say, well, this workload can only have access inside its own namespace, whereas this namespace is completely separate, allows you to get that lease privilege a little bit easier. And then lastly, there are ways to firewall off the network access to your master. So if you know that you're only going to be accessing your Kubernetes API from some set of machines, you can say, I only ever want to see traffic from this set of machines. And what that would have done in this attack scenario was made it so that our attacker couldn't just take that credential off to the comforts of his own machine where he probably has more tools, more time, more ability to probe. It would force him to maintain presence in our cluster if you wanted to continue this compromise. So let's fix up some of those and let's try to ship again. So we're going to apply two of those defenses. We're going to not do the firewalling because it makes it kind of more interesting to see how the defenses got applied. But we're going to turn on our back and we're going to segregate by namespaces. So same thing again, we left the vulnerability there, but we have a new cluster now with those defenses enabled. Same exploitation pattern, and we have our second session open now. So Metasploit actually allows you to have multiple at a time. So same thing again, we have a shell. Let's download that service account token. So downloaded it to my own machine, wrote out another cube config, and I'm going to try the same thing I did last time. So let's try and get pods. So in this, now that we have our back on this cluster, that service account is no longer privileged. So we can see here the service account in the sign up namespace, the default service account there, countless pods in the namespace default. And we can try some of the other things we tried. So stealing the payments API key also fails. And we can keep trying stuff that we did in the last demo, but by default the service account has no privileges. So this is now a dead end. So all right, what's next? Let's look around a little bit. So we're running inside unsurprisingly a Linux container. Let's have a look at the mounts that are on this system. So it's a big pile of text here, but this actually looks pretty interesting because it looks like there's a host path mount here that's dragged in a whole bunch of stuff. So host path mounting is a way to mount stuff from the host into your container. And sometimes you might use that to do some sort of writing of files or reading of files from the host. In this case, Skipper Shipper in keeping with the kind of like fairly fast and loose pace here, has accidentally mounted in the whole Kubelet directory. So the Kubelet is responsible for scheduling on the node and it's a fairly powerful agent. So let's go poke around in that directory and see what we can find. So there's a bunch of stuff in here. PKI sounds really interesting. This kube config file will probably tell us what's going on in this directory. So let's just try and cat that out. Okay, permission denied. Who owns that file? It looks like root owns the file and it's readable and writable by root. So who am I running as? I'm running as the Apache user, www.datas, fairly common setup. So I don't have privileges to read that file. And at this point, kind of in a real attack, I'd be looking for privilege escalation opportunities. But since this is Skipper Shipper and things have been pretty fast and loose so far, I'm just gonna try out this and see if this works. So essentially what we've done here is simulate a container breakout and a privilege escalation. So to make this demo a bit simpler and not require us to do all that work, we've simulated a container breakout and a privilege escalation by mounting too much stuff into the container and then being a little bit crazy with our sudo privileges. So we're gonna take the, you can see here that there's a Kublet client key reference and a Kublet client. So we're actually gonna take copies of those. So we're gonna copy them out of that host path mount, write out a kubeconfig file that uses them, and then we're gonna try using the kubelets credentials against the API and see what we can do. And to do that, I actually have to download kubecontrol because I'm running inside a container. I don't have kubecontrol in there with me. So now that I've done that, I can use the kubeconfig file that has the kubelet credentials and my copy of kubecontrol I just downloaded. So let's see if I can get secrets. So this fails. So this is the, it says this user, which is actually the kubelet for this node, countless secrets in a namespace default. I can only get individual resources of this type. So I can only get individual secrets. I can't see all secrets. Okay. How about show me all the pods then? So this is all the pods in all the namespaces, just cutting out the kube system stuff. I can do that. And the reason I can do that is kubelet needs that to do its job. So kubelet has to, is scheduling, it needs to know kind of what pods to schedule on itself. So if I can, now that I know the name of pod, maybe I can try exacting into it. That's what I did last time with the service account, that really privileged service account token. So I'm gonna try exacting into that payment spot and see if that works. So that doesn't work. So I'm, the kubelet actually isn't allowed to exact around inside the cluster. So you can see our back working here too. It's scoped the kubelet credentials down to the least privilege required for it to do its job, but it's still a reasonably powerful agent. So it can, as kind of hinted by that previous permission denied, I can actually get individual secrets because the kubelet is the thing that's responsible for mounting the secrets into the container. So I'm gonna try grabbing this secret and I can still steal that payment API key secret. And that's still a little surprising really because this is the web front end and it has no reason to be able to need to access that API back end secret. It isn't actually necessary for this front end to function. So this seems better but still not great. Yeah, so let's take a look at what happened there. Attacker found the same old challenge action. We didn't remove it. It's more fun that way. He took a look at the pod service account token. Luckily it was not able to do anything in our cluster, but we left him a nice little obvious escalation. He was able to grab ahold of the kubelet token on the node, the kubelet credentials on the node that he was running on. From there he did have some access to the API. He was luckily not able to exec directly into our payments pod again, but he was able to grab that API key and probably likely cost me a bunch of money again. So what could we have done to prevent those? Obviously limit the local escalation paths that attackers might have. You don't wanna avoid running as root, really take a look at what sort of host path mounts you really need and you can enforce these sort of things with the pod security policy. From there you wanna ensure that all of your nodes actually have least privilege. So on Kubernetes 1.7 and up, this means enabling the node authorizer and the node admission controller. And as you saw in that cluster that we were looking at, we actually did have that authorizer enabled. That's what prevented the node from being able to get all the secrets. It was only able to get the secrets that were relevant to pods co-located with that node. So once you have least privileges for nodes, you may want to go further and separate sensitive workloads from the more possibly, the web-facing type workloads that may be more vulnerable. And there's some scheduling primitives that allow you to do that. And that would have prevented our attacker from being able to grab the key if it wasn't co-scheduled with him. And then lastly, you can turn on certificate rotation for those Cubelet credentials. And what that'll do is make sure that the attacker doesn't have a credential that he can just take and use it as leisure for some long period of time. The lifetime of those credentials will be shortened. They'll be rotating out from under him. Again, to maintain this compromise, our attacker would have to maintain presence in our system. So yeah, let's fix those things up and now let's really ship this thing again. Let's just get set up here again. Same thing. We're gonna compromise pod again. We have our third session open. Let's have a look around and see how we did with the hardening and removing those problems from the previous cluster. So let's see if we still got that directory from the Cubelet. We don't have that anymore. Let's take a look at the mounts to see if there's anything crazy in there. And they look much more reasonable. We don't have all those host path mounts. So at this point, we don't have that kind of obvious Cubelet escalation path to break out of a container. We, without that vulnerability, I can't really demonstrate having the scheduling changes that we made, but now the signup pod is scheduled on a different node to the payments back end. So if we were to steal secrets, we wouldn't be able to get that payments API key. So let's assume now that we're kind of attackers landed in this cluster and we're like, oh damn, no super obvious privilege escalations, no obvious root stuff. Let's take a poke around in this cluster and see if we can propagate elsewhere. Let's look for some vulnerable services, that kind of stuff. So really like traditional obvious way to do this is using the end map utility, which is a port scanner. So I downloaded a version of end map that's statically compiled. And I'm gonna run a scan across, just looking at port 80 across the services IP address range, there's a slash 20 that is used for services in this cluster. This takes like two minutes to run, which is pretty much an eternity for this presentation. So this is the only thing that isn't live. So here's one I prepared earlier. This is the end map scam results. So you can see here that I have a signup front end running on port 80. I have a payment service running on port 80, as well as some like internal Kubernetes, other stuff, heaps to dashboard, et cetera. So payments sounds interesting. Let's go with that. So I'm just gonna curl it, apologies for the kind of big pile of unrendered HTML here, but sometimes hacking is dirty and gross. So the only really interesting part of that HTML is that it's a directory listing that shows us there's a payments PHP app. Now we could actually just kind of stop the attack here because this is kind of demonstrating the problem that I shouldn't be able to get to this payment service from this front end API, from this front end pod that I've compromised. But I think I wanna make it rain and I think we've been talking about payments a lot, but not actually getting paid, so let's keep going for a little bit. So let's hit the payments application and I'm super glad about these really helpful error messages that I just give a shiver, it's giving me because here I can see, oh, actually, would you like to make a payment? Because I have a make payment command. So let's go with that. Yes, I would like to make a payment. Oh, to make a payment, you need an account number and an amount. Okay, I can definitely do that. So let's go with my Pirate Pew Pew account number and my elite amount of dollars and make it rain, skipper. Yay! Come on, that's an applause line, come on. Okay, so yeah, not bad, but still kind of problems. Yeah, let's talk that one through. So what do we have there? Same shell injection, same service account token, still no permissions, no way to escalate directly that was obvious, so at that point our attacker starts doing the normal attack things and kind of probing around, seeing what he might have access to and unfortunately he was able to directly call this payment service. Now when I designed this wonderful app, I never imagined that my signup form would have any reason to contact my payment service directly. So this was very surprising that it would have this access, but our attacker found this, made a call, and cost me some money. So what could I have done there? As you separate your applications out into microservices, there are these sort of natural security boundaries where you get to start thinking through what pieces of my system should really be talking to which other pieces and in Kubernetes the network policy object in 1.7 and later lets you start to define and enforce that. So in our case that means an ingress policy on the payment app based on the label that says that I will only accept connections from things that have the label admin portal. So the only thing that should be directly talking to my payment processor is this admin portal that I'm helpfully providing my admins. Then the other thing that in the past was often floating around the network was the Cubelet API and you really wanna make sure that you have authentication and authorization turned onto that. There are some fairly privileged things you can do by directly hitting the Cubelet API and you definitely wanna lock those things down as well. Okay, so we've mostly focused on, well we've entirely focused on existing defenses and talking about prevention. So just wanna sum up, try and give you some kind of takeaways. We talked about a lot of different defenses here and to try and help a little bit with priority. So hopefully we fairly convincingly made an argument for turning on our back. There is that very powerful privilege escalation that you really wanna lock down and it also gives you the ability to really apply these privilege properly to your cluster if you're interacting with the API server at all. So another really important part of this is keeping up with the Kubernetes releases. You'll see in all those defensive slides we were talking about the versions where those defenses are available and a lot of them are kind of one, six, one, seven, fairly recent versions. That pace is gonna continue and so there's really a lot of value in keeping up with the latest versions of Kubernetes. So like if you're not doing either of those, I consider those kind of probably the highest priorities. From there, having a really kind of small container OS actually makes a big difference. When an attacker lands in your cluster, the kind of more comfortable they are, the easier it is for them. So it should be fairly uncomfortable landing in a cluster. So at one extreme that would be like kind of building like scratch containers but they get really hard to debug. So at least a really minimal OS like what we have with cause on GKE. And then thinking about, kind of carefully thinking about, do I really need Wget and curl in this container? Do I need a shell? And there's trade-offs with debugging there. But all of those things make attacker actions a lot easier if they're available. And obviously, as we said, no root, no host path network, where you don't need it and you can enforce those things with pod security policy at the cluster level. And then the, as kind of CJ mentioned earlier, the this decomposition into microservices gives you these really natural boundaries for segregating out your cluster and then building network policies and dedicated nodes to kind of make propagation harder. Once an attacker does land in there, making it as hard as possible to get to the other places in your cluster is a really valuable thing that is in high effort. And there's a whole bunch of companies that are doing work on kind of learning network, networking patterns and then helping you enforce network policies kind of automatically. So you don't necessarily have to write that, write those manually yourself. So we have a great community, I think, at SIGOF. We have some really stellar security engineers not talking about myself particularly, but definitely the other people who are working in SIGOF, I don't know. I'm very glad to be working with this community because I think there's some really great security expertise in it. And we'd like other people to get involved. So we meet every two weeks on Wednesdays. A bigotry shout out from my team in Seattle is hiring. So if the intersection of security and Kubernetes is interesting to you, please come talk to me. And here's a whole pile of links. So everything that we talked about today in terms of the defense is on this slide with a link. I definitely want to call out again the GKE Hardening blog post. And there's a, the second link there is a living document for securing your Kubernetes cluster that won't age like this presentation will age as things kind of change. And a couple of other call outs. So if you're expecting to see kind of security roadmap of features that SIGOF is working on in this talk, we didn't have time to do that. We kind of wanted to deliver, here's what you should do and why you should do it. But Jordan Liggett is giving a SIGOF update that has a good list of the things we're working on over the next few quarters. So definitely go to his talk. Yeah, thanks, happy to take questions. Yeah, the question is, can I use port security policy on GKE? No, you can't right now. On the slides I said coming Q1 2018, I think we should be, it should be early January, I think when you'll be able to turn that on. So I think the question is what consideration have we given to enforcing L3 level partitioning based on L7 level traffic, being able to automatically discover that sort of thing. I think from the Kubernetes standpoint, we don't wanna be really, really opinionated about that. I think different people have different opinions about how you wanna think about application security inside of your cluster. And we wanna make the tools available such that the information is there that you can build on top of it. But the end game is definitely up to you, I think. I think too, in that space, there's probably three or four vendors on the vendor floor right now that have built systems exactly to do that. So I think the Kubernetes kind of perspective is that we wanna keep things out of core where they don't really need to be in core and then provide opportunities for other people to build tooling on top. And where there's already kind of like, that's already happening and I think that's probably a good indication that we have the right extension points already and we probably don't need to build that in the Kubernetes. Can I name the vendors? No, I can't. They're on the demo floor. Over here. Yeah, so it was actually, in the second demo it was actually, so the question was, why could you get that payment secret if the payments thing wasn't scheduled on the node where you were, where we'd compromise it? So in the first demo it actually was, we made sure it was co-scheduled and then kind of like to fix it, we separated it out. But yeah, that's exactly what the node authorizer and admission does. It makes sure like, yeah, if the blast radius for compromise of a single node should be the secrets for the pods that are currently scheduled on that node and not all the other secrets in the system. So that's, yeah, as of one seven, that was the case. Yeah. Yeah, so there is a bit of a weakness in the tanks and tolerations mechanisms at the moment. So the KubeLoc can change its labels. That's exactly one of the things that, sorry, the question was, like if you have the KubeLoc credentials, could you change something to get the other stuff scheduled on you where it isn't supposed to be? That's, yeah. And so yeah, there is that weakness at the moment and that's definitely something we're working on in 2018. We're gonna close that one off. I would add that KubeLocs can't directly schedule pods onto themselves though and they can't create new pods. Yeah, they have to change the labels. So if you're using the tanks and tolerations and the label selectors to put them on different pods then the KubeLoc can change those. So I think it's still worth doing that at the moment anyway because the protections aren't gonna come and then you'll already be set up to use them. The question was, is that the case even when you download KubeCTL? And yeah, it just depends on what identity you're acting as, what credentials you grabbed. So KubeCTL is just gonna use whatever identity you were able to grab and if that's a node identity, a KubeLoc identity, you're restricted by the permissions that that KubeLoc has. Yeah, so the prerequisite would be root on the node to get access to those KubeLoc credentials. So you'd have to do container breakout, privilege escalation, steal the KubeLoc credentials, then do this. So there is definitely like a, that's not a trivial process. There's serious work involved in that. Yeah, yeah, that's basically what network policy does. So is it possible for pods in one namespace at the network level to be blocked from accessing pods in another namespace? That's what network policy does for you, yeah. One more minute. Yeah, so the question is like, how do you respond to an attack like this? Like how do you detect it? How do you respond to it? We didn't really cover that at all. There's definitely a bunch of stuff you can do. So one really basic thing that you definitely wanna do is collect the API master or audit logging. So all those interactions we were making with the API server all get logged and then you can build detections on those. On GKE, they all go into a stack driver if you enable logging and you can do really quite powerful querying and detection on the API stuff there. I think there's also the scope and a bunch of products kind of operating in the space of detecting stuff on the node itself. And so there's a bunch of people building products right now to do detection of unusual events, things that are happening on the node itself. Is there anything else you'd add to that? Yeah, okay. I think we might call it. Thanks everyone.