 All right. Talk is going to be a little bit about a container escape. Those of you that are unaware. Enjoy. What's up, Defcon? Yeah, mice are hot now. After that, I think we're going to do great. All right. I'm Dagan. Hi, I'm Will Klein. So we're all here as, you know, we're mostly like SRE, DevOps type people, setting up clusters. We're not pentesters. So, you know, but when we're going through setting up these clusters, we notice some things and kind of repeating patterns of, you know, security issues that aren't being talked about enough. And that's kind of what we decide to put this talk together for. You don't have to be a Kubernetes developer to understand what we're talking about here, but you just got to think like one and there's no zero days today here. This has all been disclosed properly. And we're going to have three different demos that we're going to run through different types of exploits that we want to tell everyone about. With that being said, with our demos at the end, I have a GitHub URL. The demo environment is set up in a local laptop using make and that file is available. So anybody that wants to can go and just clone that repo. You'll be able to step through each of the demos we do today yourself on your laptop. And there's also a step-by-step walkthrough. All right, let's get started. WTF is a Kubernetes. I like to think of Kubernetes as the operating system for the cloud. It takes anywhere from one to several thousand nodes or individual servers and turns it into a single orchestrated system. That system can span an entire data center. It can span multiple data centers and multiple clouds. It's the ultimate shift left platform. As a Kubernetes developer, not only are you able to take advantage of containerization to bring your file system and all of your dependencies with your application. You're also able to declare the exact way that your application should be instantiated. You can say I want to run this command with these args. You can include your configuration. You can say I want five instances of it and I don't want any of those instances running in the same availability zone. You can go further and declare how much CPU time should be allotted to each of your applications, how much memory is reserved for your applications. You can say this is what I want you to do to monitor the health of the application and how to respond when it becomes unhealthy. Even how to scale the application. Kubernetes is an extraordinarily powerful system. The other thing that's really nice about Kubernetes is that you basically have decades of Google's experience and other large cloud providers' experience in setting up these large scale systems. They've distilled all that down in a series of APIs to help you at any scale you want, whether it's like a Raspberry Pi up to as big as Google, deploy these applications and manage them. You get a lot of lessons learned out of that. It's really powerful system for managing your systems. Today we're going to talk about pining a cluster. I want to just think about what we consider this pining a cluster. Basically, with all of that power to schedule workloads, spin up new instances, manage the networking, what does pining mean here? We're talking about running your own workloads. This could be as simple as Bitcoin miners. This could be something that you're using as more of an implant to pivot to other containers in the cluster, stuff like that. We're modifying existing workloads. That could be if I'm running some application that's managing throughput from user data. If I can manage to modify the configuration of that pod in a way that makes it vulnerable to further exploitation, that's called pining. We're going to talk about hijacking traffic. One of the tools that we talk about a little bit is Istio, which is the most robust and interesting man in the middle proxy you've ever seen in your life. It's incredibly powerful. If you can take control of that, you can man in middle literally anything in the cluster. It's really powerful. Finally, the fourth one here is breaking out so that you're executing code on the host as root. At that point, you control everything and that's the pop calc up on the Windows box and prove you owned it. We're going to talk about three different attack angles today. Realistically, this is not a real cluster that we're going to throw all this through. These vulnerabilities probably could have existed at the same time, but it's not like we ran into a cluster that had all of these at the same time. Again, no zero days today. We're going to start with phase one here. No more secrets. I mentioned earlier Istio is a giant man in the middle proxy. Basically, the idea is that we're going to inject something into every time we launch a container. We're going to hook onto it to take all of this traffic through IP tables. We're going to use MTLS mutual TLS authentication so that when you as a developer writing a service, you don't have to figure out how to make MTLS work. You just run your pod in a system with Istio and you get the benefit of all that. That's kind of where Kubernetes becomes really powerful because we can take all of these lessons learned and hard cryptographic problems and boil it down so the developers don't have to think about it. It's kind of like just bolt on end to end encryption and it works really well most of the time. One of the other great things about it is this Keali tool, which is visualization. So now that you're intercepting all the traffic and watching it through the network, you can really figure out what you're doing and how your network is working. A quick shout out to JWT's pronounced jot JSON web token. Obviously, we're not talking that much about it, but I just want to bring it up as a concept of when you see this, it's a very common authentication protocol, how these little cookies are made. Jot.io is a really great website for if you ever get one of those and you have to interpret it. And the important part here is that it gets h-macked at the end. So as everyone knows, cryptography is perfect and very easy to implement. So we're going to talk a little bit about something we noticed. You guys ready to hear about a vulnerability? Yeah. So one day, Dagen's setting up Keali on one of these clusters and we're bringing it up, bringing it down. He's tweaking all the parameters in the Helm chart to figure out how to deploy it right. And after doing that the first time, it stands out the first time, you get that login page, hot damn, got it. And then the next time you don't get the login page and eventually he notices that he's taking these pods up and down and the login page doesn't refresh. He doesn't have to log in each time. So in Kubernetes, pods are generally ephemeral, which means that they're not storing data between sessions. And there's ways to store persistent data in Kubernetes. So we started digging into it like, okay, if it's keeping my session alive, it's storing that information, that cryptography to do the HMAC signing. So I dug in. And in Kubernetes, there's a lot of ways that you can persist data as what we're saying. In this particular case, what I would have expected to see is a Kubernetes secret where you install Keali for the first time, it looks for the secret, it doesn't exist, so it creates it. From then on it would see that same secret and use that as its signing key for the job. But there was no secret. I would delete the entire namespace, anything associated with it, and I was still logged in. The only piece of persistence was the cookie that was sitting in my browser, which meant there was two possibilities, equally terrifying. The first is there's no authentication of the cookie whatsoever. You just simply say, hey, I'm admin, and Keali's like, I'll take your word for it. The other is a hard-coded secret that's being used to sign the key. So I dug through the code and you can see the bottom-right corner. Keali, hello. Let's see how we're doing on time. I got a couple of minutes. Anyway, I went over to Jot.io, and I threw that in, and I was able to just recreate or fabricate a perfectly valid Jot that would allow me to access Keali as an admin. The reason that that happened is because they had an insecure default. It could be configured and overwritten in the Helm chart to specify, but nowhere did anybody take the time to say, when you're installing this application, you should definitely do this. So nowhere did it actually happen. So let's take a look. So here's the Keali login page. I'll go ahead and refresh it just to show I'm not logged in. You know, nothing up my sleeves. I'm going to come over to Jot.io and fabricate a Jot. The sample on the left isn't signed because there's no secret in here. I'll put in Keali as a secret. I now have a signed Jot. Just going to copy that onto the clipboard. Using developer tools on my browser, I'll add a cookie. It's just called Keali-token, and the contents of the cookie are just the Jot, which is totally valid, assuming it wasn't already signed by a completely hard-coded secret. Now when I refresh the page... Fortunately, from the attacker, Keali is extraordinarily powerful. There's a lot that I can do here. It's a powerful service mesh that allows me to completely rewrite how traffic gets managed. If I want to send 10% of your login traffic to my server instead, I can do that. If I want to add entirely new ingress points to things that shouldn't be secure or shouldn't be exposed, I can do that. I can pretty much do anything that I want to do. So, Dagen, we were sitting at the bar that day and you were telling me the story about this. One of the things you mentioned was that when you kept reloading the page the next day and the next day, it still stayed up. How did you make that work? Well, my talk on security ramifications of time travel was rejected, but I'll explain. So, here's the code that Keali uses to authenticate a cookie. Sorry, a Jot. The line 67 there, it's the second one from the top. It just says to the library, hey, can you parse this token and if it's valid, give me back the claims. What developers didn't understand is that particular library, it didn't care what time it was. It just said, yes, this is a validly signed Jot. What's not on there is any attempt to say, well, the expiration date is that in the past? It wasn't there. So, you could do a replay attack with this, no problem. So, if we imagine our fictitious cluster here, we're able to get to this Keali service, we're able to forge credentials, and with that, we can pretty much read anything in the cluster and we can write to Istio, which is the networking, you know, throbbing the nervous system of this cluster, and we can reroute that wherever we want. You know, the default for installing Keali is not read-only. So, at this point, we can manage the middle proxy, literally anything we want. We can blow up the system, we can shut it down, we can, you know, steal credentials. So, at this point, it's faced to say, we pretty much pwned that cluster. You know, not much left to do there. So, that's all right, thanks for coming to our DefCon talk. But, Dagan, we've promised them three demos. So, what if, what if I said earlier the default wasn't that it was read-only. Let's say it was read-only. What do we do with read-only now? Okay, that is admittedly a little bit of a challenge, but fortunately, this talk doesn't end there. So, let's go back and we'll say, we're going to make this harder on ourselves, and let's say that somebody responsible named Will Klein set up this instance of Keali, and so it is in view-only mode. What could we do? Enter my friend, Fleet. So, is anybody out there familiar with Fleet with Rancher? Okay, so you guys know? Two victims. Do you work for Rancher? Listen up. So, this was responsibly disclosed, and it has been resolved. But, Fleet is GitOps. So, for those not familiar with GitOps, it's essentially a really popular and great way, genuinely a good way of administering cloud infrastructure, where you put your configuration, you put whatever the manifest that you need to deploy into your cluster into a Git repository. And then you can have multiple clusters pointed at the same Git repository, and it's an easy way to build at scale. Also, it's Git, so it's actually very auditable, and you can go back and say, hey, Will, you deployed this thing. Why did you do that? There's other instances of GitOps tools for Kubernetes. Flux is a popular one. Argo CD is a little bit more mature. But anyway, that's Fleet. Well, the other thing that's great about GitOps tools like Fleet is you can basically, you know, everyone's had the situation if you're any sort of cloud admin, where you set something up, it's running good, and then, you know, somebody comes along and says, oops, needed to expand that disk a little bit, and oops, needed to redirect that and change labels here. And as time progresses, you just get farther and farther away from whatever you thought the truth of your cluster was. GitOps tools will constantly be syncing that and looking at it and saying like, nope, back to what Git says. Nope, back to what Git says. So you always can quickly understand what's running in your cluster and how it's being deployed, and that's just generally a really good thing. It makes life easy. Makes sense. Let's talk about logging. I've been professionally developing for about 15 years, and one of the things that I've come to learn is that developers either whisper or they shout when it comes to logging. Unfortunately, in this case, Fleet was one that was more of a shouter, and as I was digging through the logs one day, in fact, it's even a little bit worse than that. When you're running Fleet with multiple clusters, and that's not what we're doing in our demo, but just so you know how bad this was, if there's an error message, it gets displayed in the UI within a product called Rancher Dashboard, and so it's very, very helpful. And I was working with Fleet, and I tried to deploy something and it didn't work, so there was an error message that popped up that said, I tried to get this Git repo, and I wasn't able to. The problem was, Fleet uses a library called GoGetter from Hashicorp, and the way that GoGetter works is you provide a URL that tells it everything it needs to know to go and get whatever file you're asking for. It can work with Git repositories. It's a very valuable tool. But in order to get the Git repository, you need to give credentials. And so those credentials need to be embedded in the URL. And when the error comes back and says, I couldn't get this entire URL, and the credentials are in it, the error message that gets logged includes the credentials to the Git repository, which we just said you can use to deploy anything you want to all of your clusters. So what could we do with something like that? So as previously mentioned, Keali is super powerful. One of the things that lets you do is look at logs of individual workloads. So I'm going to take a look at the Fleet agent logs. And oh my goodness, is that a Base64 encoded SSH private key? Fortunately, it's encoded, so nobody knows how to get access to this now. It's perfectly safe. This is my actual GitHub private key, by the way. Of course, you're running low on entropy, so you only have one. Let's see, Base64 to code. Oh my gosh, it's an actual private key. So what else could I do with this? Well, one of the things that it did that the log message told me also is the actual URL to the Git repo that it was trying to read. Could I perhaps clone that and see what's there? Yep, I can. And it turns out it's a Helm chart, which I can do some interesting things with. It turns out the reason that there was an error is if you look at the branches listed, it's only main. Main is the only branch that exists, but this is looking for the develop branch. So what would happen if I created a develop branch, put my payload into it, and pushed it back up to the Git repo? Well, let's find out. I'll just create a job, and I'm going to put my malicious payload into that job. This one's set to just pop a reverse shell. Apparently I have right permissions, and now it's a waiting game. GitOps does its thing, and I have my reverse shell. So to keep this sort of going, we're going to add some more contrivances. Again, Fleet has global permissions. I could install anything that I wanted at this point. I truly own this cluster. But what if, again, somebody named will client set this up and actually constrained the default cluster role that Fleet runs under, so it can only do some things. It can deploy a pod, which would be pretty smart. Yeah, so basically, we're looking at this cluster now. If we think about it, we've gone in, and we've taken a service that's generally like kind of an available thing. We've gotten past that. We kind of locked it down a little bit retroactively there, but we can see the logs now of all the pods. We're looking at Fleet. We managed to get access to the GitOps. Now, one of the things you're thinking in your head is, like, who in the world would possibly just use their actual SSH credentials while setting up their GitOps? Well, GitOps happens pretty often, especially with GitLab tokens, where the developer will go in and generate a personal token with read-write access instead of going in and creating a deploy token with only read access. If we only had read access, this would be a lot harder to do, but there's a lot that you can do with that information. Once we have write access to the GitOps repo, now we can push and we can do whatever we want just like we were the cluster admin. It's really convenient for cluster admins, and it's also really convenient for anybody else so really, at this point, it's kind of pwned because we could literally run anything we want because the GitOps tool is just bobsledding whatever we put in to get into your cluster, and that's how it works, and it's working as planned. So what went wrong is the developers at Rancher were using a library, and they used it properly, but they didn't fully understand what contents would be in that error message, so they got an error and they logged the error, which is normally a reasonable thing to do, but sometimes that can turn into an information disclosure. It's kind of understandable to be fair. When I traced it, the error was passed up through nine different call stacks, so it was just like, here's an error, here's an error, here's an error, and when you get to the very top one, it was like, I don't know, log it. That way we know what went wrong. All right, and so we're going to talk about phase three here, so let's pretend that we had, in this fictitious cluster, we have a read-only Keali, we have a super locked-down RBAC around our GitOps tool, but let's look around and see what we can find there. So, in this cluster we're working on together, we were using this tool called Longhorn. Now, when I said earlier that Kubernetes is this whole API, really the API describes how to do all the things you want to do in your cluster, like, I want to launch a service, or I want to have an endpoint, right? But there's actually the API-specific implementations, vary cluster to cluster. So, there's a thing called the container storage interface. It represents how is persistent data going to be stored on the cluster. You know, normally the pods are completely ephemeral, so you lose it, you lose it. Longhorn is a microservice-based storage system that runs on Kubernetes, so at this point you have Kubernetes storage feeding Kubernetes storage, so it's a pretty neat project in that sense. It's made by Rancher, and you know, there's basically, there's a couple different pods running, but there are two ones that we want to talk about today. One of them is an instance manager, which means that, which is running on every single node as a daemon set, and the daemon set says, you know, when a new auto-scaled node pops up, I want to launch my pod on that one and do some things. So, it's running everywhere in the cluster, this daemon set, and then there's also the Longhorn manager, which kind of wrangles all of those instance managers running all the different instances. So, these have a communication between them, and they manage the provisioning of storage and all the different actions. So, as part of our, you know, hardening, getting things ready, you know, I pull up Longhorn, and I'm looking at some of the pod security policies, PSPs, those are being deprecated, but that's a talk for KubeCon. And I'm looking at the default ones, and I'm just like, wow, they're basically allowing everything, there's a requirement, there is a PSP, but you've done everything you can to disable anything useful in it. So, it's like having the firewall and everything's left open to all the ports, right? What's the point of having a firewall? You're just checking a box that you have one at that point. So, we started going in and trying to figure out, like okay, what can I narrow this down for, and I'm going in and out, and we're launching it and tearing it down, and just, it's taking forever because every time I try to take the least amount of privileges away from it, it crashes and won't run. And so, we're piecing this apart and it's like, well, really, there is no reasonable PSP around this. It's doing all sorts of nasty things. It's bringing in directories from the root on the host into the container on the inside, and then it's also running itself as root, which, if you're in a container and you're mounting all the important directories inside your container and running as root, you're not really a container anymore. You're just a way to run code at that point. It nullifies any of the security advantages. So, we're looking at this thinking, okay, well, at least we can try to make the network layer secure because they didn't support any network policy or MTLS. So, I started digging into that now and just more of this weird behavior where the network policies are really difficult to write because every time it does something, it opens up a new port. And fun fact that a lot of people don't know that when you're declaring your pod spec, you specify which ports you have, that's more of just road signs and information. Your pod can listen on anything. That is absolutely not a firewall. That's not a firewall rule? So, we're looking at this for people where they literally thought that those were some sort of firewall that was blocking traffic. It's really just a convenient shorthand to help you locate the port. So, we can't lock it down. We can't, you know, the inside the container is a mess. The network king is a mess. And at some point, we start talking about, like, what is this pod doing and why is it literally root on every single host? So, we started digging into it a little bit and thinking about it. And having had our conversation about the Kiali thing before, we're like, okay, let's pick this apart and see how it's authenticating because we started noticing that we weren't seeing secrets in the cluster, which, you know, that was the same hint that we had for Kiali that Kiali was up to no good. So, we're digging around and we're not seeing anything that's storing the persistent data to, you know, how would these pods authenticate each other? And, like, if it worked with MTLS, we could kind of pretend that it was doing a good job because Istio can kind of fill that gap, but we couldn't get it to work with Istio because of this weird network behavior. So, we're digging around, and sure enough, we found that they actually weren't authenticating each other. It's just a GRPC that's listening on a standard port that's not declared and away it goes. Okay. So, GRPC, for those who may not be aware, is a remote procedure protocol from Google. It runs on top of HTTP2. It uses protocol buffers to encode data. I love it. It's a very cool protocol. You should absolutely be using GRPC, like, truthfully. One of the nice things about it is you don't have to write the server and client bits yourself. You write a protocol or a protocol that defines this is what my API looks like. If you're familiar with Swagger, it's a little bit like that. But then in addition, GRPC has plugins for all sorts of languages. So, you can run that plugin in your compiler and point it at your protocol and you actually get the server or the client. With the server, it expects you to have something that implements an actual interface that does the thing, but it gives you all the connective bits in between and it's really nice. It's particularly nice if you're an endpoint developer and you want to support people that are writing in different languages because you could write and go, somebody else could write in Java. I don't know why they'd write in Java, but they could write in Java, and then they could just run the plugin and they get a Java client for your API. It's also really nice because it's easy to read. So, I was really excited when I looked at Longhorn's repo and just saw these protocols in there. I was like, oh, I can read that. That's simple. So, I started pulling it apart and I saw this service that was called Create Process. And I thought, well, I hope that the Longhorn developers have a different meaning of process than what I have. So, let me look at the parameters that get passed to this service. And there was fun little bits in there, like binary and args. And I thought, surely this is not an API that allows anybody inside the cluster because remember, cluster's networking perimeter is the cluster. It's not at the namespace. So, no ability to add a network policy, no ability to use Istio to do MTLS enforcements, no authentication. This can't be the case. They must be doing authentication somehow. Now, Longhorn Manager does generate the pods itself and push them out there so it could be generating something as an environment variable, which wouldn't be the most secure thing, but it'd be okay-ish. No. No. It wasn't authenticated at all. It took me... I work on the West Coast and Will works on the East Coast, so I have to wake up at 5 a.m. and start work. So, it was about 5.05 a.m. and I was looking at this and I'm awake now and I reached out to Will and we said, well, let's do a POC. Like, this can't be a thing. So, we started looking at it a little bit and I said, well, what can we do with this access? I mean, it's Sandbox. It's a container. Well, we'll go back to what Will was talking about, about the file systems from the host that were being mounted. So, there was two. Proc and Dev. Not the best two. And Will said, you know, in Proc, PID 1 root, that will always be the host's root file system. So, I wonder if we could just say something that's just going to touch something in host Proc 1 root. Let's get fun, Etsy, whatever, right? And so, because this was written in with GRPC, it took me about 25 minutes to slap together a real quick POC and go and run this thing and we're, you know, Will's sitting there, look at his SSH into the node and he's like, okay, you know, I just did an LS on Etsy, go, I go, I hit go, he refreshes, and there's a file that says, this should not exist. Owned by root. So, we took that a little bit further. We developed an exploit for it. It's called Rustler. It is available on GitHub. It basically allows you to point Rustler at a target that's running Longhorn and just say, do this for me, please, and it does every time. It's very reliable. Now, to Longhorn's credit, again, all the developers we worked with on this have been really amazing. We reached out to them. They pretty much immediately understood the problem and they worked to fix it. But before they did that, let's take a look at what we were able to do. Okay, so I'm back on my reverse proxy from the previous demo. I need to find Longhorn. I need to find something. So, I'm going to temporarily broadcast with NetMask. Ignore that. And I'm going to run a quick end map across the subnet that I'm working on, looking for anything listening on 8500, which is what instance manager listens on. It's not a big subnet, so it doesn't take long. There's two. I'll grab one. Remember, it's running Ubuntu as the base image. So, I have access to all of Ubuntu 20.04 that I can run commands against. So, I'm going to start up listening again, and I'm just going to do a really basic one-liner bash reverse shell. Who am I? I'm Root. So, going into the proc directory, if I change Root to host proc one root, now I'm Root on the host. And for the cube administrators out there, I can run cry control. So, basically at this point, I can do whatever I want with your cluster. If this isn't enough fun for you, Longhorn has essentially two different modes that it's running in. There's everything, and then there's Longhorn Manager, which Will was talking about before, which is creating a lot of pods. It's doing a lot of things. So, it has very permissive RBAC rules. Like, it's allowed to do a lot of things. For some reason, it's allowed to do those in any namespace it wants. It's not isolated to the Longhorn system namespace. I don't know why that's the case. It doesn't seem to be necessary at all, but it is the case. Also, at least previously, even though there was two very different security modes of I need to do a lot of things and I basically need to do nothing, there's one service account. And that service account is shared across all of the pods. So, at this point, running an instance manager, I am able to run inside that pod, cube control and create pods. I can do fun things like read any secret that's in the cluster anywhere, create, read, update, delete on almost every type of resource. Even if I didn't have that, Fleet still got my back because Fleet's permissions are API versions, all of them. Resources, all of them. Verbs, all of them. So, anything that I want, I can do. Yeah. So, basically, once you have that little foothold in the cluster that we've gotten, and really, this is just a one way to do it, but a lot of people don't think about, they think about the containers as a security mechanism, and they assume that once your application is in a container, that's going to hold back an attacker, but realistically, because namespaces are not a security boundary and there's really nothing that's a firewall in this, Kubernetes makes service discovery really easy, which is why we were able to find Longhorn pods so quickly and there's nothing holding us back from talking to them. So, it's one of those things that when you're deploying a cluster, you need to be thinking about where are the actual security barriers and where are their kind of like shadows or of like insert security barrier here, right? Like you can write. Somebody's got this for me, right? Yeah, someday someone's going to insert into the security policy that blocks this, but building out your cluster, you really need to think about that. Basically, once we're on the host, there's very little that we can't do on that host as far as getting back to the master nodes or the control plane stuff, and we don't really need to do anything on the control plane because we can launch literally any pod we want, including more incredibly vulnerable pods that mounts very dangerous directories into them. So, there's really nothing we can't do in the cluster at that point. And if there are good pod security policies in place that prevent us from doing that, it's okay. We can deploy our own pod security policies that correct that for us. All right, so some big takeaways. One thing that we do really want to emphasize is Kubernetes itself is pretty damn great. The Kubernetes developers are absolutely amazing. There's a ton of them, and they're some of the most brilliant developers that I've ever had the pleasure of working with. But not everything in the Kubernetes ecosystem is written by those Kubernetes developers. And a lot of times, to be really fair, these projects, they start off as one developer's idea, and that turns into a proof of concept, and then someone goes, cool, ship it, it's mission critical, let's go. And we don't necessarily rewind and sort of rethink, is it secure? Is it what we need it to be? I mean, you have some bullet points. One of my things that I like to drive home with people is good code does not equal good deployments. One of the advantages of Kubernetes being the ultimate shift left capability is that developers get to work on deployments, but there's also people that have been working in system administration and building all of this cluster stuff out. A lot of lessons learned about how to have least privilege and the developers aren't necessarily thinking about that. So a lot of what times what happens is you'll see a perfectly good project was actually implemented pretty well on the coding part. Then when you go to deploy it with the Helm chart, like Bitnami has a lot of this, none of the defaults are anywhere near secure, and it's difficult to secure it because of the way things are written. So just because you have a really secure project like your MySQL, your Postgres, it's up to date, doesn't mean that the deployment of that system can't shoot you in the foot. So not that there's any attackers in the audience today, but if there were, what do you do? You got your hands on a Kubernetes cluster. Maybe you found an insecure application that lets you get just a toe in. How do you pivot? Understand some of the common mistakes that people make. What are the assumptions that are just, frankly, inaccurate in a Kubernetes environment? So again, going back to service discovery, there is unfortunately commonly a presumption that name spaces are network segmentation. They're not. So look for people that just have completely insecure APIs and don't be surprised if you find it. Secrets should exist somewhere. Somehow information needs to be shared across these different applications that need to be able to communicate with each other. If that doesn't exist, ask yourself, how is this being secured? Because it might not be. Additionally, GitOps is super fun to play with. If you can get access to a GitOps repo, you own not one cluster, but lots and lots of clusters in all likelihood. So watch for that. And then finally, for me at least, don't forget to check the logs. I go back to it. Some developers whispers, other developers definitely are shouting. I think one of the things that I want to wrap up on here is that Kubernetes is pretty damn great and it has all this great stuff in there. But a lot of the bad behaviors that people do on their local development machines just has a tendency to get embedded into the deployment systems and then end up through GitOps into the cluster. So in no world does GitOps, I used to joke a lot with people that like, SecDevOps is really just that we took the developers and fired all the ops team. There's a lot of lessons learned from traditional ops that can be applied to, through DevOps type stuff, to the deployment of applications on Kubernetes. And we can't let the devs just take over the whole process because we forgot to hire an ops team. Okay, so before we go to questions, again, I mentioned that the full demo environment is automated and available. The slides are available too, by the way. Any of the main tracks, if you go to media.devcon.org, you can get all these materials, the recording, all that's there. But feel free to go there to github.com slash dagan slash devcon 30. I promise to really actually try to support folks for at least the next couple of weeks. So if you have problems working through any of this, open an issue. It's pretty easy for folks to do. Also, today we talked a lot about what can go wrong in Kubernetes. And if you're interested in knowing how to protect your clusters from that, we're going to be doing a similar talk, but in, like, the defense side at QCon, CloudNativeCon in Detroit in October. So feel free to meet up with us there. With that being said, we have, I think, just under 10 minutes or so left. So if anybody has any questions, there's a goon back there with the microphone. We'll do our best to answer them for you. Thank you. Anybody with questions? I guess we covered it all. All right, well, we'll be hanging around for a little bit and then headed out that door. So if anybody wants to ask us anything directly, again, thank you all for your time. Thanks for coming out.