 Okay, new day, new project, time to write some code. I think I'm gonna connect my local tooling to my development cluster, and then I can get started. I'm gonna do kubectl apply my usual tooling. Let's see what, oh, wait, what? Node part, part something, let me do this by hand. Fail to create node parts, the parts are already allocated. Okay, so someone's playing with my cluster parts. I told everyone 31337, that's me, don't mess around with it. Gosh darn it, okay, who did this? Let's see who's around. Okay, there's a lot of people here. What are they doing? I don't know, I can't look at it. Can I look at them one by one? No, I can't. Okay, so maybe there's some security in place here, and I can't see what everyone else is doing. Okay, you know what, I'm not calling the security people, they are not fun, I'll do this on my own. Let me check something out here. I remember seeing something funny in the onboarding documents. So I was looking around the docs, and I found this dev database controller. So the controller creates new database instances, pre-populated with fake data for using dev environments. It hangs often. You can exact into the pod, and if it's really wedged, you can restart the process by deleting and recreating the pod. And someone said something here about fixing RBAC privileges. Okay, so we're using RBAC, and RBAC is that thing where I have some permissions, there's some things I can do in the cluster, and there's some things I can't. Apparently I can't look at what my friends are doing to figure out who's messing with my stuff, but if they set RBAC incorrectly, which is very easy to do, because it's convenient to set it incorrectly, then maybe I can use that cascading effect where I am allowed to do some things, I'm not allowed to do others, but through the things I am allowed to do, I can go around and manage to do the things I'm not allowed to do just in a very roundabout way. So let's try that. So we're looking at a Dev database controller. Let's see if we can find that. What can I do normally? Let's see. Cube, Cuddle, Off, Can I? So I can do whatever I want in my own namespace, but when I try to look at someone else's, so let's see what Catherine's doing here. When I try to look at someone else's namespace, the stuff that I can do is very little, almost nothing. Let's check something out. So let's look at what's going on again, and I have this Dev environment controller. I should be able to create pods in it. Yeah, the documentation was right. So for pods, I get a star, which means I can do whatever I want with pods, and I can EXAC into pods as well. So let's try and make use of that and see what happens. Let's see, let me do Cube, Cuddle, Get Pods, and here's one. So let's try and execute something in here. So Cube, Cuddle, Execute on that namespace in this pod, and give me a shell. Okay, I got a shell inside the controller, and that means that I can probably now make use of whatever permissions the controller has. So do I have tooling here? Yes, I do. Thank you. When you leave tooling around, attackers become very happy, and speaking of attackers. So what can I do? Let's see what I can do from within this controller. Let's say again, what can I do? Would you look at that? Stars, stars, more stars. Like a weird philosophy everyone said, we're made of stars, but your RBAC shouldn't be. And oh boy, they were right. So I can do whatever I want here, which means I can probably see what my friends are doing. Yeah, I can see what everyone's doing now. So let me filter through this. A lot of people are doing a lot of things. Let's see who's using my gosh darn part. Ingress pointing here, and that's Joey. Oh, Joey, I told you to stay out of my turf, Joey. Yeah, you're getting grounded. That's enough, Joey. So Cube, Cuddle, Edit on Joey's. And here we're going to replace Joey using my port, with Joey using 420, because I know the kind of stuff Joey likes to do. So get out of here, Joey. There you go. And now I can probably get out of here and run my stuff finally. And there you go, now everything works. And now I tilt up, and my application's working. So now I can go write some code and be productive, and Joey's going to have a bad surprise on Monday, and that's his own thought. Well, here's what we might want to do. First, we want to become a cluster admin. This could happen by either compromising the cluster and becoming a cluster admin, or maybe you are a malicious admin. Hello, this is Tabitha DevSecOps enforcement. Joey, Joey, Joey, calm down. Calm down, I'm going to help you, but you've got to explain it to me. You've got to tell me calmly what's going on. Okay, so you can't get into your Dev environment. Well, let me have a look. Okay, so I'm going to look at your service first, because that's always been a problem. Oh, Joey, of course it's not going to work for you on port 31337, it says here you changed it to 3420. Joey, is this some kind of joke? Oh, you didn't change it? Oh, let me check on the Devend controller. Maybe something bad happened to it. Devend controller looks okay. Joey, I'm going to put you on hold for a moment while I do some more investigation of this. I'll be right back. If this was changed, and Joey says he didn't change it, I'm going to have to go and look in our log management system to see if the API server audit logs can tell me what's going on. Okay, so I'm going to have to first look for our Kubernetes audit logs, and let's do in the namespace Joseph Pardella. Okay, there's a lot of logs here, so let's look at what has been done inside his namespace. Okay, patch, that's going to be the ones where things were edited. Let's have a look at these. Here's one that says Devend controller recently modified Joey's Ingress. Okay, that's interesting. Why would the Devend controller have done that inappropriately? Let's see whether anything strange has gone on in the Devend controller. Okay, so here's some typical kinds of changes. Let's see who's doing them by looking at the username. A lot of service accounts, and then Ellen. What has Ellen done in the Devend controller recently? Okay, so it looks like for Dev maintenance, Ellen was allowed to execute a shell in the Devend controller. And then right afterward, the Devend controller service account modified Joey's service. Yeah, that's suspicious. Girl, I think you just got yourself in trouble. Okay, better get back to Joey and make this right for him. Hey, Joseph, thank you so much for holding. I've done some investigation and I see a simple misconfiguration. I am gonna fix that for you and then you should be well on your way. Yeah, hey, what's up? That's exactly what I was expecting. What up, yeah. Hello, Ellen. It's David, you'll recall I'm director of DevSecOps Enforcement. I have some questions I need to ask you as part of an ongoing security investigation. Do you have a moment for that? Yeah, I mean, I'm in the middle of a game, but yeah, sure, what's up? Great, thank you so much for taking the time to meet with me here. What can you tell me about Kubernetes API server audit logs? Really, you need to ask me that? Yeah, yeah, tell me about it, Ellen. Sure, yeah, so all that logs is like when you do stuff to the API server, it logs that stuff. Yeah, all of the activities taken on the API server. So like, do you know how one might enable API server audit logs? I think it's on by default on the managed stuff, but you see out there, if you're rolling around, you need to enable it by yourself, but I don't think we are. I mean, once it's enabled, like, should you just leave the logs sitting around on the server? Lots of friends of mine would say so, but I would advise you to put them somewhere. Yeah, like where could we send them? If you're doing elastic stuff, log stash, I think here we use Datadog, so you don't keep them in the cluster because that would be hard. Shift them in the Datadog, yeah, that's right. So like, what do you think I can see if I go into Datadog and I view our API server audit logs? Sure, yeah, you can see the when resources are created or modified or deleted. That activity? Yeah, like what kind of resources are we talking about here? Oh, you know, stuff like pods, deployments, jobs. Node ports? Yeah, yeah, yeah, node ports. So like, do you have any idea where I may have gotten these logs that show you modifying Joey Bardella's dev environment to point at port 3420? Do you think you'd like to take a moment to explain yourself? You gotta get your act together, Ellen. You gotta cut out this childish movie hacker like garbage behavior. You're supposed to be a professional and this ain't it. You gotta get it together. You know, if security people would listen to developers, maybe I would be writing code the past hour instead of browsing Twitter. Ah, what is this? Look it up, baby. Yeah, that was a fun one. I wonder what is this exploit that they're talking about here? CVE 11253, even rhymes, that's cute. So apparently the CVE is improper input validation on some versions of Kubernetes and causes the API server to consume excessive CPU or memory, possibly crashing. Okay, so this is a DOS. And you know, if the development cluster would be out of commission for a day, I would get the rest of the day off. Are we vulnerable to this? Cubicuttle version, we are 16.1. And what we have here is 16.2, yeah. You should be vulnerable. Where's the POC? All right, this is cool. Apparently we're dealing with a Kubernetes vulnerability here and they don't come around very often but when they do, they can ruin your day. And if we're vulnerable to this, it means the security people in our team haven't been doing their job because they should be signed up to the mailing list, the security, getting the security announcements. They should have patching and upgrading and all that stuff as part of our software lifecycle management. And if they're not doing that, then I suppose they're gonna need to learn their lesson. Yeah, they are. Okay, and we have this exploit set up. Well, I think it's time to sending a billion hunts using a flock of 400 flying geese. And is this working? Our server does not reply any more. Time to enjoy my day off. DevSecOps enforcement, this is temp. Can you please hold? DevSecOps enforcement, this is temp. Can you please hold? Oh my gosh, what is happening? Thank you so much for that great demo, Ms. Libby. Having a principal engineer like you on the staff is a huge inspiration to the entire rest of the engineering division. Now, for the last part of our monthly all-hands, I'm going to give an update about security initiatives. Let's see, how do I share my? Can everybody see my screen? Unfortunately. So, all right. Today, I need to talk to you because of some activities of a questionable nature that have been happening in our environment. We're going to need to enact certain new security policies company-wide. Ms. Seville, why just now are you saying, how long have we been without security in the company? These are meant to protect you from each other and in some cases, to protect the company from your childishness. And what were you doing the past couple years here? We're going to talk about three things today. Admission control, network policy, and container vulnerability scanning in our CI pipeline. We'll start with admission control. So admission control is what you need to be able to enforce policy about what settings are allowed to be put on a pod, such as privileged or host namespaces. So we're going to be deploying new admission control policies that are in line with the Kubernetes project pod security standards document. The one that's about to be deprecated, right? Thank you, Ellen. No, the pod security standards document describes best practices for running pods in Kubernetes. I think you're thinking of pod security policy, which is a feature that was deprecated in 1.21, but will be replaced soon with a new admission controller. And that's irrelevant to us anyway, because we're going to be using gatekeeper. But thank you for your concern. So getting back to the presentation, the importance of admission control is because our back works at the level of Kubernetes objects, like can you read or write this pod or service? But it doesn't have any opinion at all about whether a pod is a harmless web server or a privileged command shell that exposes all of the node's data. So to enable that kind of control, we need to have admission controller. And admission control then gives the API server opinions about what should be allowed and what shouldn't. Shouldn't you be doing that? Yes, and I'm enforcing my opinions using an admission controller. Thank you, Ellen. So this will help us to not develop using dangerous pod features that could help an attacker if they got shipped to prod. And it will help to isolate our dev environments from each other. The next thing that I'd like to share about is network policy. As those of you who are involved in our attempted proof of concept years ago, no, microsegmentation has been a dream of mine for years, but it was way too hard in our bare metal data center. So you know you love to dream? Yes. So in our Kubernetes environment, we already need to precisely define which services our services talk to for use by our service discovery system. So we're going to be adding. Can I get you to stop talking? I wish you'd stop talking to all of us, Ellen. As I was saying, our service discovery system will be integrated with the network policy engine. So most of the work of writing policies will be taken care of for you. And after you've added the last few touches, then Kubernetes will enforce what can talk to what within the cluster in order to reduce attack surface. The last thing that I'd like to share is that we're going to be rolling out container image vulnerability scanning as part of our CI pipeline. Now that we're Web 2.0 scale, our developers have been struggling to keep up with the over 12 containers that are deployed in production. So with this scanning, the scanner will run every time a new container is built by CI and it will send an email to the developer about any findings there. We're not going to be blocking deploys right now because of these results. This is only for your information. But if we find that these results are getting worse over time, there may need to be further follow up. So thank you all. I want to leave with one last bit of information or perhaps a warning for some of you. We will now carefully audit all logs, especially the Kubernetes API server audit logs in order to detect any malicious behavior or shenanigans that might be occurring. So thank you all so much for your time this afternoon. We will now carefully audit all logs. Yeah, over my dead body you are. I don't like working in a Panopticon and this is way too much of a dystopia. No one's auditing anything. I'm going to get in there and see what they're trying to do and if they're trying to make me uncomfortable, I'll just take some drastic measures. Now to do that, I'm going to need root access to the development environment server. And they talked about admission control but they didn't mention that it was deployed yet and if it's not deployed, it's not so hard to run a pod that gives me a root shell. So I'm going to try my favorite incantation here and let's see if it works. Let's go back to Duffy. This is such a classic and I love this forever. Is this going to work? No, it doesn't. Okay, what else can I do? What if I can mount a host path so that I can access stuff that's on the node from a pod? Would that work? Let's see, I have the script here. Let's try and run it and see if it gets true. So I'm going to apply and denied admission. Okay, so can I do something else? What if I can mount a pod onto the host network? Is that going to run? And to test this, all I need to do is run any old pod and it can be anything. It doesn't matter what it does. So long as I have this option here and yeah, it went through. This pod's probably going to crash but the point is the admission controller didn't catch it and if I have access to the host network, I can interact with the host and so far I don't know what that's going to get me but let's do some research. Aha, I found something. CV 2020 15257, container D 1.4, abstract UNIX domain socket. This will do just fine and there's even a nice write up here about ways to use this. So this is perfect. I'm going to set up my attack and then we're go. Okay, we're all set. This is going to be a two-stage attack. First, I need to set up a reverse shell so that when my exploit calls home, I can pick up the call and second, this is what I need to run. What this is going to do is I'm going to interact with the host network directly and because of this container D vulnerability, I don't have to talk to Kubernetes. I can tell container D itself to run whatever container I want. So this means I can run that host path container that Kubernetes wouldn't like me because even if Kubernetes security is set up 100% correctly, if I can exploit the vulnerability and something that Kubernetes is running on top of, I can bypass all of Kubernetes completely and then it doesn't matter. I'm in. So let's see if this works and it's running. Is this working? Yes, it phoned home. Now final stage. Did this work? Yes, I'm in. Okay, now let's see what those security folk are up to. Oh, right. I'm not using Docker, I'm using container D. This here is CTL PS. Cube API server, let's see what they're logging here. I'm gonna do CRICTL logs here and let's see what they got on me. Oh boy, they got everything. Look, the task host not stuff that I was running, it's all here. This will not stand, but wait a second. I think I didn't just see something strange. What is this? This isn't familiar and I don't like unfamiliar things. What is this pod? It's running on cube system. I don't remember seeing this on cube system before. Let's check out, let's look at the container. And oh boy, this, I don't know what this is, but it looks highly unusual. And what this is running is, wait, Natcat, no, no, no, no, no, no. I'm the one running Natcat, I'm the hacker here. Why would anyone else be running Natcat unless fascinating, a hacker. So what should I do? I mean, if there's another hacker here and they cause real trouble, I got a record. People are gonna try to pin it on me. So I think the only way forward for me is, I need to play Goody T-Shoes and at least get my re-rend out of the fire. Yeah, we need to call Tabby. You keep bothering me, what do you want now, Ellen? Help Tabitha, when I was a child, he speaks as a child, understood as a child, thought as a child when I became an adult. I put away childish things. Cute, but really, why are you interrupting me? We've been playing games where we're supposed to be on the same side, okay? Yeah, and? Are you familiar with an admission of controller on the cube system namespace, executing Natcat? I mean, no? Yeah. Why would that, why would that be that we should really start the security incident response procedure? Let me create the incident record and page, page of the forensics team for us. We can start filling out the forms together. I can take your initial report right now while we're on the call, what? Nothing, I'm impressed, okay. Okay, so here's the first question. All right, that's the last of them. Okay. Thank you so much for bringing this to our attention, Ellen. Sure. Maybe I was wrong about you.