 Thanks so much. All right, I'm gonna go ahead and get started. Thanks, Marisa. All right, so this is five of the most common Kubernetes security misconfigs and how to fix them. So I'm gonna start with a quick overview. Kubernetes has obviously taken over. Everyone knows this. The most recent survey that came out from the CNCF showed that 91% of respondents reported using Kubernetes in production and 83% of those were in production. So 91% using it in the 83% in production. And this is a steady increase from last year where 78% and the prior year of 58%. So usage is skyrocketing. The problem is that Kubernetes and open source is not secure necessarily by default. So there's a lot of security capabilities built in a lot of encryption turned on by default, a lot of things like secure configurations, but some of those configurations are not secure by default. And so we did a scan at Bridge Crew of the Helm charts in Artifact Hub and found that close to 50% of those were had insecure defaults. Additionally, unit 42 Palo Alto Networks Research Unit did a scan across the internet and found 2,100 unsecured Kubernetes clusters with 5,300 nodes, 31,000 CPUs and 75,000 pods. So these are ripe targets for bad actors to use. And some of the most exposed resources were secrets, containers, images and deployments. So let's investigate some of these. So what I did is I basically looked at our across our checkup scanning base, all the misconfigurates that have been reported back to Bridge Crew and aggregated those and found the most common low, medium, high and critical misconfigurations that were identified, plus an additional one. Now, I did a write up on this that showed up in VM blog this morning. And so you can follow along. The only difference is I swapped out the last one. So if you look at the VM blog one, I do one for EKS to kind of share the love between vanilla Kubernetes and EKS. And in this case, I talk about another really common one, which is the second most common low misconfiguration that's Tiller. All right, so basically let me show you what it looks like. So here's a very simple Kubernetes manifest. I'm provisioning a pod that's a test pod with busy box that just says that the box is running. Now, if I open up terminal, let's just see what it looks like if I do a check off scan. And check off is a free and open source tool. Bridge Crew supports it, but you can use it. It's on GitHub and GitLab. So you can check that out. So a quick scan of this file, which is a very simple manifest, shows there's already a whole bunch of misconfigurations like no CPU limits set, the image isn't using a digest. So busy box would just grab the most recent image, but that can be abused. And there's a whole bunch more misconfigurations here. Now there are a bunch that have passed by default, which is great. Kubernetes has a lot of security by default baked in, but you can see 69 pass checks and 19 failed checks. So we aggregated those scans across our customer base and came up with this list. So let's dig into it. All right, so the first one is the default namespace should not be used. Now this one is a little tricky because it's possible that some customers are going in and adding the namespace at deployment. Now we recommend, because we follow GitOps, we recommend actually doing this declaratively in the actual pod manifest, but it's possible that customers are adding it on with the dash dash name, name space and then the namespace they're doing. Now in Kubernetes, namespaces are the most common way to segregate different pods from talking to each other. So if you have multiple different teams and multiple different applications sharing a cluster, then you might use namespace to segregate those. So it's really good practice to use those namespaces and not use the default namespace. Now I know I'm guilty of this, especially if I'm tinkering and I'm the only one using a cluster like on my Raspberry Pis, I still use the default namespace, but it's good practice to actually add in the namespace. So you can add separation and that way you don't forget because we all know the power of defaults. If you forget and then you add a second application to the same default namespace, all of a sudden those two applications can start talking to each other or steal resources from each other, from their C groups. And so that exposure really hurts your security posture. All right, so the solution here, there's two different ways. There's one, the imperative way, which is you create the namespace using kubectl, then when you apply a pod, you actually add the namespace. The declarative way, which we recommend because then you can shove it through your CI, CD pipelines and follow GitOps is to create the namespace using a manifest and then in the pod secrets, config map, et cetera, add in the metadata, the namespace that you're using. So these are two different ways that you can do it. I highly recommend doing the right side. All right, so that's the low misconfiguration. Let's jump up, turn it up a notch and go to the medium. Now a quick caveat on this one, allowed privilege escalation is part of the pod security policies or PSP, which is deprecated in version 1.21 and set to be completely removed in 1.25. So this might not become an issue in the future, but there are other ways to create privilege, which you should be aware of and make sure you're using very selectively and I'll talk about that in a second. Now, normally containers are isolated with namespaces for process isolation and C groups for resource isolation plus SE Linux for some additional security wrapped around it, allow privilege escalation allows container to run in privilege mode. And so that means that the container has access to some of the host system and can basically add, act like root, so this is like pseudo, there are access to pseudo on the host. And that means that if a bad actor gets access to that container, so let's say container one, they can break out of that and start using some of the node or host capabilities to do bad things like crypto jacking. They can access some of the other pods on there, the network traffic going through. So it's a good idea to not turn on allow privilege escalation. In fact, if you look at Microsoft's Kubernetes attack matrix, the privilege container is definitely on the top and you'll notice the few others from the list that I shared before. We have access to the KubeLit API. I talked about how some misconfigurations are helping with that. And then tiller, which is obviously a problem for version two, one that almost made the list, this was the third most common low misconfiguration was exposing the Kubernetes dashboard, which in production you shouldn't be doing. That's more of a dev test environment thing. All right, so the solution in this case is to make sure you say allow privilege escalation is set to false. And so the best way to do this, even if the default is false, it's good idea to actually declare allow privilege escalation as false to really make sure that you're being secure and following those best practices. So under the specs, under the security context, add that allow privilege escalation to false. All right, so. Hey, Taylor, there looks to be a question and I was wondering if you are willing to take a stab at it. Sure. Okay, so how does defining a namespace other than default prevent some malicious pod taking with another pod in a different namespace? All right, so namespace isolation really works. This is going back to that low misconfiguration. What it does is it creates a logical separation between the different namespaces. And so unless you are actually creating an access line to between the different namespaces, they should not be able to talk to each other. And that's a built-in control that Kubernetes has. So by separating names applications by namespace, then you can also control things like RBAC. So you can only grant access to your application one team, to application one's namespace. And that way you can't get access to other things because it's all logically separated. And then can we configure the cluster to prevent allow privilege escalation? So the answer is yes. There's a couple of ways you can do this. So in OPA, the gatekeeper, you can do this. You can watch for privilege escalated pods that are being created. And there are obviously proprietary tools out there that can do the same thing and basically prevent any container that has privilege escalation from being deployed in the first place. Great questions. All right, so we're gonna change it up a bit or not really in this case. So the most common high misconfiguration is ensure that a container is not privileged. And now this probably sounds very similar to the last one and that's because they're very closely related. If you turn on privilege to true or if you grant capsis admin to a pod, it's basically like giving them allow privilege escalation. It actually flips. If you don't make any changes to the default, setting either of these to true will actually flip the allow privilege escalation to true automatically. And this again grants route privileges, which you shouldn't do. Now, I do wanna pause here for a moment and say you probably have seen some containers with privilege escalation and it seems on the surface that they need that. So there may be a monitoring tool or a tool that needs access to the host. If you have to have privilege escalation, then it's a better practice to deny all, don't grant privilege escalation and grant the Linux capabilities that it needs from the underlying host one by one. So if it's a network monitoring tool that needs access to the host, then it's a better idea to say no privilege, don't grant a privilege, don't grant it capsis admin, just give it the net raw access. So that way it can only access network. And you've basically limited the exposure. This is actually one of the things that Docker did in the early days when they moved from Docker D to container D under the hood, originally they granted way more privileges than they should have to any Docker container. Then when they realized, and they did that because they were in development and thank goodness they did because they basically created the whole container space that we have. But the problem was they were way over privileged, they gave them way too many capabilities. So if you look at more recent capabilities from Docker they've actually dialed that back. And they've basically said now you can only Docker containers have fewer privileges than they did before. So the same thing is true of any container out there. And I'm gonna stand up on a soapbox and it's very easy for me to say as somebody who's not currently writing code but I would say that there's very little reason it's very rare that you should actually need privilege escalation for a container. And it's a better idea to grant those privileges one by one. Okay, so the solution here is to remove the line privilege equals or privilege, colon true. And this is also true of capsis admin make sure you remove that. Now by default, Kubernetes is secure in this case they remove this, it's by default set to false. And so this means that enough of our customers were going in and turning on privilege to true. Now they're probably doing that like I said for some sort of monitoring tool or some other tool that needed access to the host or needed some privilege escalation. But in most cases I would say that's rare that you would actually need that. And so have a question here that's related that's about what if privilege equals true is required what should be the strategy to secure them? So a couple of things here. Again, standing up on my soapbox I would say don't grant privilege and grant the specific privileges that it needs. So if it needs net raw, just grant that. So deny all privileges, don't give capsis admin and then grant the specific things it needs. And then if it is a container that needs privilege or needs access to the host or needs access to some of the Linux capabilities then make sure that you have some sort of runtime protection in place. And then the question here about the difference between privilege and allow escalation privilege. So like I said, there's very little difference here in that if you turn on privilege equals true then that turns on allow privilege escalation in PSP. And then if you were to only allow network level access to a container, would it also expose underlying network config through BPF? So this one's a little bit complex. It depends on the underlying host that you're running. Unfortunately, I can't give a straight answer for that one but for the most part it's giving access to the full network privileges. So yes, it would be able to do that. And would it be a best practice to always have privilege equals false? I would say absolutely. That is a best practice. Again, I'm being a little bit standing up on a soapbox to say that it's a lot easier said than done. But for the most part, it'd be good to declaratively actually put in your code privilege equals true. Or sorry, privilege equals false or just remove the line because by default it's set to false. So and then how can you grant specific permissions? So that's under the security context. Under the security context you can grant specific Linux underlying Linux capabilities given and the easiest way to do that is look through the documentation. And then question for EKS specifically does Bottle Rocket help further lock down some of these top five issues? So Bottle Rocket itself is not going to lock down any of these specific issues. The next one that I'm gonna show is actually not related to EKS. It's not something you can do with EKS. So I'll talk about how that one's related but that's more EKS than Bottle Rocket underlying OS. And yes, best practice is to set privilege to false and allow privilege escalation to false. And I always recommend just doing that declaratively. So here where I have privilege just removing the line I would also just switch that to false to really make it clear. All right, so and then question about PSP being deprecated there are a lot of alternatives out there for PSP. I highly recommend just Googling it and seeing what you can do. There's also an alternative that's coming out from Kubernetes itself. So there's a new PSP alternative. Although I know that a lot of our customers are exploring because PSP was deprecated they're exploring third party solutions to replicate what PSP can do. All right, I'm gonna move on to the critical one. Now, like I said, we only have a few critical misconfigurations and most of them are related to the control plane API. And that means that if you're running a managed Kubernetes provider on a managed Kubernetes provider like EKS, AKS or GKE these are not an issue. However, if you're running your own Kubernetes cluster then these are things that you should be concerned about. Now, luckily most of them in fact, I think all of our critical misconfigurations are set to the secure default, which is great. But that does mean that we do have a few customers and I will say it's not many but we do have a few customers who are going in and turning these flags to the less secure option. So this first one is an example. This is the most common critical misconfiguration where customers are going in and ensuring that they're turning the KubeLit HGPS argument set to false. And so basically what that's doing is it's unencrypting the traffic from the main node to the worker nodes. And so you basically have no encryption between those two. And as I mentioned in the Kubernetes framework from Microsoft this is one of the ways that bad actors can exploit an API. So they can see the traffic going between the main node and the worker nodes. And they can act as man in the middles in this case. And in fact, insecure KubeLit APIs are one of the ways that team TNT one of the active bad actor groups out there are attacking clusters for a tax like Hildegard. So this is an interesting one. I honestly have not talked to a customer that has turned this off. Yeah, great question about why would anyone turn this off? Honestly, I don't know. I assume it's for debugging or it might just be to try things out. We're getting these scans back from anywhere somebody is running a scan. And so it's possible that somebody in their own demo environment that's completely and isolated they're turning this off. But for some reason enough customers there's enough mass of customers turning off a GPS. So the recommendation here there's this is a flag as you deploy your Kubernetes main cluster your Kubernetes main, your control plane. So you can do it imperatively as a flag or in the actual manifest for your main for your API server, you can set it as a command using the Kube API server, HPS to true. Yeah, one of the suggestions was maybe it's a default where somewhere on some platform may have defaulted to turning this off. And that's definitely a possibility. I honestly, when I saw this I was very confused why anyone would turn it off. All right. And so again, this is the most common critical misconfiguration. This is not one you have to worry about if you're in a managed cluster it's not a capability you can turn off in EKS. All right, so the last one is the second most common low misconfiguration. And again, I have a different misconfiguration in the VM blog right up where I talk about one of the things that you can do to make your EKS cluster more secure by making the API private access. So you actually have to either VPN in or go through a batch and host to access it. And that's just a really tight way to lock down EKS API server. Okay, so for this misconfiguration I'm gonna talk about ensure that tiller which is Helm v2 is not deployed. Okay, so basically Helm is the most common if you're not familiar, Helm is the most common way to package Kubernetes applications and it's used by 63% of the survey respondents from the 2020 survey. I have a feeling the more recent survey that's out there is gonna show a much higher number. Helm is just a really nice way to package up templates of Kubernetes manifest. It makes installing things like daemon sets really simple. So it's a really nice package manager or packaging tool for Kubernetes. The problem was that in v2 and v1 there was a component called tiller and it was basically a privileged which goes back to those first two misconfigurations I talked about, a privilege pod that can hold the state of all the Helm charts that were being used through it. So basically if you deploy Helm chart it would store who did it, things like audit histories and as well as what was deployed. So in version Kubernetes version 1.6 RBAC was enabled by default. And so that means that the history of who was installing what and the history of those jobs was no longer necessary for a third-party solution to be doing. And so the need for tiller to be a privileged pod was taken away. And so basically there was no need to have this super user pod in your cluster and anywhere you can remove those is a good idea to do that. So I recommend Bitnomi has a great write-up on the differences between v2 and v3 and it talks in really great detail about why tiller was a bad idea in the first place and what it could have done and what are some of the security implications of having tiller. So my guess is that this one will kind of fade into the dust because Helm 3 is much more secure by default and we don't have tiller anymore. If you are using v2, I highly recommend backing up and then migrating to v3. And there's a tool out there called Helm 2-3. So there's a lot of migration tools out there that can help you move. All right, now a quick summary of what we went over and then I'm gonna show in that original manifest that I showed what it looks like to make some of these fixes. So use a namespace other than default just to create that logical separation. Don't make any container privileged unless absolutely necessary and it's probably not. Deny all host capabilities and then allow select Linux capabilities only as needed to be very picky about what you allow. And then secure your Kubernetes and Cubelet APIs. Again, this is one of the ways we saw those thousands of open and exposed Kubernetes clusters out there from unit 42. It's because they didn't lock down their Kubernetes APIs. And then don't use Helm 2, use 2-3 and then I put 3 plus because I'm sure we'll get a Helm 4 soon. All right, so just jumping back in here, I added another chart here where I have the fixes in place and let me just close this. And so let's just do a comparison to show what it looks like. And so you can see I've added in the namespace. I've added in the security profile. I've done some things to make sure it's secure beyond just the five I mentioned. But you can see I've added in some resources, the image digest, some resource limits. And then down here, you can see where I've actually turned off privilege to false and read only root file system to true and allow privilege escalation to false. And I've done capabilities drop all. This is where under here under capabilities, if you absolutely need access to NetRaw or whatever Linux capability, it'd be under here under the capabilities, under the security profile, sorry, security context, that's where you would add those other Linux capabilities that you absolutely need. All right, so that's our presentation. Thank you so much for watching and again, check off as free and open source. So I highly recommend you try it out. It covers all the more popular IEC frameworks including Kubernetes and Helm charts but as well as Terraform and CloudFormation. So give that a shot, give that a shot, try it out and then join us at KubeCon, we'll be there all week and you can stop by the booth. All right, so just gonna answer some of these questions as they come in. So it looks like I already answered this first one. Maybe I missed this part, privilege is not set, does that mean the false? Yes, okay, so the question is, if privilege is not set, so that means you didn't include privilege equals false or privilege equals true, the default is false. So again, that meant that in our cases, enough people were going into their manifest and saying privilege equals true because the default is false and so it's secured by default in this case. Thank you. And then can we see check off against the updated YAML? So absolutely, let me just do this. And if I've done my work, then yeah. We actually, I was a little nervous that maybe the team added a few more misconfigurations since I ran this, but we're good. So you can see all of these are passes. We have 88 pass checks and zero failed checks and that's because I went through and I did more than just the five we mentioned. Well, the three that are applicable to a pod, the Helm chart and the cluster, the KubeLit API is not relevant in this case, but you can see I've done a lot more than just the three misconfigurations. All right, so answered that one. And then let's see. And thank you for the shout outs, appreciate it. Is check off available or deployable as a web service? So I'm not sure if I understand this question. We, you are able, check off is just a Python CLI tool. So you are able to run it as a container or locally or in a CI build run is our most common use case that we see. So just adding it to GitHub actions or Jenkins, et cetera, as just one of the build steps. And the question here about the YAML code. Yes, if you check out that blog on VM blog, it does have the YAML code that I showed here. The only exception is the Helm one, which just make sure you use one of the tools to move from Helm two to Helm three. And yeah, appreciate somebody dropping in checkoff.io. That's another place you can check it out. It's also on GitHub at just github.com slash bridge crew IO slash checkoff. And question here about checkoff and container D. So checkoff, it shouldn't matter what runtime you're running. It's actually just scanning the manifest for misconfigurations. And it actually does scan Docker files as well. But the actual container runtime should not be an issue. I think that's it. Yeah, and that's then the questions. Thank you so much, Taylor, for your time today. That's a wonderful presentation and thank you everyone for joining us. As a reminder, this recording will be on the Linux Foundation's YouTube page later today. If you wanna check back, that'll be up. You can share it, watch again, whatever you wanna do. Okay, we hope you will join us for future webinars. Have a wonderful day. Thanks everyone.