 Thank you very much. So welcome folks, thank you for joining us for this webinar today. Let's just get things moving here. So this is me, I'm director of developer relations at Sneak. And I'm joined by my co-host, Scott McCarty, who's a principal product manager at Red Hat. So what we're going to be talking about today is how we can go from an application level vulnerability into a much more widespread exploit. And if we look at the recent history of large scale exploits and data breaches. Very, very many of them have followed this pattern that a single application with a vulnerability in it gets exploited. And then that combined with infrastructure misconfiguration allows an attacker to grow the blast radius of that of that particular, that particular exploit. And so what we're going to look at today is a demonstration of how a potential attack route through a Kubernetes cluster from a starting point of a vulnerable web application. So our thought process for today's demo is that our starting point is that we found a vulnerable web application on the internet. And this is clearly a mockup of a remote command execution vulnerability and remote command execution vulnerabilities are a unfortunately fairly common class of vulnerabilities in web applications. And we've seen recent ones in application servers like Tomcat. And basically the way many of these vulnerabilities work is they allow an attacker to basically run commands remotely on the server that the web application is running on using things like malformed URLs or specially crafted requests. For the purposes of this demo. This is mocked up here and our vulnerable application will allow us to run arbitrary commands using particular formed URL which we'll see in a minute. So, we'll start here from from from setting out what we know so what we really know is that we've we've connected to this application which we know has this vulnerability in it we connected on port 80. It's exposed on the internet so we don't know very much about the, the, the context of how this application is deployed at the minute. We can also introduce our timeline of doom and the timeline of doom is how we're going to track how our exploit scope has changed over time so as we can see we've got our starting point here, which is our application vulnerability which is going to allow us to exploit this remote command execution vulnerability and run commands on the server. What do we do from here. Well, many applications these days as as we all know our running containers and typically containers are configured with environment bear. So the environment bear. The application is real tell us some some interesting things. So if we run our specially crafted URL here and we're going to run this command equals and which is the environment variables that are particular application. And this is told us interesting information. So we can see we're running in a Kubernetes cluster there are lots of environment variables here that are related to Kubernetes. So we know we're going to get internal address of the of the, the Kubernetes API. We can also assume here that the pod running our applications exposing a port via a Kubernetes service we're seeing service port references here. And so when we go back to our, our diagram, we can now start to fill in some some information here so we're going to assume we've got a pod and we've got a service because that's how we expose things within our Kubernetes cluster. And we can, we know that there is an internal IP of the of the Kubernetes API server. So let's also see maybe what we can find out about the network at this point so we're recraft our, our command vulnerability. And we can see some interesting things about the, the, the network here we can see the, the IP address of our pod and we can see the range that that's operating and then that's going to be useful information that we're going to come back to later so for now we're going to make a note of that IP that 10.244.1.9 and we'll, we'll come back to that later on. So when we update our diagram again, we've added now that we've got the IP address of our, of our pod. So, let's see what else we can do by default, every pod in a Kubernetes cluster has a service token auto mounted in it, and this service token is associated with the service account that was used to create the pod. So we can control this on the service account or the pod level by setting auto mount service token account token equals true or false. But let's have a look whether that token exists in our, in our context here. We're going to run this this or kind of cat whether we expect the token to be, and it seems that our permissions that we have whatever the permissions are that our web application is running at, are letting us access that service token. So, let's update our timeline of doom here. And so, straight away we've got some credentials. So we've got this pod token is available inside the pod, and the, the permissions of our application are letting us access that token. So, it's probably worth noting at this stage that everything we've done so far, you could also have achieved if our, if our potential vulnerability was was just a directory traversal vulnerability which let us escape the, the, the directory which our application was running in because we can also find a lot of these, the, the, the environment variables by cutting this file in in proc. All right, let's. So let's take a look at what we can actually do with the token. And we obviously we know where the internal address. Because we can see it in the in the environment variables there. So what we're going to try and do is use the token and query the API server internally in the in the Kubernetes cluster. And so we're going to try and this is succeeded we're going to put together a curl command that's going to use that service token. And because the token exists, we can make some assumptions that we also have the certificate that we're going to be able to see if we can query the endpoint of the endpoints API of the Kubernetes server and this is returned us again is to build a bigger picture of what the cluster actually looks like. I'm actually running a locally on my laptop but in a in a real cluster in lots of scenarios this will actually give you down here in these endpoints will give you the external IP address of the Kubernetes API server. So because we're running this in kind locally where exactly where this where this the external address of the API server is running. It won't be this IP address over just exposed on local host on the same port. But this is a vulnerability, and it's been created by a two permissive policy for the service token so the service token is been allowed to query the endpoint API. So we can see here we've used the gone from the pod to query the internal IP of the API server. And we can add in this permissions issue here where we've been allowed to access the endpoints API. So now we've got a token, and we've got the external address of the API server we can actually configure our local kubectl to use that token against the external endpoint of the API server. So if I go back and just grab that that token again. Let me just shift this over a little bit so we've got a little bit more room here so I've actually got a little helper script here just to set up my my kube config. But obviously you could do this manually. Just make sure that we've got that that kube config set. The plan that I'm going to try and run here is just to do a kubectl get pods. I'm in the default namespace because I haven't specified a namespace. And we can see, I've actually been unable to list anything in the default namespace, and it's saying here that we don't have permissions to list the the pods resource in the default namespace, but we have actually learned something from the output here what we've what we've learned is that the service account here is namespaced and is namespaced into this secure namespace. So, perhaps what we'll try and do now is to use that token to look at pods in the secure namespace. And you can see straight away that we've got some permissions here we've been able to see a single pod that's running in this secure namespace and clearly this pod is most likely to be the actual application that we've that we're that we've just been looking at. Let's just move this away. So, let's investigate a bit more what permissions we do have and there's a, there's a variety of ways in Kubernetes that we can, we can do that. We can use, we can do the kubectl auth command and I can run kubectl auth can I minus minus list minus minus token equals the token that we just looked. And this is going to give us for the default namespace. And you can see here that we can't actually do very much here. But if we run that command again, and we namespace that to the secure namespace. What's interesting here is that on this star resources we have a whole bunch of permissions that we can operate on almost any resource within that name space. And we can look at this in a slightly different way I have just bring this back to the top again. I actually have a kubectl plugin here called access matrix, and I can run the access matrix command. And this gives us a bit more of a readable view on what permissions we have again this is in the default namespace so we're not seeing any permissions there really. But if we run it against the secure namespace, we can see that on a lot of different objects within the secure namespace. We actually have, we actually have permissions. Sorry about that. Okay, where were we right so let's go back to our, our diagrams again, and, and see what we have found out here so. We now know some extra stuff we've connected to the external IP of the Kubernetes server. And we now know that we've got some namespaces, we've got this secure namespace where the application that we were connecting to originally is running. And we also have a default namespace we've got no idea what's running in the default namespace but we know it exists. And there are what we have discovered is that within the secure namespace. The role has been allowed to have too many permissions so we've got a lot of things that we've been able that we can do within that namespace in terms of interacting with objects within that namespace. This is a fairly common pattern where people kind of get the idea that namespaces are actually a security boundary and therefore it's okay for a service token to say, well within this namespace I'm allowed to do whatever I want but I'm not allowed to do anything in any other namespaces. And, you know, as we'll see as we go through this demonstration, that's not really a strong security position to take, because namespaces are not in themselves. A security boundary and it's very easy to escape namespaces in various ways. So adding to our timeline of doom, we now know that we've got this role which gives the service account too many permissions in that particular namespace. So let's go back to our shell again and let's just clear this down a little bit and make it a bit easier to see. We know in our secure namespace that we've got this pod running. So let's try and get a shell on this pod. So if we do kubectl exec minus it our pod name minus bin bash. Okay, so now we've managed to get a shell on our compromise pod. So let's take a look to start with who we are in terms of users within this pod. So we've run who am I there we're running as this web admin user presumably that is a restricted permissions user just being used to run the particular web application so perhaps we can see whether we can elevate our privileges by running pseudo within this within this pod. You know we don't even have the pseudo command available to us within the pod so you know that probably when the person set that this this particular pod up you know it was a restricted user can't run as route. It won't can't be, you can't run pseudo. So you know that lots of people might consider that's a fairly good security position to take when we're doing application deployment. So let's just try one other thing here. I'm going to try and create a file, and that has actually succeeded we can see that file there. And this is actually another vulnerability here. So we can create files within pods it potentially means that as an attacker, I can download new software, I might be able to change configuration. I know by the, the my ability to use curl to access the internal API endpoint that I have curl available so that definitely means that I can start downloading things into this pod, and potentially make changes. So now if we if we go back to our slides again, you know this is our next misconfiguration here this read write file system, which allows an attacker to modify the container. And we would protect against this by configuring the security context of our container and setting this read only root file system equals true, which will then prevent an attacker from being able to download. So that brings things into our container. So let's just come out of our container again for a minute, and let's try doing doing some other things here so we know we have permissions in this secure namespace so perhaps we're going to try and spawn a root pod. So I have some some YAML files here. Let's just have a look what we've got here. Let's see what our root pod.yaml will try and do so this is going to try and spawn a an alpine image alpine by default will will start as the root user. So let's apply that little bit of YAML. What happens so it looks like something's happened here we're saying that pod root pod has been created but let's actually see what's going on in the namespace. And so we've now got an error here this create container config error on our root pod. And if we use kubectl describe here we can get a bit more information on what's actually happened there. The interesting bit here is this error here so container has runners non-root and image will run as root. So something has prevented us from spawning a root container. And this is likely a pod security policy but at this point we don't actually know that we know there's some control here that has stopped us from being able to spawn a root container. So let's try something else. Let's try maybe running a privileged pod but we'll run it as non-root. And again let's have a look at our YAMLs and see what we're actually going to do so non-root privileged. So this is going to try to create to spawn a container image that I've prepared earlier. It's not running as root it's running as a normal user. But what I have done is set the privileged true flag. And I'm going to try and mount as of the host volume the host root directory into my container which I would be able to do if I was running privilege because privilege basically gives me all the permissions. So let's just try an ISF YAMLs non-root.yaml and in the secure namespace. And we've again been stopped from running this and this is given us some more information that we didn't have in the pure root case that this is actually hitting a pod security policy here. It's saying that privilege containers are specifically not allowed within our pod security policy. So we can now go back to our slides for a second. We can add some more information that we've discovered through that little exercise there. So we now know we've got a pod security policy that's running in that secure namespace and that's what stopped us from being able to spawn both root and privilege containers. So how does that stop us from extending our exploit? Well, let's try something else. So, again, we'll look at some YAML that we've got here. We're going to do non-root, non-privileged. So I'm going to drop all the root and privilege stuff. I'm just going to try and spawn my custom container image that I have in a registry on the internet. So let's see what happens if we try and run this one. And it's F, demo, YAMLs, non-root, non-privileged. And we want to run that in the secure namespace. Well, it looks certainly a lot more promising here. We'll get pods in the secure namespace. And we can actually see that this pod has succeeded. So this is kind of another misconfiguration vulnerability here as well in the sense that this cluster has not restricted me from spawning containers using images from an external repository. So that's basically allowed me to pull any random container image that I could have with whatever tools I want in there from a registry wherever, and it's allowed me to spawn it. And this image isn't running as root, and it's not running privileged. But I know that this, this particular container image has a remote shell running in it. So now I can do Cupid will port forward to that particular pod. And I know that this is running on 8080 because I built the container. So I'm going to port forward to port 8080. And now I should be able to just open a web browser to 8080. And now I've got a shell again on my newly created pod that that I now have access to all these tools. And clearly, I could have used Cupid to to to exec into that container. I just want to illustrate here how easy it is for you to then get access by different mechanisms into a random container that I happen to have spawned as an attacker. So, again, we weren't running as root when we when we created this so we can see who we are here with this sneaky user it's not a it's not a root user. And but what I can do here, even though I didn't run as privileged and didn't run as root, I can run pseudo Sue, and I can achieve a root like shell this is not going to be a full root privileged shell, because of the context that the user was spawned in, but it's still going to give me additional permissions that I did not have as that sneaky user. And the problem that's been that that's happened here is that the pod security policy didn't include allow privilege escalation equals false. And we'll look in a minute why this is important, but we can look at our next vulnerability here is that we didn't have this allow privilege escalation false set. What that means is that even though I wasn't running as root I could escalate my own privileges and in this case I use the pseudo command but you can also do these things with especially crafted binaries. And lots of people to kind of make the mistake here that the by saying, well I'm not going to allow privilege. I'm not going to allow privilege containers and you've got to run non route that you don't actually need to set the allow privilege escalation equals false, but that's really not true because and we'll see why in a second, because if I as my normal user within this container. Let's just have a quick look at the network here so I know now what the what the IP address of this particular pod is, and what I'm going to try and do now is from my pod is explore the network. Because what I really want to do is to, to find out what other things are running now in this Kubernetes cluster, maybe new attack vectors new insecure applications that I can continue to extend my exploiting. If I run the end map command to just have a look as a normal user here 10.244.1.18 flash 24 so I'm going to try to scan the, the network segment that that my particular part is in. If I try and run end map as a non privileged user, the kernel stops me from doing that because I do not have the capability, the Linux kernel permission level to be able to to access some of the low level networking features in stack to be able to do scans. But because I'm able to pseudo here. I can just copy that command. And now I'm able to scan that that the network segment that my pod is running in. And, as I noted earlier, we know that our original vulnerable pod was at this address 10.244.1.19 it was exposing a port 5000 service. And what we've now discovered is that we have another application at 10.244.1.10, which is also exposed on port 5000. Now this might be very interesting to us, you know there's the possibility here that not only do we have one vulnerable web application running within this particular cluster but perhaps this is another instance of the same application. So let's update our update our picture. So we've added our pod security policy didn't disallow us from privilege escalation so we were able to escalate privileges. And we have discovered another application on port 5000. That's in at the minute, all we know is that it exists. And then that we've been able to poke around in the works we didn't have any network controls in place that stopped us from going from one pod and scanning the network. And we'll we'll go and use that that particular vulnerability again now because we want to discover more about the, the new application that we've just discovered so. My, my sneaky pod has another tool in it called so cat, which enables me to create tunnels between particular instances. So what I'm going to do is create a tunnel from my pod to this new pod that we've that we've just discovered. So I'm going to run so cat I'm going to say, we're going to listen on port 5000 and one, and we're going to the other end of that is going to be this 10.244.1.10. And that's going to be on port 5000. And we'll just background that process. So, back on our local console now so we're going to stop port forwarding on that 8080 port which was giving us access to the remote, the remote console there, but I'm now going to put forward to that new 5001 port, which is the other end of the tunnel connecting to our mystery pod. So now we've connected to to that. And let's just have a look what's actually on that on that particular end point so we're connecting to 5001 on local host. And here we go we've got another instance of that vulnerable web application. So it looks exactly the same. It's probably go all the same vulnerabilities that we know about so let's try and and see whether we can get a new token here. And so by passing again on our formed HTTP request we've managed to expose that a different token. So, let's go and see what we can do with it. So let's just make some more space here so. First of all, let's just modify our COOP config to use this new token. So that was the original token we got. Let's just comment that one out. And let's put in this new token. So, so again, you know, I normally start whenever I'm poking around in clusters by just let's see whether we can see anything in the default namespace. So COOP cut will get pods in the default namespace, which if you remember the first time around we had no permissions in the default namespace. This new token has given us permissions in the default namespace. So if we again look at use the COOP cut will auth command. Can I list minus minus token equals our new token. So we can see here that we've got a substantial set of permissions across lots and lots of resources within the default namespace. So with this token what we've managed to do is to escape that secure namespace and have permissions in the default namespace. So if we go back again to our slides. Again, we were able to escape the network controls we were able to create that tunnel into this into this other pod that was running in the default namespace that we didn't have access to before. So what we've there's our tunnel from 5001 to 5000 and we now know that that particular pod is running in the default namespace. So let's try something else with our new token. So we know what we can do with our new token. Create minus F demo yamls. We won't try and run as root but what we'll try and do is create a privilege pod. And what we can see here is that that's now worked. So the net this namespace is not restricted by the pod security policy that we discovered before. So with the privilege pod we can do a whole lot more because we've not just launched a pod that's privileged. But if you remember from the previous look at that YAML file, non-root privileged. What we've got set in here is a host path mount, which is going to mount the root directory of the node. And these are extremely dangerous and we'll see exactly exactly why in a second. So if we now get a shell onto this new pod that we've just created, is that minus IT, non-root proof. And we're going to execute bin bash. So we've now at this point we have a sneaky user but we know we can escalate privileges to root. And if we look at the process table as this root user, this is the root user in the container. We can see the privilege, the process table here is pretty small. We're only seeing the things that were actually launched within the context of the particular container. But in our YAML file, we mounted the host slash directory into this mount path of slash to root. And so what we can do here is change root into that directory where the host file system is mounted. And here we've now got the nodes process table. So we can see here we've got instances of container D, we've got all sorts of different processes running here. In a real cluster, we might have other containers. We've got the Kubler, we've got everything that's running on the host we can now see. So when we go back to our diagram here, we've now added this privilege pod which has been able to mount things from the host. So what now we have the host file system mounted in our container, we actually have access to the token that Kubler is using. And the Kubler token has much more permissions than those service account tokens that we've been using up until this point. So if within our within our pod, we now do export kubeconfig equals etc kubernetes kubelet.com, which will give us the kubelet token in our kubeconfig. And we now run kubectl from here. And we're going to look at what pods are in the kube system namespace of the kube system namespace is where all of the control plane for the kubernetes cluster itself runs. So we can see lots of information about which pods are running the control plane. And we could dive deep into where those particular pods are we could start attacking those pods. So for the purpose of our demo today. The other thing that I'm interested in is what the names of the nodes are. And so I can find the names of the nodes. And that's going to give me the ability to now define which node I'm going to deploy a particular pod onto and that's going to be important for extending the next piece of our exploit. And just to add to our diagram here. So we've got no permissions restrictions in the default namespace so that pod security policies were only applied to the smaller namespaces. And we now know a lot more stuff. We know some stuff about the kube system namespace. And the pod that we're particularly interested in in this instance is our xcd pods. In this case it's a single pod but we're going to the next phase we're going to to try connecting to that. At this point when we've got the kube token would probably have pretty much control of the cluster but we are going to take this a little stage further, just to show you some other ways that we can potentially do stuff. The first thing I'm going to actually try and do is start a pod using my kube token. So I'm going to just run busy box and see what happens. So, straight away, we can see that this is failed because the kube token actually isn't allowed to start pods like that. We can't use that to start pods directly on our nodes. What's, what's interesting is that we get this output that the that we can create what are called mirror pods, and this, this is interesting once we have the kind of ownership of the node here in the Kubernetes cluster is to put YAML files directly into each etsy Kubernetes manifests on the node themselves and those manifest files will get automatically automatically run by the cluster. And so that would give us many, many other attack vectors against the other control plane pods that are running in the kube system namespace. We can actually sort of escape the pod security policy we know what nodes we've got, we can actually kind of attack on a different vector here. And what I'm going to do, we're going to launch a pod to the node that's hosting the etsy D, the etsy D cluster. And we're going to again use especially crafted YAML file here. Let's just come out of this pod actually exit out of that again. Right. So, what we're going to do is we're going to run an etsy D client pod. And we obviously know that we can mount the host file system we've been able to do that right now. And so we know where all the configuration for etsy D lives so what we're going to do is to mount the etsy D configuration. And then we're going to set a whole bunch of environment variables for our etsy D client, because we know where these things are going to live on the host file system. So, let's just apply this one. Client. That's running. So we've now got our etsy D client pod running so let's actually see whether we can use the etsy D client in that pod to connect to the etsy D cluster. So we're just going to run etsy D Kettle member list. And we're getting responses back from the etsy D cluster so we know that this pod is actually connecting to etsy D. And that gives us some some more things to update in our diagram. So we've launched a pod, which has the host file system mounted. It's connected to the etsy D, the etsy D cluster. Now etsy D contains a lot of interesting information about the cluster about the whole Kubernetes cluster, not least of which is secrets. So let's see what secrets this etsy D cluster actually knows about. And we can see we've got a whole bunch of secrets here that we could use for for lots of different things. And the one we're particularly interested in here is this cluster role aggregation controller secret. So, let's see if we can get the token for that particular secret. I'm just copying paste this one. It's a bit long. And we need that little things on the end sx to VV. And we have actually got another token. So if we grab this token. This up a bit so we can have a bit more room. And we run cube kettle off. Can I minus minus list minus minus token equals our new token. And what we can see here is that if I just move this over a bit that what this token gives us is the ability to modify roles in the cluster. So that's effectively cluster admin rights because I can change roles give people whatever permissions that I particularly that I want to give. And at this point, it is kind of game over. It probably was when we had the Kubler token to be honest but this just shows another mechanism by which you can access through that information. So let's just update our picture again so we've got we've got a token that gives us cluster admin rights. And at this point before I pass over to Scott, I should really say that that there's a lot of thanks and props needs to go to all the super awesome folks in the Kubernetes security community. Many of this stuff is based on on various great presentations that they've gone done in the past but particularly mark Manning in cold water and Duffy Cooley. So at this point, I will pass over to Scott. Right, thank you. That was awesome man. I love watching you escalate and then find I love that you had scenarios where were like you failed and then got around them that showed like how realistic, like, it is to basically attack one of these and escalate. So then we have to ask ourselves, okay, so you found some dead ends and then you found some ways but you know to get through around those dead ends but then how could we have prevented this right. So, so I'm going to dig into that a little bit and next slide if you don't mind. I think I like to go back to basics because I'm old, and I always think of these things from like 20 years ago that I learned and I think about how can we apply them in, you know, in the context of Kubernetes and containers so these are the three basic things we've used these forever CIA confidentiality integrity availability. And, and I asked like so if you think about what Matt did basically sorry one second. There's loads wait a second. That was what I wanted to do. That's all right, I can talk through it as your, as your, as your, as your. So, so confidentiality availability integrity right if you look at what Matt did he own the cluster right so once he owns the cluster he has full ability to compromise confidentiality availability integrity. He can literally get into databases and any app in the cluster that he wants to he can steal passwords he can do anything I mean once you have cluster admin you have everything. And so, so you look at confidentiality definitely could get access to data availability he could take the cluster down if he wants integrity he could hide something nasty and they're like a Trojan and just not let anyone know that he got into the cluster. So essentially all three of those, all three of those attack vectors have been compromised with what he showed. Now let's go to the next slide if you don't mind that. Are we seeing the slides again that's got just to make sure. Yeah, well, okay cool. No, so on this one right. No, no one more back. Sorry. Yes, so. So now we ask ourselves okay well what are the primitives we have in this new world of containers right like so I try to break it down even a little bit further than just the Kubernetes cluster we think about all right well container images is definitely one sort of new primitive if you will this is a thing that we have to think about the container host is another thing that we have to think about the registry server which honestly has its own role based access controls and all kinds of nasty things that a user could do or or you know to hack things and then the container orchestration itself now what we went through today was mostly the container orchestration, but we did actually you know if you look at the first, the initial attack vector that he used was based on a container image. And so that could be, you know that could be the app that could be libraries it could be whatever in this case we showed an app that was already hacked, but, but you have to think through the entire stack of software there because that's special I think so the next slide please map. So, one thing I didn't comment I meant to comment on the last slide though was that, you know these are the new things you need to think about a container environment but you still have all the old things to worry about to because you know there's things outside of the like layer seven firewalls and, and you may have like front doors and networking and you know IP, you know address spaces and things like that, but in the context of containers I want to say these are the four things. So now that like I mentioned though the container image is special right like that's actually how we got in the front door of this whole Kubernetes customers with it with that and so if you look there's sort of three layers I try to break it down to about three layers of content that's in there. You know there's the app that's what the developer writes themselves and this could be some code that is copy and paste it from GitHub this could be you know forked code from GitHub this could be code you write yourself and you just made a mistake and it didn't mean to do something that allowed a hacker to get in. There's also below that though there's things like the JVM, you know the Java virtual machine, the Python interpreter, the, you know the Ruby interpreter node just blah blah blah etc etc. So there's, you're inheriting some code from just the interpreter or even the language that you chose. And then below that, you know what a lot of people forget is, is that like the JVM is still written in C, and it's compiled against G lib C in most Linux distros and so is no jazz Python and Ruby etc etc php etc so sort of you got to think about that entire stack in the container image and say okay well where's somebody gonna get in right is it some, is it some exploit to the G lib C is it some exploit to the JVM is it some exploit in the app code I wrote, obviously the most likely is in the app code, and it probably gets less likely as you go down and in, but it's also much much worse the lower you go in the stack. If there is an exploit of some kind. And then I mentioned the host right you have to think about the host and and the Kubernetes cluster itself which is actually what mostly the rest of the escalations that that Matt did were around those mostly. So now next slide please. So, one of the things can we can we prevent this right so like one of the ways that we would think about preventing the attack surface. For getting in that app is again the critical execution path so so in this case we showed a web app that basically was able to just execute commands but but in a real world scenario you'd probably have some very complex set of ways that they either hacked in you know, either a transaction or a, you know, URL attack that exploited something in some text processing library that maybe relied on something in G lib C but anyway, no matter what the attack vector was in the critical execution path, it probably involved some see library some language runtime and then some code you wrote, and then how do you minimize the permutations of that critical execution code so in this example I'm showing on the left, if you just go out and choose any container image. So you're going to like and I showed a random selection of popular container images here let's say there's three different versions FIDL or 343332 Ubuntu 1604 1804 Ubuntu 20.4, you know alpine 3.123.11 and 3.9 and then say red hat universal based image seven and eight. I just kind of added up the permutations of stuff that you would have it's you got a. Three models and 11 different versions of open SSL. This is a little more attack surface than if you just selected like a single base image, and then standardized on one G lib C and one open SSL and this would be. These are libraries that I specifically picked that are in the critical execution path of almost everything we use today like every web server, you know, engine x open, you know, engine x or a patch you're going to use open SSL and G lib C. So these are these are not like, you know, hypothetical and these are ones that that quite honestly we had read up a lot of attention to because they're so critical in the execution path. But if you can minimize that set of dependencies in that critical execution path you can actually really reduce your attack factor and lower the likelihood that somebody would get in that front door through the app. So the next, the next slide please. So now going, you know, I started with a container image because I think that's how we got in the front door here and also I think that's the first thing a developer touches. But also, you know, Matt's Kubernetes cluster was actually configured decently well in certain ways, but clearly it only takes one to get in right like they did prevent the runners root user which is actually good and also prevent it starting a privilege container. And obviously the escalation of privilege kind of got us in a default, for example, open shift environment which would, you know, we do prevent those kinds of things with like our policies out of the box. So, so, but, but I mean obviously we have to have something for an example and so he had to have a way to get in but but but that matters right that entire stack and then when you go to change those defaults you want to be very careful about why you change those defaults because the common thing that I see people do is they go to run a container image off Docker hub which Matt showed, you know, if you pull a remote container image. The first thing a developer says is wow why won't this run as root you know they're annoyed they're like disabled this run allow root run it you know run as any user in the cluster and that is a really bad mistake to make. And a lot of people do it it's not uncommon to be annoyed by security controls, and then just disable them without fully understanding what the implications are and how you're letting someone have essentially an attack path like that. Yeah, in that scenario I say start with a stack that is already configured default by secure, you're secure by default and limits those things and then don't turn them off that was you know this is like the disabling I see Linux. You know in the old world, which actually still exists in the Kubernetes world but um, and honestly nobody shuts it off because you don't even notice it, but, but they do disable the security context a lot of the times which is kind of a new, the new version of that. So next slide, I think that was really all I, but I wanted to at least kind of like hammer home, you know be careful to code you download obviously that's probably the big O from a computer science perspective of like how somebody's going to get in your cluster. Start with the best trusted base image, start with a trusted Kubernetes platform don't disable the things then verify everything right so I think like sneak and red hat are good partnership because you know the once you've kind of downloaded that code and you're being careful then you scan it to double check scan the you know the container images and verify everything but you have to sort of trust but verify because you can't, you can't scan your way out of bad quality right like you can catch it and maybe remediate it but it might already be too late so so always try to start from a good place and then and then verify it is kind of what I want to hammer home. Absolutely and I mean I think that's, you know when you look at that that pathway that we showed today while the walls as bits of that they were obviously mocked up for the purposes of of this webinar, you know, you really could have prevented that at the start by not deploying that that remote command execution vulnerability I mean we see those in the wild all the time right so it's not that this is a completely unrealistic scenario that doesn't happen I mean you know that that was the that was the point in but I think the other key bit really is that the the configuration is becoming as important as the application vulnerability and you really for you know in your cloud native security posture you've really got to be thinking of all these things in in in equal terms because if you if you get one of them wrong then you've really opened yourself up to to a much a much more a much wider blast radius than than just simply having a vulnerable application. Yeah I would concur on config the more I think about it config actually perhaps is actually maybe like 5x as dangerous as even the code a lot of the time. So we've got a few questions here I know we've only got five minutes left I think but we'll try to get through through some of them. So Pavlo says what are the best practices for Helm charts to configure our back properly. Should I follow the least privilege principle. I think we're probably slightly too much scope here to talk about exactly how to configure our back in Helm charts but I would absolutely concur with you that least privilege principle is what you should always be following in these circumstances. And that applies to to you know lots of things that we've looked at today as well applies to things like setting capabilities in security context it applies to you know if you start from least privilege. You're always going to be in a better position. I'm not a car the can I hope I pronounced that correctly how does an attacker identify that the web application is hosted on a Kubernetes container. So perhaps you missed the very start of that. That hack. But we in this in this context, we looked at the environment variables the application was exposing the environment variables that told us what the context was that that container was being executed in because we had access to to a remote command execution vulnerability, but I actually showed that even with a directory traversal vulnerability which is a in some ways I guess a would be seen as being a less critical vulnerability, because it only allows you to go to get information rather than directly run attacks, but you could have found all of that information through a directory traversal as well. Paul asks to use the service account token do we need cluster machine access or coming access from any machine over the internet. If you know the as I showed there if you've got the API external IP address of the API server, then potentially you can access that from any machine over the internet. So I'm going to show you a video says are the vulnerable application endpoints on purpose or is that something Kubernetes is exposing by default. Do you mean the AP, I'm going to assume you mean the API endpoints so exposing API endpoints is an enormously common configuration I think out the box actually vanilla Kubernetes will expose those. I mean, you know, you look at all the problems people have with crypto miners on Kubernetes clusters this is exactly because the endpoints are exposed and available on the internet so hopefully that answered that one. And the last one was kind of network policy prevent that tunnel connection created from the pod in the secure name space to one and default. Absolutely. And that was the point that I was trying to make. I think, whilst you know that it kind of things like network policies or the other thing that would have prevented it would be, you know, if you've been using a proper service mesh setup where you're only allowing a pod to talk to the thing actually needs to talk to and talk to themselves. But again it comes down to that principle of least privilege. You know and I think we're really only just become beginning to scratch that one particularly around network connectivity between pods I think is, you know service mesh is a complex right Scott and I think you know, yeah, yeah they're all so easy for people to get wrong right so you know but in theory that's one of the things that that service measures are supposed to kind of give you is that defined pathway between applications. Okay folks I, I, oh we've got some more coming I don't think we're going to be able to get to the rest of these I'm afraid I think we're actually out of time. If you if any folks on here want to contact me on LinkedIn semi questions I'm on LinkedIn and on Twitter feel free. Yeah same here. Yeah, especially I noticed the one around attack surface I'd be happy to discuss that more afterwards if you find me father Linux on Twitter is easiest probably. Okay wonderful thank you so much to Matt and Scott for their time today and thank you for everyone with for joining us. I hope that you will join us for future Linux Foundation webinars and just a quick reminder that this recording will be up on the Linux Foundation YouTube page later today. Okay thank you so much have a wonderful day.