 Welcome, everyone, to this afternoon session about security and namespaces. Talk is named, Keeping Kubernetes Safe, the Lowdown on Lock Namespaces. And yeah, quickly go about the introduction. My name is Marco de Benedictis. I am a senior consultant in control plane. We do everything cloud native and DevSecOps and obviously security. You might have seen me, well, at our booth or at the capture of the flag event yesterday and today. I'm based in Italy. And yeah, I hold a PhD in computer and control engineering. I've been doing cybersecurity consultancy for a number of years. More recently, I'm also being quite involved with some of the CNCF security tag activities. As well as, yeah, I was just, that's actually a very nice thing. We actually just published a new security certification, the Kubernetes and Cloud Native Security Associate. And I was fortunate enough to be one of the developers. So yeah, check that out. So this talk is, yeah, you might have well guessed about namespaces. And yeah, the security features that are either namespace aware or bound to namespaces. And the reason I want to do this talk is basically to go through what can go wrong if our namespaces aren't configured properly from the security standpoint or someone can tamper with their configuration. And what I also want to leave you with is a number of mitigations you can effectively enforce in your clusters if you're running any in production. And yeah, you will see this is nothing expensive, actually. The talk itself is motivated by the security reviews we do, the threat models we do in Control Plane, where we see many customers relying on a lot of security features which depend on very, very secure namespace configuration. And what we see is that they don't always really realize what's the impact of a misconfiguration on a namespace level. So yeah, that's basically why I'm talking about you, about this with you. So we all know about Kubernetes namespaces. They are not a security feature by itself, but we use them for logically grouping resources in a cluster. So they definitely make our life easier for organizing resources in a cluster. They are not a security feature, as I said. So they do not provide any form of isolation. And yeah, but we will see that many, many security features depend on them. Whenever we spin up a new cluster, few namespaces are created by default. We don't need to cover them. But yeah, obviously, you will know about Kube system, which holds the Control Plane components. And we typically spin up new namespaces as we build applications on Kubernetes. Not all resources are namespaced. So yeah, this is the output of the API-resources command. And yeah, many resources are actually global to the whole cluster. You will see this with the namespace column. Yeah, interestingly enough, of course, namespaces are a global resource, right? So this is a quote which comes straight from the Kubernetes documentation. I hope they haven't changed it just before this talk, because it would be very embarrassing. But this is actually telling us that, yeah, you don't really need namespaces unless you are running like tens of apps, or you have like tens of users for your cluster. So you start using them when you need the features they provide, which is kind of true. This is another quote from another documentation page, which is very, very specific, just telling us that, yeah, using namespaces is optional. So is that true? Well, definitely from the security standpoint, yeah, the first key takeaway that I'd like you to get from this talk is that they aren't really optional. And we will now discuss a number of use cases which, well, where namespaces are very, very important. So first one I want to mention here, obviously, is role-based access control, where namespaces provide an essential feature to make sure that authorization is properly enforced in a cluster at the users or at the workload level through service accounts. Don't need to go through the different RBAC resources, obviously, but here we have an example of a role which grants us some level of access to a number of resources through verbs. And this is associated to a user through a role binding in a specific namespace. And why is that good? Because if we have two namespaces and we want to segregate access to a user to just the first namespace, we can effectively do that. And if we are not really segregating our applications in different namespaces, this becomes way, way harder. Moving on to networking, we will see a number of slides here with regards to this. And there are actually several networking features that depends on namespaces configuration. First one we see here is actually the internal DNS resolution in a Kubernetes cluster. So by default, all services and pods get the DNS name in a cluster where the namespace, obviously, is also part of that. And why is that important? Well, because this allows us, effectively, to have intercontainer network communication in our cluster, which is also across namespaces. And this all comes by default and for free. And we don't need to care about the IP addresses of our pods, right? And obviously, the services. So yeah, another important thing we want to mention here, obviously, this is a security talk. So is that by default, through these DNS names, cross namespace communication is also allowed. And yeah, there's no default network segregation in Kubernetes clusters unless we effectively enforce network policies in it. So this is an example. You see an implementation of a network policy on the left. And yeah, kind of a picture depicting it on the right. We have this pod, which belongs to a DB namespace, which has a network policy, which is basically allowing traffic to it only from specific namespaces. And this is achieved through a namespace selector configuration. In this particular example, this configuration is actually targeting other namespaces by their name through a specific label, which is Kubernetes.io slash metadata name. But we will talk about this later on. The important bit about this configuration is that the pod belonging to the other namespace, for example, the one on top, cannot really talk to our DB pod because there's no allowed network communication, ingress network communication through that. This is a bit of a newer feature. I'm not sure how many of you are familiar with the Gateway API. This is GA since October 2023. And yeah, I wasn't really familiar with it. But when I started researching into it a little bit more, I came across a very interesting feature they provide, which is called the cross namespace routing. And this basically allows us to implement separation of duties with a centralized gateway management, where the gateway is instantiated in a platform namespace. And different HTTP routes can be implemented into application namespaces. So effectively, the gateway configuration, which includes like TLS and yeah, platform level configuration is kind of centralized way away from the applications, which only manage the routes. This feature also depends on namespace selectors. So this effectively allows, in this example, only the first two applications to get public ingress, while the third application, which is supposed to be like a private application, is not allowed this level of access. So moving on from networking, let's quickly talk about security context. So this slide is obviously talking about pod security standards, which are available since 125. They are the replacement to PSPs. And they're kind of handy because they allow you to define restrictions at the pod and container security context. And you do it just by specifying different levels. So each level is mapped to a set of constraints. And we have privileged, baseline, and restricted. And for each of these levels, you can actually define an action, which tells the pod security admission controller what to do if a violation happens on the level. So you can effectively enforce a restricted level, like in this example on top, which would block a pod running as root. But you can also have a baseline level, for example, which is enforced. Or you could only audit or warn the user if a violation happens. One important thing I want to mention on this before we move on is that once again, namespace are very, very important because pod security standards are defined at the namespace level. So if you end up with an application with a ton of microservices, and maybe some of them, maybe only one of them, requires you to relax the security context restrictions because maybe as to run with some level of privileges, you will end up with lowering the level of restrictions on all the different pods in that namespace. So that's where additional namespace segregation might come into place. Yeah, we talk about RBAC, networking, and security context, but we also need to mention resource management, which also has a lot to do with namespaces. What won't do much on these slides, just want to mention that resource quotas and limit ranges are two important Kubernetes features, which are also namespace bound. With the resource quotas, you set limits to storage, compute, and object count within, let's say, at the namespace level. Sorry. And the security benefit of this is that you can effectively limit the risk of noisy neighbors in a cluster, where one application takes too many resources and basically renders other applications like non-responsive in a cluster. I've mentioned limit ranges as an additional feature on this. I see them as complementary to resource quotas because with limit ranges, you can effectively define limits for individual containers, pods, or persistent volume claims within a namespace. So while the resource quota applies to the whole namespace level, the limit ranges affect the individual instances of these resources within each namespace. And you also get an interesting bonus, which is to inject default compute requests and limits for your pods. So is that it? Have I convinced you or not about why security and namespaces are kind of something to do together? Well, it's actually more than what I've just talked about, and we don't have time to cover all the use cases, obviously. But I'd like to mention here, for example, multi-tenancy considerations, right? There's no single definition for soft multi-tenancy, but typically the way I see it is that when you are shedding the same cluster among different tenants, so you might have a tenant per namespace or set of namespaces with, I don't know, a single control plane or a number of virtualized control planes. As I said, there is no official definition, but obviously if we are considering multi-tenancy, that's where namespaces are very important, because different tenants deploy their applications into different namespaces, and we need to make sure that one application is not lowering the security of the whole cluster or compromising other applications. Other use cases are like, I don't know, secrets to pull images from private registries, which are typically defined at the namespace level as well. Custom resource definitions, which can be global or namespace. And obviously, globally defined CRDs are kind of risky, because if you are able to act on a global CRD, you can modify configuration of the cluster level rather than, let's say, at the namespace level. So that's also important. And one final example is also like PVs and PVCs, where this is kind of interesting, because persistent volumes are global, but persistent volume claims are namespaced. So you can effectively end up with some other application, like getting hold of a PV, because a PVC gets deployed before the one you are going to deploy. So there's a lot of considerations here, right? And basically what I want to tell you about this is that although Kubernetes namespaces are not as security features, they have a lot to do with security. So in particular, a lot of the things that can go wrong are about misconfiguration. And in Control Plane, we see this all the time, misconfiguration is truly a leading cause of cloud insecurity. And many, many threats are generated by either misconfiguration or lack of audit or any configuration. So we need to care about this a lot, or bad things will happen. So let's now present some examples of threats to our clusters based on namespace misconfiguration. And the first thing that I want to mention here is one interesting, let's say, operation which might not be that clear, but it's actually readily available for you in any cluster, really. So namespaces themselves are a global resource. I've said that. But a role with a role binding which allows you to patch a namespace is enough to modify an existing namespace. So this might sound counterintuitive, but effectively, if you have this level of access which is displayed here in the slide, you are able to patch a namespace and why is that a security issue? Well, we've just talked about many security and networking functionalities which are bound to namespace labels. So patching a namespace means that you can effectively modify its labels or even add new labels. And yeah, we talked about pod security standards, network policies where namespace selectors are very important. So we'll now look into more detail about a practical escalation of privilege where effectively a namespace relabeling is the attack vector. So if you are able to relabel a namespace and you are running a cluster with pod security standards, you might be able to bypass existing security context restrictions just by enforcing a privileged level, which basically bypasses all security controls. If that's the case, next step would be to run a privileged pod. And if you are actually able to run a privileged pod, then yeah, you are going to harm a cluster. And I hope I'm not saying anything which is surprising you by talking about privileged pods. But like an example here is that, for example, if you are able to run a privileged pod, you can mount the host file system, break out on the underlying node. This might be like a worker node, so you might only be able to get information from other application pods. But this might include data stores. This might be CI pipelines, so there might be secrets there. But if you are very unlucky, and this is an on-prem cluster, the pod might even be scheduled on a control plane node by using specific configurations to force a pod on a master node. And you might even be able to steal highly privileged tokens. And yeah, that's really game over for the whole cluster. Talking about network policies and labeling, well, you might be able as an attacker to bypass an existing network policy restriction through relabeling of an empty space. So in this example here, we have this other NS which wasn't previously able to talk to our DB. But if you understand that as an attacker, so if you get information about how the namespace selector is built through a network policy in the DB namespace and you can effectively relabel your namespace, you might grant yourself additional access, OK? And example on the Gateway API is very, very similar, so I don't want to spend much time on this. So yeah, basically namespace selectors are used with Gateway API cross-namespace routing, so if you can modify, for example, this public Gateway label in your namespace, you might provide an existing HTTP route public ingress. And effectively, this can result in very bad things like hijacking of shared gateways, exfiltration of data, unauthorized access to internal phasing applications. Once again, talking about networking and namespaces, this attack doesn't have to do with the labels, but with how namespaces are named. And this might not be so straightforward, so bear with me. So due the way internal DNS resolution works, we might be able to spoof DNS names by naming our namespaces as top-level domains. So in this example here, we have a namespace which is created with the com name and we create a service in it, which is called GitHub. And effectively, yeah, any pod, any client, which is just using a relative DNS lookup in our cluster, will use the local domain search list for this DNS name, which is displayed at the bottom of this slide. This is injected through core DNS. And effectively, this might end up with a DNS resolution which is local to the cluster and not on the public internet. So depending on the way DNS resolution is performed, either in a relative DNS lookup or an absolute lookup, you might end up with this local DNS lookup to actually work, which means that you can spoof an existing public domain and maybe use that for, I don't know, maybe downloading some malicious code into your pod or other bad stuff. This is actually one of the few things that are actually mentioned in the Kubernetes documentation. I want to mention that. So if you look at the documentation about namespaces, they tell you that you shouldn't allow TLD named namespaces. But obviously, there's no built-in support for that. We will see something later on about this. Last kind of threat I want to talk about is obviously a very broad category. This is denial of service. I'm pretty sure everybody knows what this means. And in the context of namespaces, this has to do with the improper resource quota limit range configuration or non-existent configuration of those things. And some of the examples we get here are, for example, noisy neighbors actually taking over of a cluster with the lack of resource quotas, which can be a big problem because even system pods running on the same cluster might become unresponsive due to noisy neighbors or controllers, operators. And effectively, this will break your cluster or make your whole cluster unresponsive. So you have to be careful with this. Talking about storage, couple of interesting examples. The first one is a bit more trivial, probably. If you don't have any kind of limit on your PVC sizes, you might end up with filling up your storage very quickly, which might be your disk if you're using like local storage drivers or even like cloud storage. And this might actually also be like a billing problem if you're using cloud storage. Another interesting thing which is also found in the Kubernetes docs is that secrets themselves can get pretty big. So you might want to have limits on the number of secrets. You can instantiate per name space because that's also a possibility for creating many, many secrets so much that you exhaust the node storage, which will once again result in rendering the cluster nonresponsive, which might make it non-functioning. Talking about limit ranges and here in this context, there's one particular example which is also interesting with regards to misconfiguration, which is that if you don't set your limits right, you might end up with causing denial of service for yourself, for all the pods whose request for a specific resource like Compute exceeds the defined limits. So if you define limits, be mindful of the actual load of your application, right? And yeah, the final thought about this is that all I've been talking about is already scary if you're running your cluster for your company, but if this is like a multi-tenants scenario, so these are different teams, these are maybe different organizations, this can get even worse. So you obviously will end up with breaking down like the isolation model within Kubernetes, rendering other tenants nonresponsive, which will definitely be a bigger issue in this context. So this is just an example, this is just a subset of threats that depend on namespace misconfiguration. So what can we do about it to make this better for us? And the good part about this is that we don't need to buy fancy software to solve most of this. This is the important bit. So first of all, I talk about this all the time as I'm doing security all the time is, least privilege or back is still probably the most important mitigation you need to put in place. So you should carefully understand if your use case requires any form of access to namespace objects, read-only namespace objects should be the default if you need access to namespaces unless your application or operator or whatever actually needs to modify them. This also means, and we have a quote here, very famous quote, this means that you should not have wildcards as much as possible on resources and verbs when defining roles in a cluster. You should also be careful about cluster role bindings. We haven't really talked about that a lot because cluster role bindings effectively only apply to cluster roles. So this will grant you some level of access on global resources most of the time and if an attacker has any access to them, this is a likely point of escalation because they will try to move towards namespaces that hold most sensitive data in a cluster. Final remark on this, regardless of you allowing changes to namespaces, you should log them or detain them regularly. So is it safe to actually grant read access to namespaces? Well, kind of. As we've seen so far, namespaces can still leak a lot of information which is not sensitive per se but might be useful for attackers. Like if you can list all namespaces in a cluster, for example, you might understand that there are specific cluster components running or that there are specific applications running and we know that namespace names are also part of the DNS names of those applications. Obviously, if you can get an understanding of how a namespace is configured, you understand if there's a pod security standard level attached to it or if there are any custom labels that might be used by attackers. Yeah, one final remark I have about namespace selectors is that, and we just touched on this in one of the earlier slides, is that whenever you have namespace selectors, you should be looking into using the Kubernetes Ion metadata name immutable label. This explicitly requires the owner of the resource which has a namespace selector to mention the namespaces which are allowed to access that resource by name. And obviously this might be a bit more maintenance effort because you cannot use just whatever label, but at least this cannot be modified or deleted by attackers. So a lot of the attacks we've seen before might be mitigated by having this approach. When talking about multi-tenancy, soft multi-tenancy, our recommendation is that namespaces shouldn't be really managed by the tenants. So we look like into internal development platforms nowadays and a lot of resources can be self-provisioned by these platforms, but namespaces shouldn't really be self-provisioned and actually tenants shouldn't really have a say on the labels which are applied to these namespaces or even the names that are applied to these namespaces. This should be assigned by the platform admins if possible. And if you want to enforce these or if you need, for example, your tenants to have some access to namespaces, that's where additional controls can be implemented through policy engines. And yeah, I'll be very quick about this. This is a, you will see now a number of examples about policies. These are all Qiverno policies. There's no particular reason for that. Opagate Keeper obviously is another important one, but what I'm going, I want to show here is that you can implement fairly easy policies which can still provide additional restrictions on what your tenants or users can do on the namespaces. And like this example here, for example, will force, well, will forbid any change to labels into namespaces which are named as like prod-something. This is like a naming convention for this example for all production namespaces. Another example I have here for you is with regards to the top-level domains we talked about earlier. You could implement a policy around that which will basically just not allow the creation of namespaces with those identifiers. Interesting thing about this is that I downloaded the full list from IAEA and A and I created a policy about this and it got like the YAML file got like 60 kilobytes long. So yeah, kind of a big one. But it works. Final policy I have for you, I haven't really talked about this but this is an additional safeguard. You might want to disallow the creation of a cube-something namespaces just as a safe measure against attackers trying to hide themselves in your cluster. Okay, so you might not realize that but that's actually something which is quite important. So yeah, we finally reached the end of this presentation. Key takeaways for you. I strongly believe like Kubernetes namespaces have grown from an optional feature as the documentation says to actually a cornerstone for several security features in our cluster. So although they do not provide any isolation by themselves, a lot of security functionalities are based off them. Misconfiguration is a real thing and therefore you should restrict what users can do and you should audit changes to your namespaces in production. List privilege is the most important mitigation alongside the use of immutable labels which sounds like a trivial thing but actually isn't and this is all for free. If you then need to implement additional safeguards and yeah, this might be more relevant, for example, for multi-tenancy scenarios, you can leverage policy engines which although complex can help you define these additional safeguards which you can't implement with our back alone. So with that said, thank you very much. And...