 So, hey everyone, I'm O'Conn. I work on identity solutions for VMware Tanzu. I have worked on Kubernetes for over five years now and have been co-chair for SigAuth for over three years. And I'm Margo Crawford. I'm an engineer for VMware Tanzu as well, working on Piniped, which simplifies the Kubernetes Auth experience. For working on Piniped, I've learned a lot more about the features and shortcomings of built-in Kubernetes Auth, so I'm excited to share some of those learnings. Cool. So in this talk, we're going to talk about some of the capabilities of Kubernetes Auth today and some of the limitations. Then we'll look at some workarounds that we did to work around some of the limitations. And then at the end, we're going to try to look forward and see what we could do in the future to make things a little better. So let's say you want to configure Auth in your Kubernetes environment, right? Seems like a pretty simple straightforward ask. So let's look at the cloud providers, right? Because this is generally the entry point for most Kubernetes users. Because the reality is, managing your own Kubernetes infrastructure is hard. Kubernetes does not handle that part of the problem. All right? So if you're using GCP, you're probably used to using the G Cloud CLI. If you're using Azure, you're used to using their CLI. And if you're using EKS, you're used to their CLI, right? So each of these platforms has their own integrated IAM solution, right? And this is totally fine if you're all in on a single platform, right? But what if you're not, right? What if you want to use a different identity provider, right? So traditional enterprises tend to have corporate active directory with their user identity, right? And it's likely that they want to continue to use that identity in their Kubernetes clusters, right? You tend to have a pretty high investment in your identity infrastructure. And it's pretty unreasonable to ask people to change that, right? And enterprises want the flexibility to use any cloud provider, right? After all, Kubernetes is meant to offer workload flexibility and mobility across the cloud platforms, right? But having a unified picture of your identity across the cloud providers is an important piece of that mobility. So if you're trying to figure out how to authenticate with something other than built-in cloud provider off, you might find that Kubernetes OIDC authentication is a nice option. OpenID Connect or OIDC is a pretty common protocol for authenticating with an external identity provider. It's an exception of OAuth 2.0. You can configure your Kubernetes clusters so that when you run kubectl commands, you can just include your ID token and the claims in the token will map to a Kubernetes username and groups. Unfortunately, this doesn't work in a lot of cases for a few reasons. A big one is that it requires configuring the API server. If you're self-hosting, this is something you can do by setting a few flags. But a lot of Kubernetes users don't have that option available because they're using cloud provider distribution and they aren't allowed to set API server flags at all. In contrast, Admission Webhook can be configured dynamically. So something like OPA Gatekeeper can be deployed by any cluster administrator at runtime in a cluster agnostic way that doesn't differ between cloud providers and self-hosted. So configuring the API server makes this option impossible on GKE and AKS. But EKS actually does have the option of configuring OIDC using their own proprietary API. It's not a bad option if you're using EKS clusters and you have an OIDC at any provider. But if you're configuring the API server, even if you can do that, you still won't have the option to update your configuration at runtime. For example, if you want to rotate OIDC client credentials without taking down the cluster. And of course, if you're using an identity provider that doesn't support OIDC, like Active Directory or LDAP, you're out of luck. Not to mention, this is just a building block. Kubernetes doesn't help your users get Jot tokens, it just validates them. Kubernetes has a bunch of extension points for authentication. So let's go through a few more. Starting with those that require API server configuration, we've got OIDC like we talked about on the previous slide. It works great if you have an OIDC compliant identity provider to work with. Another option is to create an authentication webhook. When a user provides a token to the API server, the webhook will call out to an external service to authenticate it. This is very flexible, but it requires you to write code and be responsible for that webhook. There are a couple other options that are typically enabled by default with common tools used to deploy Kubernetes, like QBADM and most cloud providers. One is certificates. You can generate client search for users based on the cluster CA. One problem with this approach is that certificates aren't revocable, but we'll talk more about that later. Service account tokens are another option, but as the name implies, they aren't meant to work with humans, they're for service accounts. So let's talk about what we would expect end users to need, right? So if we look at the bottom of the slide, we can see a kubectl command, and I would venture to say that many Kubernetes users expect something like this if they're trying to configure Active Directory in their environment, right? It's a custom resource-based API, and then you can use standard kubectl commands to try to understand the API structure and what configuration options are available to them, right? So at the highest level, right, they want to bring their own identity to Kubernetes, right? And it doesn't matter where that identity comes from, it might be Active Directory, it might be GitHub, right? They want to be able to use any identity provider that they have available to them, and they want to be able to do this even if they didn't stand up the infrastructure themselves, right, in the cloud provider environments, as a for example, right? They want to have a consistent identity across all environments. So if you're very familiar with Active Directory, you might be very used to using, like, user principal name as the username attribute in your Kubernetes RBAC roles, or perhaps you're an ultra-paranoid, and you want to use object good, but either way, you're used to a particular way of asserting identity across your infrastructure, and you want to keep doing that, right? So as we saw in the bottom of the slide, right, you want to make this stuff easy for people, right? They should be able to understand it in the same way they understand everything else in Kubernetes, right? So a custom resource-based API is really ideal for that, and you know, you want it to be a higher-order API, right, we're not talking about webhooks, we're not talking about tokens, right? We're talking about identity providers, because that's what people understand, right? And you want to keep all this stuff as safe as possible, right? So you want to limit the negative effects of an individual cluster compromise, right? So what that means from the perspective of an end user is there's no credentials that are being sent to clusters that can then be replayed across clusters, as a, for example, right? So if we step back and then instead look at what do the implementers need, right? So Margot and I are the pinniped authors, along with some others at VMware. So what do we need to solve these problems for folks, right? And it really comes down to a pretty simple requirement, right? We need some way at runtime to assert identity on a Kubernetes cluster, regardless of how the infrastructure was brought up, right? So the extension point that we need to integrate with has to be something that's always enabled, doesn't have any configuration, it behaves exactly the same way on every Kubernetes distribution, and probably most importantly, it has to be safe for this purpose, right? It has to be designed and built for this purpose. So that takes us to work around number one that we implemented. So a little background, X509 client certificate off is enabled on most Kubernetes distributions with the exception of EKS. There's an API for signing those client certificates called the certificate signing request. It's got a few problems though. A big one is that up until Kubernetes 122, when Mo added the expiration seconds configuration option, there was no way to configure the length of certificates, and the default lifetime was one year, and the certs aren't revocable. Usually when you can't revoc a cert, you'd want to keep the lifetime as short as possible so that if users shouldn't have access anymore, their client will expire quickly, and when they ask for another one, they'll be denied. The proposed workaround for when you want to revoc a cert is to rotate the whole CA, which is not ideal and breaks a lot of things. You can always go around the certificate signing request API, but it requires saving the cluster signing key pair somewhere and issuing cert space off of it. What we did is we steal your keys from inside your cluster and use them to mint shortlist certificates whenever we want. So this only works if you have access to the Kubernetes control plane, which often isn't the case on cloud-hosted Kubernetes distributions. But if we do have that access, we can look at the Kube controller manager pods, which are part of the control plane in charge of the certificate signing request controller along with some other controllers, and we can see which arguments the daemon was called with. This tells us the path to the signing key so we can grab it from the volume mounted to the Kube controller manager. Let's walk through how this works using Piniped. When you deploy your cluster, it's created with at least one Kube controller manager pod. That pod has the volumes mounted, including one that includes the cluster signing key pair. In this case, the one called Kate's certs highlighted in blue. The user can deploy Piniped or anything else using the same trick. The Piniped pod will read the metadata of the Kube controller manager to find the path to the cluster signing key. Then Piniped launches a privileged pod called the Kube cert agent and copies over the Kube controller manager's volumes. It also passes the path to the key pair as environment variables. Piniped can then ask the Kube cert agent for the key pair, which the Kube cert agent reads from its volumes and stored in memory. So when a user asks Piniped for a certificate, we can do our own authentication checks, then use the key pair to mint certificates of any duration we want. So as Margot mentioned, the Kube cert agent-based approach doesn't work in the cloud provider environments. Those are not self-hosted environments. The Kubernetes control plane isn't available through the Kubernetes API. We need a different way to handle those environments. Those are quite critical to us. So I've tried to make this like a happy rant instead of an angry rant. We'll see how this goes. So I'm pretty sure many folks are familiar with impersonation through Kube CTL. So you can see the command on the bottom of the screen. Someone's trying to get pods, but they want to run this command as a different identity. And then the curl command right under that sort of describes what is actually happening. So you're authenticating as some identity, and then you're asking the API server, hey, I want to act as some other identity. Do some authorization checks and let me act as that other identity instead. All right, so this was originally built in OpenShift. And when it was upstreamed, it actually lost some functionality. So OpenShift has a very strong concept of groups. So back in those days, when you impersonated a user identity, it would automatically impersonate all the groups that a user belonged to. That got lost when we upstreamed that years ago. So it's built with a pretty simple server-side implementation where the incoming request is processed. When the client sends these headers, the API server is effectively saying, all right, you're authenticated as Margo, but you're trying to impersonate Moe. I'm going to do some extra authorization checks to assert that Margo is allowed to do this action. And if so, I'm going to basically overwrite the user information on the request. And the rest of the request will run as if Moe had run the command himself. And most critically and importantly for us, impersonation has no configuration options, and it's always enabled in every environment. So let's walk through some use cases for impersonation. If you guys are familiar with modern IT principles, you no longer run as a privileged user unlike your Windows machine. You tend to run as a limited user. And whenever you try to perform a privileged action, you elevate yourself either with something like Pseudo or forget UACC, whatever Windows calls their thing. So you can do the same kind of flow with Kubernetes. You can be a cluster admin style user that purposely limits their own access in the normal day-to-day operations, but then they have the capability to impersonate a root-level identity to grant themselves extra permissions. So that's one common use case for impersonation. Another one is the I want to validate access use case. So you can imagine you're a cluster admin user and you want to make sure that Moe can't perform some action, or you just impersonate Moe and try to do the action, perhaps as a dry run or something, and you'll be able to see if he is or is not able to do that action. So how can we abuse this? We'll see. So let's walk through this example. Let's pretend you have like a magic box and the box can do something. And there's a user trying to get pods, and you happen to know that impersonation is a thing and you are ready to abuse it to the maximum of your capabilities. So what does the magic have to be for you to figure out what you need to do with impersonation? Well, generally speaking, that has to effectively be authentication. You have to know who is trying to make a request before you can impersonate them. So once you have that, once you've authenticated the user, now you know what headers to set. But you yourself need some identity to do the impersonation. And since you didn't know who was going to show up, you basically have to be able to impersonate any identity. So what you're talking about is some root-level account that can impersonate all other users in the system, whether they exist or not. Start making you nervous. So now we have sort of the building blocks of like an impersonation proxy. Now if we step back, what is the user seeing in all this? Well, everything behind the magic box is completely opaque to them. They have no idea that any of this exists. From their perspective, they're talking to the Kubernetes API. Just the difference is we've lied to them. We've told them to the Kubernetes API servers over here. Just ignore the real one over there. So like if we walk through a full example, you have a user, they're trying to get pods, they go to your reverse proxy, which authenticates them using whatever means they want to authenticate. And then it turns around and impersonates them on the backend to the real API server using its own root credentials while setting the impersonation headers. So in text, you're going to lie to the user and tell them the URL and CA bundle of the API server is the impersonation proxy. They're going to send all of their traffic to the impersonation proxy instead of the API server. And the proxy will authenticate them however it wants. And then it's going to reverse proxy that back to the API server with the appropriate impersonation headers and authorization header set. So what can go wrong, or sorry, what is wrong with this approach? Well, there's a lot of things. First of all, you're sending your traffic to the wrong place. So every time you type in like kubectl get secrets, all of the plain text of those secrets is flowing through this proxy. So you have basically made another root level component in your infrastructure. And this proxy now has to fully scale with all of the traffic that you pushed through it. So previously you had one API server and now you effectively have two because it has to linearly scale with all the traffic that's being flowing through it. Also, there's no way to prevent the user from trying to access the real API server. The Kubernetes default of service is available in every pod. So if they happen to try to authenticate to the real API server, it won't work because only your magic proxy understands whatever credentials it issued. The API server will very happily reply with unauthorized, which is a misnomer but it's an incredibly infuriating error message as a user. As an IT security person, you should probably freak out when someone tells you that there is an identity on the system that is allowed to impersonate all other identities because that's awful. Like this is asking for something to go wrong. And as I mentioned, the impersonation implementation was never designed for this use case. It was designed for the things I showed earlier. So imagine you're someone who's using Active Directory and you're in like a hundred groups or maybe more realistically, even a thousand groups when you turn around and do that reverse proxy to the API server in the inline request, the API server is going to linearly validate that that super user service account is allowed to impersonate individually each one of those groups. Now those are very fast checks, but it's not something that you want to sit there and do in the middle of a request. It's just not a sane approach. And probably most importantly is impersonation is the first class functionality of the Kubernetes API. It's built into Kube CTL and you've created a big problem for your users right now. How are they going to use impersonation if you're abusing it on the back end, right? Impersonation doesn't nest. You have to come up with some other strategy to allow users to keep using that functionality while you're trying to use it on the back end. So let's talk about how we implemented some of this. So the business logic of our impersonation proxy uses or abuses for a better term the API server aggregation code from the Kubernetes code. If you ignore comments and imports and that kind of stuff, it actually is very tight. It comes down to under 500 lines of incredibly dense and cryptic code, but it's not that much code. One of the things that we did because we didn't want to lose the ability to handle impersonation on the front end is we performed the same authorization checks that the API server does when you try to impersonate something, but we wanted to make sure that in the Kubernetes audit log you could always see all three actors that perform this request, right? So obviously our service account is the one making the request. The identity that you impersonated is the one that the request will be run as and then in the user info of the request and on particular key, we insert the information of the user that made the request to us. So if you ever look at the Kubernetes audit log, all three actors are maintained within that log. All right, Margot added the ability to do a UID impersonation in 122, but that's very, very recent in Bleeding Edge. And one of the core things that we wanted to maintain is one, we didn't want you to lose any functionality if you're using our proxy, but also we wanted to make sure that no information is ever lost through this layer. So service accounts happen to always have UIDs in their user info. So if you try to impersonate a service account, that information would get lost. So in this particular case, we would pass through the tokens for the service account. Maybe one day we won't have to do that when everyone is on 122 and greater. So I'm sure everyone's familiar with CubeCTL exec and Port Forward and all of those things. Those are incredibly important to the Kubernetes developer experience and we wanted to make sure all of that stuff was also maintained. So we fully support speedy and WebSockets and all of that stuff through our proxy. This is one of the few places that where we diverse from Kubernetes. We don't use the Kubernetes code to do this stuff because it happens to be the code that had the most critical CVE in the history of the Kubernetes project. So we just decided that it was better to heavily rely on the standard library instead of what Kubernetes does in this place. And of course, watches are super critical to the Kubernetes experience so we made sure those still work. So after all of that, does anyone think they can write a correct, safe, performant, and otherwise perfect impersonation proxy? But I'm going to say no. No, you cannot. And I have proof that you can't. So I went hunting, right? And there's this really cool project called Kima that builds a bunch of higher order functionality on top of Kubernetes like functions and serverless and all the cool stuff, microservice stuff. And for other reason, they use an impersonation proxy. And they forgot to validate some inputs. It didn't work out so well for them. If you don't know about Kima, I'm pretty sure you probably know about Mesosphere. And hopefully you updated it last month because they also forgot to sanitize some inputs. And the underlying proxy code they use from a company called Traffic, by the way, these guys are amazing. They had the most phenomenal security response I've seen from a company in a very long time. For them, it was a pretty trivial, low value CVE. And as they clearly state in their disclosure, that the expectation of such an issue is very unlikely because it requires that the removal of a header would lead to a privilege escalation. Well, it just so happens that if you remove impersonation headers, you get a privilege escalation. And if you don't know anything about those companies, I'm really willing to bet you know about Rancher. And they also got this wrong. So if we walk through the severity of this, we can kind of ignore the network bit, everything on Kubernetes is this network. It's not really significant. The complexity of these attacks is incredibly trivial. You're just setting some headers on the request. In terms of privilege, there's no actual privilege requirement. You just have to authenticate. So it's like the lowest possible privilege. And there's no user interaction. All right, you're just attacking the network proxy directly. So the only thing really left is the actual severity of these issues. And it just so happens in all of these cases, all the ones I described earlier, every single one of them led to a completely unprivileged user becoming full cluster admin on the environment. So if we think about this in the recent trends of ransomware, I can tell you that you have no confidentiality. The ransomware will export all of your data off your cluster to some S3 bucket somewhere. You will have no integrity because it's going to encrypt all your data. And you will have no availability because all your data is encrypted and also you're for us running Bitcoin miners or something. So the only thing that actually left is did you have a scope change? Well, I don't know. Maybe it got access to other cloud credentials and then your whole infra is destroyed or whatever. So maybe you did. So you're either talking about a high or a critical. Either way, your day is ruined. All right, so we were always very nervous about building this. So we had always designed this stuff with a bunch of security thoughts in mind. So we always set all the standard security headers that you want to set to make sure everything is safe. We always did every input validation we could think of to pedantic levels. If we thought for any reason that we weren't going to exactly impersonate your identity bit for bit, even if there was a slight case difference or something, we just failed the request. We were like, whatever. We'd rather not do anything than accidentally do something wrong. But after learning of all these issues in our ecosystem, we took some further steps to harden our implementation just in case that we had some mistake that was unknown to us today. So if you happen to use Azure Kubernetes Service, they're the main cloud provider that disables anonymous authentication. So if you run our proxy against their API servers, it will automatically detect this case and honor that configuration for you. We happen to have one API that's a pre-login API. So that one is passed through, but otherwise everything else is rejected as you would expect in those environments. We use a distinct service account to handle this impersonation stuff. So all it can do is impersonate. So if you manage to make us drop our headers, the request will run as a user with no permissions. So at least that's okay. Now, of course, if you somehow manage to trick us into running the wrong headers, then everything is sort of over anyway. Probably the most critical thing we do is we no longer rely on the API server to perform authorization checks for us. We pre-authorize every request using the subject access review API just to make sure that you're at least authorized to do the thing. We fully rely on the API server to perform all the admission checks that we needed to do, but on read requests and stuff, if we don't think you can make the request, we're not going to reverse proxy at all. So looking forward, we're hoping that one day these workarounds that we built are going to become obsolete. The ball is already rolling for certificates. So Mo added expiration seconds to the certificate signing request API, which means that we won't need our cluster signing key stealing hack workaround since we can request any link short as 10 minutes. This only came out in 122 though, which is super recent. So it'll be a while before it has widespread adoption and even then it won't work on EKS. And of course, it isn't a full solution. Even with the short-lived certificate, you still need tools built around it for logging in with your identity provider and refreshing certificates when they expire. These are all things that Piniped does and that you would need to do if you were implementing your own solution with short-lived certificates. So over a year ago, I had proposed the ability to have dynamic authentication configuration in the Kubernetes API. So that kept was rejected under the belief that it proposed a cluster configuration API and not something that should be built into the Kubernetes API server or set in a different way. The configuration belonged to the cluster operator persona and not the cluster admin persona. Of course, I disagree with this because I wouldn't have written the KEP otherwise. But I feel like there's probably some middle ground where we can try to find the good parts of impersonation without all the bad. So no proxies, no privilege escalation, no fail open behavior. So you could imagine maybe some token-based API that gives you identities from the cluster or something that either you set the token or you don't, but you can't break out of the bounds of some sign assertion of identity, as a for example. So to summarize, built-in Kubernetes auth is super flexible, but it's hard to configure and it's mostly not possible to configure it at runtime. We built some cool workarounds together on the pain points we saw by acquiring the cluster signing key pair from the control plane so we could mint short-lived certificates with it and by writing an impersonation proxy. And we're still working towards making our workarounds unnecessary. We believe that auth shouldn't be considered part of your infrastructure config under the control of the cluster operator is to be standard Kubernetes configuration under the control of the cluster admin. Having it be infrastructure config makes it hard to have unified identity across Kubernetes distributions. Pinniped uses a series of workarounds, but we don't consider them to be the safest approach to solve these problems and we think changes need to be made to core Kubernetes so it's easier and safer to build. We trust the cluster admin with near-complete control over authorization with Kubernetes RBAC and Admission Webhooks at runtime so we should extend that same trust to authorization as well. Authentication. Yes, authentication. If you're interested in learning more about Pinniped, we have learned the workarounds we talked about today and a full logging experience that lets you integrate with OEDC, LDAP, and Active Directory Identity Providers. You can read our source code to see how we did it on GitHub. You can also find us at pinniped.dev and we're active on the Pinniped channel in Kubernetes Slack where I have had you two licensed open source projects for anyone interested. Thanks for coming. Thank you.