 My name is Nathan Brahms. I'm one of the co-founders of a company called Pzero Security. I'm joined today by my co-founder, Shashwat, another founder of Pzero Security. And today, we're going to talk to you about how you can help add scalable user authentication to your Kubernetes clusters using a technology called OpenID Connect. So in today's talk, I'm first going to give a little primer on how access in Kubernetes works. I'm going to break down access in its two constituent parts, authentication and authorization. Most of today's talk is going to be consumed with authentication. I'm going to talk about what you might want to consider when you're adding authentication to a Kubernetes cluster and what the trade-offs are that you have to think about. It's probably not a big surprise, but one of the options I'm going to advocate that you highly consider is OpenID Connect or OIDC. I'll then talk about what's involved in adding OIDC to a Kubernetes cluster. As part of this talk, we're going to give you access to a public open-source repo with some Terraform templates that you can use to set up OIDC in your own Kubernetes cluster. And I'll give you a demo of setting up OIDC in a live cluster using those Terraform templates. And then finally, at the very end, we're going to have a few minutes where we can talk about how you might think about scalability for authorization. Cool, so let's get started. So let's talk about access in Kubernetes. And we're going to start with this simple command you might want to run in a Kubernetes cluster, delete a pod. You're going to use kubectl, the client library. What does kubectl delete pod do? Well, kubectl delete pod is really just a thin wrapper around a web request that's sent to the Kubernetes API server. This web request looks like this. It says delete, blah, blah, blah, pods, blah, blah, blah. And then it has an authorization header that contains some sort of credential. And that credential is going to present who is making the request. So what does Kubernetes do with this web request? So Kubernetes has to decide whether or not you can execute this web request. And in a very oversimplified pseudocode, it does something like this. First, it takes that authorization credential that you passed in the header and tries to figure out who you are. That's authentication. Kubernetes doesn't have one way it can do authentication. It has several. And you, as a Kubernetes maintainer, can choose which of those methods you want to use. After it's figured out who you are, it'll figure out what you're trying to do. It will look at the HTTP verb. And the path. Again, oversimplification. And try to figure out what it is you're trying to do. And then we'll pass that to its authorization logic to decide whether or not you get to do this. So let's start with what you might want to think about when you're setting up this authentication method for Kubernetes. So you pass a credential. Who are you? So one thing you might want to consider when you're implementing your authentication method is who can get access to that secret credential? So if that credential is stored in clear text on either the client machine or your Kubernetes server, it's vulnerable to leakage. So we want to avoid any sort of authentication method, which stores these credentials in clear text on either the Kubernetes server or the client machine. Next, if you're a Kubernetes maintainer, raise, actually, show of hands. How many people here manage more than one Kubernetes cluster? OK, great. That's probably why you came to this talk, right? But if you manage more than one Kubernetes cluster, the last thing you want to do is also manage more than one authentication source for each of those Kubernetes clusters, right? Instead, you'd much rather have a single source of truth for authentication. In fact, if you're in any sort of larger organization, you probably want that authentication source tied to your organization's identity provider. When Kubernetes gets that information about who you are from its authentication method, you may want it to just say more than a name. You may want it to give other details about that identity to use those details when making authorization decisions. For instance, if you're a member of a group, that may be useful to figuring out whether or not you're permitted to take some action. And then finally, we should think about the effort it takes both to set up the system and maintain it. And ideally, you want to reduce both the setup effort and the maintenance effort, small enough that you can sit on a beach. OK, so now I'm going to list the various authentication methods that are available in Kubernetes. And I want to point out this is a complete list, because you can just implement your own whenever you want. So I'm not going to count that. And I'm going to try to score them on each of these concerns. So this is an opinionated scoring. And we can argue about it in the post-comment debrief after the talk. But you'll notice that a number of the methods just suffer from a security problem. So let's not talk about those. Let's instead talk about the last three on this slide. So webhook or proxy systems are essentially a way you can offload authentication to another system that's not Kubernetes. Whether or not those are actually secure depends on how you implement them. There's a number of open source offerings you can use. But they all require a lot of effort on your part, both to create and maintain. One thing that's not by default included in Kubernetes is a provider-based authentication method. But you probably have it in your Kubernetes cluster if you're deploying your Kubernetes via a cloud provider. So for instance, if you're using EKS, Amazon is going to give you authentication. However, these provider-based methods suffer from a scaling problem. Specifically, if you're using AWS, the identity you're getting is different on a per-account basis. And if you're using multiple cloud providers, obviously, you're getting different provider authentication for each cloud provider. So that kind of leaves us with OIDC. So OpenID Connect is a technology that is probably going to be bundled into your identity provider. And it's going to provide us the security we want, in fact, guarantee things like multi-factor authentication, and also an ease of maintenance. If I want to update the authentication system, I just do that in my identity provider. It's probably actually done by, say, an HRMS for you. But how easy is it to set up? We'll get into that and get into how to make it easier. So let's talk about what's involved in setting up OIDC for Kubernetes. This is a graphic I've shamelessly stolen from Okta. This is all the things you have to set up to make Okta OIDC work with Kubernetes. So if you're as excited, go do this right now. Come on. All right, so let's review the major points here. So in the upper left corner, you have your Kubernetes cluster itself. Your Kubernetes cluster has to know that you're using OIDC and who the OIDC issuer is in order to verify identities. You have to create an OIDC application within your identity provider. In addition, you have to configure every single client environment to use OIDC. So this seems really intimidating, right? This seems like a lot of work. But we can make our lives easier using Terraform. So the next steps here are going to assume you're using Terraform yourself. But if you're using something related, you can apply similar principles, right? And so what we've done at P0 Security is create an open source repo called, it's a very intelligent name, Kubernetes OIDC, which is a bunch of Terraform templates that you can crib and use for yourself. And so let's do that ourselves now. Let's do a demo. So here is the repository. We're going to start by cloning this thing. So I'm just going to do that right up here. You know, when I practiced this demo, I never was holding a microphone in my hand. OK, so all we have to do is follow the read me in this repo. And I want to point out I've done two specific steps already beforehand. One is I've configured my Terraform provider, so I'm not displaying my credentials to you on the screen. The other is I've run Terraform apply already because I don't want you to sit and wait and watch Terraform apply happen. But everything else will be live. So step one on this read me was adding those credentials, which I've already done. So I'm going to go straight to step two. And so all we're going to do in this other window right here is set some environment variables. You know, maybe Shashua, you can hold this microphone. So the first thing I'm doing here is just saying, hey, where is the Terraform repo that holds my Kubernetes config? One thing I forgot to say is this is a EKS Kubernetes installation. And the other thing I have to configure is which identity provider is going to host my OpenID Connect application. In this case, we're going to use octa. And then all I'm going to do is copy these templates into my Terraform repo. So once I've done that, if I just do a git status in this repo here, you can see I now it's in Terraform stuck in here. And now I need to configure my Terraform repo to use these templates. And I'm going to follow the AWS steps because I'm using AWS. And then in my Terraform, I just need to wire everything up. So the first thing I need to wire up is just I need to tell my new repo, or sorry, the OIDC connection in my Kubernetes cluster, what my cluster name is. And that's done already here. And then I have to import this new OIDC module that's going to create my OIDC application into my Terraform repo. Cool. Now that's done. And the next step would be to apply this Terraform. Again, I've already run this Terraform because I don't want you to wait eight minutes while Terraform runs. But let me just show you what it would apply on by running a plan. So this is going to do two main things. Remember, there are three things we need to do. We need to set up Kubernetes. We need to set up an OIDC application in our IDP. And we need to configure developer access. So this Terraform here is going to do two of those. It's going to configure Kubernetes. And it's going to set up my OIDC application in my IDP. OK, so let's say I ran Terraform apply, and that's run. Let me show you the OIDC application that was created. So it's going to make this application in my IDP. If you're using Okta, it'll look like this. And now we can just say, hey, who should have access to my Kubernetes cluster? In this case, I'm just going to give it to all my engineers. So there's one third piece we need to configure, which is the developer's environment. And to do that, what we've made is a generator script that will create a script that you can just hand off to your developers or any other person who's going to access Kubernetes. And they run it once on their machine, and it will configure their environment. And I'm going to give this to you again. Thank you. So now the script is running. I'll just make it executable. And now I can just say, hey, one of those machines I want to configure it on is this machine I'm doing this demo on, right? And I'll just run that right now. This is going to install a plug-in to my kubectl called kubelogin, which will do the OIDC login for me, and then set up my Kubernetes config. And so now I can run Kubernetes commands using OIDC. So now it's taking me to my IDP. Please don't steal my credentials as I type in this number I'm getting on my phone. That's actually OK if you do, because the only thing I have access to is this Kubernetes cluster. Now I can run Kubernetes commands. And Kubernetes now knows who I am, right? But of course, I don't actually have the ability to do anything, because we haven't finished the authorization piece. So let me show you what manual authorization would look like. First, I'm going to go back to my administrator Kubernetes context. And now I'm just going to give myself all those engineers role-binding in Kubernetes. Remember, we were talking about that authorization material that you might want to get from your IDP. So in this case, Okta is going to give me all my group memberships so I can use those. Now let me switch back to my OIDC context and run that get pods command again. And we're done. Now I have access. OK, in this case, I only gave myself a view role, because I don't want every engineer to just go and willy-nilly RCE into my pods. So I won't be able to delete one of these pods. So that kind of takes us as a natural segue to the next part of this talk, which is what the heck do we do about authorization? And I'll hand it off to Shashwet. Right, so what about authorization? So we all know that authentication is important, right? But if there's one thing we've learned from all the news in the last few months around Okta and all the different breaches that have happened, it's the knowledge that whatever you do, it's more than likely that some malicious actor is going to compromise an environment of entity. It's a pretty high risk. And why is this risk? It's because, well, at the end of the day, the cloud is a very complicated place. And especially in a Kubernetes land, the number of identities, both human identities and service account identities has just exploded out of control. So it's fundamentally a very hard problem to secure your environment only using authentication. And this is where authorization comes in. You want to be able to limit the blast radius in the eventual event that some credentials somewhere are leaked. You want to be able to control what that identity can do within a Kubernetes environment. And in most places, this is also a big part of compliance checks, especially if you're trying to go for a SOC2 audit. One very big pillar is that any identity should have access to only the things that they need for their job and nothing else, right? Or in other words, the so-called principle of least privilege. So that is why authorization is important. But it's also notoriously hard to implement in practice. Why is it so hard? A quick show of hands. How many of you all have just an admin role which gives engineers very wide scope of roles and privileges to your Kubernetes clusters and nothing else? So why do people usually go and give either nobody access or a few select engineers very wide-ranging access, right? It's usually two reasons. The first reason is that Cloud IM is very difficult to understand for most people because it's not part of their day-to-day jobs, right? Most people are application developers or platform engineers who haven't grown working on Kubernetes. So the knowledge of Kubernetes IM and more broadly, Cloud IM is something that is still limited to a very limited bunch of people. So it's not something that has been democratized just yet. And secondly, it's a more fundamental problem. There are just so many attack paths and the number of principles and resources in Kubernetes is just so high that it's fundamentally a very difficult problem to collapse all of these possibilities into a bunch of tightly packed groups which grant access to the right principles, right? So it's just fundamentally a very hard problem. So this is where we come in. Our goal at P0 is to make privileged access easy. And by the way, whatever we are going to show is included in our free tier, which should be enough for most small teams to get started. Some of the more common reasons that developers need access to Kubernetes, they usually involve rolling out a deployment, a quick show of hands if you have given your engineers the ability at some point to roll out a deployment on their own. Yeah, then as we saw in Nathan's demo right now, the ability to delete a pod, quick show of hands if you've ever had to give others access to delete a pod. Yeah, this one is very common to forward traffic from a port forward traffic from a specific pod. Quick show of hands if you've ever had to do that for troubleshooting a service. See a bunch of hands over there. And last but not the least, again, a very common use case, giving shell access or the ability to run kubectl exec, right? Again, I see a bunch of hands over there, all right. So again, all of these are very granular use cases. And the way most people get around this is by creating large, loosely defined groups with permissions like cluster admin and then giving them to developers. And then most importantly, not remembering to remove those permissions once their work is done. So now what we are going to show you is how P0 can automate the process of provisioning very fine grained access and also automate the process of deprovisioning that access once the developer's job is done. So imagine you're a developer and you, again, going back to the previous demo, what you require is the ability to delete a specific pod within Kubernetes, right? So you load up P0's Slackbot and you just add a few lines. You just fill up a few fields. You specify what resource, in this case, what pod you are looking to delete. Then you specify what action you are looking to take. In this case, we have created a curated role, which is delete. You just put in your reason, which is, hey, we need to run a demo. And then it kicks up a workflow where the resource owner in most instances, somebody on the platform or the SRE team or sometimes on the security team, they get an access request. And let's pretend for a moment that I'm on the platform team. I get this access request on my phone. I can just set a one-hour expiry, hit approve, and I'm done. So as a developer, Nathan now has access to execute just that one command, which is kubectl getpods, deletepods, and nothing else. So it's severely limiting his blast radius. He does not have cluster admin. He can just perform the very tightly scoped actions and nothing else. And of course, all of this is logged and audited on the back end so that if you want a paper trail to satisfy compliance and whatnot and security teams, all of that can be done as well. Awesome. Everyone, thank you so much. It was really good to be here with you. Please follow us on social or come to our booth. And we love to chat. Awesome. And I'd love to hear what questions people have. I think a common theme that we're running into is that we have a lot of configuration tools. And the actual configuration is spread out through a lot of different sources. And we have tools like Argo CD and Flux that are trying to make our configuration declarative. Does P0 security interface with that at all? Yeah. So the question is, how would this work if you're using something like maybe a GitOps flow that's making a declarative configuration? So I assume this is more about the authorization piece. I think if you wanted to do declarative part, obviously you just commit your terraform, right? But I think there's kind of two ways you can do that. One is you can kind of think of ephemeral access as being outside the GitOps workflow. And this actually makes sense because a lot of times you want this provisioning to be quick. You want it to happen in like a few seconds and also for it to fall into a break glass framework, like if your CD is down. But I think you can also think, hey, what if I really, really, really want to apply the principle of GitOps, right? There's a lot of shops where that's like a fiercely held principle in which case you're going to want to have the output of something like an access request here instead of being a direct modification of the Kubernetes repo, be a commit and pull request to your Git repo. The product we demonstrated doesn't do that right now. Well, what other questions do people have? Well, awesome. Oh, yes, right here. Hi, do you hear me? Great. Sorry if it's a dumb question. I enjoyed the demo showing how to secure the Kubernetes pods for your developer team. Does any part of post security address like end user authentication to your application? Yeah, that was a great question. So we talked a lot about infrastructure access, right? And now the question is, how would I do something about end user? Well, there's both authentication and authorization, right? So I think when we talk about what P0 security does, we're really concerned with the product we built with authorization because there's the assumption that there are these other great tools that already do authentication. So I think for application authentication, there are both open source systems you can use, right? Involving things like OPPA and commercial systems like Auth0. I think for us, building an SDK that someone can use to alter authorization is something we're really excited about getting into. Cool, folks. Well, feel free to grab us and chat. We'd be really excited. Thank you so much.