 All right, I think it is about time for us to get started. Thank you for making the track out to the event center. I know this is probably the farthest room away from the banquet hall. This is the SIG-Auth deep dive. So I'm Mike. This is Margo. Hello, I'm Margo. I'm a software engineer for VMware Tanzu. Which is part of Tanzu's authentication sack. And I'm Mike. I'm a software engineer at Google. I've been working on GK and Kubernetes for almost seven years, actually over seven years now. And I am a co-chair and TL of SIG-Auth. So this afternoon, I'm going to give you a little bit of an intro to SIG-Auth, walk through, review what we did over the last year or year and a half, and then highlight a couple enhancements that are underway that I think are interesting. And then I'm going to hand it off to Margo. She's going to walk through an implementation of a client go exec auth plugin to give you a feel of how to extend human control client authentication. And yeah, and then we'll wrap it up and go to questions. So SIG-Auth, what we do, this is our patron XKCD comic. The Kubernetes community is very inventive. They're doing a lot of cool things all the time. I consider SIG-Auth's responsibility to make sure that operators and security administrators are able to draw, I guess, guardrails around what engineers are doing. Examples of maybe auth relevant features in Kubernetes are new flags in the security context of a pod that an administrator might want to control, maybe cross namespace, PVC references, where this is the volume claim references, maybe taints and tolerations that are used for node isolation. So this is our mission statement. So SIG-Auth is responsible for features in Kubernetes that control and protect access to the API and other core components. This includes authentication and authorization, but also encompasses features like out logging and security policy. So this is on paper what the SIG-Auth mission statement is. Over the last many years that SIG-Auth has existed and been meeting, we've adopted ownership of various things that don't really fit neatly into this definition. And we can talk about those later. If you aren't aware already, there's a newly established, as of in the last year, SIG called the SIG Security, who might take over various things that have fallen on the SIG-Auth community in the past. So this is a full list of sub-projects. On the left side, these are sub-projects that are in Kubernetes Core or Codebase, so github.com, Kubernetes, Kubernetes, audit logging sub-system, all authenticators, authorizers, including RBAC, ABAC, Node Authorizer, certificates API and related infrastructure. So any entry signers or approvers for the infrastructure that Kubelet uses to do certificate rotation for both its server and client certificates. Encryption at rest, so that would be the KMS plug-in for encrypting data stored in NCD at rest. Node isolation and identity. So the authorizer and admission controller responsible for preventing cross-node escalations. Policy management, such as RBAC and also pod security admission. And anything Kubernetes service can't relate it. So out of tree, we have kind of a multi-tenancy working group. And they have kind of incubator project. Out of that incubator came the hierarchical namespace controller, which is a project that is focused on synchronizing policy across a hierarchy of namespaces. And the secret store CSI driver, which is a CSI driver that integrates with various secret stores, I think Azure, GCP, AWS, and Vault right now all have plug-ins. And they expose kind of a secrets API, but rather than having the secrets be stored in NCD, they are stored in your secret manager of choice. And coming soon, so this is a recently, I guess, currently going through induction sub-project. So the Kube RBAC proxy is a pretty widely used, popular project. And we have recently decided to make it an official Sega sub-project. Not sure if I said it, but at any point, just raise your hand if you have questions. So what have we done in the last year and a half? So pod security admission is GA in 125, which is the upcoming release of Kubernetes. We also just recently merged APR to delete pod security policy from the Kubernetes code base. So that's a huge accomplishment. Certificates API went to GA with a couple new features, specifically like signer partitioning. The exec auth plugin, which we keep control credential providers, are now GA. That's what Margot will be speaking on later. Token request is GA, and Secret Store CSI driver is GA. So looking forward, I have a couple kept stylite. The first one is going through review right now. It's something called trust anchor sets. So right now, various things like Kubernetes itself, Istio, there's a certificate manager need to have a common need to distribute certificate bundles to all pods in a cluster. Right now, the state of the art to do that is a config map, since config maps can only be referenced by pods in the same namespace as the config map, you end up synchronizing config maps like that have the same data across many namespaces, has presented a scalability issues in clusters with many namespaces because that data is stored in SCD and has to be scanned and read into QBPi server memory. So I would think of this trust anchor set as a global object so that that data can be stored once and referenced across the entire cluster. If you've ever written a dynamic web hook configuration, you also have to embed a certificate for QBPi server to establish a connection to the web hook. So these will also be referenceable, and you won't have to do that. I know that the current mechanism is gross and not super portable. It's hard to write Helm charts that have dynamic web hook configs for that reason. The next one is KMSV2. So this is the extension point that is used for encryption at rest. The current KMS extension point has been alpha for a very long time. It had over the years of working with it, a couple of issues have come up. It is hard for an administrator to understand what the storage version of a secret that might be encrypted is what key. It was encrypted at the health check right now that QBPi server does on the KMS plug-in is to do it. Encrypt, decrypt, it's not actually potentially very thorough. And there is a fundamental scalability limit with the current model because right now when QBPi server starts and it loads its secret cache, there could be 10,000 secrets, it will serially decrypt each one of those. And each one of those has to make a round trip to the KMS provider, which could be a remote service that's expensive. So this enhancement plans to solve all of these problems. And we will be introducing a V2 alpha 1 version of that KMS plug-in API that should address most of these major issues. And so as I mentioned, token requests and bound service count tokens are now GA. So I think as of 1.23, rather than getting a secret backed token in your pod, you will get a bound service count token that is rotated by the Q-blit. It's time-bound. It's audience-bound. So while that is awesome, those secrets are still, those old service count token secrets are still created, which is not a great thing. But we can't just turn off the token controller because we expect that some users are still using, taking those tokens and maybe putting them in Jenkins or have action to authenticate to the QBPi server. So we are planning on trying to stop auto creation and remove unused tokens basically by trying to detect whether those tokens are in use somewhere else. And with that, I will hand it over to Margo. Hey, everyone. So today I'm going to be walking you through a sample implementation of a Client Go credential plugin. So what is a Client Go credential plugin? So it's a way for Client Go to call out to an external command to get user credentials out of band. So the plugin can implement some kind of protocol specific logic, like for example, an LDAP login or an OAuth login, and then pass back opaque credentials for Client Go to use. And by extension, QPCTL to use typically. Usually then on the server side, that requires a web hook of educator to interpret the credential to take that opaque token and make it into something meaningful for the Kubernetes API server. So when a user executes a QPCTL command, they're invoking the QPCTL binary, which is then in turn executing Client Go. Client Go takes that QPCTL command and makes a request to the Kubernetes API server based on that command. When you add a credential plugin into the mix, before you make that request to the Kubernetes API server, Client Go invokes the credential plugin, which then does some out-of-band authentication and returns a credential to Client Go. And so this is kind of a nice way to extend Kubernetes because Kubernetes can't necessarily natively support every OAuth protocol out there. There's just a lot of different ways that people do things, SAML, LDAP. And so people can bring their own out-of-band authentication and still be able to use that with Kubernetes. And so then once that credential plugin returns the credential to Client Go, it gives us this opaque token. Client Go makes the request to the Kubernetes API server, and now it has a bare token attached to that request. And then after that happens, the Kubernetes API server is typically set up with a Webhook Authenticator, or maybe an OIC Authenticator. The token itself is opaque to Kubernetes. So then Kubernetes makes this token review that the Webhook Authenticator is then waiting to see, and it sees that this token review has this bare token, and it's able to interpret the token so that the Kubernetes API server can use it for things like access control, take this user info that is now interpreted to tell us the username and use that for whatever it needs. Typically, the way this is set up is your user has a kube config, and that kube config has the exact field. And within that, it's calling some command. So this is a patch of some binary that exists on the machine. It's got an API version. It's got some arguments that are called with the command that you're calling. And then it's got some information about whether this needs to be interactive, so whether when kube CTL is invoked, it needs to be in an environment with a TTY. And it's also got an install hint. So when you call this binary, if it's not found in the user's path, it's just a little helper to tell them maybe where to go to install it. So I'm going to walk through the simple plugin now. This is based on my experience with Pindiped, the project that I work on. But it's pretty simplified. So this one is written in Go, but you don't necessarily have to write the plugin in Go because the inputs and outputs are all through standard out. So if you wanted to write a credential plugin in any other programming language, that's also totally possible. This one is going to be done using the OIDC password grant. So it's going to take the user's username and password from the CLI. And then it's going to take some information about the OIDC configuration. It's going to get an ID token, and then it's going to package it in an exact credential, which is the format that Clanko expects to see this credential so that it can send to the Kubernetes API server. So the entry point for this is just going to be a pretty simple Cobra command. So this is just a pretty common way for Go command line utilities to be created. It's going to have two flags. Both of them are going to be required in this case. And when you run this command, it's going to run this command called run OIDC login, taking those flags and passing them in as parameters to the function. Another thing that's typically part of that initialization is the Kubernetes exec info. So when Clanko invokes a plug-in, it passes some data via the Kubernetes exec info environment variable. So this data comes directly from the Qt config, and it contains stuff like information about the target cluster. So this should be useful, for example, if you have a cluster or some educator on your cluster that's looking for a specific audience, and that information is based on which cluster it's running on, then in order to allow your credential plug-in to craft the token correctly, it needs to know some information about the cluster. So another important thing to think about when you're creating one of these credential plug-ins is caching. So we need to cache credentials so you don't have to fetch the same token every single time you make a Qtctl request. These credential plug-ins are invoked every time that you run the command, they're short-lived. So maintaining that state somehow so you don't have to repeat yourself is pretty important. This is going to be stored on the local file system. So just like within the user's home directory, they're going to have some config.yaml file, and it's going to be keyed off of the arguments sent to the command. So in this case, it's going to be the issuer and the client ID. So if you issue the same command with the exact same arguments, you should get the same token back from your file system. So this is what we're going to do to get the file from the cache. So we're going to create this cache key based on the issuer, the client ID, and the scopes. So some of this is pretty OIDC-specific, but in any case, you want to have some way to keep track of how it was called. It's going to get the token in this cache.cat token. It's just a helper that gets it from the local file system, parses at yaml. And then if there is a value in your cache, then it's going to get it. It's going to check whether it's expired. If you have a cache, but it's been like several days or something since you've used it, then you want to make sure that you're not just passing back this out-of-date cache token just because you have a value in the cache. So you're doing that check. And then if that's the case, then we're good. We don't need to redo a network request to your OIDC identity provider, or in the case of a different identity provider than that out-of-band authentication either. And then at the end, if you did have to get a new token, then we'll come back to that. But at the end, you can put the token back at the cache key so that the next time the process is invoked, you can get it back out, and you don't have to do it all over again. Another important thing that you may need to do as part of a credential plugin is reading information from standard in. So maybe use my name and password. So if you do want to read from standard in, you can set the interactive mode in your Kube config. So that can be to if available or always. In the case of always, if you do not have a TTY available and these are not going to be able to type something into standard in, then the Kube CTL command will immediately fail, and it won't even try to invoke the plugin. If you don't want to do interactive mode, there's other ways you can get some information from a file or from environment variables. So there are other ways that you can have one of these plugins that don't involve interaction. So I'm going to actually try to do this in my IDE. So bear with me while I try to switch into GoLand, sort of. We had technical difficulties with this before the talk, so. Yeah, we tried to figure it all out, but. Yeah. All right, I feel less bad now. There we go. OK. Figure font. All right, cool. Yeah, perfect. Zoom in. Yeah. Even better, Command Plus, maybe? Sorry. I don't know why Command Plus isn't operating like I expected to. So is this, I could try to, if this is super unreadable, try to do it via. We should have caught this before. Yeah. Command Shift S? No. Close. Do you want to just talk through it verbously? Yeah, sure. I could just, yeah, try to keep it. So this getUserNameAndPassword function is, in this case, it's for OIDC password grant, so it's going to take these values, and then it's going to pass them to an OIDC identity provider that's going to give us a token back with some information about the user. This function actually works both in interactive and non-interactive environments, so there's a way to get it from an environment variable. For example, if you have some machine identity and you're trying to run something in your CI system and you want your server's account to authenticate, you can just pass these environment variables through. But if you have a human user who's just trying to log in, then we're going to prompt them for value. So that here is, it's getting, calling OS.getN under the hood, and then it's running this prompt for value there with me because this is a little bit indirect for testing reasons. So we can do dependency injection. But the actual implementation here is this prompt for value function is first checking whether we have a terminal. It's printing out a prompt saying, like, enter your username here, basically. And then it's creating a channel with the result of the user typing in their username. And then in this separate process, it's waiting here for the user to type your username and then press Enter. And then it's passing it back to that channel. And at the end of that, it's just going to trim spaces and return the value once it gets that result. There is a little bit of a different way that you have to handle passwords from usernames. You can imagine people, when they're typing their username, they want to see it as they're typing it. But that's not what you want for passwords. You want that to be hidden as you're typing it. You don't want to just see your password in plain text as you're writing it. And there's actually a built-in Go Standard library function called readPassword that hides local echo so that you don't have that displayed as you're typing it. And other than that, it's kind of similar. Time check. I'm going to attempt going back to Firefox. Command Shift F. And then hit Slide Show. Beautiful. All right. Yeah. And then the last step here is we're going to package the token into an exact credential, which is the format expected by Client Go. This is using Client Go's libraries because it's in Go Lang, which is pretty handy. We take that token that we get back, which I'm kind of going to hand wave over because that's pretty specific to OIDC. And you'd do that differently if you were using a different type of auth. But yeah, so you take that, and then you serialize it and send it to standard out. And Client Go can then do all that stuff to send it to the APS over. Awesome. Thank you, Margo. So yeah, I wanted to do one more highlight. So while what Margo just talked about is a GA feature of Kubernetes, we actually have some future plans for it. This is the last highlight. So Client Exec Proxy Auth is currently a proposal. That is under review. It allows for richer support for authentication protocols, such as request signing protocols, for example, AWS before, or Kerberos. And I kind of think of it as the kind of part of authenticating proxies, which is a model for extending Kubernetes authentication that some users use. And that concludes our content. So these slides will be posted on the website, if you guys want to click through on any of the links that you saw. But so we have a Slack channel. If you want to get ahold of us, Sigoth, we meet every two weeks, Wednesday, 11 Pacific time. We have a mailing list that you can join and a really long agenda doc. Feel free to drop an agenda item on at any time. Join our meeting. We love having new people and a lot of exciting stuff going on. So with that, I think we have five minutes for questions. Can you hear me now? Yep. All right. So one question about the credentials authentication. So I've been using it for a while now. But I've been experiencing issues when it comes to the authentication failing. So in my case, I'm using another CLI in the background to grab the token and Azure CLI in this case. And when the Azure CLI needs to be authenticated, you need to do that out of band. And this means that if I do fail this binary, the client executable will try to run it multiple times. So it will try to run five, six times in a row. And there is no way to tell it that you shouldn't retry. You should just stop and wait. Do you understand the question? Yeah. I think I recall this GitHub issue. So the idea was initially to on 401 to re-exec the plug-in. And keeping track of multiple clients within a process has been a problem. But yeah, basically, we try to fix those as we find them, like people creating two clients with two authenticators. But sometimes those re-creep in. And we haven't figured out a holistic solution for it yet. So just follow GitHub issues. And yeah. All right. Thank you. Any other questions? Kevill. Hi. Thanks for the good talk. I have a question about the trust anchor sets and kind of like a high-level question. So I understand that this is also going to be used to distribute CAs for TLS between workloads. So I'm kind of interested why does it make sense to include this entry? And like in general, how do you maybe decide? Because for upstream, there wasn't really functionality that much in upstream for workload. TLS, I'm interested whether you already have a model for which bits it makes sense to include entry and what should be out of three and why? Yeah. So that's a great question. So I think if we were trying to accomplish this for just, for example, like cert manager, maybe a CSI driver makes sense because then somebody can accomplish that out of three. I think my main motivation for implementing stuff within the Kubernetes code base is one to make use of it in core Kubernetes. So we have cert bundles that we are distributing to every single namespace. And that does present a scalability issue with Kubernetes itself. So that motivates me to have an entry. Also for an entry API to reference it, it really needs to be at least the API defined within the Kubernetes core code base. So yeah, so I guess at a high level, if we want to use it from core API groups or for core functionality of Kubernetes, we usually try to implement an entry. And then otherwise, definitely prototype outside until we find some rationale to bring it in. Maybe the other motivation might be if everybody in the world is using a specific out of three thing, then it probably makes sense to bring an entry. Yeah. Hey, thank you for the nice presentation and the demo. I'm very excited about that because I'm working on something similar at the moment for 2FA especially because that's something also a little bit, finicky topic on the command line usually. Do you have a repo or some docs available somewhere with this kind of sample code that you just showed? Like also the whole workflow with both on the client side and then also the server side, maybe what needs to be done there and what needs to be set up? So a lot of the client side code is taken from the project that I work on, which is PIDAPED, although I should warn you there's a lot of other complicated stuff that's hidden away there. I don't know if you have a better answer for some of the sample of a web hook authenticator and that side of things. So that link on the bottom is the cap. It's actually fairly thorough documentation on the design of these plugins and how we expect them to work. I think we should probably also add to this slide a link to the code that you walked through, which is on GitHub. And then we will post these slides on the schedule after this. So yeah, one other. OK, we're out of time. So if you have other questions, just come grab me. I'll be right outside. Awesome, thank you so much, everyone.