 And okay, welcome to another CNCF live webinar, everyone. Advancements in Kubernetes workload identity for Azure. I'm Libby Schultz and I'll be moderating today's webinar. I'm gonna read our code of conduct and then hand it over to Anish Ramasakar and Ernest Wong, software engineers at Microsoft. A few housekeeping items before we get started. During the webinar, you're not able to speak as an attendee, but please add all of your comments and questions to the chat box. On the right hand side, and we will get to as many questions as we can at the end. This is an official webinar of the CNCF and is such a subject to the CNCF code of conduct. Please do not add anything to the chat or questions that would be in violation of that code of conduct. And please be respectful of all of your fellow participants and presenters. Please also note that the recording and slides will be posted later today to the CNCF online programs page at community.cncf.io under online programs. They're also available via your registration link and the recording will be on our online programs YouTube playlist on YouTube. With that, I'll hand it to Anish and Ernest to kick off today's presentation. Thanks. So yeah, I'm Ernest from the Microsoft Azure Container Offstream Team. We focus on developing cloud native open source software for our customers at Azure and welcome to this presentations. Hey, I'm Anish. I'm part of the same team. So we're part of Azure Container Compute Security Team. And then we maintain a couple of open source security projects. I think some of them that you've heard is Azure Workloader Entity, Secret Store CSI Driver and Kubernetes KMS. So yeah, in today's presentation, we're gonna touch on our project, the pod identity and workload identity. And we're gonna talk about pros and cons for each project. And Anish is gonna do a quick demo on workload identity. And last but not least, I'll also talk about the roadmaps and some resource that folks can check out after the presentation to learn more about the projects. Okay, so imagine you're an application developers on Azure and your applications wants to access some Azure cloud resource. You have a couple of options. First, you can create a surface principle which contains a client ID and client secrets. That's not very secure because your client secret can be leaked and you have to manually rotate the secret every once in a while to keep your system secured. The second options, which is what AAD pod identity is mainly about is to use an Azure Managed Identity. You can think of Managed Identity as a surface principle but without the client secrets. This is beneficial because users don't have to be concerned about leaking the client secrets because there's no more client secrets and the users don't have to manually rotate the client secrets anymore. And yeah, let's apply these concepts to Kubernetes. Let's say you're Kubernetes developers and on Azure and your Kubernetes applications wants to access some Azure cloud resource. For example, fetching a secret from Azure Key Vault or downloading files from Azure Storage. You can either choose the surface principle routes which is not very secure or the Managed Identity routes which is what we recommend. However, as I mentioned earlier, you can only attach Managed Identity on the VM level or your Kubernetes cluster level. And once you assign the Managed Identity to your clusters, all the applications within the clusters will have the same access to the same set of Azure resource which is not very ideal in some situations. You might want to restrict a particular applications to only be able to fetch secrets from a Key Vault but not be able to access some unauthorized resource. So this is where ADPot Identity comes in. Not only does it help developers assigned Managed Identities to individuals Kubernetes nodes using the concepts of Kubernetes CRDs but it also acts as a gatekeeper to make sure that our Kubernetes applications are using the correct Managed Identity that they're supposed to use. So here is a quick simplified architecture for ADPot Identity. So developers who wants to use those projects will start off by creating two Kubernetes CRDs, Azure Identity and Azure Identity Bindings. Azure Identity is a representation of an Azure Managed Identity. It contains the client ID of the identity as well as the Azure resource ID of the identity that you want to assign to your workload. Next, the developers will create an Azure Identity Binding which is a Kubernetes custom resource that binds an Azure Identity with a pod using label selectors in the pods metadata. Finally, the developers will create their pods with that particular labeled defined in our binding CRDs. And once the pod is created, ADPot Identity will assign the Managed Identity defined in Azure Identity CRDs to where the pod is scheduled. And once the assignment is complete, the pod can start accessing Azure resource with Azure SDK. And under the hood, Azure SDK will fetch an access token from our instant metadata service or IMDS. And that piece of traffic is intercepted by ADPot Identity and we'll make sure that the token request is valid. That is the pod requesting the access token is indeed using a Managed Identity that they're supposed to use. And with this validations, we can block any pods or workload that are trying to use a different identity that they're not supposed to use. And after the validation is complete, IMDS will return the token back to the pod and the pod can access whatever Azure resource they're supposed to access. And with this model, it was fine, but we do have some limitations that prevents us from scaling further. First, Identity Assignments operations generally takes around 10 to 40 seconds to complete. So that means applications won't be able to access Azure resource for around 40 seconds if the Identity is not previously assigned to their underlying Azure node. Second, this solutions only works on Azure. So Kubernetes applications running in other environments will have to use surface principles, which is not very secured as I mentioned earlier. Next, it only supports Linux workload and with the increase in developers running containerized Windows workload on Azure, we tried to prototype pod identity for Windows, but we didn't have any luck with it. Last but not least, over the past years, we have encountered some IMDS caching issues which causes some of our users to not be able to fetch tokens in their applications. And with that being said, Azure workload identity, the next generations of AAD pod identity will address all of these concerns that I just mentioned. It doesn't require Identity Assignments operations. It's cloud agnostic, meaning this solutions will work in any clouds and any environments including your local environments, and it also works on your Windows workload out of the box and it doesn't rely on IMDS to fetch access token. So how do we achieve all these? There are three main pillars that enable us to move forward with the new design, Federated Identity Credentials, Surface Accounts and Miltating Netflix. Let me quickly explain what each of these components means. Federated Identity Credentials helps establish a trust relationship between an external identity providers and an identity within our Azure Active Directory. What this means is that we can turn our Kubernetes clusters into an identity providers using a built-in features and once our Kubernetes clusters can issue tokens to our workload, we can establish a trust relationship between our Kubernetes clusters and an identity in AAD. Once AAD acknowledged this relationship, we can start exchanging tokens issued by our clusters for a valid AAD token and we can use the AAD token to access our Azure resource. So how do we turn our Kubernetes clusters into an identity providers? Well, the concept of identity in Kubernetes is not anything new. In Kubernetes, a surface account represents identity that workload can use. Kubernetes administrators can set our back rules to against the surface accounts to control what their workload can access within the clusters. Last but not least, Azure Workload Identity also contains one core components, which is our mutating admission webhook. The job of the mutating webhook is to intercept any workload admissions requests from the users and project the surface account token to our containerized workload file system and it also injects several environment variables as well. And by having a mutating webhook, users don't have to manually change their workload to manifest to support this new authentication flow. So let's look at how we glued these three main components together. First, the users will create a surface account for the workload in the Kubernetes clusters. You can think of the surface account as an Azure identity CRDs in AAD pod identity. Next, the users will initiate a request to AAD to establish a trust relationship between our surface account and the identity in AAD. What this will do is that if we give AAD a valid surface account token, it should return a valid AAD token back to our workload. And once this relationship is established, we can deploy our mutating webhook and as well as our workload. The webhook will intercept the admission requests that wants to use the workload identity project and it will automatically project the surface account token to the file system as well as a couple of environment variable. And our workload can now leverage our latest and greatest Azure SDK to access our Azure resource. The benefits of using Azure SDK is that it helps abstract the token exchange steps. And under the hood, Azure SDK will read the environment variables and the projected surface account token in our file system and pass those parameters to Azure Active Directory. AAD will then verify the authenticity of the tokens and if it is valid, AAD will return a access token back to our Azure SDK. And with a valid access token, we can now access any resource in a cloud. And yeah, as you can see, we're now able to eliminate everything we mentioned a couple of slides ago. Users don't have to wait 40 seconds for the identity assignment operations and this solution is cloud and OS agnostics. So it works in any environment. Last but not least, we don't have to fetch tokens through IMDS. Instead, we now directly query AAD endpoints to exchange surface account tokens for an access token. So I'm very super excited about this project. And it was a great collaboration with AAD AKS team as well as the Azure SDK team. And with that being said, I'll hand it off to Anish for a quick demo. Thank you, Anish. Okay, so we've been working on Azure Workload Identity for a while now and we're really excited to have this finally available for our users to try out. This is based on the new Workload Identity Federation, which is currently under preview. So as part of this demo, what we're gonna see is we're gonna see Workload Identity in action with Azure AAD applications. I'm a fan of live demos, but just for this purpose of the live webinar, what I've done is I've done a recording. So what we're gonna do is run through the recording and then I will talk through what is going on. So as part of this demo, what I've done is I've created a AKS cluster with this new flag called enable OIDC issuer. So this is a one node AKS cluster and it is running Kubernetes version V122. And I've already enabled the OIDC issuer on this cluster. So this OIDC issuer feature is currently in preview on AKS. When you create a new cluster or update an existing cluster with enable OIDC issuer flag, an issuer URL will be set up for your cluster. In addition to the initial issuer URL setup, AKS will handle the rotation of the signing keys for the cluster every 90 days. There will also be an option to perform a rotation of these signing keys on demand if there has been any kind of compromise in the future. So this is what my OIDC issuer URL looks like, right? And then the issuer URL is actually part of the AKS cluster properties. So that means you can query it with AZ-AKS show command. So when we do AZ-AKS show here, we can see this is my cluster issuer, which has the FKDN and basically a unique cluster ID for each cluster. So what exactly is on this issuer URL? So there are two things, right? The first thing is the OIDC discovery document. This is the standard discovery document which has the issuer URL. And then it also has the JSON Web T-set URI, the URL where the public signing keys are available. And apart from that, a couple of other fields which is required by the OIDC spec. And then now when we try to query the JWKS, the JSON Web T-set, we can see that this has the public signing keys that's used by the cluster. So the private signing keys are available in the cluster. That is what is used to sign the service account tokens. And when AD gets the service account tokens, it can validate that using this discovery URL and the public signing keys that's available. So the next thing that we're going to do in this cluster is we're going to deploy the mutating webhook that we talked about. It simplifies config for the deployments. For the purpose of this demo, what I have done is I have installed the mutating webhook using help charts from our open-source repo. But when this is natively integrated in AKS, you would be able to enable this with enable workload identity flags similar to all the add-ons that's available today. So what this does is it installs the mutating webhook in Azure Workload Entry System namespace. So let's take a look at what's there in that namespace. So as you can see here, what we have here is a deployment and a service. The service is configured for the Workload Identity webhook. The configuration for Workload Identity is done using the Kubernetes service accounts like we talked about. So as a user, all you would have to do is annotate the service account with the client ID of the Azure AD application or the managed identity that the pod needs to use to access Azure resources. And when this pod is created, the mutating webhook will be called as part of the admission request. Based on the annotation that is there in the service account, it will inject the environment variables that can be consumed by the SDKs. And also it will add the projected service account token volume. This means you can use the same pod spec that you're using in different cloud providers and you can just deploy that in Azure without making any changes specifically to run Workload on Azure. Workload Identity is based on projected service account that we talked about. How is this different from default service accounts? So projected service account JWTs differs from traditional tokens in that they expire. They have a proper issuer and then the audience field comes filled out so that they can act like proper JWTs. These projected service account tokens aren't synced as Kubernetes secret and they can only be injected into the pod as volume. So as part of this demo, the first thing that we're gonna do is we're gonna create an Azure AD application. So here I'm doing azadspcreate for our back. This is something that we all are aware of today. As you can see, there is an app ID and there's also a password. This password gets created by default but this, at least for Workload Identity, is never used. So you can actually get into the portal, manage the app registration and delete the password because this client secret is never used anywhere with Workload Identity. The next thing that we're gonna do for the demo is we're gonna create a namespace within the cluster. So we are going to call it WI demo app and the main part now that we need is to create the annotated service account. So in this service account annotations, we are using the client ID of the Azure AD application that we created in the previous step. And then in the labels, we have Azure Workload Identity use true. This is what tells the mutating webhook that this particular service account is going to be consumed for Workload Identity with Azure. So please mutate any part that uses this particular service account. The next thing that we're gonna do is we talked about federated identity credential. So we need to establish a trust between the Azure AD application that we created in the first step and then the service account issuer and a subject. So we're gonna tie all of it together and say AD trust, this token that I send, which has all of this. So there are a couple of ways to add this federated identity credential. You can do it with easy rest. There's also support for it coming soon with Azure CLI. And there's also support for doing this with UI. So for this demo, what we're gonna do is we're gonna do it with UI. So if you see here, this is the Azure AD application that we created. And if you go under certificates and secrets, in addition to certificates and client secrets, now there is a new field which is called federated credentials. So when you click on that and say add credential, you can pick which scenario you're using this federated credential for. So this works with GitHub actions which is being deployed and that needs to access Azure resources, Kubernetes. This particular scenario we are using Kubernetes. So here the first thing we do is we add the cluster issuer URL which is the issuer URL we saw for the cluster that I have. The namespace here is the namespace where the workload is gonna be deployed and the service account is. And then the service account name is the service account that's going to be used by the pod. And then as you can see when you populate the namespace and the service account name, the subject identifier gets auto-generated. So this is based on the current Kubernetes spec which means it performs validation and makes sure that there are no invalid values. And then you add a generic name which can be used to detect what this federated credential means and then a description. The audience field, as you can see is a standard field that we're using for all Kubernetes workload. So this is not something that has to be explicitly configured by the user. But if you need to override that then you could do that as well. So we're gonna create the federated entity credential. So what this means is now there has been a trust that has been established between your Azure AD app and service account. The next thing we're gonna do is we're gonna create a workload pod which references the service account. So just to show how simple it is to deploy an application, what we're doing here is we're just using a Qtl run command. And then as you can see we're using the Azure CLI image. In the service account, we are referencing the service account that we just created in the previous step which was annotated and node selector is Linux. But as Ernest mentioned, this will work for Windows scenarios as well like out of the box. So now that the pod is created, the next thing that we're gonna see is let's see what has been injected into this pod. So these are the environment variables that we talked about. So there are four environment variables. The Azure authorities host is the endpoint where we need to fetch a token from. The client ID is from the client ID annotation in the service account. Tenant ID is the tenant ID of the Azure AD application. But if it's not specifically provided, then we would default to using the tenant ID of your cluster. And then the Azure federated token file is the path where the projected service account token can be found. Next, let's see what other changes the mutating webhook made and on the pod that was deployed. So the first set of changes is the new environment variables that we looked at that was added by the mutating webhook. And then the next thing is the projected service account token. So it's called Azure identity token and it's a projected service account token volume. And we are defaulting to the audience that we're using a standard across Kubernetes workloads. And the expiration seconds is set to the lowest possible value so that our QBlit will renew the token at 80% of expiry. So this is added by the mutating webhook and the user does not have to make any changes to their pod spec. Then the next thing that we're gonna do is try to get an access token with the federated token that's available in the month. So we are exacting into the pod. And then the first thing we're gonna do is we are doing an AZ login. So as you can see here, we are using all the environment variables that have been injected using the webhook. So there is no specific value that I'm hard coding here. And then there is this new federated token flag in AZ login where you can actually pass the federated token, which is the projected service account token. So when we do AZ login now, using all the values that have been injected with the webhook, we can see that we are able to successfully log in. And when we run AZ account, get access token. So now this is where we can confirm that we were able to exchange Kubernetes service account token for a valid AAD token. All of this by just establishing this trust using federated identity potential. And that's it for the demo. Okay. So now that we've looked at the demo and then we've also understood the changes that we've made in Azure Workloader Entity. It's very simple in terms of configuration and then it also gets rid of all these limitations that we had in pod identity. So just to have this integrated in more and more solutions because it's zero trust and then it doesn't require a client secret, we've been working on integrating Workloader Entity with some of the projects maintained by our team and our sister teams. So the first one is, a first integration is with Azure Key Vault provider for secret store CSI driver. So for those of you who are not aware of this project, it's something that enables getting secrets that stored in Azure Key Vault. And it uses the secret store CSI driver interface to mount them into Kubernetes pod. The secret store CSI driver is a Sigard sub project which supports multiple providers today and Azure Key Vault is one of them. So we have Workloader Entity integrated with that, which means you can fetch secrets from Key Vault without any kind of secrets that we talk about. And then the second one is ratify. So ratify enables validation of supply chain artifacts such as image signatures and S bonds. So ratify recently added support for Workloader Entity Federation using Workloader Entity Federation in Kubernetes. So which means you can set it up without any kind of client secret and it works out of the box. And then in terms of the roadmap where we are with Azure Workloader Entity today. So we did the open source project launch which was done in Jan. This is for the mutating webhook and also based on our experience from pod identity, something that we learned was we want to help our users by automating some of the operations. So for this purpose, we also developed a new CLI tool called Azure Workloader Entity CLI. So we launched all of this in Jan. All of this works today with Azure AD apps. And then the second thing was the AKS OIDC issuer support. So this is where AKS will set up the OIDC issuer. It will rotate the signing keys every 90 days and then also on-demand rotation. This can also be used for federation with other cloud providers if required. And then we also did a preview with Azure AD applications on AKS for some customers but they've also been doing this publicly right now with OSS. So what we have in plans next is we are working with AAD on Workloader Entity Federation with managed identity. We know most of our users today use managed identity with pod identity. So we know that it's critical and that is what we are working towards right now. We are testing and we are hoping to have something available soon for users to try out. Once we have managed identity, this would be feature complete with what pod identity supports today. So the next stage would be to have a public preview on AKS as a managed add-on for Workloader Entity. And based on the feedback on public preview and the consumption, the next thing we would do is we would eventually move the project to GA. Thanks Anish for the great demo. And that concludes our presentations and here is a quick list of resources that you can check out. That includes our documentation's website for pod identity as well as workload identity. And there is also one URL that you might be interested in checking out which is the Federation with Azure AD. This basically explained in detail the architectures of federated identity credentials which is what workload identity is based on. And yeah, and that's all we have, thank you. Okay, if anyone has questions, go ahead and pop them in the chat. Daniel, we will have the presentation attached to the webinar recording. So if Anish and Ernest can just send that to me, this final version, as soon as we're done, I will make sure that's attached when you click your registration link. So pop your questions in the chat and let us know what else you want to know. I must be thinking of something good for y'all. Anybody? There we go. Yeah, I mean, in terms of GA, we are working towards getting managed identity support out in public preview. So once that's available, the Workload Entity Federation with Azure AD apps might go into GA, but I think hopefully we are targeting end of this year to have something. But if you need concrete timelines, then we can follow up with you again on GitHub issues or maybe some internal threads. And then if there are any other feature requests or anything that you would like to see in addition to what we support today with Workload Entity Federation, that makes it easier to be able to use it with your workloads. Please feel free to open a GitHub issue, add what you need, then we can follow up with the AAD and see if this is something that we can prioritize. Anyone else going twice? All right, well, thank you everyone for joining. Thank you, Anish and Ernest for an awesome demo and presentation. If anyone else has any questions, I think you know where to hit them up. And if y'all could send me that deck, then I will make sure everything is posted online later this afternoon. And thanks again, everyone, and join us next week for another live webinar. And we can go a little early today. Thanks again. You're having us.