 Hey everyone, welcome to CNCF on-demand webinar. Today I'm going to talk about the secret store CSI driver that integrates external secret stores with Kubernetes. A brief introduction about me, I'm Anish. I'm a software engineer at Microsoft based in Seattle. I'm part of the upstream Kubernetes security team. I am a maintainer of the secret store CSI driver and the Azure Key Vault provider for the secret store CSI driver. So why are we talking about external secret storage when Kubernetes has a perfectly good one built in? First, Kubernetes secrets may not meet your data-addressed encryption requirements. By default, when you create a Kubernetes secret, the data is stored in HCD and it's just base 64 encoded, but it's not encrypted. Although there are KMS providers to enable secret data encryption at rest. Depending on your organization, you may have already standardized on a third-party secret solution and need to use that. Also, your external secret solution may have some sort of secret rotation story or integration that you're looking to leverage. Finally, if your organization has already invested in auditing and alerting on a third-party secret system, you may not want to duplicate that effort for Kubernetes secret. So what do you do? There are a few options to consume external secrets. You first might look at modifying your application to fetch the secret from external API directly using the SDKs provided by the external secret store providers. This may not be possible depending on your deployment. You may not have the code to edit or it may be prohibitively expensive to implement these changes. And if you're targeting deployments against multiple secret providers, this effort would need to be duplicated across every single storage system. You might also look at just copying the secrets and syncing them as Kubernetes secrets, perhaps with a single controller. This is portable and won't require application changes, but you may lose out on some of the benefits of your external secret stores, like using the identity of the workload for access controls. Because in this particular scenario, you are granting the single controller permissions to access the secrets instead of the workload that needs to use the secret. The third option is you could use a sidecar to fetch and write these secrets. The sidecar may be injected using a mutating webhook. Here, the identity that is being used to access the external secret store would still be the pod identity, but having a sidecar and an additional mutating webhook may add operational complexity. Finally, there is the secret store CSI driver that uses a container storage interface specification that we are here to talk about today and cover some of the features that we think is a good fit for consuming external secrets. The secret store CSI driver project was originally developed at DS Labs. This storage driver is now built and maintained as a Seagod sub-project. The driver allows Kubernetes to mount multiple secrets, keys, and certificates stored in enterprise-grade external secret store into the pods as volumes. Once the volume is attached, the data in it is mounted into the container's temporary file system. In terms of the features that the secret store CSI driver supports, it has a familiar file system mount experience to compute workloads. It's also pluggable and it can support multiple external secret providers without the application having to be modified. It can load new values of secrets throughout the lifecycle of the pod. It can also sync the mounted content as Kubernetes secret for further compatibility with existing deployments. Finally, it supports both Linux and Windows containers. In terms of the prerequisites that are required for using the driver, the driver can be used on any Kubernetes cluster that's running one 16 or above. In case of Windows, you would require a 118 plus for CSI inline volume support. We talked about the driver supporting multiple external secret providers. The supported providers that we have today with the secret store CSI driver are Azure Key Vault, Google Secret Manager, Hashicot Vault, and AWS Secret Manager. So we saw a brief introduction on what the secret store CSI driver is. So how does the secret store CSI driver work? A driver is installed as a daemon set on each node in the cluster. In addition to the driver being deployed, there needs to be a provider-specific daemon set deployed on every single node. So first, when a pod is created through the Kubernetes API, it gets scheduled onto a particular node based on scheduling decisions. The KubeLit process on the node looks at the pod spec and sees there is a volume mount request and referencing a particular CSI driver. At this point, the KubeLit issues an RPC call to the CSI driver provided in the CSI volume. The first thing that the CSI driver does at this point is it mounts the tempfs to the pod and then it issues an RPC request to the provider based on the configuration. The provider talks to the external secret store to fetch the secrets. The provider in this scenario uses the pod's identity. So it could be the pod's workload identity or any other configuration that's tied to the workload pod. The provider then fetches the secrets from the external secrets to the pod and it sends the content back to the driver as part of the RPC response. The driver then writes the secrets into the file system and at this point, the volume is successfully mounted and then the pod starts up running. So we talked about how the driver works. Now let's get into some of the YAML files that are required for the configuration. So here we have a sample pod spec. So if you look at it here in the volume mounts, we have the mount path defined which is the path inside the container and it has slash var slash secrets. And if you look at the volume, we can see that it is a CSI volume and not a secret volume. The driver name in the CSI volume is what tells Cubelet to use the secret store CSI for this particular volume. The secret provider class in the volume attribute is a namespace Kubernetes custom resource. It is used to provide driver configurations and also provide a specific parameters for the secret store CSI driver. So we've looked at the pod YAML and then the secret provider class YAML where you define all the configuration for your provider. But how do you know which version of the secret that you're getting from an external secret store is currently being used by your pod? The secret provider class pod status is a namespaced Kubernetes custom resource that is created and managed by the secret store CSI driver to track the binding between a pod and a secret provider class. This resource contains details about the current object versions that have been loaded into the pod mount. And then it can indicate whenever there has been rotation and then newer version of the secret is now being used by the pod. So let's see a small demo on all the things that we just talked about to see how the driver works. So as part of this demo, what we're gonna do is we have a kind cluster and it's running Kubernetes 1.21.1. To show how the secret store CSI driver enables pod portability across different external secret store providers, we're gonna start with deploying two applications written against specific third-party secret stores. So in this particular demo, we're gonna be using GCP and Azure. First, we're gonna see what the pod YAML looks like for these applications using the external secret store APIs to access the secrets. So this is a pod spec for a pod that's implementing API calls using the Azure SDK to talk to Key Vault. The Key Vault name is kind KV and the secret that it's trying to fetch is secret one. And this is a pod spec for a pod that is using the GCP SDKs to access a secret from Google Secret Manager. So we can see here, the secret name is the resource ID of the secret that it's trying to get. As you can see, both of these YAMLs are referencing credentials from a secret store creds which I've already pre-configured in the Azure and the GCP namespaces for the demo. So now I'm going to deploy these pods in their respective namespaces. So we have deployed the Azure pod in Azure namespace and now we're deploying the Google Secret Manager pod in the GCP namespace. Let's quickly check if both the pods are running. Okay, now we can see that both the pods are up and running. Let's check the logs to see the secret being logged. So we can see here that the Azure pod was able to successfully get the secret from Azure Key Vault using the SDK and then it logged the secret. Similarly, the GCP pod was able to get the secret from Google Secret Manager and then it was able to log back. So here we have two different pods that are implementing the APIs required for each secret store backing. Now, instead of having two different application implementation for external secret store, I have a third application that was returned to consume the secret from the file system instead. So using the secret store CSI driver, I'm going to show how this application gets the secrets from either of the secret backend, that is Azure or GCP. So as a first step for the demo, I'm going to deploy the secret store CSI driver using Helm in the CubeSystem namespace. So if you see here, I'm setting a couple of values required for the Helm charts. So I'm enabling the secret rotation feature and I'm also setting the rotation pole into the five seconds, which is aggressive, but it's only for this purpose of the demo. And then I'm also enabling the sync secret feature for the directory. We typically recommend to use a separate namespace for the CSI driver pods other than the ones that's used for your workload. The default that we set in the manifest is CubeSystem because the CSI driver pods are privileged and it's recommended to 100 in CubeSystem namespace. Okay. So now let's just quickly check what the Helm chart deployed. When we look at it, we can see that we have a driver we've demonstrated running in CubeSystem namespace and it's running three containers. One of the containers is the actual implementation for the driver. And then the other two cycle containers are for dynamic registration of the diver with Cubelet. And then the third one is the liveness probe, which ensures that the driver is functioning as expected. Okay. So the next, now that we have the driver running, we also have to deploy the providers for the demo. So I'm going to deploy the Azure provider first in CubeSystem namespace and next I'm going to deploy the GCP plugin in the CubeSystem namespace. And if we take a quick look on all the pods, so now we have a driver and then we have an Azure plugin running in CubeSystem and a GCP plugin running in CubeSystem. So let's take a look at what the pod spec looks for this application, which is just reading secrets from the file system. So we have a pod and it has a volume mount and it's mounting the volumes into mount secret store. Here we can see that it has a CSI volume. The driver name is the name of the secret store CSI driver and the secret provider class is CSISPC. So this is familiar to what we just looked at in the presentation. Let's also look at what the secret provider class looks like for this particular, for accessing secrets from Azure keyword. So when you see here, we have provider set to Azure in the secret provider class and these are the provider specific parameters that's required for accessing the secret. So here we have the keyword name, which is kind KB and object name secret one. So we're trying to retrieve secret one, which is of type secret from keyword kind KB. Let's also quickly look at what the secret provider class looks like for the Google provider. Here the provider is GCP and in the parameters for resource name, we have the resource ID of the secret that we're trying to fetch and the file name here is used for writing it in the container. So now I'm actually going to deploy the Azure secret provider class in Azure namespace and then the GCP secret provider class in the GCP namespace. Now that we have the secret provider class created in both the namespaces, I am going to deploy the exact same pod spec without any changes in Azure namespace and also the GCP namespace. So if you see here, I have the exact same pod YAML which is referencing a secret provider class that's being deployed in two different namespaces, which means it's accessing two different secret store packets. So when these pods get scheduled onto a node, the cubelet sees the volume definition and based on the CSI volume, the driver name, it invokes the CSI driver to mount the volume. The CSI driver will mount the tempfs and make an RPC called the driver to fetch and write secrets to the file system. Okay, now let's check the pod mount to see if the secret exists. So we can see that we are exiting into the CSI pod in Azure namespace and then we're looking at the volume mount path and then we can see there is a file secret one. Let's also look what's the content of secret one and we can see it's hello from Azure keyword which was the same as what we saw when we were directly trying to access using the APIs. So now let's weekly check the logs for the CSI pod in Azure namespace just to confirm it's working as expected. And then if you look at the same CSI pod which is reading from the file system in GCP namespace, it has a secret from Google secret manager. So there you go. With this, you can see that the exact same application without any changes to the pod YAML is able to fetch secrets from Google secret manager as well as from Azure keyword. So we talked about the driver also supporting sync as Kubernetes secret. The CSI driver provides an optional feature to sync the mounted content from the pod as a Kubernetes secret, a common usage of this feature is to store the TLS certificates in external secret store and have the driver sync it as a Kubernetes TLS secret and then that can be used with English controllers. So in addition to the sync Kubernetes secret, they can also be referenced in the pod spec to set environment variables if required. As you can see in the secret provider class there is an optional secret objects field which is used to indicate that the mounted content also needs to be synced as Kubernetes secret, secret auto rotation. So a generally acceptable best practice is to periodically rotate your secrets. So if your external secret store has automatic rotation feature, you may be interested in how the workload that's running on your Kubernetes cluster can get the new values of a secret whenever it changes. The driver supports rotation, automatic rotation by periodically re-issuing the RPC calls that we looked at to the provider to refresh the mount. Once the rotation is successfully done in the mount, the driver will also emit a Kubernetes event to indicate that a rotation is successfully complete. In addition, if the mounted content was also synced as a Kubernetes secret, then the driver will also go and update the values in the Kubernetes secret. So we talked about two features. Let's also jump into a demo to look at how the driver does the syncing as Kubernetes secret. So for this part of the demo, you're gonna do it with two steps. The first one that we're gonna look at is how the driver can sync the mounted content as Kubernetes secret. So the first thing we're going to do for this is we're going to enable an application to work with Nginx Ingress Controller and we're gonna store our TLS certificates in Key Vault and access them using the driver. So I've already created a certificate, a self-signed certificate for local host using the step CLI. So let's inspect the certificate before we actually import it to Azure Key Vault. So if you look at this certificate, you can see that it's for local host and in terms of expiry, this certificate is set to expire on March 10th. So I'm going to import this certificate into Azure Key Vault with the name Ingress Cert. And now for the demo, we also want to deploy the Ingress Controller. So here we are deploying the Ingress Nginx Controller on the kind cluster. While the Ingress Controller is starting up and running, let's look at what the secret provider class looks like for this sync as Kubernetes secret. So if we see here, the provider Azure, which is pretty standard as we saw in the last demo, but here now we have this additional field called secret objects. And then the secret name here signifies what the name of the Kubernetes secret should be. And the type is what the driver will set when creating the Kubernetes secret. And then the keys are what are the keys that are required for the secret type. So I'm going to deploy the secret provider class in default namespace. And then I have a single file which defines the pod, the services and the Ingress. So let's take a look at that. So here I have a pod which is called foo app and it has a volume mount and in volumes, it's referencing the secret store CSI driver. And we have a service to expose that pod. And similarly we have another pod called bar app which has a service exposing it. And then finally we have an Ingress resource which is set up to use TLS. The secret name here that we have defined is the same secret that the secret store CSI driver is going to create. And in terms of the rules, we have a slash foo and then a slash bar for validation. So now we're going to apply this YAML which deploys the pods, the services and the Ingress resources. So let's check if the pods are up and running. So here we can see that the bar app and the foo app are pods are up and running. Let's check if the CSI driver actually created the Ingress TLS CSI secret. So we see that it's there in the default namespace. Let's also look at the YAML file. So here we can see that the CSI driver created the secret with TLS.cert and TLS.key as the template provided by the user. And in the labels, you can see that this resource was created and it is being managed by the secrets store CSI driver. So the first thing we're going to try is we're going to try and curl the endpoint to see which certificate is being used. So if we see here, we can actually see it's using the certificate that we uploaded to Azure Keyworld that was mounted into the pod by the CSI driver. So the expiry date is March 10th, which is what we looked at in the certificate. So we also talked about the secret provider class pod status before in the presentation. So let's quickly look at the secret provider class pod status here to see what version of the certificate is currently in the pod. So if we see here, we can see that the secret ingress.cert is in the pod. And then the version that it's currently running has starts with 6498. Now for the second part of the demo, let's actually try the rotation with the current setup and also with the pod portability setup that we have before. So for the ingress rotation demo, I've already created another self-signed certificate with a longer validity. So when we inspect it with OpenSSL, we can see that the validity for the new certificate that I have is March 18th instead of March 10th. So I'm going to import this rotated certificate into Azure Key Vault. The auto rotation feature in the secret store CSI driver should enable us to fetch the new rotated certificate from Azure Key Vault and update the mount and the sync Kubernetes secret. So now when I try to curl the same endpoint that I called before, we expect to see the certificate returned to have an expiry of March 18th instead of March 10th. So let's try that out. Sure. So now when we look at it, we can see that the expiry date has changed to March 18th. This is because the driver updated the mount and it also updated the sync Kubernetes secret. The ingress controller picks up the updated Kubernetes secret and then re-loads the certificate. Now let's also look at our secret provider class pod status to confirm the secret version that's being used in the pod is now different. So now we can see the version is 9475 which is different from 6.4 which was before. We also, I also said that I will do the rotation demo with the first setup that we had. So let's try that. So I'm going to rotate the secrets that were used by the application that was used as part of the pod portability demo to see if it can pick up the latest value. So the application that was used in the first demo watches the file system and it basically reads the file from there and logs it. And then it also has file watcher implemented which means anytime there is a change in the file system it will automatically become the new value and then log that. So let's tell the logs before I actually go and rotate the secret in Azure Key World. So if you see here it is logged out of from Azure Key World which is during the startup. So now I'm going to go and rotate secret one to say I am rotated. So when I do this I'm updating secret in Azure Key World and if we see we can see that the driver has updated the content in the mount for the pod and it was able to pick up the latest value using the file watcher implementation. So we've looked at all these demos and I'm sure the next question is what is the current state of the project? So the secret store CSI driver core functionality is stable. This includes the interface defined for supporting multiple external secrets provider pod portability with secret provider class custom resource. So you have a single YAML file which references a secret provider class. And if you want to move it from a single secret store to a different external secret store there are no changes required for the YAML file. And in terms of the feature parity for Linux and Windows containers, the driver is stable. So what does the future look like for the secret store CSI driver? The auto rotation feature that we talked about is currently in alpha. We are working towards moving this feature to stable by reusing some of the CSI core functionality. So CSI has a requires republish feature in which scenario QBlit will automatically issue an RPC called periodically to update the month. By using this we can actually get rid of the controller that we have in the CSI driver that handles the rotation today. And then we are also looking forward to more community involvement. So if you're interested in adding a new provider for the CSI driver or have feature requests reach out to us on Slack or on GitHub. If you would like to get involved you can join the CSI secret store channel in Kubernetes Slack. We also have a mailing list for notifications on releases and security announcements. We use GitHub issues to track bugs or feature requests or to answer any questions that you might have asynchronously. Also we hold a bi-weekly community meeting. It's every Thursday at 11 p.m. Every bi-weekly Thursday at 11 p.m. EST. And in terms of the resources I share some resources from the presentation. So there is a documentation link to the driver and then also the GitHub repo link to the driver. Each provider have their own specific documentation in terms of what are required for the secret provider class so there's links to documentation for the individual providers. And for the demo I've reused some of what I've used during my QCON talk. So all the artifacts that I've used today are available in that GitHub repo. Thank you.