 Hey everyone, hello from Israel. We are super excited to be here today and talk with you about Kubernetes Secrets. We had a lot of experience working with Kubernetes Secrets and we want to share with you what we've learned and about how they work in real-world systems and the best solution to manage them. But before we dive in, we want to talk about who we are. I'm Gal, I'm back an engineer at Firefly. In my day job, I'm responsible for large-scale Kubernetes application and I'm passionate about Kubernetes concepts. Hi, I'm Lea, I'm an engineering team leader at Firefly as well. I recently had the privilege of speaking about migrating from Lambda function to Kubernetes job at Kubernetes Community Days Amsterdam and I'm super excited to have this opportunity to share with you what we have learned on our journey with Kubernetes Secrets. So let's begin. Yeah, just a second. Yesterday, it just hit me that yesterday I deployed a new service in production cluster that uses MongoDB database and they put the MongoDB credentials as a Kubernetes Secrets. As you know, Gal, that's not the most secure solution out there. It's actually not encrypted at all. Wow, then I guess we need to think about the options for changing this. Fine, that's exactly what I'm planning on covering in this talk. So here we go. Have you ever questioned the security level of your organization secrets? When I'm saying secret, I'm referring to database passwords, service access tokens, private keys. All those secrets that all your applications rely on and all your customer sensitive data being protected with, well, we did. We did it because of the importance of our client and the very own sensitive data and we think that everyone should do so. So today we'll share with you everything you need to know about safe secret management in Kubernetes. We will go through the technologies and tool that help us manage secrets in a secure way and we will wrap up with a live demo for how to use Kubernetes Secrets or CSI driver. My God, first things first. Before we get started with all that, what is a secret? A secret is an object that contains sensitive data like an access key token or passwords. As developer, we use them all the time, like for our TikTok account, but not just that. But really a more commonly used of secrets within our application is in order to connect to a database, authenticate to third party services, decrypt the message and much more. These are just a few examples. This is because secrets are essential to fulfill the goal of our applications. Without secrets, our code simply won't function as it's supposed to or expected to. However, they need to be kept secure and protected from unauthorized access. Hence, we cannot put them in our version control system, which will then expose them to the world. We have to find a way to retrieve them from an external source during runtime. When it comes to Kubernetes, secrets come in a form of a Kubernetes secret, which is a resource that stores sensitive value on a Kubernetes cluster. Basically, there are two common ways to consume secret with Kubernetes application. First one, using volume, we mount the secret value into a persistent volume in a form of a file and our application read it from the container local file system. The second one and most common one is using environment variable. The secret value is injected into our container runtime during execution. Both of them are easy to use and integrate easily with other Kubernetes resources, such as deployment stateful and stateful set and eventually a pod. Okay, so now that we know what Kubernetes secrets are and how to use them, let's talk about how we create and manage them. Kubernetes secrets can be created manually like every other Kubernetes resource using a single Qubectl command. However, honestly, that's not considered best practice though. The modern way to manage infrastructure resources is with infrastructure as code. By managing resources with infrastructure as code, we get the ability to maintain and control our environment at scale. Kubernetes secrets like any other Kubernetes resources can be created with any infrastructure as code tool out there. Helm, Terraform or even Pulumi and much more. It sounds so simple yet it is so effective but there is a problem. There is always a problem. So this is a caution taken from actual Kubernetes docs and it says that Kubernetes secrets are by default stores unencrypted in the API servers underlying Datastore ETCD. Anyone with API access can retrieve or modify a secret and so can anyone with access to ETCD. Additionally, anyone who is authorized to create a pod in a namespace can use that access to read any secrets in that namespace. So it's obvious that Kubernetes secrets are not a perfect solution for us for most of the cases and there are some reasons for that. First of all, they are unencrypted. The secrets are stores in base 64 format. Secondly, the secret role-based access control. As list privilege concept instructs, we always wish to give access to minimal scope of data and in Kubernetes, the smallest possible scope is a namespace. A Kubernetes namespace might contain many applications inside it. In addition, it doesn't handle well secrets rotations. Secret rotation is the practice of regenerating your secrets over time in order to prevent them from being compromised or outdated. It is also important from complex regulations point of view and the organization should always rotate their secrets. Another thing is that secrets tend to be shared between multiple applications and users. Secrets tend to repeat them scale and it can create an infrastructure mess. We would like to have one source of truth for all the secrets that the system. Last thing that there is no audit. It's a challenge to check who retrieved, modified or deleted a secret. Wow, Gal, those issues are very problematic but the most beautiful thing around Kubernetes and its community. And once again, the community didn't let us down. So let's jump back to the caution you mentioned before from the official Kubernetes docs and we can find a variety of action that we can take in order to use secrets in a safer way. Today, we're going to focus on the fourth option, considering using external secrets provided. But wait a second, what's an external secret provider and why should we use it? Luckily, you have us and we will tell you what it is. An external secret provider is a software component that allows Kubernetes to retrieve secrets from an external source. An external source can be a cloud provider secret store, a third party key management service or just a database. Here are some well-known providers that I'm sure you are all familiar with. Some of them are open source like Ashikov Vault and of course there is the big cloud provider ones like AWS Secrets Manager, Azure Key Vault and Google Cloud Secret Manager. Personally, I had the pleasure of contributing to CyberConjure, which is another open source external secret provider project. Secret store is a big market with a lot of solution. Now let's understand why should we even consider using one. There are several reasons why you should store your secrets in an external provider. First of all, it performs as a single source of truth. In case of updating a value, we will change the secret in one place only. Secondly, it's a secured solution. The provider keeps the data encrypted. In addition, it's easy to share secrets between application and user when using such a provider. Furthermore, it has a software developer key or SDK in multiple programming languages that we can use within our applications. Lastly, it implements the least privileged concept. The principle that states that each user should get access permission to minimal scope of data required using wall-based access control policies. That way we can reduce security risk by providing the service or the user all of its necessary data. It sounds perfect to me, pretty good. Secure centralized solution. Should we start working? Let's start by adding a safe secret pooling in all our applications, including your new service goal. I personally prefer GoLand. Should we start now? That's probably the first thing that comes in mind after realizing all the benefits of using external secret store. However, working directly with the provider is not always the best thing to do. Reading secrets directly from the provider can be problematic. First of all, by doing so, you need to make changes to adjust the code to the specific API or software developer of the particular provider. Also, we need to take into consideration that this service might have its own API limitation like rate limiting and more. In addition, you quickly become coupled to the service. You will have to invest a tremendous amount of work in order to change to another secrets manager. Eventually, what are we left with? On the one hand, secret store provide us an interface to store and use our secret in a much safer way. On the other hand, the usage of them can require a lot of effort. We want to use secrets provider, but we want to do it with minimal effort. Gal, do you know by any chance any generic solution that integrates with an external secret provider service with minimal code changes? Sure, CSI comes to the rescue. So the most recent emerging way to manage Kubernetes secret is secret store CSI driver. This solution integrates secret store with Kubernetes via a CSI volume. Let's start with what is CSI? CSI stands for container storage interface. CSI is a specification that defines a standard interface for container orchestration systems like Kubernetes and it's let it communicate with external storage systems. So secret CSI is actually that the CSI driver is what allows Kubernetes to mount multiple secrets that stored in external secret manager into their pods as a volume. Once the volume is attached, the data is mounted into the container's file system. Let's see how this magic works with a deep dive into the architecture of CSI secret store driver. The first component of our architecture is the external store secret provider. In our case, it's vault server. We found vault as the best solution for us since it's an open source and it's give us all the benefit of store provider. Connected to the vault server, we can see the vault CSI, this is a demon set and this component responsible for pulling data from vault and communicating with it. Connected to the vault CSI, we see the CSI secret store demon set. This CSI secret store demon set is a generic component that can support reading secrets from multiple external secret stores. It's a port vault, it's a HashiCorp vault, AWS secret manager, Google Cloud secret manager and Azure Key Vault. Connected to our CSI secret store demon set, we can see our application. Attached to our pod, we see service account. Next to it, we have secret provider class. This is custom resource definition and this is the magic component that's responsible for pointing the location of the secret in vault and this component holds the role with the permission to vault. Next to it, we can see the volume that actually holds the secret's value that pod will read from. So now let's demonstrate how to use the secret CSI driver on a Kubernetes cluster in order to get the secrets from vault into our application. We are going to see this architecture run in live. We will examine the demon sets of both vault and secret store. We will configure a pod that consume a secret using a secret provider class custom resource definition. We will also change the value of the secret and see changes in the application. So, let's re-share this screen because I think it didn't share an entire screen. Okay, as you can see here, this is our vault server. We created a secret in a JSON format with the key bar and now we can see the secret. This is a super important password that keeps the entire data of all our organization. Now let's jump to the cluster. We are using an EKS cluster. As you can see, we have several nodes. On our node, we have the demon sets that Gal mentioned earlier. Here we can see the vault CSI provider and the CSI secret store driver, both of them install using a single Helm command. You can take the Helm command from the Gita page of both these projects. Now, let's see the magic component Gal mentioned. Here we define the secret provider class that uses a predefined role. We can see it here in our vault server. This role is the authentication mechanism from the Kubernetes cluster into the vault server. You can see that we define this role and the vault address and the path to the secret I showed you earlier. We want to extract the value of the key bar now for our application. In this case, we will use a pod. This pod, we create a volume mount using the secret store driver. We've read only permission and using the secrets provider class we just defined. The pod, the volume will be mounted into this particular path, M and T secret store. So let's apply. We see that there is no change or already created it. Let's create our pod and take a few seconds. We can see that our pod is running live. Now, let's open a shell, this pod. Let's see if we manage to mount the secrets. Here we can see a new file called the bar. If we will print the value of the secret, we will see that the secret came successfully. It's amazing. Now, let's change the value of the secret and redeploy our application. Okay, we have some technical issues. Sorry about that. I see that we are running out of time, so let's finish the demo. We managed to bring our secrets successfully into the cluster. So we just saw that the secret side driver is such a powerful solution. We managed to safely get sensitive data out of our vault server without any code changes or any major security risks. Applying the infrastructure was so easy, we just used the existing hand charts. And if we would like to add an addition secret store, we will just need to create another external provider, CSI, and one another secret provider class, the magic component. CSI is one of the best practices to manage Kubernetes secrets using external secret store. Also, we must talk about its pain points involved in working with this. Here are a few pain points. The first thing is the complexity, adding a secret store CSI to a Kubernetes cluster can add complexity to the cluster configuration as it requires additional components and configuration to be added. Secondly, performance overhead, accessing external secret management systems via CSI can add some performance overhead to the cluster as it requires additional network calls and potentially slower access times. And the auto reload issue. There are no building solution to automatically reload pods when secrets values changes. Secrets can be expired and needed to be changing during runtime. In that cases, you will need to auto reload the value. Currently, the driver does not offer out of the box solution for such cases of auto reload. So it's a wrap. What we talked about today. Today we talked about what secrets are, what Kubernetes secrets are, how to deploy the secrets using infrastructure as code. We talked about the Kubernetes secrets problems and then we talked about how to overcome the Kubernetes secrets problem while using the external secret store providers. And the last thing, we had an architecture and a live demo of the Kubernetes CSI secret store driver. Overall, while native Kubernetes secrets provide a useful way to store and manage sensitive information within a cluster, they may not be suitable for all use cases and may require additional management and security measure to ensure that the secrets are properly protected. There are many great solutions out there and we recommended you to analyze the needs of your organization in order to choose the best solution for your organization security. Today, we gave you a quick overview of our architecture and how Firefly managed her secrets. But shh, don't tell anyone. Can you keep a secret?