 Hey everyone, I hope you're having a great time at GitOpsCon this year and thank you so much for joining me in this particular talk. In this particular one, we are going to discuss how you can unlock the secrets to effective secrets management in GitOps. Now we all definitely agree to the fact that GitOps has quickly become one of the most powerful paradigms when it comes to having a declarative approach of defining that continuous delivery part of our entire pipeline. But something that we also got to keep in mind are how are we managing the secrets that are associated with our application? Because as we all know that GitOps actually uses Git as a single source of truth. So all our applications code is basically stored in Git, right? And I don't think that's something we want with our secrets as well, right? To be publicly available for everyone to see. So that is what we are going to discuss in this particular talk. We'll first be discussing what are the different types of secrets that are associated with our application. Then we'll move on to discussing what are the problems when it comes to managing secrets in a GitOps workflow. And after that, we'll also have a discussion on some of the potential solutions that are available in the ecosystem and how you can utilize them to properly manage your secrets in a GitOps workflow. Yep, pretty excited for this one. And let's get started. So a little bit about me. My name is Kunal Verma. I am working as a DevRel at Cube Simplify, where we are aiming to simplify the cloud native and WebAssembly ecosystem for everyone. So we'll re-check us out. I'm also the maintainer for the Commiser project, which is an open source cloud agnostic resource manager. So it can help you to manage cloud resource costs in a multi cloud infrastructure. Pretty interesting tool. Make sure to check it out. Lastly, I am an open source and cloud native advocate. That means I really love helping people out in getting started with open source cloud native DevOps and all that amazing stuff because it's just amazing to get involved. And you can always reach me out on my Twitter and LinkedIn. The handles are mentioned here as well. And we can have a conversation moving ahead. Let's quickly first recap our knowledge on what is Git and how it is utilized in a GitOps workflow. So basically, Git is a version control system that you can use to store the source code of your particular application. It enables cross team collaboration. That means everyone in your particular organization or on your particular team can have the access to the source code of the entire app. The number one reason that people actually use Git is having a transparent history of the changes. So basically, Git helps you to maintain a version history. That means you can go back to the changes if anything breaks. And that's one of the coolest features that Git has. And that's why a lot of organizations have actually adopted Git as their version control system. Lastly, there are a few principles to Git. That is the few operations that Git enables us to perform. That is, you can commit your code, you can push, pull, merge, and then you can also create branches if you have a large code base and you have different teams working on a single code base. That's where the concept of branches is pretty helpful there. Now, when it comes to GitOps, it's basically a continuous deployment methodology that uses Git as a single source of truth. That means Git is actually at the center of every GitOps workflow. Now, when we talk about GitOps, we are talking about that CD part of the entire CD pipeline that is the continuous deployment. And whenever we talk about a normal GitOps workflow or a typical GitOps workflow, we talk about it with reference to communities, right? So at the end of the day, we are deploying our application to communities. And that is what is happening in the modern DevOps environment. Whenever we talk about a GitOps workflow, it often utilizes communities, right? It does not have to be, this is not a necessity. But whenever we talk about the modern DevOps, or you can say the quite famous workflow, that's how a GitOps workflow looks like, it utilizes communities. Now, whenever we talk about GitOps with communities, it's basically used to manage your Kubernetes cluster. So it helps in the cluster management. And it also helps us to automate the process of application delivery so that your entire process of application delivery is resilient, is fast, and you can quickly deliver the product to your end user. Now, as we are discussing GitOps, which is a continuous deployment methodology, the main aim over here is to compare the current state with the desired state. The desired state is declaratively mentioned by us in inside of Git. And then we have a loop, which is often called as a reconciliation loop, which basically compares the current state of the Kubernetes cluster to what we have defined in Git. And at the end of the day, it makes sure that the current state of the cluster aligns with the desired state, what we have defined inside of Git. And that's how the entire GitOps workflow works. All right, now that we are at the same page, let us talk about secrets, right, for a minute. So what is a secret? It's a piece of information. It's a piece of confidential information that is required for authentication and authorization of different services that are associated in your application, right, that can be databases, that can be API keys that can be username and passwords, we'll actually have a look at what are the different types of secrets that are associated with their application. Again, the secrets are essential for a secure deployment process and the operation of the entire application itself. So the secrets we are concerned here with are of the application, the source code that we are storing in Git. That is what we're concerned about. Now, what are the different types of secrets? So these are some of the different types of secrets that we need to keep in mind, that can be database credentials, it can be access keys of different cloud providers out there, if your application is talking to a certain cloud provider, that can be TLS certificates, API keys, SSH keys, if you want to remotely access some virtual machines and you have a configuration file that mentions some of the SSH keys, you don't really want them to be stored on Git, right? Then we have the normal good old username and password. So these are some of the examples of the different types of secrets that we are concerned with and that are associated with our application. So what does a typical GitOps workflow looks like? So this is basically the entire pipeline that we call CICD pipeline. And the first part we have the continuous integration part that is we have our application, which is utilizing Git and it may be stored in any of the code repositories, for example, GitHub, GitLab and all that particular stuff. And then we have the continuous deployment part of the pipeline that is taking our source code from Git and deploying it onto a Kubernetes cluster because right now we are talking about a GitOps workflow with Kubernetes, right? Now what we are concerned here is the database secret. For example, you have an application which is utilizing Python as the backend and it's utilizing MySQL as a database. Now the concern here is we have to store our code on Git, right? Now we have the database secrets, the MySQL secrets that our application needs in order to function. At the end of the day, we will not want to store these secrets directly on Git and that is basically the area that we'll focus on today. How we can actually manage these database secrets so that number one, they stay secure. Number two, our application can readily access them whenever it wants. So what is the problem, right? Why are we having this discussion right now and why it becomes important today? Now one of the fundamental challenges we face in GitOps is that Git itself was initially designed for collaboration. Now its core purpose is basically to provide a way for different teams to collaborate and track changes of the source code, right? And that is what exactly the problem here is. When it comes to handling secrets and sensitive information related to our application, this particular collaborative nature of Git is what introduces a potential security risk to our entire application. The second one is sometimes storing sensitive information as plain text. Now let's face it, sometimes we have the habit to just quickly get things done with and store secrets as a plain text. Now this is something that you might not see every day, but sometimes unintentionally things happen, right? And you may end up storing your secret, any kind of secret as a plain text. And at the end of the day, this is definitely one of the potential security risk for our application. Now when it comes to a typical GitOps workflow, version control is basically the king, right? Because we are utilizing Git, it's actually at the center of our GitOps workflow. But that is not something that we can say for secrets. We don't have the way to basically track changes for our particular secrets, right? The same thing that we can do for our code in Git. This is something that is again missing in our ecosystem. And this definitely poses a challenge when it comes to conducting thorough audits and having that maintained version history of the evolution of our secrets. Something which I've also noticed in the ecosystem is limited to no encryption at rest. Now a typical GitOps workflow or a GitOps tool that you're using will not actually provide you with any kind of mechanism that you can use to encrypt the data at rest. And it's one topic to talk about securing secrets which are in transit, right? But there is also another important topic that we should definitely address is basically prioritizing encryption at rest. And how are we addressing the vulnerabilities that are associated with that particular area? So this is also something that we got to address. Lastly, this is also an important point to understand that there are a lot of tools and solutions available in the ecosystem, but not understanding the capabilities and especially the limitations of each solution can definitely introduce new challenges for you. So it's essential to understand the capabilities as well as the limitations what a certain solution can bring, right? And the solutions that I'm going to talk about today in detail, we are also going to discuss what are some of the limitations and where you should not use those particular solutions and what are some of the use cases that those particular solutions are actually helpful in. Let's talk about Kubernetes secret, right? We all know it exists. It's a native Kubernetes object that Kubernetes actually provides us to store the secrets and that's how our application is able to access those, right? The question here we want to address is, is it a potential solution that we can use with our GitOps workflow? So a little bit of definition, it's a native Kubernetes object, which is used to store sensitive data such as password, a token key, and keeping these separate from the entire application code. This is something how it looks like. So we have a kind secret and then here is actually where we are giving it the service token ID, for example, right? Now, the important thing to note here is the Kubernetes secret is base 64 encoded. So the value that you see here is base 64 encoded. And that's how Kubernetes secret actually works. This is there by default behavior. Now the question that we got to ask ourselves is, are they secure? Is Kubernetes secret secure? And the answer to that is absolutely not. And it's a big absolutely not. Now, as I mentioned that this particular value of the secret is base 64 encoded. Now there is a very, very simple way to decode this particular flag. And it's right here. So you just have to do echo this particular flag and base 64 decode. And there you have got the exact value of the secret that you store. Now, many of you who didn't know about this, although the chances for that is very less. But if you didn't know, you will be like this. Oh my God, they have cracked my base 64 encoding. What happened here? So the important thing to note here is that encoded does not mean encrypted. So the value that we store inside of the native Kubernetes secret, they are encoded and not encrypted. So you can easily decode them using this particular method. Or if the secret value is a bit complex, there are numerous other base 64 decoders that are out there and the hacker can utilize them to to basically get your secrets value. So at the end of the day, what we learn is Kubernetes secret itself is not a solution that we want to go for, right? They are not secure. Now, what are the potential solutions available? And this is for real this time, not a question mark. This is a full stops. These are some of the solutions that are revolving around the ecosystem that we can utilize to manage our secrets in a GitOps workflow. Now, for this particular talk, we are going to focus on these three solutions that you can see. And number one is sealed secrets, secret store CSI driver and external secrets operator will go in depth into them and understand how they actually work. And what are the pros and the cons of using them? All right, let's start with sealed secrets. So sealed secrets is basically an open source tool, which is by a bitnami labs. And it basically provides you with a declarative Kubernetes secret management. The main highlight to note here is that sealed secrets are actually encrypted. So you can easily store them in your GitHub repository or any code repository for that matter. And you don't have to worry about anyone cracking your encrypted secrets. So that is one of the main highlights of using sealed secrets, rather than the native Kubernetes secrets, which are base 64 encoded. So this is basically how sealed secrets actually work internally. So you have an operator, which you can install in your Kubernetes cluster, either using help or any other installation methods. And then this particular operator generates two keys. Number one is the public key and the other one is the private key. So the public key is utilized by the cube seal CLI, which you install on your local system. And then you utilize this particular public key and your native Kubernetes secret to convert them into a sealed secrets file. So the cube seal CLI basically encrypts the Kubernetes secret to a new sealed secrets resource, right? And this is something how it looks like. So it's kind sealed secrets because we are creating a custom resource here. And then we have this particular data which is encrypted. Now this is way more secure than just having a native Kubernetes secrets file, right? Now comes the interesting part. Let's understand how the decryption part works. The GitOps operator takes this particular sealed secrets file and it would convert it into the native Kubernetes secret. How? Using the private key that was generated by the operator in the first place, right? So the CL secret is converted to the native Kubernetes secret and then it gets feeded to the containers that are actually running in a pod, right? The normal thing. Now let us discuss some of the quick conclusions or observations that we can make from here. So what's the use case? When do you want to use sealed secrets? You will want to use them when you want to store your secrets in Git and you want to safely store them because these are encrypted and not encoded. These are way more secure and you can definitely store them in any code repository of your choice. It's a great choice for quick start and POCs. If you want to just quickly test a few things, this is a really nice tool for you. Now something to note here is that it has some limitations as well, right? For example, it does not scale well, right? When you are talking about a production grade system or an enterprise grade system, this might not be suitable for those particular use cases. Now the next tool in line is secret store CSI driver. And from the name itself, it's suggesting that we are actually utilizing the container storage interface capabilities of Kubernetes, right? So the CSI driver actually provides you with a way to integrate external secret stores with Kubernetes via a CSI volume. Now the CSI driver actually supports AWS Azure, Google Cloud and Vault. So you can basically use these cloud providers to store your secrets, and then you can utilize those secrets via the CSI volume in your Kubernetes cluster. Now how it works is you have to install the CSI driver itself onto your Kubernetes cluster and it gets installed as a daemon set. That means on each node you will have a separate CSI driver that will be working. Inside the CSI driver, three components will be working side by side, the number one being node driver registrar. And that's how the CSI driver would register that particular node in which it's working. The second one is liveness probe, which basically is used for the health check of the CSI driver itself. The last one is the secret store, and that is what we actually use to mount it onto our particular volume. The important component to note here is the CSI provider, which is a custom resource definition. So this is something how it looks like the kind is secret provider class. And here we actually define what provider we are using. For example, in this particular one, we are utilizing vault as a secret store provider. And then we are mentioning the vault server's address so that the driver knows how to access that particular secret store. And at the end, we are actually providing it with the path of the secrets that are actually stored on our vault server. Now, once the provider class is set up, the only thing left is to mount it onto our deployment. Right. So this is something how it looks like. So here we have kept the name secret store inline. The driver name is called secret store. That is one of the components of the CSI driver that I just mentioned previously. And then we are also giving it a reference to the provider class that we want to use. And that's how the secret store actually knows from where to retrieve the secrets from. Right. And at the end of the day, it's mounting all of the secrets inside of volume. And that's how our pod will be able to have access to those particular secrets. Now, let us discuss some of the quick conclusions we have from this particular tool. Now, if you want to store your secrets outside of the communities cluster, because we are actually utilizing the external secrets provider, that is a use case that you may want to use this particular tool. Now, here we are entirely deleting basically the need for using the community secret, the native community secrets. And instead, we are using CSI volumes to mount these secrets onto our application. Right. So we want to replace the community secrets with something this is a go to tool that you can definitely try out and use. Now, again, this is a good choice for quickly testing some things and for POCs. Again, I have not seen many production use cases or any enterprise grade use cases of this particular tool right now. And I think one of the reasons of one of the roadblocks out there is many organizations want to feed in the secrets as environment variables, rather than using file mounts, right? So that is understandable. And that is one of the roadblocks or a challenge that you may see if you are looking to use this particular tool. But again, it's a good choice if you want to quick start or you want to quickly test some of the things or for POCs. The last tool in the line is external secrets operator. This is something that right now is being heavily used in the ecosystem. So let's check it out. It's a communities operator from the name itself, it suggests and it basically, again, integrates with external secret management system, such as AWS secrets manager, Hashi Cop vault, Google secrets manager, and many more. It then basically provides you a way to access those particular secrets from those external providers. Now the goal of this particular tool is to synchronize secrets from external APIs into communities. As I mentioned, your secrets are stored in external providers and you will be able to access them through using this particular operator. Now how this works is you install it as an operator onto your Kubernetes cluster. Now there are two things we need to configure here. The number one is the secret store, which is a custom resource. And here is where you'll provide the external providers information. So what provider you are using to store your particular secrets, this is where you'll provide all the information. And this is something it looks like. So here kind secret store. And here we are actually using the AWS secrets manager. So you're providing all the authentication details so that the operator is authenticated to use that particular provider, right? So now that we have authenticated ourselves with the provider will create the external secrets. And again, which it is a custom resource. And again, it is a custom resource, which looks something like this. And here we are doing kind external secret. And then we are referencing the secret store itself. That is we are referencing the external secrets provider. And then basically, we are retrieving the secrets that we want to use with our application. Now, essentially, once we create the external secret resource, in the backend, Kubernetes will actually converted into the native secrets object. And that's how it is getting utilized by the containers and the pods which are running in your particular system, right? That's how the entire workflow of the external secrets operator looks like. Two important points to note here is that the secret store custom resource is a namespace scoped. So that means you have to specify the exact namespace name, where you want to authenticate the provider with. If you don't want to do that, you can use the second type of custom resource, which is cluster secret store. And this is globally scoped. That means it is accessible from all the namespaces. So if you basically want a common gateway between your Kubernetes cluster and the secrets provider, that's the way for you to go ahead. You can use the cluster secret store. So if we jump onto the quick conclusions, again, when you want to store your secrets outside of your Kubernetes cluster, that is when you can use this particular tool, it is pretty easy to set up and configure in your Kubernetes cluster. And the number one highlight of this particular tool, it's, it's a great choice for enterprise and production grade systems. And I've actually seen this particular tool being used in production grade systems as well. The number one reason for that is it does not poses any challenges when it comes to scaling your particular infrastructure. So now that we have seen some of the solutions that are available in the ecosystem, I just want to quickly show you a demo of the seal secrets. And now you can utilize it to basically encrypt your secrets, and you can easily store them in kid. So let us have a look at that. All right, so to keep this demo short, I've already installed the seal secrets operator. So if you do Qtl get pods, so you can see the seal secrets operator is already up and running. And I've also installed the Qube seal tool, the Qube seal CLI tool, which we want for encryption. Qube seal, right now you will see that Qube seal cannot retrieve the seal secret service. So as I mentioned previously that the seal secrets operator actually generates two keys, one is the public and the private key. We basically use the public key with Qube seal to encrypt the Kubernetes secret, right? And in order to do that, we would have to make sure that Qube seal is able to actually access that public key through the operator. So for that we'll use this particular command. So it's Qube seal, use fetch sort. And we'll also give the controller name that in this case is my release seal secret. And then we'll also give the controller namespace, which is default in this case. So when we do that, it will generate our certificate that we want. And now we know that Qube seal is able to actually retrieve the public key from the operator. Now I've already created a sample secret.yml file. So if we do cat secret.yml where we are storing the database password and this particular value right here is base 64 encoded. So if we do echo this and base 64 decode, you can see that we have got the password out, which is the top scone 123. Now we don't want that and we want to encrypt this. So we'll use Qube seal to do that. And for this, we have this particular command, which is Qube seal will again give the controller name that is my release seal secrets. Again, we'll provide with the controller namespace, which is default. And now we'll use the format flag where we'll provide the syntax of the file we want to encrypt. In this case, we are using YAML. So you will give YAML here, but you can also select a JSON file if you want to encrypt JSON file. After that, we provide it with the file that we want to encrypt. In this case is the secret.yml file. And then the name of the new file that we want to create. In this case, it is seal secret.yml. Now, when we hit enter, you'll see that a new seal secret.yml file has been created. And if we go inside that, so this is a particular file. And you can see this is a kind seal secret. This is not a normal QBD secret. This is a custom resource, which is called a seal secret. And here the DP password string that is mentioned here is encrypted. So now you can easily go ahead and store this seal secret.yml file in your Git repository. And you don't have to worry about anyone cracking your particular password. Now, as I previously mentioned, when this particular file is applied to a QBD cluster, it will automatically get converted into a normal QBD secret, right? So let's go ahead and do kubectl apply minus f seal secret. So the database secret has been created. And now if we do kubectl get secret, you will see that the database secret has been created. And this is a normal QBD secret, right, that you can use with your application. So if you want to see what's the content here, you can do kubectl get secret database minus oyml. And here you can see that we have again got the same string back. This is base 64 encoded. And this works as a normal QBD secret. So that's how the entire process of encryption looks like when you're using seal secret. Yep, that is it from me today. I hope you learned something new. If you have any questions, feel free to reach me out on my Twitter and LinkedIn profile. And additionally, you can scan this particular QR code to access some of the additional resources related to this topic. Plus this slide deck as well, so that you can refer it at your personal time. Again, I really want to thank the top scorn to give me this opportunity to present here and hope you learned something new. Have a great day. Bye