 All right, good morning everyone. You all got out of bed. You've all found your way here. It's a challenge sometimes with some of the conference centers in Europe. Today's talk is 100,000 different ways to manage secrets in GitOps. My name is Andrew Block. I am a distinguished architect with Red Hat. I have been working in the GitOps space for a good amount of time, been working part of the working group that the team mentioned earlier today. I do specialize not only CICD, but also security and automation. I have two books out there. One of them is called Learn Helm. The second one, which is coming out in August, is Securing Kubernetes Secrets. So be on the lookout for that. And a little bit of Easter egg a little later on the presentation coming up. So keep an on lookout for that. I am also a helm maintainer. So really interested in the helm ecosystem, understanding how applications are managed in a Kubernetes environment. So GitOps principles, they apply to a variety of contexts. They apply to your infrastructure. This is going to be your VMs, your static machines. They're also going to apply to your applications. They're going to be database credentials, configurations, you name it. But they also apply to cloud and non-cloud assets. Yes, GitOps can apply to non-cloud assets. How many of you have servers in your data center? Raise your hand. OK, you can still apply GitOps there. It's not only for the cloud. Where are the different ways that you can manage GitOps? You can first manage them. You have to then manage them on your local machine. These are going to be using command line tools. They're going to be technically part of your CICD process. So they're going to be in your pipelines and how you manage them. We'll go into some of the details a little later on. And then most importantly, they're going to be in your operating environment where you actually using your GitOps principles, your values, how are they being used? You need to understand where and how they're being used. So your GitOps typically involves the use of sensitive assets. Where are those sensitive assets coming from? Where do they relate to? First of all, they're your source code repositories. You typically need to talk to a Git repository or some other source code to be able to implement GitOps. Next is infrastructure management. How are you connecting to these individual machines that you're trying to target? Third is application configurations. As I mentioned previously, you're going to have your database passwords. You're going to have your external resources. Where are they coming from? Certificate management. Those are all areas that you need to be very cognizant about when operating in production. And then finally, access to platform resources. So if you're talking to AWS, Google, Kubernetes, you typically need to authenticate in some form or another. You typically have to then manage that through Git or some related system. However, most importantly, improper management of GitOps can have very serious consequences. And how serious can they be? First of all, what is managing GitOps the wrong way? Number one, we've talked about it earlier today. Do not in any way put password-sensitive material, you name it, as plain text into Git. Because as soon as you commit it to Git, it's there forever, in most cases. Also be cognizant about the tooling that you're working with. Understand that if you put passwords, are they going to stay even plain text? Are they going to be encrypted? Does your tool support? And then next, understand the limitations of your tooling. So if they don't store them in encrypted values or protected resources, understand those. Understand just the limitations of what your tool can provide. This goes beyond Git. It's any type of feature of your tool. And then don't do security theater. How many of you are familiar with security theater? What is it? Anybody? It's basically going in and abstracting security, even though you're not really being secure. I'm going to go ahead and encode something 18 times. You're not really making it secure. You can just decode it 18 times. That's security theater. So what are some of the implications of improper secrets management? Number one, time and effort. If you're going to go ahead and not only implement it, but do it the wrong way, that's just wasted time. But if you do it or just probably don't do it and you get caught or there's a breach, there's going to be a lot of pain to have to un-wield all that back. You're going to have to not only fix your Git repository, if you're working in GitHub, GitLab, you got to go call them because guess what? They're going to have to help you scrub the data. If you have forks, well, you're in big trouble because that means that all your forks, repositories are now compromised. It's a lot of work to be able to come back from a breach. Next is career repercussions. Or if you want, it could be a career resume, building opportunity if you want, but in most cases you wouldn't want that. Financial implications, if it happens to cause a negative impact on your organization you're working for, you may be sanctioned or fined based on the repercussions. And then finally brand casualty. We all know of a lot of organization these days that have had security breaches. I won't name them, but in the security space we all now know of certain companies that we had no idea that existed before and what they did. There's a reason why we now know about them. So most importantly, what are the proper ways to manage and handle secrets in GitOps? So there are three areas that I'm going to touch upon. One is how you store your sensitive data. Two is how you access the sensitive data, the secrets. And three, once you have access to them, how do you consume them? So when you're storing secrets, there are two primary ways that you will go ahead and put those into your Git repository. One, you will encrypt the values. You'll use some tool, encrypt them, and store the encrypted values inside your Git repository. The second one is to use an external reference. You're gonna take the sensitive resources, store it somewhere else, but just put a pointer inside your Git repository so that when you're reconciling your state, some other tool is gonna be able to go ahead and reach out to some external system and find it at runtime. So the good news is there's a lot of tools out there to implement or to store secrets. The bad news is there's a lot of tools out there to store secrets. Which tool is the right one? I'm an architect and a consultant. My answer always is it depends. But there's a lot of options out here. I always say see what your organization feels the most comfortable with. Some are in the cloud, some are not. You have HeshiCorp Vault is a corporate option that has self hosted and cloud management. If you're in one of the three major cloud providers, each one of them has a secrets management tool. You also have the CI tools, GitHub, GitLab. They also offer secrets management capabilities. You have options out there. See which one makes the most sense for you and your organization. One of the benefits is some of these tools do offer dual capabilities of being a KMS. So you can take advantage of the key management facilities provided by these tools. Azure is one of them. So when you're storing values in your repository, there are certain traits that typically are involved. One, they're mostly going to be either CLI or GUI based capabilities. They do enable these CLI tools to encrypt and both decrypt at the same time. And I'll say it again, this applies to Kubernetes. Now we're all here at KubeCon and CNCFCon. It still applies to non-Kubernetes environments. So even though you may not be in Kubernetes, still applies. There are tools out there for you to store your secrets. A big one out there is SOPs. Another one is the sealed secrets project. So if you're using that, there's a command line tool called KubeVal. There are also other tools that are not really Kubernetes specific, which are Ansible Vault. They're using Ansible for automation and Helm if you're using Helm for package management for Kubernetes. It also has a tooling that you can use to integrate secrets management into the Helm ecosystem. The second option is storing those values externally. Where do you typically store those values externally? One, if you're using in your CI infrastructure, you're going to use Jenkins. There's secrets management capabilities in Jenkins. You're going to have GitHub actions, GitLab. Each one of them has a facility for secrets management. Next is platform infrastructure. You're going to have your IaaS solution, your OpenStacks. It has a secrets management tool. Your PaaS solution might have one. Kubernetes has a secrets management. We'll go into that a little bit later, but it also has one out there. And then as I mentioned previously, if you're using a cloud service, there are options there as well for integrating and storing secrets inside one of those platforms. So how do I reference? Once I go ahead and store something, how do I reference them? So typically you only want to store the referenceable values inside your repository. What are you actually storing in that repository? Well, there are two primary ways of doing it. One is placeholder values or platform-specific values. So a placeholder value is going to be like an environment variable. However, if you're in a certain platform, there might be certain default values that come out of the box. Like for GitHub actions, there's going to be secrets. Whatever the name of the value that you stored inside the platform. So it depends on what operating environment you're currently working in. So that's number one. Number two is how do I access my values once I've stored them? I'm good at storing, but can I access them? So three things that we need to talk about. One, which tools are available for us to be able to access our secrets? What is the access model that we should implement and employ when trying to access those secrets? And three, now once we do have to think a little bit ahead, how are these secrets going to be consumed? Because if we understand how they're going to be consumed, it may determine how we want to access them. So there are various tools out there for us to be able to leverage. They're number one as very similar to how we stored our secrets, we can access them via command line tools. You have the AWS CLI, you have KubeVal, you want to use CL secrets. There's a lot of tools out there. If you're using Kube CTL, you can go ahead and retrieve the values that way as well. Those tools can be standalone or they can be integrated into other tooling. Now, CICD tools like Jenkins GitHub, you name it, has comparable tools to be able to access the values. You have the, for GitHub, you have the GitHub CLI, but most people, myself included, will just use the UI. Just how I am. CI is great because it also accelerates the DevSecOps process because you're also starting to get security involved throughout the process. How many of you have looked into DevSecOps at all in your organization? How many you are planning on using DevSecOps? How many of you have no idea what DevSecOps is? DevSecOps really is an extension of DevOps in a way to get security closer. We're gonna shift it to the left and have security be part of the conversation upfront. Get your security team involved because as I worked with many organizations over time, you can get close to production and then the security guy comes in and says, oh, you need to make sure you implement this or fill out this form and then you have to go back and either redesign something, delay the process of getting security involved is very important at the forefront. Finally is plugins for tooling. There's a lot of plugins out there to integrate secret utilities. They are either extensions of existing systems and they also abstract and support the lifecycle of secrets management. There's plugins for Cube CTL and Helm, Helm has an extensive plugin system. Kubernetes has one as well and also there are package managers specifically designed for plugins. How many of you have heard of Crew for Kubernetes? It's a plugin management system for Kubernetes plugins. It's pretty cool. Check it out. And if you haven't played around with these CLI plugins, the way to do so is to use the tool which is basically Cube CTL, the name of the plugin and then any sub commands that can be part of the plugin. So if you have a Cube CTL secrets management extract and decrypt, I'm throwing something out there. It's just the typical convention of how these tools are integrated. Now I mentioned Kubernetes. There are solutions purposely built around the ecosystem of Kubernetes to integrate secrets management. One is standard controllers. Kubernetes has hundreds of controllers built in and extension throughout the community. These are ones that manage different resources and reactor resources within Kubernetes. You can go ahead and use a controller to apply configurations to values that are based on the configurations of the resources in Kubernetes. Second is webhooks. Now this is where the dynamic nature of Kubernetes comes in. You're able to provide either validating or mutating webhooks to change the composition of your deployments or other manifests as they're being created. So a good example is Hashtagor Vault and their sidecar injector. They install a mutating webhook configuration. When a pod comes up or a deployment comes up and is annotated with certain values, the validating and mutating webhook will go in and say, hey, I see that value. It has a reference to secrets involved and how to connect to it with credentials, et cetera. It's going to go ahead and reconfigure the deployment, add configurations to it that make the lifecycle of managing secrets easier. Third is operators. And I always want to keep operators and controllers separate. People say, well, isn't an operator a controller? Yes, it is a controller. An operator is distinct because there is a custom resource definition that it tracks against. So you would define your configuration against that custom resource definition and that controller, the operator, would go ahead and reconcile based upon that configuration. That's the key thing. So that's a great way that you can manage secrets. And we're going to talk to, we're going to talk to at least one or two projects that implement their own custom resource definition for you to then implement secrets management. So let's talk about the three that I always use in the Kubernetes ecosystem. One is sealed secrets. Sealed secrets is probably the easiest, fastest way to get started with secrets management in Kubernetes. It provides a controller with its own private key that allows you to encrypt and decrypt sensitive values. However, in practice, I have seen that it, there are limitations to sealed secrets in this multi-tenancy. So be on the lookout. It's great to start with. It's great for small projects, but if you have a large team that is using it in a large environment, it's multi-tenant, it can be a challenge because you have one secret key for the cluster. A project which I do love, external secrets. So if you're using an external secrets management tool, a cloud provider secret management tool, a hashtag or vault, you name it. This allows you to integrate a controller to grab resources and access and manage those resources into Kubernetes. And then finally, a third option out there in Kubernetes that I see widely being adopted is the Vault Secrets Agent Injector, which is a great feature-rich plugin for Hashtagorp Vault. It allows you to inject sidecars into deployments and pods as new they need to. You can also then manage the entire lifecycle of secrets within your Kubernetes cluster. So if you're synchronizing secrets from Hashtagorp Vault and you change them in Vault, it will actually go in and periodically refresh the state automatically. And the same thing when it comes to your containers too. Really cool project as well. All three here are great ones to implement. So on the right is an example of how you reference an external secret to reference a vault from AWS. We're gonna go ahead and, pardon me, it's gonna be in Hashtagorp Vault, referencing credentials to talk to AWS. So it's gonna be Hashtagorp Vault. It's basically saying in the backend type is gonna be Vault. We're gonna say we wanna go ahead and reference the AWS key ID and the secret ID, in secret key, pardon me, inside Vault and it's gonna put that in a secret in your Kubernetes cluster. How many of you are using any one of these tools right now? That's good. I mean, it's either not a good adoption and B, you are using secrets management tools and get ops. So that's great to see validation from the community here. Third is a get ops engine. Kubernetes has really moved forward how get ops tooling works. Two common projects out there are one, Flux and two, Argo CD for implementing get ops. It's not the only ones. They're just the two most popular ways of implementing get ops in Kubernetes. If you're using Flux, it has integrated SOPS functionality and by using Flux SOPS, you can then integrate an external secrets management tool. Very easily. Or you can use a GPG key, PGP, because SOPS also supports that as well. If you're using Argo CD, there's an entire plugin system that Argo CD provides and there's a Vault secrets management plugin for Argo CD. So if you're using Argo CD, you can now integrate Haskell Vault very easily. Two great options. Two doesn't really matter what tooling you're using to allow you to operate successfully and securely inside Kubernetes. Next, what we need to access the credentials. What are some considerations that you need to have when setting up how you're accessing those values? One, understanding what options and capabilities are provided by the tools you're actually using. Identify security boundaries. Ensure that you only create credentials and access models that are for your application only. Don't go ahead and create a single principle that's gonna use it for all your applications throughout your entire portfolio. That's just a breach waiting to happen because one breach happens, you have to go ahead and basically rotate your entire ecosystem. Watch out. So recommendations for that. One of them, as I mentioned, use that dedicated service principle. Two, short-lived credentials. Try to use them as short as possible. Have rotation on them. Don't go ahead and create a credential today and then I come back next year or we see y'all y'all in Detroit. I don't wanna have you talk to you and say I'm using the same credentials. We need to have a talk at that point. It's a fun talk. And then, as I mentioned, emphasize that least privilege principle. Only create credentials for individual resources. If I have application A, create a principle for application A. If I have application B, create a principle for application B. What are some common access methods for talking and accessing your secrets? One, the simplest, basic off. It is what it is. They're just an option out there. Two, most common out there is bare token. That's how you can talk to Kubernetes. One of the ways you can talk to Kubernetes. Three, which is really cool. Assumed identity, OIDC. How many of you are integrating OIDC for how you're accessing your sensitive values? Awesome. Awesome. OIDC is great. So if you're using GitHub Actions and AWS and GitLab, they're now starting to support OIDC for you to access external resources. You don't have to have and store secrets. You can use assumed identities on the platform. If you have any questions on this, this is kind of a complex topic. It's really emerging. I'm happy to sit down and talk to you about it. I'm part of another project called Sigstore. We use that extensively in the Sigstore project. So OIDC and assumed identity, look into it if you haven't. You may not be able to leverage it because you're tooling an operating environment and it may not support it now, but at least keep it in the back of your mind. It's a great way to manage secrets management in security. Okay, we talked about how we store the secrets, how we access them. Now the most important part. How do we actually use them? Because we did all the two of the three. Let's go ahead and start to consuming them. That's two common ways that you will consume secrets in your GitOps process. One, environment variables. You're gonna consume whatever it happens to be as environment variables. And number two, through a file system. And the simplest way to do that is a Kubernetes secret. That's pretty much the de facto way of storing secret values and sensitive values in Kubernetes. However, I will emphasize over and over this week. There are limitations to Kubernetes secrets. They are base 64 encoded. Base 64 encoding is not a security mechanism. Don't think it is. You can extract, you can basically decrypt that in about four seconds, not even four seconds. Tens of a seconds. Don't even bother. It's a great way to get started, but just great for development or your local machine. Not for production, at all. So, some ways that you can consume secrets. One of the most common ways to do that is a sidecar. Use a sidecar inside your Kubernetes pod. This is Kubernetes specific configurations. You have a co-located container within your pod that will manage the life cycle of a secret. An init container will go in and communicate with an external secret store and provide those values for your application at startup. Or, in addition to an init container, is a sidecar that is always running in addition to your operating container. This is great if you have a secret that may rotate frequently and you want to be able to synchronize that state inside your container. It will periodically go and talk to your secrets management tool. Many secrets management tool has a bit of an overlap in time where both secrets are valid, which allows you to update the secrets and have the new one now be active while the old one is still active. And then after a short amount of time, the old one will become deactivated so that your applications will have gotten a chance to then take action and start utilizing the new value. If you're using a sidecar paradigm, you're gonna typically mount a shared volume using an empty dur in Kubernetes. They'll be shared between the init container and the sidecar container and your operating container that they both can read. One of the best ways to manage secrets in that model that's the most secure is to load your sensitive values in memory. Read the value from the file system, remove it, then start your container up and basically start your application up so that it's only available at that initialization point and is removed immediately. So if someone gets a session inside your container, they can't actually access it if it's stored in memory within your application. And this, as I mentioned, is a very common pattern to the vault sidecar. Another one aside from sidecar is the Kubernetes CSI driver. This is a project that has early emerged in the open source and Kubernetes ecosystem recently. It basically allows you to not use secrets, not store your sensitive data as secrets values inside Kubernetes, but allows you to mount a volume using the typical supported volume types in Kubernetes to be able to manage secrets. It supports multiple backends. The common backends that you're gonna have in most operating environments, AWS, Azure, GCP, Vault, you can also, if you're a developer, create your own. It's extensible. I'm a developer. I love developing and hacking on things. So if you have some other tool that your organization is using that may not, be one of these three, one of these four or five, you can create your own. And then as I mentioned, it avoids storing sensitive assets as secrets. It will either store it, it'll store it within a volume in the container. There is an option in the CSI plugin to enable it so it will synchronize to a secret if your tooling does require it. So there is that option to do both, store it as a CSI volume, but also synchronize it as a typical traditional secret in Kubernetes. And to implement that, what does that process look like? Three steps. One, like we talked about earlier, we gotta store the value somewhere. Configure the access, how is the CSI driver gonna talk to it? Two, we're gonna create a secret provider class. This is gonna basically use and specify the specific provider that will talk to your backend and then any parameters that are needed by it. So it's gonna be if it's talking to Vault, the location within Vault that that value is stored, the location within Azure's secret manager tool where it's stored, et cetera, et cetera. And then within your Kubernetes deployment manifest, you're gonna specify the volume type of CSI, reference the driver, the CSI driver and then specify the secret provider class that you specified in step two. There is your connection from step one, connects to step two to step three. This is how you really emphasize how you access it in a secure way within Kubernetes. I'm gonna close with preventative measures. Admit it, how many of you have committed secrets to get? Raise your hand. Raise your hand if you haven't. You lie. We've all done it. I've done it. I've had it called GitHub a few times. I've gotten a couple of nasty grams from my organization saying, we noticed something inside your repository there. Gotta have a little talk. You call them up, you feel shame. We've all done it. Yeah, you have to go in and do all of the stuff we talked about earlier. Good news. There are tools there to help you detect when secrets are present. Three of them that I've found that have been very useful out there, the detect secrets project that the Yelp team has put together. AWS has its own secret management detection tool. Also, in addition, GitHub and AWS has scanning utilities that are out there already scanning repositories. So GitHub, right now, if you committed AWS secret, it will probably be revoked automatically because there are tools out there to detect when those are an AWS secret. Try it out. Just for lab environment testing, not in production, just see how that actually works. And the great part about this is security is becoming more mainstream front and center. So more of these tools are starting to become available. So keep on the lookout for more ways of being preventative to ensure you are protected in your GitHub secrets management. And then finally, I will close saying that security is continuous. It's never finished. What you store today might be vulnerable tomorrow. Always make sure you incorporate security from the start. Implement across your entire organization. Just don't have it be in your little silo. Make sure you emphasize security. Throughout every single part of your organization. Revise and remediate as necessary. What happened two years ago in the ecosystem is totally not the same as it is today. The secure software supply chain is mainstream today. I guarantee it was nothing two years ago. And then finally, identify everything you need to do. I wanna thank you for the opportunity today. Hopefully you learned a lot. I'm happy to answer any questions that are gonna be here all week. Thank you for the time today.