 Hi, and welcome to Exploring HashiCourt Vault in Argo CD, the GitOps way. I'm Tracy P. Holmes and I'm a Developer Advocate at CodeFresh. So let's get started. First off, if you're here, you probably already know about GitOps, but you may not really know about Vault. I'll get back to Vault later. That said, my title mentioned GitOps in Argo CD. So what exactly are these things? Well, if you aren't familiar with GitOps, head over to opengitOps.dev, which is the official page of the GitOps working group. In short, GitOps is a set of standards and best practices that can be applied to application configuration and delivery all the way to infrastructure management. The principles of GitOps are as follows. The system is described in a declarative manner. So in practice, this typically means Kubernetes manifests. The definition of the system is version and audited. In practice, it's typically stored in Git. A software agent automatically pulls the Git state and matches the platform state. In practice, this would mean Argo CD or Flux, and the state is continuously reconciled. That means that any changes happening in Git should also be reflected in the system. Now you will usually see GitOps paired with Kubernetes, though Kubernetes is not necessarily required. GitOps can absolutely integrate with other deployment pipelines or infrastructure, including containers and VMs. Additionally, some benefits of GitOps are deploying faster and more often, easier and quicker error handling and recovery, and self-documented deployments. And most importantly, you also get the complete elimination of configuration drift. These benefits can make it easier to handle the applications and allow teams to deliver quality software faster. Now, a GitOps tool that follows this approach of a Git-based workflow is Argo CD. It's a continuous delivery tool for Kubernetes that is essentially a GitOps controller that does two-way synchronization. The Argo CD deployment process is the embodiment of the core ideas behind GitOps that we discussed at the beginning. Remember Argo did on my shirt? All application configuration is stored in Git, so that's usually in a separate repository than the source code. Deployments are happening in a pull manner where the cluster is fetching manifests from Git, so that's instead of traditional solutions where updates are pushed to the cluster. A deployment is a unidirectional reconciliation process between the two states. So what's described in Git versus what is deployed in the cluster. And even though the sync process is vital for performing the initial deployment of the application, one of the true strengths of Argo CD is the continuous monitoring of both states, clustering Git, after the deployment takes place. This continuous monitoring is essential for solving configuration drift, a common issue in organizations with a considerable number of deployment targets. Adopting the GitOps methodology provides transparency between the application configuration deployed in the cluster and the one residing in Git. Alright, so let's talk about secrets. Why are secrets important and can I use them with GitOps? Well, a secret is anything you want to tightly control access to, such as API encryption keys, passwords, or certificates. And well, secrets are important because you want to keep your secrets, well, secret. And yes, you can use them with GitOps. But the truth is there is no single accepted practice on how secrets are managed with GitOps. We can all agree. You probably don't want to keep any secrets in Git where they're left in the open to anyone who has access in your repo. And you definitely don't want any unencrypted secrets in there. Now, I could also argue, and some security people out there may also agree, you don't really want to commit secrets to Git. Now, there's a lot of hesitance in encrypted and committing. Secrets managers are GitOps friendly. You want to commit stuff to Git and make it the source of truth. But honestly, do you want to commit everything? Wouldn't you want your secrets outside of your orchestrator? So in this instance, Kubernetes. And before we go any further, let me get Argo CD and secret management out of the way. Here's their stance straight from the website. Argo CD is unopinionated about how secrets are managed. There are many ways to do it, and there's no one-size-fits-all solution. So as you can see, they have more than a few options for you to look at on the docs page. So let's talk about some tools that I typically hear about or see in a variety of blogs. Kubernetes includes a native secret resource that you can use in your application. By default, these secrets are not encrypted in any way, and the base 64 encoding you should never be seen as a security feature. While there are many ways to encrypt the Kubernetes secrets within the cluster itself, we're more interested in encrypting them outside the cluster so that we can store them externally in Git. Another one is Bitnami-sealed secrets. So the Bitnami-sealed secrets controller is a Kubernetes controller that you can install in the cluster, and it performs a single task. It converts sealed secrets that can be committed to Git to plain secrets that can be used in your application. Sealed secrets solves the following problem. I can manage all of the Kubernetes configs in Git except secrets. So encrypt your secret into a sealed secret, which is safe to store, even to the public repository, and the sealed secret can be decrypted only by the controller running in the target cluster, and nobody else is able to obtain the original secret from the sealed secret. Oh, and we also have SOPs, and that's S-O-P as in PALL-S for those of you that want to search for it, but SOPs stands for Mozilla Secrets Operations. SOPs is an editor of encrypted files that supports YAML, JSON, ENV, binary formats encrypts with AWS, KMSG, all of the acronyms, age, and PGP. TLDR is a CLI that you can use to encrypt Kubernetes secrets, and they also have a section on encrypting using hashicourt vault. Now, there are a few disadvantages. If you use these Kubernetes secret encryption tools, then your secrets are encrypted and committed to version control, which is what GitOps says. Thou shall not use a secrets manager that doesn't commit a secret in version control. But I repeat, as a pattern, and for the other side, it's actually important to not do this, right? The brief disadvantages are questions I've heard from a few people about committing to version control. How do you know who used the secret if it did get used? What happens if you need to rotate it? Do you need to find and replace it across all of your repos? What happens if someone else or someone, period, commits a decrypted secret? These are all valid questions. So let me give you a brief overview of Vault and GitOps. So we finally made it to Vault. If you're not familiar with Vault, Vault is an identity-based secrets and encryption management system, and I don't know if I said this yet. So for you all to know, this talk was inspired by discussions I've had with two people I respect. One is adamant Vault is GitOps friendly, natively, and the other says it's not. So that's why you see both sides in this talk. So what makes Vault not GitOps friendly? But like, is it actually not GitOps friendly, though? Vault is its own controller. It has an awareness of the diff for you. From a security standpoint, you just need to know that you need to change the secret. You don't need to know what the secret actually is. It's trying to reconcile the state of what it thinks is going on versus the actual secret. It's declarative, and it will find something isn't right and overwrite and renew the secret. Also, remember principle two, version and immutable? The desired state is stored in a way that enforces immutability, versioning, and retains a complete version history. Now, there's an argument to be made that while people automatically think about Git when referencing version and immutable, you know, GitOps, does it have to be in version control? Vault does this because it does have access control and weighs the less than the blast radius. Your application and version control history associated with the app don't actually need to know the secret. And if you're using the Vault agent injector with Kubernetes, you're using Vault state store in its reconciliation mechanism, apologies. So your app doesn't need to know what the state of the secret is. See here, I deployed this app to RgoCD without having to commit any secrets to Git. I let Vault handle the rotating, issuing, managing, etc. for those secrets. And it's versioned. I use KVV2, and it's still storing secrets. You also notice there are no secrets related to the database name or password, and the only token you see is the one related to the service account. And, yes, I realized getting things to work was never my issue in my talk, but I was happy that I made it work.gunnet. And I do realize some people would still rather use environment variables. So if you like using environment variables and committed to using them, you can use a template file while you source the ENV. And for those of you who don't want to change anything you're currently doing, like your app, add another sidecar, you can either, one, use Rgo Vault plugin, which works with Kubernetes secrets, or, two, use secrets store CSI driver. I swear I sound like I'm speaking in parcel tongue. And the Vault provider for it. It will provide a volume mount to use your secrets from Vault without a sidecar container. If you have a secret store, you don't need to commit your secrets to a version control, but Vault will automatically detect any differences for you. Part of GitHub says configuration must be declarative. You can't configure Vault declaratively. Well, you can use Terraform for your configs or the Vault config operator, which is a community tool. There are two links on this page if you want to see what the Vault config operator looks or if you want to see what Rgo Sync Wave uses or how it works to sync things that need to reconcile. So, conclusion, why are secrets important and can I use them with GitOps? Well, at the end of the day, my question is, do we really need to commit our secrets in version control? Is that the only thing keeping you from GitOps? Let us know in the chat. Either way, I'm not telling you where I sit stand or fall. Thanks.