 Let's get into the big topic of open source, something that we actually have in front of us. This is so awesome. We are an open culture that is actually in place. It's that process that a developer or, let's say, as the Kubernetes ecosystem really brings. Alright, and we are live. Welcome everyone to another episode of Get Off Sky to the Galaxy. I'm Krisha Hernandez, technical marketing here at Red Hat. I'll be your captain in this journey. We had so much fun with Rosemary last time that she is... We brought her back as another co-captain, right? The co-captain for this journey. So Rosemary from Hashicorp. Rosemary, how are you doing? I'm doing great. After the last one, an hour and a half of vault. I mean, I'm surprised. That's right, yeah. It was a long one. I'm doing great. Thanks, Zimmer, and figure a few things out with Argo CD in particular, which has been fun. Yeah, so it's always fun talking about vaults, right? Because it's something that I get asked about a lot. I'm like, okay, you know what? We should probably bring someone on who knows about vaults to talk about vaults here. I actually have... I don't know what the... I know you're on the East Coast, right? You're in New York? Yes. The weather likes over there, but I have my window open today, because it's actually really hot here in LA. It's actually 80 degrees here on the coast, and I actually live on the coast, so I don't know what the more inland folks are doing, but it's pretty warm here. So if you guys hear any background noise, I apologize, but I do not want to fright a death here in my office here. So I have the windows open. I was a weather in New York. I'm always in a bubble. I just assume the weather's nice everywhere because I live in LA, but I don't know what... Yeah, you're lucky. It was so bad. The past like two weeks, it was below freezing. It was ice. I went winter hiking this weekend, and it was just ice patches. The ground was frozen. And then today, it is 50 degrees and sunny. And so you must have like your LA heat weather or heat or something must have migrated over here. Yeah, transferred over or something. Yeah, well, we can use less of it because it's technically winter and I'm like sweating here. So cool. So I'm glad you're back. I'm glad. We did part one for those of you who are watching and haven't catch part one. That is on the playlist, right? So go ahead and visit the playlist. We'll drop the link in the chat pretty soon. But if you haven't watched that, go ahead. Stay on, but watch that afterwards. You know, Rosemary was kind enough to go over just vault in general. Like you said, Rosemary took like an hour and a half, right? We went over, but it was a lot of good information. So now I asked her to come back, right? Come back again and talk about using vault with Argo CD. So not sure if we're going to take the whole hour. Maybe we'll go over. It's cool. We'll do what we can. So I'm going to ask questions, jump in the chat. You know, you know, Rosemary says, you know, she's a beginner with Argo. So maybe like we can help her out. I don't know. So, so Rosemary, what do you have for us today? So today we're going to answer the question how in the world do you use vault with all of the sort of this get-offs approach. And one of the experiences that I've had a lot of is just, well, injecting secrets from vault and a lot of the practices and the patterns that you're going to see today are very similar to how you would approach injecting a secret from vault into any application. And the great part is that Kubernetes makes it easier because it's adding a layer of abstraction. So you don't necessarily have to think about it. And as a recap from the last time, we talked about how vault works or at least a very, very, very quick introduction of how vault works, how it works on Kubernetes specifically and two ways that you can inject it. So today, what I'm going to do is synthesize some of that knowledge about how vault works, how you should securely interact with vault, but also how you inject a secret from vault to an application you might deploy with Argo. And there are a couple of ways to do it. A lot of the folks that I've talked to in the community who use sort of get-offs tooling do it many different ways. And I'm showing, I think, two variants today. It is your choice. I will come and give some disclaimers, though. So keep some of these things in mind. I'll kind of give a background as to the advantage and disadvantage of these approaches. I know for some folks, they're OK with some of these patterns and that's OK. But other folks prefer a more secure approach. So just keep that in mind. And as a disclaimer, as Christian mentioned, I'm a beginner in Argo. So if anybody gives me some tips and they're like, hey, there's a much easier way to do this, I would very much appreciate it. You know, like I said, a lot of the injection patterns are very similar across the board. So if you see this with Argo, you might consider doing this for your applications as well. Yeah, we like doing a tech exchange, right? So we like learning from each other. I had Scott Rigby. I don't know if you know him from Weaveworks on once. And he's like a helm guy. And he taught me a lot about helm. I was like, oh, you know, you can do it this way. So I'm like, oh, OK, cool, cool. It's always good to have that exchange, right? So definitely. Wrapping the helm shirt. Yes, exactly, which, you know, it's topical because I'm wearing the helm shirt. Exactly. Well, we deployed Vault via helm the last time. And so it's so apropos because we did create a Vault cluster with helm the last time. We are going to once again be using a Vault cluster. It will have one node because I'm using CRC. So you won't get the full highly available experience for Vault. And I've already pre-staged a few things. So if you don't see me walking through and deploying Vault with helm with Argo, then don't be alarmed. It's just because I already pre-deployed it. As a note, you can deploy Vault, the Vault helm chart with Argo. I have not shown an example of that. But if you have any tips or tricks, Christian, on deploying helm charts with Argo, you know, do let us know. Yes, definitely. So that I can help with. All right, cool. So as long as it's not Vault related, I can help. I know, right? OK, so I am going to switch to CodeView. And that way last week we had some interesting, entertaining. Yeah, there we go. Yeah, there we are. All right. There we are. Perfect. OK. So now your resolution should not be blurry. Because last week, last couple of weeks ago, it was pretty bad. Yeah. OK. So I modified our existing repository. It's the same repository as the last stream. J-O-A-T-M-O-N-0-8, Vault Argo CD. I don't know if I have it up on my browser, which I could pull it up if folks are interested, or we can drop it in the chat again. Yeah. But I've modified this repository for Argo CD. And one of the things that I did want to make a note is that I installed Vault with the helm chart. And it's on OpenShift. So keeping this in mind, I did not put this in Argo. You could. I could just use the values and pass it in. But I did not. We're going to really focus on the injection piece. So rather than operating Vault, we're focusing on injecting the secrets to Argo. Let's scroll down. The one difference that I will point out is that if you were tuning into the last stream or you're familiar with Vault in general, remember there was this unsealed seal workflow in which once you bring up Vault on Kubernetes, you have to unseal it. It's kind of like unlocking the Vault quite literally. I reduced the shares. So remember last time we used three keys in order to unseal the Vault. This time I'm using one. So I just set that for the sake of ease of use. So if you run this yourself, you only need the one key to unseal Vault. Don't do that. Yeah, exactly. Well, and also some people will try to automate that. And that's probably not a good idea to automate. And what you're talking about is bootstrapping, right? So that's like a separate thing, like you're fully get off automation. Yeah, exactly. And you could code that into Argo as part of your helm chart. You could define something called auto unseal and pass that as part of your helm values. But you probably don't want to set up auto unseal in production. It's one of those situations that once it's unsealed, people can access your secrets. And you may not want people to go into Vault and access the secrets right after it's gone down, right? So that's one thing to note. Okay. So today I'm going to show two different flavors of injecting secrets into Argo. There is, and both of them are community projects. So thank you, community. You're wonderful. Yay. Yeah, yeah. Those are community projects. Neither of them are maintained by HashiCorp. One is a Vault config operator, which I really enjoy the workflow that it offers. So I'll show that today. The other one that we're going to talk about is the Argo CD plugin for Vault, or the Vault, the Argo CD Vault plugin. Sorry, brain order of operations. Yeah. And shout out to the community, Argo CD community for building that. And there's also the third way that I'll show. I mentioned one of the things that I wanted to raise from the last stream was that we talked about Vault agent injection versus CSI injection. CSI, you may or may not want to run on OpenShift. So we're going to talk about Vault agent injection. And because it's pretty much deployed as part of the application, you don't really need to do much, but I did want to show the pattern. So the first thing we're going to talk about is the Vault config operator. But remember that the last time we configured Vault, we had a bunch of CLI commands. So I configured a Kubernetes auth method. I'll show you kind of what that looks like because I still needed to do it. But remember I had a whole bunch of commands that did like Vault write, Vault policy write, Vault this is, we all, we embrace GitOps, right? So is there a declarative way to do this? And the answer is yes, there is a declarative way to do this. You can choose any declarative like provisioning tool that you choose. It could be Terraform. It could be something else, but you can use something called the Vault config operator. And I really like this because it is one way that you can declare some of the... Let me pull this back up. All of these commands, right? Creating the auth method, setting up secrets engines. All of these things can be done by a config operator. And so the config operator is, I believe, on the community marketplace. I did deploy it already. I'll pull up the command. But one of the things that I did want to make note when you deploy it is that you do need to pass it a Vault token. You can set this up with some kind of bootstrap Vault token if you wanted to. Just dedicated to the Vault config operator for the sake of ease of demonstration. This is using the original Vault token, but you can issue a Vault admin token. For example, it does need admin to Vault in order to set up secrets engines and to set up auth methods. So I'll show a few of the policies you'll need to. But the basic outline of this is that once you install the Vault config plugin, it's pretty much up and running. I'll pull up the install at least so you can see what the install looks like. This is using OpenShift. So it is using that subscription object and adding all this information in. So we already got a question. So real quick, this is specifically to Vault. The question is, is Hashi Vault still paid for replication, for example? So I imagine that the question is about Vault and then HashiCorp's relationship to Vault. I imagine it's similar to Red Hat where it's like, yeah, you can use the open source version, but if you want support, then we just go to HashiCorp. Is that in my right in that assumption? So there are, and the question also asked about replication. So I'll answer that question. So an open source version of Vault and then there's an enterprise version of Vault. The enterprise version of Vault does support performance and disaster recovery replication, disaster recovery replication. The idea is that replication is very difficult. So if you want to, when you want to federate and manage across multiple Vault clusters, it is an enterprise feature. And in that case, you do have to have a license to do that. But something that has come out recently is HashiCorp Cloud Platform Vault. So if you don't want to necessarily run and set up Vault Enterprise yourself, you can use HashiCorp Cloud Platform Vault and that is a managed Vault instance. Yeah, like a SaaS service, right? Like you just use Vault as a service. Awesome. Yeah, it's neat. It's neat. Yeah, that is neat. Yeah. Well, I mean, I imagine so like, I'm like, I always like switch hats, right? You know, sometimes I have my Red Hat on, but sometimes like, I'll put my customer hat on. It's like, you know what? If I had a small team and I needed something like Vault, that's definitely something I would use, right? Because it's like, okay, you know what? Like I can just offload that. That's being managed, taken care of. I have support that's, that's actually really awesome. So I didn't know that. So I learned something. Yeah, yeah, give it a try. It's great for if you, if you want to give Vault a try and you want to experiment with it, especially with multiple Kubernetes clusters, it has a lot of flexibility to do that. And if you're familiar with sort of the Vault constructs, you know, you can set up everything that you do in your open source Vault. For example, you can kind of map it to the HashiCorp Cloud Platform Vault, right? So it all sort of maps really neatly together. And, you know, as a result, it allows you to spin up a bunch of Vault clusters and try out different things that you wouldn't otherwise be able to do in the open source Vault. For example, namespaces. Awesome. Yeah. All right. I'm going to go back to code view. So that way everybody could see this. Okay. So let's talk about the setup for the config operator. I already started up the configuration, the config operator and it sets up a bunch of CRDs. So a bunch of definitions for various, various kinds of Vault constructs. So remember the last time we had our application, our application access to database. The database had a root password and a root username. And we had Vault issue, kind of like limited access usernames and passwords for applications to use. So remember two sets of secrets. There's the static root password and root username. And then we had the more dynamic application specific username and password for the database. And what's pretty neat about this is that you can use the Vault config operator to use these custom resource definitions to define those secrets engines, right? So before we were using Vault write and choose your path and it was a little bit strange with the CLI. Yeah. If I wanted to configure something, I had to remember to delete things. It was a little painful. So now we can do this with a bunch of custom resources. But more importantly, we can commit this to version control and have Argo handle the deployment and handle some of the reconciliation of some of these resources. So in this case, remember my database root username and password. I configured as part of a key value store. So in this case, I say, hello Vault config operator. I need a secret engine mount and it is, I'm going to call it static at the path expense. So when it mounts the secret, it will mount it in Vault at the expense slash static path. And the type is KV. I'll explain why the type is KV. Vault has two different types of key value or KV secrets engines. There's version one and there's version two. Today I'm using version one because there is a neat custom resource definition here and it only works with KV version one and not KV version two. So keep that in mind. It will trip people up and the main difference between KV version one in Vault and KV version two in Vault is that there's some restructuring of certain paths and KV version two offers versioned static secrets. So that's why you're going to see a difference between the two. It does trip people up. The other thing I did. Oh, go ahead. I just like that. Like this is all declarative, right? So like, you know, last week you showed us the imperative way of doing it and it's like, okay, well now you can see how you can actually do this declaratively, right? Some of those manual steps, right? Because in my head I was thinking, man, I'm going to have to start scripting all of this. But it's really cool that you can do this declaratively. Yeah, it's great. I really like it. Usually I would otherwise do it in Terraform. And you know, there's a benefit to it. But on the other hand, if I'm already configuring my application, for example, using Kubernetes, there are some things that I would perhaps want to declare using a Kubernetes declarative format, right? Because something like secrets engines, maybe they're specific to the application, right? And you're, you know, if we're embracing sort of a DevOps culture, you would say, hey, application teams, we'll give you just enough access for you to configure your secrets engines and auth method, at least secrets engines at certain paths, right? And so this is what we're doing here. It's the idea of saying, hey, expense application team, you're empowered to now add your, add your passwords, et cetera, to vault using something you're familiar with, which is a Kubernetes manifest, right? So it's kind of neat. The other neat part that we didn't, oops, that we didn't show last time, but I was so happy to see this custom resource. Yeah, I was. So last time you remember when we did the imperative step by step, I passed a static password and everybody saw my data, this time, no one is going to see it because I am setting a password policy and creating a random secret from it. So why would I do that? I know, right? This is exciting. I'm so glad this custom resource was implemented. So in vault, there's an idea of a password policy and you can declare the kind of password you want. So if you're doing something like a managed database, most providers have a very opinionated password policy, right? You must have the alphanumeric and a symbol, a valid symbol that's not like a tilde or a backslash, for example. So the important thing is that, you know, you want to set this perhaps on certain passwords in general and in vault, you can do that. So you can declare a password policy, which is what I've done, and this creates a password policy in vault. So here it will create a password policy with a password of length 12 and use sort of these rules, including character sets and some symbols. So all of us, like once again, done by Kubernetes manifest. And if you are interested in doing this by vault, imagine you're an administrator and maybe you want to preset this before anybody manipulates it on the application side, you can set it up in vault to do that. All right. So finally, there is a random secret. So this is a really neat one. This doesn't create, well, it creates a random secret. So the config operator will create the random secret based on the vault policy, vault password policy, and then insert it for you into vault, which is really neat. That is really cool. Yeah. Yeah, exactly. You don't need to be aware of the root username and password. The reason why you might think about doing this in particular is that part of this generation, you know, password generation workflow is maybe you want to put this with the application, right? And you want to be able to reference something like this random secret, you know, all some of this information. And you don't necessarily want to generate the random secret yourself. You shouldn't need to know what the root password or username is, right? You might want to generate yourself. So in this case, this is kind of neat. It passes in a path as well as some of the password policy information. So this is what the password policy we defined above. And then some of the keys as well. So what this effectively does is store a random secret, store a database root password at the expense static slash my SQL path and it will use the key password equals. So I probably should not do this. What's, what's, what's really cool, right? So like, you know, just thinking about this more abstractly as like just a vault as a platform itself, right? Like, so even the administrators don't know what the passwords are, right? So this is really cool with like in terms of compliance, where it's like, you know, well, who knows the password and you can check off. No one does because it's, you know, managed by vault and it's all, you know, randomly generated and randomly, you know, a cycle. So we do have a question by, from IDD, by the way, who says that it's a nice step by step explanation. So he sends you kudos. But the question is, is it possible to get password without small letters in this policy, right? So I guess, I guess you just have, it appears there that you just have capitals and lowercase, right? Both? Yes. In terms of rules, okay. Exactly. You can take out a rule. So this, you'll notice this is, this is a, in the YAML, it's an escaped string, right? So it's, and, and it is declared in what is like as the HashiCorp configuration language format. So that's sort of an underlying configuration language for vault and many of the other HashiCorp tools. You can update this rule set so that you're only, I'll even, I can show by example, but you can say, I just want capital letters and, you know, the numbers and the symbols. And so that way you don't have to use the lowercase. Cool. Awesome. Yeah. You can change the length. You can change whatever symbols are valid. So it's pretty, it's a pretty neat thing to do. Okay. So let's, I don't think I can do this easily to be honest, because they're all very, very tightly coupled to each other. So we might have to hold off a bit on deploying those custom resources. Yeah. Yeah. We're going to hold off because there's a bit of an order operations here. There's a chicken or egg the way that this repository is set up. So if you follow the make file, you'll have every command in order, but they are a little bit, you know, there's a lot of complexity because there's three different methodologies kind of squished into one repository. Okay. So we talked about the vault config operator. I showed it for the root password for the root database password at least. I'll show perhaps the secrets engine for the database and then we'll deploy it after another explanation, but for let's say the more dynamic secrets. Remember last time we configured a database secrets engine and the database secrets engine connected to the database and created users and passwords. And then would effectively go and grant privileges on certain tables and then revoke them afterward. So we actually mimic the whole thing once again in a declarative form, which is much easier. Yeah, exactly. For example, secret engine mount very similar to before. Only difference is that this time I'm saying type database. The other construct that you'll need to define is a database secret engine config. The config defines the plugin vault plugin you're going to use. We're using my SQL database plugin today. So I declared my SQL database plugin, the plugin name. I also allow a role. So it's a vault role. It's not a Kubernetes role. It's not any other kind of role. It's a vault role. So I'm allowing the vault expense role to access this database secrets engine configuration and thus the credentials. So that's important to recognize. The other thing that I've set up is the connection URL. You'll remember in the last stream on vault or if you're familiar with vault, you need a connection URL for vault to connect to the database. In this case, I am able to connect by a templated username and a templated password. And actually my favorite part about this that I discovered was that thanks to the vault config operator developers and you can set your root password using and by referencing the random secret that you used to generate it, which is actually really elegant. Yeah, it is really elegant. I was going to say that's really awesome. Yeah. And so it's fully in some ways you don't have to add the password in here. You don't have to add the, you know, well, in this case, I did add the username, but you could declare the username as well if you wanted to as part of the static secret. In this case, you know, I was part of this, the knee part about this resource is that you can pull the root username into here. It will pass it as part of the template and then the random secret, if you've declared it, gets passed as part of the password. So that's really neat. And you don't need to know what the, once again, you don't need to know what the password is. And similar to the last time, I know someone asked the question, but vault does on certain secrets database engines vault does rotate the root password if you allow it to. So if you rotate the root password in vault, it will continue to allow the database secrets engines to function correctly. Right. So if you rotate the root password, it will auto insert it into this connection string for you. And, you know, that way you don't have to declare anything about your root password at all in any of your applications. Yeah. Yeah. So like you don't even have to, you can just store this and get, right? You don't have to worry about like encrypting user passwords, whatever, just like, hey, it's automatically taken care of for me by the platform. That's really good. Yeah. I mean, you know, you may not still want to do that. You know, someone who knows someone, someone may forget to rotate it. But, you know, as a, as a general rule of thumb, you know, if you, if you have it and you do tend to use a root, a static root password in general, don't put it anywhere. But if you have a very ephemeral policy, like, you know, once it's been created, you have some kind of declaration here that says rotate it, then, you know, it'll be fine. All right. So then the role is what we declared before. Remember, the role is a create, creates a, the username and password using a SQL statement. And this will grant privileges on a table. So in this case, I have an expense item table that I'm granting privileges to. And this is something that, you know, again, I wrote, I did a vault, right. And I had to do all the imperative, imperative, you know, templating and everything to do it. This time I just write it all in a manifest. And I could, I can already see like everything that you did last week, like you're just doing it now declaratively, right? Instead of, you know, you're basically just translating those imperative commands onto, you know, these custom resources. Yep. And remember, you know, there was a vault has a policy system. And so it's API authorization at its core, right? It's telling vault, you are allowing this identity to have access to this API path. And a lot of people complain that it's hard to manage policy. I completely agree. Access control is difficult. It's nuanced. It's hard to figure out. And so it's kind of nicely be able to say, as part of your application code base, you can bundle this policy with the code base, right? So you can very clearly say, my expense application has access to read dynamic database credentials from vault. So this is what this policy is expressing. If someone's like, why can't I get the, why can't I get the database password in my application? They can come and look at this policy resource and examine what their policy is. I mean, now there may be other policies that might be overlapping, but for the most part there they get visibility into debugging the kind of access control that they have. That's really cool. Policy as code, and then, you know, since it's in code, someone can reference to it, right? And that's kind of, you know, if I could step on the soapbox of GitOps, right? That's like one of the biggest thing that GitOps allows you is the visibility into what is in your cluster, right? So like, so me and Dan Garfield, so Dan Garfield from CodeFresh, we have like this running joke that we say, unless it's in Git, it's only a rumor, right? Like, you know, like what's happening in, you know, what's happening in production? Well, unless it's in Git, like who knows, but like if you're in Git, at least you can see, you know, you can interact, right? With, with different groups, right? So the security guy put this in place and the developer can look and say, oh, that's why I can't, you know, log in and then they can have a conversation of whether or not, you know, a new policy needs to be written or if it needs to be allowed or if the developer needs to do something else. So that's really, like you said, visibility, it's very important. So I'll step off on the box. No, no, it's a great thing. And I get a question. I get this question a lot actually about policies code and let's say using OPA with this, right? Yes. If you, if you wanted to control, if you wanted to use, you know, Kubernetes constructs to control, let's say like Vault access or you wanted to be really prescriptive about the kinds of roles you're going to assign, how they're named because it's helpful to be really opinionated about which identities in Kubernetes are allowed to retrieve secrets from Vault. You know, you can use OPA, you know, OPA rule sets to go in and control that and audit that as well. So you now have like two sets of, you have sort of the initial policy that someone's going to set and then you have the continued audit and the continuous evaluation of whether or not the active environment conforms to those policies too. So it's a neat way to get both and you don't have to do multiple roles to declare. Yeah, exactly. Okay. So at least I showed what the Vault config operator will help us with. But I do want to go back to one thing. You'll notice that I still have a script that installs and updates some things in Vault. There are some things that I don't recommend you, you know, there are some things that you can use the Vault config operator for and, you know, you can do this with Vault. But imagine that you have a Vault sort of this subject matter expert or you have a security team who's administering to this. They want to set up certain policies to start, right? So this is the bootstrap policy. So we talked about bootstrap vault, but you also have to bootstrap vault policy too. There are some specific things that you want to enable. So one of the things you want to enable is the Kubernetes auth method. You could do this with the Vault cluster. In this case, you know, let's say outside of the Vault cluster, you know, an admin and operator is going in and they're going to enable a couple of important policies. There's a Vault admin policy, which is what I mentioned before. Vault config operator does need admin access to Vault. So there are some end points that it needs access to and I'll be sure to show that. And then it has its own specific function. So remember before we talked about auth methods being linked to Kubernetes service accounts. What I'm doing is linking a Vault administrator service account in the application name space expense to Vault, right? So it's allowing the Vault administrator in the expenses or application name space to configure Vault using the operator. Why am I separating these policies? Why not just give Vault config operator and every single service account Vault admin access? The point is you probably don't want to give every single service account Vault admin access. You do want to give it some least privilege policy. So setting up individual roles for service accounts that are allowed to configure Vault is probably a good idea. So it's really cool to have a service account in the application name space. So it's really cool to have a service account in the application name space, even if it's a service account that actually compromises one of these accounts. So in line 21, so now I have a question. So in line 21, I see that you have a TTL. So does this binding policy or does it last for an hour and then it just access to Vault in order to configure stuff. Now, once it's over, if you allow it to reauthenticate and within Kubernetes, you can allow it, it will do it automatically. But if you run this again, like let's say Argo CD decides to reconcile or sync some of the config from the Vault, let's say the Vault custom resources back to Vault, it will handle the reauthentication under the hood. You could tell it not to do so. So you can just do it once and then revoke. But if you have something that's more continuous, which is what we're going to do, if you have something that's more continuous, then you can allow it to do it by itself and Kubernetes will reauthenticate. So what this means is that when the service account authenticates to Vault, the service account gets a Vault token back, right? That Vault token, if someone goes into Kubernetes and extracts that Vault token, it's only available for an hour. After an hour, it will get revoked. And then you get a new Vault token entirely. The service account has to reauth to Vault. So all of that is handled without you knowing it. That's really cool, especially for bootstrapping, right? It's like, hey, I just need this to happen in bootstrap, but I don't want to remember to take it away. And so this is really cool that you have some sort of a platform that handles that policy for you. Yeah, it's very convenient. And so it's called TTL, Time to Live, and the conceptual structure in Vault is called lease revocation and renewal. So if anybody is interested, you can look in more into it. It's very complicated, actually. So what vault policy am I giving to the expense application? So remember, I did say lease privilege. You want to give everybody, you don't want to give everybody all access. That's a very bad idea. So what we're doing is we're limiting the amount of access that the expense service account can use, not the expense service, the vault admin service account can use to configure Vault. Now, why am I using a separate service account? It's because that way my security team can audit it in Vault, right? The expense application can do many things. It could read credentials, but maybe I don't want the application itself being able to write credentials. So I want to separate these things. I want to really identify who is configuring stuff in Vault versus who is reading stuff in Vault. So in order to separate those two identities, I created a Vault admin policy and created a separate service account for it. So some of the things you'll need, especially if you decide to use the Vault config operator and you're assigning it to an application to create things, you'll need to assign it, of course, access to a path. You'll need to assign it access to create mounts and delete mounts, read mounts. So all of these things are pretty wide policies. You can narrow them. Yeah, I noticed that it's like a crud, right? So it follows that crud thing. But I always wonder this because this is completely different no matter what you use, right? And I'm just thinking technology just in general, right? Does one verb supersede the others, right? Or do you need to specify? So for example, in order to update, do I also need to provide read or does update include read, right? Update does include read. But it's sort of maps roughly to like a create would be a post, a read would be a get. Update would be a put. So update to a certain degree has a limited read, right? Implicit limited read, but you can't use it to read all information. Gotcha, gotcha, gotcha, gotcha. So like read is like specific, like read is like, you can read all of this for 9.25, right? Like in policy, you can read all policies, right? Whereas like an update, you can read the one that you're updating only. And it looks like IDD. So he's very active here. I has a question is like, is there a way for one user to delegate to another some like some sort of, I guess this has to do with permissioning, right? So can you delegate permissions for another one to do? So you cannot necessarily delegate permissions. You can assign multiple policy sets and the wider policy set will supersede the more minimal one. So in terms of like inheriting or like a sort of a delegation of permission, I don't believe there is one, you could possibly mimic it a bit, but Vault is pretty much like a flat policy structure for the most part. Gotcha. I know this is very painstaking, but it's very, it gets a little bit dense. Yeah, well it's important to understand, right? Like I'm taking notes here. This is at least hopefully giving you some ideas about how to structure some things. In general, from a Vault perspective, the more fine-grained you are in assigning roles and identities from your Kubernetes cluster to the associate policy of Vault, the easier it is to audit and control who has access to what, right? So it works to your benefit to set up least privilege in the first place. It's harder and it will take more effort, but it does help a lot more from an audit perspective as well, which is kind of what we're doing here, right? So the other method that I briefly mentioned, so we talked about Vault config operator setting up the secrets, but we also have a way of injecting the secrets, right? Into Argo CD or our application. In this case, we're going to use the Argo CD Vault plugin to set up a secret for the application to use, right? So it's using Argo CD to set up the secret. And in this case, Argo CD, the plugin needs its own permissions. So you'll notice I set up a special service account for this particular plugin, for the plugin to access Vault, because I want to be able to audit that. And I bind it to the OpenShift GitOps namespace, which is the default namespace that the Argo CD operator is deploying to. So I'll show the policy for that, so that at least folks have an understanding. In the case of the Argo CD plugin, I limited it just to read from the secret. And why is that? Well, the Argo CD plugin that I'm going to, the Argo CD workflow that I'm going to show you today will be done on the database root password. I don't need to give it access to that dynamic password, right? I don't need it to, you know, have the injected into the application. I just want to inject the root password to my database. So in this case, I only allow the Argo CD plugin to access the static password for the expense database. I'm not going to add more policies. It only needs to read it. And so from a standards view, I mean, this is a way just to make sure that your plugin is limited to just the secrets that it needs to read. Cool. Okay. Sorry, there's a siren going off. So if folks can hear. Yeah, no worries. Yeah. So what happens? Okay. So we're going to deploy just for, for good measure. I'm going to deploy the, let's see. I'm going to deploy that set up all these surface accounts and auth methods and whatever. Okay. So I'm going to set up all these accounts and whatever that way we can get all this configured correctly. You'll notice that I am doing this imperatively. You can choose to do this with a vault config operator. You can do this outside of it. I'm just doing this to demonstrate the separation of concern from a vault administrator perspective versus sort of your applications as a consumer perspective. So they'll also notice that I did this tricky thing where I applied and then deleted and then applied a argocd.yml file. So I'll show this because this is pretty important. So the argocd vault plugin is a community plugin. And so a big thanks Christian because you sent me some docs to help me install it. It was wonderful. Cause I was like, I can't figure out how to install it. You install it as a plugin, as an argocd plugin. You can add that to the argocd config as config management plugin. The name is argocd vault plugin and you run it by generate, right? So what it will do is start a bit of code to retrieve information from vault and then create it as a Kubernetes secret. So that's the general workflow of the vault. Plugin for argocd. One of the things, let me see if I can pull this up. One of the things that I had learned was that you have to, you can either add it as an init container or you can build it, build the plugin as an image. I found it much easier to build it as an image rather than use the init container. I don't know if there's a advise practice around that, but. I think I know our internal, we have what we call community of practice, right? So it's essentially people like me, other architects, consultants, SAs, like we all kind of get together and we say, like, what are you guys doing? It's kind of like a tech exchange, right? Like, what are you doing? What am I doing? Most people actually do what you did. It's like, let me just build an image that has all this instead of using an init container because it's just easier, right? It's just kind of easier if it's like included as part of the package here, right? So there's actually a really good question, somewhat related, right? Valentino says, hello, I know we can inject the Argo CD Vault plugin through an init container, but isn't the Argo CD deployment under GitOps operator control? What happens in the case of an operator upgrade? Yes, that's something you do have to keep in mind, right? When there's an upgrade, that image is going to be injected, right? And so this is like more of an open shift question, right? So like there's the OLM operator lifecycle manager that handles the upgrades. So if you're gonna build your own image instead of using an init container, right? If you're gonna build your own image, I would set the upgrade policy. So there's like an upgrade policy instead of automatic, set it to manual and do a manual upgrade, right? And you really can't really get around that too much because it's just gonna update the image, right? Because in building your own image, you kind of take that responsibility of keeping that image up to date yourself. And so that's just kind of the background with that, with OLM and that's kind of like the trade-off you have. I always say you have a trade-off, right? Install plan, yeah, exactly. There's always a trade-off between fully automated and fully customizable. That's like you have to like pick one, right? Right, like yeah, like you can customize it, but like then you kind of lose all the automation that cause like, obviously you can't code for every snowflake, right? We do our best here at Red Hat, but you can't code for every snowflake. So that's where that ends. Yeah, it's like there's only so much bespoke automation that you can get away with before it breaks your upgrade. Yes, exactly, yeah. Because yeah, if you want fully upgrade, yeah, it's all about balance, someone says. So it's all about balance. Balance, all right. So it's making a balance. Okay, all right. So the Argo CD plugin, yeah, so it was actually a very interesting thing to dive deeper into. I actually appreciate the explanation, Christian, because it was a good exercise. In terms of the other things that you may consider, including as part of the plugin, the plugin has some nuances in terms of configuration. I can pull it up at least. The plugin we're going to use for the database. Why are we just using it for the database root password? It is because there are some nuances to it. It does synchronize, like you will, you can change the database password and the plugin will respond and recreate a new Kubernetes secret, but it is a Kubernetes secret. And so in some situations where your applications may not be able to use the injector, this may be the way to go, right? You could opt to do other kinds of sealed secrets implementations as well, so that you don't have the secret in plain text and you have some kind of encrypted capability, but in this case, I'm just going to show the vault plugin itself and what you're going to do with it. But the vault plugin I configure as part of an Argo CD application, you have to pass a couple of important parameters. First is Vault ADDR. Vault ADDR is, fun fact, the standard environment variable that we use for declaring vault address and the endpoint that you're going to access. So if you use the Vault CLI, you will use Vault ADDR. If you use a Terraform provider for Vault, you will use Vault ADDR. And thank you, plugin, for sticking with Vault ADDR. We love standards, right? Yeah, consistency is always a big help. Exactly, so it's great. And then the other thing is that there's a K8s role. So remember, I showed how a Vault administrator, perhaps configured a special Kubernetes role for the Argo CD plugin that gave it just enough access to my static secret. And so this is the role that I'm going to use. You have to tell the plugin which Vault role you're going to use. The other thing, two things you have to say to the Vault plugin is that you have to tell the type. There are other types which I was really impressed with as well, it's kind of cool to dig through some of the other secrets management roles. Yeah. And then there's an auth type for Kubernetes. So the auth type is about the Vault auth type. There are some other Vault authentication methods that are supported by this plugin. For example, GitHub supported, which is kind of cool too. Especially if you're doing GitOps, you might consider GitHub retrieving from GitHub. Yeah, it's authentication from GitHub, yeah, awesome. Yeah, you can set up Vault to do that. So it's kind of nice that you can configure it to go through Vault and use a GitHub auth method. In this case, because we're using the Kubernetes service accounts, we're sticking with Kubernetes. Yeah. Yeah. And so then we'll also have a Vault, Argo CD Vault plugin. So now here comes the tricky part. And this is the part where I kind of got lost in the order of operations the first time I did this. We're going to create, why am I looking? I don't know. That's fine. It'll be okay. We don't need it. As long as I was asking if SSO is possible, I assume yes, right? If you're using Git. If you're using Git, yeah. So you did get this great question from some folks about SSO and I will publicly state the answer. So it will depend on your IDP, your identity provider. Vault does not offer sort of an SSO capability. You can layer it on top of it. So there's Vault auth methods and the auth methods integrate with different identity providers. So if you have, let's say, OpenShift SSO or something, it will integrate with, I guess, OIDC. So OIDC, JOT, yeah. So keep that in mind. Okay, so OIDC is essentially like, within the answers, yes, right? So like I can use DEX and then broker that with whatever identity provider I want. So that's pretty cool, yeah. Yeah, it's really neat. I actually kind of appreciate the fact that it's not so, in some ways it's a kind of agnostic, right? So you can choose however you want to authenticate to Vault. Vault itself does have two-factor auth. So if you do want to add it, you can do it. It's an enterprise feature. But most folks do the two-factor through their identity provider. To their own identity here. Exactly, yeah. Okay, so let's see what happened. All right, so this is, I'm going to go to the browser. There's probably a better way to do this. But there's a bunch of secrets now. You'll see I started syncing the secrets first. Like I said, order of operations kind of concern right here, right? I wanted to create the database root password first before I create the database, right? Because the database is going to reference it. The reason why this is kind of the important place to start is that all of this sets up the policies from using the Vault config operator. So when you see the expense secrets application, what it is doing is saying, let me set up all of the policies that are available. You could bundle the secrets with the database if you wanted to. There are a couple of different ways you can organize and in some ways restructure how you're deploying which sets of resources. But I just deploy them all at once over here into secrets. So it's easy for us to identify these other secrets and not just, you know, something else. So I've deployed the secrets first now and then we're going to deploy the database which everybody, I don't know if it will work. Yeah, it's cross our fingers, the demo gods, right? I know. It will probably get very sad at me, which it did. That's fine. Yeah, so what why are you doing that? So like for those of you watching who haven't seen other episodes, right? Cash pass episodes. If you have an order of operations sort of things, you can do something like what's built into Argo called sync waves on hooks, right? Where you can wait, you can wait until a resource is healthy before progressing to the other one and then you can actually kind of have a meta, what do you call it, app of apps where you can wait for an application to be healthy before another application gets deployed. So, so I talk about that in another stream. So again, this kind of selfish plug on my own show to catch pass episodes because I have talked about that in past episodes. That is great. That was, that's very helpful because then basically what I would recommend to anybody do if you go back and maybe this is an exercise for everybody to go back and do homework. Homework, Rosemary's give me a homework. Yeah, homework. Yeah, so in terms of order of operations, build this app first, which is your random secret, your database root password. What is that, add a sync wave, right? For the database itself. Yeah, sync wave. Yeah, sync wave for the database itself. And then you can do another app for the expense secrets because remember the expense, the secrets for the application itself depend on the database. So it goes database root password, database, dynamic database password for the application. So you can do that. There you go. That's your sync wave, zero, one, two, three. Exactly, exactly. Like I was like, I ran out of time to implement that because that would have made life so much easier. All right, let's see if this is, if the pods are gonna come up. It's in the chat. So for, thank you producer, Stephanie, put that direct link in the chat. So sync waves and hooks. Excellent. So you'll notice the database came up and the reason why it came up, it retrieved the database password from, let's see, expense DB MySQL. So remember what I talked about before, which was the Argo CD plugin will retrieve information from Vault and then write it as a Kubernetes secret. So that's where we got here. And we said, we're going to retrieve the static database password from Vault and then make it a Kubernetes secret. And as such, we have made it a Kubernetes secret here. I will not print it out because the last time I shared it. This is, I mean, for folks who are trying to move away from this, this is not, you know, it's a tough thing, right? You have to make this decision. You can either control the kind of secrets that are being stored. The other option is potentially to do with shit as sealed secret instead. But if you're looking for something that's kind of more straightforward, it's a, I would say like a second phase approach, right? So first phase is using playing Kubernetes secrets, not rotating them, I mean, managing them outside, you know, kind of separately. Then you say, like, well, I have Vault. I want to be able to issue and control dynamic secrets. Then, you know, use the plugin and then that way you keep your, you preserve the contract of your application retrieving information from the secret. So what I mean by the contract is that if you examine the database deployment, the database deployment is referencing that Kubernetes secret. So if your application is really used to this workflow of accessing from the secret, from a Kubernetes secret, this is a way for you to start using Vault or at least use Vault to control and issue and revoke and manage some of these static passwords and slowly migrate to this second phase, which is what I'll show. But this is kind of a good intermediate step. If you don't want to break everything, because as we pointed out, it's about balance, right? You don't want to refactor a million things in your application deployment, only to find it's broken. So start here, keep the contract of the application retrieving the Kubernetes secret and then move on to this next phase, which I'll show. Okay. The big reveal phase. Yeah, big reveal phase. Now, my question is how do I sync, make sure I sync everything? Yeah. It's like I started it before and then I don't know what happened. So, all right. Yeah. Tiger says refactoring is always fun. Right? Yeah. All right. So one thing I will mention is that I deployed the expense, the MySQL12 early, right? Remember I mentioned that the database didn't exist yet. So thus, we kind of had some issues. So the question is, how do I re-synchronize so that this replaces it? And I don't really remember how I did this. Yeah. If it's in get, then you would just press the sync button. Yeah. If it's in get, yeah. Let's see if this will let me do it. I think it has a access policy. That's why it won't let me do it. Ah, I got you. Yeah. That's the downside. Yeah, that's why it doesn't like me to do this. That's okay. We know what we're going to do. We're going to revert back to something. Where is this? Live demo. Live demo is always cool. Yeah, exactly. We're just like, you know, it's been a day. Okay, we're just, I'm going to log in this invalid username or password. Yeah, live demo, what can go wrong? We don't know. Right, we're going to go back. Right, so hold on. I'm not going to show the mice terminal right now because I am currently retrieving some secrets. So everybody, hold on for a second. Yeah, hold on, hold on tight. Yeah, hold on tight. I promise we will get to the secret eventually. But I don't want to show it on screen because you know, that's one of the tricky things, right? Okay, let's see. Let me just do this. Hopefully this works. So let's go back to this. Okay, so I'm re-synchronizing everything. The kind of the difficult trick to this is that when you deploy it out of order like I did, you have to check that all of these resources are actually working correctly. What I mean is that remember, you need the database to be up. So if we had the sync waves, we would have been fine. Unfortunately, you'll notice that it cannot verify the connection, right? Because it doesn't have a host. So it's not happy. And then it did actually reconcile, let me see. Yeah, so it just reconciled. That's good. There you go. Yeah, very happy. Okay, and then what we're going to do is deploy the application. So I'll deploy the application, but I'll also walk through a little bit of this, the second phase. So the first phase was use Argo CD. Make sure you don't break your applications, refactor step by step, right? So use it, write out the secrets to this group of identity secret, make sure your application doesn't break. The more ideal scenario is to use the VaultAgentInjector. We went through this last week, but I'll at least briefly show why kind of what it looks like instead. Remember, last week we talked about using a VaultAgentSciCar. The VaultAgentSciCar allows you to kind of inject this VaultAgent. It checks for information and changes in Vault and then synchronizes them to a configuration. So in this case, remember, I showed a Spring application that needed to access a MySQL database. So this is a Spring application properties file. And what it's doing is retrieving information about username and password. This pretty much stayed the same from last week. The main difference was that I used Kubernetes to configure the database secrets engine instead of doing it step by step, CLI imperative. So nothing changed here, right? But the idea is that at the very least, you don't commit your passwords and your usernames in Git. You also allow the application to dynamically respond and address to changes to secrets. If you're very familiar with application properties, in the case of the .NET space, some of the other kinds of like good file formats for setting configuration strings, this is something that you can do. Yes, it's not environment variables, but it's at least something that you can say, read from a file, read from a config file, or you can use CSI, the CSI secret store in order to, sorry, secret store CSI to retrieve secrets from Vault and then set them up as environment variables to mount and inject into your application. That's one way if you want to continue using environment variables, but in this case, you can inject using config file. So you have choices is what you're saying, so that's good. You do have choices. And if this doesn't come up, that, oh, it doesn't come up. Okay, so you'll notice the difference. Oh, there you go, yeah, the app is up, yeah. Yeah, the app is up. So the big difference, and that's kind of cool, right? All I did was sync everything up. I mean, I guess I could make the sync interval a little bit faster, but all I had to do was sync everything up and thanks to the config operator, it's checking and making sure that it's got the database config, secrets engine configured and the role set up. So my application, when it's ready to come up, it's able to use it and run with the Vault agent in place. So you'll notice there's the difference, right? So this has got the Vault agent running. That's why it says two of two. This one does not. It's using the Vault Argo CD plugin, retrieving from the secrets. Both of them will do dynamic secrets. It's just a matter of how you have your application respond. So in the case of, for sort of a GitOps approach, maybe what you do is you say, hey, I'm going to change some config on my database secret or something. And then once that change goes, then everything resyncs, you know, and then your application, you can tell your application to restart and retrieve. Which is really cool. So it's like, as a developer, right? So I'm putting on my end user hat. I don't need to know the database password. It's given to me, it's rotated for me. I don't like, as a developer, it's really cool because it's like, all right, that's not my problem, right? Like that's actually a really good thing. Cause you don't want, I mean, as a developer, I don't even want to know what the database password is. Like I don't want my hands dirty of any of that. And what's really cool, you know, just kind of giving my thoughts in, like when the password rotates, you can actually have hooks into Argo CD that just automatically triggers like a re-syncing of applications, right? So you don't have to, you don't have to wait, right? Cause Argo has a three minute wait thing. Like you just do it, you know, as a hook, as soon as it happens, you can have, you know, that rotate. So that's really, really cool that a re-sync is all it takes, right? To rotate the password. Yeah, and I think that's the, I mean, that's kind of the powerful part of this. You don't have to necessarily, you don't have to necessarily build the awareness in your application anymore. You can use some of the GitOps tooling to help you orchestrate some of this workflow. Traditionally, what happened is that, you know, if you're an app developer, you would hope that your framework had a hot reload capability. Yeah, exactly, yeah. And with like, you would have to be like, oh yeah, you know, it detected a change in the environment variable like let me reload. And this, in this situation for applications that just can't do that, you can at least surface this to, you know, Marco to figure out what they need to do and the right order to restart everything. So to re-sync everything too. So it's a, I would say, I was very excited about having the ability, like between the config and having all of this set up. It was much easier. Yeah, yeah, exactly. Well, I mean, you know, once you have the declarations and like just let Kubernetes worry about it, you know, that's awesome, right? And I say, all right, cool, that's for me to do. So that's really, really, really cool. So yeah, Tiger says, thank you, operator. Yes, thank you, exactly. Cool, cool. So I don't know if you guys have any other questions for Rosemary. I don't know if Rosemary, have anything like anything else to show us or any final thoughts while we're waiting for questions here? Yeah, you know, if you have, you know, I think that part of this trick is that it's, if you have a vault, you know, or any secrets manager and you're trying to figure out how you can deliver your secrets to your applications, it's complicated. No one is going to tell you that it's easy and a lot of folks have wondered and pondered and try to figure out what to do and how to do it. If you're looking to give it a try and to experiment and to understand the workflow, you know, spend a vault on Kubernetes and see what was challenging, right? What frictions are you getting when you try to inject or try to retrieve a secret and inject it into your application? You know, Kubernetes secrets, you know, they were originally made so that people could use them and for us to easily inject them into an application. But the reality is that they're not necessarily a secrets management tool, right? Yeah. And so we are going to seek other tools that we need to fix it or to better secure our secrets and don't be challenged. Everybody is. And if you, anybody ever has any questions, you can always reach out to me on social media. I'm happy to answer questions. And even if it's not necessarily about what you saw today and you're like with Vault replication or otherwise, I'm always happy to answer questions or find the answer. That's really cool. And it's, it reminds me, so you're talking about Kubernetes secrets, right? Like it, that reminds me just abstractly, just like as a developer or an end user. It's always important to think about what you're naming things because when Kubernetes secrets first came out, it confused a lot of people because like this is, this is encoded. This is not a secret. It's like, no, a secret is, you know, in Kubernetes is a noun, not a verb, right? It's like, it's what it is, right? Not what it's doing. But, but yeah. So, cool, cool. Thank you. Thank you, by the way, for coming on. This was great. There was a lot of chat in, a lot of chatter in the chat saying like this is a great explanation. So you went really, really in depth of what's going on. People are gonna look back. I know I'm gonna look back at the recording if I ever need to. So, yeah, so, yeah. So if you find this interesting, please share, right? Share with your networks. Like, subscribe, all that, all that good stuff, right? We appreciate it here. Thank you again, Rosemary, for coming on. We'd love to have you come on again. So if you ever wanna come on again, talk about anything else. Let me know. You have an open invite. So, cool, cool, yeah. So, yeah, we're at the top of the hour. We went a little bit over, not as much as last time, but, you know, it's cool. It's all the good stuff Valentino says thank you as well. So thank you for watching. So, yeah, so unless there's anything else, right? Everyone, thank you for watching. And as I always say, unless it's in git, it's only a rumor. And so, see you guys next time. Cheers, everyone. Bye.