 Yeah, thanks for the intro Like you said, I'm Jake. This is Josh Satellitability engineers at IBM and today we're going to talk about how we are managing our secrets via get-ups So I already got a good intro, but just a little bit more background I'm from Michigan a very small town in Michigan and I graduated from Central Michigan University with a computer science degree I started my career at IBM. I went to red app for a little bit and then I came back to IBM as an SRE Just a couple years ago Some things that I'm really into Like Argos CD like you mentioned cloud native terraform Operators and customer resources and stuff like that Yeah, and I'm Josh. Yeah, I've been here since 2018 May 2018 at IBM from NC State University I've been rallying with Carolina and I'm just happy to be here So So a little bit of background Just to talk about our journey into Kubernetes. So the team that we Are on we started out managing just Supporting just one development team that was that had this big monolith that was deployed to WebSphere I'm sorry about that And everything was being done manual, you know, we'd When the time was right we had these jars and we just manually deploy them to the server So and all that setup was done manually just by a small team of just a few But that monolith started to grow and started to get slow Performance started to suffer. So we decided let's look at you know, breaking this up into microservices and look for a platform It's a service. That's where we went into public cloud with Cloud Foundry And this worked really nice because we were able to break up the monolith But being a small team it did become a little cumbersome, especially as we got more Developers under our support. So it went from just, you know, one application with these microservices to two, three, four, five Applications and all their separate microservices. And so it was becoming hard to Orchestrate them, right? So that is where Kubernetes came into. We wanted to use Kubernetes to help simplify this process which Simplify and not simplify, but You know, it helped orchestrate these and these different applications and helped us, you know, to be a small team and manage multiple Developer teams, but with that, we did notice, you know, a lot more resources. There's a lot more managing with these clusters You know, a lot of these resources that were necessary to be there for these applications pull secrets Rbac all these different resources that we didn't really have to manage with Cloud Foundry These are now all under control and all being done manually. We were, you know, applying these things Manually into clusters. So we had an observation or observability problem where we weren't sure what was in what cluster You know, who who needed what Rbac? We even had problems or we deployed an open pull secret wasn't there and nothing was pulling in production. So a lot of these problems and so We tried to look how can we solve this problem? How could we have this observability? How could we be able to check these things into code and to get right? You know and get these things into our cluster in a stable and reliable way and so That leads us to get ops, right? And I can go deep into get ops. I know there's a lot of you know, there's get ops kind yesterday And there's a lot more going on with get ops Earlier and later, but just I just want to talk about the like the four main principles that were developed by the get ops working group Starting with the declarative state, right? This is This is where like something like Kubernetes is really great because right you have these manifests. They're declarative by nature You can check these into source code and it's telling you exactly what you want Kubernetes to do Which is really nice and it doesn't have to be Kubernetes I can be any sort of any sort of system that you are having this declarative state defined And then second the mutable desired state. This is Right, this is get right. This is the get part of it And again, it doesn't necessarily have to be get it can be some sort of other version control but you know having this You know these the history of the commits and being able to It's fine Right get sorry about that The third part this is the part that to me differentiates from something like infrastructure as code It's the reconciliation having something like a controller that is watching your get it's watching your cluster or your other system And it's it's constantly checking. Oh, is there something new and get do I need to put that in a cluster? Is there something new in the cluster that's not in get what do I do with those? And it's constantly checking and doing that that reconciliation. So that's Really big part of get off and then the last part is the Operations through declaration. This is like to me the something that really like resonates with me here It's it's that if you're doing something like In order to like be doing get ops, right? You're you're going through the principles of get ops to do get ops, right? It's you're not applying things directly into a cluster. You are always following these principles So that if someone is manually applying into a cluster get ops is able to with your Reconciliation is able to detect that say this is not supposed to be here and it can remove it if you like have some the correct policies in place And for me, that's pretty powerful so I went a little faster there, but so we have these clusters that we want to manage these resources via get ops We did some You know experimenting some searching trying to understand what a tool for us would work and for us Argo CD was that solution With this we chose Argo because you know, we did like the user interface aspect of it You know, it's really nice and clean baby being able to kind of see these things without having to maybe do a little bit of guesswork the We wanted to because we're a small team. We wanted to have one Argosy instance and many clusters. So being able to have a that multi-tenant Cluster aspect of Argosy was really nice the extensibility This one is key for this talk. So I will come back to that later But Argos CD allows you to be able to use different tooling You know within our city, which is nice CNCF incubating, you know, we that's something that when I was looking at doing this. I wanted to make sure as Trying to push toward that CNCF You know tooling so that was really important and also we were looking to move from something like a plain yaml to Helm to be able to template out our resources So that was also very important for us So, yeah, Jake are we gonna talk about secrets? Yeah, we're gonna talk about secrets just a little bit of background So the biggest question I always get when I talk about get ops is what do we do about secrets? That's a valid question right because we have right everything and get and I'm telling you to put everything you get but you know We've always been told not to put our secrets and get so You know, so we that's something we had to try and solve we wanted to do these resources We want to Argosy but how do we do secrets and Argosy doesn't necessarily have a native secret solution they kind of go with the mentality of You know However, you want to do it right there. There's different solutions. It's not our we're not opinionated opinionated about it So, you know, you can you can kind of do what makes sense for you So with that in mind, we had a certain set of requirements that we were going to need for this secret solution Yeah, obviously we talked about Argo, so we wanted to be able to work with Argo CD Our our secret manager solution was to use actually core vault Because that's you know what we had internally and Also, you know search products that we wanted to continue to work with that as well as IBM cloud was building a Secret manager solution based off vault. So we figured that was the way so we wanted to continue with fault We wanted to be we wanted this to be able to work with all resources So we had a couple of use cases internally that we were we were Putting things in the vault that would then be added into something like a config map or a custom resource in a lot of solutions That we were looking for looking at they were just about this secrets themselves So we just decided So we just decided let's just make it work for all resources We didn't want to have an operator to install this. We wanted to be very simple So the idea was we wanted to just have Like environment variables or a config file to be able to run this thing without having to do this big installation set of a cluster and Again, you know, we wanted to be able to use it with helm and other things So the idea was very simple we wanted to have YAML resource in a gate repository where the Value where the value would normally go we would template it out with this this kind of like placeholder syntax something that we had used on a legacy System so we wanted to continue using this because it would also allow us to use our already defined yamls We wanted then our go CD to be able to pull this do something Apply it in a cluster and then we get our value when it applies it so Very simple idea right yaml and get with a placeholder our go CD and then the cluster we get our You know secret and the way we were able to do this Talking about that our go CD extensibility was the custom tools Ability that our go CD has so our go CD right they have their native like Transformation tools like helm customized plain yamls, but they also allow you to define your own tool so This is really nice because you can basically just either you know run an it can in an it container with the our go CD repo server or You can just bake it into your repo server image And then you put it in a certain directory which our go CD will look into for these custom tools You then register it Looking the step two up in the top right or you just then you can register it with Exactly what you want this command to do So you can give it a name and you tell oh I want to run this command with these arguments and Then in the UI you'll see that you know your custom plug And then is part of the plug is dropped down because when you in the UI you'll have like the different native Options when you're creating an application, but they have a plug-in option where you can then select from your list of custom tools And so with all that being said that is You know Our requirements how we wanted it to work how we how we were able to get it work And with that we built the Argo CD vault plug-in mentioned in our intro And this is a plug-in that allows us to Take these yamls Run them through Argo CD reach out to different secret managers pull the values Replace the the placeholders and from the couple slide before and then apply them into the cluster We started off with just hashicorp vault and the IBM cloud secret manager again, which is based off the hashicorp vault Yeah, but then we realized there's much more secret managers You know that are not Hashicorp vault based so we wanted to do some of the other main ones So we then built in support for AWS secret manager and then GCP and Azure were actually added via the community Which is really nice So that's something that we didn't actually have to do so thank you community and with this with this tool some of the features is we We do versioning we have multiple different ways of defining Where you want to like where you want to tell Argo CD or I'm sorry We want to tell the plug-in to go look for the secrets So you can either add an annotation or you can build it right into the the placeholder itself We call inline path To tell you know to tell the plug-in where to go find these secrets it can be used with Helman customize and There's different authentication methods Josh Josh is going to show you I'm using the Kubernetes auth within a Hashicorp vault and how you can get that set up So just a high level to try and Kind of paint this full picture So and hopefully you can see this in the back But so in get we put our yaml with the annotation and our placeholder So the annotation is our path to vault and the placeholder is the The key right so involved you have your key value So that's the key from vault that you put in the placeholder. We have Argo CD, which is watching that repo and When it detects a new change right Argo CD does this thing it pulls it in We You specify that you want to use the Argo CD vault plug-in as the custom tool So it knows to run the binary that you've set up in your You know in the Argo CD custom tool plugins it then makes the call out to vault based on your path returns the values Does the replacement outputs yaml back to Argo CD and then Argo CD will apply it into your cluster So you know at a high level that's what that's what it does and Josh is going to show how to set that up and a couple different useful use cases that we found for our CV all plug It's logic. All right, so here we go doing the demo. Let's hope this works Okay, so Today I'm basically going to go through this and I'm going to show you how we can use the Argo CD ball plug-in I'll call I'll probably refer to it as ADP at some point shorter This repo is publicly available the link will be on the last slide. I'll show later It'll show everything here in good detail and there's read me is all over so you can follow along what we're talking about So to get started, let me zoom in here so we can see We need to get our go CD deployed with AVP installed, right? So Jake pretty much explained how that's done, but I'm going to show you in a little more detail What I'll show you is doing it with customize. I'm basically going to use a customized bundle Make a few changes that we have the repo server plug-in and then I'll have that I'll show you how you know that works out So if we go to my repo in this Argo CD overlays folder and we look at the customization yaml What you'll see is we're basically taking the upstream manifest from the Argo CD project at version 201 And we're making two important changes. We're changing the repo server deployment The repo server is where is the context in which an Argo custom plug-in runs in and then we're going to change the config map resource So the repo servers so we can actually download our plug-in and the config map is so we can register it as something Argo CD can use So to show you what that looks like if we go over here to the repo server yaml We'll see basically what we're doing as we just have this you know Empty directory Kubernetes volume that we're creating the femoral volume We have our init container in this case what we're doing is we're basically downloading the latest release of the plug-in from github releases So this version 140 here and we're basically putting it in this custom tools volume What we'll then do is we'll take our repo server container Which again is the actual thing we want to have our plug-in in and we're just going to mount it here at user Local bin, so it's available on our path and we can then reference it as an Argos CD plug-in in the config map So at this point we have a be downloaded installed It's just a matter of configuring it right and there are multiple ways to actually configure a VP to be able to communicate with your secret Manager like Jake mentioned what we're going to do today because it's keep gone is we're going to show keep your name He's off. So what that means here is basically we're going to have a vault instance Running in our keep your name is clustered here and we're going to have the two communicate to each other using keep your name He's off. So what I'm doing here is I'm basically telling a VP that we're communicating with a hash a core vault instance That's our secret manager back end. We're telling it where it's located. So it's just going to be here in our cluster I'm gonna make that we're going to use cuba nanny's off to be authenticated to it And we're going to use the Argos CD vault rule and I'm going to show you how we set up the roles in vault So that cuba nanny's off works and then finally because the repo server It doesn't usually have the service count token mounted and you need a service account token This is actually do keep renaissance. We just you know, make sure you have one mounted So all that to say you have this custom. Oh, sorry one more thing. I just showed you the repo server I didn't show you the config map. So the last thing is just actually being able to register it as an Argo plug-in What you'll see here is I have two different plugins sort of described here I'll talk about the helmet in a second or a little bit later But the simple one here is just called Argos CD vault and what you'll see is all it does is it just runs our Vault plug-in binary generate command on dot slash dot slash beating the yaml's that you have Of the Argo app that you're trying to deploy So it'll run, you know the plug-in through those yaml's replace the placeholders and send the rest to standard out so that Argo can apply them Yep, and I've got you know credentials so I can actually pull from git Cool. So now I think I've shown you everything. I think I've already gone ahead and set up port forward So I can actually show you the Argos CD UI Looks like we're good. I have this bolt instance deployed We'll talk about that in a second But just to show you that the plug-in's been installed correctly You can do what Jake was describing where this is the Argos CD app form where you can fill this out to make an Argo app And here at the bottom you can choose to do a plug-in and you'll see that we have our two plugins Here so we are good to go. I'll zoom in a little bit here. Okay So that was Argo. We have to plug in we're good now. We just need vault, right? And so this is again for the purpose of the demo. We're just going to deploy a simple vault in our cluster here This is not how you should do it in production, but just an example So I have vaults I'm using the vault helm chart to keep my life simple here And if you look at how I've deployed the vault helm chart here in the parameter section What you'll basically see is that I'm going to run the dev server which you should not do in production and I have this post art script and what this post art script is doing is It's basically waiting for vault to start up. We are creating a vault policy That allows reading secrets at the secret path We're basically so we're creating that policy in vault, right? And then we're going to enable cuban is off and we're going to tell vault where our cuban is host is so we can actually talk to the cuban is master and most importantly Maybe I have to zoom out so I can show all this. Yeah, we are creating that Argo CD role So this is the role that we can't configured AVP with earlier, right? So we could tell AVP, you know logging with vault and use this role and we're granting that role to the default service account in the default Namespace because in this case, this is where the repo server is running and most importantly We are granting that demo policy that was created up here for that service account So we have everything we need we have the repo server and we have vaults again. Don't do it like this in production This is just an example So we've got Argo. We've got vault now. We can actually start to deploy things So I'll start off with a simple example of a basic get app So just a collection manifest and get what we're going to do is we're going to deploy an engine X instance And what we're going to do with this engine X instance is we're just going to have it basically print out the value of a secret That we've deployed through AVP So I've got a bunch of the animals here first off This is config map that basically contains an engine X configuration file Don't fret if you're not familiar with engine X just know that basically what this says is we listen on 8080 and for all requests We're going to return a 200 with the value of this my secret environment variable Please report So that's that right. That's the engine X. Where's the secret come from? Well, it's going to come from this Kubernetes secret and it's Kubernetes secret is what's going to be Created through a VP and what you'll notice here is that this secret has a special annotation This is the annotation Jake was referring to earlier where we tell a VP where to look in vault for our secrets The way you read this example is there is a my engine X secret at this path secret data Because this is a vault using KV the KV V2 engine and inside of the secret There's a password key and I want to interpolate the value of that password key as part of the my next secret for this variable That's how you read the syntax here. That's what we're doing So we have our config map We have our secret and then we just need an actual engine X image so we can run this and actually you know make this work Right, so we've got a little open rusty image here using a port 8080 like we said earlier We're going to mount our secrets through environment variables in this engine X instance and we're going to mount the config file where Engine X expects it to be in this image So I put all this stuff and get already you can see them here in the apps Slash get slash engine X slash manifest folder. So these are all the files that we're going to be deploying here So I've already created them So we have the code and get great That's the first step forget ops now. We can you make our secret not in get we're going to put that in the secret manager So let's go ahead and do that So first what I'm going to do is I'm going to port forward to the vault service that I showed you I was running already So let's go ahead and do that. I'll just zoom in so you can see here. We are so we've got vault Next I'm going to basically set some vault variables so that I can talk to my vault instance You should never use the vault the root vault token should probably be disabled in any production Deployment of vault, but again, this is just a demo So, okay, we should have everything we need I'm going to create a secret called the mind gen X because that's what you know We said we're going to do in our secret YAML with this key password and a value secret password So Here it is. All right We've created a secret and I'm going to read it back just so we can all agree that yep I put secret password. Okay Now there's a way you can create Argo apps You know because Argo CD comes with the custom resource definition You can create custom or you can create apps through YAML, but I'm going to show it to you in the UI You know just to make it hit home a bit better, but you can easily do it Just YAML so here we are we're going to make an app called mind gen X how fitting. I'm just going to put it in the default project Not going to change in the sync policies here I'm going to use the one repo this very repo that I have configured here the path that contains my manifest is going to Be this this one here apps get engine X manifests apps get engine X Manifests and I'm going to deploy to my local cluster and my default namespace And the most important thing here is I got to tell Argo CD. I want to use our plug-in to deploy this So Argo CD vault, so I've got everything right here. I think so I'm going to go ahead and create this Argo app Okay, so we see is we're currently in out of sync state So basically what Argos done is is looked at the those files that were in get it looked at what's in clustering realized Hey, we have files that are not in git. We have you know resources are not in our cluster and we're out of sync So that's the yellow here, so we can easily go ahead and just synchronize these And what's it's going to do is that's actually going to run our plug-in Actually interpolate the secrets create the resulting secret in our cluster and deploy an engine X instance That's going to figure it to read from it. So why don't we make sure that we did this right? Yeah Let's actually try to talk to our engine X and see if it's returning the right values So I'm gonna port forward to this engine X instance. I have deployed And if I go over to local host 8081, which is where I'm running it Great, you see here. We've got secret password great So we've enabled show that we can deploy secrets and we can deploy apps. I use them with Argo and AVP Now we just need to actually try modifying the secret in our secret manager, right? Because you know we I can show you how to create one but a big part of secret management is rotating them, you know for reasons And it's an important use case to be able to handle So what I'm going to do is I'm going to simulate a secret rotation by just doing a volt KB put So I'm going to put a new value for the secret in this case We're just gonna put the phrase edited at the end Just read it back out for us real quick And now the question is how do we tell Argo CD that's that the secret changed and this is what a concept of a hard Refresh comes in during a hard refresh of our an Argo app Argo CD will take any custom plugins that you're using to deploy the app Do a dry run up those plugins and compare the output to what's in cluster if there's a difference Then you'll be out of sync and you have the opportunity to resync Once we do that We'll also need to restart deployments since we're using environment variables in a pod and they only get refreshed when you restart those pods So I'm gonna zoom out here real quick. I'm gonna do that hard refresh All right, our secrets out of sync because we change it in volt the plug-in ran We realize that what's in cluster does not match what we want it to be and we can easily remedy that by syncing So we've gone ahead and synchronized and we need to restart the deployment like I said And we're gonna get to have a new engine X starting up here soon And now if I just report forward back to our app We'll be able to then refresh and we see here secret password edited. So great. We did a secret rotation Thank you You can Yeah Thanks a lot. I'm glad it worked, too Cool. So that was just a simple example of you know a bundle of manifest and get But you can also use this tool with pretty much any sort of YAML JSON templating thing It's really just a matter of having the placeholders in the first sort of tool have something rendered Then it had ABP run through that. So what we're gonna do is we're gonna deploy a helm chart, right? So helm charts basically template out a bunch of YAML and we're basically going to use a helm chart with ABP to deploy You know an app with secrets. So I'm gonna do a redis instance here So first I'm going to create a redis a secret for my redis instance I'm gonna set the global password to what three H's so I'm gonna go ahead and do that here And I'll just read it back just to be sure I did that right cool So we have that and now just a matter of creating the Argo app and this is a very simple example I wouldn't recommend actually doing this for any real reason But what you'll see here is that we've created a secret in our secret manager and then we're just going to create an Argo app nothing else and What you when you look at this Argo app What you'll see is that we're basically going to point to the bit Nami redis helm chart for redis and that we're gonna use the Vault plug-in and then we're gonna set set some special Environment variables for that plug-in to explain this let me go back to the config map where I showed you where there was the second Plug-in I registered. Let me do that real quick So if I go to our duty overlays again, and I go to this dash cm All right, I talked about this one and this is what we've already seen There's also a version with how and essentially this is kind of showing the idea I said earlier right where we basically need to just render out You know our templates or our customized things or JSON or whatever and then run a VP on the result to get YAML with secrets and so what we're doing is we're basically just adding our helm repo and downloading the YAMLs that comprise the redis chart And then we're just going to run helm template filling in some stuff with Argo variables And then actually generate the manifests with a VP and you'll see this little helm arcs thing This is basically our way of passing things. We want to pass to helm the command line invocation Since you know we have to do this because it's a custom plug-in. So this is just our way being we'll pass special flags to helm and We're taking advantage of that here So that we can basically require off on our redis and use the inline path placeholder The Jake was talking about so this is the path and we're going to use the password key and that's equally created And we're going to use that to set this little redis password So real quickly here because I want to be able to answer questions. I'm going to apply this Okay, let's try that one more time. I'm going to apply this redis app okay, and If I go back and zoom out basically what we'll see here is oh, no, what is this indication required? Not sure what I did wrong here. Oh I think I I think I might know I don't have this repo configured with any credentials So, okay, so I probably won't be able to fix this in time But just know that basically if I had done this, you know, we'd have what you have seen is a set of Manifests just like we did here But you know for all the animals that comprise a helm chart and then if we had just synchronized them You know the plug in would have ran replace the placeholders and we would have had a redis instance deployed with our secret set to sh So sorry, I can't show you that I must have messed with this the night before so I'm gonna stop here so that we can take questions So thanks for your time. Thank you Oh One more thing this is the slide with the links so you've got the repo link there We've got documentation, you know as part of our repo our github pages and there's a demo repository Where if you don't take my work for you can be sure the redis example works So and some blogs that have been written on this so thanks So something we have about two minutes for questions If you have any questions, please raise your hand and I'll run over with the microphone and When you have a state out of sync like when you detect it is there a way to actually check what the discrepancy is Good question. So because it was a secret that changed the Argo UI will always redact the values of secrets You won't actually be able to see oh the secret was previously secret password and now it's secret password edited I get I don't know what the difference is you won't be able to do that Really, I guess in that case what I'd recommend is just going into the secret manager and taking a look Yeah Hello How can we limit what secret path can Argo application can use? Well, let's say I want up to be able to read this namespace involved for secrets But not the other namespace from the other application A good question so When you're setting up Cuberin Okay, when you're setting up Cuberin is off I believe there are ways you can actually configure that especially you can basically set like which service accounts can you know talk to vote? Yeah, so Yeah, so what you can do is because it's based off environment variables Jesus, sorry you can go into the When you're setting them up you can set the specific environment variables for each You know Deployment and so you can like you can have certain applications use certain environment variables that point to the different Service accounts that you set up and that way you don't have like one set of credentials You can have these multiple set of credentials pass throughout environment variables for each Same each different application that you deploy That makes sense Is is there a way to automate the part after you cycle the secrets? We have to go kill the pot. That's a good question So, you know, Argos he does have a rest API So if you do have a way of like I know you change a secret and then you have that call some sort of web hook Right, you can tell Argos to be hey do a hard refresh for this app and that if you have automated sync Then you could have an automatic complete rotation That's a good question Think we're time for one last one first-hand up all So Is that a way not to put the bash scripts and stuff inside the YAML because I mean we moved to YAML definitions not to tell, you know, we just want to have the state right not tell it how to do things You know what I mean sort of We're just defining the data inside a YAML, but now we're back telling them Oh, this is how you fetch a secret. This is how you render it. So I think to me I mean, this is a great presentation, but it's kind of messy when I looked at the YAML file So that's a fair So if I understand correctly you are talking about the when we register the config when you actually register the plug-in in the Argos City config map and how we have to have that bash stuff to be able to tell it how it works So that is fair. I will say that is a one-time thing, right? Because once you have once you have this set I mean you have it forever You don't you know once you have an Argos CD that is using this config map It's it's good to go you can deploy helm charts all you want and you just have to say use this plug That's really it like the complexity is yeah at the end of the day You just have to I mean you just have to be able to tell Argos CD what to run in this in this case It's just you have to tell it that it's wants to run this command So there's really no way around that unless Argos CD adopts a different way So that's just the only method we have to register the plug-in to Argos CD, right Cool, I think that concludes the Q&A. So if you have any more questions Please meet up with the presenters outside in the hall and don't forget to rate the session. Thanks everyone. Thank you