 All right, welcome everybody. I appreciate y'all coming through to everybody on the live stream, welcome as well. Today we're gonna be talking about Kubernetes and Vault. We're gonna really take it more so from the application perspective, and we're gonna talk about how we manage our secrets, how those secrets change, and how our application consumes them. Let's go ahead and jump into it, but we're gonna spend a lot more time because I don't like slides just in the code and walking through that stuff. There's not a ton of folks here, so if y'all have questions, I feel free to raise your hand and ask. I like a more interactive situation if possible. All right, so our main objectives of this talk are to write an application that A, gets a secret from secrets management, and the second part reloads when a secret changes. We're gonna be utilizing Vault to do this. Let's talk about some initial setup and we'll look through kind of what we got going. So if you all wanted to, you could follow along. First, let's outline our application architecture. In the middle there, we have our payments app. That's the app that we're writing. That's the Golang application using Gen. That's going to interact with two main pieces. We have an external payment processor. So when we submit a payment, it goes to this external payment processor. The payment processor returns me some data saying, hey, you're good, the payment went through. And then we go from there to recording that successful payment in our Postgres database. The demo uses Kubernetes. So specifically, we're gonna be using Minicube and the Vault CLI we'll need. And if you want to, you can go to that repo and clone it as well. Everything Kubernetes related is in the Kubernetes folder. So we're gonna jump into that and actually see what we got going on. And when you run make setup, it'll go through and set up everything. So let's go ahead and take a look at that real quick. I have my code here. All right, so I'm gonna go to my Kubernetes folder. You see, once you go in there, you'll see a basic make file here. And at the top, I have a setup here, which goes through configures Minicube, does all the things it needs to do. And then at the end, we have this bash script setup here. Let's go ahead and take a look at that. Look at the setup script. In the setup script, the first thing we do is we set up Vault. So Vault will be deployed in Kubernetes with the agent injector option configured. So we're gonna roll out Vault and the agent injector. The agent injector handles when I deploy my pod, automatically deploying the agent as a sidecar to that pod to manage the writing of secrets. We're gonna get more into what the Vault agent is and how it works later on. After that, we handle the port forward so I can actually interact with Vault and we'll show you kinda how that stuff works. We handle the Vault address and Vault token situation, do status check, and then we enable two secrets engines. And this is gonna be critical as we go throughout this. So we have a static secrets engine called payment slash secrets, which is just using KV. And we have the database secrets engine we configured which is going to be our dynamic credentials use case and really be the root of why we need this reload. So go down a little bit. Now we're gonna go into configuring Vault. This way when you all, if you wanna spin it up, Vault's already preconfigured with all this stuff. So we have payments, username and password we put into KV. Then we write that config for the database. We give it a connection URL and we tell it what role here. So you should be able to see that, yeah. We tell it what role, which in this case is the payments app that gives it the right to actually utilize the secrets engine. And the nice thing about this dynamic secrets setup in Vault is that with that role, if you access Vault and you access the database secrets engine, it'll execute this creation statement on your behalf. It'll go in, it'll create that user and it'll provide back those credentials in real time with whatever time to live you define. It could be, I've tested it down to like 10 seconds. It gets kinda crazy. But we have, it might be about two to three minutes with this demo. Now when it's done, Vault will go in and actually delete those credentials for you and manage all that behind the scenes. Sweet, so we're also doing transit engine. We're not gonna mess with that too much. That's with Vault encryption stuff. And then we write our Vault policy, which I did wanna show you all. So the Vault policy on our Vault folder just gives access to this payment role and to every token that we assign this policy to to read from our database credentials and from our payments secrets KV. So we have all those there. You can check that out as well. Go back in the setup. Now we go down, we configure Kubernetes auth here, which we're using. And we're also writing a payments app role and binding it to our Kubernetes service account. We handle the right of our policy and then the last step here is we handle that Kubernetes role as well. For the CSI. So that's the basic setup. All that stuff has been set up. I wrote and did all that beforehand just because the Wi-Fi here isn't great. So it handled all that downloads and all that fun stuff. Let's jump back. So that's the make setup portion. Now the first step that we're gonna talk about here is getting our secret from secrets management. There are two types of secrets. You mentioned it before, we'll clarify here. We have our static secrets, which usually is mapped to KV, right? Where you store, you kinda have to manually rotate them. But Vault has some nice controls around the KV secrets engine. Then we have our dynamic secrets, which Vault will handle the refresh and rotation for you. Vault secrets engines handle both of those. The secrets engine is the concept you can configure and we have different secrets engines that handle both static and dynamic credentials. So first, let's look at one way we can interact with Vault, which is via the API. We have some, we have a root token assigned here and I'm actually using Postman over here, and I have my Vault, this is also linked in the repo as well. So I have Vault here and I'm gonna just use the Vault API to grab these static credentials. I can see I have my password and I have my username. This is static, so these are gonna be the same every time. Now, with my database credentials that we set up in our setup script, we'll go ahead and run this. Creates a username. The last bit there is 4885. And once I call again, it should change that. Yep. So now I call it again and it's 4893. And behind the scenes, that was changed as well in the database, right? And each of these have a specific time to live. So after three minutes, if you actually log into your database, you'll see those users, you'll see them existing. And then Vault, after that time to live is done, we'll go in and clean that up for you. Right. So now, we got our static secret. We did example of the Vault API. Behind the scenes in your organizations, your Vault team will have configured these authentication methods for you. All right, so you have various authentication methods, everything from simple username and password to Kubernetes specific authentication methods, app role for like machine and apps automation. We're not gonna go deep into the authentication method. All of those steps that you would follow all end up kind of resolving down to this token, right? That then you will use to interact with Vault from the perspective of your application. So thinking about the API, and we now kind of mentioned this SDK, in order to interact with Vault, do I need to refactor my app? Do I need to write code that utilizes the SDK heavily? That's the question. Not necessarily, but we're actually gonna show both approaches in this talk. I'm gonna show you how you would do it with the SDK. And then we're gonna show you how that would look to outsource that and use Vault agent as well. So first let's look at the Vault SDK. And we're going to get the dynamic credentials we saw with our Postgres database using this SDK. So I have this code here, which we're gonna jump into the actual code, but this is the portion of the code that matters the most. With the SDK, we get our Vault address. I'm pulling in my Vault address for the environment variable. Then we go ahead and configure our client. Once we have our client, we get this token, right? Now here, for the sake of this demonstration, I'm getting the token from a file, but you don't have to, I was pulling, when I was just using the SDK, I was pulling this MV environment variable, but you just have to get this token initially through one of your authentication methods beforehand. Then I go ahead and set my token, so that the client uses the appropriate token. And then we have this get database credentials function here, which returns our database creds. So let's jump into that function and talk about that. As you can see at the top, and then get database credentials, and actually for this portion, I'm just gonna jump into the code, so we can do that. So here, can you all see that? Let's go down a little bit, yeah, it looks big enough. Let me know if you all can't see it. So as you can see, I take in two things here. I take in the client itself that we configured with the Vault token, and I take in our role name. The role name is key here because the role is what has the policy associated with it to actually give me access to what I need access to. So I have my role name, I have my client token. My token authentication method is also linked via my policy to give me the access I need to read these secrets. Now, if I go down, I'm just simply using this line here to go about reading my secret at a particular path. You'll see this trend within Vault that we track everything by path. So we can define multiple paths like we did with our payment processor, and we did with our database credentials, and have different policies associated to who can access what path. So Vault needs that path to understand location, and then it validates if you actually have access via your authentication to that path. At the end here, because that's the location, we just grab our username and password as well as our lease duration, and then I print this back. Now, for the sake of this demo, this is what I do in the beginning of my app. I actually go out, I grab this, let's look at the main.go, the main.go here. So you can see in the beginning of my main.go I have some environment variables which you'll see also those being configured in our Kubernetes config. And then I assign and create my Vault client here in the beginning. I get the token from this token file. Then I set that token, I do all that stuff here in the beginning of my main function. And then I do the get database credentials here. Now we'll get into why I'm doing this, but I'm doing this just to show, hey, you can do this with SDK, you can do this programmatically, that's all great. But then you run into a couple of challenges. Specifically here, now I also have to do it via some type of interval or monitoring for signal interruption, et cetera, to then refresh or rerun my application at certain intervals because of that expiring of the database credentials. But we're gonna get into that and show you how we handle that later on. At this point, just since we're here, we'll go through and tell you I then use those credentials to create my connection string to my database. And go ahead and use the, I think the SQL, it's the Postgres SQL driver here, to then open it up, open up a connection. I do a simple test and I'm using this config object, which you'll see as currently to kind of track state within my application. So that as I get updates and as we in the future we'll show the live reload that we end up doing using this config to keep track of all that stuff. So to further close, we do go ahead and test the DB with a ping and then we're into using JIN. Now JIN is a basic kind of go-lang web framework, has some benefits. I mean, it's pretty cool. I was messing with it just because I knew I needed to write this app and it actually has been pretty nice. So with JIN, we can define these endpoints. I have a basic health check endpoint. I have this reload endpoint, which I'm not going to show you right now because I'll show you that later. The main thing you need to know is we have our payments get and our payments post. That's the interaction with our database. The get just gets me a list of the payments from a payments table in our database and then I have a post here that actually allows me to submit a payment. Handling that workflow of what we talked about before of submitting to the payments processor and then recording a successful payment. Sweet. So that's kind of the vault SDK approach. I have to write code. I have to write a lot of manual code to handle the refresh of my credentials. If I am using dynamic credentials, I have to handle all those things in my application. That's a lot for a lot of development teams to do. It can become challenging to just maintain all the code that's around your vault situation. Now, some teams want to do that. Some teams like it. Some teams want to take that on and that's why we enable all that functionality. But there are another group of devs that really don't want to have to worry about writing all this code that isn't related to features. They would like to focus more on writing features and outsource the credential management part and rotation part and sync part to something else. This is where Vault Agent comes into play. So now we're going to talk about Vault Agent. Vault Agent runs as a separate process and it gets those secrets, writes it to a file that you've templatized, right? So you write a template, then Vault Agent will go ahead and handle writing that initial token that your app can use so that you can just pull it in. That's why I'm pulling it in as a file if you remember that from earlier. And it also handles at that least duration time or that time to live that we defined every few minutes updating that file. So we talk about this a little bit, but running another process gives a few advantages. We avoid having to add a lot more code around managing our interactions with Vault. It automatically handles all the retries and secret refreshes and it kind of separates the concerns of, hey, this is my application. I would like to focus on writing features too. I need to focus on managing secrets and writing all the code that does that, right? It allows you to more so just focus on things maybe you're more concerned about. The Vault Agent itself comes with the Vault binary. So if you installed Vault via the instructions, you'll already have access to Vault Agent. So what the Agent is doing in this case, let's get more specific to our use case now, is I have the payment processor, which needs a username and password, and I have my database, which needs a username and password. The payment processor, I'm using KV, a static secret. So in this case, it's taking those static secrets and it's writing them to a file so that I have access and can just read in that file in my code. It's using that template we defined and we're gonna go ahead and look at that template when we look at our Kubernetes deployment here in a few minutes. The result of this is nice. Instead of all that code I had to write, which I try to simplify so it wasn't taking too much, that git database credentials function, now all I have to do is read these from a file and it's pretty easy. I don't have to think about as much the Vault logic that I'm writing. So now we're actually gonna get into our Kubernetes config and show how you would configure the Vault agent to handle some of these things. Because outside of Kubernetes, there's actually a vault-agent.htl that you would configure, but inside Kubernetes, you gotta configure Vault agent with annotations. So we're gonna go ahead and discuss some of those annotations and jump into all that fun stuff. Oh, make sure, all right. So let's go ahead and look at that. We're gonna jump into our Kubernetes folder where we're at now and we're gonna look at our applications. Here in our applications folder, we have our payments database and the payment processor and we have our payments app. This is what's linked to the code that I've been showing you, right? Now, hopefully we can have some fun and break some stuff. We'll see, but I wanna show you all some of these annotations and what they mean, right? The first annotation, which is key, is the vault.hashgroup.com slash agent inject, which says I want you to put the agent as a sidecar for my application. Great. Then there's a defined role, which we configured earlier, that we need to give our config so that it makes sure that it's requesting with the right role. Next, we have the agent inject token, which also handles how I'm getting the client token from my app, because it's writing that to a file as well. And then we have the next one, which is the inject secret processor properties. All right. This is where I get to find, and I'll show you the part of the annotation here because it's all in the annotation docs, but let's go through it. So this agent inject secret, everything after that kind of defines what I'm actually going to be injecting. In this case, I'm gonna be injecting the processor dot properties. I supply this annotation with the definition with where it's at in vault, which is our path, the payment secrets data processor. So that's how that kind of breaks down. The next thing I supply is a template. This template is the core annotation and then everything I put after that is what's actually gonna be written. Here we have our template where we're simply, this is a kind of go templating syntax. So the main thing to pay attention to is my username and password here. That's what we're actually gonna be pulling into our application. Now, let's have some fun and break this through. I don't want to tell you about this part yet. This part is where it gets interesting later in the live reload. But I'm actually, just to show you that it is working. I'm gonna delete this. Everything that I just told you for that, the payment processor information is the same for our database properties. So I'll define the same thing. I define the template, I define the path of where to get those database credentials here. And then I have this special annotation that we're gonna save later for our reload. Everything else in here is pretty vanilla, Kubernetes stuff. I also have those environment variables that you see at the bottom, kind of telling my app where those locations of those files are, postgres stuff, all that fun stuff. All right. Now let me handle this rebuild. Let me do this. I'm gonna go ahead and delete my payments app itself. Oh, you guys can see that? Sweet. And now I'm going to run make debug, and I'll show you what that does here in a second to redeploy that bad boy. And hopefully that goes faster than the first time, which it should. All right, so if I show you my make file, all I have in debug is just the building of my Docker container, which is just my go app Docker container. And then I apply that app, wait for the deployment to roll out and handle the port forwarding at the end. So that's already done. Now, before my time runs out here. Oh, see the angle is the challenge here. I plugged it back in. Let's go out and then something's happening. All right, thank you all for that. So now if I go into postman here and I go into my payments app, I should be able to hit my payments application running on port 8081 and get some back, but I didn't get anything back. See, I told you I wanted to break stuff and we successfully broke it. So I feel like this is a win. Now, what I want to do is look at the why to that. So let's look at our pods. kubectl, get pods. All right, we got our app deployment here. No errors. So let's look at these logs. kubectl, logs, boom. Whoa, we got red. Okay, here we go. Got a panic here. This is fun. Now what I'm curious about because I deleted the annotation which I will admit is not something I did before this talk. I was just being brave, but we're gonna go ahead and add that back and see if it does it'll be interesting. But we're gonna see here in a second. All right, so clear out of that. Stop the port forward, handle that delete again. There we go. Do my make debug situation. Debug, all right. That's rebuilding. Now will it happen again? Oh, this is gonna be interesting. It didn't happen again. Okay, I have some ideas as to why that's a thing. And it's because if I look at my logs again, let's look at our logs, kubectl. This will actually probably be a good segue into our live reload. Get pods. Sorry, you guys can hear me breathing on this thing. Looking down at this angle. All right, so I'm gonna do kubectl, logs, boom. Yeah, all right, make this bigger. So a couple things interesting here. And this is gonna lead us into our live reload. But you see I have an endpoint configured to reload. And this is actually what's being called by the vault agent. So that's why we did it, because it tried to trigger a reload, but it didn't work. That's the high level of it. Because I didn't have that command, so I was wondering what the heck it was supposed to do. That leads us into this live reload. So we're airing out before. I handled that, we added the annotation back, that's fine. But what happens usually, let's step back a second, what happens usually when you reload or when you make a change to secrets? We talk about those two types, static and dynamic. This example, we're gonna be talking heavily about the dynamic side. And the vault agent we kind of reviewed is doing the same thing as it was for the parent processor. All it's doing is saying, hey, I have these dynamic credentials I just generated them for you, here they are. Let me write them to that file. And the path to these files, this is just a fun fact and useful if you're doing vault stuff, is slash vault, slash secrets, and then whatever your secret is there. If you want to actually exact into your container as it was going on. So then it handles the rendering of that new file with those credentials. Now your team, again, I'm not gonna go into configuring all that, I did that with the setup script, you can look at that. Your team behind the scenes is probably the one setting up dynamic credentials for you. In reality, they're probably working with the DBA team to make sure the creation statements are right or whatever it is, depending upon the level responsibility that team has, to make sure that the user that's being created has the appropriate permissions. So now let's talk about the application. This is where it is on. So I have my secret, my secret changes because I'm using dynamic credentials. I initially load my app, boom, I have database credentials, fantastic. Then vault agent, because vault agent's awesome, goes out and creates this new file. It creates a new file with my new credentials after they expired in like the few minutes. Cool. Now my application though, has no idea what's going on, is still using that old secret. And this is what we have to address because there was no reload type of operation. And this again, is where you had to handle this a couple different ways, a couple options. One of those options is via interrupt signals. Like you can have an interrupt signal sent to your application, sig in or something like that and have it reloaded, and then you'll get it back, right? Not ideal, I didn't love it. That's why I didn't put in this demo. So I actually have the code for it and I tested it. In Golang, if you all are Golang folks, I'll show you all just for fun. The code for that interrupt was handled here. So just you know, spun up a go routine, had a signal channel listening. As soon as I got a control C in this case, I handled a reload of my application. So that's cool, that's fun, that's an option, right? But what I wanna talk about is live reload. Oh, scroll down on my code. This is that endpoint I wasn't telling you all about, okay? So I set up the endpoint instead that is just slash reload, which handles reading those files again and refreshing our application state, which is that higher level config object in this case. And I think I added, no, I said just catconfig.go, but I'll bring it up here. It's pretty simplistic. It was appconfig.go. It's just a basic struct here that I'm managing globally. And I have a couple scan functions for that file. All right, now this was just me throwing some of the data, don't make fun of my code not being good, because I was getting this done and wrapping this up probably a couple days ago. So great, now if we go back, right? I can see my reload function here. The first thing that you wanna do and kind of any, whether you're getting an interrupt signal or you're handling it via this live reload method, is you want to handle the closing down of those previous connections to your database, kind of like handle all that type of stuff. Open up a bunch of connections. Same with handling the sig in. Close those connections here. Like handle all that stuff in this space so that when your app exits or refreshes, you're good. Cause that, oh, there's VIM key bindings. We revealed, I'm sorry, I just had like a very happy nerd moment. There's VIM key bindings. Use the reveal.js, sorry, we're in. So now I'm looking, right? I closed that old connection, which I'm handling in my kind of global config object here. Then I'm refreshing the database properties, which again is at that path slash vault slash secrets database.properties. All right, I read that in. I defer that file close. I scan that file and read it for my username and password. Then I configure my new connection string and handle all that logic I did in my initial main using the SDK here again as well. And then I assign that and refresh my global config with that database connection to be used. So I do that with the processor. I do that with the database. And I also read back in my client token file and set that token here as well. At the end of it, I get this token successfully refreshed. Great. When I got this working, I was really excited. I was very happy about this. Cause now it gives us the ability to do some other fun stuff. But first, we have to configure our vault agent to actually reload our application on our behalf. This is part of the benefit of having an external process managing these things. Cause now I can do interesting stuff like this. So the command I use, which you saw is this, just W get, I'm doing a simple W get. You can do whatever you want here. Everyone can kind of do whatever. One way or another, you can use curl, whatever. You have to call this reload endpoint. Now, a little context on this talk. I'm covering for a colleague of mine and she was using Java Spring, okay? And I refuse to use Java Spring. So we wrote it in Golang and that was part of why this took some time. No hate on Java Spring folks, okay? It's just not me. It's not me. So now I'm here, I'm doing this. But one of the things it had, the real reason I chose, oh, come on. Don't do this. The real reason why I chose this live reload functionality is cause in Java Spring, they have a actuator dash refresh endpoint that literally does the same thing. So like, if you want to handle that in Java Spring, you have to edit a bunch of config files and do all that stuff and enable the actuator and then you don't even know what's going on. That's the thing I don't love about Java Spring is I don't know what's going on half the time. They're just doing magic stuff. So this is what pretty much is doing. It has this endpoint that then live reloads these configuration files, right? Little fun fact for you. Now this is the portion of our annotations. I try to condense it a little bit for us. That is affected by this, okay? So I have my database properties here. That's the same one we saw and this last line, the magic line I was telling you about, is using this agent dash inject command. Let's just open it up in the code because reveal.js is annoying me a little bit. All right, so if I do my, oh, no, that's not right. Open up my payments apps, payment apps, boom. So this here, we have agent dash inject dash command. That part there is the actual base annotation and then we're doing the definition of that which is database.properties. Now I can pass it any command that I want it to run to handle whatever. This is external process awesomeness. Now you can do this with the SDK. You can handle the global refresh of your state. You just have to write all the code to handle that and figure it out on what interval, which again, I would probably recommend if you're gonna do that. Don't like set a time interval, like handle it via the interrupt or whatever. So here we have database properties and I'm passing this wget which calls my reload function and handles all of that stuff. So now we've given enough time and tried to talk long enough for this to actually show now. So now if I look at my kubectl logs again, I wanted to add colors here for you all but it just didn't work out. So you see the username at the beginning which was kind of the username and password for our database. After a few minutes, we got a new username and password because the Vault agent refreshed that file, it automatically called the reload endpoint and now I have as part of my global state this new username and password. And I don't even have to think about it anymore. The live reloads automatically happening behind the scenes while agents taking care of it and I can just focus on writing my application code the way that I want it to be, all right? Sweet, so that worked out. Now beforehand, before I added the annotation I added a bunch of other code being brave, right? Because I have my payments and then I'll submit a payment. I just want to submit a payment just to show you all that it actually works and myself that it actually works. There we go, it's at the bottom there, nice. So here, I submit that payment. I can also manually call the reload if I want to. Let's look back here. It's not what I want to be. All right, anyway, the live reload functionality is kind of I think the direction that you want to go if you have an application team that isn't interested in maintaining their own vault abstractions. That's kind of like the main decision point I'm seeing on teams where they go full SDK or they go vault agent. How much capacity does that particular development team have to manage vault? Because when I was writing vault SDK stuff I ended up adding a lot of abstractions on top of vault. We ended up having this whole company specific vault library that we kind of did some fun stuff with. It's just an organizational decision point that you got to think about. But the vault agent we see in large organizations removes a ton of that friction. All right, so I don't think that was near my hour and a half. I went through pretty quick, so sorry about that. But that's the general synopsis and end. We went through static and dynamic credentials. We went through handling that in our go application. We went through facilitating the refresh of dynamic credentials because that's really where it matters the most. And also being able to handle the static ones as well using an external process. In this case, vault agent. And yeah, we have time. Anyone have any questions? Or thoughts or anything like that? What's up? Yeah, it lives, so there's two pieces of it, right? So I have, I'll show you them here. Specifically within Kubernetes, when you're doing this in Kubernetes, you have the, let's just get pods, vault typing and speaking, difficult. So I have a vault namespace that's here. And when I enter that, I see a vault running, right? And then the second one there, I have the vault agent injector. So this isn't the vault agent directly, but the vault agent injector handles injecting this directly into your pods. Now, to answer your question, but I just want to give you that context, when I do a kubectl, get pods, I see there's two containers here running. So this is where the actual vault agent is running. And if I do kubectl logs here, now scroll up, it's just choosing to give me the payment app because it's kind of seeing that's your app, right? It's not official, like a sidecar, but you can see here that I have payments app running and the vault agent running. And vault agent in it, which just handles some knit stuff. So the vault agent is literally just running as a sidecar to your application. Anyone else got any questions? Capacity in what sense? So I don't have a hard number for you as far as like how much compute it takes up, but I've seen this, the vault agent and vault agent injector workflow running across thousands and thousands of services. So that we haven't had an issue with scale directly, but yeah, I don't know exactly how much compute it would take up. The nice thing is it's pretty low management. That's more so what I think about it from because the compute hasn't been a problem, is whenever I have a sidecar that I bring into my app, I'm like, how much work I gotta do with this sidecar? This is how much maintenance is this thing gonna be? And that with the annotations, once you figure out the vault annotations and just that initial configuration, companies end up usually just kind of templatizing almost that piece of it and giving their app teams different options of how they can consume the vault agent secrets from different application perspectives. So in this case, like with go is like, oh, our go convention is slash secret slash vault, whatever at this folder, this is where it's stored so you know where to go and blah, blah, blah. But yeah, I would say the maintenance is low. Once you figure out that initial config part and then think about the nuances of how you wanna handle that in your apps. That makes sense. Any other questions, thoughts, concerns, feedback? Sweet. All right, I'm good. My name is Gerald. I don't even think I told you all my name, but my name is Gerald. I work at Hashicorp and this was a lot of fun. So I appreciate you all coming through and hopefully see you all around after the talk.