 Good morning, good afternoon, good evening, wherever you're healing from, welcome back to another Data Services Office Hour. I am Chris Short, host and showrunner of Red Hat Live Streaming. I am here with the one and only Michelle De Palma. Michelle, how are you today? I'm good, how are you? I am good. Before the show, we were talking a lot about disaster recovery scenarios, but like human disaster recovery scenarios. So it's interesting to be on the phone and we're on the horn with you right now to talk about, you know, data. It's kind of, you know, importance. If I had known, we could have switched to like a disaster recovery. No, I'm done with disaster recovery for this week. You don't know, like take questions down and pull the power plug. Good. So this is, there's nothing disastrous about this, except we are, it is a live demo, and it is OpenShift 4.8, which is GA. Right, GA yesterday. GA yesterday, which is good. Good. But it's ODF 4.8, which is not GA. So we're living on the edge. We are living on the edge here. You've been doing for a few days. Yeah, pretty much. It felt like I was on the edge of existence for a little bit, but yeah. So hopefully this will, I actually, I've done it a few times, but you know, it's live, things happen. So, but I at least we're not starting out with the intention of breaking anything. I'm actually like everything to work properly, but you know, but that's okay. Oh yeah, like I know where that goes. Sometimes I come in with that, like, let's break something. And now I'm like, oh, let's have everything not break. So let's lighten the burden for you because you've had enough happened to over the last few days. So, okay. So I'm going to present my screen and we're going to talk. I'm going to do the whole desktop here. Yeah, so just to let you know, people have been having some sharing issues with Zoom lately, like the tab sharing has gotten weird, I think, or something's changed there. Oh, all right. So let me know. Do you see this? Yes. Okay, font size is very, everything. Okay, so today, what was our subject today? I mean, I know what it is. Oh my gosh. But we titled it... Using vault with ODF for better security. Thank you. Okay, all right. Just tracking it. All right. Sorry. You're fine. So we have gone through, this page should look familiar, right? We've done, this is our OCS training page done by people in my team, the practitioners, we kind of write this up. So I actually have only contributed a little bit. This is mostly JC and Chris Flum and then all kinds of wonderful people contributing this. So normally when I talk to you, we're up here in general deploying use and we're going through some new feature. So we did a little bit of encryption last show, maybe the show before, I don't know, but we were down here in external Canvas encryption. Did we do any of it? Anyway, it's been a while. No, we did do it. No, we definitely did do it. I was just trying to remember when it was because it seems like a month or... And that, yeah, you're right. Yeah, it was like six weeks ago. Okay, so we did a couple of these sections here. I think we did all the way up to six. So I've added this new section. I know that's why it's such a magical... Brandyler persistent value in that rest encryption. That's what I'm talking about. It just goes on and on. Okay, so I just want to let everyone know it's here. However, I have a new version of this that I haven't finished the PR yet. This is the, I think a better section on the same thing. So granular persistent volume at rest encryption without cluster wide encryption, Kubernetes, Auth methods, service accounts. That's what we're doing today, but nothing should break. So just a level set with... So are you a vault guy? How's your vault? My vault was like minimal when I went into this. I mean, I've used it in the past, but it's not like I have like a vault constantly running someplace, right? Right, and I've used vault as a user. I've never been on a vault admin team or anything like that. So this is a learning curve for me. The admin team, like trying to run the infrastructure and everything. So I mean, it's a great product. I like it a lot. And it's the first one we're going with, I guess due to popularity, I know we're doing more. I don't know the roadmap for that, but I know it's planned. So here's how this kind of flows, right? So we're under data security. We have data at rest, and we have like at emotion. We're not dealing with emotions. We're strictly talking about at rest. And once you come in, you know what you have once you're in that rest, it's like you can do the whole storage cluster, the whole thing, cluster wide, or you can do storage class. And then that would be, that would be PV. That's because then you can create a PVC that uses a particular storage class, that storage class, some crypts. So we're on the storage class side, right? We're not doing the whole cluster on the storage class side. So with both of them though, and we're only gonna talk about storage class today, you could have a vault instance running in the cluster, but chances are you don't. Chances are it's external to the cluster. So that when I say external here and external KMS, that's what I mean. It's not a service in your OpenShift cluster. It's running somewhere else. Lots of reasons for that, like it's existing and that's where your security team wanted it to run. They're doing some of the BHA that they couldn't, I don't know. So- A lot of people put their vault server on its own like VLAN and everything, right? Like it's off to the side in this highly protected environment with very granular controls. Yeah, exactly, it's blowing. Yeah, you go to that VPC and AWS and all of a sudden it's like, oh, it's magical. What's going on around me? Exactly, it's your vault universe. Run by some team you've never heard of and don't know how to contact. So for that point, there are a couple things about this. Wow, so a couple of things about this demo. So when we come in, there is obviously in this situation, I have my own vault dev instance because my 30 day trial ran out. So there's some things I can't do but I have the root token. So that is probably not typical, right? That is rare. Yeah. That is rare. And because I'm running the dev version of vault, I don't have namespaces, which I think is also rare. If you're in a large enterprise solution with another team running it, they're gonna do namespaces. So we can talk about it and I can show you it, but I can't actually do it that part. So we'll just do it kind of like I'm all powerful because I'm feeling that way this morning. Yeah, you're omnipotent, that's fine. I am, right? So here, so the other thing, so we've come down through, we're not doing the whole cluster of doing the storage class and we know we're external, but the key point in this one is that we're going to use a service account token and use the Kubernetes auth method. So if you've done anything with Kubernetes auth method, so here's our cluster, here's Vault. In our namespace, there are gonna be some service accounts, every service account has a token. This guy is gonna wake up and say to Vault, I wanna act as blah, blah, blah, Vault's gonna say, who are you? What service account, what token, what namespace? And then come back to the cluster and say, is this service account allowed or not? And we have to set up that conversation properly. And that's just authentication. That's not about writing your keys out or anything yet. So it's kind of like a two-part thing. And in that case, I will set it up and make it work and then I'll show you what it looks like when it doesn't work, because inevitably I forget something and you'll see permission tonight, but it's good to see it. So let me... You know I like stuff breaking. I know you do, I know, but all right, let's... Where am I? Where did I go? Oh, oh, oh, I know what it is. See, a problem's happened already. It's not that. It's the, the, the, the, the. I want to kick off. It's not enterprise and amorphous. When it, when you know, like this morning, at like two o'clock in the morning. Okay. There we go. Oh, there we go. Yeah, so, but I want to run the Dev instance in particular. So this is now running on, I have a jump post. Here you have, you see my root token. I don't have to un-steal it. Like there's, if you have a real role instance, you have to do a lot of spike stuff with it. I don't have to do any of that because I'm running Dev. And on this side, just to show you that this is a real thing. We have that's my, this in the, in the document I'll show you, I show you what information you have to gather among that information. It's you have to know your vault address. So, and we're going to, yeah. So we're going to do just check out the status of stuff, that of that. So here, do you, you hold people out there, it's initialized and it's, it's not sealed anymore. So that is unusual. Yeah. Also in a, in a real setup, you wouldn't have to do this at all. It would only be set up and they would just tell you, this is your vault address, so here we are. I want to go back to the document for just a second. So the vault instance is external to the OpenShift cluster. We're going to use a service account token. You can use a namespace that as I said, I can't show it, really demonstrate that. I'll just talk about it. And I'll show you some other stuff about some optional things we have. You can have a really complicated vault setup. And I'm interested in from anyone listening, what is a typical vault setup? If we have anybody out there who's like, are more on the security side. I'm just, I, I definitely, I work for the client where the authentication to vault happened on one endpoint, but then the namespace was something else, which I understood because as a team, you would want to be able to control authentication in one place, maybe, but then you could, right? Like I could, I'm like, okay, so I'm learning a little more about vaults. I'm interested in that. Like, what is a typical setup for vaults? And I believe that we are already pretty much set up to handle that from the ODF side, but I want to see it in action so that we can show people like, oh, you need to set the blah, blah, blah in order to make this kind of scenario in vault hack. Does that make sense? So anyway, you'll see more. So let's go down. So here we have, please gather the following information. I've got my vault address. You saw it over here. This is it, blah, blah, blah. And I know my cluster endpoints. You can get it from OC config view if you need to, but we are generally, you know, your cluster endpoint. And then number three is important. If you don't have the root token for vault, you need someone who does because we're gonna have to set up some policies. And we're not doing this, but I can talk about that if, sorry, the vault CA certificate. Yeah, I mean, I don't blame you for not setting up a CA certificate during a demo. Exactly. It's a lot of work. Yeah, it is, a lot of work. So, okay. So the first part of member, we talked about this conversation, the conversation around authentication. So we're gonna set up our site first, which means I'm gonna need a service account. It's set up here. This is the, I'm just gonna call it token review because it's too long. I'm gonna have a cluster role, and then we're gonna bind some other services accounts to it. So down below, here's your look, here's your provision and your stuff. So you can cut and paste it. So let's apply it. I've got it in here already. It's the last way to do it. I do, uh-huh, uh-huh, uh-huh. Right, so here it is here. And just before I, let me do this before I forget because I often do. Oh yeah, good point. Yeah. Okay. Oh, yeah. I see, are you all ready? Oh. Yeah, that'll do. All right. Oh, okay, there we go. Hang on a second. It's number one. I labeled them for review. Oh, sure. So these get created. Nice. Right, like I was trying to make it easy for you, right? So that was all this. That's all that was all that in there. And then the next piece is to, we're actually setting up the vault side, but we're going to, again, you would not normally do this because you would have a team that does this, not have the token, et cetera, et cetera. Or if you're a small place, maybe you are doing this. Yeah. I think it depends on size. So maybe you are cluster admin, open chip cluster admin, and you're the vault person as well. So anyway, in order to run this job that I have below, I have to set up some pod security policy stuff, what have you. And that's all this is. This runs once, and then it's done. So that's what is in two. And this is all about setting up the job that I'm going to run right after. So you do it once, you don't have to worry about it yet. Okay, so. Created. Right. The centers here are. But you did get, okay. You did get my favorite deprecation warning and one, two, five PSPs are going away. Remember that folks, this was announced a little while ago. So PSPs will only work. But you know what? We're not going to change until they really, really make us vote. I know. And one, two, five isn't going to be out until middle of next year, I think. So you've got some time, right? But then I'll forget that they've even told me. Okay. So at this point, our side is, should be set up, but the vault side is not. So if you have a vault team, what you would do is you'd bring them right here into the documentation and say, okay, I won't wait, I got to go down. Look at this job. And this is going to set up policies. Can you do this for me? That's all. So we're going to do it here, but very, exactly. So it's provided, I thought about maybe pulling this out of YAML and just have it as a like, cut and paste this in an email to your vault admin team, maybe, but yeah, I'm just making it so. A copy of it, attach a text file or something, anything. Exactly. So in here, once this, you see these little numbers here because it has not been rendered by Intori yet, but it will be as soon as I submit it. There are a couple of things you have to change. Okay. So. At least the vault token, that makes sense. So what happens is that, hang on a second. Here we go. So up here, actually. So the vault address you would put in your vault address, the sample root token, actually, I'm going to show you the one I have because mine is set up properly. And notice I have another one there for namespaces, we'll go over that after. Okay. So let's go to the bottom. Okay. So I happen to know that my sample root token is root token ID and you can see it here because I made it that way. And then I've got my vault address. I've got, I didn't change anything else. The only other, it's like three things literally change. And then it's your endpoint right there. Endpoints matter. That's it. Endpoints matter. So what this is going to do is it's going to run this job with this in its script. It actually uses the vault image to run it. And it's going to run hopefully these policies up here. This is the vault piece. So all you vault people speak up if anything looks in this. We log in, we enable the Kubernetes authentication method on the path and then we do the policies below. So if you actually look, you can see here it's going to take in the service account token path and use that token. That is the token. That's that RBD, blah, blah, blah, token review service account. This job runs as that service account that we created. It doesn't run as this job does not run as the root plugin or anything deeper inside of the ODF service accounts stuff. So we're setting ourselves, because the idea is that you have this SA account, you have this service account that you're going to thank you, that you're going to masquerade as. So if I come in as the root plugin, it's basically saying, am I allowed to authenticate? I'm bound to this role. So anyway, this gets all set up here. You can go through it. It's pretty nice. Also of note to vault people out there, the dev vault instance is already KV version two. So apparently that matters. Yeah. I don't know why. No, I mean if you want to version two. Yeah, it's like your new version. So it already is there. So if you have a look at this, some people need to change this because for instance, the cluster identifier, I believe we have is like CSI Kubernetes or something like that. And it actually has to be different. So your vault team would know that. They would say, no, no, no, you can't do this here. You have to do this over there. Also, I just wanted to show you, I'm not going to run it, but I want it for vault namespaces, which I really think everyone probably uses, it would look like this. So when you go down, sorry, when you go down to these vault commands, everything's got some namespace attached to it. Namespace, namespace, okay, cool, yeah. Right. So one thing, not on the namespace. So someone can correct me if I'm wrong. Can you hear me? You froze for a second there. Am I bad? Just like say the last sentence. So this vault enabled piece can happen. It depends on how your vault is set up. So if you have been given a namespace and it's totally yours, then it makes sense to enable Kubernetes authentication method on your Indian namespace. But I did work with someone who didn't have that option. Authentication happened in another area, even though they had a namespace and they were writing keys to that namespace. So does that make sense? Separation responsibilities, that makes sense, yeah. Yeah. So I understood it from the vault team's point of view. They're like, we wanna like corral all of you to authenticate here. So we have this point of control here about who's even talked to us. But once you've been authenticated, you need to go right over there. So this is where I had to like actually read the documentation. Yay, distributed systems. Oh gosh, how does this work exactly? So just know that that is available to people. I don't go through all of the different things, but you can do it and I've done it. You can do a lot of different things with off-in vault and making that kind of. Exactly. That glowing to VPC we were talking about earlier. So I am interested in what is common. Like if you're a large institution enterprise with very particular security controls, what is the common setup for vault? What about how many, how large does it get? That's all. I would like to do more common setups. Those are good questions, right? Because like, where do you run your vault? Do you run it on cluster? Do you run it on another cluster? Do you run it on another cluster? You know, some server in your private cloud or whatever, right? Like there's a lot of different ways to run vault. There's open source vault, there's enterprise vault, there's all these different options too. Yeah, and like not even thinking about that. There's tons of ways you can run vault in your environment. So we try to say, hey, this is how we would do it, but that doesn't often meet everybody's needs, right? Everybody has their own requirements when it comes to security. And a lot of that is driven by regulations. So, you know. Exactly. So I'm like, if you're in this industry, you are kind of brilliant. And also, what about vault HA? Like I would like to, do I need to know anything from the OD? I just don't know. So I'm definitely- Yeah, I need some HA for everything over the weekend. Yeah, yeah. Sorry. Okay, so let's run this. Let's see. Okay. All right, so. These, see the vault, I'm gonna just, sorry, I had to make my- Oh, I know. I'm not sure if you can- Okay, so hopefully there. So we've completed. So- Nice. Let's see what the job did. And also, we can actually to some extent see it here. Okay. Oh. Ta-da. Everybody was happy. Okay. So, yeah, but normally, so someone doing this probably wouldn't have access to the logs from vault. It's actually really nice to do, but- That's pretty weird. Yeah, but it's good to show off it. Okay, so just- Pipeline. Okay, so I just looked at the logs for the pod that ran the job. So it's showing you the output of the script that ran and da-da-da-da-da. You're now authenticated. You don't have to log in again. I'm looking for failures. Here's my keys. Nothing failed. Everything looks- Yay, exclamation point. Okay. Yes. So at least now, the kind of the off part should be set up. Right? Should. Right? So we're, okay. Should. All right. And let's go then here. And here, commands for how to look at what you need to create a storage class. Aha. Okay. Here's the fun part. Yes. Let's do, you've done this before. It looks like your video first. I can do the canvas detail first. There we go. Again, can you hear me? Yeah, we can still hear you. It's- Can I make faces so you see me? I'm like- Oh. Okay. Hang on. This is a, my video is a little weird. Let's talk about, let's talk about please this config map first because it's kind of an important one actually. All right. So now that we believe, we believe authentication is set up properly, we're moving down into how keys are actually gonna get stored, right? And where we need to go. So we need to, so ODF needs to know this information. It needs to know where's your vault, what's my path to login, what's my role. Like so as you, there's a config map called, where is it at the bottom? Yeah. CSI, KMS connecting details, which contains exactly that. So it's actually in the OCS training doc earlier and it looks a little different. They changed the format. But basically this is everything you need to set up this connection. So I'm gonna take out, I was messing around with this yesterday. I'm gonna take out the namespace and I'm gonna take out the rest because I don't need it. We're not gonna do it today. So that's a vault. Oh, I see what I did. Up, up, up, up, up, up, up, up. Let's keep it simple. We can break it after. Okay. So this is it. This is all I should need to connect. And what I wanted to point out here is this. So I'm going to create a storage class. The storage class is gonna have an entry called encryption KMS ID. This is where, it's all in the document that's to show you. That ID will map to something in this config map. It's gonna map to, I'm gonna put in vault test. You'll see that in a second. In turn, vault test has a KMS type. And this is, you can go look at the code to see what this maps to. There's actually a vault.go file. It goes in, it does all the things it should. So you have two mappings to kind of pass through. But this is where that information is contained. So I could have multiple instances, multiple vault instances I want to connect to. I just cut and paste this, change the information. And I'm often running, but each one would need a new name so that I can look it up here. This will become more clear in a second. You've got the look, you're like, what is she talking about? No, no, no, I was gonna, I was pausing to ask, can you share with me, just drop it in Zoom chat here, the link to the repo. Yes. And I'm sorry if you shared that beforehand. No, no, no, it is what it is. Okay, hand. Oh, you have this third, where are you guys? Okay, there we go. Hang on, I gotta bring it back, chat. I'll take it to you here. Ta-da, there, did you get that? Yes, I'll grab that and share with the audience. They always like to phone along. So all of this, it does get a little hairy, like it's a little like, okay, we're down in the weeds, like it's very, but once I have it going, it'll make more sense. Oh, sorry, hang on a second, see? I need to do my own, I messed it up, hang on a second. Then it's just kind, let me, let me look at my own. TVK is always gonna get you. Group version kind. Yeah, here we go. Let's do this instead. Let's do CMKMS, I need to take a second, let's just play this part. Okay, I just want to get rid of these. My little documentation notes. Does everything else look good? Wanna try it? You need the vault address. Thank you. That would not have been fun. Okay. It would have been an easy find though. Okay, let's do this a little over far enough. You know, when like the, hang on a second. There we go. Okay, zero, comma, and that's not comma, that's a comma. That's it, that's it. Oh, and I'm not gonna verify it. Let me turn off this true business here. Oh yeah, that's right. Okay, so, oh, sorry, that's not what I want. I changed it tonight. Oh, you changed it, yeah. I changed it because I didn't want to go back. CMKMS, there you go. Are we happy? Okay, so let's just, I just want to look at it, make sure it's okay. It's going on. Details. Yes. Okay, we like it. All right, okay. So this is where the information about connecting to our vault instance is contained. We look it up by referring to whatever the key is here. So, let's see what I have in number five. New storage class. I'm calling it external vault SC. Nice. And this is what I want to, is it encrypted? Yes. Here's my lookup, vault test. So KMS ID is a lookup into CSI, KMS connection details. That entry should have everything I need to connect to my external vault instance. Nice. Let's see if I actually did it right. We'll go back into that in a second. Do you want, what, you had a question? Let's see it again. Well, I was just, I was just looking at the various values there. The parameters, especially, right? Like, We will, we can change them for some, yeah, yeah. So there's a lot of options here. You can kind of step your toe on or configure to your delights. Either way, you want to rephrase to it. There's some yummies in there. So for instance, if we, if you, if you want to verify your vault CA certificate, you actually do have to change some things. You have to go in and you can't use the different secrets. You create a new secret and that secret has a couple of entries. Like it starts to get a little, you can do it. Yeah. I was like, this just works fine. Typically you wouldn't do that. So maybe you would, but I have done it with a client. So that was kind of in the weeds. Fine. That's okay. We like weeds. Yeah. We like weeds. That's what we do. Okay. That should be created just fine. Okay. So now let's switch to, so I like to switch to the console. Hang on. I have to, do you know how to make that there? It goes away by itself. It's something that I want to push it out of the way. So. Yeah. You can actually grab the toolbar thing and move it off screen or whatever. Okay. Did I offer proof of what I'm running here? That I was on Odia. Yeah. Bleeding edge right here. It's got things going on. We're not going to deal with the problems. Yeah. Like these we're not dealing with that. Sorry. But here it is. 4.8 to 2. Nice. Nice. And then on this side, here we are. 4.8. So it's there. Yep. Now remember, 4.8 for ODF is not out yet. It's not out yet. Flash that future thing again. Real quick. Future thing. Okay. We've got our new storage class here. Right? That we just created. That's got our details. Just so, sorry, I'm sure this font is too small. Here. This is what matters. Can you see it? Yeah. Okay. If you can't see folks, let me know we'll increase the font size. Right. And this ball test, we can go up into the config map. Have a look in our CSI, KMS connection details. And there it is. All tests. All right. So if I, if I didn't forget anything, which we know how, and I can be in any way. So I do this. Let's just do it here. Cause it's easy. So I select this new empty, this new storage class. Let's call it external SC. Just do whatever. See what I felt. Oh, I didn't forget anything. Cause it's bad. So now we will break some stuff. So. All right. Yeah. So man, you can look at the events if you want. So this is what it looks like. Well, here's the thing. Like even with explaining it, this only took you 30 minutes. Right? Like, it's going to be a lot of back and forth if you have a security team that's managing your vault instance, but in general. Absolutely. Yeah. You know, once you have the process down, you can kind of, you know, do some automation around this. Maybe, right? Like, if you're using something like ServiceNow, just put in a ticket, I need a new key, automate that process somehow, maybe with Ansible, a little, you know, my mind is just wandering off. Sorry. Yeah. So what's also interesting is that I believe with this particular customer, the plan was to have a, for every namespace in OpenShift, they were going to have a vault namespace. So this opportunity, yeah, so you could, I thought, oh, well, that's actually really organized. So, and then they were the ones who had authentication happening at a different endpoint than where the namespaces were. And I was like, that's actually really organized. I get where they're going with that. So you can do your own application tokens, you can store stuff in it and you can encrypt your own PDs if you create a storage class that does that for you. So that was kind of nice. And they wanted all of that automated. So this is what it looks like working. If we were deeper into vault, we could see keys being created and all that kind of stuff. And the document actually goes through all of that for any other ways. So I didn't want to do that here necessarily, but I just wanted to show you this is what it looks like working. So now let's break it. My favorite part of the show. All right, let's see what we could. Well, so I want to show you some errors, but that's really kind of like, I felt like, do I have anything that I can edit? No, not you. Do I need, no. Okay, so the first thing that goes wrong is this. So let's look at our, this was what was a cluster role binding. If this isn't set up properly, you will see a permission denied error. So let's have a look. Let me edit right here. I forget what the, there we go. Okay, so if I go to the bottom here and let's just say I remove something. Well, so let's create a new PVC. Cause I'm doing torturing us today. We've created a new PVC. I should see, I'm gonna be upset if it works. It shouldn't work. It shouldn't, no. There you go, call it what it is. Yeah. Okay, so you saw how quickly it bound. Nothing should be, this should be fast, fast, fast. So here you see permission denied. Anytime you see any sort of, yeah. And, and, there you go. You can see it right here as it retries to see, yeah. So it's telling you, open shift, Rook provisioner cannot create resources because it's not bound properly. This is the point in time where you should expect your phone to rain. Cause someone's gonna see this and be like, huh, I wonder where this is coming from. I wanted to show this because also when you're dealing with another team and everybody's busy, maybe you're doing something email, right? And that's life, right? I get it, but you should, but this is so you can say, no, no, no, no. It's, it's not the vault side can go, it's you. It's not us. See, look, we're asking and we're getting the permission, getting that you're not allowed to be here. So that, that's something to see. Like, and that's the authentication conversation. There's other stuff with the CA and all that other, that can go wrong, but that's just making sure the authentication pieces is okay. So we can add a fact. Yeah, it was this, right? This is what I'm gonna add. Yeah, you just took out the last one. Yeah, okay. Wow. Okay. Do, do, do. Oh, there we go. Got none. All right. Let's make it. Oops. Oh, nice. For my own. All the spaces. Okay. All right. So now, do we have to create a new one? That I don't remember. I thought, or it may have backed off at this point, but if I go and do a new one, but I expect it's just, I suppose it backs off for a little bit. And then once, let's see, we'll leave it there. See if it comes back after. Right? It's not trying. Let's see. Oh, it's not trying maybe. There it is. Yeah. It's something just got out of that. That was me. That was me. Yeah. See now it's found. It backs off a little bit and it tries again. So it's just like, yeah. So there we are. So our bad PVC is now fixed, but permission denied is generally the cluster role bindings on the open shift side. Let me think. What else was in here? There's lots of good stuff. We can talk about creating, I create the PVC here. There's some troubleshooting stuff you can do, but if you have access to the logs of vault, it's much easier. Right? Like I included this just so people can, yeah. Yeah. You are kind of dependent upon an administrator of that vault server or system at that point, right? And aren't you busy enough? Aren't you busy enough with your own open shift stuff? You need to run home. I think it's like that. You should have encryption at rest flow, please. No. I don't want hundreds and zeros out in the breeze. That's all I know. Oh, and here is, I can't do names, but I wanted to just show you what, if we were dealing with namespaces, all that would change here is I would have to add another entry in that config map that has same connection details, but then it has a vault namespace as well. So for instance, here it is. So again, your storage class is using encryption KMS ID to look into the config map, CSI KMS connection details to get information about how it's supposed to connect. So you can run many of them. You can have a nice mix and match kind of thing. And if we look at this, I move the other one, but you could have another one. There would be, it's like, I think it's actually just vault namespace here and works exactly as you can give would, right? And you could put, you know, my namespace, my vault namespace, blah, blah, blah, blah, blah, blah. So that works. I tested it. It just, you have to worry about it getting more complicated where you authenticate. Do you want me to talk? Is there any questions? Does anyone want me to go over? Any questions? The audience is kind of quiet and that's fine. Okay. Maybe they don't have power. Maybe they're like, Oh, you're doing this all wrong. Who knows? Who said a vault like that? What is wrong with these people? And that's entirely possible because this is a demo, right? Like, Okay. So let's get a little more real world. If you had to verify your vault CA, you need to do a few things. First off, you have to have it and you have to set the time file and you have to, we're going to put it in a secret, but it's a special secret. And it's going to be a special secret. I sort of get complicated. Okay. So let's, if you have a look at, where did I put it? Oh yeah. Oh yeah. You actually mentioned it when we were looking at these storage costs under normal circumstances, you leave all of these secret names alone and you just don't bother with them. In our case, when we're going to have to verify the CA we need to keep it somewhere. We actually have to create a secret and we have to name it here and it's going to have a few things in it. So it looks like this. All right. So three important entries in our secret that we're going to name in our storage class. First one is this user ID and this user key. I'll explain how you get that in a second. That actually allows you to talk to the sub cluster. And then there's this, the OCS KMS CA secret. This is your base 64 encoded CA. You don't have a choice about this. This is how it has to be. That's the name of it. That's how it goes. If you need, every time I create a cluster obviously the keys change for this. So there's, what is it? Is it no? It's a, is that it? Yeah. You can get them like so. I believe I put that in the documentation. If I, it will be, if it's not, but I think it's there already. So here's the user ID and key that I'm going to use in this secret. Below it, I'm going to put that entry for OCS. Let's go up a little bit. OCS KMS CA secret. And here's my encoded CA. And then I add it to the storage class. By saying, I just say vault CA verify truth. Does that make sense? It's tricky. I don't want to, it's actually something. So, it's cryptid, sorry, I meant to say, but even I said this wrong. What, I will put that secret name here, because I've created a new secret with that information with those three items. And then in my KMS ID, I'm going to name a KMS ID that in it has vault CA verify truth. They're trying to keep the configuration around, the information around connecting to your sort of vault is completely contained in the KMS ID. It's not intermingled with the storage class itself. They want to keep it in that, config map, the KMS connection details. So, you just name it, you put it over there. So I could have, here, I can have more than one, I haven't done that, but I'm sure you can. And then here, you could add another one below this. And underneath here, I say false, I say true. And then I think, there's a, you can add the name for it as well to be specific what secret you're looking at. And it all kind of flows from there. But that's like a whole nother layer of getting in the week, just so you know. So Chris, just for CA verification. Oh, which, I know this first. You know this first. First, he says, you're doing a great job. I enjoy this. That's why I'm being quiet. Oh, here it comes. But did you talk about why we need this and why this is better than doing it another way? Here's another way. Any other, well, I mean, well, so I actually like the storage class level, encryption, PV method. First off, I understand, and I hope anyone listening understands why we want to encrypt sometimes, right? I think it's actually, I personally, I love the, I like the flexibility of it. Here are a bunch of storage classes. Here's where, here's how it encrypts and where it connects to, I think gives you like a lot of flexibility. So for instance, if you're doing some secret of stuff, or if you have a particular namespace and project where they need to encrypt things, it's available to you. You don't have to do it for the whole cluster. And I also like that you can do it after the fact, right? So when you go to encrypt your whole cluster, I believe that's only a decision when you go to install. When you create the storage cluster, that is where you make that decision if you're going cluster wide or not. So if you do it at the storage class level, you're okay. You can say, well, no, I'm not encrypting yet, but we'll get there. Like you can do it later. So I like options. I like all that flexibility. So in addition, I like using an external vault because of the, it's more flexibility, right? Like it's even, you can have your, the Holy Grail of setups that we've talked about before. You can have the AJ. And I like how you're not really managing tokens. You're not really managing your octopons. You're kind of letting OpenShift manage them, right? Vault is just saying, so OpenShift, is this service account allowed or not? So you don't want to deal with expiration. You don't want to do it with recycling them. Like you just don't. That's a good point. Like doing this manually, right? Like, oh, no. Don't drive people crazy. So why do you need somebody like an FTE to just manage your certs and stuff? Yeah. Right. So you could set this up so that you have the tokens that are managed by OpenShift per name space. You've got them. You can have storage classes associated with them. And you're never going into vault and saying, you know, do we delete it's part or whatever it is? Like it's all like nicely managed. It's just, I just think it's a better way to manage at scale. Is that, do you know what I'm saying? Like, I didn't think of when I think of managing vault, I think of like door keys. I'm like, oh, can you imagine recalling door keys and all this like, oh, would be a nightmare. Oh yeah, like changing locks on doors? When you put it like that, that's kind of brilliant, right? Like, all of a sudden my house has a thousand doors and I have to go change the locks by myself. Right? Right, right. And then you can go like, well, I want to delegate, I want to delegate the kids rooms to them. I want them to be able to manage their own keys or whatever. Or maybe you don't, maybe some kids you don't, I don't know. But so if you think of it that way, you're saying, oh, I could see why they would want to have you authenticate over here and have name states over there, like that's all. So, but more importantly, you really want to do it with service accounts and you want the Kubernetes authentication method so that you're not in the business of normally doing any of this stuff. You're managing it within. So, is Chris like that? Well, so, yeah, I mean, I'm assuming he does, he hasn't said anything and we've got another good thumbs up to another audience member. But like, so you mentioned you can do your whole cluster like this, right? Like, you can have every source class everywhere encrypted at rest, no problem, right? Like this, but this layer of flexibility does give you, right? Like, I would do this in dev maybe and stage and then that prod environment would be that fully locked down everything, some encryption at rest kind of deal. That way I can say, hey, listen, I know these keys and vaults and stuff, everything else is good to go in dev. I know it's good to go on stage. Let's make it happen in prod. And that way, you could do both, yeah. For the really power, yeah, you can. Tell me more. Well, so as long as you decide when you go, when you create your service cluster to actually encrypt, it's like a little button, quick. And then, and you can do both. And then later on, you could do, do per name space and stuff like that. So the dev encryption, I guess in that scenario, you think of it as you're just trying to keep others from looking at your stuff. I think I haven't met any kind that wanted to do both yet, but I'm sure they exist. Or maybe you have to just, maybe there's a regulation you have to meet or something like that or an audit requirements. But personally, I like to stay with the not close to wide and just do the PV. Flexibility. That's really nice. Yeah, yeah, yeah. Option is great. If you're mandated, you can mandate it as well. Yeah, good call. I'm thinking, what else could I ask here? And I'm gonna change the document to, I would like to, I'm going to run with what I think is a common setup, what I learned from this client as a common setup for both. So I am gonna do how you verify the CA, I am gonna add in namespaces. But again, I'm really not sure that that's actually the way most people are running it. Maybe there's a better way, or maybe someone can point me to a vault enterprise, best practices stuff that isn't like 18 pages long. So I'm interested in like, what are the common things you need to see in a typical vault setup for sure? 18 pages is a lot. I don't know. You just read a number didn't you? I did, I was like, I should have said 180. No, no, I'm sure there's a tutorial that takes you through it. But then there's also that, like that practitioner's knowledge, right? Like kind of what we bring here. It's like, oh, okay, you read this, but then how we really do it is, da, da, da, da, da, da. So I'm kind of looking for that kind of vault practitioner's knowledge. So there is a run vault on OpenShift doc from the vault project itself. So that's internal. Right, like they're using a how to install, they're using a home chart and all that fun stuff, but it's like running it on OpenShift. Yeah, that's actually a little easier. Yeah, so it's, we have, they are in tech for dev preview, I think, I forget which, it's from the future. So in the future, you could have a single node OpenShift cluster running your vault and it could just be off to the side here and running other services, hopefully too, that your security team might want. You can give them the option to have that OpenShift experience as well. That way, everybody's playing on the same playing field, right? Like OpenShift is the underlying infrastructure. It's not like we have a VM over here of a certain version running a certain OS that needs patches and everything else, right? Like you can just change the underlying image and run it like that. You could run vault in your dev cluster just to make sure that this all flows properly. And then when you go and let's say your production vault is external, then you're just dealing with that office, right? Like it's sort of, it's just a simplifying in dev and stage and how you want to lay it out. But yeah, so I'm interested in common vault practices. Yeah, I mean. I had another question for people. I don't know if I've seen the same vault practice in practice, like from business to business, it's business, right? Or organization to organization. There's always some wrinkle of difference, right? Okay. I mean, the most common scenarios are the ones like this that you describe here where there's some trusted CA in the organization that the vault server uses and so forth, so on. But that's for big words, right? Like most people that need, you know, the open source version of vaults at least are probably not thinking massive, right? Or at a scale beyond a certain number of clusters. So, you know, on the open source side, I think there's a lot of different ways to run vault. On the enterprise size maybe less, but you know, I'm curious, you know, I might dig into it myself now. Like what does vault enterprise give you? Is there any difference? Does it simplify anything? Are there different endpoints or something? I don't know. I'm used to the open source version of vault, yeah. So, I would look into that, Docs, but I'm sure they have like, wouldn't they have like some reference, like architectures or something? They do, I mean, okay, I look at it a little, but it's like- So, super simple management, data encryption and identity. Right. And the multi-tenancy, like, okay, that obviously can happen. Like it's just an understanding more about how they manage that. So, yeah. No, I'm definitely interested in what, I just feel like sometimes a practitioner, they know. They're like, oh yeah, we tried that. We do it differently because they didn't see it. Like that kind of thing. So, yeah, yeah. So, once this document is fully out, it would be nice to see some comments from people saying, can you please do the following use case because we run this in our organization and maybe we try it, right? We try some else out there. So, anyway, cool. Cool. Well, this is awesome. So, can I break something else? I don't think I, yeah. Like I- I wouldn't have you to break whatever you want. I'll give you extra time to break stuff. I mean- I was just gonna say, everything that does break here tends to be about, it's just configuration. It's really, it's a typo somewhere. It's a miss, it's all about this config map. Once the office setup, it's just very, it's super precise. You just have to like, flow through it. And it's easy to do that if you have access to the vault logs for sure. But that's it. And knowing that the vault site has been done already, right? I expect a lot of time back and forth just making sure that the whole policies are set up and that you on your side understand what your namespace is and instead of that of us. So, yeah, that's it. And in the document, I was gonna say the, the troubleshooting piece at the end actually has you drop in to an image and try and curl the external KMF. Yeah, yeah, that's a real test, right? Yeah, like just get in there and go see and you wanna run it as one of the service accounts that's supposed to be able to authenticate. So that's this line, right? Like this is- Yeah, you've got to make sure you do it too. Become that thing that is allowed to- Yes, right, right, right. That's the full test. Accessable. So you could run this and when you curl it, if it doesn't work, send it to your vault team and work it out. There it is, ta-da. Look, oh, here, when you're in your pod, this is where your service account token is. Every time. Just so you know. Do you know how, like how that, not that so it's there, it's, what else? Any more questions from anybody? Anyone want to chime in? No, no, no, no questions. Hi. Good, okay. There is meeting of people. I do like the locks on doors analogy. Like use that one in the future, because that's a good one. This is why you want to use vault, because you don't want to have to go change a thousand locks at once. Right, right, imagine walking around. Remember how I remember seeing people in buildings with like a key ring, yay big, and a thousand keys off of it. And I'm like, oh wow, how do you manage that? So you can think of vault the same way and as well as like passing cards and everything else. So once you have that image in your, that model in your head, you can see why they, why they have namespaces, why they have you authenticate in one place, but don't give you full access to your namespace to do it, et cetera, et cetera. So yeah, thank you, that's what I've got, Chris. Cool, now that's great. Thank you, Michelle. That's a wonderful demo. And you know, tons of people use vault and tons of people should be using encryption at rest. So great show today. Really, really appreciate those. And it's awesome that we've got this out there for folks to consume on their own. So if you have any feedback, how would you like people to give feedback to your scenario here? Do we have an issue or email you? That's fine, yeah, open issue, yeah, yeah, yeah. And certainly comment on the video. Yeah, I'm interested in what would you like to see? What's the missing scenario here? What's the missing use case that we should be doing to make it easier for people to understand? It's involved, it's just got a lot of detail and it's involved. So I would like to do the most common one and help people up first, that's all. Yeah, that's it, it's great. Fantastic, thank you. Cool, awesome. You've had power. I have power, I had internet the whole time. I didn't know. It's a miracle almost, modern technology. So thank you, Michelle. Thank you. Later today on the show, calendar don't freeze on me now. We have it in the clouds. We are going to be talking about DevSecOps and shifting the left, what that means. Part two, by the way, because we did part one on the level up hour last week. So we're gonna do part two on in the clouds this week. We'll have Kirsten Newcomer, security extraordinaire and Jamie Scott on board to talk a little bit about shifting left and DevSecOps. DevNations on tap today. And DevSecOps is the way, we'll be talking about data controls. So similar vein as what we started with today. That's pretty cool. And then of course everyone's favorite get-ups guide to the galaxy. We're actually gonna be interviewing Dan Garfield from Codefresh. And we've got some really cool stuff that we're cooking up with get-ups, open get-ups. There's a get-up today coming. There's all kinds of fun stuff. So yeah, full lineup today for you folks. So let me know if you need, if you wanna see it for yourself, you can go get it by looking at the calendar. And please don't forget to like, subscribe and share as you see fit this video because I think it was helpful for folks and might that repo itself might make life a little bit easier for people. Like to get the concept or at their mind around the concept, right? Sometimes you need to see the config to actually get it. And BaconFork says, thank you, thank you. So thank you, BaconFork. You're doing it. Oh, you're welcome. And we'll see you all soon, I'm sure. So stay safe up there, Michelle, out near your neck of the woods. And I'll try and stay safe here. We'll keep on pushing. So thanks, folks. Thank you.