 All right, so the next section is on the topic optimized secret management by, if you're Lua, oh, please don't kill me after I pronounce your name. But yes, if you're Lua is going to be taking that, I don't want to murder the second name. He is fondly called, if he devops. So that's what I'm going to be calling throughout the section. This section, he has over 80 years experience and across various sectors, including banking, telecommunication, media, security, and consulting, if he loves devops, just like the name, if he devops, just because of its broad coverage and dynamics. And there's never a dull moment with if he, of course, car racing games and other musical instruments are things that if I find fun and exciting. So we're going to be hearing from Ife in a bit. Ife, can you hear us? It seems like to be muted. Oh, okay, awesome. We can hear you now. Yeah, for great. It's a pleasure to be here. Thank you so much. So, welcome everyone. I'm sure you had a nice break. Oh, I mean, I'm just just to dive right in. Secrets, it's a simple concept sometimes, could also be complex. Are you sharing and trying to, we are only seeing a loop of. Oh, okay, yeah. Let me, nothing to share, but I could just leave it on my, yeah, just okay. Maybe I could leave it here. Okay, if you're not sharing anything now, we can just take the screen off then, when you want it or no, put it back. Okay, sounds good. Okay, yeah, so, yeah, just, I mean, the brief intro. Secrets could be a simple concept, could be a complex concept at times, but over time, I noticed that the approach to it affects how, you know, DevOps teams perform, you know, in terms of security breaches or even in terms of how holistic or how optimal secrets are managed. So I thought to myself, okay, based on a few challenges and scenarios I found myself in, of course, based on several use cases, how should secrets be handled? Can they be handled, you know, within the cluster? Should they be handled by a secret management tool? How do we refresh and rotate secrets? Should they be part of Git? I mean, with the whole new, you know, GitOps model, how should we approach secrets? So based on the challenges I faced and how to efficiently manage secrets, I encountered some solutions, you know, came up with a stack in how to, you know, make this seamless and easy to manage. So this is not going to be the conventional techie-techie breakdown, but I'll just try to make it as simple as possible enough to relate to it. Okay, so I'm going to just demo something simple using some tools that make up this tech stack, just to ensure that we can optimally manage secrets. I mean, at the end of the day, the whole takeaway from this is whatever stack we choose, whatever backend management tools we choose, something we can relate to it, something we can easily manage. And, you know, most importantly, something that meets the needs of, you know, the team and the organization in terms of managing those, you know, sensitive artifacts in our environment. Okay, so I'm going to share my screen now. So I just have a short, I mean, I just did some brainstorming, some typical challenges that I had faced and maybe what, you know, others might be facing in terms of the overall concept of secrets, you know, how to manage the entire secrets lifecycle from when they are created, to how they are rotated, to how they are even injected into the clusters, you know, the environment, to how they are deleted. Also disaster recovery strategies for secrets. So, you know, we have several solutions that could back up the Kubernetes clusters, but how well are the secrets also backed up? How do we have, how do we ensure that if you lose our secrets, you can recover them? Things around GitOps, you know, how should we commit secrets to Git? Things around show the actual secrets, you know, sits in the cluster. Why not, I mean, it should be managed by an external secret management to maybe properly encrypted and also guard against accidental deletion. So things like that, you know, some of the concerns that made me come up with this, with this idea. So the stack that will be used here, I'll be using our secrets operator, formerly called Kubernetes external secrets, but that's been deprecated. So it's not an external secrets operator, but, you know, a whole lot of improvements. So that will basically just be an abstraction layer. So on a high level as well, there's a stack, it will be using called Reloader by Starkator. I think it's a Swedish company also. So they also shared this solution with the community. And in terms of the cloud provider, we'll be using EKS on Amazon and our backend secret management will be Secrets Manager, AWS Secrets Manager. So some of these words, some of these terms or tools might be new, but I mean, some of us might, you know, resonate with some of them, but nothing to get worried or get scared of. I'll just deploy this in a very simple fashion and then maybe we can run through questions, you know, observations, concerns. Yeah. So I already have my EKS cluster setup. I think one of the sessions in the morning went through, you know, setting up a mini cube cluster or so any of that works as well. If you have, you know, connection setup to your secret management tool. But before now, I have my EKS cluster setup already. That's part of the prerequisite. So the first thing I'd like to have will be to set up Reloader. The charts are available in the GitHub repository. So I mean, I could probably share this later. So I'll just be installing, adding the Reloader's home chart to my local repo, Helm repo, just do some updates and install Reloader. So I'll just grab this and press here. So it's grabbing the latest charts and we'll install. Now, downline will still come to that, but maybe I should just mention this now. Setting up Reloader is pretty easy. It involves just adding an annotation. No, most of the heavy lifting has been done in the setup. So once you install the charts in your cluster, once you add them, you know, Reloader and your cluster, you are good to go. Okay. So Reloader is added. Check to be sure. So I can tell you keep sitting here will get all. Just to see that my Reloader app is running. Okay. So I see it's running. Deployment object looks good and the pod is also running fine. Now the next tool to install here will be, or to deploy will be external secrets operator. But one of the prerequisites or the requirements will be to set up or define your AWS credentials. Why do we need your AWS credentials? Because you need external secrets operator to be able to communicate with your secret management tool. In this case, I'm using AWS secrets manager. So that means I need AWS credentials. You could use an Azure Key Vault or maybe GCP's Vault or even Haskell Vault. Any of those other systems can walk behind the scenes with ESO. But in this case, I'm using AWS. And interestingly, I'm using IRSC. This is a concept I won't dive deep in. It's just IAM roles for service accounts. So rather than me defining access keys and secret keys, I have a role defined. To define the role, this is my policy here. I just need four permissions. I'm able to describe a secret, list secrets and get secrets. I can just show you that in the console. I have AWS single sign and set up just in case you're wondering what this is. So this gives me a more easy and streamlined access to my environment. Okay, so let's see what we have set up for external secrets operator. Okay, so I have a role called external secrets and the permissions right here. Get resource, get secrets, describe secrets and list secrets. So, and it can only access my secret manager resource in AWS. This is all you need. Just the user, but in this case, I'm using a role with a permission that just needs access to AWS secrets manager. You can streamline your resource for the secret but we'll get to this part in a bit. So I have my role set up. I have my permission set up and I think I'm good to go at this point to deploy external secrets operator. Straight forward as well. You can grab the helm chart for external secrets and the helm charts and repository for external secrets. I have a repo added locally already. So it's optional for me to run this. I mean, it's probably said already exists. Yep, already exists. So I will install external secrets. Now take note, I'm parsing in my values file here and this is simply the same, the default from the charts. And the only tweak here is I am running this the only tweak here is I am adding my role. Like I said, I'm using IAM role for service account. So this enables me to grant a service account in Kubernetes to grant it permission to assume or work with an IAM role in AWS. So with this, my service account can access AWS as long as I have this annotation here. And that's all. The rest of the values file is the same. Just use the default. Yeah, except you have some other interesting levels of configuration. Okay, so I will install this. I'm doing this in a namespace by the way before now I created an external secrets namespace. But this can be your default namespace. I mean, I recommend it to be in a separate namespace. It's good to always use a proper namespace strategy. So this will install the external secrets operator it's CRDs as well, I installed. Okay, hold on, I think, okay, condition already exists, okay. So I think that is pretty short already installed this but let me just be double-share, okay. So I think, yes, it's installed, great. So yeah, external secrets installed successfully. Now the concept of external secrets is after deploying the operator, you need to set up a secret store. What's the store? The store is simply a backend object that establishes connection to your secret management tool. So it could be HashiCore Vault. It could be Azure Key Vault, like I said. It could be, in our case, you need to be a secret manager. And this is the manifest here. All of these charts and manifests are in the home chart repository, but I will just explain what the secret store does. So this is the manifest. I'm giving it a name AWS secrets manager because that would be the name for AWS. I mean, it's just defined. It can be anything, you can call it anything. The region of my secret manager will be EUS so that's the London region. So that's the region I will be using which is this region here, secret manager. Right here, I think I should be in the, okay. Okay, so let's go to the London region, EUS too. That's where we should be. Okay, great. Next up, the service is secret manager. So in external secrets, when you say secret manager, that means you're referring to AWS secret manager. I think if you're working with Azure Key Vault, it has to be maybe something like Azure, maybe KV. And I can't recall, but it's there. So depending on the backend tool, that will determine the kind of name that should be here. But of course, this can be any name because this is just a friendly name for you to identify it as one of your Kubernetes subjects. So I'll deploy the cluster secret store now. And while that will be deploying, I will explain one concept. Now in external secrets, we have what we call the cluster secret store or secret store. And you have what we call cluster external secrets or external secrets. If you notice, it's just the word cluster that is different in both. And why is that? It means you can choose your object to be accessible throughout the Kubernetes cluster or you can choose to tie it to a particular name space. In this case, I'll just like to have one secret store to serve all secrets in my cluster. So I'll go with the cluster secret store. So this is defined, this is created, okay? So let's see, let's be sure, let's describe this object, let's see what's happening now. We have a cluster store created. Of course, with backend connectivity to AWS. Although the QCTL gets cluster secret store, let's see what we have, okay? We have this created. I think I'll like to describe this as well. I missed the word described. Okay, so when I see an event like this, this is just a way to validate that your external secret operator successfully, you know, can successfully authenticate to AWS using the role or the credential you provided, maybe an access on a secret key stored somewhere or your role. In this case, I'm using IRSA like I said. So this is good. This is a good sign that we can interact with AWS and we can also interact with the AWS secrets manager. Now we have, we load that deployed. We have external secrets operator deployed. We have this store created and deployed as well, which connects to AWS. I think it's, let's get to the fun part where we start creating secrets. Now like I explained the concept of cluster external secret or external secrets. I want to be able to create secrets from anywhere and map to any name space. So that's why I'll opt for the cluster external secrets. Now, before I create my cluster external secrets, they'll just be one peruquisite. And I'll explain that now. This is what the cluster external secret looks like. This is what the manifest looks like. I'll just run through this so that you can understand what each section means. Of course, the name is the name of your cluster external secret objects. Now the external secret name is the name it's used to create or it will give you external secrets. What's the difference there? The external secret will be created in whichever name space you define. So remember, you are managing the Kubernetes infrastructure. You're using the name space strategy to properly isolate workloads, your environments. I mean, maybe you have a name space for dev, a name space for staging, or you have a name space for critical apps and less critical apps with several levels of isolation. At the same time, whatever, which of our secrets are consumed in a name space has to be within that name space. That's the whole idea of the virtual isolation, which regards to name spaces. So here I'm saying creates an external secrets and provision it to this name space. In this case, I'm using the default name space. This can be any name space. This can be maybe an ABC app name space. What does refresh time mean here? Refresh time simply says external secrets operator. Make sure that these external secrets always exists in this name space. And who is in charge of that? It's the cluster external secrets. So every one minutes, the cluster external secrets checks to make sure the external secrets is in the default name space. But it gets even more interesting. I mean, there's another layer just to ensure 100% or maybe near 100% protection or availability of your secrets. The next layer there is external secrets in turn also creates a secret within the same name space. So you'll end up seeing three objects. You'll see an object of kind external secrets. You'll see an object of kind cluster external secrets. And you'll also see an object of kind secrets. Of course, the one we're all familiar with is secrets, which that is what our application will be consuming. We'll get to that part in a bit. So for the external secrets, it now creates a secrets called test secrets. Of course, this secret story is just saying, hey, remember I created a secret store named AWS secrets manager. That's the story should connect with back end to fetch my secrets. So that's this just like a reference call for synchronization. Hey, check my AWS secrets store fetch all my secrets or fetch my secrets named this. You'll see a section called them data from or extract. Like I said, this might be new, this might be strange, but I'll advise you check the docs as well. I'll just try to do justice to explain a bit further, but the more you relate to it, you'll understand what each section is all about. So three kinds of objects will be created when I apply this manifest, which I'll do now. So let's create our cluster external secrets. Oh, before that, like I said, create secrets and secret manager. And why am I doing that? At the point when I'm telling it to fetch the secrets, I need to make sure that I have a secrets named this in AWS or in my secret management tool and it's all I'm using. So I'm going to show you that. So I have the secrets defined and this is the name of the secrets. So what I'm telling it here is extract the content of this from AWS and please create a secret for me in Kubernetes with those contents. And how do you connect? Please use this to connect the secret store. Now this year I'm achieving so many things here. So I'm going to apply this and we'll see what happens. So let's just create this. I just have the syntax handy just to save time and efforts. Okay. So I have the cluster external secret created. So let's see. We should have something called cluster external secrets. Yes. Which in turn creates an external secrets in the full name space right here. Which in turn creates a normal, the regular, the usual secret object which is what the application will consume. So now let's see get secrets. And here we have test it. I mean, we are going to see something fun down the line. I could choose to delete the secrets. I mean, this is one of the pros of external secrets because of the synchronization. I can choose to delete the secret and it will create it's QCTO, delete secrets, test secrets. So to refresh every, I think one minute or even immediately, let's see. Yeah, so you see, let me just clear this. Okay. Secret here, 50 seconds old was deleted. And here it automatically recreated it. So let's say someone earnestly, deletes maybe just a typo and deleted the secrets. You can be rest assured that that secret will be recreated. If you're not using an external secret management tool and if you're not using external secrets operator, there's no way to recall a secret on like deployment objects or maybe stateful sets that have maybe a test like revision history limits where you can roll back objects or depending on if you do upgrades, there's no way to roll back or recover you know, objects like config maps or even secrets once they're deleted unless you have them saved somewhere maybe locally or in the privates in the repository to yourself. Okay, so we have created our external secrets. We've also indirectly created our secrets. So I think we are ready to test this. Let's deploy a test application. I just have a busy box here. Which I'll run, which I'll deploy. And I made two additions to the busy box. The first one is the annotation for reloader. And this will simply ensure if I update my secrets, I don't have to manually do a rolling restart. It automatically redeploys, restarts all the pods. Now that will come in handy when you have lots of, it may not mean a lot if you just have one deployment object or two pods. But when you start dealing with hundreds of deployment objects or stateful sets and you start managing a huge number of microservices and that's when this will come in very, very handy. So I have this annotation here that says, hey, just look out for all my maps, secrets and config maps. If you notice any of them is updated automatically, restart my pods. That's the first addition I have here. The final addition I have is, of course, I'm bringing in my secrets. So I'm just saying, hey, fetch some environment variables from these secrets. Remember, this secret is the third secret that was created from this, which is right here. So this is the third object that was created by me applying the cluster external secrets. So let's deploy our test app. Let's see what happens. I don't have a very decent task for this. So I'll just run this. Yeah, let's apply. Okay, let me just clear this to have a clear screen. I'll get all now, let's just see what's happening. Okay, I have my pods running. Let's, okay, I think let me exec into one of them. Just to, let's see what's going on. We just want to validate our environment variables. Okay, so I'll say print ENV. Okay, I think let me just, set this properly, let's say, okay. Let's want to see what I need. Okay, so this environment variables in the pod, let's see, and they match what we created here. So what's the entire flow of this? I'll just run through it. External secrets operator connects to AWS through the cluster secret store. That's the first part. The second part is the cluster external secrets creates an external secret in your defined namespace. And cluster external secrets always makes sure that the external secret exists. In turn, the external secrets created the secret object, which I assigned to the pod or to the deployment. And it always makes sure that that secret object always exists. So if I delete the secret objects, external secret automatically recreates it. If I delete external secrets, cluster external secrets automatically recreates the external secrets. So you see there are more than one level of protection here to prevent an accidental deletion. So let's say, let's see what happens now. I think this is the final interesting part. I'm going to modify a secret. Let's say I add maybe one of the locations here. So let me update the secrets. Let me add something here. Let me say KCD, let me say city. So let me just add the main capital here in Lego. So I'll just say, let me say Kejha. Kejha and I will save. Okay. So this is saved. So let's monitor what's happening here. If I do it, get all, you might observe something. Let's see if that's what happened. I expect Reloada to go into action here. Okay. So I think the refresh time is one minute. So just wait for a few more seconds and you should expect any deployment to kick in. Okay. Reloada is active. So we see here that the last set of pods are terminating while new pods are running. Remember this happens in a rolling fashion. So there's no downtime, uptime is guaranteed. And of course, this also respects your update strategy. So if you're using maybe a fresh redeployment of everything or a rolling restart or a rolling upgrade, that's what will happen. Reloada, respect, respect, respect, respect, respect your update strategy. But by default, the default Kubernetes of the strategy, which most, I mean, everyone uses this, the rolling upgrade. So it ensures that it iteratively shuts down each pod one at a time and then spins up each new pod. So it takes out the one, brings up a new pod, takes out pod two and brings up a new pod just to be sure that all our new pods are running and to be double sure, I will just connect into one of the new pods and be certain that we have we have our new environment variable. So I'm going to print the environment variables in the, and search for a CD. And I have KCDCT, Keja. Voila, so this, I mean, it looks like a lot of heavy lifting has been done here and this achieves the goal which you want to achieve. Now, I mean, the approach of this stack, I just did some brainstorming and I said, okay, secrets cannot be part of the GitOps model. I mean, you don't want to be committing actual secret manifest into Git but with external secrets operator and with this entire stack as a matter of fact, you can have your external secrets manifest in Git. In fact, depending on your GitOps strategy, in some cases, if you're using tools like Algo CD or Jenkins and JX3, you can have some levels of integration where when you make a change, Algo CD automatically does, you know, recognizes that new artifact. By the same time, if you're using JX3, you can, you know, build a model that says automatically connect to AWS and create secrets. So that's the next layer. Maybe the next, I mean, maybe the next time KCD who knows maybe that will be featured but that's something also, you know, that's possible. You can automatically create secrets from AWS without manually defining your secrets. So just have something and just properly define your GitOps model and that is very much possible. Automatic secrets rotation. I mean, the option is here. You can configure, imagine these are database credentials or any other form of credential. You can configure automatic rotation, DR strategy, I think adjust to safe cost since it's just a test but you can also set up some replication if I attempt to create a new secret or I think it's even here. You can choose to really get secrets to other regions. So you're not scared of the fact that the region may go down or your cluster might be on a, you know, unreachable or your cluster crashes and all of that. That automatically handles your disaster recovery strategy. Auto secret synchronization, like we saw, you delete a secret object, it automatically gets recreated. You update a secret, it automatically gets, you know, the pods gets redeployed in a rolling fashion. I mean, a couple of that. And the concept of the stack, nothing is perfect. There's always room for improvement. So to me, so far so good. I feel it's just one more layer of operational management which happens just once, which is at the point of setup because the rest of the operations are basically, you know, I'm automated. So if I'm to deploy a new application, I just have to create my secret manifest, create my backend secrets and AWS and redeploy. Whenever I update my secrets, reload it automatically, reloads. So I feel with this, I think optimizing secrets, you know, should be more user friendly. It shouldn't be that scared topic handled by only the maybe experts or the gurus in the Kubernetes community. I think everyone can relate to it. Everyone can just deploy a simple stock depending on maybe your cloud provider or your secret ment tool of choice. And you can achieve an optimal secret management lifecycle, you know, without being an expert. So I think that's the end of my demo. I'm going to stop sharing. Yes. Thank you guys. So if you have any questions. Awesome, that was a great session. I've always looked forward to secrets and different other options around the secrets management. I think one of the most popular ones is Vault. I've never seen a demo of that of AWS. I don't, we don't have any questions yet from the audience. I don't know if anyone has any questions if we can share for, if you're lower to answer. Okay, someone is asking, so there is Kubernetes service that dots this. Kubernetes service. I guess you're referring to the operator, external secrets operator. Before now it was Kubernetes external secrets, but that has been deprecated and improved. So we now have the external secrets operator. So for now we can see that is the Kubernetes controller of service that does, that is the backbone of this entire stack, basically. Sir, I think you're on mute. Don't mind me. I'm muted from the microphone, but I was seeing the screen as, I see everything was fine. Okay. Sir, thank you very much for the awesome presentation. I mean, look forward to having you now subsequent event. It was a very great presentation. Thanks, the pleasure. Thank you. Okay.