 Everyone, can you hear me? Cool. Awesome. Yeah, welcome, everyone, to my talk, Protecting Your Crown Jewels with External Secrets Operator. My name is Moritz Iona. I am one of the maintainers of the External Secrets Operator project. And for my day job, I work as a senior software engineer at Form 3, where I work in the platform compute containers team. And we build the Kubernetes infrastructure for all of the other application teams across three different cloud providers and on-premises, which is a huge task. Yeah, there's also Lucas and Gustavo, which are here today, there in the first row. And we also have a booth in the CNCF project, Pevel Young, in Hall 5. So if you have any questions afterwards, I don't have the time to do a demo today. If you have questions afterwards, you can feel free to just talk here. Or afterwards, just in the booth. So yeah, without further ado, we are here to raise some awareness that this project exists, because I don't know if people are aware of it. And I would like to do a quick show of hands and ask you, who of you is aware that this project exists and have a rough idea of what it does? Cool, awesome. That's quite a few people. Awesome. And who of you guys are actually using it? Way less, but still, a couple of people. Awesome. So if you have the time, feel free to come by our booth. We would like to learn more about how you use it, with what kind of stuff you struggle with, and how we can improve. So feel free to come by. So what I have for the agenda for today is, first, the foremost, I would like to give you a rough overview of what Secret Management is, what kind of the challenges you have typically when you want to implement Secret Management. This will be roughly about 10-ish minutes for those watching the VODs, and you're already familiar with it. Just skip the 10 minutes. After that, I'll talk about External Secrets Operator. Everyone else who's familiar with Secret Management, feel free to pass out, and I'm going to catch you later. So what is Secret Management, and why should you care? In every organization, we have some sorts of secrets or credentials that allow members of your organization to access internal services. That could be a monitoring system. That could be production access to a database. There could be a lot of different things. So these are the kinds of secrets we're talking about, mostly credentials, username, password combinations, API keys, all that kind of stuff. And that's important, and you should care about it, because if they end up in the wrong hands, they could potentially have a big negative impact on your business. Like if an adversary gets access to your production data, he or she would be able to exfiltrate the data and sell it on the black market, and that will have a huge impact on your business due to bad reputation, et cetera. So you really should care about that. For short definition, Secret Management deals with everything regarding the life cycle of secrets. So how are they created in the first place? How are they stored securely? How are these secrets audited, changed to it, audited? How are these secrets transferred between systems? And how are you rotating these secrets, and how do you retire secrets? So that's all about managing the life cycle of these bespoke secrets and credentials. I'd like to give you a little bit of a structure to think about secrets, just four attributes that many secrets have. First, their expiry date. Some secrets, they live eternally. They don't expire, and you can just use them today, use them tomorrow, and in the next 30 years, potentially. And other secrets, they are just ephemeral that have a limited lifetime, like one minute, one hour, or one day. Second, their creation. Some secrets need to be created manually with human action. Others are created automatically as part of a CI-CD pipeline, as part of a protocol, like OpenID Connect. Three, some secrets have dependencies. For instance, if you would like to get a new certificate and you want to get that signed by a publicly trusted certificate authority, you need to hand over the certificate signing request over to that publicly trusted CA. They're going to issue the certificate, sign it, and hand it over back to you. In the moment when you would like to rotate the secret and you want to be as fast as possible, you depend on them, that they are available, that they respond in time, et cetera. Other dependencies, they have other secrets, maybe have a dependency on your internal organization, on your same team. Others have just no dependency at all, which makes things way easier. Fourth, the consumer. The question, who uses that credential matters to you and your secrets management procedures? Because users tend to click on links on emails. They like to open attachments and enable macros and stuff like that, so they are more likely to do these kinds of bad things, and they may get phished. So you need to have special care for these kinds of people, not these kinds of people, but human users out of made of flesh and bones. On the other hand, applications, they're just program code. They do whatever they are programmed to do, so probably a little bit easier to manage. So where do we need secrets? We have them on development laptops. You have your AWS CLI, your GCP CLI Azure, your Terraform, et cetera, these kinds of tools. They need secrets in order to fulfill their job. If you do a Terraform plan or apply, you'll need access to your cloud provider. Second, you have some sort of UI logins, like your monitoring dashboards, like Grafana, or your monitoring system that you'd like to access. I don't know, all that kind of stuff. You need secrets there as well. Second, in CI-CD systems, it's pretty much the same thing, but within a given CI-CD system. So you do the same things there, Terraform, Ansible, USB to boxes, do whatever you need to do there automatically. And also, for all these application-related workloads, you build a container image, and you push it off to a container registry. And of course, you need credentials for that. Third, which is the obvious one, operational environment, your infrastructure applications, your software as a service products, your cloud and on-premises stuff. You also need secrets there as well. So what I'd like to show here is that you have secrets basically everywhere. It's not only the tech or engineering department that deals with secrets. It's also like the marketing department who also may have access to a monitoring system or some other internal system that provide them admin capabilities to enable or disable the user or provide them support. So it's basically everywhere. And I hope it comes clear that this is not a simple task. You have so many different kinds of secrets, certificates, API keys, which need to be rotated for different environments and use cases. So that's not an easy task. What do you do today? You just centralize and win. You have, in theory or in an ideal world, you have one vault where you store all your secrets, all your credentials in, and it provides secure storage, auditing capabilities, when which secret was changed by which user. It provides authentication and authorization layer. It provides APIs so you can rotate secrets easily by the click of a button. And it provides some way of integration into your application or infrastructure landscape. There are different tools with different purposes. Some of them are software as a service. Like, I don't really endorse a particular vendor, but just so you get a rough idea. You have tools like LastPass or OnePass, which are mostly focuses for users, where you store these kinds of shared secrets for say a monitoring system or stuff like that. So these are SaaS solutions that you can have. On the other hand, you have cloud-provided APIs like AWS Secrets Manager, GCP Secret Manager, Azure Key Vault. The same goes for IBM and Oracle Cloud. They all have a pretty similar API that you can use to store your secrets, audit them, et cetera. And others are hosted on-premises, so you host them and deploy them on your own. But in the end, provide the same functionality as just an API that allows you to read and write secrets in them. But only storing them is not enough. You'll face a lot of challenges, which still need to be solved, even if you store them in a central vault, one of which is Secret Sprawl, because organizations, they evolve over time. Teams and departments get restructured. That means that developers and access permissions also need to move. And also sometimes developers, they just need to cut corners because you have delivery times that you'd like to fulfill and so on. So sometimes secrets are stored in source code. They are stored maybe as a hard-coded environment variable somewhere. That's just happened in this real life. And in the end, your secrets are scattered all over and you don't have proper auditing capabilities for these kinds of secrets. So that's still something that you need to keep in mind. Second, you have to implement the actual lifecycle management and access control, which is not easy, like rotating a secret. Sure, you can do that, write a little bit of code for it, can test it and apply it and rotate it in an automated fashion. But you have to do this for every secret, for every certificate, for every kind of secret you have, for every API key. That's not a simple task and requires a lot of engineering and effort and time that must go into that. Third, integration and tooling. Yeah, you don't get that one for free. You have to do it yourself. You have to integrate all that stuff. You need to integrate authentication mechanisms between your Kubernetes cluster and your secure vault. You need to authenticate. You need to set up the roles and policies so you're able to pull these secrets into your cluster. Fourth, yeah, bad practices. If you have bad passwords that can be guessed, then, yeah, it doesn't help you. So all right, that was the first part. Time to wake up now. Now I'd like to talk about secret management with external secrets operator. But before I dive into the technical details, I'd like to give you a brief overview of the history of the project and how it came into existence. So for that, I'd like to go back like four years in time. Slaya, it's 2019. That was a year of Kubernetes 1.14, 15, and 1.16. That was quite a while ago. And during that point in time, you had basically like two different secret management operators that you can use, one of which is sealed secrets, which is still very popular today. The other one was GoDaddy's Kubernetes external secrets. And at that point in time, it kind of became evident that the project, which was written in Node.js, which is good and which is fine, but the Kubernetes client implementation at that point in time was unmaintained. And a complete rewrite of this operator was needed in order for this project to move forward. In 2019, around, yeah, roughly around 2019, there were many projects which pretty much did the very same thing. They just talked to some provider, AWS Secrets Manager, GCP, whatever. They pulled secrets out, and they store them as a Kubernetes secret resource so that applications can pick them up as a file, environment, et cetera. There were many projects which were not so popular, like maybe 30 stars on GitHub, 50 stars on GitHub. And in issue number 47 in that project, that's where I met Lukas the first time, we discussed that we actually would like to gather all the different projects together to come up with one solution that does it and does it well and to build up a community out of that, because you have multiple maintainers from different companies and so on. And that's essentially what we did in 2020. We joined forces, came up with a CRD proposal that we'd like to implement with all the abstraction and all the different lessons that other projects have learned at that time. And as of today, we have like 2,100 commits, like two and a half years later now, and we have over 200 contributors. We're like five maintainers from different companies and we joined the CNCF as a sandbox project last year. So that's where we are today. We have a project booth, thanks to the CNCF for that. And we now have the opportunity to get to know our users a little bit better. So everyone who uses it or would like to use it, feel free to come by at their booth and we'd like to have a talk with you. So let's take a look at how this thing works. So you have this operator, which is the blue robot that runs in this Kubernetes cluster, that blue box, and everything that the operator does, it talks to the provider, AWS Secrets Manager, GCP, HashiCorp world, what have you. It talks to the provider, says give me a secret, and it stores it as a Kubernetes secret resource. And it does that not only once, but like in a regular interval that you can configure. This is like the basic abstraction, like on a very basic level, how it works. And around that abstraction, we've built custom resources. On the one hand side, we have the secret store, which represents the provider, like which provider would you like to use, AWS or GCP? Two, it also contains the authentication mechanism that should be used to connect to that provider. You have like different mechanism that you can use, which varies from provider to provider. As of today, the best, I would say, that's maybe a little bit opinionated by the best way to authenticate with the provider is to use a Kubernetes service account, which is supported across all the different major cloud providers. So that's the secret store, which provider should be used and which authentication method should be used. On the other hand, we have the external secret resource, which points to the secret store. I'll show you an example in a bit. And it contains information, which secret should actually be fetched, like give it a name, back end API key, what have you. And the external secret also describes how the Kubernetes resource should look like, what kind of key value pairs would like to have in there, what annotations and labels and whatnot. So yeah, that's on a basic level, that's how that works. So here's an example. On the right hand side, you can see the secret store. It's pretty easy and straightforward. You can just look at it. So we have a secret store resource. You define a provider, which is AWS, and you define a service, secrets manager, a particular region that you would like to talk to, and then a simple authentication block, where you just point to a service account and then that is used to authenticate with AWS. And then on the left hand side, you can see the external secret resource, just as mentioned with a secret store reference, which just points to that secret store. You define a refresh interval, so that's the interval how often the controller should ask for updates. So every 30 minutes it will ask, hey, do you have an update for me? And it just fetches the value and writes it into the Kubernetes secrets. Last but not least, you have this data block. That's just for this example. And what you can do there is it's a pretty basic mapping of secrets with their name on the provider side. So you have secrets in HashiCorp Vault with the name backend slash grafana slash API key. That's the one here at the bottom, remote ref key. And you have the secret key. It's a little bit confusing, but I hope you get it. And once you did it, then it's kind of obvious that defines the key in the Kubernetes secret resource. So you have the remote ref, that's the provider side, how this thing is named there. And then you have the secret key side, which is the Kubernetes secret resource, how it should be named there. There was a lot of secrets in that sentence. Sorry for that, but I'm not going to stop with that. So yeah, that's like the most basic example. You just map some keys from the provider into Kubernetes. There's a little bit more to it. That was just the most basic use case that with which we started like two and a half years ago. As of today, we have a little bit more custom resources. We have for both secret store and external secret, also like a cluster variant that you can use to define them on a cluster level. So that allows you to reuse the secret stores and external secrets across different namespaces, which is handy if you provide these secrets as a service. These APIs are on the left-hand side because they're beta today and we'd like to push forward to a GA release in the next couple of months because we have a lot of feedback from users already and a lot of production burn time and we're pretty confident that we can ship that what we have today as a GA release. And on the right-hand side, we have the push secret and the generator API group. These are like two other use cases. One is the push secret, which basically turns around this whole process. It reads from a Kubernetes secret resource and it pushes that value to a provider. I'll show your use case later on. And the other is the generator API group, which is, as you might have guessed, you can generate values with it so you can generate a random string. You can generate short-lived tokens with it. So this is still an alpha. It is not yet, it's a little bit rough around the edges, but yeah, we're slowly getting there and once it finds a more adoption, we get more input from users and going further stabilize it. Here's a rough overview of features, pretty opinionated list that should give you like a rough overview or rough idea of what it does. First and foremost, the zero configuration authentication part, which is the most important one to me, because if you run this operator in Kubernetes, you need one initial secret that allows you to access the secure vault and the only way to do that, or the best way to do that is to use Kubernetes service account to authenticate with your vault. On AWS, this is called IAM roles for service account or short-IRSA. On GCP and Azure, it's called Worklet Identity, and you can simply use that to authenticate with a secure vault, which is pretty, pretty handy and it allows you to not need to define this initial secret that provides access. So the operator also helps you with lifecycle management. It doesn't rotate on its own. All rotation must be happening on the vault side where you store these secrets, because this is the source of truth, so it has to be rotated there, for sure. It allows you to merge secrets into existing Kubernetes secret resources, which is handy if you have some help chart that cannot rely on some extra resource, but it comes with the secret itself. But these are just minor details. So the operator also helps you with secret distribution across namespaces, so you can define a secret one external secret once, and you can distribute that secret across different namespaces. Best example would be you wanna have an image pool secret, and you wanna have that, for sure, in every namespace or just in a namespace that matches a particular label selector, all that is possible. You can do cross cluster synchronization, so you can pull secrets from other clusters, and you can also, in the future, people start working on that, also push secrets to other clusters, which helps you if you have one central cluster that bootstraps other clusters with, say, a crossplane or cluster API, and you can also seed secrets to that cluster. You can do secret templating, so if you have an application like, say, Grafana, that requires some configuration file, which also contains secrets, for sure, like some username or passwords that need to be in this config file, you can template these config files in the external secret resource, and the operator will fetch the secrets, render it, and store it as a secret resource. You can fetch multiple secrets, you can aggregate secrets based on their name, based on a regular expression, based on tags, I'll show an example on that later, and the operator is designed for multi-tenancy purposes from the get go, so if you have like a larger team structure, and you want to apply different rules for different tenants within your Kubernetes cluster, this is like the best tool to do that, it allows you pretty much fine-grained policies on a namespace level, so team A is not able to read secrets from team B. So here's an example regarding the template, here we have an external secret resource, we just ignore all of that stuff, so at some point you have this multi-line string, which is a simple goal-length template, and you have these double curly braces expressions, and what the operator does, it fetches these values here, applications for password, applications for user, and it renders this template with the given secret values, and then in the end you have a configuration YAML file that contains the passwords and users in clear text, and this is stored as a Kubernetes resource, and then you can mount that as a file into your application and you're good to go. You don't have to do this as an inline string, which sounds a little bit hacky, you can also store that in a dedicated config map also, so that helps you with a little bit of cleaner design. Here's another example, so the idea is that your application requires multiple different secrets, here you can pretty easily get an idea about the pattern, so backend slash database slash crats, backend slash cache slash crats, these are all secrets in your vault, they have some value which we don't care about now, and what you would like to do is you wanna fetch all the secrets matching a specific regular expression, and the operator should fetch all these secrets and store them in a Kubernetes secret resource, and that's what you can do with this example here at the bottom right, you just specify a data from, let's say find by name, specify a regex, and everything, all the secrets that match that particular regular expression will then end up in a secret, so when you add a secret, it will automatically be stored in a secret resource, eventually when the operator decides to do a reconciliation right now. A different example, here you have one key database slash crats, and you have a structured JSON, and what you would like to do as an application or as a user, you would like to have environment variables with this pattern, MySQL user, pass, host, port, blah, blah, you would like to have these environment variables in your application, in your container. You can simply do this with external secrets operator, you can specify this data from extract, and specify the same key that contains that information, and what that does, it reads structured information, JSON in the end, and stores each individual key value pairs in the Kubernetes secret. Here are two use cases that you can do with external secrets operator as well. On the left-hand side, you have a tooling cluster, which contains some sort of secret that you would like to pull in into your workloads cluster. To do that, you can deploy ESO into the workload cluster, pull that secret from that other cluster and distribute it across different namespaces or whatever your need is. So this is like a pull-based approach, deploy ESO in there, pull it and distribute it, simple. And on the right-hand side, you can see that inside this tooling cluster, that's also where you can deploy ESO, and ESO is able to push secrets to other clusters, to these workload clusters. So that's the use case when you have cluster API or cross-plane, some central cluster that bootstraps your infrastructure, it can also bootstrap or seed secrets into these newly created clusters. This is not yet supported. There's an implementation missing for the push secrets feature. People started working on it actually like two days ago, so I hope it will be available in the next couple of months, weeks, hopefully. So let's see how that goes. Here's an example of how this multi-tenancy thing could look like. It always depends on the organization because every organization is different. Their role-based or their role access model is always different. Here in this example, I just implemented a simple color coding. We have yellow on the left-hand side, that's one team or one logical unit within your organization. You have the green one, which is just a different one. So what you can do is that you have inside a namespace, a secret store, and that secret store provides access to their AWS secrets manager account. So in this example, we have different AWS accounts, account one for the yellow team, account two for the green team, et cetera. And that secret store provides access to only that account one, one, one, one, the left one. And with that, the yellow team basically is able to read all the secrets only in their account, but is not able to read the secrets in the other account. What you also see quite often is that you only have one single account and you share your secrets manager instance with different teams. You can also implement that and still provide proper separation of secret access. You can do this. This is the example for AWS secrets manager, but it's also valid for GCP, Hushacrow Vault, et cetera. You can do that there as well. What you can do is that you create a secret store resource and you deny access to developers to change that secret store with a tool like OPA, Gatekeeper, Kyberno, what have you. So they are not able to change it or at least change parts of it. So with that, you can basically specify a policy or a role on that secret store that is assumed when ESO fetches the secret. So as a result, the yellow team is only able to pull in the yellow secrets from AWS secrets manager, the same goes for the green one, et cetera. Yeah, and that's pretty much it as a rough idea. So again, as I just briefly mentioned, we would like to target a GA this year, hopefully the next couple of months kind of depends on how much time we have because this is still like a project as a side project from like many of us. And we'd like to welcome people who want to like to learn about open source. I'd like to learn how to contribute to open source to come in, feel free to talk at us at your booth. Further, we'd like to stabilize the push secret and generate the features which are still relatively fresh and new. So yeah, please reach out to us. We can find us at the CNCF project pavilion in hall five and thank you so much. Any questions? I have a question. Hello, I've got a question over here to your right. Ah, hi. Can you help me understand the operator, the RBAC permissions that it needs from a secret standpoint? Does it need to be able to read and write the secrets across all namespaces of the cluster? So you have different ways to run the operator. The usual way is that you run it in a highly privileged mode similar to a search manager that it has access, read and write access to all the secrets across the cluster. But there are like different, like other deployment topologies that we support. One is namespace based. So you're deployed into a namespace and then only stuff from that namespace can be read and write to. Great, yeah, that's what I was hoping that you'd have both options there. Thank you. Yeah, sure. Hey there. Yeah, what should we be thinking for extensions to the provider model? Like if I have a sort of a backend storage system that is either not something you already have, may not even be generalizable because it's something internal to a company, how should we be adding support for that into the operator? Yeah, good point. So this is basically like, I think the first or second item on our list is to provide some sort of plugin mechanism. So that allows you to develop something internally and then just use it together with extra secrets operator. What you can do today is we provide a webhook provider which just sends out webhooks and then you just run a service that fulfills the contract and then that would also work, but that doesn't work for like all the use cases we have today. So this is pretty much very high on our to-do list and feel free to jump in, yeah. Thank you very much. Sure, welcome. I have another question here. So it may be a dumb question, but so the external secrets operator creates Kubernetes secrets, right? Yeah, it creates Kubernetes secret resources, yeah. Which are, I guess, base 64 encoded. That's right. Yes, that's right. Any plan to add any encryption there or? We do not provide any encryption. You have to solve that on the Kubernetes level. Like you would like to, you should do encryption on an LCD. Okay. And yeah. Is this one on? Yeah. So I guess that my question is like, why would you do this replication in the first place? If you have like a workload that entity and you have an external secrets provider, why not provide access to the external secrets provider directly? Yeah, that's a good point because that works in 90% of use cases, but still you have some cases that are not covered for say like platform applications like Prometheus, Grafana, some other maybe vendor solutions that you just deploy into the cluster and then there's no way how you can get that functionality into the tool. Maybe you can create something like an init container that just fetches that and then these are all like sort of workarounds, but they're true. And also you can use Kubernetes secrets and it is better because you don't have to implement it in each and every application. So that's like another platform service that you provide and you don't have to implement it yourself if you are able to trust at CD as a secret store. Yeah, yeah. Which is like always a concern for Infosack. I guess it's kind of changing in the landscape today where I feel that secret resources are fine to be used but that's like depends pretty much. Thank you. Yeah, welcome. Is it possible to generate anything else than time secret? I can repeat the beginning, I didn't get that. Is it possible to generate anything else than kind secret? What do you mean by unite? Than kind the secret. Generate a CD object or something like that. I guess you can use Helm? No, but we want to generate it from the same way when Helm doesn't have access to the secrets when we run in a GitHub's way. But is it possible, do you have any plans of extending the template language to support to generate anything else than a Kubernetes secret? You mean like instead of storing it in a secret, storing it, for instance, in a config map? Not a config map, but some customer resources. Some customer resources. Don't have that heard before. So there are no plans yet, but hearing us the first time soon. Coming with a GitHub issue then. Yeah, sure, that would be great. I have a question. Considering that your external provider is your single source of truth for the secrets, does this mean that when you update a secret path or put the secret in a different namespace, does it synchronize back? Do you watch over it? Or maybe rephrase it that is it possible that I specify that all my secrets, bunch of secrets, are under this namespace? And then you automatically create those? Or do I have to single one by one map them as my resources? Yeah, so you can do an explicit mapping, and you must know the name beforehand. But there's also the way to use this find semantics. And then you can filter by a regular expression, like the name, or you can use tags if it's supported at the provider. And that way you don't need to know beforehand what kind of secrets you have. You can just pull all of them, stuff them into a secret resource, and you're done. All right, thanks. Here's a question. If I update the secret, who will update my port, actually? There's another project called Staccato Reloader, which does pretty much that. I know it was figuring if it's something else or not. Yeah, no, we actually had a discussion in the community, and there's an open issue regarding that. Want or should implement that pretty much the very same functionality? And I think if there's already a solution, but that's just my opinion if there's already a solution, just use that instead of implementing the very same thing. But that's just my opinion if other people would like to have that thing in the project. And sure, like it. Also GitOps tool can do that for you as well. Well, if there's an annotation with a checksum. Yeah. My question is related also about the kind of resources you suggest not to use only secret. But comparing to, for example, CSI secret store, which at the top of that allows just to mount secret directly to the port, do you plan to have anything similar just compete on that? We are in close contact with the secret sources IDRABER people. And I would say that we do not like to interfere with their set of features. It just doesn't make sense to build the same thing out. So if you want to use CSI, then I guess that project would be better. We were also thinking about providing some sort of mechanism like compatibility mechanism. So they are able to use our features and we may be able to use their features. But that's pretty, pretty tough to do. Well, because basically there are two sides. One is where you fetching stuff and how you fetching secrets and how you deliver. Exactly. They do differ in how they fetching stuff. That's the problem. So you cannot just interchange. Yeah. Yeah. OK. Another question. So one last question. Yeah, regarding the other question, like maybe if you had an image post secret, you might want to generate that. But anyway.