 So we have Swaroop and Chirag. Swaroop is a software engineer at Red Hat, who currently works on developing scalable programs for Red Hat OpenShift Container Platform. Chirag is a software engineer at Red Hat, working with OpenShift Customer Focus Engineering team on various things related to the OpenShift Container Platform. Welcome to you guys. Over to you. Hello, everyone. Good afternoon. Thanks for joining here today for this talk. I hope you guys have a great lunch. I have. I like the food. So my name is Chirag Kyaal. And join me with my friend and colleagues, Swaroop Ghosh. We both are working at Red Hat at Software Engineers. We basically work in OpenShift Engineering team. So today, we are going to discuss about encrypting secrets in Kubernetes cluster using KMS, OK? Before I go too deep into the discussion, let me ask you a couple of questions. So how many of you guys are using Kubernetes secret? I hope most of you, right? OK. How many of you are using in a production cluster? A couple of you. Are you managing your secrets securely in the production cluster? Or ever come across questions like, how can I securely manage my secrets? Whether my secrets are secure or not? If you ever come across these questions, I hope this talk can help you understand bits and pieces of that. So without further ado, let's get started. So first, I will start with some basics of Kubernetes secret. So anything that is sensitive, be it credential, your username, password, token, API keys. So if you wanted to use and store such kind of sensitive data inside your Kubernetes cluster, then Kubernetes has this awesome mechanism of Kubernetes secrets where you do not have to include any of your sensitive data directly inside your container images. Rather, you can reference these secrets during runtime inside your container, be it as an environment variable or you can use any file system mount and you can mount your Kubernetes secrets directly to your containers at runtime. So this is the advantage of using Kubernetes secret. So whenever you try to create any secret, what happens is that your API server gets a request. It reads that request and store your secret as a key value pair inside its database, that is HCD. So inside HCD, all your secrets are getting stored as a key value pair. So with this setup, I hope everyone is happy here. Like your API server is happy, it has stored your secrets inside HCD, your container is happy because it is getting all the secrets from there, you are also happy. Do you feel like there is some problem with this setup? I feel there it is. There is some security problem with this default setup because by default, Kubernetes store all the secrets as unencrypted format inside the HCD database. So if you try to visualize what is there inside your HCD database using some HCD-CTL tool, you will get to know that all the secrets inside it are getting stored as a plain test format. They are not encrypted. That is the default implementation that Kubernetes has. So if any attacker, by any chance, get the access of your HCD database, they can basically gain all the sensitive data that is present inside it. And they can happily create a backup of your HCD database and walk away with all the sensitive data that is present inside that. So all your sensitive data, all your username, password, be it anything, get compromised or can get leaked with the default implementation. So that is the problem of the default implementation of Kubernetes, that secrets are not encrypted by default. So what is the solution for this problem? I hope you might have guessed it right, looking at the picture at the right hand side that you need to encrypt your secrets at rest inside the HCD database, OK? Now, if you search Kubernetes documentation that how can I encrypt my secrets, you will land this page. And you will see Kubernetes is giving different types of encryption providers, like AES, CVC, AES, GCM, KMS, V1, V2. There are quite a few. These encryption providers are basically classified into two types. One is called as local encryption provider, and another one is called as remote or external encryption provider. I will go a bit deep into these topics. First, let's discuss about local encryption provider. So in order to use any encryption provider, you have to use something called as encryption config, OK? So this is a configuration YAML that basically sits inside the API server. And it instructs the API server how to encrypt any secret. How it does is that you need to add this encryption configuration YAML inside the HCD database. And you have to write all the configurations, like which provider you wanted to use. So for local encryption, you can choose between AES, GCM, AES, CVC, secret box. Doesn't matter which one you are picking. So along with that provider name, you have to give keys. So these keys are basically used by the API server to encrypt all the secrets. So now what happens when you do kubectl create secret? Your API server reads these encrypts and configs in YAML. It uses that encryption key that is basically present inside that configuration YAML and stores your secret as encrypted format inside the HCD database, OK? Now during decryption or when you do kubectl get secret, the reverse process happened. Your API server get the encrypted secret from the HCD database, uses the same encryption key that is present in the configuration YAML and decrypts the secret and give it back to you, OK? So this is the mechanism of local encryption provider. Still now with me? Any question with local encryption provider? All right. So local encryption provider, it mitigates one major concern. That is the HCD compromise problem because all the secrets inside your HCD is now encrypted. So if any attacker, even if they try to get access to your HCD database, they won't get much out of it because all the secrets are encrypted now inside that. But it doesn't mitigate one major problem. That is your host compromise problem. Think about this way. Suppose your attacker tries to gain the access of the host itself, or they get the access of your disk of the node where your API server is running. What will happen is that they will get the encryption key. You see over here the encryption key that is present inside your configuration YAML. So they can easily grab that encryption key, take a backup of your HCD database, and happily decrypt all the secrets. So now your secrets are again compromised. So local encryption provider, it mitigates one major problem. That is HCD compromise problem, but it doesn't mitigate the host compromise problem. So to mitigate both the problems, we have something called as KMS envelope encryption, or using the external encryption provider. First, let's discuss about what is KMS. So KMS stands for key management system or service. So it is an automated system provided by all the major cloud providers to automatically manage your cryptographic keys and its metadata. So all the major cloud providers like GCP has their Cloud KMS, Azure has their Key Vault, then AWS has their AWS KMS. So all the major cloud providers have their own flavor of KMS. This KMS uses something called as envelope encryption. How many of you guys know about envelope encryption? No? OK. So in case of envelope encryption, there are two types of keys. One is called as de-case, that is data encryption key, and another one is called as KEK, that is key encryption key. So in case of envelope encryption, your data are basically getting encrypted using the data encryption key. After encrypting your data, your de-key goes to the remote KMS server. Get that de-key encrypted using the key encryption key, that is KEK. Get a ciphertext back and store that encrypted de-key along with the encrypted data inside your disk. So this is how the flow works during the encryption for your envelope encryption provider. During decryption, the exact reverse process happened. So your encrypted de-key goes to the remote KMS server, get it decrypted from there, and using that decrypted de-key, it basically decrypts all the data. So you see with envelope encryption, the advantage is that your key encryption key never gets stored along with your data. It always present inside some external remote server. So your attacker cannot get the access of KEK because that is backed by all the cloud providers. So that acts as a root of trust. Now let's see how you can use KMS in case of Kubernetes. So in order to use KMS, you need to have something called as KMS plugin. So this basically acts as a bridge between your API server and the external KMS. It's an GRPC server that basically talks to your API server and the KMS. Now what happens when you do kubectl create secret? Your API server sends a signal to the KMS plugin. KMS plugin then generates a de-key. So that de-key basically used to encrypt the data. Once the data is encrypted, KMS plugin talks to the cloud KMS, decrypt the de-key using the KEK that is present inside the KMS service, get a ciphertext back, and writes the encrypted de-key along with the encrypted secret inside the HCD. So you see your encryption key, that is your KEK never writes, never get written inside the HCD database. It always present outside to your cluster. So using your external secrets provider or KMS, we can mitigate both the security problems that I have been talking about. It mitigates the HCD compromise problem, and it also mitigates the host compromise problem. Now Sorup will discuss about the architecture of KMS. Well, Chirag, excellent to you about the different encryption providers that you get with Kubernetes these days. So basically we are at a point where we have an understanding that unencrypted secrets are anyways not a good idea. And if you move one step forward when you move to something like a local encryption provider, you have the risk of getting your host compromised and still you get attacked onto your secrets. So we are here with KMS encryption provider where the keys are stored remotely in some remote location managed by the cloud provider. Or you can also use your, what would I say, like a custom key management solution running on a bare metal kind of a setup within your environment where you can manage the keys securely and keep it as isolated from your actual control plane and your actual cluster. So let's dive deep into what happens when we use Kubernetes KMS. And I'll also walk you through a demo with it, how it looks like. So as we see that there is a lot of encryption and decryption going on. And at a very high level, there's API server, HCD. And in between them, there's a new agent. It's called KMS plugin. The KMS plugin's job is to talk to the KMS provider and ensure that keys are encrypted and decrypted when the API server wants to. That's all the plugin does. So communication essentially happens over a GRPC connection. So typically you would run the plugin as a static pod or as a, like on the host itself, any way you want. Basically, what QAPI server needs from the control plane is a unique socket where it can communicate with the plugin and make those encryption decryption calls. And if you see this representation where the secret is getting stored in HCD, it is getting stored in a format where it's encrypted and it's safe for anyone to whoever has access to the disk or those kind of things. They still cannot get the actual secret itself. So moving on, we have two implementations of KMS in Kubernetes as of today. One is the V1 implementation, which has existed for quite some time. And there is V2 as well, which we will discuss later on. So V1 has been there for quite some time. I think, yeah, V1.10 or V1.30, I don't remember. Yeah. And the new V2 is there from V1.27 onwards. It's available as a beta. Well, I walk you through what happens when you create a secret using V1. So there's a client that tries to talk to API server to give it the contents of the secret. Usually, it's base64 encoded. Even if you use the new string data field, if it's not base64 encoded, the API server receives it if required, decodes it from the normal base64 to the normal string. And what it does is it randomly generates a number, which we call the dek, or the data encryption key. This dek will be used to encrypt the secret. So this dek will be ciphered upon the actual secret itself. And you will get an encrypted secret, which will be stored on the disk. So on the disk, two things are stored. One is the encrypted secret itself. And this dek is also sent to that KMS plugin, which sends encrypted dek back. And once the encrypted dek is available, the encrypted dek, along with the encrypted secret, is stored on your HCD. Another good part is that your secret and its actual contents doesn't leave the node. So it doesn't go to KMS. This is why envelope encryption is helpful. Apart from that, there is also another interesting bit, which is, if you see over here, this is the cache size. This cache size, you can configure from your end when you are configuring the encryption provider. What it will basically do is, whenever making these encryptions happen, they will also cache the dek's encrypted, like the plain text dek, as well as the encrypted dek. Keep them mapping. It's essentially an LRU cache. What it helps to do is avoid, or I would say decrease the number of KMS calls later on when we use it for decryption. It's up to you if you want a very secure setup where you do not want anything to be running on the memory itself. You can use something like cache 0. But that would add a lot of network overhead between KMS plug-in and your API server. And we could just make a note that for this particular KMS v1, there's a 1 dek for each secret. So when we decrypt it, that CD data is red. And basically, you get the encrypted dek and the encrypted secret. So when you get the encrypted dek, you send it back to the KMS plug-in to get the plain text dek. Use it to decrypt the encrypted data and give it to the client. If it is available in the cache, you save yourself from, I mean, Kube API server saves itself from the extra KMS plug-in call it had to made. Depends upon what is the cache size and how the cache works. So this is how it works at a high level. I'll also walk you through v2. So things change in v2 a bit. It's basically tries to reduce the number of calls to KMS that's happening, and also the total number of deks that get generated. So before in KMS, v1, what used to happen was we used to have that many number of deks as the number of secrets. But now, usually, when the API server starts up, it creates one dek and caches it. So it doesn't create one dek per secret anymore. But it only does this when it is required. By when it is required, I mean that this happens on API server startup, and if the KMS key is rotated. So if the KMS key is rotated, this is how it will go on. If not, then if it doesn't get rotated, then the dek will remain the same. Internally, the cache is a time-limited cache. So after a point of time, it does talk to KMS again. But shouldn't be a problem since it's just one key or maybe two, three keys. So essentially, you see now the flow is simpler. When you create a secret, you need not make a KMS plug-in call. You have the dek available in the cache itself, that one particular dek. Use it to encrypt the secret and store it at CDDB. That's it. Another difference is if you check out the representation on the CD data itself, previously it used to append the dek and encrypted secret. Now it uses a protocol buffer. So you can also store other information like annotations and any things that the KMS plug-in provider provides with you. But you shouldn't be concerned about these from a cluster administrator perspective until and unless you are developing a KMS plug-in yourself. There's this field, the key ID. This is changed only when the KMS master key is rotated. So Kube API server gets a signal that the master key for the KMS has changed. And now it is time for me to re-encrypt everything, basically. Same thing when we have the secret being read from the KMS V2. What happens is it takes the data from the cache. It already has the dek as plain text version in the cache. So it doesn't need to make a plug-in call. It will just directly use it and decrypt it from at CD and give it back to the user. If there's a key ID change, it will perform re-encryption. OK, so we'll move to the more interesting part to the demo. I hope you can see my screen. So what I did is I created a cluster, not a cluster, a single node Kubernetes VM, a VM where single node Kubernetes is learning using QBadium. Nothing special. If you check, so I'll just show you what you need to actually change to enable this KMS thing which I told you. So Kubernetes would store its static pod manifests in this URL, in this directory, etc-cubanities-manifests. You can check the QBAPI server pod YAML from here, if you want. Yes, pretty much this one. So this doesn't have a flag encryption provider config yet. I will give one of that to this. And I'll also add another static pod which will run that KMS plugin. So I already have these things stored. Yeah, so this is my updated QBAPI server static pod. What it has add on is this last line over here, this encryption provider config. I'll also show you what's in this config. Yeah, this is what is basically there in the config, the path to the Unix socket and the cache size. This is KMS v1. So this is where it should be running. And for this to run, I will spin up another static pod. So I'll quickly copy these. And if I restart Qubelet, my Kubernetes should now start using KMS. Yeah, not an issue. Now what I will try to do is, yeah, we have them running. OK. Yeah, this is using the older, just give me a minute. This is the older Qube config running at my end, which will not work. I think it should work now. Yeah, so this is, so what I will show is create a secret. Yeah, I'm trying to create this secret generic secret DB user pass just some dummy values, admin and password whatever. So this went through KMS internally. I'll show you that. So if you see, this is not the plain text secret. This is some encrypted. And if I do a hex dump over it, it will give a better representation of it, basically. But yeah, this is what is contained inside the secret. I'll also show you another secret that was there from before, which is not encrypted and how it looks like. This is this bootstrap. I think I messed up with the, OK. Oh yeah, this should be Qube system. Yeah, you see this secret is there as plain text from whatever was there in the API server. So this is the difference, essentially, from KMS and without KMS. So this is how it works if you do manage your cluster from your end and you have access to the control plane. What happens is there can be scenarios where your control plane is managed by someone else for something like managed services. You can use GKE AKS that has off the fly KMS encryption. That is one alternative. Apart from that, if you have some tooling on top of it which manages the control plane for you, you would typically need to change the Qube API server manifest through code, whatever tooling you use, and add a container for that KMS plugin and change that encryption provider config from inside. And it should work. It should look something similar to this basis of GCP or AWS wherever you are running. So this is the basic change that you need to do to your Qube API server if you want to do it in a managed kind of an environment where there's a control plane manager and those kind of things. So right now, this feature is not there within OpenShift, but we are actively investigating how we bring this in. But you can definitely use it on a regular Kubernetes cluster on AKS, on EKS, on GKE. And if you are running something on a bare metal kind of a deployment, what you could do is you could directly use another KMS provider. That's called TruSO, that's not a KMS provider, my bad. That's a KMS plugin that connects to Vault. So if you're running Vault internally in your bare metal or in your network, you can essentially use TruSO KMS plugin that will connect to Vault very well and would run very well for your bare metal clusters. We discussed a lot about securing and encrypting Kubernetes secrets using KMS. But what if you are trying to do something differently and you still want to use security, or just to discuss on the other approaches that you get. So HashiCore Vault has a Vault sidecar injector that injects the secrets into the containers, into the pods directly through a sidecar container. There's no involvement of Kubernetes as such, the Kubernetes API server as such. But it's secure because it directly talks to Vault and bypasses QAPI server. External secrets operator, this one is more like a synchronization point between an external system or an external API where your secrets are stored and then it injects into Kubernetes secrets. Sealed secrets is another thing. It's similar to KMS, but it works a bit differently. That is, it actually changes the original secret itself. And there's a controller running that decrypts it at runtime. So even if you run on a GitOps environment or something, if you are using a public Git repo, if you are using sealed secrets, you can keep your secrets even on GitHub. It seals and keeps your encryptions running that way. That's it. So yeah, thanks pretty much. That's what I had. We had. And if you want to contact us, we have our handles on the screen. And I would give you the opportunity to question back whatever you want or relating to whatever we discussed today. Any questions? Peter presentation. So there is one question that I have. We have a similar scenario. And what we did is we actually used the external secrets operator to resolve that issue. So I don't understand where or what kind of use cases this KMS-based stuff will be useful. And because the external secrets also, it connects to the cloud provider secret itself and then get it synchronized and store it in a secure way. So external secret providers do not encrypt your secret. So it just creates a synchronization between your storage, your remote storage, and reads your secret from there and creates a Kubernetes secret itself inside your cluster. So it's creating a secret only. And it's creating a synchronization. But it is not encrypting any of the secret. So even if you are using the external secrets provider, your secrets are not encrypted. They are just synchronizing between the external remote server and your Kubernetes cluster. So if you do the hextam of your HCE database, you will see the secrets are still plain text. They are not encrypted. OK. Thanks. HTTPS mode only, so it will be automatically secure. So you are saying, while doing the synchronization, it won't be a kind of a plain text. But in HTTPS mode, if I'm getting from the external world, it will automatically secure. But at rest, inside the HCE database, it is not. During transition, it can be encrypted. I'm talking about at rest what is happening inside the cluster. If external secrets operator will create a secret itself, and that secret will be stored inside the HCE database only. The problem is, inside the HCE database, it is plain text. Inside HCE, it is not encrypted. So external secrets provider won't solve the encryption and decryption problem. This entire presentation is for rest inside HCE. The data that is present inside the HCE that is not encrypted. Entire presentation is talking about at rest problem. Also another thing, if that is not the solution you're looking for, you can also consider encrypting your nodes that does some point of the exercise that we are doing. But yeah, it doesn't touch anything at cube level. This is another thing. You can encrypt your nodes disk. Basically, it offers similar kind of encryption. But it doesn't always work. If you have multi-tenant workloads or some kind of a thing, then it doesn't work. For a multi-tenant workload, each tenant can have their own KMS plugins and those kind of things. So if you are using open-sift cluster, we are sorry. It's not launched yet. Internally, we are working heavily to get it G8 and beta version. So just bear some time to get it released in open-sift. This is the first kind of encryption. So it is a kind of pseudo-random number generator which you have. And on top of the KMS is there. So what if the cache is invalidated? So how do we describe the actual secret out of the KMS? So from KMS, so that the key is there, which is in the node order cluster. So that is used as a kind of encryption on the secret. And then the secret is shared to the KMS. No, secrets are not shared to the KMS. The DK is shared to the KMS. The DK goes to the KMS, get it encrypted, and stored with the encrypted secret inside the SA database. Your secret never goes to the remote KMS server. Only the DK goes to the remote KMS server, get it encrypted, and stores along with your encrypted secret. If the cache is invalidated, so how the DK, the encrypted the DK is decrypted? So if your cache is, so you can configure your cache size. So if your cache is zero, so your API server will then talk to the KMS plug-in. So KMS plug-in won't find any cache. It will talk to your remote KMS and pick the decrypted DK from there. So your cache is null, right? Your cache is nothing. So the KMS plug-in have to go to the KMS service again and again. There will be latency issue that we talked about for KMS v1. And the KMS v2 solves that problem. Because basically the cache will stay in the memory. It will not leave the node again. Yeah. So that is the main point. Your cache is the lifetime of the API server process. Not your disk. If we have more questions, maybe we could take it offline. Thanks, everyone. Thank you, guys. In the interest of time, we'll take the Q&A offstage. So I request money to give away the token of appreciation to Chetak and Swarov. Please go ahead.