 All right, let's begin. Welcome, everyone. Thanks for coming. This is Extending Kubernetes with Storage Transformers. My name is Andrew Lutinov, and I'm a software engineer on the Google Kubernetes Engine Security Team. So today, I'm going to explain to you briefly what transformers are, why they exist in Kubernetes, and then most of the session will span actually walking through the guide of how to implement one. So starting from a base transformer built into QBAPI server, then two envelope transformers that help us with key rotation and management, and then into KMS plugins. And lastly, I want to talk a little bit about how we secure KMS plugins that we end up at the end with. So to understand what storage transformers are, I want to look at QBAPI servers consisting of two layers. You have the RPC layer, where your requests come in and out, handles authentication, authorization, parsing, validation, all those things. And then there's the storage layer, which is essentially a QBAPI server talking to HCD and doing any serialization and deserialization. At the RPC layer, Kubernetes provides you with extensibility in the form of web hooks. So you may have heard about admission web hooks. The idea is pretty simple. You just provide some custom piece of logic that is injected in the path of all of the objects to traverse the API. The same thing at the storage layer is provided by transformers. So KMS plugin is an example of a transformer. But the idea is the same. You inject some custom logic that sits in between your objects as it can come in and out of HCD. By convention, transformers look at all the data as opaque byte blobs, while web hooks look at them as sexual parsed objects. So they will look at individual fields and maybe change their behavior, maybe mutate the fields based on some logic. Obviously, that's just a convention. Transformers are free to parse all the objects as they come in and out. But if you find yourself needing to do that, web hooks are probably a better fit for your use case. So why? Why I talk about transformers at all at KeepCon? First of all, transformers are somewhat under-documented. It's kind of difficult to figure out how the code base is structured, how to implement one. And my colleague, Aleksir Nyokovsky, has spent a lot of time working with transformers for GKE. And there's some valuable lessons to be learned from his work. And finally, with this all new shared knowledge, I want to encourage some new contributions, maybe to just improve the code base, maybe to redesign the entire feature, maybe to implement some new kinds of transformers for use case that are not obvious right now. The reason transformers were created in the first place in Kubernetes is because by default, QBPI server does not encrypt any of your secrets at rest. So that means all of your data that you submit to your API is stored in your HCD data file, encoded in some binary format, but essentially plain text. And transformers exist to solve that for at least the secrets by encrypting them before storing to HCD. The reason it's a problem is if you look at someone, for example, creating a new secret in your API, they send it to QBPI server, then QBPI server encodes that in some format but essentially stores it in plain text. So if an attacker gets a hold of your HCD data, they have access to all your secrets, anything that your workloads running in your cluster have access to. You might think, well, you spend a lot of time hardening your master VMs, your databases, your API, you have all the RBAC rules, everything is bulletproof, but consider this case for offline attacks. We have a cluster here where we have a credential stored in our HCD that is just a credential to authenticate to some remote database that stores a bunch of sensitive data. Our attacker comes along, tries to mock around with our VM or API and nothing works. It's indeed like a really hardened setup, a very well protected cluster. But we also configured our cluster to do a daily offline backup of HCD data for disaster recovery. And the backup server where this backup ends up happens to be not as well hardened as your master VM. So if an attacker gets access to that, they can use the credential to access the data store and basically it's game over. And notice that the attacker didn't have to compromise your API, your cluster VMs, your database, nothing. They still got the access to sensitive data just because one of these backup servers happened to be not well hardened. So the point here is that there's many vectors to accessing and exfiltrating HCD data that may not be obvious right away. So you might think that, okay, someone got the dump of this huge database of HCD. There's a bunch of objects and it's really, really hard probably to parse everything out, figure out which objects are secrets or not, figure out what the format is that Kubernetes stores the data in. But let me show you this. So I'm gonna run a little helper that will do the typing for me, but it's all running live. So I have this HCD data directory dumped from a running cluster. After a quick search in the documentation, I know that members snapdb is where the data is actually stored. So I'm gonna run the strings utility on this file. And strings utility is essentially available on any Linux distribution pretty much built in and it will print out any data in the file provided that's textual. So anything that's binary is discarded. Anything that looks like a piece of text will print out. So here, like if you scroll around, you can see there's some object URL looking things. There are some full objects in JSON. So that looks promising. I'm just gonna grab this output for a string secret. And immediately find one finding. It's called davdbsecret01, okay? So I'm gonna pass the dash c flag to grab, which will just show me the lines around where this match was found. And here at the end, you can see that we have our secret name, namespace name and the contents of the secret in plain text. So I'm trying to point out is that this is all the tooling I needed to extract secrets in plain text out of my AC data dump. That's applicable to any standard Kubernetes installation that doesn't do encryption at rest. All right, so hopefully I kinda convinced you that encrypting secrets at rest is a useful thing to do. Now let's jump into the main section and how do you actually implement one? How do you create it if you have a new idea for a transformer? So this URL here, I have a fork of Kubernetes where I built a transformer that's following exactly the same steps I'm gonna show you right now. The steps I'm gonna be showing in the slides are more abstract. I'll just use some code snippets for demonstration purposes. But on this fork I implemented a transformer that does SM4 encryption at rest alongside all the other available transformers. And if you're not familiar, SM4 is an encryption algorithm similar to AES that's been designed and approved by the Chinese scientific community. Before we jump into the steps, at the bottom right of each slide will be the exact commit on that fork that implements the step. And I'll also try, these are actual links to the directories and files where the changes I'm gonna talk about are need to be made. So first step, obviously we need to implement the actual core of our transformer, what it does. And that's done by implementing this transformer interface. This is a pretty simple interface, just two methods. You get some data before or right after is being read from, oops, sorry. But after being read from its CD and you have to transform it back and same thing in the opposite direction. So that's specific to what your transformer does but should be pretty straightforward. Next step, transformers don't really exist in isolation. Usually they need some input. So for encryption transformers, there'll be encryption keys, there may be some other data you might need to provide. All this configuration is passed to QBAPI server through a YAML file at startup. This file contains an encryption configuration object. And essentially it lists the types of resources that the transformer applies to and the specific transformers. The providers is just to send in for transformers here. And here I sort of sketched out what I want my transformer to look like. It has a key and a field of some other type as input. So now I convert that last chunk into a ghost struct into which I'm gonna parse out the YAML. It should basically mirror the exact YAML object that you expect to get. And you may notice there are two links to the files here, and yes, you need to implement the exact same struct in two places. You might also notice that we're using JSON tags in our struct fields while we are parsing out YAML. But we don't have time to make our code base make sense. We just have transformers to write, so let's go next. Step three, you have your configuration. Now you need to register it as a possible transformer in the encryption configuration YAML I showed before. And to do that, you just add it to an existing struct at the end. So here I just added as a last fields in the same two files. It's pretty straightforward. Next, I need to explain what prefix transformers are. Prefix transformers exist for the purpose of not having any conflicts between multiple transformers. So when a prefix transformer writes some data to ICD, it prepands the static prefix that's highlighted here to the data. And then when it reads it back, it actually checks the prefix, strips it. But if the prefix does not match, you know that there's some misconfiguration, there's something wrong, and instead of returning corrupt data, it will throw in error. So here, yeah, I just grabbed another ICD data file and the prefix looks roughly like this. You don't need to implement any of that yourself. Prefix transformer is already implemented. You just need to instantiate it and give it a prefix and your actual transformer. So you register your prefix in this const block and the linked file. The const block is at the top and lists prefixes for all the other transformers that already exist in Kubernetes. It's pretty simple. So last step, we have our configuration structs with all the inputs for our transformer. We have our transformer implementation. Now we just need to instantiate that implementation from the config in the linked file. You create a new function that takes your configuration as input, takes your prefix as input. That would be a good place to validate the configuration. So if you have any keys you need to parse, if you have any other configs that need to be validated, that's a place to do it. Then you instantiate your transformer implementation from this config. And right before returning, you wrap it with the prefix transformer so it handles all this prefix thing for you. So great. Let me show you how that looks in practice. So here on this other tab, I'm gonna start my local cluster that is built. So it just uses a modified QBAPI server binary that was built from my fork. Here I have my encryption configuration that I passed to that cluster. You can see this as some for transformer config field here. And I just give it a single key, the encryption key. So start the cluster in a separate tab and gonna make sure that it's actually running. So it's running on my local host live. I'm gonna create a secret. I'm gonna give it two fields, username and password. And I'm gonna name it DevDB secret 01. So I'm just gonna, this is how it looks like in YAML when you print it out as a client. You will notice that these fields don't look like the actual fields I set, but these are just base 64 encoded. So this base 64 decoded is just password zero. And this is working as intended because we're not doing encryption at the client layer. So not at the API layer, we're doing it as storage layer. So clients should have no difference in experience. Now for comparison, from the previous demo, this is the command we ran and this is the output we got. So we got our username and password in plain text here. Now the same command ran against this new cluster. We see that we still find the secret identifier, but then we have our prefix that I used for my transformer and a bunch of binary data that's not printable and that's the encrypted data. So okay, great, that works, but we have two problems. The first problem is that we provide this encryption key at startup via the YAML file. So that means if you want to rotate the key, you need to restart QBAPI server, you need to update this encryption configuration. It's a lot of manual work and it's a little bit of API downtime, so that's not great. But the bigger problem is that our key is stored in plain text and disk. So the encryption configuration YAML is passed to QBAPI server at startup as a file path and odds are that that file is stored on exactly the same disk where you store your HCD data. So if the attacker compromises the entire disk, not just the directory, then they have all the keys they need to decrypt all the secrets. To help us with those two problems, we're gonna use envelope encryption and envelope transformers. So it's a bit of a tricky concept. I'm gonna walk it step by step. So bear with me. The idea behind envelope transformation is that you use two keys instead of one to encrypt every piece of data. So you have your data encryption key or DEC which is used to encrypt the actual piece of data and then you have your key encryption key or CAC which is used to encrypt the DEC. When you actually store the data, you encrypt it with the DEC, you encrypt the DEC with the CAC, you concatenate those two parts and that's called the envelope. That's what you actually put into HCD. Well, key encryption key lives outside of the master VM completely in a separate machine, in separate infrastructure. Key management services or KMSs are a good fit for handling CACs. That's their whole idea is that they store your plain text keys in some secure infrastructure and only expose the operations that those keys can perform like encrypt, decrypt sign and so on. So the idea here is that we use KMS or something similar to host the CACs and never let them land in plain text on our master VM. So master only has these like encrypted blobs in the envelope. So that solves our second problem of keys being in plain text. The first one of key rotation and management. I'm just gonna walk by example, so it's a little clearer. So we have our KMS provider with the CAC and most KMS providers allow you to automate key rotation. So here we have configured to create a new version of the CAC every month. When you write your first secret, it gets its own DEC, DEC one and then that DEC one is encrypted with CAC one. If you create another secret, you get a separate dedicated DEC for every single secret and then that DEC is encrypted with CAC one again. Now sometime passes, our automation kicks in, it creates a second version of the CAC, some more time passes, it gets another version. Now at this point, if you create a new secret, it will get its own DEC as before, DEC three, but this DEC will be encrypted with the latest version of the CAC. So CAC v3, if you update an existing secret, it will notice that the CAC used was an old one and it will create a brand new DEC for it and then encrypt that DEC with the latest CAC, CAC v3. So that's how you can gradually roll and update all of your secrets to use the latest version of the CAC without having to do any manual rotation and your restarts. You can also manually trigger a secret update to sort of indirectly update to the latest CAC so you can clean up some older versions. So that sounds like it solves our problem, but it's pretty complicated. There's a lot of keys spying around, a lot of encoding and creating envelopes and whatnot. Well, luckily, you don't need to do any of that yourself. That's already implemented for the most part in Kubernetes. There's this envelope transformer type that takes two pluggable pieces of functionality. It has the envelope service and the base transformer funk. So the base transformer funk is what's actually used to encrypt the data with the DEC. Envelope transformer would create the DEC for you and then it will call this function with the DEC so you can return the initialized transformer from it. So you can reuse the same transformer from step one. The other part is the envelope service and that's actually an interface. It has just two methods and it's used to encrypt the DEC with the CAC. So the idea is that service is some sort of remote service that doesn't expose the CAC to you in plain text. But, so we could potentially implement the service interface for a lot of possible KMS providers like Vault or Amazon Key Service, but following the Kubernetes philosophy, we don't actually want to put any third party, any vendor code into the upstream Kubernetes repository. Instead, we want to provide a standardized extension point and sort of hide this third party code outside of the tree somewhere. For that, there exists KMS plugins. The idea behind KMS plugins is that they are lightweight proxies, local proxies on the master that expose a standardized interface that QBAPI server knows how to talk to and then they proxy those requests to the KMS provider using whatever protocol that provider happens to be using. They already built into Kubernetes. You can specify them in the encryption configuration. You just have to give it a name and the UNIX domain socket to talk to. And UNIX domain socket is an actual explicit limitation. You cannot have a remote KMS plugin. The reason for that is this basically eliminates a problem of needing to authenticate from QBAPI server to the KMS plugin. So, and usually there's no reason to have a remote KMS plugin anyway. This UNIX endpoint should serve a GRPC service that implements this definition. It's basically the same as the interface we looked at a few slides back but with an extra version method for just some metadata. Just to visualize it all together how the entire flow works with the KMS plugin. Say your user, your operator submits a new secret, pushes it to the API. QBAPI server then generates a new deck. It encrypts the data with the deck. Then it calls the encrypt RPC towards the KMS plugin with the plain text deck that gets proxied over to the KMS provider using that specific API. KMS provider encrypts the deck with the kek, returns back the encrypted blob, that gets returned back to the QBAPI server. QBAPI server appends the encrypted data in an encrypted deck and stores that envelope in HCD. So, nothing unencrypted ever touches the disk and that can exist in the memory of QBAPI server. Okay, like two other steps that are optional by default, KMS plugin will use AES encryption for decks. So, if I'm encrypting data for the deck, if you want something else and in this case I wanted my deck to be an SM4 key, you need to do a little bit extra plumbing. So, here I added a deck type field to the existing KMS configuration and that's the YAML configuration blob that's being parsed. And then I added a switch to the function that initializes KMS plugin where based on the type I will replace the function that initializes the transformer so I can reuse the same transformer from the first step. Lastly, most likely you don't need to implement your own KMS plugin. There's plenty of officially supported plugins from a bunch of providers out there. Here's just a small selection of them. So, if you search around, you can probably find one and just use it. But if not, it's pretty easy to implement. Okay, now you should roughly understand how the whole code-based structure around transformers how to implement one. The last thing I want to mention is a FAT model for KMS plugins or basically how to help secure the KMS plugin. So the problem here is that KMS plugin talks to a remote KMS service. So it needs to authenticate somehow. It needs to provide some credential. In this case, let's say it uses a static token that's passed with a request for authentication. We also have our cluster configured to do a full disk backup of the master for disaster recovery instead of just a data database. So it might be easy to overlook and just store the token on disk in plain text for KMS plugin to read and use. But if our attacker compromises the offline disk backup, they essentially own everything they need to decrypt the data. And as long as they can figure out this whole envelope and decryption dense, they can get access to your secrets. So our goal must be to not ever let the plain text token touch the disk. And to do that, one approach I want to propose is using trusted platform modules or TPMs. If you're not familiar with TPMs, they are little crypto co-processors. They're usually built into all of the server motherboards as a non-board chip. There are also virtual versions of them so you can create a virtual TPM for every VM running on a server. And the main idea behind them is that they have a little region of protected memory. And that memory is completely isolated from the host. So if you have full root privileges on the machine or if you have compromised the kernel, you can still not read that memory. And you can only interact with it indirectly through commands that manipulate it, but you can get in plain text. And also TPMs are bound to the specific host they're on so you can not just like move it over to a different server or copy it over. You have to be physically on the server where this TPM exists to access anything that's inside of it. Okay, so first idea might be to just put the token in the TPM because it's protected memory, right? But we actually need to read the token in plain text in order to be able to pass it to the KMS provider. So we cannot put it directly in a TPM but what we can do is we can use a encryption key stored in the TPM and encrypt the token with it before storing it on disk. And this is called ceiling. And basically in order to unseal the token, you need to be physically on the master VM having access to the actual TPM to get access to this key and decrypted. You might think, why don't we just like skip this whole thing and use TPM as a KMS? The problem is that TPMs are really low-powered devices. They have very limited memory. So you can only start a handful of keys there and at least physical TPMs will be really slow in encryption and decryption. So they'll really slow down your API. So they're not a good fit for like in being something in request path. So yeah, having this token sealed should help us protect against compromises of offline backups. In summary, transformers are a way for you to add some pluggable custom logic in the path of data that's being stored into HCD. And there's multiple layers of transformers. They're sort of, we're built historically. There's the standard transformer that just does the thing to data. Then there's envelope transformers that use multiple keys to encrypt the data and help from exaltation of the root key. And then there are KMS plugins that are just a type of envelope transformer that talk to a KMS provider for hosting keks. And finally, TPMs are one possible option for protecting the credentials that KMS plugins might use. Now I want to encourage anyone to strongly consider encrypting your secrets at rest if you're not doing it already. Hopefully there's enough evidence that this is a problem worth addressing. And also, I encourage the audience with the new information about transformers to actually go out and contribute to the transformer code base, maybe propose design changes, maybe implement new types of transformers that don't only do secrets encryption, and maybe implement a KMS plugin for a provider that doesn't have one already. Finally, here's a few links to the previous talks and the KMS plugins and TPMs from previous KubeCons that might be interesting if you actually want to dive into this deeper and implement your own transformers. Right, thank you very much. And if there's any questions, I believe we have a few minutes. There's a microphone behind you. I'm wondering why this kind of functionality is not implemented in GCD. And where? It is not implemented in GCD. GCD? The boundary you have. Sorry? The boundary you have is between Kubernetes and GCD, right? GCD. Oh, GCD? Yeah, yeah. Why is it not implemented on GCD layer? You mean? Yeah, on GCD instead. On the lower layer. I'm sorry. The storage layer, I mean. Oh, why does GCD does not do that by itself? Yeah, yeah. So there's also something called like file system layer encryption or dis-encryption. And that usually is done by most cloud providers already. The problem, so those two things are not achieving the same goals necessarily. They are supplemental to both are needed. You, the thing that full dis-encryption provides you is protection against someone physically running and like stealing your hard drive. Like someone running into the data center and grabbing the hard drive and mounting and reading data. As long as that hard drive can be put online in a data center and decrypted at runtime, which usually happens with backups, you don't even see that the disk was encrypted in the first place. You usually see the data that's like at a higher level that was already decrypted and provided for you. So full dis-encryption is a supplemental feature. Now, HCD could in theory implement encryption of its own. Just say like everything that's written to HCD is being encrypted before it touches the disk. That could be done. I'm not been in touch with HCD folks about it, but I assume it has exactly the same problems is that you have to specify or provide some key coming from somewhere to do in the encryption. Usually the problem with the encryption at trust is not the actual fact of encryption, but it's the key management that's associated with it. And theoretically HCD could implement something similar, but they would have the same exact problems facing them. Another reason is that encrypting everything might affect performance. So if you have encryption that's based on some remote service, not the local key, then all of your objects that are being rather written will incur this penalty of some remote call. And that most likely is a big penalty. All right, no more questions. Thank you so much.