 volume encryption, for real this time. I'm Eric Harney, I'm a Cinder engineer at RedHut and this is what I've been working on lately. I'm going to use this to go over three new features related to encryption in Cinder and use those to kind of give a technical overview of how volume encryption actually works at the volume level in OpenStack with Cinder. Why volume encryption? For those of you that were just in the Barbican talk, this was covered pretty well. A lot of organizations have real requirements around encrypting data to meet federal regulations from various governments, various government standards and most of my work in this area has been around filling in gaps that customers have asked for to meet these standards and to have a more full-fledged volume encryption setup. So Cinder volume encryption is pretty much all about protecting data at rest. It gets a little complex because there's a lot of pieces that are moving just going over the network but the primary point of it is to protect data at rest in the data center and one kind of nice feature we get out of this scheme that we use is to also be able to more safely destroy tenant data when we delete Cinder volumes. People often ask if you can rely on this for various more complex things but the answer is generally this is what it's for, at least so far. So how volume encryption works in Cinder? So a user creates a volume. We create a record in Cinder that says there's a volume that has some ID. It's got some size and gigabytes associated with it. And at creation time the user is specified via the Cinder volume type that they want this volume to be encrypted. So if we want to encrypt a volume, we have to actually put this key somewhere, right? For many years the only way to really do this in Cinder was to use the conf key manager or the fixed key manager which basically means you have to write your volume encryption key in cinder.conf on every Cinder node that accesses volumes and every Nova compute node. And all volumes are encrypted with the same key and don't accidentally lose it or share it is generally the idea. So not great. It worked for some definition of works. But it didn't really give anybody what they actually wanted. So Barbican integration has been completed in Cinder over the last few releases. Barbican is a service that lets you have an API in OpenStack to generate and store secrets such as volume encryption keys. So instead of having these sitting in plain text in config files that have to be handled by the admin, Cinder, whenever it's creating volumes, will talk to Barbican, create a key. And the other nice feature that this gives you with Barbican is that Barbican can actually safely secure these keys in a hardware security manager instead of just storing them on your OpenStack nodes, which is a pretty crucial part of most people's deployments. So as far as actually adding support to Cinder, sorry, tripping over cables here. So as far as actually adding support to Cinder, we had a key manager interface somewhat in Cinder which gave us a way to plug this in. So I think for a while we had the old conf key manager and Barbican options in Cinder. So Cinder was built with this idea in mind that we would eventually add a real key management interface to it, or a real key manager to it via this interface. And this interface is now still basically there, but a lot of it has been taken up into a standard library called Castlin, which is an OpenStack project that kind of establishes a defined interface for create key, get key, delete key operations that you want to call from Cinder and then have handled by backends like Barbican. And the nice thing about this is that it lets you then use different key managers in Cinder without having to change Cinder code. So far, Barbican is, I think, by far the most popular one, but it's good to have this flexibility for the future. I kind of just said all of this stuff already. Castlin is a library. It supports different backends. So we did this in Cinder, and a lot of things went sideways very quickly because it turned out that Cinder, while it had this theory of a plug-able key manager interface, made a bunch of assumptions in how keys work in backups, transferring volumes to other users, retype. Most of this stuff has been fixed up now. Transfer, you can't do. The end result of this now is that instead of having all your volumes encrypted with the same key, volumes have unique encryption keys. You don't have keys stored in plain text sitting in con files. They get stored securely, and it helps meet people's actual security needs. So when you actually create a Cinder volume, we have a create operation coming in from the REST API on Cinder. Cinder then talks to the Barbican service, gets a key, attaches it to the volume via the key ID, and then anytime we need to do things with this volume, we will need to retrieve this key from Barbican again, right? Attached flow is the same thing. So in this case, we're looking at Nova attaching a volume. So Nova gets a request from Cinder, or Nova gets a request to attach a Cinder volume. There's connection info associated with how to connect to that volume that Nova has gotten from Cinder. And one of the pieces of info in that connection info from Cinder is that there's an encryption key ID associated with the volume. So then Nova will do a get operation from Barbican to get the key and actually pass it in to Libvert to attach the volume. It's pretty much the same as any normal attach process. So the basic functionality was written in, I forget, Pike or Queens in Cinder. But we had a problem of a bunch of customers already using the old Cinder conf key manager and had encrypted volumes that were relying on the previous system of having a key stored in a conf file. So we had to figure out how we actually moved them to Barbican. So the way this works is when you start up the Cinder volume service, if you have volumes that are still encrypted with the old Cinder conf key manager key, we will take that key, create a Barbican secret that holds that key data, and then change the volume to consume that key from Barbican with the config file that was used previously. This part happens automatically. When it gets done, there's a message in the log for the admin that says, we moved all of the volumes to the new Barbican key store. But then it's left to the admin to go and actually remove the keys from Cinder.conf once they are sure they're not needed anymore. We don't try to purge them automatically or anything. So anybody that actually goes through this migration process needs to be aware of kind of Cinder does the legwork as far as doing the migration of the keys themselves, but then you have to really clean up the config part or you're still going to have keys that are laying around and not being used anymore in your config files. So another feature that a lot of people have been asking for for a long time was the ability to use Cinder volume encryption with CIFRBD volumes. For a long time, we've supported this only basically on iSCSI and Fiber. And this is due to the fact that these are attached in different ways when you're using an iSCSI blocked volume versus a CIFRBD volume. You don't necessarily have to absorb all of this, but the idea here, starting from the upper left, is you create an iSCSI connection, get a block device for the one on your sand back end. On the compute node hypervisor level, we use LUX to create a new device that is the decrypted version of the volume, which is where we've passed in the key during the attached process. And so you get a device on the host that's now the decrypted, directly usable data for this volume. And so that gets attached to the VM through Libert and QEMU and used like a disk. The tricky part with CIF is that CIF attachments don't actually operate the same way at all because they're not attached at the hypervisor with a device node in OpenStack. So when you attach a CIFRBD volume, we pass in all of the CIF, how to connect to it, credentials, location, and encryption key to Libert itself instead of mounting it on the host and then passing in a device. And this is the part that was missing for years, the ability to actually run the LUX encryption-decryption layer inside of the QEMU process here. So when you're using this feature with CIF, QEMU actually makes the connection to CIF itself with the RBD client to get the RBD volume itself inside of QEMU and then attaches a LUX QEMU block layer on top of it, which then ends up in the end presenting in the VM the same way it did for iSCSI. But all of the connectivity and encryption pieces are running in QEMU instead of on the host. So this works now in fairly recent versions of Libert and QEMU. I forget what they are, but they're documented in the December work for this. Going forward, there's an idea that we may end up using this same mechanism with iSCSI as well in some cases. It's somewhat limited now in that you can connect to iSCSI the same way via QEMU with the NQEMU storage layer, but it doesn't support some features like multi-path, so it's not widely used yet. But the idea is that this is going to end up working the same way at least for deployments where that's a useful thing. So third new feature we've got lately related to this was the ability to handle the intersection of Cinder encrypted volumes and Glance images. So Cinder supports uploading a Cinder volume to a Glance image and creating a Cinder volume from a Glance image. And when you didn't yet have Barbican, this kind of worked mostly by luck because everything had the same encryption key and so you didn't actually have to keep track of anything. You could kind of shovel data to Glance and back down and it would all work out okay. But as we added Barbican support to Cinder, this was no longer the case since all the volumes have different keys. If you upload a volume to Glance, you have to somehow attach a record that tells it what it's encrypted with and how to find that key later so that when you create a volume from the image, you actually can clone the encryption key and Barbican tie the two ends together again and have a volume that's decrypted with the correct key. Through all of this, there's a one-to-one relationship between volumes and keys and images and keys. So the model is that any time you create a volume or you create an image from a volume, you're creating a new key and then any time you delete a volume or delete an image that's been created this way, you delete the key from Barbican because it was the only user of that key. There's a couple other things going on in this area. We also recently added image signing, which is the ability and sender to validate signatures for Glance images. So this is usually an important part of trying to have a secured data path. If you're going to download a Glance image, this helps ensure that it wasn't tampered with, basically, in the Glance storage or over the network or whatever. There's some ongoing discussion right now about adding encryption in Glance itself for images. We're still working out how this is going to overlap with what I just described, but I think they're going to fit together into a pretty good picture here, where you'll eventually be able to actually encrypt an image from Glance instead of the model here is that you can only create an encrypted image by uploading a sender volume to Glance that's sender already encrypted. So pretty much what I described here, sender creating a Glance image from an encrypted sender volume, sender is going to talk to Barbican, get the key that we are using currently for that volume, create it as a new secret in Barbican to maintain this one-to-one relationship between volumes and keys, and then upload the actual volume data to Glance and attach metadata to the image that tells it to point to encryption key B, which we've just created here. And then when you download it, it's kind of the expected inverse, right? So you download the image, get the key from Barbican, and then again to maintain this one-to-one model we'll clone the key into Barbican again for the new volume. So you've got a separate instance of the same key for that volume instead of the one on the image. Sender backups, there's a whole bunch of stuff here, but it basically is exactly the same thing as Glance images. When you create a backup, you have to clone a key, store it somewhere. When you restore a backup, you have to go find that key again, run it for the new volume. So you have one key per volume, one key per image, one key per backup. And if you, you know, you have to have Barbican functioning to restore these backups. So if you're assuming a model where if your whole cloud falls apart, you can go grab your sender data out of sender backups. It's only going to work if Barbican is still functioning too. So it's important to understand that sender backup is really for protecting user volume data in this instance and not, you know, not the cloud itself. So gaps in sender around encryption still. Sender has managed and unmanaged operations, which is the ability to basically, you know, the unmanaged operation tells sender to no longer keep track of a volume. So you leave it out wherever it's stored, but sender just no longer knows about it and manages the inverse of this where you have something that's on your backend that you want to import into sender. We don't support this with encrypted volumes now because if you unmanage a volume and then you weren't really sure to keep track of what encryption key was associated with it and somehow knew how to put it back together when you wanted to import it again, you would end up basically losing data. So this is probably something we could fix at some point. Nobody's asked for it so far. I don't know how important it's going to be priority-wise, but the thing to be aware of since, you know, when you're not using encryption, you can do this. Rekeying volumes. If you noticed when I was talking about volume clone and cloning to and from images, we're currently kind of doing the silly thing where every time we copy data, we copy the encryption key along with the data, which works but is not a terribly security, optimal way to do this. What we really should be doing is every time we do a volume clone or a volume create from image is stamping a new key into the Lux encryption system on that volume so that all your volumes have actually different keys and a leak of one key for one volume doesn't compromise any other volumes. I started looking into this. I think it's something we can reasonably start working on, but it's kind of the next step to making this Barbican support work the way it really should in Cinder. Volume transfer is also something we probably can do and just haven't done yet. Transfer is a Cinder feature where you want to change ownership of a volume from one user to another, which gets more complicated when you're using Barbican to store your encryption keys because then when you change the Cinder volumes owner, you have to also make sure the new owner can actually access the encryption key for the volume, and there's no reason we can't wire this up. We just haven't done this part yet. There's some use cases where people actually want multi-attach read-only support for encrypted volumes. Currently, multi-attach is turned off for any encrypted volumes, mostly just because it's a complex area that nobody's spent time looking at yet, but also on the to-do list. And file-based drivers are the last kind of driver gap. So we filled in CefRBD, a release or two ago. NFS volume driver has the same problem where since we don't have block devices, we have to wire up the LUX encryption to actually work with files in the case of NFS. And the platform-level work that was added to enable that for RBD enables it in Libvert to do it for NFS, but we haven't done the sender and nova work for that part yet. I've got some patches up where I've started working on this, but it's been pretty slow so far. So I think that's about it. Glad to take any questions on pretty much anything related to volume encryption, if anybody has any. Basic questions. First is just how... What is the scope of the Barbican key? Is it tenant level or user level? It is at the user level. So when we create the Barbican keys as part of the volume create process, it's done with the user credentials. So several users inside of the same tenant cannot really use volumes of each other or something like that? Yes, the same way they can with unencrypted volumes, right? So if you have a volume inside of a tenant, anyone who's having a member on that project can take ownership of that volume. Can't want to attach it to his VM. Okay, but I think sender volumes are too, right? I think sender volumes are as well. So I don't think it changes anything from how the permissions model works today unless I'm misunderstanding something. Maybe discuss that later. Another one I have is... So you said that there is a kind of QMU Oliver support how to decrypt or how do you provide encryption for all of these? Right. That also means that if... where do you store that essential information? Is it stored in the domain XML or something like that? It would be... I don't know what actually ends up stored in the domain XML, so when you do the attach call, it gets passed into QMU through Lidvert. I'm not sure if the Lidvert domain XML actually is going to hang on to the key or not. I think it ends up in a Lidvert secret store of some sort, but I'm not too sure. So just the basic question is that if someone is just having access to the compute node or someone is having root on the compute node, is whether he can just do retrieve these key kind of keys? I mean, if you have root on the compute node, there's probably some way you're going to end up seeing these keys, but that would also be true for the iSCSI attach case because they're getting passed into... In the iSCSI case, they get passed into command line operations via temporary files, I think. The key gets written to a file and passed in and then deleted after the call to the LUX system finishes. Yeah, so practically, all we need is a little bit safer than iSCSI. There you have an unencrypted little device so you can retrieve whatever you want. It has the potential to be safer because you end up with all of the security-related stuff confined in the QEMU Levert area instead of all over the whole system with different devices. So there's a lot less area for stuff to go wrong. I wouldn't necessarily say it's automatically more secure, but it definitely could be a better model. Okay, thank you. All right, well, thank you very much.