 Good morning, everyone. Apologize for the late start. And thank you to the tech guys for sorting out the power problems up here. So thanks for coming along to this session. We're going to be talking about Swift Objects encryption. My name is Alistair Coles. I'm with Hewlett-Packard Enterprise. And I'm also a core reviewer on the Swift project. I'm Janie Richling. I work for iBamCloud. I've been working on Swift development for a little over a year. OK, I start my stopwatch. So there's plenty of really good talks on Swift at this week's summit. Some of those talks will be describing to you the features of Swift, what you can do with it. Some of them are from operators that are telling you about their experiences of using Swift. In this session, we're going to be taking a little bit of a look into the future and telling you about some work that's going on in the upstream Swift developer community to bring you a new feature in the future, which is the ability to encrypt Swift object data that's at rest in back-end storage. So I should just be very clear about that at the start. What we're talking about is work that's still in development. It's not yet been released as a feature for Swift. But that said, we are working really hard on it and hope to bring it to you soon. Also, I want to acknowledge upfront that although Janie and I are the lucky ones to come here and speak to you this morning, the work we're describing is a team effort. And so right up front, we want to thank the other contributors from the Swift developer community to this work. You can see them there hard at work at one of our recent mid-cycle meetups. OK, so I'm going to give you a very brief, high-level overview of Swift before we get into some of the more detailed discussion of how we're encrypting data at rest. Swift is an object store. It has a REST API. It's accessed via HTTP. It offers the usual CRUD operations that as you can create, read, update, and delete your objects in Swift through that REST API. Swift is not a file system. It's not a hierarchical file system. It has a very shallow naming hierarchy. Objects are grouped into containers. Containers belong to accounts. And in the context of an open stack environment, those accounts have a one-to-one mapping to Keystone tenants. And Swift is perfect for storing blobs of unstructured data. So maybe it's a virtual machine image, photographs, media clips, whatever it might be. Swift's great for storing that stuff. There's a number of clients available for Swift, both command line and libraries, bindings to various languages. Just showing you here how you'd use a very simple CURL command to make a put request to create an object in Swift and use GET to retrieve that object. If we look behind the scenes, Swift has been designed to be highly scalable. The evidence for that, there was talk yesterday in the summit, and in previous summits, we've had talks from operators who've described how they've successfully scaled their production Swift clusters to capacities of many tens of petabytes, storing billions of objects. How does Swift achieve that scalability? Well, there's many contributing factors, but one of them is the way in which it does a great job of distributing load throughout the Swift cluster. So when an object is put into Swift, it's first handled by a proxy server at the front end. And one of the roles of that proxy server is to decide where that object will be located in a larger pool of storage nodes in the back end. And the proxy server does this using a mechanism known as consistent hashing, a slightly modified form of consistent hashing. And much as I'd love to go in the detail of describing to you how that works, really all I have time for is to tell you that the outcome is that objects are distributed uniformly across that pool of back end storage nodes, which means that load is distributed uniformly across the pool of storage nodes. Another property of consistent hashing is that it's deterministic. That means that any proxy server can independently determine the location of an object in the pool of back end storage nodes. So that means that we can also very easily spread load across the front end proxy servers. There's very little need for them to communicate or share state with each other. So that's one of the ways in which Swift has been designed to scale. One consequence of that is that any particular storage node, in fact, any disk in a storage node, is highly likely to be storing objects that belong to more than one tenant of the Swift cluster. Now, the significance of that and why I've drawn that out now, I will come back and explain later in the context of encryption. Swift is also a highly durable storage service. So in fact, when that object is put into the Swift cluster and arrives at the proxy server, the proxy server actually spreads the content across multiple back end storage nodes. And it does this with a configurable redundancy factor. So this is achieved either by using a rager coding or just straightforward replication of the object content. And on my slide here, I've kind of illustrated a very typical replication policy where three copies of the object are stored on three storage nodes for durability. The choice between a rager coding or replication depends upon the application. But whichever it is, the outcome is that Swift is able to continue to serve object data back, to serve get requests, even in the face of node failures or network partitions. So your data is very durable in Swift. What about security, though? Is Swift secure? Obviously, we're here this morning to tell you about some work we're doing to make Swift more secure. But it's worth pointing out that Swift, as it stands today, is not an insecure service. I think many of you will be familiar with the Keystone Identity Service. Swift uses that to authenticate every request that's received at the API. And Swift has authorization middleware that will either grant or deny access to resources for those requests. Also, it's a feature of the Swift architecture that only those front-end proxy servers are exposed to externally-facing networks. The larger pool of storage nodes in the back end are isolated from those external networks. So those are measures that are taken to prevent the unintentional leaking of data out of Swift through its API, and they do a great job at that. But there are other ways in which data might leak from our Swift clusters. And the one that we're specifically focused on in the work that we're doing is the risk that a third-party gains physical access to a disk from one of the back-end storage nodes. So in other words, how do we protect ourselves from this guy? This is Fonzi, I believe he's a friend of Janie's. And you can see here that Fonzi is making his way out of a data center with a truckload of disks. Now, it could be that Fonzi is a trusted, authorized member of staff that is in the process of disposing of disks that have been retired from a Swift cluster. If that's the case, then we hope that those disks are going to be thoroughly scrubbed if not destroyed before they leave a secure physical environment. So the story actually of how I obtain these disks just demonstrates the need for encryption. Went to a computer store that shall remain nameless and smiled real pretty and said, I needed some disks for an art project and left the store with a basket full of somebody's used hard drives. All right. And I read an estimate recently, in fact, the estimate is now several years at a date, so I'm sure it's a very conservative number, that each day, something in the order of 50,000 drives are retired from data centers globally. And the process of ensuring that those drives are scrubbed is not straightforward and may not be particularly quick. So there is a risk that these disks find their way out of a data center with our user's objects stored on them at rest in the clear. Of course, it could be that Fonzi is a malicious intruder, and that actually he's in the process of stealing these disks. And if that's the case, then we have even more reason to be concerned about the destination of the data on them. So what we're seeking to do in Swift is to add the ability to ensure that all of the content of our user's objects is encrypted while it's at rest on those disks. And then, assuming that we do a good job of keeping the encryption key separate and secure, we have a lot more confidence that should one of these disks find its way out of the data center unclean, that in fact, our user data is effectively inaccessible to third parties. In a moment, I'm gonna hand over to Janie, and she's going to talk in more detail about how we're building that into Swift. But first, we're just looking at a couple of existing mechanisms that people actually use to achieve at rest encryption. So the first might be to use some kind of hardware encryption technique. This could be self-encrypting drives, or it could be that you have a disk controller that has encryption capabilities built into it. And presumably, this is gonna give you great performance. The encryption algorithms are implemented in hardware. And also it means that everything that is written to those disks will be encrypted. So not just the content of Swift objects, but actually all of Swift's metadata, and in fact, the metadata of the file systems that Swift is writing its objects into, everything will be encrypted as it's written to those disks. But each of those disks will have a single encryption key that is used to encrypt all of that content. So this means that, let's say for example, we wanted to have separate keys for separate tenants that are used in our cluster. Well, that's not something that we can do using just this kind of hardware encryption approach. The key that's used to encrypt is shared across all of the tenant's data, and this is why I pointed out to you earlier, the fact that some of these drives and all of these drives are likely to be storing objects that belong to multiple tenants. And why might we like more flexibility? Well, let's say you actually wanted to render a tenant's data effectively inaccessible, or effectively erase a tenant's data by erasing their encryption key. But not erasing any other tenant's data. Well, that's not something that we can achieve with this kind of approach. Or it might be that you wanna use what's known as a bring your own key model, where the user actually provides Swift with the key that's used to encrypt and decrypt their data. Well, again, that's not something that we can support using this kind of hardware encryption approach. Moving away from a hardware solution, we could use a software encryption solution. So for example, we could use technology such as DMCrypt and build ourselves some encrypted virtual block devices. Again, this is gonna have the same features and the same inflexibility, if you like, with key management as I described for the hardware solution. Also, just a reminder, I told you how Swift stores multiple copies of objects on multiple storage nodes. So with a software encryption solution that's implemented on those storage nodes, we end up encrypting the same data, in this case three times in three different places, which is not particularly efficient and these nodes are often not optimized for that kind of CPU intensive workload. So, what we've been building in Swift is a software encryption solution in Swift itself that gives us much greater flexibility over the management of keys, including supporting those bring your own key use cases as I mentioned, and also only encrypts the data once as it passes into Swift and decrypts it once as it passes out of Swift. So Jane is gonna come and go into some more detail about that now. So yeah, what we've been working on is encryption and decryption middleware that'll live in the proxy, and so that means that only one copy of the data is going to be encrypted as it passes through the proxy server before it gets distributed to the storage nodes. So one thing is the proxy holds the authentication middleware, so it already has access to the user's identity and credentials, and those can be used to be passed to a key manager service like Barbican. And another thing this design provides is it allows to support user provided keys that would be included into the request as headers, and so this was the BYOK that Alistair mentioned a minute ago. And notice with this design, you can upgrade to encrypt new objects without changing your hardware or backend virtual disk provisioning. And so a side effect of doing it this way is that your data is encrypted during the ingress into the Swift cluster, so the data between the proxy server and the storage nodes will be encrypted in flight. Notice that the in-flight encryption is not one of our primary goals, but it's definitely worth noting as a benefit. So to enable encryption, you want to add three pieces of middleware. For those of you familiar with Ghostbusters, we named one of ours after the Keymaster of Gozer. We have another middleware piece called the Gatekeeper and we use a gating system called Zool, so definitely seemed fitting. The role of the Keymasters to, of course, provide keys to the encryptor and decryptor middleware, so it does this via a callback method. And another role is it decides when it's necessary to encrypt or decrypt an object. So if it decides that it's not necessary, then it sets an override in the environment. For our first version of encryption, the Keymaster is probably gonna remain pretty simple. There's a root secret for the cluster and it's provided to the Keymaster, and from that it derives separate keys for each container and each different object. Note that the keys aren't put in the request environment and they're never cached or persisted in the system. So under this model, the key derivation you see is sort of hierarchy. The inputs to the derivation algorithm are the cluster root secret and the unique path to the container or account. We're using HMAC, which will create a hash that's a SHA-256 and it makes use of the root secret to then achieve an appropriate level of security that we need for this. So as we mentioned, we're gonna support the BYOK. This is the push model where the user supplies the necessary keys on the request and just wanna reiterate that those keys are not cached or persisted. And this has a lot of greater interest or interest in the greater community. There's several talks this summit for this topic. Yesterday, Nathan Reller gave a great talk on bring your own keys. Today, the security team has a fishbowl and tomorrow, a Barbican has a fishbowl. So those would be great to attend if you're interested. So this is the pull model. The only differences instead of these are providing the keys on the request. The Keymaster can call out to a key manager such as Barbican in order to get the keys. And so the client can be managing their own keys in that key manager. So let's move away from key derivation or key management for a little while and just talk about the encryption itself. So on the left, you see a few values from the request as it is in plain text. And then on the right side, you have the corresponding values as they will be at rest. So we have the eTag, which in Swift is the MD5 hash of the object contents. That will be encrypted and encoded, as well as all of the user metadata on an object is encrypted and encoded. So in this example, we have bank account password. That string will be encrypted. And of course, the object body. And you see here, the length of the body remains the same. And that's an actual special property of the cipher mode that we're using. So we chose to use AES encryption with 256 bit keys. And this is an industry standard, widely accepted to be secure, despite the fact that the NSA agrees with that. And AES, just like any other block cipher, has to be adapted to be able to work on an object size larger than a block. So in order to do that, we use CTR mode. And so there's several different ways to adapt an encryption algorithm to string data. So in Swift, we have a specific requirement to be able to decrypt an arbitrary sub-range of the object. And that's because we have a feature called a range to get. And so the user can request to have request to have a sub-range of bytes from the object. Actually, they can supply a request for multiple ranges. And CTR mode allows for efficiently decrypting those. So this is a picture of CTR mode. I wish we had time to get into it, but I really just wanna say the important part is the plain text is only X-Ord with the key stream so that what happens is the cipher text ends up being positionally equivalent to the plain text. And that allows for the efficient ranged gets. And now we're gonna watch a demo of how to encrypt your Swift data. Okay, thanks, Janie. How are we doing on time? Just, we're good. Okay, so for this demonstration, I am using a mini Swift cluster that is running entirely in a single virtual machine on my laptop. One of the advantages, it's one of the reasons I'm doing that is it means that all of my services, particularly my storage node services, are writing their data into directories on that virtual machine's file system. So it's really easy for me to show you where the data is at rest. And we can go and kind of dig around in that back end and have a look at stuff. So, yeah, so all of the essential services are running in this virtual machine, including a little pool of back end storage node services. The code I'm running, all the work we're doing is being developed on an upstream feature branch. So the code I'm running is from there. And as Janie mentions, we've worked really hard to not break any of Swift's API. So this code passes all of our Swift functional test suite. Okay, so the first thing I need to do is get myself a token that I can use throughout this demo to authenticate all of the requests that I make to the Swift API. And I'm just gonna kind of label that point. So every request I make to the API is authenticated. Therefore, I expect, and it's perfectly okay for me to be seeing my data in the clear, to be seeing plain text come back through the API because I have authenticated myself. Unlike when I go and look at the file system and start opening up files. So I have a token and I need to create a container first into which I'm gonna put an object. And so I just use a curl command. And again, throughout the demo, I'll just be using curl to make put and get requests of Swift API. I pass in that token as a header and have the storage URL there, appended with slash C. C is the name of my container. And that worked, so I start to relax. Good. So I'm gonna show you three scenarios. In the first scenario, I'm actually going to override our encryption middleware, disable encryption, and just show you plain text being stored at rest in the backend. The second scenario, we're then going to use the encryption middleware that's installed in this mini Swift cluster. And use the service managed key model. So Swift will be applying the encryption keys to the data that I pass in. And then in the third scenario, I'm gonna show you how we can do bring your own key where the user that the request will actually be supplying the keys to Swift. So first of all, I wanna disable the encryption. So we've just put a little feature in here where we can override our encryption middleware by adding this header, xcrypto override, set its value to true. And that just informs the encryption middleware to ignore this request and pass data on through without taking any action. And I'm gonna put an object in. I'm gonna be reading the object content out of a file, secret.toxt. So that's gonna go into Swift. The object gets created. And as a sanity check, I can do a get request to the API and get my object back and have a look at the content. And there you go, there's my secret. I love Swift. So now let's go and look at what's happened on disk. So as I said, I'm running a pool of storage nodes. I've actually chosen to use a triple replication redundancy policy. So I'm gonna have three storage nodes that are each storing a copy of that object in the file system. There you go. At the leaf of each of those directory paths is a file with a .data extension. That's where my object is stored, three copies of it. The file name is actually a timestamp, second since the Unix epoch. So the more observant of you will realize that I'm actually running a scripted demo here, but it is live. And if you don't believe me, you take that timestamp, run it through a date command, and it's about 30 seconds ago, or however long it took me to say that in UTC. So let's just select find one of those data files in the back end. There's the path to it. And take a look inside. And sure enough, there's my object data in the clear. That's what I get if I manage to steal this laptop, get this disk, what kind of data center. I could just look in that file and find it. Now I mentioned at the start that Swift organizes or groups objects into containers. So we can do a get on the container URL, and it's gonna return us a list of all of the objects in that container. But that list for each object also has some metadata about the object. So let's do that and see what we get. There's only one object in this container. So there we see a little dictionary that has some metadata, it has the size and bytes of the object when it was last modified. But it also has this field, the hash. So somewhat confusingly, this is what is also referred to as the ETag, the entity tag. This is the MD5 sum hash of my object content. So once we start encrypting our objects, this particular piece of metadata is something that we would also like to have encrypted. We don't really want to be revealing the hash of the plain text of our objects. And this is also stored in the back end. So again, if I filter now for the container part of those storage node directories, at the leaf of each one, there's a database that's holding that list of objects. Again, it's triple replicated, so you see three copies of it. And if we look inside that container database, so I just find one, one of those paths, and run an SQL select query on that database. Sure enough, there's the metadata for that object. So this means that as well as encrypting the object data file that I first showed you, we also need to encrypt at least some of the metadata that finds its way into the container database. Okay, so now I'm gonna actually enable the encryption middleware in this cluster. So I go back to the beginning of the sequence of curl commands. I'm gonna put my secret object in once more using a put request. This time, I'm simply omitting the X Crypto override header. So I'm gonna be using the default mode of the cluster, which is to encrypt my object. That succeeds, and now we're gonna find the object file in the back end. And there's one of the data files. You'll see that file name, that timestamp has advanced because I've just put a new version of the object into the cluster. So that's now the current time. And if we look inside this file, that's ciphertext, okay? So my object's now been encrypted. If I do a get to the API as I emphasize at the start, my requests to the API are authenticated. I expect to get back the plain text. So sure enough, I do get my plain text back through the API because the middleware is decrypting this object on its way back out of Swift. And just to check, if I do a get to the container URL to get that listing with the metadata of the object, because that's an authenticated request, the container metadata has been decrypted, and I see that object hash, that entity tag, the MD5 sum unchanged, still in the clear there. However, if I now go and inspect the content of the container database that's at rest and run that SQL query again, you'll see that what we put into the database is also an encrypted version of the object entity tag here. There's also a bunch of metadata there that just informs the decryptor to do the right thing as the object comes out. But this is the ciphertext of the MD5 hash of our object. Okay, so that was showing you an object being put and retrieved from the cluster using the inbuilt encryption mechanism and the cluster managed keys. I'm now gonna show you the third scenario where I use bring your own key model. So to do this, I need to generate myself some keys. They're just 32 byte string sequences. I need to base 64 encode it so I can send it as an HTTP header. I have an object key and I make myself a container key. And then I construct same curl put command to put my secret in, but this time I'm adding two headers to pass in those two keys that I just generated. That succeeds. I'm gonna take a look at the object file at rest on disk. Again, the file name timestamp is advanced. We have another new version of the object. And the ciphertext has changed because we're using different keys. We're now using the key that I passed in rather than the key that Swift would have allocated from its own key master. And finally, just final sanity check. If I do a get and again provide my own keys with the get request, then Swift is gonna decrypt this object and give me back my plain text. Okay, that's the end of our demo. Back to Janie. Great, thanks. Okay, so why is this hard? Well, it turns out if you take Swift as it is upstream and add a couple middleware to encrypt and decrypt just the object body itself, the API is not maintained. In other words, functional tests would pass, I mean, would fail. So part of the reason that is is because the eTag or the MD5 hash is not what's expected. So the team has found a solution to that and to several other problems I would love to have time to get into, but they're listed here. If you're interested in talking about them later, we can do that. We started late, so we gotta kinda go through this. We're working on the feature branch upstream, very close to landing. We hope for that to be in the Newton release. No promises, of course. And we're interested in knowing feedback about use cases and integrating with the other projects. So please let us know if you're interested in talking. And just reiterate thank you to everyone who worked on this, and thank you for everyone, to everyone who came. And so, questions. Hopefully we have some time for, we have a few minutes left for some questions. Yes. Great work, nice presentation. Any thoughts on integrating the pull model into external key managers like that support KMAP or stuff like that? So IBM's done a little prototyping using that pull model and talking about the KMAP use. And we have someone working in Barbican, there's Fernando Diaz is probably the person you wanna talk to to get some more details. And for upstream, there's a lot of challenges with again maintaining the advanced features. When Swift Cluster doesn't have access to the keys at all times, there's some problems. Actually the talk yesterday on bring your own keys had a great comparison of different models for bring your own keys and the advantages and disadvantages. And we kind of have a similar story here with, yeah, playing with Swift. Okay, so I'll just repeat the question briefly. Anyone didn't hear it? The question was have we given any thought to secure wipe concepts? For example, down to the level of being able to delete the key for an individual object. So it's effectively securely erased. Yes, we've given a lot of thought to that. It is one of the more, as Jane mentioned, one of the more challenging aspects of key management. I mentioned at the start that one of the reasons we wanted to work towards supporting more flexible key management was, well the example I gave was actually securely wiping an entire tenant's content without having to impinge upon other tenants' content. We have given it a lot of thought right now. That's kind of future on the future track. Our key master is a little simpler than that. But I think I'm confident that what we're building allows that flexibility, allows us to start to explore those kind of concepts. Another question? Yeah, so you mentioned being able to integrate with something like Barbecan when working with the key master. Have you also thought about working with Castilon as a generic key manager interface? Is that what you're using? Yes, essentially it would be Castilon but I didn't have time to explain. Okay, I was just checking. Okay, one more. I was just curious if this introduces any additional overhead from a CPU perspective or how lightweight it is? Yeah, it does. It doesn't come for free. So it's a great question. And we're in the process of characterizing the performance impact of doing the encryption in the proxy. As I'm sure you'll understand, Swift is a pretty complex system. So benchmarking, performance benchmarking is something you have to do carefully and present the results in a way that doesn't actually confuse so I apologize if we haven't quite got that ready for this summit but some interesting early indicators. So one is that actually the encryption algorithm is real quick and that is not where we see the bulk of our performance impact come from. It's actually that, we keep mentioning the entity tag, that MD5 hash. So as we transform the data and do the encryption we have to kind of like calculate new versions of the attack. And so that's actually where we see cycles being lost is in doing that calculation rather than in doing the actual encryption engine. Very, very ham wavy. It depends on the object size but the impact that we're seeing is equivalent to if not a little bit less than the impact we see if for example we choose to use a range of coding which is also implemented in the proxy. For the bring your own key, do you store any information about the key that the user provides? So I'm thinking about the use case where I have my own key, I encrypt it and then when I want to retrieve the object back I accidentally give you the wrong key. Will you just give me a garbled text or will you throw me an error message? No, so the answer is right now we don't but we know that we need to and it's a patch away. And actually it'd be great to chat about it if you've got ideas, it's the best way to do that. So I think what we realize we need some kind of identifier, something where we can basically correlate the key that was used when we encrypted and stored the data with the one that's provided, with the get request and then take the appropriate action if the two don't match. And my guess is the appropriate thing is not to just simply return garbage. So probably some kind of 400 response. So yeah. Okay, I think we probably need to stop. Sorry if you had a question. Come and grab us afterwards. And once again, thanks for coming along to the talk.