 No problem. So just FYI Let's get started the the sooner we start the sooner everyone gets to go home. This is very exciting So thank you guys for coming to the common Barbican use cases and options for your open-stack deployment I know it for many of you is the last day. It is the last talk on Thursday for everyone So we do appreciate you guys being in attendance. So we didn't have to just present to one another again My name is Sheena Gregson. I'm a product manager at Mirantis in a very recent previous life I was the product manager for Barbican at rack space. So this talk is near and dear to my heart With me presenting today are Doug Mendes of all who is the Barbican PTL Adam Harwell who is a core contributor on neutron L bass and has been working on the integration between neutron L bass and Barbican John Dickinson who is the PTL for Swift and is going to be explaining the spec for the Swift Barbican integration and Joel Kaufman from Johns Hopkins who has implemented an integration between Barbican cinder and Nova and I Know that for some of you if you've looked at the schedule, it looks like we're having a panel We're actually not that's a sneaky switcheroo on everyone. They listed me as a moderator But I was mostly just the person that made sure that we had a slide deck and presenters So I'm going to happily step off the stage after I give you guys the run-through of the topics and let these guys take it away So today we'll be covering five topics more or less For those of you who aren't terribly familiar with Barbican will go over some of the basics terminology that you might need to know What the projects intended to do how to use it? We'll also go over the common integration libraries So if you're a developer and you're interested in Using Barbican some ways that you can use it without actually talking to the API We'll go through three use cases integrations with Swift, Neutron, Elbas and Cinder and Nova We'll talk about a couple of other pieces of information that you may want to look at after and then we will definitely have time for Q&A Or for you to escape quickly So without further ado, I'll hand it off to Doug to go over the Barbican basics All righty. Thank you, Sheena Thanks everybody. So I'm just gonna give you sort of the quick five-minute elevator pitch for Barbican what it is and and what it can do So Barbican is a RESTful API for key management in a secure storage and generation of secrets And we're in official open-sac project. We're part of open-sac in the big tent now and we started to sign the Design the service from the ground up to be part of open-sac. So it's got stuff built-in like multi-tenancy It's designed for scalability So you should be able theoretically it'll handle hundreds of thousands of millions of keys or secrets as we like to call those It's also got a configurable back-end architecture And so what this means is a Deployer will have their option to talk to their security guys and sort of figure out what the best option is for Deploying Barbican and there's some trade-offs on sort of the the back-ends that we have Some of the options include dedicated hardware Modules like a HSM sort of hardware security modules that provide sort of high security assurances for the storage of your secrets and We've also got Configurable CA back-ends one of the cool things that Barbican can do is provision TLS certificates And so you you sort of have your choice of what CA to use in the back in the back end for those So some terms so that we're all sort of on the same page We call basically any kind of sensitive data We call those secrets and secrets could be a bunch of different things you could have Symmetric keys for encryption Asymmetric private keys so some like your SSH key or an RSA key or something like that You could also have passwords if you wanted to you can store passwords in here And of course x bar by 9 certificates for TLS encryption and such We've also got this concept of orders, which is the way that Barbican is able to provision these things for you So if you want to provision a certificate you would submit an order to Barbican and then Barbican would process that Asynchronously eventually would produce a secret from that order and that's that's how you retrieve your stuff It's just like any other secret We've also got a concept of containers now This is just a logical grouping of secrets in Barbican and some of the containers could be for example a certificate Could be represented by a container if you want to group it together with its private key As well as its public part of the certificate and stuff like that also RSA key pairs could be a container You could have the public key and the private key and if you choose to encrypt your private key You could also add a passphrase to that so you could have all three secrets Related to each other in a container We also got a concept of consumers Which is a way to add metadata to a secret and sort of add information about other systems that may be using that secret and Adam will talk a little bit more about that when he's talking about the LBAS integration so if you're interested in using Barbican in your Application or in your service or something like that. There's a couple of choices for sort of what library you want to use for that integration And so the first library I want to talk about is Python Barbican client Know this is your typical open-set client. It's going to include a Python API as well as a command line interface And if you're familiar with any other Python-project client, it's going to look very similar How we use the keystone client and keystone sessions for authentication? So you don't have to learn anything new there Our latest release is going to be version 3.11 which is available on PyPI now And it supports secret storage for both the Juno and Kilo release of Barbican Unfortunately, it's laying a little bit behind it does not have some of the features that were released with a kilo release a couple of weeks ago But we are actively working on those so there should be upcoming in a future release It's a couple of advantages and disadvantages that you're going to have to consider when you're thinking about how to integrate Barbican one of the advantages of using the Barbican client is that it uses all these Barbican conventions that I just mentioned like secrets orders containers all those things so you don't have to really relearn like if you know Barbican terms you should know how to use Barbican client after that Also, even though we're lagging that is where we implement new features first So if you want some new feature in Barbican best bet is it's going to be implemented in Barbican client first One of the disadvantages though is that it tightly couples your application with Barbican So if you decide to use Barbican client Then you are basically saying that your key manager is going to be Barbican no matter what now for some people That's not a problem For some other people it may be a problem For example people that are working in a government setting where the key manager has to comply with certain compliance regimes like VIP certification or something like that Barbican may not be the right key manager in that situation And so if you find yourself Sort of needing alternatives for a key manager if Barbican is not the right solution. We have another library called castlin Now this was born sort of a pattern We started to see with services that were integrating with Barbican where they were putting an abstraction layer of a sort of a generic Key manager in front of Barbican clients. So we we took those all those interfaces and combined them into sort of a general You could think of it as like the lowest common denominator key manager And we call that castlin and it'll allow for Different implementations to key managers. Obviously Barbican is one of them And the disadvantage to this is that it only implements a subset of Barbican functionality Specifically a certificate provisioning will likely not be available in castlin And so with that in mind, I'm gonna bring Adam Harwell up here to talk about how he's using Barbican in Neutron al-pass All right, so why did we choose Barbican? So as I'm sure most of you are familiar secret storage is not an easy problem So actually it's it's so challenging because of the guarantees You have to make in a lot of cases that for Neutron al-pass v1 They actually chose to not have SSL termination rather than try to solve this so Fortunately now with al-pass v2. We do do we do Support to al-pass termination and we do that via Barbican So another thing Besides just the fact that Barbican will store our keys for us is that the type containers that they have so Doug talked a little bit about this there are containers that are generic that just hold any kind of secrets But there are also type containers which hold very specific kinds of secrets with specific names So in our case we use a certificate container and that guarantees us that we'll have several named secrets in it We'll have the certificate itself. We'll have the private key for it We'll have intermediates that might be necessary and a passphrase if the private key has one And so that allows us to really conveniently keep that In one package so that when users are using our service, they don't have to figure things out. It's just it's right there for them We're also making use of a rather new Feature in Barbican per secret ACLs Access control lists So normally when a user puts their secrets in Barbican only they'll be able to access them by default So for our service to access them, we actually utilize this feature so that the user can allow our service account to access specific Certificate containers so that Elbas can go in and pull them on their behalf And also consumers are awesome or rad as Sheena has here on my slide Consumers allow us to do sort of a basic reference counting Thing so that we can tell and actually it's a little bit more than that. We can actually link back whatever is using This secret container or this certificate container by its URI And store that in the metadata for the container and we can do that atomically when we Retrieve the data so when neutron Elbas gets a user certificate we atomically register as a consumer So there's no chance that the user will Accident well, there's still a chance, but there's a lower chance that a user will accidentally Delete a certificate container out from under us Which could cause problems for us down the line And it's also that can also be really useful for like horizon or other UI integration So that you could see right on your secret Hey, this is in use by and actually have a link straight to the thing that's using it So you could see this secret can or this Certificate container is in use by this load balancer Possibly in the future maybe a firewall. I mean the it should be really useful for Allowing users to keep track of what's using all of their secrets So our actual use case is pretty straightforward So we actually allow the user to store their own Certificate information so their own cert and private key in Barbican either using Barbican's API or hopefully soon there will be a horizon UI We're crossing our fingers on that one And that that way the user never has to give their cert to us. They only trust Barbican with it or their private key so when they Store that in Barbican, they'll get a container ID which represents all of those seek necessary secrets together Then they'll make a request to our service Neutron Elbas and they'll pass in that container ID Along with their TLS termination create or update At which point we will use Python Barbican client to go get that data Register and then validate the data and pass it on to our load balancing appliance in the back end which then uses that certificate data to do TLS termination so it's a fairly straightforward, but Definitely adds a much needed functionality to Neutron Elbas in v2 I'm actually that's all for me. So I'm going to pass it to John to talk about Swift's use case Thanks So one of the use cases that we have in Swift is well generally we're here to store data and lots and lots of it and when people start doing that there's new use cases that come into the community and it's Really to me personally one of those exciting things are like hey, we're using this but we need Feature X or Y so that really gives us really great incentive to start working on solving this so one of the things that has come up a few times in the community recently is How do we store our data, but we need to make sure that's for various reasons that it's encrypted while it's stored and That's a huge. That's a huge problem space that you can start trying to you can you can tackle and talk about it for years Before you actually get around to doing it. So we've constrained the problem space down to this When somebody stores data in Swift regardless of the client that they're using or any or anything like that when it is actually Persisted to a storage device The plaintext will not be there neither the metadata nor the actual data of the object is being stored and this is really nice because the specific attack vector that we're planning on shutting down the one that we need to account for is What if a hard drive walks out of a data center Because there's a lot of people who are using Swift today in very sensitive places that have a lot to do with personally identifiable information specifically medical records and financial records and Then even you've got corporate information that you want to make sure you keep inside of your your own Facility so it's really bad because if a hard drive walks out and somebody can read the data That's the kind of thing that for some of these people ends up on the front page of CNN or something like that And they just cannot happen So what we need to do is make sure that regardless of the client they're putting that into They're they're encrypting that when it gets onto a drive So the problem of course is that when you're storing hundreds of petabytes of data and billions and billions of objects key management gets tricky so That's why we're looking at things like barbecue. So this is something that is An in-progress thing in fact just I just came from one of our working sessions where we were specifically talking about some of the Outstanding work we had to do this and we saw a little demo of this Working in a you know kind of a demo since yesterday, so we've made some great progress on it And it's something that's an active area of discussion so the basic idea is we're saying that we need something out there to provide us the The assurances that we can get from a key management without a key management system without us having to deal with securely Doing that because it's hard and that's outside of the problem scope of object storage So that's why we're looking at Barbican and saying this is what we need to use so We use Barbican we don't have to worry about what proprietary thing is out there we can trust that that's going to be Part of the overall ecosystem, so that's where we are and the basic idea that we're working through is this at some point When an account inside of Swift is provision That's just a storage area for where you're going to end up putting your data in Swift when that is Created at that point. We're going to this is remember. This is one possible implementation of what we're working towards What we're going to do is we're going to request a secret to be generated and at that point We will store that with Barbican and ask Barbican to generate that secret so then we can get this the secret back and Store some unique identifier about that either we could deterministically ask for it later store a key ID or something like that With that account later on a user of that that Swift account is going to want to store some data And at that point we can find the key that we used to store things for this account Ask Barbican for that particular key and we can get the key and encrypt the data and then When it's persisted to the disk it's encrypted data and metadata cannot be read Later on somebody's going to ask for the data back to actually read it And once they're authorized to do so then we can request to the same exact process we request the key back from Barbican we are able to decrypt that and Return the plaintext back to the user so the end of the overall goal here, which is really exciting I think is that the end users don't even have to worry about this. You can have any sort of client you need Maybe it's something that's official upstream that knows exactly. Hey, the right way to do stuff Encrypt stuff on the client side before we push stuff all the way down clearly that's going to be the best way to encrypt data However, that's not the only use cases out there and all that and you can't always trust all of the clients They're talking to the storage system. So you need to actually make sure that the data is encrypted so specifically for the problem space that we're trying to solve for this is something that's going to be really good and We're working through this now and the so far these kind of patterns that we're looking at here Provide us both the high security that we need, but it also is going to be able to continue to scale For the use cases that we have Storing massive amounts of data and things like that. So like I said Working on it now trying to figure out what all the details are and get all those those pieces worked on and we expect to have this done Soon, I mean as soon as possible. It will be released and it'll emerge into mastering and released in Swift So I'm hoping that this is done before the Tokyo Summit. We'll see what happens there But you know, maybe maybe after that when we get to Austin We'll just kind of have to play it by ear and see what when we have something that's validated tested and things like this So it's really great. We're working with the Barbican team very closely They've been a pleasure to work with as well as HP and IBM kind of leading the way on Implementing this into into Swift. So it's been really it's a really fantastic thing and every single time I hear about Barbican I learned something new and I'm thinking this is actually kind of cool It's like well, I was wondering how we're gonna do this this particular scalable thing. Oh, we've got castellan over here It's gonna it's gonna solve that problem or hey, did you think about this and oh, yeah Well, we've got that crypto library over there that's automatically implemented that for you And it's it's just been a really great thing because every time you come out There's like this this is exactly what we need. So anyway, I'm very excited about what's going on and I'm looking forward to it and I hope that you're able to use that so with that I think Joel's next I'm gonna tell you about some stuff at John's office So I'm going to talk about some of the work that APL has been involved in Across several different services in OpenStack We started in OpenStack as part of the grizzly development cycle and the primary Situation that we were initially trying to solve was a very similar use case to John's However, instead of working with object storage, we were interested in securing block storage across OpenStack So the first place that we started was Cinder and Nova to provide encryption across all of the persistent storage provided by Cinder So as part of this we knew that we needed key management But we really did not want to write key management ourselves at the time because key management is really hard It's hard to do securely. It's hard to do well Unfortunately when we started this work, there was no barbecue on project that was still about Three or four months away from the initial Prototype that was going to be displayed as part of the Havana summit in Portland So what we started with instead was what would become the precursor of the Castellan library We implemented a very generic interface it supported features like creating and getting keys and The idea was we didn't know what the form the eventual key manager would take But as soon as the key manager would be stood up It would presumably have an implementation that satisfied these various functions and Barbican certainly Provides that functionality for us. There's other compliance regimes like Doug mentioned previously such as FIP certification that may be required for Governments or even financial information or health care information And we want to be able to plug much more high security Devices that do satisfy these compliance regimes Behind Barbican or independently and Castellan gives us a way to do this We also have a pluggable back end for Barbican that uses the key management interoperability protocol or came up for short In order to satisfy these various things So what we're getting out of? Barbican or key management Castellan in general is secure key storage separate from the data itself When we were starting the work, we knew we couldn't store keys either in the Nova database or Cinder database Very nice for a prototype not good from the eventual security that you want to have for one of these features And the other thing is that we needed all the keys to be accessible across various services So we couldn't embed this as part of an isolated Single service because in our case as you'll see momentarily What we actually do is we generates the keys from Cinder and ask for a key to be generated But Cinder never actually handles the key material itself That key is stored in Barbican in this case, and then Nova is retrieving it later on In terms of the implementation we completed the majority of this work as part of the Havana release of open stack There is at the Hong Kong summit a talk entitled encrypted block storage technical walkthrough which explains all of the details in At great length So you can feel free to watch that presentation if you're interested in the work And we have continued to improve on that sense if you go back and look there's a lot of command-line interface driven stuff in that particular Presentation all of that's been replaced now. We're fully integrated with the horizon dashboard As part of the Keeler release We were also fully integrated with Barbican for all of the key management as part of the Keeler release as well So the way this this actually works is the user is going to create a new Cinder volume So that request is going to go to Cinder Cinder is going to actually take this request Recognized that it needs to create an encrypted volume based on some of the parameters of the request Cinder is going to then ask Castelon or the key manager interface to create an encryption key Castelon will pass this request on to Barbican assuming the Barbican is the back end for Castelon Which is the primary back end currently? Barbican is going to create this key It's going to return a reference to that key which is going to be stored as metadata Within the Cinder database. So again Cinder is not actually handling the key material itself. It simply knows that the key exists Then later on when the user is going to attach the volume to an instance Nova is going to receive this request. It is going to recognize that it is an encrypted volume when it retrieves all the connection information from Cinder Then it will go back and ask for the encryption key from Barbican again going through the Castelon interface to do so The encryption key will be sent back to Nova once Nova has it it will be used or passed to DM Crypt in order to provide Transparent encryption and decryption of all the data being written to and read from the volume and so to the virtual machine instance it has no Actual knowledge that the volume is being encrypted. It is completely transparent to the end user Apart from some additional enhancements that we've added to the horizon UI in order to indicate with a very simple Yes, no that the volume itself is encrypted But the user can't accidentally turn any of this off if you're hosting a database And you just simply need to make sure that everything in this database is encrypted You can turn this on as a configuration option for example, and there's no possible way that the data would ever be stored on disk And it's plain text So with that I am going to hand it back to Sheena in order to Cover some additional information as well as handle questions My handle questions. He means handle deferring questions just for clarification I also noticed on here. So we cut down the number of steps Significantly, so you'll notice that the last two go 914 I actually am better at counting than that But we were trying to cut down on the number of steps to make it a little bit easier to read So that's not that should be 10 So just a couple of bits of information for those of you who actually access the link at the beginning of the presentation There are two specs that you may want to take a look at based on the the presentation today Swift spec for the implementation between Swift and Barbican and the Elbas neutron Octavia spec Which uses the new load balancing appliance that that team is working on there's also a Barbican contributor meetup tomorrow So if you haven't gotten enough Barbican You can find these fine folks tomorrow at 1 20 p.m And they would be more than happy to talk to you about Potentially using Barbican for your project or open stack deployment And with that we've finished and it's time for Q&A if you'd like to run away Everyone's email address is at the bottom take a picture make a run for it. Otherwise, we'll be happy to stay and take questions a Question about the Nova Cinder Barbican integration If it does the encrypt I assume it's not completely Back-end agnostic. So for example, it might not work with Seth or does it? It will not interoperate with Seth currently. It is relying on I SCSI connections That are formed by the other cinder drivers any future plans to somehow magic this into being We would like it to happen the Probably best way to do this is add encryption to Much lower in the stack say to Kimu. We have investigated that possibility We aren't pursuing it immediately But it looks like there is some motion on the Kimu development list that this could happen within the next six to twelve months If someone else does make progress in this area, we would certainly be interested in tying this into the other feature So I'm gonna just do plus one for that right there. We want that integration with Seth Please My question though the next question though is you know working working with John on the encrypted Swift stuff One of the things we want to be able to do is when Swift the key master hits Barbican it goes This key is old. We want to be able to recycle or deprecate that key and So my question is how do we do that in the status because I want to have a list of keys One is active and some are deactivated. They're not deleted, but they're still there So when I come across the container in this container is has a deactivated key I can go. Oh, it doesn't have the current active key So I need to unwrap the container key with the deactivated key and rewrap it with the active key So kind of how do I do that as far as like status? How do I how can I look up the key and say this is the one that matches this particular container that hasn't been accessed for a year? You know what I mean? I think that's a great question No, I think it's here so they I don't know Really what the what the implementation is going to be because it is not yet written yet So we can make it whatever it needs to be that being said One of the what you described there is certainly one way that we could do that. However, I have some Concerns with my understanding of how that would work and what the effective effectiveness of that would be Specifically around the scales of keys that would be needed Yeah, and we're not talking often like three years as NIST guidance for the keys We're looking at so John will continue to work with you Janie and try to figure this all out as we go One of the things that we're specifically talking about is having some Something that might be a little bit more nested. So we have we can lower the scope of the keys. So we have some sort of Derived keys key derivation stuff to make that go down. So that's maybe we can Store a little more limited numbers of keys that can be we can use to derive those little lower level things say a container Or an object level thing and if that's being said I don't know how we'll do key rotation or if we will because that's hard with that much data Re-encrypting all of the data. So yeah, yeah, I don't want to re-encrypt all the data I want to do it at a very high simple Encrypt, you know, you like key encrypting key sort of thing The short answer honestly is it's being written and that's That inside of Swift we're the way we're doing this is creating something called a key master Yeah, the key master is responsible for talking to Barbara can cast along that kind of or whatever else is out there and The key master API is currently being defined. That's what we've been working on this week and we'll be working on over the next few weeks so and the implementation is certainly not done at all and The we want to support various and many different key masters. So the if you need one that does a particular thing Yeah, we want to enable that And I hope it will have a key master that will support most of what you need And that will have the tools available so that you can write one that has their specific requirements Super. Thank you, sir question about Getting the infrastructure we need to have tenant based logging in open stack So one of the people one of the things is manasca right now has API's to say Oh as a as a VM you can send logs up, but it's requiring the open stack Token as a way to validate and we need a secret to be a long-term secret that the VM has kind of Pertinent and not something that's going to expire so that the VM can keep sending logs and metrics up into a shared infrastructure but we can make sure that coax VMs can't fake Pepsi's data and And So I guess a question I had on that is do we have anybody kind of looking at that? Infrastructure an integration with Nova to try to have a kind of tenant based logging password pushed into the VMs and stored in Barbican So I think this is more of a general problem that we've been discussing on the mailing list sort of last week There's actually a design session going on at the same time as this talk Kind of talking about there, but it's basically how to get How to assign Identity to a VM is sort of the base problem We're still hashing that out. We don't know what that what the answer is now I'm hoping that we'll figure at least a plan for work before the end of the summit But it's definitely a problem. We're aware of the problem and we're working on a solution But we don't have one yet. Okay. Oh So what's your room was that other go to D. No, the other one was in the Sakaar the sign summit is whatever wherever room Sakaar was meeting in right now It's where they're and we actually sent some Barbican folks over there to participate on that. All right. Thanks If we could go back to the Swift implementation proposed, I guess Swift implementation and We're talking about medical data. We're talking about if you're Calling up a study. You're gonna be calling up not one object, but several in the case of an x-ray hundreds in the case of an MRI and Potentially thousands in the case of a CT scan. So I'm wondering what kind of if you have any I Guess predictions for what kind of overhead that would introduce into the system to As far as it being multiple objects versus right. Yeah so As I I mean and we're leaving aside just the overhead of decrypting encrypting the data, right, right I Don't anticipate there being significant additional overhead to Beyond the CPU needed for doing the encryption decryption. Okay. The reason is because Again looking at the use case we're storing so I'm what I'm about to say is not solving other use cases But it works efficiently for this one for the use case that we're solving we are solving the Data at rest keeping drives from walking out and exposing that data so one of the things we're doing especially to improve performance and keep It working at large scale is we when we request that token at that key I'm sorry that the secret that's what it is the secret from Barbican and Use that for encryption decryption. Let's suppose that that is stored on a container resolution So every you know the million objects that you have the million files that you have inside of one container in Swift It would be sharing the same key potentially That being said we can cash that for a while We can store that internally in the cluster without having to go do that fetch again so that we can have very very fast access to it For subsequent requests to objects in that container and basically in this way it works It would work very very similarly to how we do off tokens and keep those As performance possible inside of Swift so I don't anticipate a high level of additional overhead But if you do want to talk more about that I'd be happy to introduce you to some other people who are more directly involved with it Sure. Thank you This question is for both the Swift and Cinder implementation at what level is the key generated is it a per tenant? Is it a per VM from the Cinder case or is it per container in this which case that kind of thing? Pervolume per volume. Okay Yet to be defined I The key generation will be done inside of Swift at some logical layer of Swift So like that's gonna be like a cluster a cluster the whole cluster for account or container or object and we maybe do something with key derivation and figuring out how to keep how to do that rather than request doing a request for creating a new key every single time But the resolution will be Configurable on there depending on that key master implementation Okay, so the the configuration part is kind of what I was digging out there as well because for example our application we actually Might run multiple versions of the app and for every customer within our app We're also multi-tenant so we need to be able to encrypt things per customer and Even to the level of per account within a customer. Yeah, I think that makes complete sense. Yeah I have to support that And actually I meant to mention this earlier, but I had someone from the heat team stop me I think yesterday or the day before and tell me that they've got an integration working as well So it wasn't featured here, but these are not the only use cases We have a lot of folks inside the community making use of barbecue and functionality And I'm super excited to be a part of the presentation Showcasing all the hard work these guys have done so with that with that be free Thank you guys so much for coming and have a wonderful rest of the summer