 Hi everybody Thank you for coming to this talk on the last day of the summit It's nice to see everybody filing in for a security talk and making the time to be here. So we appreciate that My name is Rob Clark. I'm the lead security architect for HP Healy on and I'm also the open stack security project PTL Presenting with me today will be Arvin Tuari and Dave McCowan We're gonna talk to you about a few different things and I'm just gonna run you through Some of the crypto stuff that's going on in open stack today So in this talk, we're gonna talk about encryption capabilities about key management And then about some new projects which involve localized VM key management, which accelerates your path to having encrypted VMs and A new method of doing full-disk encryption for cloud deployments, which should allow you to deploy entirely self-encrypting clouds So I'll talk to you a little bit about encryption and where we are in open stack today So This is how products sees open stack Lots of cameras come up This is more like how developers see open stack and this is how security people tend to see open stack and Of course, we all know that the way to solve security problems is to encrypt everything I encrypt all the data at rest. We're encrypt all the data in transit and then everything will be nice and secure Unfortunately encryption isn't really a silver bullet It's a useful tool and it's the focus of this talk But just when these technologies come through the path to securing open stack will not be complete There's a lot more left to do but encryption technologies are something that we're constantly asked for So in clouds you really have three different levels for encryption these three tiers You have application level open stack native and full-disk encryption. So application level protects you The application layer it protects the data in your applications in the cloud It's very useful for migrating data between cloud securely. They're a good Good products out there that allow you to do application level encryption Open stack native that I'm going to talk to you a little bit more about Allows you to protect entire workloads at rest allows you to cryptographically isolate tenant data Which means that in certain situations you may be happy enough to have say Pepsi and Coke running on the same devices and Can't in some situations with appropriate key management protect you against operator intrusion Full-disk encryption encrypts everything that touches disk It's generally a requirement for a lot of data center operations, but it's normally an unmet one and I'll go through why It allows you to do a number of things But basically you're protecting Disks when they're removed from a data center be it through an RMA process or anything else And those open stack has this horrible habit of spraying credentials all over drives It's nice if they're encrypted before they leave your organization So open stack native encryption actually overlaps a little bit It can help with some of the application level challenges and can help with some of the challenges that full-disk encryption does as well So where are we with solutions on these today application level it's pretty much been solved in the commercial space already So I'm not gonna spend a huge amount of time on that because we've got loads to cover But those are three companies you might want to go look at Open-stack native encryption So here we're really looking at the most core services in open stack be it Cinder Nova and Swift and in a minute or two I'll go through the status of each of those services and where they want to be by the end of The next release and full-disk encryption So I work for HP HP cells very expensive Secure encryption hardware that is awesome and great and you should use it if you don't want to use it I'll tell you how you do it for free in a minute. So this is basically how How a service any one of these core services is going to operate They can operate obviously they can drift from this. So your service be it an overall or whatever will run encryption through Through its own encryption. So all the encryption will happen within the service keys will be brought in probably via cast land Extracting keys from Barbican, which will in turn probably be talking to an HSM You see here. We've also got salameter in this orange audit trail It's really important to know what's using keys and what's performing operations in your data center And this is just kind of the general framework for all of the open-stack natives encryption services So Swift today There's some experimental code Swift turns out to be one of the hardest services to encrypt You would think it's probably one of the easiest ones But because of the way Swift manages chunks of data spread out across many many platters It's very difficult to pull back an individual chunk and Decrypt it on its own or do a key rotation on it because your entire object in a naive solution would have would have been encrypted before it Is sprayed across the disks you don't want to have to reassemble a terabyte sized object to be able to rotate individual bits So it becomes quite messy They're aiming to have encryption be a real big priority for this cycle. So hopefully we'll see more out of them Much further down the road is Nova ephemeral encryption. You can boot encrypted Nova drives today Not many people support it in their production products, but if you go through the documentation and there's a doc a security docs Article in review at the moment that addresses this it will allow you to boot Nova encrypted drives however, it relies on live that Live at LVM which at the moment doesn't support migration. So you have a choice between being able to migrate or being able to encrypt Lastly the last core service I'm going to touch on here is Cinder Cinder again will allow you to create Access Encrypted drives that you attach and remove or encrypted volumes that you can attach and remove There seems to be some problems at the moment with booting encrypted volumes My understanding is that both the roadblocks here so creating encrypting volumes from images and actually booting encrypting Encrypted volumes directly are being addressed in the next release So with that I'm going to hand over to Dave. I have no idea if I've run along or short So I'll try and work that out before I speak again, and I'll pass over to Dave for key management All right, so at this point as Rob described your data is encrypted. It's in one of the services and it's safe and secure That was the easy part apparently the hard part then is what do you do with the key? Your data is encrypted is protected with a key. You shove it under the mat and you're good to go Unfortunately, there is no mat in in the cloud So that that's the question is where do you put the key and the good news is that if a black hat does manage to Capture your data without your key He's a lot for the bunch of garbage the bad news is that if you somehow lose your key You've also lost your data, and that's also a bad scenario, so we need to have a key manager that works with your cloud So you want to have you know the following attributes in your key manager one it needs to enforce some sort of access policy You want to make sure that the black hats cannot get a copy of your key, but you can always get a copy when you need it And to that end it's highly available You don't want your data to be available just sometimes you want it to be available whenever you ask for it So your key needs to be also available Whenever you ask for it you want your storage for keys to be durable. You don't want to your key to be lost if there's a power outage or Server crashes they need to make sure that your key storage is durable, and you want to have some sort of auditing and logging you want to know If there's a compliance check or an audit of some sort You want to be able to have a record of when keys were accessed and by whom and you want it to scale to your needs Depending on your cloud this may be once a day you want to start up or it could be every time you access data and Of course whenever you're building a system you want it to be easy to use and low-cost and high-performance Of course you can't have all of these things So you need to figure out what your use case is and design for that Two options that are available today for your cloud For a key storage. So Barbican is the official open-stack key manager It keeps copies of your keys. You have an option You can have a back-end plug-in to a hardware service module which protects your keys in a FIPS compliant manner Or you can tie it to just an encrypted database In this case, it's you know compatible with with open-stack. It uses keystone You can choose to use roll base or ACLs to protect who has access to your keys And it ties in well with keystone and Oslo and the open-stack command line at SDK. So you're all set there If instead of encrypting at the open-stack native layer, you're encrypting out the Infrastructure layer then you need to put your keys somewhere else. Maybe directly into an HSM and And manage the access to that separately away from open-stack protocols But just having a place to put it's not enough It's not just a project that needs to design a strategy that meets your needs and an architecture that meets your needs And in this case the type things you need to be thinking about As a customer, do you want to keep control of the keys yourself and take on? That responsibility to have an HSM on your own site or do you trust having the key store in the cloud? With your data and protected with the cloud availability Another question, do you want to have one key or many keys? Does each of your data records need to know their own key? Or is it one per project or one per cloud? Another question you want to think about is is key rotation or replacement if a key is compromised can you change to a new key? You want to build an architecture that has redundant storage and then need to figure out Do you need some sort of backup or recovery? Scenario, do you want to keep a copy of your key that's entirely offline in a disaster scenario? And again, you can't have all of these so you need to pick which ones you do You want and prioritize as you design your system So here's some specific examples of how you can turn encryption on in your cloud In dev stack or your production available today. So here's an example of sender volume encryption Simplest way. I want to put my key somewhere. We'll just stick it my configuration file That's great for dev stack There's other secret bits. Maybe your rabbit passwords in there anyway, so you might think that's a reasonable idea So no judgment. You have to decide what works best for your system Another option if you don't think that people have access to your configuration file should also have access to your data Here's how you would tie sender in with with barbican again in your Uh sender nova configuration files just point to your your barbican instance the url and you'll have encryption for your cinder volumes Talking inside the bm's themselves if they need a copy of an encryption key The way to do that is just embed the key right into the image that you boot Probably not a good idea if you want to share your your image with someone else Or if just because someone has access to your image if you don't want them to have access to a particular volume Um, so in this case you can go back to barbican and keep your keys inside the key manager Um a couple different ways uh using code available today to pull the key the key um at a barbican into your vm You can use the open stack command line as in the first example Um the downside there is then you have to give keystone credentials Inside your vm. So if you want to avoid that here's another example If you use ansible for deployment you can use the keystone credentials Of ansible to pull a copy of the keyword stick a copy of it inside the vm and then the vm can can boot and it's ready to go But you can see from these examples that didn't meet all our use cases You know lots of things that you have to trade off But we definitely need uh more options to cover all our use cases And uh, that's what we're going to present next Two uh complimentary proposals one's project marshal and project leason The first one project marshal. Um, urban's gonna come up now and uh and present this It's an example of an open stack native encryption strategy And then rob is going to come back and talk about project leason, which is an example of a full disk encryption So here's Thanks, jav Hello everybody We have been looking for a key management solution for sysco cloud services And we choose barbican that doesn't mean we are in production. We are serious about it So we have to solve multiple encryption use cases and Volume encryption is one of the important one So we started looking into the current encryption support in open stack and we felt that Not all of the use cases can be solved So we started investigating and uh, we are working on an agent based solution So if not all but it's going to solve a lot of problems. So that is what I'm going to solve Talk about and So we have done some work over there and we thought okay, let's bring it to the community and discuss with the Smart folks here and get feedbacks and enhance it and make it usable for everybody so that we can use So here is a marshal. So marshal is a key injection An encryption agent and that's a slogan is keep calm marshal is on So what is marshal? So marshal is an agent or demons process running inside the VM And it features keys from key management system And barbican is the main focus, but idea is not to depend on one Try to make it and there are regions for it. So and it interface with encryption encryption subsystem of operating system and uh Inject get the key from the key management system and inject to the to the encryption subsystem and dm crypt and bit locker are the Everybody knows that It is an abstract abstraction. So we can use any encryption subsystem Maybe proprietary or we can also integrate with the key management system as long as You are talking in rest or k-mip. So we haven't done anything in the with the k-mip. So we are we are testing with the rest so far And this is the high level flow how the whole encryption will work. So user will Go to the horizon and create key as per their need and key will be stored in hsm and Next user will go and Try to spin up Some cinder volume or maybe they have their own existing volume. So they can have a number of volume and Now next they want to attach that volume with the new Vm or they want to attach with the existing vm So they can have a number of vm. So marshal is uh is a Component cooked up with the with the image. So it will be it will be running over there and So after attaching the volume with the vm So marshal provide a command line interface and from there We can say okay marshal. I want to encrypt this particular device And it has it will have some some information which i'll i'll show in in my demo And based on that information it will go and pick the key and inject it to the to the dm creep model and forget about the key From there onward the Disciple will be encrypted and protected. So basically marshal is going to provide more transparency flexibility and control and features And it is abstraction as I mentioned so we can integrate with other key management system if needed And uh, so why marshal? So as I mentioned current encryption is pretty rigid. So it uh made Open stack barbican and it is very well defined flow And uh in in multiple use cases And the customer environment they don't use barbican and they want to keep their existing key management solution So we thought okay. Let's not Ask them to migrate everything. So try to use their existing key management solution another problem with the existence. It only support Linux version machines and there is no support for containers or mics of windows vm It's support full disk encryption only nor would this uh disk partition So marshal can fill those kind of gap And uh, it is open stack centric. It means it Expect barbican and pretty much open open stack architecture. It required and on on prem key Integration is not the focus right now, but it practically it can be integrated And benefit it is transparent user can choose their own key and it is Uh, we can support linux as well as mics of virtual machine It is cost-effective. What that means is if you have your existing key management system so We can we can utilize that without migrating your cool migration is costly. So And it is extensible. So we can we can use it for root disk encryption or partition And monitoring target certificate is these are not the focus of this project But that can be doable and containers and bare metal are the uh next focus for this thing So these are the roadmap. We have done some work. Uh, not everything is done. So full disk encryption We are supporting it and uh for the fine grained. We are we have tested with the Partition containers are the next thing which we are we want to work on Microsoft windows. We haven't done much, but that is the next thing Ability to fetch fetch key from the on-prem. That is again, we haven't done anything. So these are the roadmap for marshal and Okay, so gaps which is why we are here to discuss and anybody can guess with this kind of solution authentication is biggest issue and You can expose Way to the key management system So there are some option we are thinking. So first thing is One-time trust. So in the keystone, uh, there is a One-time trust so we can use it to get the key and after that there is no way to access the Key management system short-lived token another thing which we need to discuss with the key management system and with Right set of automated script. We can also fill though in that gap and Barbican community is working on Persecret policy, which is also going to help here And we are trying to introduce a concept of license key license is something which is Which is which will be generated by Barbican through the api and idea is marshal Okay, so idea is marshal is going to use that license to access the key And horizon integration. We have done something, but which is a kind of hack, but We can share if that looks good automation next thing which we are going to focus on And security of key in transit. So there is a feature in transport key which can be utilized for this These are the standard deployment model. So generic image or standard packaging can be utilized. So Next expectation we are looking for support from all the communities and inputs and Okay, we have already Shared the code as well as wiki. So go ahead and look into it and We have also opened a blueprint for Barbican so Next is a demo. So we are going to show a short demo, which is mostly manual We are not using the automated heat Temperature something. So these are the agenda for the demo and let's see if I can Oh, okay Okay, so So keep calm marshal is on again and so these are the end of we are going to create key through horizon And we are going to use Barbican orders and secret resources And we have also tested with one of the Other key management system called hashika vault And we are going to show full-disk encryption as well as the single partition So this is the interface we have done some work on the horizon. So from there you can create an order and You can choose your different parameter. So you can see order is created and from there you can Get the secret reference or you can go and list the Secrets and get the secret ID So End goal is to create a license. So we are creating a license. It's pretty clear and we are putting a Credential which is not good. So there I mentioned we can use different techniques But the idea is to create a license. It is it has to be it should not be that open We can we can do a better job over there so License is going to have metadata information about the key for example key id and the end point or Some kind of credential information so that it can access it So here we are using the horizon to spin up a VM and we are providing Image with marshal cooked into it And we are providing the license to the cloud config So these are the manual way of doing it. But see VM is up and running and license is embedded into the VM and from there Okay, this is the license format for the hashika And we are attaching the the volume now and These are the command for Marshall these are the marshal the specific commands and now we are Giving a command to encrypt the device. So so this is We and it provides multiple options to choose from so you can choose your cipher and And see We are checking the status and it is the encrypted it is encrypted now Now next we are doing the partition of device and we have created two partition and we are Trying to encrypt Particular partition from there. So it is it is encrypted and you can see the status in the next screen Okay, so you can see that it can encrypt the particular particular partition also So that's all Let me close this. Thanks Cool. Thank you very much. How awesome was that see encryption works today? All right, so I'm going to talk to you a bit about leason Leason is a project that we've we've worked on for a little while that we're just going to socialize now really Um, it's a crazy idea for managing full disk encryption and Linux at scale So it's fairly decoupled from open stack um, so I'm going to talk to you a little bit About the state of the art in linux full disk encryption today And that is to say that your laptop could well be encrypted and you type the key in And your machine boots great Unfortunately, that doesn't really scale too well at data center level Because you don't have people running around your data center typing in key material all the time and um Beyond a certain scale you expect that machines are basically going to be in a constant state of reboot There's always going to be something rebooting somewhere So the state of the art is someone something some system will notice that machine is rebooted and in the very best example I can find will ipmi into that box over what is probably an unsecure network And then type in the key material over the serial console on the box And then eventually the machine will boot Unfortunately, what normally happens is it's it's they're waiting for a key Waiting for a key And you end up with something a friend of mine called cryptodos, which I quite like It's a state in which your data set your um machines in your data center Failing to operate because they're waiting for key material So to address some of these challenges, we've come up with a project called leason Um and addresses, you know linux full disk encryption is over 10 years old hasn't changed an awful mark awful lot in most models it requires external intervention Someone to go in and type something or some process to trigger to go and bang a key in over ipmi So what leason does is and It's a really scary idea. So you're gonna have to bear with me while I explain to you how it works Um, it reverses normal key ops So we go to a pull model instead of a push model and that Means we have to change some things about the way parts of the system come up But it allows us to make some Interesting guarantees against certain threaters But basically what it's doing is binding disk crypto to the network And the strap line for this if you're trying to get product people involved Is it will allow you to build self-encrypting clouds that is to say your installers can make sure every single part of your system Comes up encrypted and and if you couple that with open stack native encryption through marshal or through the other services that are coming And add application layer on top Then you'll have that wonderful full encrypted stack So I warned you this was going to be a scary idea And you just kind of have to bear with me and just remember the base we're working from So our threat model, which I'll talk about after I've told you a little bit more about leason Basically involves people stealing entire chassis And when they do that, you can't trust anything that's stored anywhere on that chassis The only things we know That we can trust really are the ip address the machine comes from if it comes over tls because tls guarantees. We're talking to who we think we are and The uuid's of the volumes in the box that need to be booted Which we can take as reasonably trustworthy as their uuid falls are not reasonably guessable within our lifetimes, etc So leason has two phases. It has a registration phase and a retreat retrieval phase So registration as you can tell in this very very technical sequence diagram Involves a client box coming up for signing uuid's and this will be like the first time it's come out The first time it's been pixie booted or whatever And it will go and register an incredibly secure key in this case dead beef Against a uuid with a leason broker service And leason has been connected to over tls using a publicly signed cert So it knows that this client at least right now that it's talking to is 10 dot 2 dot 45 dot 1 2 2 And it stores this mapping of ip address to uuid to key maps And you know everything's good and fine Then the machine reboots because machines are constantly rebooting in your cloud When it reboots it sits there and it tries to mount a machine mount a disk and realizes it needs keys So it starts to leason again and says can I have the key? For this uuid remember I can't store credentials on this box I can't have a usb stick in this box with key material or credentials I can't use a tpm for a couple of reasons that i'll go into further on but basically for now I can't guarantee that all these machines will have tpm's So I ask leason for a key leason looks at where the request has come from And looks at the mappings that it has now remember a uuid isn't reasonably guessable So it looks at uuid and uses a proof of possession so that the machine that's requesting it has the disk that's requesting it And it looks to see if the request comes from the same place that made the registration If it does it gives back the key hugs everyone's happy This is kind of what leason looks like in a more typical deployment This is taken from you know, it's vaguely healing unflavored But should look familiar to anybody replace control node with director or whatever and you translate to whatever your distribution is Um, and basically we end up with a simple architecture like this Where leason's running on the control nodes Initially it would run on the installer for the registration phase And I mentioned the threat model and why we can't necessarily trust things most disk encryption schemas Work by protecting the individual disk It's fine to do system measurements or use tpm's to derive Encryption keys from system properties if the only thing you're trying to protect against is somebody stealing a disk Or a disk going missing through rma or something like that Unfortunately Sometimes things go very wrong Our threat analysis And our threat model for this includes entire chassis or even entire racks including the hsm going missing And I'll tell you why I once worked at an organization that paid very very close attention to how much state you brought in and out of the building Any disks any cds any hard drives especially needed a lot of paperwork exactly what was on the disks Security had a very well-defined process for doing this right If you wanted to walk out of that building With five chassis on a wheelie truck Wearing highly visible clothing the security person would put down their check board We'll come across and open the door to help you out of the building This can and does happen So we can't rely on any system properties to guarantee it. So we tie it to the network. This is why we have this lease and service We can't use hsm credentials Even if we wanted to if you could guarantee they couldn't be stolen or they're in a separate locked compartment Um HSMs don't like it when you try and set up 10 000 users on them They some of them don't like it if you try and store 10 000 keys on them. So we can't really use those We can't store the credentials a bottom up attacks. I'm going to mention in a minute And I've just brought up everyone's favorite crypto related Uh cartoon not that there are many um, which is to say that It's important to keep in mind the human factor when you're trying to build crypto systems and understand what's actually being protected against And this is one of the things we're looking at with leason is is an insider Who has the ability to influence the network to some extent? Who has the ability to physically access and remove hardware? um, and this this insider can operate in a number of different ways and Years ago people might have thought this was being extra paranoid but you know snowed and releases around tailored access operations in the us and how Discs were tampered going between manufacturers and sources or on the way to rma facilities and other things shows that these sorts of attacks can and do happen and We've always traditionally like to draw this very strong boundary between what data at rest protections will protect against and what live system protections Cover and normally they don't mix and with leason we started to mix them and say That we do care about how you get your key material even though we only ever care about this being called data We still don't want this guy to be able to retrieve key material for another disk Even though it should only be protected when it leaves the data center um So that's why leason uses uuids to guarantee that it is giving you Some close mapping to to what you should have it should mean that he can't Reasonably guess the key material and retrieve it for drives and before he takes them out of the center He might be able to he looks like a competent fellow But we're trying to raise the bar and like I say remember where we're starting from Manually punching in keys over the serial consoles Right, so I think we've got time To do this right so effect this is only an interesting demo if you really like like watching boot screens so the point of this You know leason is proof of concept code So this is post registration and what we're going to do here is just show in this case It's a vm, but I just imagine it's a machine in your data center booting and This will only be interesting to people who Have run into this problem at scale right so here's my machine booting It's coming up. It's doing its various checks um This was run on on tim's laptop and this demo would be faster if you bought him a nicer laptop So I've made a note to do that But the key the key request is going to come up in a second and The nice thing about this is you've got network bound requests For key id with proof of possession the key passes back and then you've got a full Debian system booting with full disk encryption for each of the volumes it has And you're interlogging and that would that scales to 10 20 000 machines in your data center So we've left a bit of time for questions because Some of this is a bit crazy So yeah, so that's it basically that's our talk I think I can do a slide advance somehow there we go So yeah, so leason binds crypto keys to the network protects against entire chassis theft Protects against your hsm being missing in the case where everything gets torn down um Your your bootstrap for bringing your cloud back up is you only need to manually punch in a key for one of the leason nodes Because everything is encrypted in this including the lease and service It's running and the data that it has stored So if you switched off an entire data center and switched it back on everything will be sat in crypto dos Waiting for key material and you go to leason and you give the leason server It's key to boot the other leason servers because you'll be running in ha we'll be Pinging each other waiting for one to come up as soon as it comes up They can boot and then everything else in the data center can come up on its own So in the event of like a big disaster recovery or a big failure where everything has disk material Because this is nice and ha and scales out and it's all done in software It will all come back up pretty damn fast And The last point here is just the one i've tried to reiterate through L this talk It is probably the least worst way to do looks at the moment And that's actually I think a reasonable step forward So I think we're on to questions So dave arvind For marshal and key management has anyone got any questions for any of us on what we've discussed Okay, why can't you just stuff an ssh server into the net ram and ssh in and drop a key or decrypt the drive and then continue boot You can that's still a push model though. So you still require that. Yeah, that's a push Yeah, so you still require something to know that it's now time to go in ssh into this box And i've never seen a data center Actually do ssh key authorities appropriately and you can never revoke ssh keys Yeah, and you can do tls with the ssh as well. So Well, you can do ssh. So it's they're not tls Google i'm fairly sure that google is doing tls ssh is like how they're doing their security internally So one of the newer versions of ssh does introduce support for ssh certificates They are slightly different to more normal tls certificates. In fact, it's not tls because it's ssh But they might be using certificates in it for for or then Um, yeah, like I said, but Right, that were actually one thing I meant to mention. I see nathan there. Um, there is also another project I became aware of while socializing this earlier in the conference called dio, which is by red hat The first version of what they do is very similar to what we do today in our proof of concept The next iteration, which is something we're going to be throwing a lot of attention to to start off with and hopefully get involved with Actually uses some some homomorphic encryption approaches to to really extend this technology out. Yeah, you still work from the same base that How do you bootstrap trust into a node that you don't have any trust anchor on? Yeah, um, but it's you know, so there's it's interesting that it validates the approach we're taking and The two can coexist, but I'm really excited by how we might be able to mix the two together and do something really clever with it And thank you for the question We got a couple down here. Maybe once you're done at the back, you can bring the microphone down Sure, uh, you assume that the that's a ui ui d and ip mapping there, right? Yeah, what if on a reboot because of dhcp. I got a different ip address So that's an excellent question. Uh, we do pretty much need static ip's for this to work at the moment. Um If you're happy enough having your dhcp be, um So earlier in the in the diagram I showed mapping to a cmdb and to an hsm That's what the cmdb mapping is there for basically it's whatever your system of authority that knows how your cloud is configured is It can be mapped against there So in our proof of concept absolutely needs static ip's If we were to build it out a little bit further with a cmdb integration That's exactly where dynamic ip addressing would come into it Yeah, that's because I'm thinking I'm from the south side of things You've got lots of disks and the ui d tends to come You can take a disk out move into another chassis because that chassis failed So when you've got lots and lots of disks that becomes a problem ip changes and ui d mapping is really changing frequently So for sef this would you need add extra light you just tell yeah, you absolutely do I mean we can you know, you can introduce api extensions to allow an administrator to say I've going to move this from here to there, but actually the ui ds would change when you did so you need some sort of Short overlapping registration phase That's what the cmdb is really there for is to do those sort of things It's disturbing to me how many data centers don't know what disks are in watchmen what machines So we will kind of get hung up on that But yeah, it's absolutely one of the problems we need to tackle moving forward And I think there was a question down towards the front They mentioned Hello, how do you How do you integrate marshal into an existing open stack cloud? I mean is it something you put on the nova compute nodes or Or what yeah, as I mentioned like we are going to support the standard packaging Deviant package or meredhat packaging so we can do apricot and we can embed it Is that at the host level or in the VMs or inside the VM? Okay Yeah, that is one way yeah or Other ways we can cook up images with marshal and further new VM can use them I saw that lisin supports high availability. Is it a particularly bad idea to run that geographically distributed? As long as you can either make strong assertions around the ip's or that you have a good cmdb that can Do that sort of geo distribution as well that can keep track of it. It should work in fact lisin solves Solves this problem for me today. I have it running somewhere. I have my file server at home runs All my nas stuff. It's on a static ip Um, I switched it over to lisin recently before then you could guarantee whenever I went to a summit The power would go off at my home and it would restart and then the state of lenox encryption being what it was I would have to ring home and get my girlfriend to punch in my encryption key while I read it out to her Okay, now it reaches out to a server In the us east pulls back the key material An extra layer that we could put in that I didn't I didn't mention because I was trying to keep all the diagrams nice and simple We do have a pre encryption model for it as well where like in the case of my nas The keys that are kept in lisin are encrypted with local keys that are kept on that box So even if lisin was completely compromised or somehow compelled to give keys out to the wrong machine So even though a uid isn't guessable Let's say there's a vulnerability in lisin and they're able to retrieve it the keys would still be entirely useless It does frustrate the key moving operation though because you have an extra step of wrapping and unwrapping So it's a It's a trade-off that you have to decide whether or not you want to make But thank you for asking the question because that reminded me to tell people about key wrapping as well related to the distribution geolocation and in the To protect in the data for that storage the swift community is discussion about the Secret share method to use to protect data in the distributed strategies So the this technique can be used for the another project. So how do you think about this technique? Secret share method to protect data in the distributed discuss So the can you? Integrate the project this approach to your project Can we integrate that towards lisin or towards the I'm not and I think it's a complex question. Maybe we can talk about it afterwards And so I can go run through a bit more of it really Okay, I see Okay, I think that's everyone. Thank you so much for coming in for asking so many great questions