 Great. Hi, my name is Shawn Michael Kerner and you guys are in the open stack security session So you picked the right session because there is no more important session Than this one because if anybody tells you that they're not interested in security They might be misleading you because everybody is I'm an editor at e-week and internet news But you're not here to listen to me. You're here to listen to this panel We've got a panel of experts on open-stack security from multiple aspects from the internal project VMT project from Barbican and then people that just work on open-stack security at the largest public open-stack cloud vendor So it's a good mix because when we think about security security means a couple different things, right? There's the open-stack security project There's the VMT the vulnerability management team. There are open-stack security Projects which are things like Barbican and we have people from Barbican the Barbican project here the Ptl And then there's how we actually implement things. So all those things mean security and everything in between So just to introduce the panel and we'll just go on the line and everyone will introduce themselves Awesome. So my name is Douglas Mendes-Aval. I've been working on Barbican for about four years Formerly at Rackspace currently in between jobs And I was Barbican Ptl for a while not any longer also I've been working with the security team Sort of for fun on my off-time So I know a bunch of those guys My name is Jeremy a Lot of the community technical community probably knows me as Fungi I'm a longtime member of the community vulnerability management team since 2012 or somewhere like that and Hold a lot of other roles in the community technical committee Ptl of the infra team Foundation staff member, etc My name is Dave McCowan and I work for Cisco in our cloud solutions business unit my focus there is securing our NF BI which is the foundation for our NFV solutions and Upstream, I'm the current Ptl of Barbican taking over from from Doug working on on key management. I'm surrounded So I'm major Hayden I work at Rackspace on our private cloud product So I focus on just security among other things in general a lot of around upgrading and maintaining And lately the biggest project for me has been the open stack ansible security project which Applied stick hardening standards to any host that you have open stack or not Great, and I'm hoping that this session will be as interactive as it can be because security is not a spectator sport So feel free to to us take up one of the mics if you ever have questions. I've got a few Leading questions, but you've got experts up here. So if you have questions by all mean and to those of you They're watching in in the future in the recording. Sorry, you'll have to email us afterwards And just to go right to left And talk to you Doug first and then everyone can answer in line if you want to jump in What do you guys think is working well across open stack security today? And that's a big question because there could be the various pieces that you're working on whether it's the Barbican project other projects VMT or in general Wow, okay So one of the things that I think has been really cool in the last few cycles is the adoption of Bandit So then it's one of those security teams tools that does static code analysis and I think the community has been very receptive about adding new gates to have bandit test their code pieces Bandit for those of you that don't know One of the most amazing projects in my opinion with static analysis that fits in Garrett So that when people contribute code all the code is actually scanned for vulnerabilities if I understand it correctly Which is amazing because when you think about the vast majority of all projects in GitHub people post it Then people consume it and you have no idea whatsoever. It's if I've been audited for any kind of security But as Doug was saying a lot of open-stack projects are now running through a bandit And that's a huge difference between open-stack and just about any other infrastructure project. I'm aware of So I guess it's hard to follow that one up. That's awesome Bandit is really cool probably I know as far as things that that I touch on a regular basis from the vulnerability management perspective The recent Assistance that the the broader security team in open-stack has has given us and the community in putting together a plan for security analysis standards so that I Don't know how how many of you are familiar with VMT policy internals, but we basically have a relatively restricted set of projects that we feel comfortable taking responsibility for triaging vulnerability reports on and and providing advisories and We we need some additional level of vetting of new projects that come to us and want our assistance in an ongoing basis with these things mainly so we can Make sure that they're in a good enough shape security wise that they're not going to eat up a lot of our time and that that security analysis process that The security team has started to pioneer is providing a really good outlet for these newer projects who are looking for our help to be able to know the sorts of things they should look at in their software and come up with detailed documentation of The the security relevant Areas within their deployment configuration and so on so that then we have something we can go back to later To quickly assess impact if we get a vulnerability report that might relate to one of those areas So what I think is is awesome about open-stack is the the documentation. That's available when you're deploying a Cloud you need to secure every layer if you're going to be successful. So securing The hardware the hypervisor operating system The software running in and out of VMs For any layer and for any project. There's really good documentation and tools And best practice is available to tell you how to configure it properly to harden Your your cloud in general and each component specifically and there's a lot of warnings What happens if you don't do it correctly so the available documentation is great I think for me it's just been actually starting to have conversations around security So I'll say you know I work on a deployment project primarily open-stack ansible and it's been really nice recently because the The keystone team for example has added some scripts into their repo that'll do sanity checks on how you've configured keystone So if you've made really terrible life choices about how you configured keystone or you forgot to change the ownership of a configuration file or something Keystone has a little script that you can run that'll dump out all that data And so what we were actually able to do is implement that in our gate jobs So if anybody ever screwed up, you know a configuration The gate job would fail and we'd go in find out what's going on fix it so that it wouldn't go into production that way That's great. And by all means if anyone wants to ask questions. Otherwise, I've got a long list But it is great that we are talking about security the first Open-stack security session I went to was at the Portland Summit and that was just before there even was a security guide It's hard to believe that there wasn't even a guide in the beginning, but now it's an integrated part of the project Just just to go a little bit further aside I think there's sometimes people in my line of work not necessarily me But people that misunderstand Open-stack security and vulnerability management, which I know is something that you work on Jeremy But we'll just go from from major back. I guess What do you think is most often misunderstood when you see media reports again? Not me because I'm usually pretty good But everyone else just general misconceptions about cloud security vulnerability management And perhaps even certificate management in the case of Barbican Yeah, so I think There was always kind of that knee-jerk reaction when you do cloud whether it's in your data center or somebody else's data center that you know Oh, I'm losing a little bit of control or you know You know, maybe VMs are close together or there's things happening in my system that I can't reach out and touch You know, for example, if you use other virtualization platforms where you have to Individually do each step. It's a little bit different than using OpenStack where you have some services that will go and take care of things for you For storage or networking and that kind of thing. So I think it's it's getting past that and then also understanding how to deploy You know how to deploy things securely so that there's certainly a way where you can go and say Oh, I'm gonna go deploy 200 VMs and just put them all on the same network and use the same SSH key for all of them That's you're gonna have a bad time But if you use, you know projects appropriately networks appropriately and and segregate the different things that you deploy I think you'll have a good experience So I think a popular misconception is that security can somehow just be configured and that you can set it and then forget it and feel comfortable you have a Secure a cloud environment. I think any ongoing security is gonna take ongoing Monitoring ongoing care and feeding most importantly keeping up-to-date on all the software packages for me, I think It seems like probably the the biggest issue is I'm gonna say it's it's related to a lack of real education and Engagement that we have as a as a technical community broader technical community even outside of OpenStack with the general public that There's this perception that security is a binary state either things are secure or they're vulnerable and really security is a broad gradient and Things are always at some point on that gradient And and frequently people perceive them to be in a different place on the gradient that they are But you know the the idea that when there's a vulnerability announced Details are frequently alighted or just completely wrong and the the readers Consumers of of these reports don't really understand the the risk The impact of the vulnerabilities are being announced And that leads them to Panic fear You know those the sorts of things that you can imagine when when faced with the unknown Okay, and I would say probably I Don't know if I've heard it a lot recently, but certainly a few years ago A lot of people looking into the cloud to go deploy our applications Sort of have this impression of you can't secure things in the cloud because you don't own the infrastructure Which I think that's changed. I think we have a lot of good tools now that I'll enable you to secure your workloads on the cloud stuff like the open stack ansible security is a great set of ansible things to help you harden your systems on the cloud so Great and then talking about Ansible because when I think ansible I think you know play books and security configurations and the like Just just to play Davos advocate open stack then by default if I understand you Dave is not secure So if I download upstream plain vanilla open stack Deploy it on my bare metal just because As at day zero minute one am I vulnerable at that point? Or do I need to make some changes whether it's installing a certain barbican? Requirements doing some ansible configuration making sure I followed the VMT advisories or whatever to make sure that I'm secure at minute one Just go from you Dave if you want or whichever way whoever wants to jump in by all means. Yeah, I'll start so You want to start before you install it? You want to make sure you design it? So open stack has has internal interfaces where we talked to the database we talked to the message broker We talked the services talk to each other that network you want to be isolated You don't want to have any external users have access to that network Whereas the rest apis where you talked to open stack they are going to be open by nest by necessarily to untrusted sources So those you want to make sure you secure with with TLS so accessing those only with HTTPS and install them with valid certificates So starts with it with a good design and treating each endpoint differently. I Think that's where the deployment projects help as well. So I mean there's like open stack ansible. There's cola There's chef there. I mean you name it There's quite a few different ways to deploy it But one of the nice things you're going to find in there is that generally the deployers will steer you in a guide of deploying in a Little bit more secure way. I'll say for example with open stack ansible we We actually go and make a lot of decisions like for example use TLS by the fault and then you know Nova doesn't use the same username password to talk to my SQL that neutron does So that way if one part of your environment becomes compromised It doesn't mean that they can take over the entire environment So sometimes it is nice. I mean My employer we we certainly do have a very opinionated set of defaults when it comes to security So sometimes if you're not entirely sure which way you should go look to one of those deployment projects and look at what their Defaults are and it's usually a good starting point And this is an answer that a lot of people generally hate but It's not really a cynical as it sounds All systems are always vulnerable. So so I mean to directly answer your question Yes, when you just play to play open stack or any software you are vulnerable And there's actually a lot you can do about it, but you can never completely solve it like I was saying security is not a binary state one of the probably the most significant challenge with system security is that you are always trying to balance security with convenience and For every new security control you ratchet on you make the system a little less flexible so without an actual risk analysis and an understanding of Your particular risk profile for your use case. There's no one way to secure a system And so for open stack or for any software the the developers can't really I mean They can make some good guesses as to what a good default deployment security Stance should be but but there's no one answer It's really gonna come down to you having to be educated and informed about the software that you're using and have a good understanding of how its risks affect you and I think to well from the keynote this morning From one of the things that Edward Snowden said when when he said was you have to eliminate a whole class of bugs Like you have to find a way to implement security so that you know Okay, if if maybe there's a black box type of your infrastructure or a part of the infrastructure that you don't understand or it's very complex Find a way to keep people out of it or find a way to isolate that from the rest of the environment So that even if like let's say someone broke in and and did something to neutron Well, they should be confined within that area and not have a way to get out and spread somewhere else in the environment So if you're not entirely sure what's secure and what's not isolated Yeah, the Snowden connection is really quite interesting for those that may not know and for the recorded audience At that same Portland summit. There was a guy Nathaniel Hawthorne, I believe From the NSA who spoke about how the NSA was deploying open stack. This was six eight months. I guess Nathaniel Bert thank you. Thank you. I first name right last name wrong 50 50, you know, it's not bad good odds So the NSA is a open stack contributor and user So which is kind of interesting whether or not Mr. Snowden was able to get documents out of a cinder or Swift store I cannot say but it's kind of ironic No pun intended on the word ironic for him to advocate for silos when the lack of a silo in an improperly configured perhaps I don't know open stack configuration allowed him to Reveal interesting secrets but from that perspective and it didn't really want to get necessary to threat attack vectors and that kind of thing but Usually in my line of work people talk about, you know, the malicious outsider Thanks to mr. Snowden people think about the insider malicious whistleblower or otherwise When people are thinking about security, whether it's a barbecue and VMT Ansible open stack just open stack in general. How do you do you take the? Zero trust lease privilege model. Is that the right way to to define things to prevent that malicious insider? Yeah, I would say yes That's one of the things that we were thinking about Iraq space with our barbecue and deployment is How can we configure this so that no one person can? Bring it down or the story or data right like yeah, we trust everybody in our team But who's to say, you know Johnny Whatever it's gonna get real upset about something tomorrow and try to take our whole system down So I think it's definitely something that you should keep in mind and plan for Yeah from a vulnerability management process perspective To take this from a different angle. We also Apply the principle of lease privilege in in our workflow So, you know General vulnerability report intake we we instruct people educate people to provide those Reports to us in private so that only the three people currently seated on the vulnerability management team know Along with the reporter presumably The details of the vulnerability that they've discovered we attempt to confirm whether they're reporting it against the correct project and Once we've done that then we let a select subset of core reviewers for that particular project in on the report So that they can comment on it vet whether or not this is a legitimate vulnerability You know and there's this progressive disclosure model where we we continue to try to limit the the information flow for for embargoed bug reports like that just to the people who are working on at least getting some sort of minimal fix Developed so they can can stem the bleeding when we eventually go public with that particular vulnerability And then at that point, you know, it can it can be further short up and improved, but at least once it's in the public eye There's also a fix handy We we also do a bit of advanced disclosure to a select set of vetted downstream consumers some major public cloud providers a lot of distribution packages so that they can incorporate those fixes into their software in preparation for that announcement the typical Coordinated disclosure model really Yeah, and and you absolutely want to think about defense in depth as you Set up your cloud assume that one of those layers of defense is gonna fail and the next one's gonna minimize the impact that that vulnerability exploit That the damage that can happen and then you want to augment your defensive measures with Comprehensive audit logs and paying attention to them so you if a breach does happen You can identify it and plug it and stop it as soon as possible And I think the zero trust is critical for other reasons like obviously you can have a malicious insider But you can have an unintentional malicious insider So for example, you have five people on a team and they're great people and they love their job But one of them, you know took their computer somewhere and plugged a random USB stick into it And now all of a sudden there's data being siphoned out and they don't even know it So you got like I affectionately called an unintentional insider attack that kind of thing So you always have to find a way and I think the audit logs help with that as well But find a way to just you know have the least amount of access possible And then when there is access just audit it audit every single action Yeah, in my line of work at least half the breaches that I report on It's usually a privilege escalation that was chained to something else and that's just the way it is So if you're not monitoring those privileged administrative accounts, that's not good never never run open stack as root I would assume But that's probably not a good idea in Linux in general pseudo not aside, but we won't get into a pseudo SU conversation We've got a good 18 minutes left if anyone has any questions for the audience feel free to we've got two mics in the corner Otherwise, we'll we'll jump into lightning round and I've got a few interesting questions because when I When I think of a certificate security specifically Barbara can and that kind of thing couple years ago It wasn't considered part of Def core and when I asked people the vast majority weren't running certificates What's happened in Barbican recently and just for the Barbican guy specifically? What's happened to Barbican recently to make it easier such that everyone is actually deploying it as they probably should Yeah, one thing that's improved over the last few cycles is services other open stack services adopting it and and building in support with Barbican so Octavia or neutron load balancer as a service can You know pull the certificates that it needs when it creates a load balancer You know out of Barbican and that there's no user interaction required once it's configured Octavia pulls it right from Barbican and Nova and Cinder and Glance can all have extra security features So that integration that takes place directly service to service. I think Helps with adoption as once it's configured it You get that that support Yeah, I think maybe one of the the problems we've had with Barbican adoption is that people are maybe not thinking about the key Management problem yet or are not ready to to solve it I think adding Barbican to as a core required service would make great strides into getting bigger adoption of the service We're starting to make progress that way As of this cycle the Oslo team is going to adopt the castelon library Which you can think of as Oslo dot key manager and so moving forward any open-sec project that has key management Requirements it's going to be able to use castelon and be confident that Something that works with castelon is going to be available in their cloud Is castelon a sub project at Barbican? is Castelon yes, it's a it's been owned by the Barbican team And and it really you can think of it as also dot DB right like also dot DB doesn't tell you what database to use Other than it has to be sequel compliant Same thing with castelon. It's Like an Oslo dot key manager that lets you use a key manager with a certain feature set But doesn't mandate that it has to be Barbican Fair enough, and then the only other question on the Barbican related stuff is sometimes when I talk to people there's confusion between Keystone and identity management and certificate management in Barbican is am I the only one that hears that? I don't know if anyone else hears that or do you guys hear it? I hear it every day yeah, so the key stand keystone is the identity manager and That's where you you have your user ID. You have your roles. You have your domain That tells open stack who you are what Barbican provides is a secret storage for that user and The way it works is if I want to get access to my secrets I'll tell keystone. Hey, my name is Dave and here's my password and I get a token from keystone That's keystone's job to give tokens based on identity Then I'll get a Barbican and say please give me my secrets and here's my token and Barbican can check that keystone token and say I see you are Dave and I see that I have some secrets for Dave's and then it can return to me the secrets that I have access to so so Barbican is for secure storage of Very particular objects and keystone is for identity and they they work together One way I like to think about that is Maybe you have a safety deposit box keystone is sort of like the key to your safety deposit box But it's not the contents of your safety deposit box And I know you have a session tomorrow. So a little plug for Dave session on Barbican versus vault I believe yeah that later this afternoon. We have a hands-on workshop for Barbican So if you want to spend some time hands-on with Barbican that that's this afternoon I recommend it. I'll watch it on video and then from a vulnerability management perspective How does the VMT actually work? So if I'll give you a wild card? Let's say Tavis Ormandy decides to tweet something Friday at six o'clock. Do you guys you care? But what happens? Well first off I Probably don't find out about it right away because I don't have a Twitter account but Generally, I mean so you know Reporting things on Twitter is not really our recommended way of passing along vulnerability information It's certainly a way and it will eventually get to us through other people who are paying attention to Twitter specifically But yeah, so someone points out something like that pretty much our first order of business is to turn it into a Actual bug report in the defect tracker that we're using Which currently launchpad for the majority of projects launchpad net And I mean if it's if it's already reported in the open like that Well, we can actually move on it a lot faster because the embargo process that we follow and again I'm for those of you who are familiar with the differences between coordinated disclosure and full disclosure I tend to lean more toward the full disclosure community I think that that reporting these things in the open sooner actually solves things for a lot more people but But we do try to satisfy a lot of different Aspects of the community that would prefer a slower more Coordinated disclosure approach and so the embargo process when we get a Private vulnerability report actually does slow us down quite a bit because we're having to carefully involve just the people who might be able To solve the problem and they may have to involve other people and that whole game of telephone takes time If it happens in the open, it's a bit more of a snowball approach We we just ingest every bit of information we have about it into a bug report We immediately start working in the open on trying to get to the bottom of whatever the cause of that vulnerability is if we can confirm that it's actually vulnerable and You know the developer community generally engages far faster on on those problems So, you know the fixes get pushed right into a public code review in that case reviewed by the normal core viewers merged to the source code repositories Generally in parallel with that. We're already drafting an advisory within it, you know accurate impact description that's been confirmed by the developers and the reporter and we send that out to to every Corner of the internet As soon as we've got fixes pushed up for it and those are almost immediately incorporated into point releases for the various stable branches picked up by Downstream deployers packaged maintainers for distributions and so on and just for coordinated disclosure Is there a standard 90 or 120 a policy or is it a case-by-case? It it's it's actually I don't know flexible is probably the wrong term, but arbitrary it mostly has to do with developer bandwidth It's a lot easier and this this is one of the places where I think that the embargoed and coordinated disclosure process falls down is that it's a lot easier for developers to procrastinate on fixing vulnerabilities if they don't think they're widely known so it Because the vulnerability management team while our responsibility is to coordinate You know the the fixes and the reporting for these bugs We're not you know generally directly involved in developing the fixes. We still need to engage Developers who then find the time to provide? patches that you know those those core review teams are gonna agree on so trying to rally the the troops within those Security fixed developer sub teams within the the different projects can can sometimes be a bit of a challenge and slow the process down And so if you're running a private cloud I think these vulnerabilities closures like a full disclosure vulnerability process wouldn't be so painful because hopefully you've isolated The control plane, you know all the API is in horizon interfaces Hopefully you've isolated that so a very small amount of people in your company would actually be able to reach it So even if they said hey, there's a vulnerability in Nova where you can show up and build a VM without a token Okay, well, yeah, that's that's pretty gross like that would be better be pretty bad But on the same hand if you only have five people that can access that API you can go to them and say look this came out Here's the issue no granting any access to anyone else until this is solved But I would say on the public cloud side as you know working for a vendor that runs a public cloud You know a vulnerability like that that would allow someone to create let's say a Nova VM without having a token Would be a complete disaster so a coordinated release would really help because that could potentially allow us to get a patch put in place or Disable part of the code or change the configuration value Before that goes out to the public and then you have you know kids at home trying to write their own scripts and take down the cloud so the the place where we strike a bit of a compromised balance between the two full disclosure coordinated disclosure positions is that for Embargo private reports where we've developed the the fix in secret We we do follow a fairly aggressive disclosure schedule once we have the fix And and to that is generally that we notify the list of vetted downstream stakeholders like You know majors employer and others With five business days Advanced well three to five business days it depends a little bit on the severity as well Advanced notice and and then we go ahead and we've provided them with a copy of the fix We're going to send and as long as we don't discover within those few days That there are any changes that need to be made then we go forward with the the scheduled date that we've told them We're gonna disclose it on but that at least gives them a few days heads up to try to confirm that the fix isn't going to destroy their systems And hopefully have it ready to be rolled out when the announcement is made But also bear in mind there's more vulnerabilities within a cloud than just within OpenStack So occasionally there may be a vulnerability in Libvert Which is what Nova talks to to get VMs taken care of and that kind of thing or there could be a vulnerability in Galera that runs the database there could be something like that. So You always have to make sure you're watching your OS vendors Release lists. So if you're using red hat or a devian or a bunch or wherever make sure that you're following Their feeds so because there might be something that comes up with Libvert that no one in the OpenStack community may be talking about at the time And it may not come through proper OpenStack channels. So that's just something to keep in mind We we also not so much from the VMT but the the security team in general bridges that gap a little bit with their security notes process for some of the more esoteric related vulnerabilities like majors talking about where there may be a Vulnerability and some commonly used component that a lot of people are likely to have within their OpenStack deployment But that's not OpenStack software. So the vulnerability management team isn't coordinating any sort of fix for it or anything we do sort of cherry-pick but the The security notes editors in situations where they think it's particularly beneficial will draft a security note specifically about That related vulnerability so that hopefully at least if people have missed the report from their Distribution vendor or somewhere else, you know, they might see that security note And it gives them a bit of a heads up that that this is something they should be keeping an eye out for And if you're not running SE Linux in enforcing mode, please turn it on Thanks and or app armor, right if you happen to be an app armor person We have about ten minutes or a little bit less left if anyone has any questions We've got mics on both sides questions about how The security project VMT Barbara can security in general works And We can do that no questions, which means everybody is running securely, which is great. Oh, we've got a question. Thank you, sir had a quick question about Some of the operating system agents that are available are actually only know just a few like Marshall for grabbing the key And then using it to enable like file system encryption Is there any other tools or plans for you know updating those to be able to do file level? Or application level instead of just the whole operating system level As far as as far as open stack is concerned So some open stack projects are already looking to integrate with Barbican to provide application level encryption And so Swift is one of those projects that it's working on on having an encrypted Swift storage In which they use those encryption key or they use Barbican to store the encryption keys that are being used for that encryption I Believe cinder has also integrated to where you can get It's kind of OS level encryption because they encrypt the entire block thing But the encryption keys that are being used to do that are being stored in Barbican So it's one of the reasons we would like to see higher adoption of Barbican I think it it helps with a lot of these workflows and to being able to provide Encryption at different layers of the stack Also, be sure to fact check me on this because I'm not a hundred percent certain But I think I want to say freezer has some encrypted backup capabilities as well At least worth looking at and they probably should if they don't Any other questions? Yeah, go ahead So the question was some of the projects are are saving passwords in clear text There's there a way to make sure that that doesn't happen so There's something I would like to see there. There's a conversation that needs to happen So I think a lot of open stack projects realize this is a problem. I think Probably also config would be a good place to fix this right One of the things that we've talked about on the Barbican team is you know, could we work something out with the also folks To have Barbican support and also configs so that you wouldn't have to change every single project the projects are all already using also config and and just instead of reading Those configs from a file to be able to reach out to Barbican and grab those One of the challenges there is that because Barbican depends on keystone. You'll still need To get your keystone credentials from somewhere Right and sort of the best practice for that is to use some sort of Configuration management to provide those so that you don't have to write them to disk Yeah, there was a I want to say it was earlier today, and I missed it. There was a forum session on clear-text passwords in There's a hand waving in the background you were there Can you come up to the to the mic and and fill us in a little bit? I'm curious if there were any resolutions out of that So we actually just did a proof of concept nothing I think we're gonna start pushing up specifically starting with Oslo config basically we use custodia to Interact between Barbican and Keystone So it would pull the secrets from Barbican and then just authenticate with Keystone And then you could actually encrypt or not encrypt you would use the Barbican container Reference and the key name instead of the actual password itself in the configuration file Awesome, so that that's certainly one route There have been very recent discussions as of this week also about Mostly using for lock management, but integrating at CD into the you know Set of expected services that we can use and that it's been discussed also that it would be possible for us to store specific Configuration elements within at CD we could use it for things like passwords, so they're not kept on disk on the on the systems Those systems still need access to those passwords in some way though So it's not really buying you a whole lot of additional security if they need a clear-text password The the alternatives that we've discussed Architecturally over the years are finding ways to get rid of password based authentication for these services anyway I mean that ultimately the solution should look more like you know and Not specifically this People would probably shoot me for suggesting it though. I'm a big fan, but you know rolling out Kerberos So the services authenticate via some ticket granting system rather than relying on some fixed secret Yeah, and in the meantime until there is a solution You know make sure your config files that have sensitive information are locked down with SC Linux So only the Nova process can read the Nova config file and that'll help mitigate the concern about the clear-text passwords Policy policy is always the way SC Linux armor etc I think we're out of time right so we're right at 420. So if everyone will thank me thank me Thank the panelists for being here and thank you all for for being here today. Thank you. Thank you strong