 Good afternoon, everyone. My name is Tin Lam, working as a consultant for Ascension with AT&T. My name is Gabe Chudeau. I am a developer at AT&T. My name is Sam Pill, and I also am a developer at AT&T. So this afternoon, we want to present a problem that I know that a lot of operator express interest in is how to actually deal with the issue of having plain text password in the various configuration file in open sacs. So for this talk, we're going to go over what the problem in AT&T face. A proof of concept that we did, kind of delivering what I know yesterday as a talk, and also config modifying it, and what we tried, how we can solve this, and then we'll have a live demo, and hopefully that will work. So the problem right now is, if you wanted to plan open-style glands, there's these configuration files sitting in SC glands, and in there, there is database connection information, the keystone of the service account passwords, and those are all in plain text so that the ASL config can read and make the connection. While we try to protect these things using file permissions of people who have access to the box, we cannot read these passwords, often times when your cloud is having issues, and you call the support, and the folks who actually have access to these configuration files will sometimes send these configuration files out because as a developer, you're going to troubleshoot this and say, okay, something is wrong, maybe the configuration is messed up, can you tell me what the information is? And oftentimes they send these files without being redacted, and then the whole company knows what the data production database password or use ID is. And then, attitude that is, in a large cloud deployment, sometimes changing a password across all the node requires that you are relying on the deployment which takes time to propagate and restart all the services. So what this, oh, here's a, right now, an example of what we have today, right? So in a normal cloud deployment, you have your keystone providing authentication to the various core services, Neutron Overs with Cinder and Glance, and then in each one of those services, there is always a section called the keystone off tokens that has a URL, but in there there's a password sitting there in plain text. The same goes with the database, right, that you have the MySQL connection in there, and the connection in the password in plain text, otherwise they are so config is not capable of connecting. So some of the possible solution I think that they have been talking about in community is perhaps we can just encrypt the entire pass up in configuration file and then we can decrypt it or even partially encrypt a sensitive information, like the connection password. The issue with that is when we later on want to update those passwords, we need to then find the correct encryption and decryption house the way that if we have an encrypted password in the configuration file, if we want to update it to a new one, then we have to have the correct password and using the same encryption algorithm and replace it is a hassle from the deployment standpoint. Another thing that we try is to use some kind of a CMDB configuration, a management database kind of to store all the potential in the database, but then as the database in general isn't secured, you need to do the add a security layer on the database. So what we for this talk, they propose is to use some kind of a secret management or key management store which OpenSat provides, called it Barbican. Thank you, Ten. So for this proof of concept, the whole idea is pull the password out of the configuration file and store it in our proof of concept for instance, Barbican. So it begins in also config, also config loads of configuration files. As you can see on the third bullet point there, instead of having just your plain text password, instead just have some kind of secret reference and then the key name, immediately the password's gone. All also configured will do will be to parse out that string, realize, I need to call out to the key management store and then retrieve that value. Also another idea we toured around with was using environment variables. So the project we used for this proof of concept is something called custodia, which I'll mention more in a second, but this is the general workflow between also config and custodia. Also config will parse the configuration file, read the secret string with the reference and key name. It will also pass those along when it makes this call out, authenticated with the certificate to custodia, which is a, as I will explain here, sorry, custodia is also another type of key management store. But for this proof of concept, we kind of tweaked it a bit. So according to the internet, custodia is called a PIX or a container for the host. But custodia really is an open source project created by some Red Hat developers. I think there's only really two that are actively contributing to it. But it's a tool for managing secrets. It basically works the same as Barbican. But what custodia does offer and they actively advertise is you can create your own plugins and use something else beside custodia, like Barbican or even Vault if you wanted to. Right now, when we tackled this project, when we tackled this problem, custodia did not support Barbican back end. So that was one of the things we had to overcome. So the main issue was getting it to work with Barbican is custodia is not an open stack project, it has no idea what Keystone is. So we had to go in, allow custodia to authenticate with Keystone. We also had the, so custodia kind of works like Barbican, where there's containers, but there's not container references. Whereas Barbican deals solely in container references. So that's another small tweak we had to make is pass in these container references to custodian, custodia had to recognize that. And then of course, we had to create the Barbican store plugin. Then after custodia was all done, we got custodia to authenticate with Keystone. And then custodia would actually call out to Barbican after being authenticated, and then it will actually retrieve your secret value, which is your password, and it brings it all back. And I will pass off to Sam to talk about the store back ends we tried. Okay, so Barbican is the open stack secret vault using, as Gage was talking about, the container reference and key name. So this is the one that we went with just because it's supported by open stack. Like we talked about, they are containers, kind of the overloaded term. We're pushing all of our secrets in there and retrieving them all with custodia. And then, sorry, and this is all done through the configuration files. Just wanna make it clear that we chose Barbican, but like we talked about, you can choose vault, free IPA. It's just that the tweaks that we made work best for custodia talk to Barbican. And speaking of vault, it is very similar to Barbican. It's probably a little further on in development. Hashicorp actually is the ones that developed this. It is also free as far as I know. As you see on the right, it's really easy to just fire up a dev server and start throwing in some secrets with key value pairs. And then, if you wanna know anything further, just specifically about vault versus Barbican, not necessarily the other secret management software that's out there right now, there is a talk on Thursday at 9.50 titled Comparing the Barbican and Vault Security Models. Just letting you all know about that. And so right now, I'm actually gonna jump to a demo to sort of show off the kind of stuff that we tweaked with. Everybody see that okay? Yep. So we're gonna quick just source so that we can have our admin credentials. Yes, and we're gonna jump into the Glance configuration file, which is where we put the line. So as you can see right here, we have our password directly in the configuration file. Super secure. Just like normal. And so if we quick just run and open stack, image list may take a little bit. But it'll print out your typical list. We just have one, Cero's booted up right now. So let's tweak this a little. And let's just pull out that password so that we'll comment it out. Just pretend like there's no password in there now. We'll make it fail so it doesn't look like we're just pretending. Yeah, this is essentially to show that you can't just pull the password out of the configuration files that doesn't work. And then if we call that image list again, it's gonna throw back, I believe, a 403, 503. So sadly, you can't just pull your passwords out of your configuration files and call it good. But our solution, as Gage showed on that slide is we have now, our password is now equal to this string right here is the container reference for Barber Can. And then the key name is the glance password, just the glance password, that it's gonna pull from Barber Can and use to authenticate. Yeah, your key name can be anything. It's probably recommended to not just use the glance path. All right, so if we do that, restart glance again and then run that image list command one more time. Give it a sec, voila. So this is sort of a really quick run through of our tweaks to glance, Oslo, config, Custodia, and Barber Can to sort of be able to pull your passwords out of your text files and not have to worry about people getting a hold of them through various ways. And we will jump back to Q&A. Make sure you go up to the microphones to ask. Just so everyone not here and here can hear the question. Sure, Greg. Hey, what is AT&T doing in the interim before this gets, comes to fruition? What kind of safeguards or steps are they taking to make sure that passwords don't get passed around? Thank you, I'll sit down and listen. Right now, it's difficult as one of the use cases that the folks who are troubleshooting have tendency to send those configurations. They just kind of copy it and then just send the whole thing through. In terms of that, other than informing and educating the folks who are doing dealing with these troubleshooting to carefully redacted the password, right, is really just a file permission. So right now is an education issue because these passwords are actually needed in the configuration file. We can't take them out at the moment and expect the cloud to work. So that's why this, this proof of concept was done to help south, to alleviate this problem. And also part of that is also to kind of centralize the password management because I know a lot of folks have these service accounts. I believe the last survey in the open site is the 80, something percent is still using local user, but on people who are having some kind of IDP and LDAP where your password kind of expires and things like that, changing password is a tedious process or having the ability to centralize and then kind of save up on one location, these passwords, it will, I mean, there is a benefit and gain in that, in that we got. I don't know whether that answered your question, Greg. Since you're using Barbican as the back end for Custodia, how are you protecting secrets in Keystone.com or in Barbican's configuration file? That is a very good question because part of the proof of concept in here was like, yeah, that could lead to a chicken and an eight issue. For that, that's why one of the slides, and I believe one of the architects has asked that question is that they asked to put in an environment variable. I know environment variable isn't the safest thing, but in the code, the modification to also config is that it is allowed to actually pick stuff up from the environment variable, so you can just set that environment with low Keystone and then move on because it's a one-time site. Again, there's a proof of concept, there's probably some stuff that need to be ironed out, because ideally the end goal would be to just get rid of password, like moving some kind of certificate base. I know Keystone already support tokenless authentication, but that was at the scope of this proof of concept. One thing we talked about in Custodia, so I work with the Custodia engineering team, and one thing that we have talked about is having different back ends for different classes of secrets. We from the get-go didn't look at using Barbican as a back end, but using IDM ball or Hashicourt ball or something else, and using TLS instead of Keystone, but maybe you can use that as bootstrap and still put everything else in Barbican too, so something to consider. Awesome. Hey, I was wondering if you guys had looked into the Castelon project or you're aware of it, and if you were or were not, why you would pick Custodia over Castelon? Thanks. I personally have not heard of Castelon. Yeah, I've heard of it before. I think it was more just with just Custodia, it looked something similar. It looked like something we could work with better. I honestly haven't looked too much in the Castelon. I've heard of it before. It's something we probably should look into. Yeah, it just seemed like from the slides that, I mean, it seemed like you guys kind of re-implemented what Castelon already does. I was just curious if there was some distinction I was missing. No, I think, so yeah, we just sort of picked Custodia and then like Barbican's Opus Stack supported. So mostly just keeping with the Opus Stack theme with like the options of, oh, you can use Vault, but we haven't heard of Castelon, or I haven't heard of Castelon. So we'll have to look into it after this. Thank you. Hello. Very cool. Thanks for showing this. I was not able to understand very well what is the role of Custodia on this. Cannot you use only Barbican for all of this, integratively dozzled? I've answered that question. It is similar to why we are not using Castelon. I heard of it, never really worked with it. The point was that to provide some form of abstraction in caves we later on don't want to use Barbican, right? Because for this proof of concept, we just decided to choose Barbican, but let's say if somewhere when we finally do the architecture, they say we want to use some other vendor, right, then we just want an API to be able to extract and store secrets and not tie specifically to Barbican. So we kind of, Castodia and I believe Castelon provide that abstraction layer to the underlying management system. And that's why we just provided abstraction there. I mean, otherwise we can just, out of I also can fake, call Python, Barbican client, and then just start using Barbican. But then that tightly couples the underlying key management system with whatever it is calling it, right? Okay, I see. All these changes that are required in OsloConf and in each project, are you guys putting this upstream? We're going to put on, I believe that there is talk yesterday and I also can say people are interested in writing Pluck into, to modify it, because I think in the grand secret aside, we wanted to be, they asked, based on the talk, was that they wanted to be able to store configuration in a database and to be able to extract the configurations. I mean, there's technical difficulty because, you know, configuration right now in the INI files and, and, and they wanted to like convert it to YAML. But the code will be, it gets up and it will need it to work with the Oslo, at least their initiative to kind of drive and move that upstream. I think that they just interest from folks to, to be able to modify Oslo to, to either store config from some external sources, either be barbecue and database or, or whatever people want to write drivers for. Okay, thank you. In the example that you did, you were just using the sender password. I'm glad, I'm sorry. You're fine. Just that, but it was the service password. Does it work the same for the database passwords? Yes, because what we will do, well, we didn't demonstrate the database password because we are modifying Oslo config and the init parser at that level. It will, it will, I mean, obviously the, the specification and the technical detail can be modified and change. So long it detects a certain pattern, then we will go out and actually pull the, extract the secret from the, from the secret store. So if you want to store the secret of max length, something like password or pattern, that's fine because it's not specific to, to adjust in the configuration. We are just kind of hiding away parts of whatever you want to, you know, kept secret. What are you exactly trying to protect against with this? Because, well, you will get the, you can always fetch the passwords if you get to the machine anyway. So it's mainly like leaky hard drives. The hard drive contest gets like published somewhere. Or is it actually an attacker actively getting entry to the machine or what attacks do you see it is protecting against? So I know we were like, it was on one of the slides. A specific example is when you're passing around the configuration files, say by email or however, there's always a chance that that gets intercepted. If someone gets into your machine, there's, and they're going to get to the configuration file, they can probably get to most of the stuff that we showed off. This is just sort of adding that layer of abstraction that is like an extra step that they have to take. Do you guys have any other examples? Yeah, because what happened is that as a developer, sometimes if I have access to the machine, I can let's say SU and read that file, right? And then I say, oh wait, I think something is wrong with the database. Normally I wouldn't be able to access the database, but if I can read that SQL line there, I can connect to the database and start. It is kind of just providing the extra steps. Okay, well now I would not be able to do it because, like I said, a frequent use case is that when you troubleshoot, people say the environment is down and then you're like, okay, well, can you send me the configuration file, then the ops or the support team would then just go in and just immediately grab the entire configuration file that you asked for and just send it to you for email. And then then a wide distribution list in the company, which who knows who have access to what people would do with it knowing that information. So it prevents that scenario. So how would the solution compare to token assault? So the token, this thing, because barbecue and vault actually provides also encryption on the password, right? Because we are not really calling keys though because then you have to set up the actually the certificate with within the configuration. So we really just want to provide that encryption because if you have token us off, then you will have to put all those certificates everywhere anyway. So you end up managing certificate versus password. But part of that, the reason for that is also we can centralize and manage these password and credential in one single location and kind of do both. Thank you very much.