 All right. Let's get started. Thank you everyone for joining for this one of the security track talks. We are here together to talk about secure Keystone deployment, and I'm Preeti Desai. I work for Symantec. Let's start talking I'll give you just a little bit of overview of what Symantec is doing with OpenStack. So we are building consolidated Cloud for providing infrastructure and also platform services, so that the intelligent, the application security inside applications can run on our Cloud. A little bit about myself. I started working on Norton Insight products, where I was part of reputation-based technology, which is tracking millions of files even today and rating their applications based on the file attributes like age, frequency, location, etc. Then I was working on the Symantec data analytics platform, which is a big data platform where we are hosting security intelligence metadata for protecting mobile devices and PCs both. Then I moved to get started working on and involved with OpenStack as we were forming this group within Symantec. So I work as an OpenStack engineer in our group, and I primarily work on Keystone. I'm also part of OpenStack security group. It's a very cool group and we just heard about that group in last presentation, what we are doing. I'm also working as the open-source community of practice. I'm leading this practice so that we have a strong commitment of giving back to the community. So we are hosting hackathons, meetups. We have blogs where people can share their expertise. So let's get start first before moving on. A big kudos to OpenStack security group for OpenStack security guide and OpenStack security nodes. These are wonderful and very helpful. Also, one more kudos to Sohail for contributing and leading the effort on threat analysis on Keystone. It's very helpful when we are going there and trying and running threat analysis on Keystone, all the work has been done and led by Sohail. So why is Keystone so critical? Why do we want to secure? Before going into there, let's just understand and set common ground on what is Keystone and how the authentication and authorization mechanism is implemented. So here is the sequence diagram that we are going to walk through. On the left, we have the Cloud User, Keystone, and then the three different backends. Identity, which can be SQL, LAP, Assignment, SQL backend, and Token, which we are using SQL backend. So let's just go over how the authentication piece is implemented here. User comes in and requests for a token in Keystone. Basically, he's trying to authenticate himself. He sends username and password. Now, Keystone verifies that username and password with their identity backend, and it sends the hash of the password. It's not a clear plain text password. Then, after the successful verification, it requests the metadata for the user-tenant relationship. Does this user belong to this particular tenant or project or domain or not? After the successful verification and getting the information, what kind of role, it now basically goes to token backend and says, okay, now create a token for this particular user based on the scope that he has requested. So it responds back with the token. Here on the token backend, that token is being registered so that it can be revoked when the token is expired. So now the user has basically he has authenticated himself in our Cloud and has the authentication token. Now, let's look at how the authorization piece work. Now, with this token, Cloud User goes back to any of the OpenStack service or any external service and says, okay, here is my token. I want to access these resources. Let's say I want to boot a VM on NOVA, I want to access Object Star, my objects on Swift. So now, based on the token format, OpenStack service goes back to Keystone and validates this token. Is this the token valid? Has it expired already? No. Does it allow the service usage and etc.? So now, after the successful verification, so OpenStack service has to go back to Keystone in case of UUID. If the token format is set to PKI, then it can basically validate the token itself. Now, OpenStack service after verifying and validating the token, it basically executes the request and sends back the response. Now, here we have said the common ground of understanding what the Keystone is and how the authentication and authorization piece works in Keystone. Keystone is nothing but it's basically a single point of interaction and integration within all the OpenStack services. It maintains users, roles, tenants, all the services, catalog, all the endpoints. So imagine, I mean, this service is pretty, it's very, we have to secure it. It's very important, it's very crucial. So now, what kind of cloud assets does it contain? Is there any type of attack possible on Keystone? Can it bring down the entire cloud? If Keystone is attacked, can it bring down entire cloud? Now, like I said, it is a gatekeeper for accessing your entire cloud. It basically allows you to access the OpenStack cloud. It has all these assets and also, it is vulnerable to DOS. Let's look at what was our approach, what was our strategy to identify the vulnerabilities behind our deployment of Keystone. So basically, what happened was, we engaged Global Security Office at Symantec, we call it GSO, and there was interview sessions going on, like in threat analysis, there are interviews going on. So we basically met with technical experts and also the use case experts from the Keystone perspective, and we ran threat analysis and the penetration testing. Then we used the traceability matrix for coming up with the plan. What should we do once we identify the vulnerabilities? What should our approach be to mitigate those? So we ran since all the threat modeling work was pretty much done by the OSSG group, we didn't have to spend that much effort. We came up with this particular threat model diagram of our deployment, and it was based on Stride, Robin, so he'll pretty much covered the details of threat modeling. So what kind of security deficiencies did we discover using this threat model, this data flow diagram? Let's look at the data flow diagram again, and if you notice, there are two Keystone instances. This is incorrect, we have three instances. Then we have the jump host from where Edmund can log into the Keystone instances, we have the LDAP backend for the identity, and we have something called CPE LMM instance for gathering and retrieving all the logs which are sent by Keystone and all the other services that we are using. So and the red dotted lines are basically, it's pretty much covering the management network, and outside is the VPN network. So here, if you think about it, now we found out a vulnerability where the user credentials were very easy, and it was very easy to basically hijack those credentials, because the communication between the clients and the Keystone instances was going over port number 5,000, TCP 5,000, which was not encrypted. So anybody can pretty much run a sniffing tool and hijack the credentials. Without knowing that person. Now, so this was one of the attacks. Next, so the access to Cloud Edmund privileges for almost free. Now this one, in Keystone, if you deploy Keystone using the service token, or it's sometimes called shared secret, it can pretty much act as a Cloud Edmund. It can give you the Cloud Edmund privileges. With just that token, you don't have to get yourself authenticated. You just use that token and run Edmund privileges. So we had configured that in a production environment, and in addition to, so anybody can, for example, log in to the jump host and get into the Keystone instance and get to that admin token, but it's not that easy. But both of these has to be secured. So it's not that easy, but imagine the service token. If anybody has wrote it down on posted because supposed to be secured and pretty long, so we have service token, which is pretty long and very difficult to remember. So I have a colleague, for example, he wrote it down on posted. Now anybody can get to that and run the admin privileges. So that was the another attack that we discovered. Next was the denial of service. There was CVE discovered by Sohel again. To come up with a denial of service attack in just single request. So let's say you can think about sending multiple requests and you can still have the DOS, but if you limit the number of requests per second, then you can pretty much avoid the DOS attack. But with this vulnerability, it doesn't matter. I mean, you can just have the DOS in just one single request. Then next, we found out that the Keystone.conf file, which is pretty much, which has all the good stuff. Adapt credentials, MySQL credentials. It's like a keys to the kingdom. Now, that particular file, should we keep it secured or how secure should that be? Now, next we found out about leaking the sensitive information in our log files. This was possible and I will go into detail of how, but we found out that username, passwords and tokens were getting logged in plain text in our log files. And last but not the least, the unauthorized access to the MySQL server. If somebody, if a malicious user can somehow get into the management network, he was able to get into directly into the MySQL database and run the malicious activities. And we found out many more, but just for this talk, let's go into each of these and see how did we mitigate. So, we came up with our security plan based on the type of attack and the kind of risks that it brings. We laid it down on the traceability matrix and prioritized based on the threat agents and vectors. So, we decided that, okay, let's look at these three and then see how much it can help and make our keystone deployment secure. So, starting with the very first attack that we just saw, user credentials were very easy to see and hijacked. So, just from the VPN network, anybody can run the sniffing tool and get to the request. This is just over REST API. We have username and password and it's exposed. So, we decided, okay, let's encrypt our traffic going back and forth to Keystone. And this is the approach that we followed. So, we have a user. We have a public APIs and private APIs, the load balancer. We have the hardware load balancer whips, two separate whips, one for public and one for admin. And we have three Keystone instances. The public APIs are being served on 5,000 and the admin APIs are being served on 35357. So, we have now the SSL clients and SSL servers on the hardware load balancer. When the traffic comes from a user, it's being encrypted and hardware, the load balancer decrypts that traffic. We are using the certificate, the trusted CA certificate. Now, it decrypts the traffic and then re-encrypts it for the Apache mode SSL module to be decrypted again. So, we are using two different types of certificates, the trusted CA authority, and also the self-signed. We wanted to encrypt basically the end-to-end traffic without having any, yes. Then we looked at the another attack which was the Keystone.con file, which is pretty much has all the good information. And we found out that it has read permission by everybody who can log into that server. So, we decided to secure it with creating a service account, a specific service account for Keystone and then restricting privileges for that file to just read and write by the service account, by the owner. So, restrict the ownership to the service user and then restrict to read and write by the owner. But it's just adding one more layer of security. This is better to have it read permission by everybody. It's just a little more secure. We looked at the service token that we were using. You don't have to keep your service token this long. You can pretty much use like password ABC or whatever. It's up to you. This is actually fake. I have just created it for this presentation and this is the MD5 of Keystone text itself. So, it's not that difficult to get to. So Keystone service token, it's basically used for bootstrapping the Keystone. It gives you the cloud admin privileges and if attacker gets to that token, then it pretty much can create, register any bad services and run that service against you and you would never know. So, what we decided to do is basically disable service token and this is how you disable it. Just comment out the admin token from your Confile, Keystone Confile, or be better just remove this line. Also comment out or remove the lines from the token or middleware. The above it should be suffice. So, you remove this admin token or middleware from the Keystone paste file. Now, so who is the cloud admin now? We got rid of the service token. Who is the cloud admin? Nobody. So, before even you're disabled that token be sure you have somebody to be cloud admin. So, this is how we approached to be cloud admin and we followed Florence blog. I'm sure he's not here. He's presenting on Swift at the very same time. But he has written a very nice blog on creating cloud admin and domain admin using the existing source code and you don't have to apply any patches. So, this is how we created the cloud admin. We have a separate domain dedicated just for the admins. So, it's called, for example, this is not the name again, I've just morphed it. Cloud admin domain. Then you can grant admin role on this particular domain to appropriate user or appropriate group. And update the Keystone policy.json file. There is a line, basically, there is a rule for the cloud admin in the policy.json file. It says admin domain ID, basically to place this particular ID with the actual domain ID. And this will bring you the privileges to cloud admin. Yes. What you're talking about here, you're talking about JSON, is it not going to fall? No. V3 policy cloud sample. So, we are using V3 APIs and that's what we had to copy over the policy file from the cloud sample. It exists in the same directory under Etsy Keystone. So, it's very easy to find. Now, we found out that some, there is a role admin already. Adam Young, Keystone core. Sorry for the, I have a head cold, so I sound worse than usual. There's, the roles are not supposed to be global. What you're seeing there by is admin going, do you have the admin role on anything that comes in here is a throwback to the battle days when roles were global. And now roles are scoped to something. And what the cloud admin, or excuse me, the cloud sample policy file took as the default is, for doing things that are domain level like adding new users, because you're adding a user to a domain, you should have admin on that domain. So, what she's aligning is that you want admin to be admin on the thing that you're managing and you need to have a separate policy file, the better policy file in order to have those more fine grain roles, role assignments. Clear as mud? Too easy? Okay, cool. Thank you. Thanks Adam. So, then we found out that we were leaking some of the sensitive information into our log files, for example, passwords and tokens. How was this possible? Why were we doing this? What it made passwords and tokens appear in log files? Because we were using debug mode. It's obvious. Then we had to disable the debug mode. And also there is not just the debug mode that it makes the passwords and tokens appear in log files, but there is one more. So, before going to there, if you have the debug mode on, for some reason, if you want to keep that mode on, you better upgrade to the Keystone client to this release, because the patch was applied to not log the sensitive information like passwords and tokens to the log files. And there is an open stack security note already out there. So, now if I disable debug mode, I should be pretty confident that there are no passwords or tokens logged into the log messages, but which was not the case. So, even the info level logs, they were contained, there were the tokens going out. So, again, there is a security note, OSSN, on this particular risk. And you can basically mitigate the logging of the tokens by setting the log level to one instead of info. And this can be done in the logging.conf file. So, Keystone has the, basically you can specify what is your logging configuration in Keystone, and you can specify this. Now, tokens get cached. They are cached in, so that, like Adam was saying, tokens are cached from the other open stack services. They cached it, but not the Keystone. And it's basically, there is a revocation cycle going on, and you can set the expiry time in the Keystone Conf file itself for expiry, the token validity. So, now, DOS, with the single request, we found out that it was possible based on the CVE that you can actually apply, have the DOS attack on the Keystone instance. And this is how, basically, this is the JSON payload for generating, for basically generating the token. Now, if you tweak this JSON payload, you can actually have the DOS attack on Keystone. Just repeat the methods for some, just repeat the methods, same method, and it was possible. So, now, Keystone team basically applied the patch and sanitized the methods list to remove the duplicates. So, it's not no longer a risk. And the impacted versions were mostly the 2013 versions, and the patch was applied during the Icehouse release too, and also it was backported to Havana. Havana isn't of life now, but it was backported to Havana. So, upgrade the Keystone version and use the new, any version about this. So, thank you, everyone. Any questions? Let me start off with an answer. We asked about encrypting in the Keystone.con file. I would like to go further. So, there are two resources that pretty referred to in there. The second one was the LDAP credentials, and this is the easier one to mitigate. The general approach should be that LDAP is read-only anyway, because you're using a corporate AD to read the stuff out of. The use cases where you have to have read-write access to LDAP are minuscule, and I just kind of forget that that exists, that LDAP is really for identity and it's read-only. In that case, you can do one of two things. If your corporate LDAP is globally readable anyway, just do an anonymous bind. If you drop the user ID and password out of there, then it will do anonymous bind, and there's no credential in there to attack. The second is to create a non-privileged user, and actually for free IPA, this is what we have to do, because we don't have that. So you put a user in there specifically to only be able to read, and again, eliminate credentials, and then if that gets compromised, it's not really an issue, because all they can do at that point is to read it. So first, mitigate at that level, because it should be read-only data. The sequel one's a little bit more problematic, because you just keep pushing back, and pushing back, and pushing back, and then you get to the answer of all things, security, which is Kerberos. Yeah. I would love it if I could use Kerberos credentials to talk to the database. You can if you use the Postgres backend. Now, it's not full sassel, so you can't use it to encrypt and talk, so you still have to do TLS, but if you use a Postgres backend, you could do a key tab. But again, you have to correctly protect the key tab, so it's the same set of issues. So with the Postgres backend, you can do that. Unfortunately, we don't have that for my sequel, but I don't know what we could do with the password. It probably should be in a different config file, at a minimum. That was one of the things that we were talking about is being able to separately configure a bunch of different components. I wish I had a better answer for you there. And the other thing is, you heard me, she's awesome, so I don't want to ever come across like I was typing it. This is the first time I've actually found something actually Chastise E4, and that is that the whole remove the service token from the Keystone config file. That is in the deployment guide, and that is normal deployment mechanism. And if you're not doing that, you are in a state of sin and go and ask for penance and go edit your Keystone.conf file to get rid of that. So that is baseline, okay, cool. Thanks, Adam. Any more questions? So the question is, what is the best approach for the two-factor authentication? Well, you come to Symantec. We already have two-factor authentication. No, I'm just kidding. We are actually exploring. We haven't, it's still in design phase, and we were basically trying to use something like with Radius plugin, but we haven't been able to come up with a concrete solution. But you can get back to us and we'll provide you more details, no, no. No. To rest one other thing, as a backup to that. So talking about the Port 5000 and insecure thing there. If you've run DevStack recently, you'll notice that we've changed the default deployment mechanism for Keystone from being deployed in a Python specific web server called Eventlet to being deployed behind Apache HTPD. And one of the things that this gives us is it gives us the ability to use the native support in the Apache modules for authentication. So for instance, for Kerberos, I don't wanna have to implement that in Python because doing crypto in Python is really, really slow and painful, and then there's issues with threading. So we inherit the Kerberos authentication from the modules. So that should actually be the answer for two-factor auth. That if we can get two-factor auth on the web server, then it's not really a question of Keystone doing that. Now, that's speaking strictly in terms of Keystone the server. Now you still need to find a way to make all that stuff happen at Keystone, the client and Horizon talking to Keystone. And that's always a question of things there. So it's not really an answer, even if we had two-factor auth to Keystone, that would only work if you were doing a custom script or something like that. With the methods thing there too, that was originally going to be for multi-factor. We never went in and said that all these have to pass in order to get a token, which would be pretty trivial to add. All right, thank you. Thanks, everyone. Oh, you have a question.