 So thanks everybody for coming. My name is Keith Neustadt and we're going to be talking about Keystone security Before I get started how many people are somehow involved in the security field. It's pretty good And how many people are pretty familiar with Keystone while I'm at it All right. Great. Sorry about that. So before I get started, I want to start by saying that There's a lot of really good information out there in the community on this topic And I would recommend that everybody start by reading the open stack security guide. There's really good information in there And as a matter of fact the community has already recently has also recently done a threat model of Keystone and That's out there as well. So it's pretty interesting reading. I Would also like to point out that Everybody's deployment everybody's use of Keystone is going to be a little bit different Everybody's environment is a little bit different so it can be a hard to come up with a boilerplate recipe or checklist set of rules of what you need to do to make sure that you are secure Security is really a mindset and a process and So we thought it would be interesting to share with you guys our process and the things that we've been thinking of as we at semantic Go through the process of deploying Keystone into production. So before I get started I just want to give you a some context of you know, what it is that that semantic is doing We are building A very large cloud for semantic We're building the next generation semantic cloud that all of our products and services are going to be deployed on And it's a really exciting project It's a Greenfield project all the way up from brand new data centers up to the infrastructure and platform services And we're heavily committed to OpenStack. So again multiple data centers thousands of nodes and our approach is that we're going to use OpenStack in every way that we can and In areas where we have needs that We don't find yet in OpenStack. We're going to Build in features. We might build entirely new services. We're going to contribute all that back to community So I'll give you a quick example In partnership with Mirantis, we have started a new OpenStack Service called MagnetoDB Which is a no sequel database as a service and as a matter of fact, we've got a design session tomorrow on that So if anybody's interested in that, I invite you to come and you can also talk to me after the presentation So again, I'm Keith Neustadt. I've been in security for Coming up on 15 years I Was recently the architect for the various Norton web services that includes the Norton identity provider Which is in a lot of ways very similar to Keystone These are you know fairly large-scale services. We handle billions of requests today North of a hundred million users and also north of a hundred million endpoints And being semantic we are a really attractive target people like to go after us and so we are basically constantly under attack and sometimes it's a low level of attack and Sometimes those attacks kind of ramp up, but they're always happening and we definitely understand Security from a practical production sense So I've now moved on to again to to cloud platform engineering building the next-generation semantic cloud But this presentation just isn't just about me. There's a larger team at semantic that worked on this I can see some of those folks in this room So if you see people wearing a semantic shirt like this one in the hallway Feel free to pull them aside and say hi asking any questions you've got Now I know that most of you guys have a pretty good sense of what Keystone is But to kick us off. I really wanted to you know, just start with a really brief Overview of what it is and how it works Keystone is open stacks identity provider This is the thing that allows you to log in that identifies your users and here's how it works The user goes to Keystone and authenticates using some form of credentials like a username and a password Keystone then responds with some sort of an identity token that says that I keystone vouch for this person They are who they say they are And now that user can go and take that identity token to any other service in open stack and Use it to authorize themselves without that service service might Validate the token itself Using something like PKI or it might pass that token through some back channel to Keystone to get validated For example with you ID tokens But either way this is this is really good stuff This is a it's a very simple pattern, but it's a pretty powerful pattern It gives you a lot of things for example It gives you one point for authentication for all of your services. So your users Always know where to go. They always have the same rest interface They always know how to authenticate and when they authenticate once they're authenticated to all of the services inside of open stack There's a common API layer that you to put on top of all of your authentication protocols It doesn't matter if you're using LDAP or MySQL or something else And here's I think the most important thing is that it reduces the exposure of your credentials Right because rather than passing your you know users passing your their credentials to every single service They only pass them to Keystone Keystone is the only service that get to see the credentials We all know that the credentials are very important gives us a place to focus our efforts in securing those credentials And of course Keystone is a lot more than that, but for the purpose of this presentation I'm gonna I'm gonna focus on this flow. So here we are all gathered together to talk about Open stack and security Why are we focusing on Keystone? Why is Keystone so so critical? and the answer comes from Basically looking at the assets and if we look at open stack and we look at the assets we see things like passwords keys certificates tokens and Keystone is basically the gatekeeper for open stack and If you can get through that gate you have a lot of access inside of open stack and these things we're talking about They're keys to the gate. They're basically keys to the kingdom. So if these things are stolen, that's a big problem And also if somebody wanted to DOS your open stack Infrastructure this would be a good place to start right because if you can shut the gate Then you've shut down access to a lot of open stack So for semantic as we determine that Keystone is particularly important We start to look at how we're gonna secure it and we have a tendency to look at things in layers All right So for example, we'll start by looking at our process for securing Keystone And then we'll start to look at our environment where we're gonna deploy and run it And then finally we look at the application itself and figure out how are we going to secure that application? How are we gonna secure Keystone? So I'm gonna use this as kind of a framework to go through the presentation and We'll start with process now security to a large extent is about learning to ask the right questions and so I thought it would be fun to share some of the questions that we're asking ourselves as we're Going through this process of securing Keystone and the first question you tend to ask yourself when securing anything is What is it that I'm trying to protect? What are my assets? Where am I deploying? And where am I likely to be attacked? And we use the security development cycle life cycle as our security process There's others out there. That's the one that we've chosen for us and that's a pretty broad topic I'm not gonna talk about all that but I do want to talk a little bit about one aspect of it, which is threat modeling So who here has done threat modeling has had experience with threat modeling Okay a few So threat modeling is cool. It's really cool. It's a lot of fun actually it Takes us away from that typical perspective of how do I secure what I have and it flips it around and allows us to take the perspective of How would I attack what I have if I wanted to do that? And this is a really important exercise because the fact is that if you don't do it your attackers are going to anyway, right? so you might as well know what they're thinking and We're right now going through the process of threat modeling keystone and open stack as we're going to deploy it for ourselves and as I mentioned the community is already working on some of those activities and we're actually getting involved in those Threat modeling activities of the community too But I just wanted to pick just one just to give you a sense of how this works one little snippet of a potential Keystone threat model So up here in the top you've got a user who's logging in they're passing their credentials into a trust zone For a keystone and eventually those credentials find their way to the LDAP driver And the LDAP driver passes those credentials to the LDAP server LDAP server authenticates them sends back the user's ID maybe some roles other information It's a pretty straightforward interaction and now that we've got that interaction understood mapped out We can apply stride to it And stride is an acronym for different categories of attacks that somebody might launch against you So let's take an example and see how this works What would happen if somebody tried to spoof the LDAP server? Maybe they got into the network some DNS attack and they were able to replace our DNR LDAP server with a fake one of their own and what could they do? Well, they can see all the user credentials every user that logs in they can see that that username and password All right, and those are the keys to the gate. It's the keys to the kingdom, right? They could also elevate privileges they could look for certain users and they could pass back additional roles if they saw them All right, so this would be a very powerful attack that we find from this very simple Interaction so we ask ourselves now that we understand the attack and we understand what's at stake How do we mitigate it and the answer when it comes to spoofing usually comes down to some kind of authentication? How does the LDAP client? authenticate the LDAP server and you know, that's likely maybe through TLS and server sites to certificates for example Am I running what I think I'm running? Did I get the right distros and even if I got the right distros? Is it the same thing that I actually deployed into into into production or did somebody inject something malicious into it? And am I running the right patches? So this is pretty standard standard stuff I don't want to spend a whole lot of time talking about this, but I do want to call out that Since this process of getting the distros building the distros deploying the distros since it's not production Sometimes we have a tendency to overlook it And I'll give you an example is a story that a friend of mine told me recently about a previous job previous place where he worked and at this place They had their CICD server as a machine sitting underneath somebody's desk and They realized that they had a physical security issue with the server because Every once in a while somebody would sit down at that desk and they would cross their legs and they would kick that big Power button on the front of the machine and they would basically DOS themselves unintentionally So in realizing that they had a physical security issue They went to mitigate it. They took a Snapple cap and they taped it on top of the power button and Now no more DOS attack So again, I have I think we have a tendency to think since it's not production I don't really have to worry about it But if that machine can basically be shut down if you've lost physical security of that machine What else is going on that machine? What else could somebody do on that machine? You know, maybe what I'm deploying a production isn't really what I think it is. Let's go on to environment Is my system hardened against attacks? Once I've deployed could somebody change it What could somebody steal from it? And after it's happened, do I know what happened? Can I figure out what happened after the fact? So this is really about compliance and we're talking about, you know, two sides of the same coin with hardening and auditing All right, and you're looking at how do you harden the important config files log files the ports executables your environment itself So that they can't be changed Or at least that they can't be changed or read by anyone who shouldn't be And then since all these things are going to be changed eventually, you know when it's appropriate How do you audit those events? How do you know who changed it when they changed it what they changed? So In terms of Keystone I'll go back to it every keystone deployment is a little bit different And so the process of hardening and auditing it, you know exactly what it is that you need to harden It's gonna be different for everybody, but what I'd recommend is that everybody start with the keystone.com file open it up and read it from top to bottom and I think the first thing that you'll find is that there's really good stuff in the keystone.com file If you wanted to steal something that would be the first place that you would go because there's admin passwords There's keys. There's ports There's locations to other files like your cert files your CA files your key files are all sorts of really good stuff in there So you certainly want to make sure that that is hardened and audited But you can also use it as a trail You can just follow the trail from the keystone.com because it tells you where all the other assets Related to Keystone are on your system and while you're at it You might want to think about changing those default locations because you don't want to make it too easy for someone who manages to Get on the keystone system to find all that stuff change the default ports change the default file locations in Terms of compliance There's a lot of tools out there. It's meant to happen to make one And so that's what we're using is a data center security It kind of automatically goes out there and makes sure that you've hardened Linux and they're also working on Open stack policy. So also goes out there and make sure that you've hardened Open stack, but you don't have to use our tools. There's other tools out there A lot of people use SE Linux these tripwire things like that So just you know pick a tool and go with it is my data secure while it's in motion What high-value assets am I sending out in the wire and what would happen will be the repercussions if somebody managed to intercept those and I think you know, this is an important one. How much of my environment do I really trust? I think we have a tendency to think of our environments as everything that's inside is Trusted and good and everything that's outside is untrusted and we don't know The reality is the way these attacks happen is they get a tiny little foothold into your environment And then from there they slowly drill their way in so you have to ask yourself How much do you want to or do you even need to trust some of these internal parts of your environment? So now when we talk about data security motion, I think we probably all agree You know we're talking about SSL right and we I think we hopefully we all agree that cells important We should be using it, but I want to point out that it's not just about SSL But how you use it? So let's go through an example Up here at the top. We've got a client that's logging in they're passing credentials down to Keystone and As you can see those credentials are completely unprotected through the internet So we're going to add SSL and now that path is good and they're protected on that path But again, we're not talking about just attack vectors on the public internet There are also attack vectors internally what happens if somebody manages to Compromise a process that's running on one of your open stack servers Or what if you happen to have deployed something that's not open stack? Maybe it's a monitoring application Or maybe there's some sort of web service that you deployed either intentionally or maybe unintentionally on the same subnet But you can see that After that SSL termination those credentials are in the clear and anybody who's on that subnet can see those credentials And those credentials are the keys to the kingdom, right? So the way we're dealing with this it's semantic as we're going to push those as cell Endpoints down to the open stack servers, and then we know that credentials are always alright Very small round of applause. So So now we know that all the credentials are always in every case. They're encrypted. They're never on the wire without being encrypted But there's cost to this right and you have to kind of assess your own risk and your own cost and figure out What you what you need to do? Now you've got issues and you know things like deployment of certs You know deployment of these endpoints Maybe it affects your ability to use hardware low balancing and SSL termination So you have to think about what works for you But at the very least think about the any clear text network that your keystone server is on and think about Segregating it now onto keystone Well, I know when I'm under attack and you got to assume that it's gonna happen Who's attacking me? How are they attacking me? What are they after? And do I know how to stop them? Do I know that I can stop them? So there's two sides of this point also there's prevention and there's forensic How do I how resilient is my environment in production against these kinds of attacks? And then after the attack happens do I have the information I need to figure out what happened and maybe what was lost? So I know what I need to do to remediate Now again working on the Norton identity provider We were always under attack and we saw the kinds of attacks that that tend to come to us And for the most part with an identity provider, you're gonna see a lot of brute force attacks They're looking for those passwords and they're gonna try to use your service to try to guess those passwords running through dictionaries things like that So when looking at how you foil a Bruce for a brute force attack you need to slow it down So you need to look at things like rate limit because if you can prevent these attacks from Making thousands of requests per second against your identity service You're basically going to disable the attack. It's the only way they work The other thing you might want to ask is Is this a human user? This is a real person that's that's making these requests or is it an automated process because again These brute force attacks don't work if they can't use automation So there's certain challenges you could send back like CAPTCHA You need the ability to be able to black the blacklist bad IPs, you know, maybe You've determined that traffic you're seeing from a particular IP is bad You know blacklist or maybe you get a feed from somewhere, but you need to be able to block certain IPs and Then you know the other thing that that we started to do is sort of look for anomalous patterns All right anomalous patterns and logging in but also maybe a non anomalous patterns and how users use your Your applications so maybe you're seeing a lot of users from the same geographic area Log in and follow a particular and unusual pattern through the other services Maybe that's an indication that again you've got somebody coming from a particular area. That's that means you harm And then after the attack you need to make sure that you've got the data that you need to figure out what happened And that's a lot of data It's things like users that logged in users that failed to log in Maybe hashes of the tokens you gave out and so you can track them through other systems. You want the source IP addresses? You need to pull all those logs into a central location where you can do some heavy-duty analytics and correlation And then finally you need to realize that all this information that you've collected that has value to right you're tracking things like users Patterns for the day, maybe you can figure out when they get up when they go to bed You certainly know where they are geographically. So you've created this new very highly Highly valuable asset that somebody might want to steal and so it's something new that you need to figure out how you're gonna protect So keystone includes a lot of you know, really good logging. We're looking into that But if there's other information that you need to collect and certainly to be able to to support Things like rate limiting and challenges There's a couple of options. You can build a filter for keystone You can add a proxy. There's some, you know, load balancers hardware load balancers that offer layer seven Rate limiting so you can look at other options like that So is my user validation effective? Is it working? Am I really determining that my users are who they say they are or passwords enough and If they're not enough, what other kinds of authentication should I support and how should I implement it? So we're really we're talking about two fact two factor off here Everybody knows how to factor off works, right? I log in I have to provide a one-time password That gets texted to my phone or I've got some sort of a fob or something like that So if you don't have my phone, but you have my password, you still can't log in So two factor off for semantic is really important. We're gonna have two factor off in production so we start to go out and we look at options for how we support that with keystone an open stack and We see a couple of common deployment scenarios for authentication For the most part people either use the LDAP driver to hook up to an LDAP server Or use a sequel driver to hook up to a mysql database to do the authentication and From what we've seen we don't see a solution that works for us for two factor And so when we look at this diagram we think about how would we want it to work? Really what we ideally like is a radius driver Radius is a low-level authentication protocol that's supported by a lot of different authentication servers that are out there And so once you have this radius driver, you've got choices For what you're gonna use for your back-end authentication and use RSA Semantic has a solution. There's all sorts of open source authentication servers that support radius Now once you've got that choice you can go out there and you can pick something that supports two factor And we happen to have something but again, there's other options that are out there So this is this is what ideally we would like and actually in talking informally to some other people in the community We kind of get the sense that this driver might have been implemented by others before but just not shared not open source So what we're gonna do is we're gonna look for an implementation that seems to work Or if it's not out there we'll partner or we'll build it ourselves But whatever we'll do whatever we do we'll share So how do my services and scripts? authenticate themselves I think that a Lot of the time when we think about authentication We think about the case where the user is sitting down at the computer and they're typing in their password All right, and we know how the credentials are secured, right? They've got them up here in their head and hopefully they're not writing them down on the sticky note Or they're telling their friends or whatever it is So it's pretty straightforward, but there are there are times when Machines need to be able to authenticate themselves when there isn't a person around to help them do it For example open-stack services authenticate themselves with Keystone and There are clients out there that might need to authenticate themselves with Keystone All right, maybe it's a cron job that does a health check or maybe it's a deployment All right that needs to authenticate itself so it can run some sort of script And so in looking at these kinds of user scenarios We think about a couple of things So first of all since we're assuming since there's no you know human That's got the credentials in their head that there's some sort of credential that's being cashed on those machines So how do you secure those particularly given the fact that some of these machines? They're not owned by us. They're owned by our customers, you know on their phones or on their laptops and How do you? How do you how do you limit the damage that can be done if they're stolen? like how do you limit the scope of access that these things have or the amount of time they might live and and Finally at the management of these I like if you're talking about things like keys Where do you store all those keys at a central place so that they're protected and how do you distribute them in a secure way? So in a sense what we're really talking about is we're talking about delegation I mean I'm I've got a certain amount of access and I want to delegate some small portion of that access for some Limited amount of time to some machine so it can act on my behalf in a limited way So we looked at some of the different potential solutions that's available with Keystone We started with cash passwords, which I think is where most people start and I think we know why that's a problem We looked at the EC2 key, which is a little bit better than cash passwords But just a little bit because if you steal the EC2 key, you've got all the same access right You almost might as well have the password We looked at trusts which is a lot closer It does a lot of the things that we're looking for you can log into Keystone you can ask for delegation To give to a certain user with a certain scope a certain amount of time and now that other user can act on your behalf But where we get stuck is that I can only delegate to another user Which means there has to be a user that's logged into these other machines Which means that these machines need to have cash credentials and those cash credentials also have a scope Then there's other solutions like keys and certificates and now you get into Manageability issues, right? So we're this is something that we're still looking at about how we want to go about this And we'd be interested in hearing if there's other people out there They're asking these same questions of themselves and what you guys are thinking and then finally I want to just make a comment about standards So and the Norton identity provider we started in a similar place where you know Keystone did which is it's a REST API Something that's proprietary that you can use to log in for single sign-on get tokens and We eventually took a step back and we decided that it was important for us to support industry standard protocols And so we did that we made that migration and I'll tell you it was hard. It was not easy Because we had lots of clients hundreds of you know hundreds of millions of clients out there that used our old protocol And we were moving them to something new and something web-based. So it's not easy But we decided that it was important because it had specific benefits So let's say that Keystone supported identity pro identity standards. You could log in single sign-on to Not just open stack, but to maybe Google apps maybe sales force so you get broader single sign-on Yet some improved integration both when you're bringing new services into your open stack environment But also if you wanted to integrate external identity providers like Like Google or Microsoft There's actually a little bit more control over the credentials in these protocols and You also get a unified not just API but a used unified user experience for logging in so your users know what the login page looks like they're familiar with the Graphics and Whoever's running that identity provider has a lot of ability to add new features to it in a multi-factor retina scan Security questions identity proofing things like that. So there's a lot of benefits We're not the only ones that brought this up there's a lot of discussion going on in the community There's blueprints out there So I know it's being talked about we just wanted to throw that out there It's something we've got experience with and we'll look to participate when the time comes So a few parting thoughts Credentials are the keys to the kingdom right there Among the most important things that you've got so you got to make sure that you protect them threat model your deployment Watch the credentials whether their keys or passwords or whatever it is watch them work their way through the system Look at every interaction think about how they could be attacked and how you can mitigate it Securing your use of keystone is going to be an ongoing process All right, everybody's deployment is a little bit different everybody's got different considerations different production different tools different services So you want to do that threat model and then every once in a while when you change your deployment you add some services You maybe upgrade keystone dust off that old Threat model update it look at the new interactions And then finally share security is a team effort and This is the team right it's the community so ask yourselves your own questions Share with us the questions you're asking the things that you're thinking about the solutions that you're deciding on Maybe the solutions that you implement And again, feel free to reach out to me or others in the semantic team That's it Any questions? Hi, I'm Adam young with red hat keystone core And I just want to say this fantastic presentation really really got a lot out of it and I took notes One thing I'd like to I Think you might have oversimplified on the statement where you said, you know change the standard locations and stuff like that I'd like to put a caveat on that which is Use SE Linux and if you use SE Linux don't change the standard locations because what's gonna end up happening is you're gonna have to manage your own policy so I think what you said is in a vacuum a really good piece of advice, but understanding that When you deploy keystone Especially under htbd, which is the the approach that I highly recommend htbd already has an sd Linux policy for Securing port 443 you don't want to put that on a non-standard port and then have to deal with telling htbd How to deal you don't want to do sd Linux almost your radius commit some some time to it But I think the the the the concept behind the advice is really good You With when you start talking about radius and the the ability to externalize the authentication One of the things I was wondering if you looked into it, you know again with htbd and the ability to use the external Off-plugins if perhaps that that might come closer to what you're looking for that you let htbd Say who you are and then keystone's job becomes more what you can do. Have did you investigate like along those lines? Yeah, we so our off team You know separate are you the basically the the verisign off team did take a look at that? I'd like to talk to you about it a little bit more than let's let's let's talk about it And I actually let me just point out that Adam has a killer blog out there So I was talking about About resources that are out there Adam's blog is one of them It's it's not just good information is actually a really good read So maybe we'll talk and that'll become a post and to do address your last point about share It would be great if we could see Symantec You know on the code reviews on the design reviews up front for the stuff that we're talking through if you guys Come on over the dev lounge participate there I'm assuming you you're planning on it But I'm gonna get the shout out for doing it now and that would be great. Sounds great I have a question about what network is used data marriage data network or Management network is there a use case where you can use keystone on the data network. I had another question about Multi-region support in keystone. How does this? How does that work? What was the second question multi-region support? So How do you how how does keystone support multi-region deployment? See I'm not sure about that Maybe Adam can respond to that Well, the question was about multi-region support in keystone. It sounds like it sounds like they're still working on it. Okay? All right, let me try to just to summarize that it sounds like there's been some work done at the API layer But there's still underlying functionality that's still being worked on for that Okay, try now. So when you say multi-region support, I'm assuming you're talking about the Ability to annotate on an endpoint which region it's in is that what you mean? So if your services are distributed across multiple regions, right then does the key store? Are the are the authentication tokens sort of replicated does that? It was across multiple keystones, right? No, they're not and our goal in one of our design summit sessions here is to get rid of Having to persist the tokens altogether. We want to move to ephemeral tokens with pki There really is no need to go back to keystone to to validate them So if we can get it such that A two different keystone servers Can issue tokens independently and without sharing keys because obviously that's a bad practice stuff like that Then what should be able to happen is it doesn't matter which one you get it issued from Everybody should be able to authenticate You know validate the key the the tokens. Okay. Is that what you're looking for? It's on my roadmap. Okay So another so sort of my first question I wanted to sort of talk about the use cases for using key store or the data network is that I Mean typically when you do this service level token request or authentication You're using the management network. Is that is that right and is other use cases where? You're building services on a stack and you want to make use of key store. Is that does that does that work? We're talking about you have Other services outside of open stack that you want to provide authentication to using keystone. That's right. Yeah, right? Yeah, I mean keystone is a basic. It's a it's a it's got its own identity provider pattern that you can plug Various things into it. As a matter of fact, we've done it ourselves. We've added services to open stack And used keystone is the way to log into them and you know this, you know particularly this new one Magneto DB that we're working on It's not yet open stack core, but you plugs in just like any other open stack service does Would you agree with that? No And the reason I say no is not that you can't do that, but I kind of feel like you should One of the patterns that we're seeing time and time again is that Keystone's Roll has has shifted over time originally. It was a central point for Passwords so you didn't have to replicate them across all the different open stack Services and what we've seen is people already have their own identity providers People already have LDAP or people already have this way of ending if you're any non-trivial organization and We're trying to get out of the business of managing passwords and trying to make it so that if you have some other service out there Use a standard like the list that you had up there like Sam L and in curb Roseburg, you know the enterprise X509 client service, whatever it is a cryptographically secure way of authenticating it and think of keystones Reason for being as an additional layer of authorization on top of a secure authentication layer Okay, so you should not have to go in And make your application keystone aware Unless what you want to do is tie in with the role-based access control view of the world the keystone does in and say that the Roles that I'm going to use are going to be consistent between this application and say Nova I understand what you're saying. I think it seems like maybe the division is Are you providing authentication to something new that you're providing to your customers as a service? If it's a service similar to you know the way you might provide Nova You might provide Swift or something like that then certainly that makes sense But if this is you know something for your own back-end authentication where you could just as easily use something else That would make more sense Use this use the standards and the cryptographically secure off authentication Mechanisms where possible. Okay. Thank you. Great. Thanks a lot