 Thank you for making the time to come to this guy. Just ask quickly, who was in my crazy PKI talk earlier today? Brilliant, OK. Maybe a small amount of slide reuse. I'll try and get through it as quickly as possible. So my name's Robert Clark. I'm the lead security architect for HP. I deal with all our security in the cloud. Charlie is from Ericsson, and I'll let him introduce himself a little bit more in a minute. So I'm going to take a minute or two to talk to you very quickly about the OpenStat Security Group, which we're both members of. The OpenStat Security Group has been around for two or three years now. We're responsible for a whole bunch of different security initiatives inside of OpenStat. If you care about security in OpenStat, if you really want to contribute to security in OpenStat, or if you just want a really good place to go and complain about the things that are horribly broken in OpenStat, then you need to come speak to us. So the security group's been responsible for a number of things. We've written a security guide. We produce security notes, which are bits of security guidance around how best to deploy OpenStat in a secure way, and will help steer you away from making bad configuration choices. We do threat analysis, which is the subject of this talk. I'm going to let you all speak mainly about that. I'm going to introduce some security concepts and some of the ways in which, as industry best practice, we can look at OpenStat code. So probably everybody here, probably not that many people that do threat analysis every day. So just going to touch on some of the principles there. Look at a couple of vulnerabilities and how the systems we use to appraise them would rank them in terms of their danger and in terms of the effect that they would have. And then I'm going to pass a mic over and we'll run through some of the threat analysis work that's been done already. So threat analysis really comes down to the management of three different properties around any type of data or any type of network connection, any type of interface, which is the confidentiality, integrity, and availability of your data. So whenever developers are building systems, we'd love to see these three things being considered. A lot of the time availability actually does get considered because everybody has to deal with large-scale problems in OpenStat. But often confidentiality and integrity aren't. When these things are missed, it becomes quite difficult to add them in later on. In a threat analysis context, CIA is a little bit broad. So when we're trying to classify different types of attacks that we might see on a particular interface, we use a system called Stride. We didn't invent this as far as I can tell. It was Microsoft in the secure development lifecycle. I'm happy to be corrected on that. But either way, it's been around for a long time. A lot of people use it. Stride consists of a number of different properties. And you apply consideration for each of these whenever you're building something. So spoofing, can an identity be changed? Can an identity be stolen or masqueraded? Good example of this. I have here a phone dialer where if you have a telephone number for a celebrity, you can dial into their voicemail because they use the telephone number that's dialing in as their authentication. And it turns out to be trivial to spoof. These are the sorts of things we need to think about when we're designing all systems, not just open stack everything that you're going to build. Tampering with data. So your typical man in the middle type attacks can fit in here. Typical sort of coffee shop attacks where you're messing around with people's data. Similarly, this is what a lot of secure transport tries to protect against. So if you're looking at building TLS, a lot of the time you're looking at building protections to people changing or observing data on the wire. Repudiation is important. So who changed what, when, and why? You need to know when changes happened. You need to know who caused them to happen and why. Transaction information, you know, there is work going through open stack right now. But it's still not necessarily the easiest thing to map an API call that somebody made on a public interface to a file change that happened somewhere in Cinder or something like that. Information disclosure. So this is what the system has caused to disclose information that it shouldn't otherwise do in any number of ways. This can be as simple as misconfigured horizon interfaces. So when you've left configuration panels open and things like that through to, or Heartbleed, which we'll talk about in a minute, is actually an information disclosure vulnerability. It's just a really, really horrible one. Elevation of privilege. So any mechanism by which you can gain a higher level of privilege in the system than you were supposed to have by the designers. In the case of open stack, this could be a failure in an API node to correctly evaluate you and allow you to execute things on the node that you shouldn't be able to. In a hypervisor context, this could be a hypervisor breakout where you've gone from having a level of access on a virtual machine to having a level of access on a compute host which you were never supposed to have. So again, stride. One of the things you have to consider along with stride is exactly who's trying to attack your data. Now, this lovely diagram is from the open stack security guide, which again, I encourage everybody to go and have a look at, find all the problems, and email me. And then I'll pass them all on to Brian, and Brian will get them all fixed. Basically, when you're designing a system, you have to consider who's going to try and attack it. And everybody's familiar with the idea of a script kitty. Someone that just download those tools and runs them. Unfortunately, the reality is now that those tools are actually incredibly powerful. So you really have to worry quite a lot about the person and even on the bottom rung. Above that, you have motivated individuals. These are people that understand what they're doing, maybe capable of building some of their own tooling. Above that, highly capable groups of individuals. These are your typical sort of hacktivist types, people with an agenda who will come after you with some real force. Beyond that, you really get into serious organized crime and intelligence services, both foreign and domestic. The line between those two used to be quite broad, and now it's very much less. So serious organized crime. We're really talking about a lot of Eastern mafias and how they're deploying malware into the world and where they're trying to gain point of presence and leverage. So we've spoken a little bit about the different types of interface and the different bits of systems that you need to evaluate using stride. When you get to a vulnerability or when you discover a vulnerability hopefully before somebody else does, you need a way of actually establishing how important that vulnerability is. How much do you care about it? Is it going to be fixed in your next release cycle, or do you need to go and unplug the internet immediately? So DRED is a simple scoring system. Damage potential, reproducibility, exploitability, affected users, and discoverability. Doesn't matter that much. High number matters a lot. So what I'm going to do is talk to you a little bit about a couple of vulnerabilities that you will be aware of. I've chosen three fairly high-criticality ones, so I'm just going to quickly run through them. So Heartbleed was affected by Heartbleed. I assume everybody here knows about Heartbleed. Cool, OK. So a few people. So Heartbleed was a vulnerability in OpenSSL to do with the heartbeat plugin whereby you could instruct the server to give you back more data than it meant to. The result was that you were getting memory that belonged to the web processor was running, which meant that you could receive all bits of interesting information that would run in a web server. So these bits of information could be private keys that were used for the certificates. Could be session information from people logging in, or depending on your authentication scheme, you could even just grab people's passwords out of memory. This was really, really bad. So it was actually only an information disclosure vulnerability. So on stride, it comes up under there. But also, because of how trivially you could get these pieces of information, there's a good case to be made that we're actually be a spoofing and elevation of privileged vulnerability as well. Now, the dread score comes out pretty high. It's very easy to find the vulnerability. It's very easy to exploit it and happened on a pretty reliable basis. So it comes out with a score of 4.1. 4.1 is a bad score. Anything red is bad. Anything above 4 is very, very bad. So Heartbleed caused a lot of headaches for a lot of people. A lot of people had to go and replace all their certificates. You have to upgrade their entire real estates. People had to ask themselves some very hard questions about what material they thought would be lost from their systems. It was a very bad vulnerability for a lot of people. So I presume everyone here heard about shell shock. I love how vulnerabilities all have cool names now. OK, so shell shock was an interesting one, and that split the community a little bit. So this is really going to be my interpretation of how important shell shock was. Shell shock, in broad terms, allowed you to define inside environment variables functions that would be executed when the environment was evaluated. What that meant was that you could do some very interesting things. Some of the first examples were that people could abuse CGI servers. Now, if people are doing CGI pass-through straight to shells, then they probably deserve a little bit of pain. But there were other interesting cases that came up. I mean, this was a really insidious one in reality. Like, DHCP clients would get some information back from a DHCP server, which isn't authenticated in any way. And they would put that into environment variables when they were calling other parts on the system to set up a system when it first boots. That was a really interesting way to compromise an entire data center with one bad machine. So this actually gets a very high dread score. And the reason I think it gets a high dread score is, well, the reasons are outlined. But the reason I think it ended up being slightly more important than Heart Lead is because it allowed you to subvert the system itself. Heart Lead was terrible, but all Heart Lead allowed you to recover credentials and then interact with the system in the way it was designed to be interacted with. If you grabbed credentials for a root user, that's really bad. But you're still interacting with the system by logging in as root and doing whatever it is that the system allows that user to do. Shell Shock allowed you to completely subvert running systems in really dangerous ways. And it was really, really hard to work out what parts of your real estate were affected. You basically had to assume that everything that had bash installed could be compromised. Who here knows about XSA 108? Yeah, I think a few more people than raising their hands, perhaps. Who heard about Rackspace and AWS having to reboot large portions of their clouds a little while ago? OK, that was because of this vulnerability. It was very bad. It allowed reading from memory from one guest to another and could cause enter crash into all sorts of horribleness. These vulnerabilities are really, really bad for us. I'm only bringing these out to demonstrate how we apply stride and dread primarily. Cheryl's going to talk a lot more about different parts of OpenStack and how they fit together. I just picked some exciting vulnerabilities from the headlines recently to illustrate it. It affected a huge number of customers. This is, as a cloud provider, this is your worst nightmare. As a cloud provider, this is my worst nightmare. We weren't affected by this because we run KVM. I think everybody that runs a Zen-based cloud was probably affected in some way. We did a talk in Hong Kong on mechanisms you can use to try and contain hypervisor breakouts and control what people can do on the system. If you're interested in that stuff, I can send you a link to it. But fundamentally, this is a really, really bad problem for cloud providers of all scales. We need to continue to work upstream to improve our options with hypervisor and improve some of the security characteristics that they have by continuing to do threat analysis like this. I just wanted to run through those, explain a little bit of stride and dread and how they can apply to different things. Encourage you all, if you're involved with designing stuff, if you're at the product level, consider confidentiality and integrity in your systems as well as availability, how people are building things, what they're building them for, what sort of data is going to be stored inside it because it is much easier to design this stuff in up front than to have to try and retrofit it afterwards. With that, I'm going to hand over to Shole who's done some amazing work on the threat analysis, all the threat analysis work he's led. I've been on the sidelines kind of cheerleading with my OSSG flags, but he's been leading all this stuff and if you want to get involved, he's the guy to talk to you. So I'll pass over now. Thank you. So we have started threat analysis work almost one years earlier when we see that OpenState is becoming one of the most popular platform to manage your cloud. And we see that new deployers, developers are coming up and it's hard to know what's going on inside each of the OpenState projects. So threat analysis is one tool that can provide any deployers the tool to understand what are the security issues from a big picture point of view. To go, before going to the threat analysis, we will first use the same categorization just introduced by Robert and use it against historical security tech bugs, including the OSSG which has been released against a target OpenState project to understand that what internal components of a target project can be affected by the weakness, security weakness, and what if there is some kind of pattern exist on those threats. So for our analysis, we have taken Keystone, that is the OpenState identity API, and Keystone middleware, which in conjunction with Keystone actually provides identity and access management solution for the whole OpenState or different OpenState projects. So let's look at this. So when you see Keystone, Keystone is not just a one monolithic box, it's actually a combination of different services inside. And as you can see here, there are many components. So we picked up the major services from the Keystone, such as the identity assembly service token service and to end controller and so on. And for this analysis, we have taken 63 published already reported security bugs and tried to map which internal components are prone to more security weakness. So as you can see here, some of the red colors on our more prone to security weakness compared to the blue and the green one. The same thing can be seen in using some bar chart where we can see there are some clear cases such as identity and assignment service for token controller, token service. So some cases get more security weakness and due to the security next, some attacks and threats compared to others. Now what are the reasons for that? That's another investigation part, like why these things happens. Some of the components get more security tech bugs compared to some others. Maybe because they are introducing new features because of that but that's another angle of the analysis which we could do but we haven't done yet. From another point of view is from all these reported security bugs, what are the consequences from these reported security bugs? Now as you can see here, most of them lead to spoofing of your identity. That means that someone else can pretend as who you are. So which is actually quite serious thing. So most of the security bugs then lead to spoofing identity and then the next one is related to information disclosure and the third one is elevation of privileges. We also try to find out pattern from the historical data. So from this historical data we find two major pattern of weakness. One come from sensitive.exposure in logging or output. So we have seen a whole lot of OSSA and OSSA and security bugs related to such as your token is written in the log file, your password is written in the log file or someone is getting admin token using endpoint like endpoint URL creating. So these sorts of weakness come because of input output filtering weakness in the project. Then the second class of pattern we see is related to revoked or expired token can be used as a valid token. So this is due to the fact that recently there are a lot of additions such as adding of a domain as an addition of trust. So such as like when you invalidate a domain, you invalidate a tenant, invalidated a user or user changes password, you remove the trust. So in all those cases, tokens used to be revoked and there are some mismatch happens and due to those mismatches, we see that this security properties that is revoked token is still used as a valid token. So this kind of security weakness come through. So from this historical analysis, we see that most of the threats are coming or threats consequences are related to spoofing of identity and information disclosure and then the second category of things that from here we see some patterns are out there. The most of the patterns are input, output, filtering weakness and access control weakness. We have also seen that from which sources this weakness are generated. Is it because of some design fault or is it because of implementation weakness or is it because some crypto error? In most cases, it is because of implementation failure. That's why these threats are generated. There are the second category is the design weakness and the last one is the crypto error that comes there. Now this is all related to based on historical analysis. But from historical analysis, we can get a view of where we can put more effort in future and what type of things we should look in future security weaknesses there. But to get an overall idea, we need a comprehensive understanding of the system and threat modeling can provide us a comprehensive understanding of the system. So when you say threat modeling, what does it mean? A threat is something which violates your security properties, so that is a threat. So what does it mean is if you have a system or if an application where, say, you store a file and if someone can break that property, someone like some user can access another user's file or he can store file in another directory, then it is breaking the system properties. So it creates a threat. So in threat modeling, in a systematic approach, we try to identify the threats to the whole system, identify attacks or successful attacks which can cause those threats. We identify the weakness related to the attacks, like which the attacker is actually exploiting. We identify the weakness and also we identify the assets related to a particular attack that is attacker is targeting. For example, attacker can target probably the token or attacker is targeting probably the user password. In the end, we also do the classification, try to find out the pattern. What is the pattern? Exist there and that gives a guidance that this type of things we probably need to look in future in our development phase. So for threat modeling, we usually follow a process and in this case, for this analysis, we have followed a six-step process which is a simplified or adapted version from OSAP analysis process they use there. We'll go through these things using an example case which for which we have done it, one iteration of threat modeling for a target component. And in this case, we have used Keystone Middleware. So Keystone Middleware is the one that actually it is there in every OpenStack service you use. So it's in the pipeline of your Nova, it's in the pipeline of your Glance. So what it does, it's a proxy that intercepts and you call it is coming and then from there, it checks the token, it takes the token and then verifies the token whether the token is correct or not. And if the token is correct, then it populates some more headers and let it pass through or block it there. So that it does there. So two main things the Keystone Middleware does is it verifies that incoming client requests have valid tokens by communicating with the UID. In case of UID token, it communicates with the Keystone server. For PKI and PKIZ case, it just checks the signature. However, there are some other, actually some other sub-use cases are there. So even if your token is invalid by these two main cases, if there is a delay authentication flux there, it can go a different path. If there are token caching is there, it can probably go some intermediate path there and so on. So from this thing like what does the Keystone Middleware provides, what is the security guarantee it tries to provide? The security guarantee it provides is verifies the integrity of the token and it helps us for access control. Now that's a high level understanding of Keystone Middleware what it does. For threat modeling we also need to understand what are the internal components of any component out there. So in this case this is a one level deeper understanding of Keystone Middleware which we can see here is OTH token is the main component which is or sub-component which is actually doing all the operations and then there are a lot of this dependent part that is such as HTTP request, hashlips, CMS, memcache, scribe and so on. And then there are some interacting up and down side. There are some interacting parties are there that is the client side and INTPs are there. You can also see there are some dotted lines out here. So this dotted line actually interprets the trust boundary. So the trust boundary implies that it creates a trust zone. So you trust everything within that zone but you don't trust anything coming outside from the zone. So you don't trust anything coming from the client side when a client makes a request to this module inside OTH token module. So you don't trust anything coming from the client side. Similarly probably if you want to place the lower dotted line then you don't trust anything coming from that side. But it depends on your setup and deployment cases. If you believe that your deployment case that, okay, I believe the underlying conflicts, RD store and memcache store are trustworthy, then I can remove it and we can then put it in one trust zone. So then we go for a detailed analysis. This is a deep dive but we can then probably make those detailed analysis a simplified version. So this is a simplified version of PKI token validation. Here you can see there are major assets in the green color which are used during the verification or PKI token verification phase. And you can also see the attacker, the attacker where the attacker tries to attack. The attacker tries to attack the client, attacker tries to attack the config file, the memcache, keystone server and so on. So now we have a pretty good understanding about the keystone middleer, what are the internal components, how the data flow is going through, what are the critical assets are there, and what is the attacker motivation here. The next thing is to identify the threats. That is what we are looking for. Now to identify threats we have to look into what are the system assumptions and security controls, existing security controls in place for a component. Now say that we see that there are probably, we trust someone can trust the client and we just reduce that assumption. No, we will not trust the client. What becomes if a client becomes an attacker? It could be that we are someone thinks that the communication channel or tunnel between the client and the keystone middleer is trusted. So what happens if something goes wrong there? What happens, our keystone middleer is not checking the CA certificate or it is not matching with the host name of the certificate which we've seen for STDB live 2K. So if those things happens, how the scenario changes? What kind of threats can come from those scenarios? So the next step would be to actually to have a guidance report kind of like accumulate all the threats from the analysis and for this case we have using, or we are using like this kind of three step model. So we first take one threat consequences from the threat consequence we identify what are the successful threats that can cause these kind of consequences and for each threat we look for the attacks, like what successful attacks can cause the threat. So let's look at an example. So we are taking spoofing of identity and that can be done by unauthorized access to the target service using a keystone middleer. So we are following a tree like pattern. So how many ways you can do it? So you can do it probably multiple ways. One option could be attacker pass auth headers and the keystone middleer fails to remove the auth headers. And then we need to think about, okay, is it a possible case? If it's not a possible case, we can just check it out. It's a cross case. Then we take another op path is like, okay, revoke token can be used as a valid token. So this is a possible path and then we can go dig down to another path how that can be possible. It could be possible by attacker using a cast token and there is in the configuration file you haven't set revocation forecast. So that can cause within that opportunity window a revoke token can be used in that opportunity window. Even if you set the revocation forecast token, it can be further one step down where if your revocation list has been cast during that time the token can be a revoke token can be used as a valid token. So that's one path. Now then we can go for another path. So another path is if we have a compromised revocation list that can cause a revoke token used as a valid token. So how that can be possible? There are multiple ways that can be possible. Signing being compromised, we can assume that, okay, this will not happen. But then we can go for another path which path says, okay, a stale revocation list is sent by a network attacker or a man in the middle attacker in between. So is it a possible scenario? So let's assume there is a possible scenario. Why is a possible scenario? If there are two security weakness exist in your system, in that case it's a possible scenario. So one of the possible scenarios is one of the weaknesses to be there is your server certificate validation from the Keystone middleware side is incomplete. So if there is a bug or if it is running in insecure mode, in that case, the server certificate will not be validated or if new bugs such as STTB leap case comes where the CS certificate is not checked by the Python STTB leap to libraries, in that case it can happen. Then the second weakness needs to be there is freshness check missing in the revocation list. That is you just create a revocation list but there is no expiry time for the revocation list or there is no counter exist to say that this is the current fresh list out there. So if these two weakness exist in that case we can say that an attacker can create a replay attack or man in the middle attack and then all the way down you can get an unauthorized access to the system. So but then again we have to create some or we have to give some scoring here is scoring about whether it's a possible attack what is the probability here. That's something we have to still have to work on how is the possibility of these such kind of attacks in scenarios. So all in all threat analysis can provide us deeper understanding about the system, internal system and it can give us a comprehensive view of the system. In some times it can feel like there's a lot of documentation and scaling of the work, how to like if it's too much documentation how can we work with the OpenStick Security Group how to actually implement it across different OpenStick projects that become sometimes challenging which we are still working on with the Security Group and then that's something we are still discussing. So that's I guess we can now take a question. So just before everybody leaves I need somebody to give this man a camera for a second. So while you all think of the amazing questions you're gonna ask I'm also the chair for the security track at the summit and I think we've probably got good calls for another room as we're packed out through the doors every single time. So I need like Tim or Ben to give Joel a camera so he can take a picture from over here or come over and take a picture of everyone that's here. In the meantime I wanna talk a little bit about what we're trying to do to get the community involved in this. So the Security Group has been pushing threat analysis now for I guess coming up on a year I guess we've been trying to make it an active project. It's very difficult to do because you need a lot of domain expertise you need we've been pushing a lot to get other people involved in OpenStack. So at HP we do a lot of threat analysis work. Some of it is internal we're trying to move more of it to be external. Other big players in OpenStack do the same sort of stuff. People like APL have a mandate to make OpenStack more secure. So there are people out there doing similar stuff obviously Erikson as well I should mention. So if you have people who are concerned about security in OpenStack or if you have people who are trying to do threat analysis work in OpenStack we'd encourage you to come talk to us and see if we can do it in the open see if we can start publishing the designs. I mean the decomposition slide, the big scary one that's where a lot of the work takes place which is an unfortunate thing. I mean threat analysis is hard but there are enough people making enough money out of OpenStack that everybody can contribute a little bit to this and make it more secure. We're trying to push this into enterprise we're building big everyone's building big products based on top of it. We need to try and come together on this a little bit more. So I've asked at other summits and people have become more involved and I'm just asking again if you're interested in this if you're either getting involved or in contributing stuff you've already done please come find me, find Joel, find anyone from the OSSG and let's have a bit of a conversation about how to do that. Okay, thank you.