 There's a couple more people filtering in. We'll just let them come in. I quite cunningly only have about 30 minutes of content, so we've got a bit of space for slippage. So my talks on securing the public cloud. OpenStack is an amazing tool, an amazing framework for building up cloud infrastructure. There's a lot of problems in terms of trying to leverage that for a public cloud, because a lot of the different assumptions that have been made when OpenStack was built and the initial designs. So I talk to a lot of that and go through. So my name's Rob Clark. I'm a security architect for HP Cloud. I've been with HP Cloud for about two years, so right up from the decision we started to take on OpenStack and build that up into a proper product. My background is in secure design, architecture, and cryptography. I was one of the founding members, along with Brian from Nebula and the OpenStack security group, and also one of the founding members of the OpenStack vulnerability management team. If you find a vulnerability in OpenStack, blog about it if you really want to, but what would be nice is if you sent it to VMT so we could all patch our stuff before it becomes live. So say come for the security, stay for the shiny. Today we're gonna give away a HP Envy super thin, super awesome laptop. I can't win it, unfortunately, because I'm an HP employee, other HP employees here can't take part, unfortunately. But yeah, we have it here. So those of you that can manage another 28 minutes of me will get the chance to be rewarded. Okay, so how many people here are responsible in some measure for the security of their OpenStack involvement? Cool. So that's about two thirds, maybe three quarters. So you should have some interesting stuff here. The rest of you are probably interested in security or in the wrong room. Some of this stuff will probably scare you a little bit. Right, so here's an amazing graph that tells you nothing, but what it actually describes, going from the left side is where your private clouds are. As you move across to the right, the threat and attack surface of your cloud deployment increases. So somewhere in the middle, you have clouds that are set up to do web-facing operations, clouds that have interactions with different other clouds or different interfaces. And then on the right, you have public cloud, which is an interesting place to work right now. This is about where the controls in OpenStack get you too. OpenStack gives you a certain level of preparedness, but it assumes a lot of other controls are in place. It assumes a deployment in a reasonably safe and non-hostile place. There are a lot of assumptions and a lot of bits of the design that need to be changed or augmented. So what do you have in the private cloud? In the private cloud, you know who your users are. More importantly, you can fire them if they do something really stupid. Knowing who your users are means that you can give them training. You can give them frameworks. They understand who they're working for and what they're doing. Very, very different to having customers. Networks in which clouds are typically deployed are already secured. There are well-understood ways of doing this. Clouds are normally deployed by large companies with global real estates. They have policies, enforcement. They have appliances, application, gateways, they have all this stuff. They have all this stuff that protects all of their other network assets. And in that situation, it's not so much of a headache to just pop up OpenStack and a few wraps inside of this nice secure area. You can't do so much of that in the public cloud and I'll go through why in a moment. You have policies. Your company understands what they want this cloud to be used for. Some places will have multiple clouds to service different business needs. These are easily enforced and checked. Private clouds can inspect their own data. One of the biggest concerns people have with a public cloud is how safe their data is. They don't want rack space. They don't want HP going through and trawling through all of their files, inspecting them, understanding them syntactically. The reason is because they like to have a measure of security and privacy, but what do malware detection systems do? What do antiviruses do? They go through and do all this stuff. You can do that in the private cloud with no problem. You can snoop, manage, use a traffic. You have these well-defined border controls that we've already spoken about. You can have them throughout your data center to detect and control stranger anomalous traffic. You have very simple ways and very stringent rules about what can and can't go across your network. You understand and protect for your business needs. So in the private cloud, you really have a lot of control. Not all this control is necessarily built into OpenStack, but you have the ability to apply other compensating and preventative controls because of the nature of the business you're doing. You can, by default, restrict what people do. Okay, if they don't like it, then it will get bubbled up through your management chain and a policy decision will be made. But that's the way it works. So we've just been over. Users are accountable. There are established routes for responsibility and attribution of actions. Define network access controls and corporate policies constrain usage. Users can be held to specific use cases. If you fill out a use case or something for spinning up 10 instances on the cloud, it may say this is being used as an FTP server. The security networking guys will bring up your FTP server policy and apply it for that group of nodes. If that starts spitting out SMTP or something else, you know you've got a problem. There are many options for user control and discipline. So this comes from everything, from training, understanding, awareness to discipline. So making people realize that they are being constrained in what they can do. This isn't a free resource to your company. You're still having people consume resources that others could, and you need to make sure they're used correctly. IP reputation protected by clearly defined border policies. So you have an IP block given to your company. Things like mailability of email, spam filters, and a few other things, take this into consideration when making the decision about what to do about data that's come from your network. You need to be worried about that. And you have established abuse handling mechanisms, both internal and external. And the public cloud, we can't make many assertions about what people are gonna do with our cloud. In fact, it's a really bad idea to do that. We don't know what the next innovation, web 5.6 or whatever the next web innovation is gonna be. We don't know how this data's gonna flow. We can't detect anomalous stuff in the way you would like. We operate at a huge scale. We have big pipes. We're constantly expanding in terms of capacity and network support. This makes it very difficult to take normal static boundary controls, either boundaries of internal parts of your network or right out on your edge and apply them to a public cloud. We have a very robust set of fraud business logic components. But there's really no way of escaping the fact that new identities are cheap and easily obtained. And the best fraud systems in the world still have to be told or have mechanisms for detecting malicious use or the fact that credentials have been lost. So if we have certain malicious actors who may want to come onto our users, come onto our service, use it for six hours to do whatever dastardly deed they want and then leave. And if they pick up the right set of credentials and things, that's really difficult stuff to control. We have a global user base across any and every service. We see very, very different packet shapes go across our networks. We have no idea what the next big thing will be. And detecting abuse is really, really hard. So I'll go into that a little bit more later. In the private cloud, you have clearly defined policies. I'm highlighting this again because it's really important. You have clearly defined policies. You can spot very easily when something steps outside of that when it breaches the norm. When you have only a vague idea of what's going down your pipes, working out which one of them is the right one and which one is the wrong is tricky. So in the public cloud, we can't make very many assertions. Our users are largely unaccountable. You can't fire a user. You can ask them to politely leave your service, but they'll just go and use another one. That's not a preventative control where the public are concerned. You have open network requirements. We cannot set and arbitrarily make decisions about what data users can put across our network. Users have complete control of what they use their instances for almost over all behavior types and profiles are presented. Identities are cheap. Few data controls we can realistically apply at the edge. A constant battle to identify and control abuse of users. So here we go on to some specific use cases. In a private cloud, you know what data is gonna be stored. You have the ability, if you choose to, to apply analysis to that data to detect malware, to detect. If you have a multi-classification hierarchy in your data system, you can use that to pick up documents that are tagged and in the wrong stores. There are a whole bunch of other out of ban security stuff you can put around these because you have the capability to inspect, to look closely at user's data. And they're okay with it because they work for you. They share the data with you. You all have the same level of privilege. We have a whole bunch of terms of service, but they're largely permissive in terms of what you can do because they have to be because we can't predict what our service is gonna be used for. Privacy is a major concern for customers so we can't go sneaking around. We can't go looking in their files. Inspecting data is very difficult. There's very good cases and legitimate uses for encryption of files and we see this. One of our big partners, Panzora, they work with Dreamhost and they have an appliance that you have at multiple sites and it presents you with a sort of a global file store. And that encrypts everything before it puts it on the cloud and our cloud backs up the Panzora appliance. But everything that's stored in our cloud from them is encrypted and we as HP can't go in and look at it. Nova has a whole bunch of similar problems. There's a whole bunch of interesting things you can do to try and control what people do with Nova. So uses controlled by policy. Internal deployment options mean that you can configure who can deploy what sorts of image. But one of the really interesting things here is that you can force your users to deploy certain images of certain types with certain controls. So all of your endpoint security. So these can have encryption on them. Perhaps they can have IDS, they can have antivirus, they can have all the things that your IT company tells you you need to have in terms of endpoint security. And you can have this all on their VMs and it'll make them slower, but the corpora has made a decision that they accept the cost of that to bring it out. We can't do any of that. We have people spinning up all sorts of stuff, installing all sorts of stuff because they see it as a disposable resource. It's out there, it's on the public cloud. Most of the time they're spinning up things can be transient in nature just to test something. So we get insecure VMs popping up all the time. And some of them do get abused and get into problems. We have constantly expanding capacity, especially in networking and other areas, which makes it very difficult to use common sort of static controls. The cloud is designed there to deal with anomalous behavior. So we can't sit in a room and say, well we are going to have exactly this much usage over this much period of time. We have to plan big, we have to plan and do things at scale. So here's some abuse that we see people trying to do with HP Cloud. So we have to get people storing things that our terms of service say we don't want. Okay, might not necessarily be illegal, but take it somewhere else. We have copyright stuff, sometimes people try and pipe copyright stuff on the cloud. Okay, that's not new, that happens everywhere. We need ways of dealing it, dealing with it and detecting it. Exploit payloads are a really interesting thing to have. So if your browser gets an exploit, normally that exploit is a shim exploit, it's very small, it's slipped past an antivirus, could do things. The first thing that does is it goes and downloads a big lump of data to explain what it should do, whether it's an executable or the rest of it. If you were to combine Swift with CDN for doing exploit delivery, then you would have a lightning fast delivery coming from very different network points around the world, distributing your exploits for you. That's cool. And of course these guys can encrypt it on desks, they can just use basic symmetric encryption. So even if you have antivirus, you won't be able to get it. File dumps, sometimes we can see this sort of behavior where a compromised machine will just be used, the cloud will be used for exfiltration of data for storage and later retrieval from a different place. It kind of makes sense. Nova can have all of these problems, plus a whole bunch more. The reason it can have all of those is because you can use Nova for storing files. You can spin up a VM, spin up some persistent storage, run NFS, FTP, whatever you want, and now you've got a file store. But we see a lot of malicious use going out from VMs, or we see a lot of attempted malicious use going out from VMs, putting port scans, SSH, brute force, SQL injection, denial of service stuff, and spam. And that's, you know, a lot of people don't have to deal with this or at least it gets caught with their borders. Private controls, you can have a rack. You can make a decision that at the top of that rack, you're gonna have IDS, you're gonna have an application firewall because you know that that rack is a cloud, but it's a cloud-serving SAP or serving something out. You know what sort of data's going in and out, you can build stuff for it. And that's a one-time cost because you're building something out for a purpose. If we were to do that in that way, there's a really big scale impact of that because we end up paying for the IDS twice. These IDS need to be fast and very performant because we don't get in the way of our users. So they're expensive. Now we have HP properties like Tipping Point and ArcSight and other technologies we'll talk about later, but they still have revenue representation that they need to have within the company and so it's not like we can just go and raid their warehouse and steal all their stuff. And even if we did, we'd avoid the first cost but not the second, which is that we have very high-density racks. If we have to take three or four of you out of that to put a security appliance in, that's cost us a whole bunch in compute or in storage that could be used to generate revenue for the company. We have some ways of dealing with some of this stuff. The reason that we're not releasing it right now, as is, is because it's very tactical. Some of it is not code that's ready for consumption by the outside world. Some of it isn't necessarily an appropriate way of doing this in OpenStack. These security initiatives have been running for a while and they work, but they were built against older versions of OpenStack that were doing older stuff. The first thing we did in the quantum was a glint in someone's eye when we started doing this and now it's an amazing thing with people talking about having firewall as a service. Some really cool stuff there, but what we can bring to that party is what we've learned in the last year in terms of how we can protect this and the sort of abuse profiles we actually see. Yes, for tracking. So most of the people we use for the IP reputation is just for spam stuff. I can't call them out individually because somebody else actually deals with a lot of that stuff. But if you're interested, drop me an email and we can talk more about it because we've got a whole bunch of information on this sort of stuff. So we have controls that are very tactical in nature. They're at places in the stack that you might not necessarily want them to be because hindsight's a wonderful thing. But what we wanna do is work with people to understand and build these controls into OpenStack. So the firewall as a service stuff that some guys are working on is really cool. Other stuff like just securing internals which I'll speak a little bit more in a little while we can do right now and are doing with people like the OpenStack Security Group with the Nova hardening project that started maybe a year ago. And I think it's fizzled out a bit now but actually bought some good security innovation for a short period of time. So what I'm gonna talk about here is just a little bit on how we deal with malicious virtual machines. So we have a whole bunch of things we're worried about coming out and going across our border because at the end of the day that packet leaves your network with HP's name on it. Okay, it's got an IP address that IP address says HP. We don't want HP packets landing on SSH brute forcing undoing all this other stuff. We just don't want it, it makes us look bad. So we have a mechanism that scales out with Nova and it scales out because it sits on the compute host and it messes around with IP tables. There's a few other things to look at traffic as it leaves. We don't do deep inspection. At the moment we made a very deliberate decision, not to, one it's computationally expensive and IP tables isn't the tool to do that. And two, we have this general ethos that we don't want to go snooping on people's data. So what we do is we look at the sign packets, we look at all the number of different new connections that are going out. We found with the little bits of profiling we can understand when all these things go out. Now one of the obvious problems with this is that you have a localized knowledge base. So you have your IP tables and your decision engine or whatever running behind that. But that only knows what's going on this host. So if I was a smart hacker and wanted to do some bad things with the cloud, I might write something that uses the APIs to spin up 30 or so VMs in a premium layer, ultra small, they send out small bits of data. Soon as they start getting rate limited, they crash out, you keep the floating VMs so they don't get reassigned to you from your pool, spin up a whole bunch more, do it again. That's hard to detect with this and we're looking at kind of interesting ways of leveraging bits of HP's property like ArcSight to start doing correlation across the cloud and saying, well, this tenant's doing kind of slow traffic over here but they're doing slow traffic from 30 other places over here and it's all going to the same place, okay? This is what we wanna avoid. We wanna avoid our services getting compromised. And the reason I bring this up is because I'm now shifting to looking at individual services rather than border controls. OpenStack inside is very, very vulnerable, okay? On the outside, it's reasonably hard to get in. There are reasonably well understood interfaces but on the inside, if you were to have somebody gain a point of presence within your network, then you've got a really big, painful problem for a number of reasons. Architecturally, most things inside OpenStack trust the other things inside OpenStack. What that means is that somebody on the side, somebody in your network can start representing themselves as other parts of your system and there is at the moment no way of verifying it and even in the places where OpenStack allows you to turn on SSL point to point, most of the time it's not doing any sort of endpoint verification and where it does a lot of the time, well, there is no R back inside of OpenStack at all. There's no way for OpenStack to understand conceptually that say a network node shouldn't send a certain type of message to your billing system or something like that. It's just not there. So these are a few things I wanted to call out specifically that we've been working on and other people have been looking at. One of the things I wanna mention in a minute is how well security is improved. So just keep these in mind, crunchy, squishy, loads of stuff that breaks. We've been talking about this for a while and internally I have a few different tactical things that I'll deploy to try and make the pain a little bit less. This is just finger in the air interpretation of how I've seen security and the discussions I've had with people. I now go into a room at a design summit and I don't have to put my hand in the air and say, well, there's no security there because people are already talking about it. They might be talking about it in slightly naive ways and that's kind of where the OpenStack Security Group comes in as a function to allow people to consult on security stuff to say, is this a good idea of dealing with this? What's the best way of securing this type of message? But I just want, you know, this room is full pretty much. It wouldn't have been a year ago. Okay, people care now. This is good. Cloud scaling, I've been fixing the message signing. Eric's doing a talk right after this one that I recommend you all go and see because he's gonna highlight exactly how bad some of that stuff is and what he's done to try and fix it. It needs a little bit of development to get to be production ready, but it's good. Nova messaging with RBAC, this is a really big thing. Nova is so big and so complex, the idea that one component doesn't know who's talking to it and whether or not it should be asking for something is terrible. Now, Nova's got a lot of good things going for it, but the security side isn't one and we're gonna work and change that. Swift Stack, in the last couple of days, we've been talking to them about doing the message authentication between the different components of Swift, which is a lot easier because Swift, pretty much everything has ultimate level of privilege and what you're protecting against is someone on the side injecting something on your network. At Memcash Deep Protection, asking for integrity, there are a whole bunch of configuration things in the different parts that we have deployed that can just be turned on. There's a whole bunch of stuff. So how many people here would like to see an open stack hardening guide come out of the security group? How many people would like to see a document on best practices for setting up and configuring SSL across all the different components in open stack? Cool. This is where normally someone goes, yeah, but it'll make it all really slow. No? Cool, cool. It will, by the way, but there you have to make a cost managed decision and you can put other compensating controls in place in areas where you don't feel the SSL's appropriate. And again, we'd encourage you to, if you've got ideas, come join the open stack security group and come talk to us about this stuff. So that's basically it. I don't wanna scare too many people with the public cloud. I do have a laptop to give out to someone. There's a sticker, an HP Cloud services sticker under one of your chairs. Someone finds a sticker, they'll get a really shiny thing. If you wanna help, please consider joining the open stack security group. If you wanna help more, come work for us. I'm sure there's a sticker somewhere, I know roughly where it is. No, it says HP Cloud on the sticker, sorry. Good try though. I think we've got a winner. There we go, congratulations. You got better applause than I did. Cool, there you go. That's why we all put conference stickers over the webcam, right? Cool, yeah, questions. So there's a few options. It depends what you're defining as privacy there. If you're defining their privacy from other users, their privacy from you, privacy from us as the cloud operator, you're absolutely right, you can. But if you are genuinely concerned about a service provider abusing you in that way, then the public cloud just isn't what you're looking for, right? You always have a hostile process of problem. You are captive within our system and to a certain extent we can do what we want. You can do, there are various things you can try and put in place like encryption and other stuff, but at the end of the day, everything's running in memory, everything is accessible. So if you're genuinely worried about that, then to use a compute example, public cloud probably isn't what you want. For the object storage example, there are absolutely a whole bunch of interesting ways you can work around that. I mentioned Panzerra earlier because they're doing this at a really big scale with us. They're taking data from companies that have massive corporate IP concerns. They're encrypting it and they're storing it in our cloud encrypted. And they do some cool stuff with PGP or some cool asymmetric encryption to make it work. But yeah, and even people who do, well, that's because it doesn't work. No, it's a very valid concern and there are even potentials around there's some interesting stuff going on in terms of trusted computing, but that actually bubbling up to something you can deploy in the cloud on a production hypervisor is a long way away. We have a ArcSight deployment at the moment that we use for a number of the things that you would use ArcSight for. We're still learning really what the best way to use it is. I mean, ArcSight is incredibly powerful and gives you so many different options. And one of the things we need to do is get some more people on board who can sit down and play with all the shiny toys that HP gives us. So yeah. There was a question over there? What we do at the moment is use them internally. We have various properties that interact with customers externally. Fortify is a good example of a company that interacts with customers externally. And there would be good reasons for them to come and look at the cloud. And we have talked with them about that sort of thing. But really, as a deliverer of services like that, unless you were gonna put it kind of into our password framework or something else, that wouldn't be something we'd drive directly from HP Cloud so much as we are a service provider and we would provide that service to various properties within HP and otherwise they'll wanna come and use it and deploy it. But if you wanna talk more about that, if you grab me at the end and we can run through it all because there's stuff I can't present, so yeah. Cool, anyone else? Great, well thank you very much. Good work. Thank you.