 Okay, so welcome to our talk on the OpenStack security group. My name is Robert Clark. I'm a cloud security architect for HP Cloud Services. I've been involved in OpenStack now for coming up on two years. And I'm going to speak to you at some length about some of the complexities in OpenStack and some of the challenges that we face moving forward. My name is Brian Payne. I'm the director of security research at Nebula. I've actually been involved with the OpenStack community for about a year, so I'm still a little bit of a newbie. But I'm learning quickly. I have a much longer history in the world of security in general. And so I'm trying to bring some of that skill set into the OpenStack community to see what we can do to make things better. So without further ado, for those of you that were at the previous Summit back in fall, we introduced this OpenStack security group effort. And so basically today we're just here to give you a status update on where we are. So first I wanted to start by telling you a little bit about our motivations and what we said last time so that you can keep us honest as we talk about what we've actually done and actually where things are headed. So last time, and I'm not going to rehash all of this, but last time we talked about why we needed a security group, we felt that there was a lot of people that were starting to think about security with OpenStack, that were working in disparate ways and that would benefit from having some security assistance. So we tried to pull together a lot of the security teams from all the companies that are involved in OpenStack, anyone that sort of had a security hat and get them into one unified setting where we could then work together to help improve things and have a slightly more common voice. And I think that's been relatively effective so far. So we can talk a little bit about what we've done. In general, we talked about this high level plan of trying to lead in the security space through various things, implementing, designing, writing. So some of it would be code, but a lot of it would be procedural. So getting the right people together, doing reviews of things, setting up the nuts and bolts that allows security reviews to just be part of the framework of how OpenStack works. And so we'll talk more about that today as well. So again, we just kicked this effort off about six months ago and there was a lot of interest right away. So what I try to do is the canonical logo slide here. And I hope I didn't miss too many people. But as you can see, we do have a lot of involvement already. When I checked earlier today, I think we had about 36 members in the security group as it is right now. And it's not intended to be a closed group, although we do like to get a pain from people that are interested in joining. And so that way we can welcome you to the group, tell you what mailing lists to join, and just generally know who's around and link you into all the right spots. So if anyone is interested in joining and you think you have something you can offer, then please touch base with us after this, send us an email, anything that works. So I'm going to walk you through some of the administrative things that we've done to just get the group up and rolling over the past six months. And this will give you a bit of a flavor of where we're at today. Rob will talk in the second half of the presentation about some of our plans for the next six months and where we hope to see all this going. So since it's open stack, a lot of things are done launchpad. We thought we'd set up a launchpad group. So here's a little screenshot of what we have. Basically just an overview of sort of the core mission of the group and all the members are listed there. There's a mailing list associated with this launchpad web page. We actually eventually set up a real mailing list, which I'll tell you about in a moment as well. But this allows us to have nice linkage to other bugs and things throughout launchpad so that you can have a common place to see some of these security relevant things. We started doing weekly meetings in January, so just a couple of months ago. We meet every Thursday at 1,800 DTC. It's just a brief 30 minute meeting. And it allows us to sync up if anyone has security concerns, things that the security community should be aware of. It's a nice place to raise those concerns. If you want to see a security review of something, it's a great place to raise that as well. Most recently, we've been working on a variety of different efforts and just trying to rally the troops. And we'll be discussing some of those specifically later on. All the minutes are posted online. Since it's IRC, this is done automatically, so you can actually see word for word where everyone said. And when we remember to do so, there's hashtags that we can use throughout the IRC conversation that can help the minutes be structured a little more into an outline and action items and stuff like that. So it's relatively easy to parse through what's been going on. We've also been working on something called OpenStack Security Notes. I'm going to let Rob describe this piece a little bit. So a little while ago, it became obvious that a number of people were getting into security issues. And a number of people were perhaps making the same mistakes in terms of the way they were configuring some of the underlying features in OpenStack and some of the supporting software in OpenStack. A lot of this stuff wasn't directly a vulnerability in OpenStack, but it was really something where we needed to start pushing out security information to people who would be interested. So that's what we've done with the OpenStack Security Notes. They exist primarily as a foundation to the OSSA stuff that comes out of the VMT. What they allow us to do is we will discuss a problem, suggest the right direction for some solutions, and over time these will build up into a reasonably good database of things that will be incorporated into things like the hardening guide, which we will talk about later. So they're similar to vulnerability disclosures. They don't directly affect OpenStack core projects, but they are worth paying attention to. They are released. They're drafted on the OpenStack Security mailing list, which we'll talk more about in a moment. And they are released, in general, on the ML and to dash dev. So everybody should be made aware of these things, and it's stuff to keep in mind in future. At the moment, we only have two. We have our LXC vulnerability, or LXC configuration issue, and a similar thing with Keystone. Again, these were either chosen to be released as OpenStack Security Notes because they were either discussed in public, and therefore didn't necessarily warrant an OSSA, or they were in foundation stuff like LXC was. And oftentimes, these OSNs are more associated with deployment guidance than a specific vulnerability to the product. So the OpenStack component might be just fine, but it might be better if you deploy it a certain way. And so the idea with these OSNs is just little short snippets that will help guide people to make the right decisions when they deploy it. And then we're going to put it on the mailing list, like Rob said, but there's also going to be this web page on Launchpad where you can go and just read all of them if you were starting with a new deployment or something you wanted to get up to speed on the latest. So these will be a fast way to get little snippets of information out. And we will be working on a hardening guide that will be a more comprehensive piece as well. So I was just going to highlight some of the more technical work that's come out of the group as well. There's been a few design sessions this summit around encryption for volumes, particularly the storage encryption side of things. And this has been work that's been led by Johns Hopkins APL. Some of those folks are in the audience today. And they are continuing to push this forward and working to get it into Havana. So I just wanted to highlight that this is a nice piece of work that's been done. The security group has sort of helped where we can to get it through. Sometimes these things aren't easy. And we also tried to help by getting qualified people to review the code and provide impact studies on how to best do these things. Another example, last summit I was up here talking about how the support of HTTPS and all of the clients for the open stack pieces was really, really poor. There was not proper name checking. There was not proper certificate checking. You couldn't provide your own certificate chain if you wanted to for various pieces. And so there was work that was actually done at Nebula by Dean Troyer. And he went through, effectively what he did, that are familiar with the Python side of things, is he re-engineered all of these clients to use Python requests, which has a nice HTTPS implementation. So he got a lot of these things sort of for free, if you will, after switching the underlying networking library to use requests. So that was fantastic. Now the clients work much better in a proper environment that has SSL or TLS for your service endpoints. And it allows people to deploy clouds in a much more secure manner. We've also been trying to get involved in review efforts wherever we can. So there's a ton of work going on the open stack community, and it's very, very challenging to track it all. So even with a security group that has 30 some odd members at this point, it's almost impossible to track every single thing. So one of the things that we started doing is on bugs. And very shortly on the actual code reviews in Garrett, you'll be able to tag things as having a security impact. So when such a tag goes on one of these items, then there's an email that goes out to the open stack security mailing list. And that will alert people that are interested to specifically put their eyes on these items. So it's a little bit of a user initiated system. It's not perfect, but it is a nice way to allow anyone out there in the community to bring in security help when you need it. And so we hope to really leverage this and advertise it so that people can use it to its greatest benefit. We've also been involved in a variety of mailing discussions that have sort of a security-centric theme. And occasionally for vulnerabilities, the VMT team will pull in someone from OSSG who has specific security expertise that can help out with a vulnerability assessment. So we've sort of alluded to it a few times. There is a launchpad group mailing list for the team that you can hit everyone on the team. But for a variety of reasons, it's actually nicer to have a mailing list on lists.openstack.org. And so we went ahead and set up one there as well. This is the mailing list. And we just named it OpenStack Security. And it can be more sort of general security discussion place for OpenStack. This is the mailing list where you will receive notifications when someone tags something with a security impact. So it's a nice place if you're interested in that as well. So this is somewhat recently set up, just early April. And so it's been a little quiet so far. We've been using it for some security group communications. And there's been a little bit of chatter. I suspect that will grow a little bit over time as people realize that it's there. So it's not all roses. We're six months in. There's definitely some challenges. And I'm just going to highlight a few of them. It's very challenging to come in at this stage in a very massive open source effort and say, hey, let's start thinking about security. Not that no one was thinking about security before, but I think with each summit, the number of people you see that are interested in security-related topics is just growing. And that's fantastic. We've had key management talks and encryption talks and things like this. And so making sure that people are making appropriately good security decisions is not always easy. So we're rethinking at all different levels. We're trying to get the right people in the right places so we can sort of help. But we can also benefit from everyone else if you see a place that needs help. Raise your hand, call us in, and see what we can do. We've also noticed very quickly with the group that there's a gap between those that have open stack experience and those that have security experience. I actually fit into that boat at some level, where a lot of the folks in the security group are really good security folks, but they may not necessarily be open stack core contributors, for example. And so we sometimes struggle with, oh, I see something that needs to be addressed, but I don't know the politics, or I don't even know the code base well enough to get in there and make this happen in a week. Whereas if we had the right contributor from a specific team, then we could go in and probably make that happen very, very quickly. And so I think this is a common thing with anyone that's getting involved with open stack. It's a huge community and sort of making your way in. Anyone that's out there that wants to help, core developers that want to assist the security team and provide a ramp up for us in that case would be fantastic. But even without that, we're gonna continue to whittle our weight anyway, so we'll do what we can. And finally time, right? So everyone that's involved in OSSG is just really, really busy with security back at whatever company that they're involved with. And so not surprisingly, it's sometimes difficult to get the cycles out of everyone, but that's just what it is. So we do what we can to work around that. Okay, I'm gonna pass it off to Rob. Do a little switch here, very professional. Okay, so Brian mentioned a moment ago about how hard it is to try and retrofit security onto OpenStack. So I'm just gonna have a quick look at this and we'll kind of investigate it together. So OpenStack is an IAS cloud. So that means we're fundamentally talking about compute and storage, except actually even in a moderately small deployment, you're probably looking at some or most of these components. That's okay, we can use well-defined security practices to lock this down. So one well-understood way of doing this is to create security domains and create gated access between the security domains. So this means that you sit there with your IDS, IPS, WAF, whatever, and you control through policies and detection what travels between these security domains. So what you do is you put your controls in place, you run your data paths, and all those data paths are running across these controls. Great, secure design complete. You can go home, check down the OpenStack group. Everything's nice and secure, right? Turns out OpenStack's a little bit more complicated. There's lots of glue required. So let's have a look at these data paths again. So for each of these, you're gonna wanna be securing either end. You need a good understanding of what's gonna happen in the middle. You need to understand what's going on and how everything interacts with the other components in this whole stack, in this whole ecosystem. So let's have a look. So we need message plumbing. Everything's got to talk to everything else. Great centralized points to hack things. Not the best place for security, doing ACLs and stuff. It's actually not necessarily easy at this scale with messaging platforms that are around at the moment. Billing, everybody needs billing. Everybody's doing private clouds, public clouds, whatever, even if you're not monetizing your cloud, you're still charging back for the time on it. Alarming, you need to know when things go down. You need to know when you need to wake up your network operations team and start shouting at people to get stuff done. That has to be plumbed into everything. Certificate authorities, NSA, we're talking about PKI and CA stuff yesterday. We're at HP Cloud, pretty much everything goes over TLS. That means everything needs to be able to talk to your CA, to your RA, everything needs to be able to do SCAP, OCSP. Everything needs to be able to look at CRLs. You need to think about how you're gonna do all that securely. Account maintenance, you've got teams of people working on all this stuff. If you've got DevOps or multiple regions where you're doing all your R&D and all that, you've got to understand how that affects everything. You've got to understand how all these systems interlink, how all happens across your entire state. So very quickly, you end up with not quite so pretty a picture that your security architect may have drawn for you right at the start. And understanding how to secure all of these parts becomes very, very hard. Everybody does it in different ways and there are some good ways and some bad ways. We are attempting to take on a number of projects to help make some of this slightly more sane. So you could panic and yeah, I'm sure a few of us probably have, but there are ways to fix all of this, okay? The OpenStack security group is here to try and orchestrate as much security effort as possible to approach this from the right angles to make sure that we're doing the right solutions that we bake security in where it's needed and that hopefully we can reduce replication of effort. So HB crowd, we've done all sorts of cool stuff. Nebula have, everyone else has. Rack, you know, Rack are coming in with some great security innovations this week. And it's really good that we have the OpenStack security group to try and bring all these people together and push stuff forward. But thinking back to that messy plumbing diagram, it's pretty hard to know where to start. So we've had a starboard kind of breaking things up from polling people at previous summits, from having discussions within the group about which way this is gonna go. So we're gonna take it in a number of directions. We're gonna develop a security hardening guide that Brian's gonna speak about in a moment. We're recruiting into the security group from different areas, from security heavy areas, from OpenStack heavy areas where you have developers who enjoy security. The security notes will improve in terms of cadence. We will be releasing them more, more regularly. This is one of the things newer recruits can help with. Consultancy and support, Brian again spoke to us briefly. We will be assisting more teams in a consultive capacity that is to say, and it's already starting to happen, developers are writing things, we're coming up with blueprints and then coming to the OpenStack security group and saying, can you take a look at this? Is this a good idea? A lot of the time we're able to give some good constructive advice on how to make these designs better first time around, which puts us in a great position. On top of that, we're gonna be looking at doing critical code review. One of the things HP has committed to is putting some resource towards really doing critical code review in an automated way against critical components in OpenStack and eventually through all of it. So last summit, we asked a question and I'll ask again here. Let's imagine there was a hardening guide out there that would help you to deploy OpenStack securely. How many of you would find that to be a useful guide to have? Excellent, so that's about what we saw last time. So a lot of folks seem interested in it. And so we were all excited about this and we said, okay, we're gonna go make this happen. And six months later, we had a really nice outline and realized this is really hard to do. Not so much that it's hard to do, but it's really hard to get the people to have the time to come together and to really do what needs to be done to make this guide good. And then I had this aha moment when I saw this really nice DevOps guide that was done in a week long documentation sprint. And so I spoke with some folks that were involved with that. And long story short, we are tentatively planning to do something similar in the June timeframe to create a security hardening guide. So we're gonna try to get together 10 to 15 security and OpenStack experts, lock them in a room and turn out a guide that will be a very useful reference for people that wanna know how to deploy this securely. We have several very good people on board already to put this together, but we would like to get a few more people. So if you're interested in helping out with such an effort, then come talk to Rob or myself and we can get you linked in as well. We're gonna try to keep the group to a manageable size. We're using a documentation facilitator and he sort of gave us some guidelines to how many people work and how many are too many. So our hope is to get the right size group and to make it come together nicely so that next time we're up here talking to you guys, we can have an actual document to show you and to end the point you have. So yeah, I think that's enough about that. Okay, so how can you help? How can you help? So get involved, we've got a whole bunch of interesting things that need to be worked on from, as I say, right through the stack, right through the range of capabilities that people have. If you're just interested in security but don't know where to start, open stack security group is a great place to start. We have people who can guide you through the early stages and if there are any ninjas out there, please come and teach us all about it. So we want people to share their knowledge. Have you been working on open stack security projects? Have you been trying to patch things or grade things? What have you come into? What issues have you encountered when trying to do that? Secure deployment options speaks very much to the OSNs that we're putting out at the moment. There are so many interesting ways in which you can just flip configuration options in open stack or choose different technologies that result in you having an incredibly insecure infrastructure. We really need to get a better handle on that and it needs a lot of input because a lot of people are putting out things in very different ways. Architectural security concerns, there are a few issues in open stack that are fairly systemic that will take significant effort to fix. A key one that's worth mentioning is know of a message signing. Most people who've looked at the way Nova works would agree that it needs to be better. There are people that are working on it but these people have been working on it for a while now and I think need some support to really get behind it. And that's the sort of thing the open stack security group can do as a whole. Bring in security experts, open stack experts and support things where they need support. So we are recruiting enthusiasts and I chose the word enthusiasts because it's nicer than noobs. So we're putting out a hardening guide. Okay, so that means we need people who have got good reading and writing skills. We need people who've got good technical documentation skills. We're gonna be putting out more and more documentation on some of this stuff in the future. So we're gonna need people to help us get it right first time. We're gonna need people to help maintain it. Security note editors and issue researchers. These kind of go together. Security notes can be quite esoteric sometimes. They might relate to one small product that some people are using that they use in a very insecure way. So it's worth bringing out an OSN to say, well actually you need to be careful about the way you do that. Sometimes you need people to go away and research that. And it's a really good way to contribute and if you're starting off on a security career, it's a really good way to get your name on a few documents and start showing that you're actually contributing out there to the community. This brings us right onto next point which is community engagement. That's going out, working with developers, mentioning security issues when you're at your open stack meetups, all that sort of thing, selling the OSSG or just selling security in general. And a security group tooling. If we've got any good web UI developers or anything like that, we are thinking about a whole suite of tools that will be very useful to anybody who's trying to assess for themselves what the security stature of open stack is at any one point. There are a few guys who got interesting work in this space at the moment. Matt Joyce, I'm not sure if he's in this room at the moment but you want to go track and down to talk about his OSSG database project because that's really interesting. Open stack devs. We need people who really understand open stack to come in and review code and design. The blueprint process can be a bit tricky especially if you're not used to putting insecure blueprints. We need to take some of these nudes or some of our security guys and we need, some of them will need guidance on driving blueprints through the process the first couple of times. There are plenty of people in open stack who can contribute to that even if they're not totally OFA with all the security issues. We need help with patching. So when we see more and more vulnerabilities come in or there will be more issues that perhaps aren't considered by the VNT to be vulnerabilities but we feel as a whole need to be patched and dealt with and improved. Those are things that guys come in with an open stack background can help us with. And again, community engagements are really common theme. The open stack security group exists to try and encourage as many people to come together and work on security together, right? All the big organizations are out there doing open stack at big scale at the moment are doing a lot of things behind closed doors using a lot of private compensating controls. And then to some extent that's going to continue but very much in the spirit of open stack a number of us are prepared to contribute a lot of this back out and improve the quality of the product for everybody. So community engagement is very important. Info second enders. These are guys that are coming in with a very good security background like Brian and to some extent myself but perhaps haven't been in open stack long enough to really understand all the politics and of course, these guys won't have broad spectrum understanding of how all the different projects work. And previously to get things done in open stack you really needed to know who to go to to float ideas especially if you're very new to the project and you might only want to contribute in a very specific area that can be quite tricky if you don't know how. So this is where nine ninjas can work together with our open stack developers. In terms of things that individual things I'll bring to the group, threat analysis and architectural security reviews. I'm not sure how many people were at a summit a couple two or three summits ago now where I met Brian and he and another Nebula engineer were presenting some threat analysis and attack surface stuff for open stack. And it was very cool and very shiny and it got a lot of people in the room very excited and we need more projects like that. Code reviews, Brian already mentioned that security tagged things are gonna start popping up to open stack security group members a lot more regularly. A lot of these things need to be looked at with people by people who've got a good eye for looking for problems. So you know, last thing we want is to introduce new vulnerability while trying to patch an old one. Automated code review, I already spoke about this previously, we are interested in working out the best way to automate some of the code review processes. Now, Python isn't necessarily the best language to do essay type approaches, but there are people doing interesting work in this space. We are working with them and we expect to have some kind of cool stuff to be able to show you at some point in the near future. VNT engagement is an important point. The vulnerability management team are the marshals for all the security vulnerabilities and bugs that come into open stack. They decide how to address them if it needs to be dealt with behind closed doors because it's severe. If perhaps it just needs an open release, stuff like that. The VMT is very much made up of open stack devs and the OSSG exists to be able to bring some more Infosec specialism to the VMT as and when they want it. This means that we can look at vulnerabilities for a number of different angles because we have security experts from public clouds and experts from private clouds and everything in between. So a lot of how important a vulnerability is very much context dependent. So we can provide a lot of that context and attack surface analysis to different vulnerabilities that come in. The Infosec guys will be responsible also for security evangelism and awareness. Looking back through open stack, there are in many, many places quite glaring vulnerabilities or omission of security controls that will come back to bite you. Things like noble or keystone logging, user credentials in debug mode. Things that just, they didn't need to be there in the first place, right? Anyone who's ever worked on secure products before will not have perhaps approached it in the same way. But what we can do is we can get out there and we can start talking to people about this stuff early. We can hold up old things as examples of areas where code could have been done better. And we want more security people to be there to be able to have those sorts of discussions and to be open and just kind of move everything forward. So in summary, what do we want to do with the open-stat security group? Well, we're gonna do this documentation sprint in June. The hardening guide is something that we are repeatedly asked about. There is a lot of interest. We have a bootstrap sort of template for it at the moment. We know which direction we want it to go in. But finding the time from the right number of people with the right level of expertise is incredibly hard. Everybody in this room, I think, is working on OpenStack. And I imagine all of your calendars are booked out at least until June. So everybody is very busy trying to do all this. We have a lot of work to be done in areas like recruiting and areas like tooling. But we are getting more and more people on board. We've got coming up on 40 people now with nearly 100 people on the OpenStack security general mailing list. So what we want to do, this is really a rallying cry. If you're interested, if you're in this room, you will have something to contribute. You will more than likely fit into one, if not more of those groups that I just went through. You do have something to contribute to OpenStack security. And we'd love to have you on board and improve the product for the next summer. So with that, we have a few minutes left and we're happy to answer any questions or just have general security chats if folks would like as well. And if you could, if you have a question, come to a mic or we'll try to repeat it for you so that everyone can hear. I'm curious what you're seeing from the user groups, or the OpenStack users doing deployments right now. Obviously the hardening guys of high interest and understanding how all these components fit together. What other areas are people doing private or public deployments looking at? Sorry, private or providers? So what areas of OpenStack security in terms of what they're developing or in terms of what they're worried about? What they're worried about. So for example, has the issue of using OpenStack as a base for a compliant cloud for whatever regulatory? Oh sure, so compliance is a good question. Compliance is causing problems for a lot of people who are trying to deploy OpenStack at a big scale in a moment. And the main reason goes back to, I'm not gonna go through the slides, but goes back to that nice pretty architecture diagram I showed. That is what auditors expect to see. When you show them the reality of what's going on in OpenStack, it gets more complicated. And it forces you to do a few things. And in many ways it forces you to actually create walled garden models where you have a lot of repetition of services because you don't feel that you can share them between the gardens and things. So what we would like to do is get to a point where you didn't have to apply all these extra compensating controls like ordering off things, putting different levels of IDS and detection between them to kind of tick all these boxes. We are finding in some of our engagements that there are some auditors coming around to this and some perhaps lagging behind a little bit. This is a problem I think a lot of people are gonna face. Do you wanna add to that, Brian? Yeah, I mean, what I was gonna say is that a lot of people are solving these problems behind closed doors. And then it's leaving people that are sort of new to OpenStack or trying to do their own private deployments to fend for themselves. And so there's a lot of reinventing the wheel that's going on. And there are certain pieces that are, encryption is a classic example. It's very easy to encrypt something. It's very hard to manage the keys properly and to have all of it built up just right. And so to have some commonality of expertise around how to do these really difficult things that are sort of deceptively simple, I think would improve OpenStack quite nicely. These are the kinds of things I just hear over and over. Well thanks for the insight. Thanks for all the hard work on this. Sure, sure. Any other questions? Okay. Well, we'll stick around for a few minutes. I think we're moving into a break, so happy to field some one-on-ones up here as well. Thank you everyone. Thank you. Thank you.