 All right. Good morning, everyone. Welcome to the OpenStack Summit here in Atlanta. We are kicking off a security track today by talking about the OpenStack Security Group. And if you don't know what that is, you'll find out in a couple of minutes. If you do know what that is, wonderful thank you and welcome for coming. I know this room was a little tricky to find. I know I was running around in circles for a few. So I suspect we'll have a few people trickling in over the next few minutes. But I think it makes sense to go ahead and get started. So first of all, what we're going to talk about today. I've got myself, Brian Payne, Rob Clark from HP, and Nathan Kinder from Red Hat all here to talk to you. So we'll be rotating a little bit. Hopefully that will keep it a little more interesting and dynamic for you guys as well. First, I'm going to kick off by telling you a little bit about what the OpenStack Security Group is. Who we are, what we've been doing. And Rob will tell you about some of our plans for the upcoming release cycle. And we'll get into some more depth about some of the things we're doing and what we're planning to do next on each of those specific topics. And then finally, we'll close by showing you all the wonderful ways that you can pitch in and help. We're still sort of a new and growing organization with an OpenStack. And we'd really, really benefit from your help and involvement. So first of all, a little bit about who we are. We are a rag tag grassroots kind of group. We've come together to try to improve the security in OpenStack. We're working on documentation, deployment, guidance, steps for hardening OpenStack cloud deployments. And trying to also work with each of the core projects and even some of the fringe projects so they can improve their security along the way. We currently have about 150 members. I count that by the number of people that have joined our launchpad group. There's also maybe a subset of that who's really regularly active through our IRC meetings, through our mailing list, and through the various projects that we're working on. If you're interested in learning more, getting involved and getting pointers to all the various things that we're doing, you can go to the URL at the bottom of this page. That's our launchpad group and from there you can request to join. It's currently listed as a private group, but trust me, we will probably accept you if you say that you want to join. Some of the main things that we've been working on is documentation. So actually, this wasn't really expected, but when we first started work with the security group about two years ago now, we kind of thought we'd just dive right in and start making the code better. But we realized there was a big gap in documentation and letting people know what to do with the existing security features, how to deploy things securely in OpenStack. And so we actually started this journey by getting involved in writing a book on how to do all this stuff well, by working on OpenStack security notes, which is things that we'll talk a little bit more about later, and some other documentation efforts. We also do a lot of code review. For those that are developers, you can tag your commits with security impact, and that actually sends an email to the OpenStack security mailing list. And with luck, we'll get in there and give you some security review on the code. I know we're not perfect with getting all those reviews in, but we do our best to touch the ones where people request it. More recently, we've been actually going through the code at a slightly higher level, sort of at an architectural level, to do thread analysis and review. And we've been finding some very interesting things there, and we're going to talk about that in more depth later in this presentation. And finally, occasionally the vulnerability management team will call in members from the OpenStack security group to help sort of triage bug reports to come in. So we can assess whether or not something is severe enough to require an OpenStack security advisory or a code change, or if it's something that could be handled through documentation or just more information about the source code. So if you look back across the most recent six months, we've had a lot of improvements in how we're handling OpenStack security notes, and we'll talk more about that today. I mentioned the thread analysis project. We're going to be talking more about that. And we also finally formalize ourselves a little bit by having leadership elections. To give you a sense of our history, we sort of started this group, I said, about two years ago, and it was sort of led by Fiat because Rob and I sort of just declared ourselves leaders and started growing the group. We finally decided as we grew a little bit bigger, it was important to have an actual elected leader. And I was also needing to step aside to spend some time on some additional other tasks. And so we had a lead election, and Rob here was actually elected lead for the next cycle. And so, congrats, thank you. So one of my first acts is sort of outgoing co-lead will be to speak a little less to you today and let Rob speak a little bit more. And so on that note, I have Rob coming in to talk to you about the plans coming forward. So when we decided to look at how the OpenStat Security Group was going to be run and the things we were going to try and achieve in the next six months, it turned out there were actually quite a lot of things that we wanted to do. We end up with a large number of projects, and prioritizing them becomes quite important for us. They have various levels of impact, and some of them you're already well aware of. So our three key projects, the three things that the OpenStat Security Group is going to be focusing on over the next release is threat analysis, the OpenStat Security Guide, and the OpenStat Security Notes. These are three things that we're already delivering and already providing value to OpenStat with. Following on from that, there is less of a focus, but something we'd like to deliver within this timeframe are the developer security guidelines and the cryptographic review of OpenStat. It is very difficult right now for somebody to look at OpenStat and say, my business, my company, my government demands that I use the following cryptographic standards. Does OpenStat meet them? Well, yes, probably, no, probably. It's very difficult to just say quickly what is and isn't being used in different parts of OpenStat. Our stretch goals, and a couple of these are waiting for technology to catch up with us, are to do with automating some of the other things we've done. So where we have consistent findings from threat analysis, where we have consistent vulnerabilities being identified through security advisories and security notes, is looking at how we can codify some of those things and put them into the various bits of infrastructure we have so that we can start catching them while they're being committed. So threat analysis, security guide, security notes. These are the three main things we're focusing on. They already provide value to OpenStat. The guide is available in tree form. You can download it for free. You can contribute to it by correcting some of the many mistakes. It's a great way to get into contributing to OpenStat security, actually, because the guide is very large. It was written in a short period of time and I think Brian's going to speak a little bit more about that. Our best practices are interesting opportunities to improve OpenStat from a security point of view. They are started. Nathan started some of the cryptographic review stuff recently. The developer security guidelines are already up on the wiki, but a lot of them are stubbed out. The idea with the developer security guidelines is that what we have done in Fat My Talk after this is about a review of every security advisory that has affected OpenStat in the last five releases. We've taken a bunch of the things that happen a lot and put them into guidelines, and we'd like to get developers and PTLs to sign up to try and follow these things. The reason I call this out separately as a low bar for entry is that I don't necessarily require you to be a hands-on active technical contributor. So you don't need to be au-fait with using Garrett and the whole Git process, which is quite a high bar to entry if all you want to do is add a few lines to something or contribute in a little way. The stretch goals, I don't really anticipate us hitting these in this cycle. There are interesting opportunities where we can and can do some basic things, like checking for bad functions and checking for known mistakes that we've seen happen in the past within OpenStat. One of the things we noticed on the various reviews we did is actually developers are pretty good at putting in tests to make sure that they don't regress security issues. So the stretch goals are really here to identify where we want to take things in the future. So threat analysis, I'm going to speak a little bit about threat analysis in OpenStat, because it's something that hasn't really happened before, or at least it hasn't happened in public before. Rackspace, HP, Nebula, Red Hat, millions and millions of dollars' worth of other big companies are all doing their individual threat analyses of OpenStat. They're all looking at individual services. They're all finding interesting bugs, and some of them are disclosing them and some of them aren't. And they're all missing stuff. Every single one of us will have missed stuff in our threat analysis. Anyone who thinks they didn't did a bad threat analysis. So one of the things I'd like to see moving forward is bringing together of the different threat analysis efforts that are going on in the community, such that core projects can be reviewed in the OpenStat. There's already a process in place for doing some of this stuff. The guys that are working on this are working incredibly hard. And the idea is that we will be able to... This is just proof that people have actually done work and drawn diagrams and stuff. The idea is that once everybody comes together and starts contributing to these reviews in public, a number of things happen. Firstly, the individual amount of resource spent by each organization goes down. That's great. If I don't have to use three people on my threat analysis, if I can put two on it and have somebody doing something else that benefits OpenStat, like contributing to one of these other things, then that's a win for me, and it's a win for OpenStat if every other company does the same thing. Similarly, we'll bring together our combined knowledge. We'll get some better combined coverage. And then what I anticipate happening is that the various companies that are involved will then end up just doing spun out delta reviews for their little bits of secret source and their value ads. At this point, I'm going to pass on to Nathan who's going to talk to you about the security notes project which is something he's done amazing work on. How many people here are familiar with the security notes or have seen them? Good. Fair amount of you. I'm going to give an overview of what security notes are and then get into some process around it and where they're going. Security notes are not advisories. Basically, VMT is responsible for advisories and advisories usually end up being something that has a real code fix. Here's this vulnerability. Here's what you need to install to fix it. Security notes cover more things at the advice level or the documentation level. You need to be aware of these configuration settings. The default could be insecure. You know, we're just pointing out important things you must be aware of from a security perspective in the code that you're running. So it's really intended to raise awareness of these issues that are out there that you may not be aware of. You can think of it as a knowledge base but only for security-related topics. And so here's some examples of notes that have been published within the year, basically. And you can see it covers all sorts of things. Things from Heartbleed, which isn't really an open-stack-specific issue, but it does affect open-stack to actual configuration issues within the projects, like potential token revocation abuse, for example. And it'll give advice of, here's how you can configure around this issue and tell you how to inspect your deployment to actually see if you're vulnerable or not. So the notes are published to a few locations right now. They're published to the mailing list, the comment list, and the regular user's list. And we also publish them on the open-stack wiki to make it a little more consumable. If you missed things on the mailing list, then it's pretty easy to do, especially on the dev list, because the just sheer amount of traffic there, you can go to one location and see if anything new is there on the wiki. And they're also listed within the weekly newsletter that Stefano mails out. So there have been a lot of process changes since Havana. Basically, the way that security notes were written and published in the past were that everything went through Launchpad bugs. So we determined that a security note was needed for a particular issue. Someone would write up a draft, put it in as a bug comment. Maybe you get some review feedback there and it goes back and forth. And then it would get sent out to the mailing list, no wiki at that point. That's been formalized a lot more in order to try to bring the quality up. So instead of just using Launchpad, we now actually are keeping security notes in Git and using Garrett review, just like the development process uses. And we actually have guidelines in place for who must review a security note before it can go live. So two of us here on the stage have to review every security note and a PTL or a core member of any affected project for the particular issue. Publishing, also the wiki publishing is new. That wasn't there before. So that makes it a little more consumable, I think. And then we have unique identifiers for security notes, just more for convenience purposes to be able to refer to them. In the past we just used Launchpad bug numbers, but we actually have some sort of unique identifiers similar to how the OSSAs, the security advisories are. So the results of these changes that I've seen are that the output has increased considerably. During Havana there were three OSSMs published. It was a new thing at the time. I think that Havana, right around when Havana cycle was starting was when the first security note was actually published. But there were three of them at that point. We had 10 published in the Ice House cycle and we have some already in progress for Juno right now. So the output is higher. Probably still should be higher than that. I think we could be documenting even more issues that are raised, but it's a good start. And the quality, I feel, has gone up as well. The reviews have been a lot more in depth and there's a lot more back and forth and adjustment of it and actually having the criteria for reviews for them to get published I think has really helped there. And I mentioned the criteria that we've been following there. And so going forward, further ways to improve the security note process. We want to get them published into the OpenStack Security Guide, likely in an appendix, and we've been working with the doc team on that. So I think ideally if the Security Guide is released for every OpenStack release, say the Ice House Security Guide update, it would be great if only the security notes that actually affect Ice House are listed in there as an appendix and updated live throughout the cycle as new issues are found, as usually happens. So that's a big goal and I think that will make things even more consumable and raise even more awareness to the notes. Adding automatic gate jobs to keep the quality up, so starting to tie into all of this great development infrastructure that's built around OpenStack, getting that into our process here as well for security notes. Right now we don't have any particular jobs that do things like check for formatting or they could potentially check for things better than just spell check and formatting changes, but we could actually have some other checks in place potentially on that. And another great improvement similar to reviewing security advisories is reviewing security notes, because a lot of times what we're finding with the security notes are it may have been a poor development decision in a default or just a design of a particular feature and that is confusing and could have security ramifications. So reviewing those and starting to establish guidelines based off of what we've learned I think can really be beneficial. And as we start to get into some of the other efforts that were just discussed, we can really go through driving those security guidelines to best practices for developers based out of what we've learned here. And of course one of the other future improvements I didn't list here is just trying to increase the output. That requires getting more people involved just in reviewing and pointing out issues that should be documented in writing notes. You don't need to be able to write code. You can simply know of issues and maybe something bugs you from a security standpoint of this is really confusing and we should write it up, contribute it. Absolutely, I think that's what we need to get more people on board with to really raise the cadence of the notes. And so that's what I wanted to cover security notes and Brian was going to talk about the security guide. All right, so how many people here are familiar with the OpenStack security guide? Quite a few hands, that's actually fantastic. So this is basically what we call the book in the sense that it's actually a book that you can buy in published bound form. You can also go to the website that's on the bottom of this slide and you can view the book in HTML. You can download a PDF version of it and those online versions are continuously up to date. So every time we edit a single word in the book we publish a new online version. So you're always getting the latest and greatest. The book was actually created through an interesting process called a book sprint. Basically this meant that last summer we took about 10 to 15 people that were involved in OpenStack security and somehow convinced them to come to a hotel in Annapolis, Maryland and we locked ourselves in a room for a week and it was all in good nature. But literally on Monday we were putting post-its on the wall trying to figure out what the table of contents should look like and by Friday we were doing final edits on all of the chapters that we had written. So this whole book was written in one week and that alone I think was just an amazing effort. It was a really great way to spearhead something like this. The end result is relatively comprehensive but what we did at the end is we sort of stepped back and we realized we had spent all this time talking about a lot of what we called the glue, these extraneous configuration pieces, things like TLS, things like the database, things like the message queue. All these things that you need to deploy OpenStack that aren't actually OpenStack themselves. We do have chapters in there on the various OpenStack projects as well, at least the ones where we had experts in the room. So we had chapters talking about configuration details for some of the main projects as well. But so much of the security for deploying OpenStack deployment is around the glue and how you deploy this thing. So now what we want to do is we want to maintain this and we want to make sure that it's always up to date as changes happen in OpenStack or as changes happen with these side projects that are required for deploying a cloud that the guide always represents the best thing to do. And so going forward, what we did is we actually converted the source code for the book into a doc book format. If anyone here is familiar with that, it's basically an XML-based format but it's not quite as bad as it sounds. That allows you to have a structured text around how the book is deployed. That way we can do reviews in Garrett. We can templatize it. We can do all sorts of fun things for deployment. And so now that we're a little shy of a year into having this out there, we're really starting to ramp up the efforts to do a full-scale editing to the book. We want to get a common voice since it was originally authored by 10 people. If you read through it from top to bottom today, you'll see a few little changes in voice throughout and things like this. So we just want to smooth it up, professional, update all the content, make sure it's current with Ice House as all that work is happening as well. And so that's work that I'll be working on and part of my reason here telling you about it today is this is actually a great task for folks to get involved with if you're interested in getting involved in security and OpenStack. You can learn a lot by doing it and you can also contribute a lot by having this book out there. A lot of people are reading it. I frequently hear back from people that it's so I think maintaining this and keeping it as a valuable resource for the community is a great idea. So a common theme throughout our various speaking points today has been about how to get involved in OpenStack security. So how many people in the room are involved in running an OpenStack cloud of some size of description? Brilliant. Okay. How many of you are worried about the security of that? Okay. A lot of people getting sleepless nights. There are a lot of ways that you can contribute to OpenStack security in the same way that, you know, we've just gone through a bunch of ways that you consume the security contributions that other people have made. So how many people in the room from your respective organizations have security teams dedicated to working on OpenStack? Brilliant. Keep your hands up. How many of you have those security guys making contributions to the Open Security efforts that are going on in OpenStack right now? Okay, that's pretty cool. There's a lot of people in here right now that are worried about security, but for whatever reason aren't in a position to be able to contribute yet. And that's fine. We can work on more projects like the ones we've spoken about to bring people in. So we have, as I say, we have these three key projects that we're pushing forward on, which are the threat analysis, the security guide and the OpenStack security notes. And you will see great improvements in all three of those. Any of those of you that a moment ago indicated you had security teams working towards OpenStack, I would really strongly encourage you, if some of you are in the room from those teams or if you know the people in those teams, get them to start talking to us about the threat analysis. Because I guarantee we will have found things that you haven't and we will have missed things that you have caught. And I'd quite like it if we had a massive breach of OpenStack on the front of TechCrunch and stuff, because that would be bad. So there are a whole bunch of ways to participate. We have IRC meetings every Thursday at 1800 UTC. That's right. I often start the meetings at five or seven, but it's at six. We do code reviews as Brian has already called out. There are a lot of interesting opportunities for people to get involved. And one of the things I wanted to do was a little bit of a question and answer session, because we've got the three of us up here. Myself and Brian have somewhat steered the OpenStack group for the last year and a half and Nathan has come in and been doing some amazing work with the security notes. And this is just a good opportunity for any of you guys. If you have any questions or if there's anything you want to discuss, I know this is certainly, for me, a lot less technical than the talks I normally give at the summit. So if anyone's got any questions, now's an opportunity to talk to us or any comments on the threat analysis or how you can get involved in OpenStack now is a good opportunity to talk about it. Please do come up to a mic because the acoustics in this room are horrendous, as you might have noticed already. I think on the first slide you had governance and you talked a little about configuration management and compliance. Is there any effort to actually define a hardening guide for OpenStack KBM combined? Because there's specific hardening templates like desistics or other hardening CIS as benchmarks for specific operating systems but there isn't one that is for the full stack. So the OpenStack security guide, the book that I mentioned earlier, some of the folks that were involved in that were actually talking about later taking the guide as it evolves and turning it into an OpenStack stick, for example. Right? So it already has some of that flavor. It's not as formalized but I think that that's something that could certainly happen if the right people cared and wanted it to help push it forward. But even without that formalism that nature of guidance there, all the way through to the very last chapter is actually on compliance itself. I have a more general question. You discussed that several either corporate entities or private entities, whether it be HP, Dell, whatever are performing security evaluations on OpenStack or either core features of OpenStack clients, whatever. The results from those are varied and disclosed or not disclosed, right? But if I was kind of asking, do you know of any more if they have been published generally is that information typically included in your notes or where would an individual get information on specific security evaluations that have been done on individual components? That's a very good question. To date, the only avenue that has been open to organizations like one of ours where they find something on a security review is just to open a bug with a VMT which becomes a private bug until it is dealt with and then eventually it's released as an OpenStack security advisory or they decide, well that's kind of bad but either we can live with it or it's more of a configuration issue or something we're not going to fix and then it becomes a security note. At the moment, there's no nice parallel process where people can contribute things from threat analyses and say these are the things we found and these are how we found them and that's the next step that's where we need to be. So the direct answer to your question is no but that's where we want to get to. However, the threat analysis work that we mentioned while it's not being published through the OpenStack security notes or the guide yet because it's somewhat immature there are public and get repositories that have some of those efforts. So they started with Keystone and there's already a fair bit of information available on their Keystone review. So I don't have the URL handy but it's been mentioned in IRC meetings so there's logs of it and if you came to one of us afterwards related this week we could certainly point you in the right direction to find some of that information. But you can stay for my talk which discusses that below. So as Brian said, the threat analysis going on in public is reasonably well documented and accessible but things coming in from other organizations that aren't really plugged in to this threat analysis process that we're both strapping at the moment they just, the only real avenue they have is to go through the bug process at the moment. And I'll say my threat analysis question, threat analysis. Are there any other questions? It's a pretty big group there's got to be some questions. Alright, we've got one coming back here. So I wanted to find out about how your work intersect with other developing standards in this area like cloud security alliance, FedRAM and IST or ISO cloud security initiatives. You guys want to take that? So was this, so the question was around which various standards like the cloud security alliance star or all that stuff. Was it pertaining to the book or to the work that we're doing in general and whether or not we're aligned with those? So with the work that you guys are doing? Okay. I wouldn't say that much of it is directly aligned with anyone's standard because you just rattled off five that you could think of during the time we were talking and it's difficult to keep all those different people happy at the same time. But what we can do or what we do say is that in the book we call out things that are relevant. The CSA have produced various matrix that kind of show you all the different controls you need to apply. So whether or not we would choose to, I mean we could look at putting those into the guidelines. The thing is most of these like ISO and all the others they're generally covering a lot of the same things anyway and if we pick one we'll end up upsetting a bunch of other people. So it's actually not something that, it's not something I'm a hundred percent swayed on either way. I don't know what you guys think. I wouldn't want to pick one particular standard and everyone's going to have different requirements for which one they're going to want to follow. I mean that's one of the things I like about the security group because we've got people from so many different organizations I mean it was initially bootstrapped by myself and Brian. At the time I was very much focused on the public cloud and Brian from Nebula obviously very much focused on the private side of things and as time's gone on both of our scopes have changed slightly but it gave us a good opportunity to provide these two very different perspectives on what security meant inside of OpenStat and if you go back two years ago and ask us which standards we needed to meet they would have probably been very different and now perhaps there's a little bit more convergence so things like the security guidelines what I would say is that if you pick your chosen compliance targets that you have to hit and go and have a look at things like our guidelines I would be confident that 99% of them would be met already and where they are we'd really value your contribution to improve the security guidelines I mean the other thing that's worth mentioning is that in my view when you think about security of a system like OpenStack compliance is one slice of that picture but there's another slice that's all about just general hardening right how do you make this thing harder to hack into and I would argue on the compliance side there's actually a fair bit of organization out there through groups like CSA and such like that and so we have largely like through our book tried to reference those things and talk about how they apply to OpenStack but I would say a big bulk of the effort that's gone on by the security group is really on the hardening which kind of becomes the foundation that you need for any compliance effort just to generally make your cloud better and stronger So I think I noticed you talked a lot about stuff kind of maybe more after the fact rather than maybe up front and trying to get the architectures and stuff right I know and Nathan had just posted on how much is the security group actually involved in that sort of helping other developers make sure that they're getting those additional architectures right. I think it's an area that we're trying to get more overlap between developers and the security group so it doesn't fall into one of those three main efforts that Robert pointed out but the code reviews and security reviews so every time a patch goes through if a developer remembers to check security impact instead of the keyword then it goes to our list now having actual process of how many of those get reviewed and having official sign off from the security group that's not all really in place but I think there is a lot of behind the scenes reviewing going on there I know we've been reviewing there were some blueprints a couple weeks ago around NOVA but we have to be asked at this point hey can you look at this and tell me if there's anything you guys think should be done differently. A good example of where we've been doing stuff like that is with Barbican because it's very easy for a security team to come in any security team in any organization to just walk into a room annoy a whole bunch of developers and then get ignored for the rest of the time so Barbican is a nice enough small enough project and it's easy enough for security people to get their heads around because it's key management that we can start we do review a lot more of their blueprints and do a lot of those things we ramp up in that way also, threat analysis doesn't necessarily have to be retrospective so at the moment it pretty much is because there's so much code already out there but the idea is that once you build up a certain cadence of review you can start reviewing deltas and those deltas can be on pose changes rather than necessarily looking at code that's already out there it's definitely something we need to work on a lot more you're absolutely right it wasn't highlighted in those slides but it's something I know we're all concerned about how we can best influence and steer OpenStack to make smart choices the talk I have coming up after this really focuses on places where people make bad choices in OpenStack and in a lot of these situations it wouldn't have taken much if the right person had been in the room at the right time things could have been steered differently so I think it's definitely something we need to focus on more it wasn't called out as much in the slide deck as it should have been perhaps and it's something that I'm quite confident that if we carry on doing what we're doing and bring more people in and start focusing on bigger and bigger projects then we will hopefully get to that point where we have the level of influence that we want the other thing to remember is if you look at the history and the timeline all the core projects that you know and love right? Nova Keystone, Horizon, Swift Glance, Sender these were in existence before the security group was in existence so it was hard for us to really influence the initial architecture for sure of course things are constantly evolving and as we sort of get more integrated with the OpenStack community at large one of our big goals will be to be more integrated with not just a code review step but an architecture review step at the blueprint stage and one of the early things that we've done there the Nova team has started to formalize their blueprint process through a blueprint template and all that and now there is a section in their template for security impact just like you'd see in an RFC or something like that and it sounds minor but it's an important step because just by putting that in over the past month or so we've already had several people paying us for hey can you look at the architecture that I'm proposing here and that's really a great step forward for us trying to make improvements before there are problems I'll follow that up how has to win the response from the developers when you've come and approached them maybe that's always a challenge being kind of in a security opposition it is a challenge and I think that's one of the reasons that we didn't come in two years ago and have an iron fist and say hey developers security is the only thing that matters you have to listen to us right we've been very careful to be sort of nuanced with how we integrate with projects I feel like we've generally been well received and we've been coming in slowly so you know we'll just have to see how it progresses with time but the projects that we work with we even have people that come to us to ask security questions whether they're asking us on the mailing list or they're asking us on the side I feel like we have a pretty good relationship it's just that there doesn't exist a formalism today that says you can't release Icehouse before it's approved by security I mean we're so far from that but they're starting to realize that we're here, that we're a useful resource and I think that will only benefit everyone in the long run well thank you everyone I think it's about all we had time for today we can take some more questions on the side if you'd like but really appreciate you coming out Thank you