 We good? Awesome. So if we can just maybe close the door there when you're finished in the bends, that'd be great. So hi, my name's Rob Clark. I'm going to first off start by apologizing for the clickbait. This talk is about saving your ass since Essex is about the vulnerability management team and the security project. What it really should have been titled was something along the lines of please, please, help with security. But the title I actually ended up with is OpenStack Security Project, securing the world's fastest, largest open source project. It is a non-trivial effort undertaken by many people for whom security might not actually be the primary function of their job. And I'm going to try and guide you through what it is that we do and how you might be able to get involved and be able to help us along. So quickly I should speak about myself. I'm the OpenStack Security PTL, and I have been for a couple of years. The security project actually started about four years ago myself. And Dr. Brian Payne created something called the OpenStack Security Group, and I'll tell you a little bit more about the genesis of that. I was one of the early VMT members. Those guys were all too smart for me, so they kicked me out at some point. I'm a security track chair, so the security track here at the conference and many others I've helped put together. I'm a cloud security architect and an IBM Distinguished Technologist. I've been at IBM now for eight days, so I have literally no idea what that means. But there you go. So I'm going to go through a few bits and pieces with you. I'm going to talk about the state of OpenStack Security, the different initiatives that we have ongoing and how you can contribute. So one of the things I think was interesting was to see some repeats of this slide at this conference. And there we have sort of 85% or so of surveyed enterprises saying that security is the number one blocker for adopting production-based OpenStack clouds, the number one blocker. This should not be a surprise, but it seems to be a bit of a challenge for OpenStack in general. We have these two security organizations, the VMT and the security project. The VMT is retroactive. The security project is very much proactive. And we complement each other quite well. But one of the areas we struggle with is bringing new people in. There aren't many people who are paid full time to do security for OpenStack. And it would be great if we could bring more people in if some of the major contributing organizations could throw a little bit more resource our way. Yeah, this entire presentation is just me begging for security resource. So there you go. So OpenStack security is a little bit like herding angry cats that just want to write code. This is how product people see security, see OpenStack, as you say, sorry. So we have various services. We can build little walled gardens around them. Those are the blue dotted lines. We can have a gated secure interconnects between them. Those are the bright green arrows. And then we map data paths. And that is generally how product people like to say that OpenStack works. If you ask a developer, it kind of looks a little bit more like this, keeping in mind, of course, we're missing all of the big tent niceness that's come in recently. So this tends to be how developers see OpenStack. And this is kind of how security people see OpenStack. And we're running around fighting fires. And we're doing our best. And I'm going to go through a little bit of the things we're doing to try and fight some of these fires. So I've done a lot of number crunching and some very big data analytics. And basically, we've come up with a graph like this that shows, it's very scientific, that shows the accumulative effect of vulnerabilities being detected in OpenStack. This is just a very general guide. But basically what we're going through here is the fact that when OpenStack started, there really wasn't much of an effort in terms of security. And when you look at some of the early vulnerabilities that were found, they were pretty bad. Security started to ramp up. We started to peak a little bit at some of their problems. And we saw a much bigger uptick in reported vulnerabilities. And now we're at a point where there are a couple of people dedicated to just looking at vulnerabilities in OpenStack and finding things. So we are finding more vulnerabilities in OpenStack as we continue to go forward. And this graph will never level off. But the things we find tend to be a little bit deeper. A lot of the low-hanging fruit has been picked from the OpenStack tree. So there are plenty of vulnerabilities to pick from in OpenStack. But this just shows you how vulnerabilities have evolved over time. So back in 2012, we were talking about really basic direct reach-a-versals and arbitrary file creation vulnerabilities within OpenStack. A lot of the time, it was unauthenticated users could just bring things into existence on your cloud, which is bad. Improper sanitization in instance names leading to cross-site scripting. 2014 had a whole rash of SSL and TLS failures right across OpenStack. And a lot of that traced back to Python implementations. And some of it drove the move across to using the Request Library, for example. Then we move on to 2014, 2015. And the vulnerabilities we find are a little bit more complex. And if you want a list of every single security vulnerability that has an advisory, where there's a link at the bottom of this slide, and I'll share the slides later. Also, if you Google OpenStack OSSA, you'll find a list. So what we're finding now is that the number of reported vulnerabilities is going down. The number of issued advisories is going down. And I expect to see similar numbers for 2016 and 2017. So the OpenStack security project started as the OpenStack security group. It was an informal group that myself and Brian Payne started about four years ago. The idea was that we would consult and try and work with teams to improve the state of OpenStack security as opposed to just reacting to vulnerabilities. It's made up of security engineers, architects, and consultants. We also have a number of OpenStack developers now who are just involved in the project. Michael McKeown, amongst others, people who've just kind of dropped in and said, hey, I want to do security things. And we're more than happy to buy the handoff and work them till they die. Consultants on security. So we consult on security advisories with the vulnerability management team. The VMT itself only consists of a few people, and they can't be expert in everything. We have a much wider group of people that are running public clouds, private hybrid, so we can provide extra perspectives and things on vulnerabilities. We have four to five calls at any one time. Around about 25 active participants. We have a couple of mid-cycles each year, and they're normally very well attended. And there's a lot of different security initiatives that we take on. So the vulnerability management team I'm going to tell you about in a little while, they exist organizationally as part of the security project. Previously, I think they were release management. They are largely autonomous in and of themselves. This just makes more sense from an organizational point of view. So I don't get to see any of the confidential vulnerabilities that the VMT manages, unless they bring me into consult on a specific issue. So we have developer guidance and security notes, the security guide threat analysis, lots of documentation efforts. We also have active code efforts like Bandit, Anchor, and Cintra Boss. I'll tell you a little bit more about those. And we still have an aspiration to do cryptographic audit and tracking. That is, being able to say to, if you're a cloud deployer, being able to say to your customer exactly which algorithms are being used at any one time with any set of configurations. Sounds easy. It actually gets slightly messy. And one of the things we're hoping to do is leverage Bandit for some of that. So around the false and time frame, we came together. And this is a bit of a timeline since then. So security notes started after Grizzly. The security guide followed threat analysis. An agreement to do VMT work. And then our code projects have followed on in the last sort of year and a half, two years. So vulnerability management is really important in OpenStack. And the vulnerability management team actually has a very well-established process for managing vulnerabilities that come in. So they receive, investigate, triage, document, and publish. Their actual official documentation looks like this, but I wasn't sure how well it would project. It's a very well-documented process. You could just Google the VMT, and you'll find it easily enough. And if you do find vulnerabilities, please follow that process. Opening a public launchpad bug makes everybody's life a lot harder, because we have to run around much faster to try and patch things in the open. And if it's a particularly bad vulnerability, it can be difficult for deployers to fix in short amounts of time. So we ask that people, if you find a vulnerability, that's brilliant. But if you could submit it as a private bug, that would make everyone's lives a bit easier. There were plenty of OSSAs to pick from. And this is just an example of one that's gone through the system. So this was in 2013, the 36th OSSA to be issued. And it was a cross-site reported by a Cisco employee. It was fixed, publicly disclosed, bug with marked public, which isn't great. Backports were completed. Got a CVE in the advisory, got published. Backports completed is a really important point. So vulnerability management team only release security advisories for things that can be fixed in supported versions, which leaves a really big gap for things that are just horribly broken, but we haven't managed to fix, like, Nova RPC. And I'm going to talk a little bit more about that with security notes. So security notes document common mistakes, bad configurations, and the mitigations for some of them. They're up on the wiki, much like most things a security project does. We have, in the order of 67 or so, security notes at the moment with a few more in the pipeline waiting to be taken on. Security note 64 actually went live, or is complete as of a review that was done earlier today. And the vulnerabilities themselves can be different things, but they're normally things that won't or can't be fixed in an existing release, or where it can't be fixed with a patch. So we create this documentation and goes through them. So it's written and managed by the security project. They very much are intended to complement security advisories. And that's a link to where all the issued notes are. The idea here is that we create a one-stop shop for the deployer. So although the security guide exists, if you're deploying an OpenStack Cloud, you should go and take a look at the set of OpenStack security notes and see which apply to your deployment. Because there's a lot of gotchas, there's a lot of considerations that need to be made things. Good examples are database caching and flushing with Keystone and things that you're just not necessarily going to think about. But you need them to deploy OpenStack securely. So they're linked into a couple of places. They're linked into the security guide, and they're online for review. And we do actually have deployers coming to us now and asking us questions about the OSSNs, which is great, because it means that people are actually going looking them, consuming them, and overall helping us improve them. So OSSN 64 is going to be published in the next day or two. It's in public review. This was a great vulnerability in Keystone, whereby deployers were leaving the admin token auth method enabled. That's an auth method that's generally used for bootstrapping the rest interface on Keystone when you're bringing things up. And it basically meant that if you set that in your header and gave the passphrase admin, because it just uses a default one, you would get admin and Keystone, which is bad, which is very bad. And it's not something that you could easily fix with a patch without breaking a lot of people's systems. So we created a security note for it that discusses the issue and different ways to fix it, like removing that piece of middleware after you start your cloud running, which is what you're supposed to do, but it turned out people weren't doing it. And that's a good example of the sorts of things that we try to protect against with security notes. So I mentioned before that this talk is about trying to get people involved with what we're doing in the security project. Security notes are a good way of doing that. They involve taking a bug from Launchpad, working through a reasonably well-documented process that is here. It's up on the wiki in far more of the both terms. The security project cause will absolutely support anyone who wants to take on writing a security note. There's always a back list of like three or four that need doing. Sometimes it gets a little bit longer. And we're definitely very keen to have more people come on, even if they're just coming on as reviewers to start off with. That's always very useful contribution. So like I said, security advisories and security notes complement each other. So security advisories are generally, this is a bug, there is a patch, it's bad, you need to go fix it. Whereas the security notes are more along the lines of, if you configure it like this, you're gonna have a bad time. So I mentioned the security guide a few times. I'm not sure if we've got Nathaniel in the room or Michael McCune, but those guys have been doing an excellent job on updating and pushing forward the security guide. So the security guide we created in 2013, it's available, or it will be when they fix the weird RST issue, be available again in PDF form. It's all online, it's all in the security docs, which is part of the OpenStack manuals. It reflects the current state of security within the OpenStack community. It's basically inspired by the Ops guide, which was released before it. And it's designed for deployers, auditors and even customers who are looking at different clouds who want a good set of questions to go and ask a provider, have you configured X or Y in a secure way as per the security guide? Again, this is another place people can get involved. It's great, if you would just wanna go and pick holes in the security guide, because something changed in Keystone or Nova works differently now or something else, then that's absolutely encouraged. Spelling errors, everything else, all fine. You'll get your ATC tag, you'll be very happy, you'll get an extra little thing on your budget in the next conference. We really, really do encourage people to get involved with these documentation type efforts. They're very good because they do reach quite far and wide. The security guide has been consumed a lot and we've had a lot of good feedback on it. A couple of example topics there. We go through a very wide swath of thoughts and choices you have to make when deploying an open stack cloud and the live links to the guide are there. We also have a set of security guidelines. The tag line there is don't write sucky code. I shouldn't possibly say please don't write sucky code. These are a list of things that happened a lot that we saw coming up in multiple vulnerabilities again and again. This actually was generated out of one of our mid cycles and inspired by something at the time I was working at HP and we'd created a set of coding guidelines specifically for Python, specifically for cloud type operations. We kind of forced it on our developers and we thought it was unfair not to force it on the rest of the community. So we put these together and made them available. The idea is that Bandit, our static analyzer, can actually tie back to these findings as well when you find people making the same mistakes again. So the guidelines are pretty simple. They're up on the open stack security page and it's a reasonably accessible list of things and they're written in a very conversational format. So it's example-based and unvalidated redirects. Here's an example of bad code that we see from time to time when people write things using Flask. And here's the example of how you might do it in a more appropriate way. They are like most things open stack community driven so if there are multiple ways of doing it, the author will probably write it in whichever way they're most comfortable. In this case, it was someone who's more comfortable with Flask as opposed to say Pekan or something else. Again, this is somewhere if you wanna be involved with what we're doing, if you work with a different framework, you could come in and tell us all about that. So that was the documentation efforts that we're taking on. I wanna talk to you a little bit about some of the code projects we have as well. I'm gonna talk to you about Anchor. A few people here will know that Anchor's one of my pet projects. And PKI has a really big problem that we can help you with. I can do about an hour and a half talking about either the problem or how we can help with it. So I'm gonna try and give you a really condensed version and at the end there's a link to a far more elaborate deck that goes through things about Anchor. So Adam Langley wrote this recently in his blog. Adam Langley at the time I think was a Google Chrome engineer. Revocation checking is useless. It doesn't stop attacks. He's not wrong. But I'm gonna try and take you through very quickly why he's not wrong. And then some of the things we can do to try and improve it. So TLS, SSL, it's all the security that you have in a certificate, if it's on a certificate being revocable. That is to say that I no longer trust this certificate and you shouldn't either. There are a number of ways of doing this. The first one was certificate revocation lists. A bit essentially a signed list of certificate fingerprints that should no longer be trusted. Turned out it didn't scale. So the online certificate status protocol was created. Nice lightweight UDP protocol. You reach out to some machine on the internet and say is the certificate still valid? And you get a yes, no, maybe message back. Unfortunately, that didn't scale. So then we have OCSP stapling, which is when a server can fetch its own OCSP attestation, bundle it together and pass it to you when you do a TLS handshake. Because they're individually signed, it means that you can be sure that the certificate is still valid. There are a couple of problems. Firstly, if a man in the middle can remove that. The second is that it doesn't work with intermediate certificate hierarchies because there's only space in the protocol for one. Then we get onto something called OCSP must staple, which is, I think it's still an RFC, but it's a change to the certificate specification that means that when you get a certificate, it says that it has to have an OCSP attestation with it and if it doesn't, then you should reject it. Which gives you kind of short life certificates because OCSP assertions are only valid for a short period of time. Most browsers don't actually support any of that, at least not for general certificates. So especially Chrome. Chrome has CRL sets, so Google creates an audited list of a very small number of certificates that they believe you need to care about as being compromised, and those are all checked. And this is easy to check on your machine. You can go to, if you set up an open SSL, CA, create a little website, give it a certificate, and then revoke it with a CRL or with OCSP, and then go to it with Chrome. Chrome will just say it's perfectly valid. It just doesn't check, which is a little bit difficult to swallow sometimes, and I bring this up only to encourage people to go and actually have a look at what TLS is doing because I don't think it's doing what you think it's doing. So once CRL is Mozilla's approach to CRL sets, although they still do some OCSP, depending on the type of certificate they get. So, PKI is pretty broken inside the browser, and it's completely broken outside of the browser. We don't support any of those more advanced technologies in any of the Python libraries we use in OpenStack. And in fact, most of the C libraries that you can do crypto suite stuff with don't support them either. So, this is kind of a metric of what is supported outside of the browser. You can just about get OCSP stapling with some C libraries, but basically, the only thing that works at all across the board, and the only thing that works at all for Python is CRLs, which, as we discussed before, don't scale. So we created Anchor. Anchor create gives you very short life certificates. I'm not gonna go very far into this because I'm wary of the amount of time we have to go through everything. Basically, it solves a lot of these problems by only giving you a very short lifetime certificate. And if you need to prove that you are who you say you are in another eight, 12, 24 hours time, you request a new certificate and you attest again that you are who you say you are. And this gets rid of revocation problems because the one control that works universally across all the crypto seats you tested is the checking the expiry date on a certificate. So by giving you a very short life one, we guarantee that we know when things will be revoked. It gives us a lot of nice properties, like by looking at the audit logs from Anchor, we know exactly how many machines in our infrastructure are using certificate at any given time, which is actually quite a nice quality and a bunch of other stuff as well. So it's an ephemeral PKI system, super lightweight stack. The idea of Anchor was that it could be deployed, say designate needs certificate services internally, it can just deploy Anchor internally. It doesn't require an environment where you have a big corporate style PKI that it needs to be configured to plug into. It's designed to just be spun up by individual projects as they require it. This is not our actual logo. It is some public domain image I found. I wish our logo was that cool. So short life certificates, rule-based issuing. A key point is that the rule-based issuing allows you to give it some understanding of how OpenStack works so it can understand naming schemas for different bits of OpenStack, that sort of thing. Yes, so there is an Anchor presentation. This is a wonderful screenshot from that presentation. If you're at all interested in PKI and the problems it can solve and the way Anchor can help you with that, then there's a Google Shorten link at the bottom that will take you to that deck. Cintra Bus is an awesome new project that we've just come brought into the security project. It is a further, which I know a few people here enjoy, for OpenStack APIs and services. If finding security issues in the RESTful API, it allows you to create all sorts of interesting first payloads. It logs all the requests and responses. Nicely, it supports keystone authentication so it can start working its way fairly deeply through branching logic within OpenStack rather than just banging on the front door. It detects common defects and it's generating some useful output. It's going through a whole bunch of changes at the moment, but if you're interested in a more up-and-coming project where you can have a bit more impact and get really involved, then Cintra Bus is a really interesting one to take a closer look at. So here's an example payload where we've just got a fairly simple descriptor below about how something's going to be interacted with and it generates reasonably parsable output at the moment. There is work going on, or there was work going on to give it multiple formats for output as well, which is similar to what Bandit has. The idea being that you can start collating output from all these different tools with a sort of a similar look and feel. So it's open source, it's easy to extend. It's very much, it's open stack-centric, but much like Anchor, it's not too tightly coupled to open stack. So if you want to apply this to some of the tooling that you've built to work with open stack, which actually seems like a fairly smart thing to do, then you have the ability to do that as well. So Bandit, Bandit is probably our most successful technical security project. It is a dynamic static analysis, which doesn't make sense. It is a dynamic static analysis engine for Python, what it actually is, is a static analysis engine for Python. Bandit finds issues in Python code, it can find a whole bunch of issues, I'm not gonna read them all off the slide there. It does it by first taking the Python code, putting it into the Python AST, de-offuscating certain things, and then looking for certain patterns. It's found a whole bunch of stuff. Travis, who is in the room here, has found a whole bunch of things and he's got a bunch of vulnerabilities to his name in Launchpad. It's had a whole bunch of development done. It's, like I say, it's probably our most successful project we have in the order of four or five big open stack projects now running it in their gate. And our ambition is to have every open stack project that uses Python have it running inside their gate. So this is an example of output from the Bandit test suite. And here you can see it uses pretty colors to tell you which things you think you need to care about. It gives you severity and confidence metrics. This is something I ran a day or two ago against Kola. These are some of the issues that it brought up. In fact, these were the only two issues it brought up. And the good thing about this is when this is in the gate, you'll get these results. You can then go and look. And if it's not a vulnerability, if it's a false positive, then you can mark it. And Bandit will know to skip it next time. But more likely than not, when you have things coming back with a high confidence, unlike some other static analysis engines, it's pretty good at finding vulnerabilities. So I'm gonna show you something now. So I can't really see any of you, so I won't really bother asking. But if you've used any static analysis engines before, you'll know they're incredibly difficult to install. So I'm gonna go ahead and hopefully we won't run out of time and show you how to install Bandit. Okay, so Bandit's now installed. Now, obviously that's just pipped. One of the big issues that you have with any static analysis engine is actually working on configuring it and tuning it and making sure you don't get too many false positives because they tend to be incredibly noisy out of the box. Sometimes that's because they rely on heuristics over time and sometimes that's just because they need a lot of tuning. They're very delicate machines. And obviously Bandit's no different. So as this video goes on, I'm gonna try and show you some of the more complicated ways you can configure Bandit before you can get useful output. The dash R there is very important for getting useful output. And there you go. That's static analysis with Bandit running over a thousand lines of code in a fraction of a second, finding some useful output. The point there is that it is not static analysis in the way that you might have experienced before. It is incredibly quick. It has good defaults and it is finding vulnerabilities and stopping vulnerabilities going into OpenStack code base today. So it's open source. It's plugin based. It's super easy to write new plugins. You can read a whole bunch more about it on the wiki or go straight to have a look at the Git repo. And as I demonstrated a second ago, it runs very, very quickly with just a standard configuration. So I wanna talk to you a little bit about where we are. We have a security blog. It's up on GitHub because that was the path of least resistance at the time. We're always looking for new people to contribute and the blog can be somewhere that's quite useful to get involved as well. We've only started blogging about things since the Mitaka mid-cycle and we have a significant number of posts on there now. If you're at all interested in any of the threat analysis work that's going on, then I would love to talk to you more about that. There's a lot of good information about that on the blog. We actually don't have threat analysis in this deck, which is possibly an omission. But there's a lot of good information up on the blog and we're very, very actively chasing the threat analysis stuff this week. So if you wanna understand more about that, then talk to myself or talk to Doug after this. So security project in summary. We have around about 250 listed sort of lurking people on Launchpad. Around 20, 30 people active at any one time. There's really opportunities to get involved at any level. If you just wanna help us out improving our documentation, if you wanna help us out by working on these interesting projects. I mean, Bandit's getting picked up all over the place. Ironically, two of the biggest companies I know that run it outside of OpenStack are Uber and Lyft. I'm not sure why it's sort of taxi companies that like it, but they do. And other people are contributing all the time. Anker helps solve a really big problem, not just for OpenStack, for other people as well. It's such a different way of approaching PKI that sometimes it's difficult to get traction on. So if anyone's interested in working on or adopting Anker, then please get in touch. And Cintra Boss, which is the new upcoming project we have very much on the coattails of Bandit, really looking at trying to actively improve security in OpenStack. There's scope for writing your own tools. The OpenStack Ansible Security Project is a great example of some works going on outside of Security Project as a whole because it should sit more with Ansible. But it's definitely something that we dip in and out of and help with. This talk is really, like I said at the start, really about trying to get more people interested in contributing to OpenStack security. We have, again and again, analysts, enterprises, everyone telling us, yes, your cloud is wonderful and I really like the way Nova does this and all the rest of it, but it needs to be secure. We need it to have certain security properties before we can take it forward. We see that time and time again. And there are a few companies like IBM, HPE, Rackspace who have thrown some money at making this work. I was fortunate enough to be pretty much dedicated to OpenStack for the last couple of years. But it requires more effort from more contributors in order to really get it to the point where we can make everyone really happy that we can deliver on security. So thank you for coming to my talk. Thank you to Travis, Michael and others that contributed to this. This is a collaborative deck, the intention being that anyone who wants to go out and spread the world about OpenStack security and try and recruit people can. So thank you and I'll take questions if there are any. Okay, well thank you for coming along and please contribute.