 Good afternoon. Hopefully, everyone had a nice lunch. We're starting to roll in a little bit, and it's about time to get started for our next session. My name is Brian Payne. I'm from Nebula, and I'll be here to talk to you today about security and private clouds. So really, the premise of this talk is to think a little bit about why security matters in a private cloud. That might sound funny to some of you, and to other of you, you might wonder why security does matter in a private cloud. I've had a lot of conversations, even with security minded folks, that say, well, in a public cloud, it makes sense. It's on the internet. You have a lot of security concerns that you need to be thinking about. But in a private cloud, it's behind your firewall. Maybe it's in a dev center somewhere, and perhaps it just doesn't matter so much. And so today, we'll talk through what a private cloud is, what it looks like, and why, in many cases, security does matter very deeply in a private cloud. I say, in many cases, because I can certainly imagine various private cloud deployments where maybe security is not the topmost on your mind, and that's OK. But the important thing is to recognize when it should matter so that you can do the right things about it to make it better. So let's dig a little deeper. If we look at how NIST defines public and private clouds, it's pretty simple. A public cloud is effectively accessible to anyone. In fact, what they say is it's provisioned for open use by the general public. Typically today, what that means is you have a credit card, and you can provide that to your cloud provider. And in return, they will provide you some instances. A private cloud, NIST says it's provisioned for exclusive use by a single organization. So in this case, you might be a member of a company, and therefore you can use the cloud. You might be a member of a nonprofit or some club, or you might just deploy a private cloud at your house because you have a need for a lot of computing resources. These are all examples of private clouds. The problem with the NIST definition, I think it's correct, and it's a nice, very high-level view of it, but it sort of doesn't get into what does this very high-level difference mean in terms of how these clouds actually get deployed and how they might look different behind the scenes. So let's talk about that a little bit today. Excuse me. OK. So first off, some private clouds can look a lot like public clouds. There's no doubt about that. So you could imagine a private cloud set up for a company. You can hang it off the internet, and you could access that resource just like any public clouds, with the only difference being that you're accessing it because you're a member of that organization. That's perfectly fine, and you could even do that private cloud in a complete and total vacuum so that it's not connected to any other corporate resources or anything like that. It's just a cloud that the company members can use. And in that case, your security concerns are, in fact, pretty much exactly the same as that of a public cloud because from a technology standpoint, presenting a public cloud is just limited in who can use it. A lot of private clouds are different, though. A lot of private clouds are deployed within a company or within some sort of existing network. The cloud probably isn't the very first piece of IT infrastructure that you brought into this place. You probably have existing things stood up, like DNS infrastructure, PKI, email servers, SIM solutions, NTP servers. It just goes on and on, right? So you have all of this already set up because you've been doing IT for decades. And now, within the confines of this private network, you're going to add a cloud. Is this cloud going to stand by completely separate and isolated from all this infrastructure, or is it going to integrate in with the infrastructure? I would argue that most of the time you're going to want it to integrate in because that's how it actually turns around and adds value to your corporation or to whatever the group is that's deploying it. So what does that look like? Well, if your cloud is integrated in with the email services, maybe it's using those servers to send notifications to the cloud admins when something's going wrong in the cloud. If it's integrated into LDAP, maybe it's using your existing directory to allow people to log into the cloud. In fact, maybe the integration with LDAP goes deeper. Maybe you can write to the LDAP server to create new groups, to allow people to provision groups and projects from the cloud that then propagate through LDAP and get saved there. Maybe use an NTP service just to keep your clocks in sync in the cloud. Maybe you send logs to your corporate logging infrastructure so that you can monitor those and understand when there's problems in the cloud. Maybe you have a DNS service where the cloud can actually modify the DNS so that you can provide DNS as a service. So sometimes people like to spin up instances and provide a name for it. And so that would require the cloud to have a deeper integration with DNS than a typical DNS client would. Similarly for PKI, maybe the cloud is using certificates to provide TLS to protect all the network communications going into the cloud. Maybe the cloud is doing rapid provisioning of certificates and has some sort of provisioning authority through your PKI system. And so it has, again, a stronger relationship with that service than a typical client would. And also with storage, how many of you work at a company where there is some sort of network attack storage, be it a NetApp or something there where you can go drop your files on and come back the next day and they're still there? Probably someone's one or two? I bet there's more than that. You guys are probably still sleeping from lunch. So what I have found is that as people want to deploy clouds into their corporation and to their enterprise, they're seeing that, well, hey, we've already got terabytes, pentabytes of storage that are set up. And we're not going to move that data into the cloud. We just want to connect it so the cloud can use that data as it does its computation. Sometimes those connections, again, become more trusted than a typical client because the cloud will need to talk to the management plane of the storage system. So what happens then is an attacker can actually come in to these systems. And if the cloud isn't set up properly, the attacker could leverage the cloud to do bad things, not just in the cloud, but throughout the enterprise. So going back to something I said in the beginning, I've heard countless times people say in private cloud security, probably doesn't matter so much because you're sitting behind a firewall. So one of the things that this diagram is showing you is that your firewall doesn't really matter. I really hate to tell this to you, but it just doesn't matter. Because what an attacker is going to do is they're going to do some sort of spear phishing campaign where they send an email to someone at your company who has access to the resource inside the network. That person's going to open up the email on their computer. It's going to run some sort of an attack. And from there, they're going to access the cloud. And then from the cloud, they're going to access whatever other resources they want. And all of a sudden, your firewall just doesn't matter anymore. And so if you built everything inside the firewall, thinking with the mindset, well, I've got a firewall. So it doesn't matter, then your services will probably fall down pretty quick. So in this case, I'm showing an attacker launches an attack just like I described. They might go through a desktop or a laptop that's inside your corporate network. They're leveraging that connection to get into the cloud. And because the cloud in this case maybe has a trusted relationship with the directory service, they're able to go in and modify the users or groups inside that directory service. Now they've got access to a lot of resources throughout the company. And they got it because someone deployed a cloud that wasn't set up properly. So what can we do to make this better? We'll get to some of the specific hardening things that we can do in a little bit. But first, I want to step back and look at a, this actually is a diagram that came out of the OpenStack Security Guide, which is a great resource for those of you that aren't aware of it. It's available online for free. And in the beginning of that guide, we talk about threat models and attackers. And this particular diagram was actually drawn by Rob, who's in the back laughing at me right now. It's a great diagram. And so I put it in here. I redrew it so it looked a little prettier, of course. On the bottom, you can see we have script kitties. These are like your really basic people. They don't even know how to code. They just run someone else's attack, right? We go up to individuals that are motivated. More capable groups. They have more resources, more skills. Then you get an organized crime and even the intelligence people, intelligence services. If you look at the width of this triangle at the bottom, you see the likelihood of attack. So how many of you operate a server somewhere? Say like an SSH server. And you actually look at the logs. Cool, hands stayed up. All right, so you've probably seen brute force password attempts, right? Where people are just trying to log into your service. Let's try admin 123. Let's try password, password, all these things. These are script kitties. They're just hitting everything on the entire internet hoping that something works. And they probably do get a certain amount of success. But they're just launching tons of attacks. All the way up to the top, how many of you have seen an intelligence service break into your network? Okay, I see some hands. I'd be curious to talk to you later. These attacks are going to be much less common. But it's very likely that if these attackers are going after you, they will succeed because they do their homework. And so you have everything in between. And so when you're starting to think about the level of security you need, the first step is to think about, well, what's one of these groups am I interested in actually protecting against? From there, you can start to think about, well, what is that group actually able to do? So what I drew here is a simple attack graph. You can actually imagine expanding this to be much richer. But I didn't want to make it so convoluted that you couldn't see it on the slide. So the idea here is you start on the left and you sort of work your way across. The example that I showed earlier started with a compromised user system somewhere within that private network. So that's the upper left there. And then from there, maybe you can access the cloud as either an admin or a user, depending on what sort of level of access that particular computer had when you get onto it. And once you're into the cloud, maybe you can launch an instance and do a VM breakout and find yourself on the control plane. Maybe there's a vulnerability in one of the APIs. And so you can exploit that and, once again, find yourself on the control plane. Maybe there's like a cross-site scripting vulnerability in the dashboard that you can exploit. So there's all sorts of options. Once you actually get into the cloud, then you can do all sorts of things. You can look at instances from other tenants. You can abuse the cloud resources. If you need a botnet, all of a sudden you've got a lot of nodes at your disposal. You can view all the data that's in the cloud, which I apparently have listed twice. The other one is supposed to be to view some of the computation in the cloud. You can leverage the resources that are within that private network. So we talked earlier about how the cloud may have a trusted footing into the private network and to some of those services. So we talked about how you can modify LDAP. But you could also view data on this external data store, like a NAS that's attached to the cloud. If there's VLANs that are going in and out of your cloud and to other portions of your corporation, perhaps you could then just tunnel through those VLANs and end up wherever those networks take you. That could be useful. And you could do spear phishing attacks. If you're already able to send emails through the corporate email server legitimately, well, that makes it much easier for you to send emails that look legit to people to then further your own exploit campaign. So this is just a simple set of ideas. I sat down and sketched this out in a few minutes. If someone really, really wanted to do some damage, you could imagine building out an attack tree like this that has many more nodes. Many more ways in, and just many more opportunities for the attacker. So the trick, as a defender, as a security person, is we want to make sure that the attacker runs into obstacles at each one of these steps along the way. So your cloud might be vulnerable to something that you don't realize, but you want to have a lot of compensating controls in place so that that single vulnerability doesn't allow them to walk through and own your entire cloud, and therefore own your corporate infrastructure as well. So I've been criticized in the past for scaring people about things when I do security talks. And so I'm going to try not too hard to scare you too much today. So I want to let you know that there's some good news out there. And for me, cloud is a really interesting opportunity to do some great security work. And it comes about because of a few basic things that almost anyone that's working in open stack is probably using. The first is orchestration tools. I listed a few of the more common ones up here, but in a sense, it doesn't matter what orchestration tool you're using. I'm not advocating for one or the other. What I'm saying is this allows you to roll out configurations very consistently. It allows you to update your software very consistently. And that's what's really important. If you read a security note that says the default configuration for Keystone had a weakness in it, and you go back and you check your notes and you say, wait, I happen to be using that default configuration. I need to change this. And I need to change it on all 100 of my nodes where I'm running Keystone right now. Well, it would be nice if you had a configuration management system, an orchestration system that lets you make that change, push it out reliably, and know that it's everywhere. It also perhaps would let you go back and audit and make sure that what you expect to be deployed is actually there. The next piece is known hardware and software. If you think about the job of a typical systems administrator, now way back a long time ago, I used to administer some Windows networks and some Linux networks. And I recall one of the pain points being that each of those systems was always a little bit different. You might image your Windows systems with a disk imaging utility like Ghost, and they start out straight out of the IT shop exactly like you expected, and everything's just wonderful. And then like a day later, the users install the browser of their choice. They've customized all these little checkboxes. And before you know it, you have all this divergence. And every system out there is just a little bit different. In some cases, they're quite a bit different. And then a couple years later, you buy a new rollout of different hardware, and someone brings in hardware from another division. Before you know it, like, there's just so much diversity. It's hard to wrap your mind around it all. Typically, when people are rolling out infrastructure for clouds, they have a little more control over what the software looks like when it's installed everywhere. They have a little more control over what the hardware looks like everywhere. Sure, you're gonna see some diversity in time and some evolution, but this knowledge of what's rolled out everywhere is really a security dream. Think about a lot of the classic security tools, like intrusion detection systems. The job of these tools is to look at your system and understand when something is happening that is different from what you would expect. So the more you know about your system, the better you can configure it and the better your results will be as far as determining what's wrong. Similarly, if you look at tools like SE Linux, mandatory access controls, where you put in really fine-grained security policies that say what's allowed to happen on the system, these typically really, really frustrate end users on laptops running Linux because they wanna do all sorts of different things than what the policy creator had in mind. In a cloud, you know what the cloud should be doing. It's running Nova. It's running some sort of networking service. It's running some instances. And you know this in advance and so you can set up policy correctly and expect it to work. So the story here goes that if you have nice orchestration tools, if you know what software you're rolling out, there's some really good opportunities for security. And I'm not saying it's easy, but you can take the time to do it right and it's worthwhile. So what does right look like? This is a somewhat generic picture I put together, sort of inspired from ideas from that OpenStack security guide that I mentioned earlier. The idea here is step one is to make sure that you have separation of concerns. So you have different logical networks. Maybe they're actually different physical networks depending on how you're deploying it. For outside of the cloud, the external, you have a control plane, you have a data plane, you have a guest network. You probably have a lot more than this depending on exactly how you're deploying your systems. And then what you really wanna be focusing on in addition to hardening everything is the bridge points, right? So think about an API service that's running in your cloud. Think about Keystone. So a user from the internet is talking to Keystone in your cloud. That's going from the external network right onto the control plane or the management plane. So that's a bridge. It's going across two different security domains and so you wanna be very careful with what happens there. And so Keystone or any service that runs one of these API endpoints becomes a service that you care about from a security standpoint and you wanna make sure it's implemented properly and it's not vulnerable to attack. Excuse me. Similarly, if you look at where the instances run, there's a boundary there. The instance is running within a hypervisor, say KVM, QMU, ZIN, LXC's, right? LXC's aren't really a hypervisor but it's running in some sort of container or hypervisor and again, it's sitting on a security boundary. So a user can come launch an instance and from there they could do a breakout, they could get onto your control plane and they could do some damage. So thinking about how you separate out these concerns helps you identify where in the cloud you need to be thinking about protecting it. Now I mentioned some of the OpenStack projects but then I also mentioned hypervisors. One thing that's important from a security perspective, I mean I know we're at an OpenStack conference and so we should be thinking about all of the OpenStack pieces but it's also really important to keep in mind that your system, your cloud will not be secure unless you get all the other pieces good too. All of the glue, right? So here we've got the Linux Insta, excuse me. All right, I think I'll make it through this alive. You've got the Linux distribution that you've installed your cloud on. You've got your hypervisor, your messaging queue, your database, you've got all these different pieces that come together to form your cloud and you wanna make sure that they're all installed and secured properly. Interesting side story, when we were writing the security, the OpenStack security guide one of the suggestions that came out of the room from all the authors as we were sort of brainstorming this was that when you're connecting to a database every node that connects to the database should probably use a different username and a different password and if you're provisioning multiple clouds those passwords should be unique across all the different clouds too because this allows you to do really good auditing and all this kind of stuff but then there were some people in the room that said well geez that's exceptionally hard. How do I deploy a database with all these different accounts and manage this and handle all of this? And the answer of course is you go right back to what I was talking about before your orchestration tools and you let them do it for you. So you develop it once and then you let it happen multiple times for free and that's the beauty of it and that's where we can really get some security wins as we deploy these clouds. All right, so digging a little bit deeper we've got these threat actors, right? These bad guys that we talked about before everything from the intelligence services all the way down to the script kitties. We've got all the various ways that they could get into our cloud and we started to think about how can we protect the cloud by breaking us out in different security domains. Now the next piece is to think about what are these guys actually gonna be doing to me and how do I need to protect myself so that they can't do what I showed in the beginning. They can't break into the cloud and then hop around and break into the rest of your enterprise from there and the way that I like to think about this is to start to walk through all the various places that someone could attack you and then think about what do I do to mitigate that. So I've listed some here. This is I think hopefully just big enough that you guys can read it all in the back but I'm gonna walk through some more specific threat things on the next few slides and dive in a little bit deeper to give you some more concrete content here. Okay, so one of the biggest threats is information leakage and you would actually be surprised perhaps at how many OpenStack clouds are deployed today that do not use TLS to protect their network endpoints. So I will say this once and I'll say it again every five minutes until the rest of my talk. You should use TLS for everything. Protect your endpoints with it. Protect your web dashboard with it. Protect your log feeds with it. Yes, we just went through Heartbleed and there are complexities and complications with the protocols around TLS but it's still basically the best thing we have. So use it and keep it patched is the best advice out there but that's not the only way to leak information is across the network. There's interesting attacks that you can get from basically looking at cash effects on a single host and there aren't really good answers for this today. So if you're worried of things at this level you might need to keep different tenants on different nodes, for example. But for example, recent research has showed that using these cash effects and timing effects you can actually do things like read AES keys from a pure VM on the same node. So that's kind of bad and it does sort of influence how you might think about deploying a cloud. There are ways that you can actually set up security policies so that different tenants would never be allowed to deploy VMs on the same node and so if you're thinking that your users of the cloud might have these kinds of sophistications and capabilities that might be something to consider. VM breakouts, I've mentioned this a little bit before but I haven't really gotten into defining it. So who here is familiar with the notion of a VM breakout? Also known as like a VM escape attack. All right, only a few. So this is an interesting one. Virtual machine, a lot of people refer to it as a security boundary of itself and it can help but it's not a panacea. The thing about the virtual machines is that it's software like everything else in this world and so, and not only is it software but it's relatively complex software. So you're emulating hardware in software and you gotta figure there's gonna be some bugs in there somewhere. What a VM breakout means is I can run code inside an instance that will actually exploit something on my virtualization layer that will allow me to then turn around and run code on the host OS itself. Now, if you were on a desktop system maybe this wouldn't be so bad, right? Maybe you're running Windows in a VM on top of your Mac OS and you're just doing it for convenience and if someone broke out from Windows and got into your Mac, eh, maybe you don't care so much. If you're in a cloud, not only did someone just cross into the management plane but now they're in a position of power where they can potentially see all of the other instances running on that node and if your management plane is not locked down properly perhaps they can get from that node to any other node in your system with relative ease and so the problem just sort of unfolds like dominoes. So what can you do? In the KVM world, there's ways that you can use things like SE Linux which is a mandatory access control policy to provide containment around each virtual machine. So each virtual machine would get its own context and one context cannot talk to another context. So the idea being that if you were to break out you're still confined by the SE Linux policy and very limited to what you could do on the system. There are similar technologies available for things like Xen so I definitely encourage you to look into them. Anytime you're using virtualization in a multi-tenancy environment these kinds of controls are really really critical and again since we understand the software that it's being rolled out on it's not gonna be quite as painful as rolling out things like SE Linux on a desktop machine. There's additional hardening steps you can do. You can make sure that things like QMU which provides all your device models doesn't have extra software in it. Strip out all the drivers that you don't need. Use compiler hardening techniques. Just do everything you can to harden this layer because this is an important layer. The breaking down the security of this layer pretty much breaks down the security of your whole cloud and it's stuff like this that really destroys confidence in cloud systems. Finally, at some point in time you have to assume that someone's going to actually maybe get out and you wanna make sure that in the event that they do get out they can't get very far. So you wanna deep privilege your node so that it can't do much in the cloud. You wanna make sure that you're doing boot time attestations if you can to verify that that node came up and is still running the right kernel, the right hypervisor, the right boot loader, the right firmware, all these pieces. So all of this together can provide really strong defenses against someone who's attacking your node but nothing's perfect and so you still need good security update policies as well. Excuse me. So we just talked a little bit about what happens if you have a node compromise but more generally you can have a control plane compromise or a management plane compromise. In these cases what you're actually seeing is maybe someone came in through a breakout like we just discussed or maybe they came in through an API endpoint that wasn't configured properly. For example, what do you think the chances are that one of the open stack services somewhere has code that says exec shell equals true. Anyone seen something like that? It happens, right? And when it happens if what they're executive is user input then you might have yourself a vulnerability. So when someone gets into your control plane you don't want them to be able to get very far. Again this comes down to you want to have specific passwords that are used on each system. You don't want to be able to just sudo and log in everywhere. You want to have firewalls that restrict the flow of traffic to only what's necessary. All sorts of just really best practices, hopefully nothing that I'm seeing here is really groundbreaking in this particular slide but we need to remember to do it. I think often times when we're deploying open stack we just start celebrating and popping champagne quirks when we spin up an instance because it worked which is really quite great but there's a lot of work beyond just getting it to work. Primary focus here, limit the damage from a bad actor on the control plane. Today you basically have to assume that someone's going to get there and make sure that there's not a whole lot they can do when they are there. You want to be tracking vulnerabilities upstream. This could be an open stack or in any of those other glue pieces that you're using and be prepared to roll them out quickly. Be prepared to triage them quickly. And I would also argue that if you have things like compensating controls in place it helps a whole lot here, right? Because if every single vulnerability that comes across is a critical level of vulnerability for your cloud because there's nothing that mitigates it then you're going to really be in for a lot of long nights as you're constantly rolling out security updates. It'd be nicer if you see a breakout vulnerability come through and you can say, well, I'm actually okay with this one for now. It doesn't have to be critical because I have all these compensating controls in place and I'll roll this out in the next day or two instead of not going home right now. Okay, this is the last one. This sort of starts to bridge the boundary a little bit. I actually spoke with some people earlier today and this topic has been coming up a lot so I think it's on people's minds. We talk a lot about what you can do to secure the cloud itself and then often we punt on what you do to secure the instances. It's easy enough to say, well, you have all these instances that are running and they're not at all different from all the systems that are running across your enterprise and so you just apply the same sort of policies and controls that you do there and you make sure that they're patched and they're running all the right things. It's a nice thing to say but sometimes it's not entirely true because instances can be extremely transient. It can be hard to track what's going on there and frankly, shouldn't we be able to use the cloud to make the instances more secure? Shouldn't we be able to use the cloud to simplify this process of keeping them up to date of auditing what's running? It seems like we should and so I think there's a lot of services that the cloud can provide to make auditing and managing your instances better and this is an area that I think we need to do a lot of work in to be thinking about what sort of innovations we can do there. One such thing that you can do is you can provide entropy to your instances. Now, entropy is just random data, random numbers. Turns out it's extremely useful for things like key generation for cryptography. You might think, excuse me, you might think, well, I don't actually ever have to do cryptography. I'm not like a hardcore security guy except I bet you SSH into your instances and so you are doing cryptography. And I bet you're doing other things that you don't even think of as cryptography but in the shadows behind the scenes, crypto's happening. And so entropy is very important to protect all these things. Turns out when you're running a virtual machine, entropy is usually not too easy to find. So as a cloud, it's something that you can provide. You can take hardware random number generators, you can wire them through your cloud. If you do it properly, maybe you have different sources, multiple vendors and then you find a way to get these to your instances so that they can actually get a good stream of entropy. It's just one example of ways that we can provide cloud level services that can make securing your instances a little easier. Okay, so to come back full circle, if the cloud is gonna be deeply integrated in with the enterprise network, then it becomes really, really important that your cloud itself remains at least as secure as everything else in your enterprise. Now, if you run an enterprise network where you don't care about the security of anything on there already, then hey, you're golden because your job's really easy. You add in a cloud and you still don't care about security and you're good to go. The thing is, I don't think that's where most of us are today. I think most of us are deeply concerned about security across the enterprise. We don't want corporate espionage to happen on our watch. We don't want information stolen and we don't want resources taken away from us. And so when the cloud comes in, the bar in the cloud needs to be higher because the cloud integrates in with all these pieces of your infrastructure often at a privileged level. And so you need to think about what are the communications going in and out of the cloud and how do you protect all of those? And you need to be thinking about how can the cloud itself be locked down so that it doesn't become yet another stepping stone across an attacker's attack tree like I drew earlier. And I think if you think about it that way and you can think about it holistically like that, then you'll be good to go. Just remember getting the cloud up and running is step one. Securing the cloud is step two. And I think step two is oftentimes a little bit harder than step one. With that, I'm happy to take some questions. And please do use the mics because as you might have noticed, the audio in this room is not too good. I've got a question. Have you thought through like VM uploads, right? So VMs get uploaded into a cloud. How do you do things like looking through them before they are unable to be provisioned or what not do? Is there a place where you stick them until a scan can be performed against those and then they get elevated into being accessible by the cloud and put it into a catalog or what not? So you're talking about the images that you would then turn around and run in the cloud. Correct, yeah. Right. Yeah, so there's a variety of things that one could do here. And I think there's actually a lot of really interesting opportunities here related to this. I'll sidetrack and then I'll come back. One thing that I would love to see, and I'm gonna be trying to work with people to make it happen, is imagine if you could take your image service, Glance, and just point it at your Ubuntu or your Fedora upstream server where all the cloud images are and give it maybe a GPG key so it can validate them and say just make sure that I'm always running the most up-to-date trustee cloud image or the most up-to-date Fedora 20 cloud image or something like that. And so that whenever someone launches a Fedora 20, they get the one with all the latest security patches, all the fixes. That, I think, could help quite a bit. But then you've got the custom images, which I think is more of what you're getting at. So someone has some random custom image, the upload, and you wanna make sure that it's not malicious. So yeah, I mean you could imagine, I actually haven't seen that fully integrated yet like a automatic upload an image, scan it, but it's not a big stretch to imagine doing something like that. The one caution is that it would be very easy for me as an attacker to upload a benign image, you approve it, I launch it, and then I inject my stuff over SSH or something, you never saw it coming. So just don't wanna get a false sense of security from something like that. Yeah. Go ahead, question about, OpenStep used a lot of open source projects. So what's the best practice to track the security what I believe from those open source projects? Another question in the future, are we gonna conduct some secure review about those open source projects before we use them? Okay, so I think first question was, how do we track the security of the OpenStep projects? Wonderful, so I'm involved with the OpenStep security group which does a lot of work in this front. There's basically two primary resources that you wanna watch for. One, which is actually put out by the vulnerability management team of OpenStack, which is a real small group. They assess vulnerabilities in the product and then put out what's called an OpenStack security advisory. Those advisories are kind of like CVEs. They tell you when there's a vulnerability and you need to go update your code right now. So be watching for the OSSAs because they tell you when you have to update your code. The security group complements that by putting out OpenStack security notes. These are sort of little pieces of configuration guidance that are useful for deployment of clouds. So maybe the default configuration wasn't as strong as it should be, or maybe there's a concern about using Linux containers as a security barrier, things like this. These little notes of guidance will come out to give you steps along the way that can help out. And so between the OSSA and the OSSN, I think you have a nice collection of resources there. Your second question was about threat modeling, looking at the existing projects. And actually the security group has been working for about the past six months and we've been spinning up projects to do threat modeling and assessments of the existing OpenStack projects. They've started with Keystone, perhaps for obvious reasons, being a very security critical service. And they are already like shoulder deep and Keystone threat analysis. There's documents out there that are available. I actually just tweeted links to it today. So you can, oh, my slides are gone. You can go find them. BDP security is my Twitter handle. And there we go. So there's work on Keystone. They're gonna be working on Novinnext and the idea is just to keep stepping down through the projects. That'd be a great thing for anyone in this room to be involved with if you're interested in threat analysis work. Just a heart bleed related question. I know that there are patches, but the second piece of that is the certificate revocation. There is no certificate store in the core of OpenStack. So what's the best practice for OpenStack implementers on certificate handling since I know you recommend TLS everywhere? Yeah, so how TLS is handled throughout the cloud really varies. There was actually a nice blog post by Nathan Kinder at Red Hat recently about the various ways you can deploy TLS in the cloud. And so the right answer in terms of how you manage your certificates is gonna depend greatly on how TLS is deployed. So that's kind of a non-answer. Maybe a better answer is there's a lot of people looking at trying to standardize some of those ways using things like Barbican as a backend secret store combined with some sort of middleware layer depending on how people prefer to deploy their clouds that would actually present those certificates to stud, pound, Apache, Nginx, whatever it is that you want to use, and then have a nice way that you can actually auto those and quickly update them and that kind of stuff. But again, it's gonna depend on your specific deployment. So now we did actually release a OpenStack security note that talked about Heartbleed and talked about how you can go through your cloud and do the proper updating and stuff like that that gets into some of the OpenStack specific details. So that's probably worth referencing if you have questions. All right, thank you everyone so much for coming out. I'm happy to take additional questions on the side.