 I'm pretty loud talker, so I don't know if I really need the mic, even though it is being recorded. I guess that makes sense. I guess we're good to go. Keep it on track. Hello. My name is Andrew Hay. I am the Chief Evangelist at Cloud Passage, and we're going to talk today a little bit about, and I apologize for the guy with the camera, I pace when I present. Talk about securing OpenStack infrastructure as a service environment, mainly for someone who wants to run, not necessarily a SaaS application, but run a server in OpenStack. So a little bit about me. I am a former industry analyst. I was with 451 Research, and I covered security, all security. Cloud didn't come up that much in my group because there were just so many other security things to tackle at the time in different areas. I'll just talk about that. I used to work as a security practitioner. I worked in the Information Security Office of a university in Western Canada at the University of Lethbridge. You've probably never heard of it. I also worked at a bank in Bermuda, which you might think would be awesome, but it wasn't, and I can tell you later about that. I also was a product program and engineering manager at Q1 Labs, which was acquired by IBM last year, didn't see a dime, and I was a Linux guy. I worked at Nokia doing a lot of support for their security products on the various BSD flavored Ipso versions and a lot of Red Hat servers. Before that, I worked at a bunch of ISPs doing general network system administration for various flavors of Linux. I've also written or co-authored three books, three books that you've probably never read, so we'll move on. Like I said, I work at Cloud Passage. What we have is a security product built for cloud environments to secure it. So we can do firewall, server vulnerability management, and so that Misha, where'd Misha go? Oh, good, front and center. So that Misha feels better. This is not a product pitch. I don't generally like doing product pitches, especially not to technical people, so we're not going to do that today. And these are some of the things that we're going to cover. So a very brief overview of OpenStack security, and I've got a kind of funny anecdote when I was emailing back and forth with Rob who was just up here. And then why security's hard, and we'll get into this a little bit. We do have a very short period of time before the break. So whirlwind tour of the architecture. And I am Canadian, so no making fun of my accent. Apparently I have an accent that only Americans can hear. They have a fine tune to hear for it. So this is the general architecture that I'm sure you've all seen on the OpenStack documentation site. Of course, you probably know it more by the individual code names, as with any project code names are much easier to remember. Now, the security in quantum, very network level. So we can have individual, we can have a physical router and we can have a shared network for all of our servers. Very convenient, not so secure. We can have some logical segmentation so that we can, in the essence of VLANs. We can have virtual routers that are separating people around. Actually, this is probably a good place to explain my email conversation with Rob. So I went through my presentation when I was creating the slides and I'm like, okay, I've got three mechanisms for security in OpenStack. That can't be right. There's gotta be more. So I email Rob and I'm like, hey, do these sound right to you? And I could kind of hear him sigh from across the pond and respond with, yep, that's it. Those are the three, great. So probably the coolest thing about quantum is you have this API that you can plug third party products into, such as NYSERA. Everyone's heard about NYSERA? You can do virtual port isolation. You can do a lot of these really cool things that you could always do on your physical infrastructure, now in VM form. Very cool, very helpful for security, but, as we'll see in a little bit, this, probably a bad idea unless you're running a private cloud, where it's very small and you control everything. This, a little bit more secure. This, very ideal if you're doing public cloud environment. But, not easy to maintain. Some of the other things, Keystone. Who here has heard of Keystone? See, you guys have been here all week, I just got in last night, so. If we wanna start doing authorization, authentication, Keystone's our go-to thing. Does anyone have concerns that these technologies or these security mechanisms thus far are just for the open stack product? Nothing higher up, we're not really securing end user type things. Like operating systems, applications, data, anything like that. Yeah, one person's concerned, awesome. You were here in the last session too? No? Yeah, you were, all right. So, I'm not gonna go over in great detail of Keystone, cuz I would only assume that someone had a huge Keystone overview earlier this week that I couldn't make it to. But, definitely a mechanism to employ additional security for open stack. Then we have security groups. We have our IP tables firewall. My colleague of mine actually referred to open stack as a big ball of Python. Which I thought was kind of apropos. And so we have security groups where we're doing firewalling using IP tables. This is great, but at scale, doing these manual rules, not all that great for end tenants who have, if you're running a public cloud on open stack and you have, let's say a thousand customers, each with a thousand servers. I doubt they're gonna wanna call you and say, hey, can you allow just this? Or can you block these certain applications and ports for me? I'll wait, it's fine. Whatever you can turn the ticket around, that'd be great. That's not scalable. VLANs, fixed range, before, we can actually do some separation, some logical segmentation. Who here trusts VLANs? Explicitly. See, you had your hand up and then you put it down when I said explicitly. Why is that? Why do you not trust VLANs explicitly? Do you trust anything explicitly? That's probably not a safe question. Then we have this, has anyone seen this Python Keystone Client script? It's in the documentation. It's pretty cool. It will help you set up your authentication. This is great. Your OS password of secret, admin, admin. Gives you, again, another layer of security. But what is missing is this concept of host-based security. Because unless you're in, so who here works for an infrastructure provider that provides cloud services via OpenStack to customers that are outside of their own organization? So who runs like a public cloud? Okay, quite a few hands. Who here just runs OpenStack internally? Private Cloud, no one else touches it. So it's a good mix, roughly of the people that put their hands up, 50-50. Who here uses OpenStack to secure the operating system up the stack? So from hypervisor, up. Not a single hand, and that's because there are no mechanisms in place to help you do that within OpenStack. But that's not unlike most cloud providers. Amazon, for example, they are not, they have explicitly stated in their, I can't remember the terms of use or some other documentation, that states you are responsible as the intent to secure the operating system up. They will take care of the virtualization layer down to the plugs. That's their responsibility, but they're putting the rest in your hands. That hands-off approach appeals to some people, but for people moving to cloud environments, if it were me, I'd be a little leery to just say, okay, yeah, okay, I'll handle it. I'd like a little bit of help from my provider. That's just me. So this, who here believes that OpenStack security groups, the three mechanisms that I said earlier, who believes that that's enough to secure their OpenStack cloud for both their end users and the infrastructure itself? Anyone? No one's so bold as to say that they believe that? All right, so Mike Tyson had a great comment. Has anyone ever heard this? So there's a lot of, say what you want about Mike Tyson, but this is the smartest thing someone has ever said. Everyone has a plan until they get punched in the mouth. And it's been used several times. It's actually accredited earlier on. I can't remember the boxer, might have been Joe Frazier. But when people think of Mike Tyson, they don't think of something this profound ever. Now, why do we want to secure the images? Well, securing the images, it's really about the lack of network-based security in cloud environments. Network-based security is only so good. Who here runs a traditional data center for their organization? Yeah, do you rely exclusively on network-based controls? No, you employ some measure of host-based controls, right? Why? It's defense in depth. It's good practice. It just makes sense. So why wouldn't you do that same thing in cloud environments? You're not going to exclusively rely on network. Sure, network controls are a helper. They have a very broad protection net that they're throwing out there. But it's like an umbrella. It will protect the rain from falling on your head, but you're not going to be protected if the wind starts blowing and blows rain up in your face, your pants are still gonna get wet. Now, the ultimate target is your endpoint. So why don't you secure it? Again, it makes sense. And who here, so like I said, I'm Canadian. Who's seen Bowling for Columbine? A couple years ago, Michael Moore movie. So his big thing was, I hear that Canadians don't lock their doors. You can just go up and you can open the door. I'm watching, I'm like, yeah, so my door's unlocked right now. I look over at the door, I'm like, yeah, unlocked, middle of the day. And then he goes up and he starts going into some people's houses. And I was like, oh, I couldn't believe your door's unlocked. And they're all like, yeah, we always leave our door unlocked. But then you keep watching the movie and you think, crap, I should probably lock my doors after everything that I read. As I now live in San Francisco, I lock my door when I'm in the house. When I'm running to the laundry room, I'll lock it and I'll bring the key with me. Because I'm definitely more paranoid living in a big city than in a small city of 80,000 people. And that's just like public cloud. You're living in a big community now of people. And a lot more attackers than when you had just your individual data center hidden behind your technical security controls that you had beforehand. And because it is the 20th anniversary of a few good men, if you want to feel old, please feel free, I felt old when I heard yesterday. We need technical controls where people are standing there with the guns on the wall, watching, seeing what's going on. But we also have to protect the munitions dump. We have to protect everything behind those walls. The reason is, well, our servers are more exposed than ever, especially if we're talking about public cloud environments. We don't have the luxury of those technical controls or the tangible network controls that we once had. At least not in the same manner. So servers are exposed, which is great for teleworkers, remote workers, because they can get to these cloud servers quite easily. The problem with allowing everyone to get to the server quite easily is that if we have configuration issues, any sort of security holes that can be exploited, odds are, given enough time, they will be exploited. Well, we could using our IP tables and our OpenStack level security controls block access to everything. We'll firewall it, so people can get in. We'll allow people that we explicitly know should have access to get access in here. And that's great, because that's gonna deflect the unknowns that are trying to get at the servers. And it's going to allow people who need access to have access. But what about the guy sitting in Starbucks? Well, they're really not gonna get in. Has anyone here worked in a support capacity before? Show of hands? Yeah, quite a few people in various roles. If you've not worked support, it's really something that I recommend everyone do at the very beginning of their career, because it kind of level sets your expectation of humanity for the rest of your career and the problems that they have, and the way that they convey that life ending problem to you, like getting a call through the morning, I can't reset my password, can you fix it? Sure, I would love to get out of bed and reset your password for you. There's nothing I would like better at three in the morning than to do that. So when you get the CEO who's over in China trying to connect to the CRM server or some sort of server that you're hosting in your cloud environment, you get that really nice call at three in the morning that I always hated to get. And what happens when we really start scaling out our deployment and start cloning servers and adding complexity to the environment? How scalable is using OpenStack level security controls and IP tables rules to block, to firewall off access, to prevent access to these servers? It just doesn't scale. It's not a scalable solution, especially for the tenant. So let's just say we've added application servers, additional servers, so we've got all these interconnected mesh type communications. Are you going to really like sitting on the phone or sending in an email to say, hey, can you open up access between this network and this network? Or this cloud and this cloud and I'll wait. It's fine. A couple days is not really critical. No, you probably want to be able to configure this by yourself. And for dynamic deployments, well, this is definitely more of a public cloud. So this goes back to that shared model if we had an OpenStack deployment where everyone was using a shared slash 22 network or whatever. And we didn't have any sort of logical segmentation in place. Well, the dynamic nature of DHCP, if you drop offline and someone drops, someone comes online, it's only a matter of time before someone can get that same IP address. And if you have IP tables rules in place to allow the communications back and forth, that's kind of an issue. Because now, Nasty McBaggy here has access to your slave database server. Not ideal. So, my talk really is about securing the, kind of a long preamble, but there's the whirlwind tour, securing the images, the actual operating systems that people are installing in these environments. Now, you might be installing them for people. You might be allowing your customers to install them. It doesn't really matter. You should either be able to handle these things on your side or at least provide people guidance. Give them some tips on how to secure from the hypervisor up. So let's pretend that there really isn't a network. Because a lot of people say, sorry, public cloud, it's adjacent to the internet. And I use that all the time. We don't have those network barriers to cut people off or block them from coming into our networks. Because we're sitting on the network edge. We are adjacent to the bigger cloud. Our cloud is adjacent to the big cloud. Now, if we have no mechanisms on the network to prevent our servers, prevent our servers from being scanned, probed, attacked, taken out, what do we do if we do not have these controls? Well, so I got this, isn't that a great picture? I got this from a university somewhere in Southern Alabama. I'm not sure where. And this was actually there in their policy for secure server image deployment. So basically building their gold standard and what they're going to roll out. And this is probably the most important part of this section here. By definition, any machine that is accessible on a network and running services is potentially insecure, pretty profound. And then this was actually in it, i.e., pretty much any server. That was in there. So basically, they're saying, hey, if you have a server and we have it connected to the network, it's potentially a target and probably or has the likelihood of being insecure. So in the last session, we were talking about attack surface area a little bit. What the main goal of securing or locking down the operating system image is not to have complete security. Because if you believe you can have complete security, you're probably a little bit, you're setting yourself up to fail. I was gonna say delusional, but you're setting yourself up to fail. What you wanna do is reduce the attack surface area as much as possible. And as much within the tolerance of the business for restricting that attack surface area as you can. So these are just five things that you can recommend or implement on your operating system images, running in your public, private, hybrid, any sort of cloud environment, physical server environment, virtualization environment. As you can see, nothing here is installing a tool. You're not installing any tools to do any of this. These are things since the dawn of time that we have been doing that a lot of people have forgotten because of their heavy reliance on network controls and or their inability to keep up with doing these kind of things on every server because let's be honest, a lot of security in IT shops are one person operations. And there's so many fires to put out. When I was in Bermuda, I was the security guy. And I was running around, I was gonna say with my hair on fire, but no one's gonna believe that. I was running around like crazy trying to fight all of the problems at once. And it was just a prioritization, which as you know, in any organization changes daily. So it just shuffles around. So of these five things, I'm gonna go through and we'll just talk a little bit about it. And whenever I look at this slide, so this is the kind of thing that in the last session they were talking about. We have to go out and we have to talk to security people. Security people don't really talk to other people outside of security circles. And there's this concept of the echo chamber where we just sit in debate, usually on Twitter, about how great or how society and companies should be doing X, Y, Z because it makes sense. And they should know better than this. And it's our job to educate them. We don't educate them. We just debate it even more saying, yeah, we totally agree. And then it just turns into a Dr. Phil episode of the, I used to love watching highlights of Dr. Phil because he had these very astute comments like, hey, you're fat because you keep eating. Really, well, thanks, Dr. Phil, that's changed my life. As you can see, it did not change my life. So we'll disable some unnecessary services. Well, if you are just installing your operating system and leaving the default config, default things installed. So I saw some people from Red Hat walking around. Remember back when Red Hat Enterprise Linux 2 came out and you do like the base install, it installed way too much for just a base install. Ubuntu's kind of gotten a little bit better with that. And Red Hat definitely has gotten better. They're definitely more enterprising now than they were back when I got my RHC years ago. But there was a lot of packages that were installed that really shouldn't have been on there, but they were on there for convenience. So that people didn't have to search around and install them themselves. But if you're running a web server that has no requirement for a database server, why do you have that on there? Why is that service enabled? What about our services? Anyone here still use our sync in their corporate environment? Misha, no? You still use our sync, really? You work at a university? Cuz that's usually where I see it. Ah, pin pointed. All right, so yeah, our sync really should have gone the way of the dodo. And some of the other services because there's so many secure replacements right now. I bet you have a lot of Perl too, don't you? A lot of Perl apps developed internally. Maybe some cobalt kicking around. No, clipper. FTP, there are secure alternatives now. It is 2012 to FTP. Telnet, again, if you tell me that you need Telnet to update your routers, my response is you should probably update your routers, either the physical hardware or you should update the iOS that supports something like SFDP or SSH or the secure file transfers. Because the latest and greatest versions, and I know they don't work on old hardware. But odds are if you're running hardware that can't be upgraded, you either don't have time, which I understand. Or you're just running these things without a service contract and you're cool with that. Remove unneeded packages. Again, if you're running a web server that doesn't have a database requirement, uninstall MySQL, uninstall Postgres. If you're using Apache or if you're using Nginx, get rid of the other. You don't need everything on there just in case you might use it. That's bad, that's really bad. That's a bad practice. Securing access files and directories. Who here knows what the chmod command does? Yeah, what does it do? Anyone? Yeah, change file, access permissions. What about ownership? How do we change ownership in the Linux world? C.H.O.N. Yes. How many people verify this on their servers that they have deployed in the organization on a daily, hourly, infrequent, every couple of year basis? It's not something that a lot of people still do because once they, it's the Ron Popio method of deploying servers. You set it and you forget it and you walk away and go to the next thing. Unless you have this sort of continuous monitoring and continuous checks, you're not gonna be able to catch this if someone's going in and changing ownership, changing permissions. That could be indicative of someone doing something either malicious or something that they're not supposed to be doing, not necessarily malicious. They could just be doing it either out of spite or by mistake. But you need to be alert on this and you need to make sure that these things are continuously monitored. Now what about insecure default configurations? How many people launch OpenStack instances and just let them run? Or better yet, who has heard the argument that we're just going to run this and it's only gonna be up for about 15, 20 minutes so we don't really need to do much to it? Anyone? I've heard that quite a bit, actually. People say, especially people that are doing big data, analytics, some sort of massive number crunching or like very short term spot instance kind of thing. Like, let's launch this, we'll do our work and we'll shut it down. So we don't really have to care about security. Well, the thing is you're launching these things and they're staying up for a period of time. And in that period of time, someone could just happen to be scanning the network, get some sort of alert that hey, this thing is susceptible. Like, now I can go in and if it's something that is, let's say it's a pharma company that's doing something with an intellectual property or driving data that could be worth billions of dollars. Well, in that 15 minutes, what's to say that they can't steal a chunk of that compute data and then derive something from it or get something meaningful from it? It's the window at which it's up there. I don't think that is a strong enough deterrent to not secure your systems. Apache, anyone here administer Apache servers at all? Yeah, do you know what these mean? Disable server tokens and server signature directories? So this is all about fingerprinting your Apache server. So this will tell you exactly what, if you have these enabled, these will tell you what version of Apache is running. And if so, it will be susceptible to X number of vulnerabilities if it matches all the various search engines out there. There's quite a few scary search engines out there where you can just search for versions of this. You can use Google to search for a particular Apache version coming back in the banner, kind of scary. Loadable modules, if you're using Apache, do you need to load every module that's in the default config? Probably not. You should probably only load the modules that you need to operate the web application that you're serving up to your customers, your employees, whatever. Don't load it just because it's the default. Go in and actually change things. Now, this whole concept of access only from trusted sources, like I said, it doesn't scale well, but it does give you some measure of control. Whether you use the firewall on the operating system, which I highly recommend, because again, as I said traditionally or conceptually, we are now operating a servers at the edge of the network. And if you're an end customer, you probably don't want to put all your eggs in the service provider basket to make sure that their virtualization stack is providing you the security that you need to operate your servers. So use the firewall. Linux firewalls, sorry, Linux firewalls, Windows firewalls all have operating system-based firewalls that will prevent people from accessing services, will likely deny, if configured properly, people scanning. Like it'll just deflect them. It'll let them move on their merry way. It'll obfuscate these servers from potential or would-be attackers. Anyone use failed to ban on any of their Linux servers? It's kind of cool, after a certain number of criteria are met, it'll block the IP address. Usually SSH, someone trying to hammer SSH over and over and over again, hits a threshold like, yeah, we're gonna block them. So it puts an IP table's rule in, that IP cannot get there anymore. Who here has a large environment where they allow servers to hop to one another for convenience sake, right? And it's extremely convenient. I don't know about you, but I am terrible at remembering my passwords. So whenever I used to deploy servers, I'd make sure that I would just have to remember for one, and then I can just hop around my environment using keys instead of having to remember password. Which made a whole lot of sense for transferring config files, transferring log files around to do analysis. Was it secure? No, because you're creating a single point at which someone can get into your network, be it physical or cloud oriented, and then hop to the other servers without a password if they're using your key. That's kind of a big problem. Now, if you have the ability to do on-demand access with two-factor authentication that could, let's say, open up a firewall hole after you authenticate, that's very convenient. This helps with that problem of not having the ability to block everyone, but still facilitate access. So the two-factor authentication mechanism would allow you to log in whitelist your IP address for a certain period of time and give you the access you need. So it gives you that obfuscation and also the added security of two-factor authentication. So what do we want? We want security, most of us anyway. And we want it now. But how do we get it? Not a lot of people really know how to do this security thing. I gave you concrete examples how to reduce that attack surface area. Well, doing it manually is a pain. So I spoke at the Puppet Conference a couple weeks ago, and there were so many people there, and it was all about automation, automation, automation. I noticed the ops code guys have a booth out there with Chef. So if you have not taken a look at Chef, it's pretty cool. So you don't have to be an Uber geek programmer to be able to automate the configuration and the locking down of these servers anymore because people have written libraries. There are people far lazier than you out there that have already done this. And I mean lazy in a good way. Like there's two types of lazy. There's lazy, I don't really want to go for a walk because the Hunt for Red October is on TBS and it's only on TBS every Saturday. So I think I should lay on the couch, which is me. And then there's the good lazy where it's, oh, I really don't want to go and manually configure a thousand servers. I'm going to automate it so that I can just go boop and it's going to go out and it's going to verify all the directories, have the right permissions. It's going to change them if they don't have the right permissions. It's going to disable and uninstall packages that shouldn't be there based on my security policy. These are very good tools. And I'll note, I don't work for any of these companies. CF Engine, Puppet Labs, Opscode Chef, these are really cool. If you haven't tried them, try them. They're awesome. There's user communities popping up in pretty much every major city around these tools. Now, we're definitely running out of time. So in terms of locking down your systems, these are some great resources. Don't worry about scribbling them down because I'm sure these slides are going to be released. But these are most of the highly referenced pieces of documentation for creating a secure baseline or hardening your server. With the exception of this one, the Halo Configuration Policy Rule Checks. So these are our Cloud Passage Halo security checks. These will give you a very good idea of the certain things that you should be checking on servers. So we have a whole bunch of canned ones. Definitely good to check out to see if you're doing these with your own automation tools or just configuring manually. Kind of neat to check. So now that we've talked about just kind of a hat tip to my employer who sends me these things, you should be, once you have this solid baseline, you need to have the continuous monitoring tools on top to actually do things and make sure that it is continuously secure. You have the ability to do dynamic firewalling, for example, like I was talking with the two-factor authentication. Very important in Cloud environments. So if you can do these traditional tools in Cloud environments, it really brings up your security level or raises the security bar quite a bit. So in summary, I think Cloud does... It doesn't necessarily need a new approach to security. It draws on a lot of the old things that we forgot or that we stopped doing because of our reliance on additional tools that had a much broader protection layer. So now we're getting down back to the host. That's the endpoint we need to protect the endpoint. You really do need to get your house in order before installing any sort of security tool on your server. This shouldn't be new, but your job will be a hell of a lot easier if you get your house in order before installing stuff on top. And then really, once you have your house in order, install the tools on top that are going to work for your environment. So like I said, operating systems have inherent firewalls that you can configure, use them. And if there's anything I can impart to you, if you're an end customer and you're thinking about Cloud, don't just run to Cloud because you hear it's the coolest thing ever. I was at the PCI conference last month, and this woman came up to me, and she was responsible for security at a university in Florida. And she's like, so what do you guys do for security? So I told her, and she's like, oh, that sounds pretty cool. I'm like, oh, so what's your Cloud use case? And her response was, well, we don't really have one right now, but every department head comes to me at least once a week saying, we need to get our stuff in Cloud. And she says, well, what does that mean? And he says, I have no idea, but we need to get there. I read about Cloud, and it sounds like it'll fix all of our problems. And then she's like, great, yeah, I'll put that on the roadmap. We'll get you to Cloud right now. So you will trip up. Has anyone done this playing baseball before? I have. It doesn't feel too good. You don't want to rush. You have to take your time. Know what you can protect. Know your mechanisms. Know what your Cloud service provider can give you. Know what your Cloud provisioning stack can do for you, can do for your customers. So with that, definitely if you want to try our product, it takes about five minutes to install. This is where you'd go. We have a really, really broad scope. We will work on any Cloud provider, including OpenStack. We're actually members of OpenStack as well. So big fans. And with that, I encourage you to ask some questions. I know we're running out of time. The mic's right there. Please feel free to step up to the mic and ask me any questions. If you want to email me, it's androidcloudpassage.com. Follow me on Twitter. I rant quite a bit. Andrew SM, hey. And with that, thank you. So are there any questions at all? No, but it's definitely the less secure of several better alternatives. My problem with people using R-Sync is that a lot of people won't take the additional step to wrap it with security. They'll just use standard R-Sync because, oh, it's there. I'll just transfer files around because it's a path of least resistance to get files transferred around. And that's how we use it. That's one case. I'm not a big fan of just jumping from server to server anyway. And again, it comes back to being lazy because I will use the share keys if given the option. I'm a security guy, but every security guy has his breaking point. And it's, yeah, can you give me a good use case? Other than the ones that I gave? Right. OK, I can see that as a use case. Everyone does it. Or a lot of people do it. It's that whole path of least resistance thing. Any other questions at all? Try to be Canadian as much as I can. So with that, I'll let you guys run and get coffee. I think that's the next step is the break. So thanks for attending.