 Welcome, everybody. We're here to have a chat about working with security groups. My name is Jacob Walsik. I work for Rackspace. I'm a principal solution architect with our private cloud group. So what we're going to talk about is applicable to most private clouds, not today, to the Rackspace public cloud. But everything we're going to talk about will be true of just about any OpenStack cloud that you're going to consume that is using Neutron for networking. Quick introduction. I've been with Rackspace since 2007. I've been working with OpenStack since 2010. I started playing with the early Diablo releases and then have stuck with that ever since. I'm not sure how many of you have worked with security groups with Nova Network versus Neutron. But one thing to be clear on, we are just talking about Neutron today. Some of the functionality is different. We're also not talking about firewall as a service. So if you were here hoping to learn about firewall as a service that is not going to be covered today, I did not see another session on firewall as a service. There is a Neutron networking hands-on at 4.30 tomorrow that may have some content. I was unable to reach the authors and find out. And we also won't be looking at any live demos. There's going to be plenty of lovely screenshots of command line input and output. There's not really much to demo when you're creating, destroying, modifying security groups. So I just stuck with the screenshots. I'll upload this to SlideShare after the presentation. And of course, they record these. And they'll be available within a couple of days. All of the commands that you see have been verified to work on IceHouse. So first, what are security groups? This is the definition from the docs.openstack.org site. Security groups and specifically security group rules are what allow us to define access into or out of instances. And it's the security group rules that actually do the work. I can create a security group. I can call it whatever. I can assign that security group to VMs. But it's just going to use the default behavior that all security groups have. And the default is no traffic in, all traffic out. It's not until I start defining rules, which will be some combination of a protocol, a series of ports, ingress or egress, a IP range, or even a different security group for my source or target. That's when I start being able to actually do filtering or to allow access into my VMs. The majority of the time we're going to spend in here today is going to be in this section here on creating security groups. When I set down to really dig into the difference in how security groups behave with Nova Network and Neutron, I discovered that there's not a whole lot of interesting stuff that you do other than creating the groups themselves. Once they're created, you start assigning them to instances. But that piece is pretty trivial and is more or less stayed the same as we've moved from Nova Network to Neutron. So security groups themselves are implemented, assuming you're using the KVM hypervisor with IP tables rules on the hypervisor. You'll see these security group rules show up as individual chains for every single VM. Every VM on that hypervisor will have three chains. If you look at the names of the chains that are on the screen right now, the top one starts with the letter I. This is where the ingress rules go. The bottom one starts with the letter O. This is where the outbound or egress rules go. There is a third chain that you see referenced there. It starts with an S. That is where the MAC address is added for any outgoing packets so that they have the appropriate source. When we go to create security groups, there are a couple of different tools we can use. The Nova commands for creating security groups still work. They just pass that functionality on through to Neutron. The Neutron tool also works and exposes the full complement of functionality that's available once you start working with Neutron. One of the biggest differences is that if you want to create egress security groups, you need to use the Neutron tool to do it. It allows you to specify whether it's an incoming rule or an outgoing rule. They're with the direction flag. If you have automation that already uses the Nova commands, that's great. Like I said, it'll keep working. But if you want to start using some of the different functionality, you will have to move at some point in time and start using Neutron to create the security groups. So when you create security groups, Nova will not enforce a unique name or Neutron will not enforce a unique name on the security group. I would recommend not creating a bunch of security groups with the same name. Anytime you create a tenant, there's going to be a security group called default. That security group will allow traffic between all the VMs that are part of that tenant that are a member of the security group default. It won't allow any incoming traffic, and it will allow all outgoing traffic. If you want to allow any additional traffic into those VMs, you have to explicitly allow it through a security group rule. The UU IDs that are shown here are what you end up having to use if you start creating security groups that have conflicting names. So if you like playing with your UU IDs, and I'm sure everyone does, then by all means call all of your security groups Bob. If not, then give them something more meaningful. When you start adding rules to security groups, you're generally using either a single port, as we see here, with 80 and 443, or you're going to be using a range. So if you have a service that uses a range of 10 ports, you can specify the first number will be the beginning of the range, the second number is the end of the range, and you can get away with just creating one rule. You're also going to specify the cider for where the traffic can come from. So here we have two rules that are opening up ports 80 and 443 globally for all incoming traffic. That could very easily be a different networking range within our cloud, a different networking range for another application environment in our data center, or it could be some public range that we're allowing in because there's routes that get through into this piece of our private cloud. Here is the creation of a security group using Neutron. On this slide, we've got the Nova Network commands. Here we've got the Neutron command. When you're creating them, it's obviously very different. I've had to truncate some of the output here because you get a big chunk of JSON back when you're creating them via the Neutron tool. And then here is adding that same rule in. Again, we're going to start getting a big chunk of UUIDs. If you've been working with the Neutron utilities, you're going to start to become accustomed to this. When we start looking at applying security groups with Neutron, you'll notice that we start applying to ports that are going to be a unique ID. We're not applying them to individual VMs anymore. When you create a security group, rather than specifying a source-sider range, you can also specify another security group. This allows us to start creating tiered applications. As we look through this presentation, you'll notice that there is a security group for proxy servers and a security group for application servers. And the application servers only accept incoming traffic from the proxy servers. If you are accustomed to deploying three-tier applications, setting up a DMZ, this is the exact same model that you can bring into your security groups. When you boot an instance up, you have the ability to specify a list of security groups that you would like for it to be a member of. If you're using the Nova command line tool, it's just dash, dash security groups, and then a common delineated list. Once an instance is up and running, you can modify which security groups are attached to it. The security groups you see here are whatever. I've added a security group to the proxy01 instance. I've added the Linux servers security group. And the command here is shown, both the Nova command, which, as I mentioned before, is what most of you are probably used to working with, and the Neutron command. Once again, the Neutron command, we get a lovely handful of UUIDs. Neutron won't let you delete a security group that has any instances attached to it. So if you have a security group and you decide that you want to get rid of it, you don't want to allow global SSH anymore. You don't want to allow PING, whatever the case may be. And you go in and delete that security group. It will come back and tell you no. So you'll have to figure out which instances are still configured to consume that security group. And you'll have to go through and either remove that security group or change the rules within the group itself. So through the process of creating this presentation, as I mentioned, I learned quite a bit about what to do with the security groups themselves and how they apply. One thing to keep in mind if you are using Nova Network is there is a flag that's in your NovaConf called allowSameNetTraffic that has a substantial impact on the behavior of security groups. If you're using Nova Network and you have multiple tenants that are consuming the same network and you have allowSameNetTraffic turned on, then it will allow traffic between VMs across tenants. This behavior does not appear to apply in Neutron. The docs today don't reflect that. I could not find a way to have that flag set to true or false and have it influence the output of the security rules that were created. So that is something to be aware of. The documented behavior is that if it's set to true and you have the same network used by multiple tenants, then the traffic should be allowed to pass through. And that is definitely not the behavior that I saw in practice. Do we have questions? Yes. Enabling IP tables logging? There's not a way to do it that I'm aware of. So the logging that you would set for catching all of the bad behavior that comes in, say, hey, I want to see everybody that tried to hit 22 on this that wasn't explicitly allowed in, I'm not aware of a way to do it with the native security group tools that are available through Nova or Neutron. I mean, the solution that I would propose to that would be filter that at the edge and do your logging there. But that doesn't help if you have multiple tenants and you're worried about your tenants misbehaving when they're communicating with each other or about an application that might go rogue within your cloud. I'm sorry, I can't hear you. You know, VMs communicating on the same tenant. You know, if you have rules in place to restrict that with security groups, there's no visibility, right? Yeah, so the default security group, which will be applied to every new VM that you create within a tenant, allows all of the traffic from those VMs to communicate. So you have to explicitly say, I don't want the default security group. I just want this one in my proxy servers security group. Right. In which case, only the, whatever, only the incoming 84, 43 should be allowed. But I'm not aware of a way to log like an incoming attempt to connect to 22 on that VM. I'd assume you have to do it through IP table somehow. You would have to do it through IP tables. It can get pretty dicey when you start injecting security group rules or injecting your own IP tables rules in conjunction with security groups. All of your security groups get managed by, you know, all those IP tables rules are getting managed automatically. And so when you have a, when I say I've got my application servers security group and I'm allowing incoming traffic for my proxy servers security group, every time I add a new VM to the proxy servers security group, every VM that's in the application servers security group gets its current IP tables rules updated to include the additional source address. So unless you had a very static configuration, those rules are going to get stomped on if you try and start putting them on the interfaces that are getting managed by, by Neutron. And how do you see the kind of the evolution here with firewall as a service then the evolution of security groups? Do they go away? Do they work with each other? Security groups stick around. Firewall as a service is intended to provide protection at the edge of the tenant. So if I have multiple tenants and I've got a set of developers in each tenant and maybe they're each running a piece of my application they can decide what traffic they're going to accept from the other bits of my application. There's been talk though about, you know, firewall as a service moving down to like layer two but I don't think anything's there yet. I definitely don't think it's there yet. And with the problems that you already have, I mentioned that you can do security groups can use another security group as their source address. So when you do that, you start the behavior of every time I add a new security group to proxy servers, all my app servers get their security group rules updated. So if I've got 500 proxy servers and 20 or 200 or whatever application servers, I start this cascading effect. Every time I add another one to either group, all those rules have to change. Which kind of comes down to the fact it's not super, I guess it's scalable but it's not in the long term a great solution. Yeah, it's scalable but it's no more scalable than IP tables is natively. I mean IP tables is what IP tables is. That's the tool that was used to implement this. I can't think of a better one that they could throw in there tomorrow. So that's using security groups in a limited basis because of the default behaviors is easier. By default, nothing gets in. So I'm only setting up one IP tables rule that says I allow SSH or I allow HTTP or I allow whatever my SQL, whatever the service is that VM is running or that family of VMs are running. And support is still just limited to KVM, right? It varies based on the hypervisors. If you go out and look at the hypervisor compatibility matrix that's on openstack.org, there are several other hypervisors that list it as being a thing. I have no idea how they're implemented. I know some of the older VMware stuff required a Linux VM running on every ESX host. I don't think that it would be practical to funnel all the traffic for that hypervisor through that one VM. So I'm not sure how they go about implementing it with something other than KVM. Yeah, I mean, I thought it was only KVM still, but maybe. I mean, like what you're saying you can do, but it doesn't scale at all. Yeah, the funneling all traffic through one VM creates the same problem that we have with L3 and Neutron where we're now funneling all traffic through one physical host. And it's bad enough when it's a full blown box with 10 gig NICs. Thank you. Yes. So the question is, is there, am I aware of any plans to expand the security group functionality to include more like IDS or application firewall style functionality? With the way that they're implemented now, you couldn't filter for anything that IP tables couldn't filter for. It's like you couldn't do any sort of layer seven stuff where you're watching for a SQL injection or a WordPress vulnerability or anything like that. That would require some alternate implementation of security groups that use the technology that was able to do that sort of inspection. Either some sort of proxy service or integration with an appliance like what, alert logic cells, and then like that. If you have that type of service, that's something that's going to have to be implemented elsewhere. It's not something that security groups is going to do for you. So you'd either have to do it inside of each of your VMs or you'd have to do it externally with some sort of security appliance, the common way that you would today. So inside of your VM, instead of accepting direct HTTP requests, you could have some rules with something like varnish that rejects any traffic that matches a certain pattern. So if you know that there's a common, if you're running WordPress and you know that there's this new SQL injection that hasn't been patched yet, you could match that traffic in a proxy service and just return a 500 and say, nope, go away, instead of passing it through to your application server. Yes, then you would have to use some sort of tool to manage all those proxy servers in their configuration. So that's when you would be getting into what kind of automation are you using to deploy your application itself versus leveraging the infrastructure to provide security. I'm not familiar with that specific phrase, but there are various proxies available for most of the common services that allow you to do that deep inspection of requests for that service. And so you could use that to say, you know what, I'm going to proxy that off to DevNull, essentially, whether that's an HTTP service and you just return an error code, or it's a SQL connection string, and you just drop it. If you used an admin user to modify the default security group and then didn't give the tenant access to have any additional security groups, with their quota, I believe that would work. I'm not 100% sure. Yeah, I think this is whatever. There's not a lot of really good role-based access control there. I mean, if you have a user within that tenant that is an admin for that tenant, they're going to be able to modify that rule. So conclusion. Security groups are the best way to provide security within a OpenStack cloud between the VMs that are within a tenant and to manage the incoming traffic that may be coming into that tenant from outside your cloud. You can manage some of this with Edge devices. You can manage some of it by how you control your network. As you saw from some of the questions here, however, there is always the concern about what happens when the bad traffic is coming from inside the cloud. So following the rule of having a policy that allows the least amount of necessary traffic into your VMs is always recommended. That's all I have today. If anybody has any questions, please feel free to step up to one of the microphones or I'll be outside here after we wrap up.