 All right, I think we're going to get started. I'm going to try and actually make this a little bit quicker just because we are running behind. And I do want to leave room time for a lot of questions. I like to do that in my talk. So I'm going to try and move through this pretty quickly. Same thing with my other talks. If you could hold the questions till the end, that would be really helpful. All right, so to get started, I'm going to give a talk about Nova Network. And primarily, I wanted to do this, even though quantum's here, quantum is ready to go and grizzly. There's a lot of people that are still using Nova Network. There might be a lot of people deploying Folsom or grizzly with Nova Network and not quantum. So I wanted to spend some time. This might be the last open stack conference where we talk about Nova Network at all. But I wanted to give people some idea of what's actually happening behind the scenes. If you've already been running open stack for a while, six months, eight months, you probably have already learned most of the things I'm going to tell you today. But for some people, there may be some information. So like I was saying, Folsom deployment's pre-existing installations. Hopefully this talk will be of use to you. A lot of the quantum information, I just wanted to provide some links so you know where to find the configuration options, things like that for quantum. It's worth looking into and understanding what quantum is going to bring and how they differ between Nova Network and quantum. That's not primarily what this talk is going to be for. So those are some of the links you guys can use. These slides will be up on SlideShare. Under SlideShare.net, Ryan Richard 07. They'll also be posted by OpenStack. All right, so what is Nova Network? Quick overview of it. It basically provides networking for your instances. So it is the piece of open stack that says I'm going to provision an IP from an available pool and I'm going to give it to this instance. It actually does a number of other things behind the scenes, which we'll talk about. So the main choices with Nova Network are basically flat VLAN, flat DHCP. Flat VLAN and flat DHCP aren't terribly different other than the concept of VLANs are added into it. But ultimately they function in a very similar fashion. Flat, I don't know if many people are actually using flat, you don't hear about it too often. We at Rackspace, or at least with Private Cloud, we're consuming DHCP until we move to quantum. The flat DHCP that is, whereas flat is essentially for injecting your own networking into it. Nova Network is also responsible for handling the most of the IP tables, EV tables, and Linux bridge pieces of it. There's other implementations on how things are actually being consumed, but this is what I'm going to focus on today. It's what I have most of my experience with. One important thing to note is that Nova Network doesn't have a direct API like quantum. It is actually part of Nova. So all of the calls to it, all the everything that is the configuration options, they all really fall under Nova. It's not a separate major project until quantum. That's what quantum was born out of is because some of the limitations of Nova Network needed to be surpassed. So that's why quantum even exists. But it is, there is no direct API. There are Nova API calls for networking specifically, but it is not its own project. Let's see. Quick thing on IP tables. I didn't have a slide on this, but I tore it up. Just so you guys are aware that there's a lot of IP tables going on at the host level. There's about 20 chains in use for just the netting portion, not even including the actual security groups implementation. So just an IP table is known for netting. There's about 20 chains in play. And which can grow immensely, the number of rules that are actually in play over time. Quick definition. The way I'm gonna use definitions are, a host network is basically the network space, or interface network space that the Nova compute nodes, the physical machines are gonna talk to each other on, as well as in our case, it's where Nova, the open stack processes, and we also manage the machines from that network space. The fixed network is essentially the layer three network that instances will get their IPs from. It is also where instance to instance communication happens. So if you certainly have the possibility to run these all in one interface on VLAN, but if you choose to separate them out, then that is the dedicated interface where instances will communicate to each other on. So I wanted to, I try to like think about how I wanted to put this in slides, and it ends up being really terrible. So I tried to draw pictures, and hopefully that will make everyone's life a little bit better. So this is an overview of, I don't think anyone wants to stare at IP tables chains all day long. So hopefully the pictures will give some help. So this is my world, this is a small little example network that I have. I'm basically defining that I'm not really showing interfaces right now, just saying that I sort of have a network space, and my compute nodes and controllers are living on this 10 dot network. That's how I communicate with them. I'm also showing that on our compute nodes, we chose to run Nova compute and Nova network processes, as well as the metadata and DNS mask on every compute node. DNS mask is what's responsible for handling all the DHCP for instances. So when instance boots up, does a DHCP request, it's local DNS mask is gonna respond and say your IP is X, your default route is this, and your host name is Y. One interesting thing to note is that Nova network, the Nova network on your local compute node is not necessarily the one responsible for responding to a Nova network API call. So if you have different Nova configurations on different nodes, you could actually, that could cause some serious confusion, because a Nova network process running on a different node could be the one that responds, actually provisions the IP. So that's something worth noting. All right, so initially when I'm booting an instance, first thing that happens, it says it's an extremely new environment. I boot an instance, first thing that's gonna happen is the compute node actually needs an IP for itself. That's gonna act as the gateway for your instances. So it's gonna make a series of calls and get an IP provisioned out of the fixed network, and that will be stored in the same database. There's this very specific database field, the host field, that is going to actually have the host name of the compute node and really know other information in it. Second step in the process is gonna create the bridge. It's gonna put the interface in the bridge that you specified. I'm not really specifying one here, I'm just saying that that's the process that it takes. It's also gonna assign the IP from the fixed network that it chose in the database to the bridge. It's also gonna set up some initial IP tables, EV tables, some rules to prevent some spoofing. I'm gonna actually go into these a little bit more later. But the IP tables, the security groups, and the NAT rules, they're all actually running on the compute host, not within the instances themselves. So that's worth being aware of. Oh, just so you know, the space that I've defined here for my fixed VMs is the 192.168 space. So once my instances are created, it's gonna set up some additional security groups, which are also IP tables rules on the compute host. And by default, when OpenStack deploys, it's a extremely restrictive model, nothing's allowed, right? Next step in the process, the instance is going to send a DHCP request and it's gonna get a response from DNSmask running on its machine. DNSmask is gonna know, it's gonna keep data about all the instances that are running and what their MAC address for their virtual NICs and their IP address. And then we have our instances up. The instance is gonna get, it's the first available IP out of the fixed range and its gateway is actually going to be the IP on the bridge from the compute node. That's something worth making sure you're really aware of. Again, a lot of people probably realize this if you're running already, but it's something that most people don't understand initially. There are our options on actually to do this a different way, which I'll talk about later. This is kind of the standard default way though. Actually, I guess, does anyone have any questions? I said we could wait till the end, but just about this setup or what's going on here. Any initial questions? No, okay. Port forwarding for like an application in an instance or so, oh, to go outside. So for an instance, so instance to the world. Okay, so from here, if an instance is gonna talk to the world, it's gonna go through its default gateway, which is the computers. There's an automatic rule put in place in the IP tables NAT chain that says when my traffic is destined for something that is not another instance, I'm gonna get NADed to the IP of the compute host and then it's gonna get sent out its default gateway of the actual physical server. So in this world, if I were talking to Google, I would, from VM1, I would get NADed at the compute host and I would appear to be coming from 10.003 and I would go out whatever default gateway that compute host has, assuming you don't have some other route in place. So there's not actually a need for report forwarding. Open stack handles that for you. All right, so as far as options go, I tried to count the number of options possible. It's 50 plus, I'm sure it's actually much higher than that. Once you really start getting into it, that's not even including any of the quantum options. So there's a lot of ways to customize how this looks and there's not really any way it could go through all of them realistically and tell you what each and every single one of them means. So it's worth noting that with anything in open stack is extremely customizable for your needs. So make sure you look and do it before you try and build one. We run multi-host everywhere. It's just a way of basically, so you're not having one machine act as your gateway for all your instances. Multi-host is basically what runs Nova Network everywhere. It allows your gateway to be the compute host. Basically, you don't have a single point of failure. Obviously, if that compute host goes down, those instances lose their gateway, but the host is down anyways. Some of the other major options that I'm gonna talk about real quick are some of the DNS stuff, the DHCP options, what public interface is and what DMZ CIDR is. Public interface as an option basically decides which interface the default SNAT rule's gonna apply. So that's the rule that I was mentioning when your traffic is leaving your instance. If it's destined for somewhere else that Opusack doesn't know about, there's a default SNAT rule. That public interface basically says, watch, apply this rule to this interface. And that's what you're gonna have to set that. Specifically, yes. Well, default, if you don't set that, it defaults EED zero. But when you do set it, it affects the chain. Just put this in here just so you guys can go back and look at these chains. The specific chain it affects is called Nover Network SNAT. The rule there is automatically there. If you do not set public interface, it'll watch EED zero. It's also responsible for, excuse me, you want to associate it with the interface that can reach the internet. So if EED zero is the one that can reach out to the world, that's probably what you're gonna want to set public interface to. Well, so we, the NovaConf runs in a, we have the same NovaConf everywhere, but this is for Nova Network. So wherever Nova Network is running, that configuration option needs to be there. So it just probably your compute node. Yeah, so I explained earlier basically the default SNAT rule says that for any traffic, not destined for another instance, or if it's not in the DMZ sider, SNAT that traffic will make it look like it's coming from the compute node. Some of the DNS mask options, again, I'm not gonna go through all of them, but I just kind of want to explain some of the bigger ones that you might look at. DHCP, at least time. Obviously, people are gonna want to customize that. There's an option for that. The big one is the interesting one that we've done for a few customers is the hardware gateway option, which is kind of neat. So it basically says use a real gateway device somewhere further up in the network, not don't use my compute node. So whether it's router or something else. You know, you have the ability to do that. You can do it with the VLAN mode as well, so you may want to look into it. There are certainly caveats to doing it that way, but it's one way of, you know, some of our customers that have done this is because they wanted to maintain routes to other parts of their infrastructure. And so, otherwise you'd have to do it on every compute node, right? If you wanted to have special routes to reach some other backend network, you'd have to do it on every compute node. If you point to like a hardware gateway device that knows about those routes, you then don't have to maintain those in a bunch of different places. Sorry, I don't follow. So if you're using a hardware gateway, you basically need to tell OpenStack not to apply that rule. And then further up, you need to deal with that in some way. So you need to deal with the fact that your instances may not, so if it's public space and I'm not getting mad into something that has internet access, like through router, I have to, you know, a pad or something out there to basically say to do what Nova's trying to do for you. But if you do point into a hardware gateway device, you basically need to stop Nova from doing that automatic snap rule. Because else it's gonna say, oh, you're trying to reach this other network I don't know about, and it's gonna try and get snatted and that's probably not what you want. I'm gonna tell you how to do that right now, actually. Well, actually, DNS mask also, just so you guys are aware, by default it forwards all DNS queries to whatever the underlying hypervisor installation has set. It will just forward those queries on. So that's just one thing to take note. All right, so DMZ Sider is exactly what I was just talking about just now. It is basically a NAT exclusion list. It is an accept rule in your IP table's NAT chain. It is actually a bunch of accept rules. So as you grow this, that chain gets longer and it's more accept rules. It is further up in the process than before some of the SNAT rules get applied. So basically, it says if my destination is some other network, don't perform that SNAT. I don't want you to perform that SNAT. So maybe that's the other end of a VPN, right? Or maybe it's just some other network in your data center that you want access to. If you wanna look at what is set for your installation, that's the chain to look at, the Nova Network Post-Routing chain. All right, let's see, so the IP tables, it's basically not talking, so it is responsible for that. I've been talking about that a lot. It's also responsible for security groups. So when you define a security group in Nova, it stores it in the database. When you boot an instance, it applies very specific IP tables rules to that compute host to limit access to your instances. Default, like I mentioned earlier, is restrict everything. And an example, if you look at your IP tables rules, it's gonna be Nova compute instance, and then the instance ID in decimal, not hexadecimal. There's just one thing if you're trying to figure out which instance to look at. Take the instance ID and convert it. That's where it's gonna be. Some of the default EB table stuff, at least in the liver implementation, does IP Mac, ARP spoofing, doesn't let you assign multiple IPs inside an instance. It's trying to prevent you doing that. It's really following the public cloud model where people aren't gonna be able to have multiple IPs inside of an instance. It's basically there for protection. You can actually break these. The directory is etsylibertnwfilter, if you want. I don't necessarily recommend it, but you can turn off all these protections if you really wanted to. And then floating IPs is one of the final things I'm really gonna talk about is floating IP pools are extremely easy to add. You can define them on the fly. They must be associated with the public interface, though. You don't really have an option right now, or at least Nova Network, to have them go out some other interface. So they have to be accessible on whatever you define as your public interface. They don't actually get assigned inside the instance. I think that's probably one of the biggest misconceptions about floating IPs. They don't get assigned inside the instance. They are actually SNAT and DNAT rules. Well, it's a combination of, it gets assigned to the compute host, and it's a combination of DNAT and SNAT rules that get applied on the compute host to make it happen. They're dynamic and that I can apply them to an instance, I can assign it to an instance, I can unassign it and assign it to another instance to make them dynamic. So let's say I have a, in this model, I've added a floating IP of 10.00203 to my bridge on computer one. That's where my floating IP is actually gonna be assigned. So if you assign a floating IP and you're like, why didn't my instance nothing changed? It's good if you look on your compute host, look on the bridge, that's where it's gonna be. And then the SNAT rules are what affect traffic flow, right? So traffic flow coming into the instance is gonna hit a destination at rule that says anything, traffic that was for 10.00203, getting added to the IP of the fixed IP, and then vice versa, going out an instance. So that means traffic from an instance is now going to look like it's coming from a floating IP instead of the fixed IP. The DMZ SIDER is the exception there. So again, if you set DMZ SIDER rules, they are except rules further higher up in the chains and they'll affect flow. Let's see, and then finally, integrating with your existing network, this is the biggest headache we run into, I think, is that it's actually really difficult to do, mainly because you have to define your fixed IP space. You say, well, I think I may have 2000 IPs at some point, I need to define the correct network. A lot of companies don't like that. They don't like having to define the 4,000 or 2,000 IPs that can only be consumed here at some point in the future, right? So it is kind of difficult. And a lot of people want to know about integration with their existing IPAM. The problem there is that OpenStack really is IPAM, so it wants to know everything about networking, like it wants to decide everything. So in trying to integrate with one of your existing ones is going to be a headache. I'm not sure if anyone's successfully doing that real well or not, but it doesn't handle all the things you expect IPAM. It can handle like DNS. So just be aware of kind of what's going on there. All right, and then some example architectures. Basically, this is one that we use pretty commonly. I'm running my fixed and my management on one interface and floating IPs are pretty much all running on one interface. It's a pretty standard looking environment. And then another common setup that we see is where you essentially have a dedicated interface for your fixed. So for instance communications happening, dedicated interface, and all management traffic is happening on say ETH0 versus ETH1. And then one thing I did mention in my other talks is really another common use case is there's going to be connection between the sender nodes and the compute nodes for maybe like 10 gigabit throughput or something like that if you needed quicker. This isn't a sender talk, but it's certainly part of the networking overall thoughts. All right, so that's pretty much it. Just opened up to questions now. Yeah. So it doesn't work when you assign it to a compute. I mean, that's probably a specific implementation. Like we'd have to sit down and look at it. It's probably too hard to try and troubleshoot up here. Yeah. No, so this is not actually, that's a good question. My traffic's not flowing through the controller. I'm really just trying to say that they're all in the same B-lan. But my traffic's not flowing through my controller. We don't run the controller as a compute node, so we don't have bridges. The controller is a different model. It just needs accessibility to your compute nodes. Doesn't actually need to be able to reach your instances. It's responsible for the control plane, not actual stuff going on inside the instances. Yeah, so the floating IPs pool will be stored in the database on the controller. But when you actually assign it, the work is done by the compute node. So the controller tells the compute node, you need to assign this floating IP to this instance. The Nova compute process, Nova network processes will go to work and make that happen. Yeah, I did talk about that earlier. I am assuming that multi-hosts is turned on and you're running Nova network on all your compute nodes and not in a single point. Yeah, yeah. Yes. Yeah, it's because that's all happening before the VLANs come into play, right? So, yeah, and that's, I'm not sure, I mean, there's a flag that you can set that will not allow that, right? I think it's like a lack of accept rules. It's, I think there's a default behavior in setting that flag actually removed. Yeah, we don't run VLAN mode, so. I don't think there's another session so we can probably just keep asking questions unless you guys wanna go eat food without up to. You can based on how your networking set up. So you can define any number of floating IP pools. They're really easy to add after the fact. But if, right, but they're not, yeah, it's routing further up. But if your network lets you do that, then so be it. So, commonly, we actually assign multiple pools and we actually map them at the firewall to some private IP. And then that way, it just lets us assign new pools extremely easily. So it doesn't even really need to know about different network spaces. Yeah, but you can, absolutely, yeah. I don't know, well, I don't know about Havana, right? And Havana six months away, but in Grizzly? I believe it is. OpenStack tends to let deprecated stuff last for two or three releases. So it is still an option. And then I believe, I haven't looked into it, but I believe quantum allows you, and Evan might know this also, but I believe quantum allows you to mimic the exact same behavior that Nova Network will do if you want just a very basic out-of-the-box thing. There was an initial talk that would let you actually use Nova Network as like the plug-in to quantum. So it was really doing the work. I don't know if that actually made it in or not. I don't know if some other people might know. I think so, yes. It also depends, yes, it is. It is much different in the design and then the idea of vendors come into play which changes everything as well, which is why I didn't wanna focus on it because it's a completely different talk and different architecture, different design, all of it. And then depending on the vendor that you choose or if you don't choose a vendor and use OBS, that's a different design, right? It's big, yeah, and the quantum's big. Yeah, back there in the back. That's a good question. I don't think we've run into that yet and we have some customers running a significant number of instances. But certainly, there were in the past, like in Essex, there was some pretty serious performance problems if you're using a lot of security groups that were sourcing other security groups. That used to be a performance issue. Now, I don't know the answer, to be honest. It's no different than regular IP tables. So, at some number of chains and rules, they're gonna start to impact performance. But even customers running 30 VMs on a host haven't really, they haven't complained about it. So, we haven't hit that limit yet. No, sorry, say that again? So, no, no, the interface will be in the bridge and the IP for the fixed IP will still be inside the instance. But at that point, if you've set up your DMZ side appropriately, your computer is essentially on the wire through the bridge and the host is really not doing anything besides performing IP table security groups. So, you'll communicate directly through the bridge to the hardware gateway device. Assuming it's accessible on that VM, right? You mentioned earlier, EV tables prevents the VM from using more than one IP. Now, is that, so in essence, if you were to go into that VM and assign a secondary IP within the instance, that prevents it from talking to the gateway? Yeah, you could certainly put one in there, but the compute node's not gonna let you do anything with it. I actually think, well, if you do, you might be able to communicate with all other instances on that compute node, but basically you need to break those restrictions if you wanna do that. So, where that's come up is people trying to do traditional HA on here, right? And they need floating IPs, but they want traditional floating IPs and get assigned by like pacemaker or whatever and not open stack floating IPs, but you can't do that because you need two IPs inside that instance and you can't do that with these restrictions. So, I know some people have looked at actually writing fencing mechanisms that talk to open stack and move open stack floating IP across, but there's, you know, doing, just trying to do a traditional HA, it gets, yeah, it's probably not the right. In our instance, be like, if you were to port like, say you convert a physical web server into a VM and then put it inside open stack, well, traditionally like our web servers that have multiple IPs on them, and so that would be a use case for us. But you're saying that's, you said that was under, so is that, in W filter. Now, with no network, you can't do that. Basically, unless you break those, but with quantum, you can. So, you know, the, I didn't really touch on quantum at all, but it is a huge step forward in networking, right? So, we tried that with quantum and it didn't take like the gateway that you wouldn't talk to it. Just setting a, setting up things. Yeah, that might be a implementation. Well, there's the quantum workshop later. Oh, and here? Yeah. So that's worth bringing up. Yeah, that's worth noting. I didn't talk on that at all, but you could certainly define multiple fixed networks and every time you boot an instance, you get an IP from all those networks and that will work. Yeah, that was what Evan brought up. You can do that. You can even, you can even do some things like you can do that. And when you're spinning up an instance, tell it to use this network. Instead of this network, so you can kind of get some of the quantum functionality, but the dashboard doesn't respect that. So, you'd have to always use the API or CLI if you wanted to do something like that. Other questions, I think? I would, I mean, I would go with quantum, grizzly forward no matter what, just because that's the way open stack's going. Yeah, into the, into the mainland. Yeah. There's, there's so many ways of building these. Like it's, it's actually kind of, right? Yeah. You know, we, we chose, like I said, we chose flat DHCP as our standard until we moved to quantum. And anyway, basically just what Evan said, if we have to deal with VLANs and the NAS as the direction we would go, we tag them on the interface and put that into a separate bridge. Who else? Anything? More questions? You got one? All right. Thanks guys.