 So, can you hear us? Perfect. So, good morning. Thank you for attending our session. Today we are going to be talking about the implementation of Router Networks. This is work that part of this implementation took part in the Newton cycle. And there's some piece that is going to take place in Nocada. And so, most of what we are going to show today, it's already implemented, but there are some pieces that, even though we are going to show them, it's still in implementation, okay? So, my co-presenter is Carl Baldwin. Let's, yeah, do introductions. This is me, Gulloval. I'm Carl Baldwin. We both work for IBM. It was a recent change for me. Not so recent change for me. I've been with IBM for one year. And for today, we have a pretty simple agenda. We are going to talk about the motivation for Router Networks. Why is that important? Why we took the time and the effort to implement this in Newton? We are going to go into very good detail as to how Router Networks is implemented in Newton, the changes that we did in Newton, and the interaction that needs to take place between Newton and Nova for this to be useful. We are going to provide some guidelines as far as configuration. And then we are going to try to do a demo and show you some of the functionalities that we implemented. So, with that... Yeah, so, most of us are probably familiar with the complexity that Newton provides in terms of allowing you to create complex network topologies as a tenant or a project with an open stack. Routers, tenant networks, floating IPs and such. But there's a class of deployers out there that just want to provide like a provider, a shared provider network that all the tenants boot to a shared medium. They're all on the same network. They just want access to the internet and from the internet. So, Neutron also allows you to do this with a Layer 2 provider network. So, you can set this up. It's pretty simple for users to use. They go in, they boot their VM, they select a network, and as soon as it boots up, it's on the network. It's visible. The problem is, what if we wanted to create a deployment with a very large provider network, one that's scaled up to possibly many, you know, hundreds or thousands of hypervisors and thousands and thousands of VMs all sharing this one shared network. Then we start to run into scale issues that are kind of inherent with the way L2 networks work. In L2 network addressing, you think about it, it's a big flat 48-bit address space. And I won't go into the scaling problems, but you go read about it, you'll find, or if many of you have probably had some experience with scaling L2 networks, there are all kinds of problems that happen when you try to make these networks bigger and bigger. So, what do we do with Neutron? If we want the provider network model, the simplicity that that provides, just boot to it and your VM is accessible, and what if we wanted to scale up to very large? This is the problem that we aim to solve with routed provider networks. So, there were two ways you could scale this up in Neutron. We talked about scaling up the L2 network to very large. The other way you could do it before is to create multiple provider networks. You know, just segment your network into multiple networks. The problem with this from a user standpoint is that if you make your users come in and choose between a set of different networks without any real differentiation between the networks for them to choose which one to boot to, that just creates a confusing environment for users. You know, imagine you go in, you say, okay, I want to boot to the public network. I just, you know, I want a public IP address. Now I've got 12 choices. I've got public network one, public network two through public network 12. That's not ideal for users. From a user standpoint, you just want one. You just, I just want to boot to the public network, right? So, this is what routed provider networks gives us. We can create one network from the user's point of view. So they choose just the one public network. But behind the scenes, we can segment this network into multiple L2 domains that are easier to scale. And then we provide the routing between them to make it one contiguous L3 domain. So that was the motivation. I'll go into how we implemented this in Neutron. So we started with segments. Now, segments are not new in Neutron. In fact, segments have been around in the ML2 plugin for quite a while. And you could even have multiple segments within a network. Now, the problem was the assumption with multi-segmented networks in ML2 was that there was an underlying assumption that all of the segments were bridged together into a large L2 domain. And that was what we didn't want. So we had to have some way to distinguish between legacy multi-segment networks in ML2 and a segmented network that's going to become a routing network. And in fact, if you're using the ML2 plugin, your network has a segment associated to it, at least one, even if you haven't used the multi-segment extension to ML2. So if you're unfamiliar with ML2 segments, a segment is defined by three pieces of information. The first one is the physical network. That's really just a name that you give to the physical network that hosts the L2 domain. And then there's a segmentation type or network type, and the segmentation ID. So for example, you might call your physical network, my network, and you're using network type VLAN and segmentation ID 2016. That's an example. So what we did was we decided to take this segment idea from ML2, and we actually kind of promoted segments into neutron core. So they're no longer part of the ML2 plugin, they're no longer only part of the ML2 plugin, but they're part of neutron core. So the neutron core model now has a segment, and we provided an API for segments. So now you can, a segment is now a first-class thing. It's not just a piece, an implementation dutel under the network. It's now a first-class thing, and you can do CRUD operations on it. And whereas before, you could only create segments along with creating a network, now we can create a network with segments, and later on we can add segments to the network, and we can delete segments from the network. So the network can grow and shrink along with the segments that we need to have with the network. And there's a lot of detail here, I'm not sure we need to go over, but it's important to know that there is now the REST API to interact with segments. Now this is actually a pretty important slide. This is the thing that distinguishes a routed provider network from a legacy multi-segmented network. And it's that we can now associate subnets to individual segments. Before this, subnets were just a part of the network, and it was assumed that anywhere since the network was one contiguous L2 domain, it was assumed that any subnet that was on the network was available anywhere on the network. Well, we're breaking that assumption now. And what we did that by adding a new field to the subnet that is the, it's kind of squeaky, we added a new field to the subnet that is the segmentation ID that that subnet is associated with. So once you take a subnet and you lock it onto one of many segments within a network, that's the way you distinguish the routed provider network from a traditional network. And this, you know, the diagram just shows that we've got different subnets in different areas. So this first one, the way you imagine it is the subnet is now confined to a small part of the network instead of available everywhere in the network. And we added another attribute that's kind of important here. We added an attribute to the network called L2 adjacency. Now what this does, forever in neutron since the beginning, networks were L2 things. And there's been a lot of discussion, I bet there's been discussion at every single summit since neutron began, is network just an L2 thing or is it, can it be something else? Well this, adding this attribute L2 adjacency, this is formally saying now a network doesn't necessarily have to be just an L2 thing. And we're going to give you this attribute on the network so that you can tell if it is an L2 network or if it isn't. So if you read a network and you get L2 adjacency, if you get a false back in that field, that means that your ports may not be able to send L2 traffic to each other because on a routed provider network it might be that your ports landed in different segments and L2 traffic isn't going to be carried across them. If you have for example a Calico network, a Calico network will always show L2 adjacency false when this flag is implemented in that plug-in because it sets the expectation that L2 traffic is not going to be carried from port to port which is actually breaking an assumption that's been in neutron for a long time. And the other consequence of breaking L2 into parts is that now we need to think about providing DHCP in each of the segments because DHCP is a protocol that requires L2 adjacency, right? And we just broke that. And so given the way that neutron provides DHCP with a DHCP agent that connects to the network, now we have to have at least one DHCP agent in each segment. So that's kind of a consequence of deploying provider networks is now you have to think about providing the DHCP server for each of your segments. Now, the next thing we had to deal with in order to implement routed provider networks is how does a port behave? What happens when you create a port on a routed provider network? So if I go and I say I still have the neutron command line in my head, I don't have the open stack command line in my head, but if I say neutron port create and then, you know, routed provider network what happens, what do you get back? Well, you can't get back an IP address on that port because it doesn't know which segment you want to be on. The port just exists on the network, but the subnets exist on segments, right? So what we did was we give you back a port and we set a new attribute. The attribute is called IP allocation. We set that IP allocation. Think of it as kind of a policy when and how IPs are allocated. We set that to deferred, and then the fixed IPs is empty. Now, doing this allows you to take that port and hand it to Nova without Nova complaining that, hey, you gave me a port without an IP. What's up? This communicates to Nova that, hey, I know I gave you a port without an IP address, but go ahead and try and boot a VM on it and bind it to a host, and when you bind it to the host, then you'll get an IP and everything will be okay. So there's this concept, new concept of a deferred IP port that allows us to do this. And we had to go in and we had to change a bunch of stuff in Neutron IPAM so that we triggered IP allocation at the right time, which is basically when you hand the port to Nova and it tries to bind that to a particular host where it's going to boot a VM. And in order to do this, we added another model to Neutron which maps hosts to the segments they're on. Now, we tried to do this in a way that is kind of operator-friendly by reusing some configuration that you already have in your deployment. So if you've got agents running on your compute hosts for ML2, say either an OBS or a Linux bridge agent, then you've already got mappings in your configuration that allow us to create this host to segment mapping. And those are, in OBS, those are called bridge mappings. In Linux bridge, I believe they're called interface mappings. And those you configure the agent with, the bridge mappings and the agent actually sends those to the Neutron server. And this already happens so this is nothing new. But what is new is on the Neutron server side, we actually open up that mapping and we say, hey, there's a mapping. There's an agent, there's... My mind just went blank. The mapping has the agent's hosts and... And the bridge mapping. And the bridge mapping, right? And we actually take that and we resolve that to which segment that host is connected to. So we keep that mapping. And when you go to bind a port that has no IP address because it's deferred, we take that mapping and we say, okay, now I can find a subnet to get that IP address from. So this is jumping a little bit off topic. This is actually something we have not yet completed. Everything in the slides before this is done and merged in Newton. The issue here is that we can't yet use a routed provider network as a router external network. Now, if you're familiar with the router external, setting router external on a Neutron router, what that means basically is that this router can host floating IPs. So imagine if you took a router and you set that router's gateway to a routed provider network. That's saying that this routed provider network can get my floating IP to this router interface. Well, the problem is now we have a routed network out there and we have floating IPs that aren't really... You don't want your floating IPs to be stuck on one segment. You want them floating. And that's a problem we haven't quite solved yet for Newton. And we hope to get it solved for Okada. So the other point I wanted to make on this slide was that with a routed provider network, we actually could take the Neutron router out of the picture for floating IPs. And in fact, there's one operator, Godaddy, that I've talked to a lot about this. They actually have an implementation of this in their data center where they boot VMs to a routed provider network-like thing that's very similar to what we have here. And then they have the ability to create floating IPs and associate them directly to those VMs. And they do that... I don't remember exactly the motivation, but one example is you could boot your VM with a non-public, non-routable IP address, and then your floating IPs could be routable public IP addresses that you could selectively associate with VMs on that provider network. Which I think would be pretty cool. And I'm going to pass on to Miguel to talk about what we had to do with NOVA and scheduling. So the other side that we need to address in this implementation is that now when NOVA creates a port for an instance, it has to make sure that the compute node where that instance is scheduled is connected to a segment where there's still IP address availability. Segments might run out of IP addresses, so we need to make sure that NOVA has the information necessary to schedule that instance in a host, in a compute node, where we are going to find IPv4 addresses available. So that's pretty much what we want to say here. And essentially in this graphic what we are saying is that segment 3 has no IPv4 addresses anymore, so therefore next instance needs to be scheduled to any of the other racks. So on the NOVA side we are moving right now, up until now in NOVA, all the wiring of the instance sports has taken place in the compute manager. In other words, NOVA schedules the instance to a specific compute host and code the compute manager running in that compute host takes care of wiring the neutron ports for that instance. What we are doing now is moving that out of the compute manager, we are moving all that functionality to the conductor, and it's going to take place in the NOVA server. And once we do that, we have the possibility of having four use cases when we provide ports to an instance. The user might create a port manually, and the boot provides that port ID to the boot command. And let's say that port has no IP address. In that scenario, what NOVA can do is schedule the instance to any segment, to any rack where available IP addresses. Where IP addresses are available. Now let's second use cases, one in which the user creates a port and assigns an IP address to that port. So when we give that port ID to the NOVA boot, NOVA now has to find where that IP address is located in the segments of the routing network and schedule the instance to a compute host connected to that segment. Third use case is one in which we boot the typical case where we boot the instance just providing the network ID. And in that scenario again, NOVA is going to be able to schedule the instance to any segment where IP addresses are available. And finally we have the instance move case where the instance already has IP addresses. So when we are going to move the instance from one host to another host, that host needs to be in the same segment. Otherwise the IP addresses are not going to work. So for all this to happen on the NOVA side, we need to provide NOVA with information of the topology of segments on the neutron side. The NOVA scheduler now needs to know how the segments are mapped, how the compute host are mapped to segments, and using that information then the NOVA scheduler can do the proper placement of the instance. So to accomplish that, we are using a new API on the NOVA side and that's the generic resource pools API. That's work that has been done already on the NOVA side and essentially what we do is we create a resource pool and a resource pool is anything that can provide resources to an instance. Those resources can be storage blocks, can be IPv4 addresses, and another type of resources. So we create a resource pool and with those resource pools we associate inventories. Inventories of those resources available for the instance. And we also associate with a resource pool aggregates. In other words, collection of compute hosts that are associated with that resource pool. Specifically in neutron what we are doing is we are creating a resource pool for each segment in each router network. Every time we create a segment in a router network, we create a resource pool and every time we associate a subnet with a segment, with that segment, we are going to update the inventory of IPv4 addresses available in that segment. So let's say I have segment A and I associate subnet number one. At that point in time what we are doing is we are sending information to NOVA saying from this point on this segment ID, this segment in this router network has this many IPv4 addresses. And then let's say I associate a second subnet to that segment. So what we are going to do is we are going to increase the inventory of IPv4 addresses available in that segment. And also with each segment we associate an aggregate where in that aggregate we update the hosts that are connected to that segment. And that's where the segment host mapping that we are keeping in neutron that Carl mentioned a few minutes ago becomes very, very important. Because on the neutron side we know which segments are connected with which hosts. And that information we communicate to NOVA so NOVA can do the proper placement decision. So as far as configuration essentially there's some physical preparation that needs to take place. First of all we need a unique physical net per segment. We are going to assign a VLAN number per physical net and those VLAN numbers can be reused across physical net. As Carl already mentioned we are going to need a DHCP agent with each physical net and the routing architecture that is used for the router network is really up to you. It's really up to the employer. From a neutron configuration point of view you need to configure the service plugin which is not configured by default. You need to set up your bridge mappings and in the case of OVN another mechanism is being implemented. Then you create your network and then you start creating your subnets associating your subnets to segments. And that's the operation that makes this network a routing network. So let's see how that works. So we are going to show you a little demo that we put together. Essentially in this demo we have four nodes. We have an all-in-one node where we have all the services running. We have three computes. In Compute 2 we also have another DHCP server and we have two segments that makes this router network, segment one and segment two. All-in-one and Compute 1 are connected to segment one, Compute 2 and Compute 3 are connected to segment two. And we have an IP router which in our case is just a VM doing IP routing between the different segments. What we are going to try to do is to show you how this works. So this is the network that we are using and as you can see we have the two segments on physicalnet one and physicalnet two and we are using segment ID 2016 for the router network. And now what we are going to do we are going to create a couple of subnets and as you can see in the subnet create command I associate a segment ID to the subnet. That's the last part of the command. And that's reflected in the subnet that was created close to the bottom. You can see that now we have a segment ID attribute and that segment is connected to associated with my segment number one. Same thing with the second subnet and now I'm going to show you that we created two aggregates on the nova side. We have one aggregate associated with my first segment and another with the second segment and within those segments as you can see I have in the first segment I have host all in one and compute one associated to it and compute two and compute three to the other one. Another little piece of information that I wanted to show you is that we have two new resource providers on the nova side. This is something that is happening on the nova side. We don't have a CLI client yet so I'm using PDV to play with the Python client to show you the information. And as you can see we have a resource provider for both segments and I'm going to show you let's speak that with each one of those segments there is an inventory of IPv4 addresses. In this case this inventory has 2253 IP addresses available for instances. So now let's create a port and in this case let's create an onbound port and there are two important things to note here. Number one in the fixed IPs attribute we don't have any IP addresses. And in the IP allocation attribute this port is marked as deferred because this port is going to get its IP addresses once we bound we bind the port to a host. Now let's do a port create but let's create a bound port and in this case indeed we get IP addresses and the IP allocation is immediate. Why? Because we are telling we are telling when we created the port in the port command I'm specifying the host ID where I want that port to be bound. So now let's create a couple of instances so our instances are ready and good to go so I'm going to use the HCP name and space to log into one of those and we are going to see that we have traffic between them. Oops. I'll show you some tricks later. What? I'll show you some tricks later. What line is this? You might just remove the file. So we should have traffic to the other instance that's going to be... Yes, we have traffic to that. So what's happening here is that we have one instance on one segment the other instance is in the other segment and we have traffic here. So that's pretty much the presentation we prepared for you guys. I think we're at time. Do we have time for questions? We're at time, 12.30. So if you have questions we'll be up here for a few minutes. And this demo is in my GitHub account so feel free to download it and play with it and it completely configures the whole router networks for you so you can play with the functionality. All right, thanks everyone.