 Good morning and welcome to our talk on Adrescopes and Neutron My name is Carl Baldwin from Hewlett Packard Enterprise I've been working on OpenStack for about three years and All pretty much all in Neutron a little bit in Nova too most recently Just getting into some Nova things But I've been a Neutron core since For about two two of those years and in fact it was right after the Atlanta Summit that I that I was made a core reviewer and currently I am I'm still participating very actively in Neutron as part of the Neutron drivers team and I'm also what they call the L3 Lieutenant in Neutron So I oversee anything L3 and routing related to Neutron That that's a little bit about my background. I'll let my colleague introduce himself Thanks, Carl. Hello everyone. My name is Hong Huixiao. I am a software engineer from IBM in China Mostly I worked in Neutron project Honghui has has been a great help in getting this implemented and helping us to Make the Mihtaka release with this new functionality So Neutron addrescopes I Get a lot of questions like What are these for? What purpose does this serve? Why are you adding these and I've had to explain it a lot So when I explain what these are for I usually go back to When Neutron started It was envisioned as kind of a well, we're gonna have we're gonna have tenant networking and we're gonna have an external network and We're gonna model this after kind of the kind of how you do your home internet Your home internet you have a router probably Your router probably has at most one public address on the outside and on the inside You use whatever addressing you want, right? probably something from from RFC 1918 on the inside and most likely There's a lot of overlap Between you and your neighbors and everyone else and in the world This this was kind of how Neutron was originally modeled We'll let tenants do tenant networking. We'll let them use whatever addresses they want and And we'll make that okay on the external network because we're gonna stick Nat in between the tenant networks and The external network and it'll all be okay. So There's a couple of problems with that Number one We don't do any Nat in IPv6 and we don't really want to We have enough addresses and we have Even if you're not using public addresses, even if you're using private addresses, we have a way to to to almost guarantee that you're not gonna overlap with anyone else and The other problem is that some people Some organizations don't want to put that layer of Nat between their tenant networks and their external networks Since you know a long time ago we we had to start creating lots of little routing domains with IPv4 because We've been running out of addresses for a long time, right? so There are millions maybe even billions of routing domains out there if you consider all the little ones here and there And some of them aren't aren't so starved for addresses that we need to stick a Nat in between The tenant networks and the external network so we've had Customers ask us Hey, can we can we just route into our internal networks or our tenant networks? Instead of doing this Nat thing And well that when you when you consider doing that and you consider The model that neutron uses where tenants bring their own addresses and put in whatever they want to a subnet create It's kind of a problem So We need we need a way we need a model to isolate the routing domains a little more precisely and To prevent Address overlap within a routing domain. So Actually this we started adding this feature Well over a year ago in fact subnet pools which is part of this feature has been in neutron since the kilo release And it hasn't changed much since the kilo release It's it was added it just kind of worked people actually use it and it it's really cool if you haven't played with subnet pools You ought to go play with them and and in a few slides. I'll have a short little demo showing how Showing how you can start playing with these but just as an overview of what subnet pools are it it's an object that just holds a range of addresses that That you want to use to allocate subnets and it within that range it keeps track of what's been used and what's available So that when you go do your subnet create, you know, you type neutron subnet create at the command line Instead of giving a cider That you have to come up with on your you know, I always imagine somebody with a pat pad and paper Scratching down what addresses they've used and which ones are available instead of consulting that pad of paper you you say Neutron subnet create dash dash subnet pool my subnet pool name and then optionally a a prefix size so you say just Just give me a slash 26 out of this subnet pool and I don't care what the addresses are just give them to me and Within it it does all the bookkeeping it makes sure it that those Addresses that you get are unique within that subnet pool and no one else is using them Subnet pools are already used in in a couple of different use cases in Neutron in me talk another new feature is What we call the auto allocated topology extension or Get me a network You might have heard the term get me a network We've even shortened it to g-man, but that's just kind of a silly name for it but the the subnet pools are used within get me a network to uniquely allocate addresses to each new network that's That's automatically allocated by Neutron It's also I've learned its project courier uses subnet pools And I've even seen mention of address scopes in In a blog from project courier and hopefully some of your projects are using them already Mostly it's it's a handy feature that that takes a little bit of burden off of you and Puts it on a Neutron so Subnet pools support address scopes and you might be thinking well What's the difference? A subnet pool is kind of like an address scope. It's a Subnet pool is a thing Within which you don't want address overlap and you want addresses to be coordinated and and all that and that's kind of What address scopes are? So how are they different and this is the way I see it The subnet pool is a mechanism for for managing the allocation of addresses and address scopes they map to to routing domains and When when I was first designing that I thought well, maybe we could just have a subnet pool Map to a routing domain. Maybe that'd be okay But I had a couple of reasons that I I thought it was beneficial to insert another Class of object in between the subnet pool and the routing domain and the first one is that Subnet pools kind of existed before we had routing domains and actually Before we had address scopes the concept in in Neutron and Actually, there's there's a thing that I'll go into in the next slide Called the no scope scope At least that's what I call it And within the no scope scope, there's lots of subnets and subnet pools That all kind of overlap, but they're they're in the same routing domain. And so it didn't make sense to start tying them to Directly to routing domains And so I decided to keep the the accounting mechanism the subnet pool and the concept of an address scope Separate and also just to make it I wanted subnet pools to be useful Even if you didn't want Even if you didn't have a use case for address scopes so then there was In thinking of how to add both subnet pools and address scopes There was a question of how to maintain compatibility How to keep Neutron working the way it works for people who are using it and happy with it and Also introduce this new feature that that you may have a use case for And there there are kind of there are some conflicting requirements First Neutron allows overlapping subnets one and two it allows you to use anything you want really any address as you wanted and When we map to routing domains we want we want to first Get rid of the overlap we want to prevent it and We can't just let you use any address you want because We have to use addresses that are available and viable within within a given routing domain So given those conflicting requirements We couldn't add subnet pools and address scopes and and make them man mandatory right off the bat so If I were to go back and redesign it all from scratch and I had everything was greenfield I might have I Might be tempted to make them required make Subnet pools composed of subnets where subnets don't really exist without a subnet pool and address scope Address scopes composed of subnet pools Where subnet pools don't exist without an address scope, but we couldn't do that So instead we used more of an aggregation relationship where the subnets can exist without anything Just like they do today But then if if you want to get the benefits of using a subnet pool You can create one and use it to aggregate some subnets and the same thing with address scopes If you want if you have a use case for address scopes Then you can create one and use that address scope to aggregate some subnet pools But it's it's not a requirement so this This lets us have what I call the no scope scope, which is everything without an address scope and then Also have address scopes and they all work together So here's my little demo. It's a short one and it shows that these work just as well for IPv6 is for IPv4 So the first thing to do is to create an address scope So it's Super simple Neutron address scope create will create it with it with the Python neutron client If you're an admin you have an option of creating a shared address scope Which just means that it's visible to all the projects Within your deployment You give it a name and then you give it the IP version and that's it you get You get an address scope with an ID Given that address scope you can now create one or more subnet pools So this shows the neutron subnet procreate again if you're admin you have the option to create a shared one You can actually associate Non-shared subnet pools with shared address scopes And that that's one of the benefits of decoupling the subnet pool from the address scope is You can actually With multiple subnet pools owned by different projects You can you can divvy up your address space differently for different projects and Once you have a subnet pool You can do a neutron Subnet pool create or subnet create Given that you have a network. I didn't show that step, but so here I've created subnets and I don't specify the cider I just specify dash dash subnet pool and In fact, I don't even specify What size of subnet I want in that case? I just get the default for that particular pool Which for IPv6 is Practically always slash 64 and with IPv4 can vary in this case. I think I made it a slash 26 And once you've done that If I do a net show on My demo network I Can see that there are a couple of new attributes on the network One for IPv4 and one for IPv6 Given that they're they're completely orthogonal the two address spaces I Can see now that I've created subnets from an address scope on my network. I can see that that That my network has that Address scope is as an attribute so now I wanted to get into how we implemented this and And Honghui my colleague was was involved with a lot of that so I wanted to pass the microphone over to him and have him explain how What changes with address scopes in in L3 routing in neutron and what that means Okay, thanks car Let's see the implementation details of adjust goals in the L3 agent On the hood it is the IP tables, which is working In the router name space each little port will be associated with an address scope by looking at its related subnet as Car has mentioned Subnet has information of subnet pool and the subnet pool has information of address scope So we can get the address scope by looking at subnet and And Every network traffic will be marked according to the address scope always originating interface at the IP tables pre-routing chain When a network packet tries to leave an interface and go in a long scope It will be broken as the IP tables for water chain In the case of network address translation connection mark will be used so that The returning packet can be marked with a router just go and go through the forward chain We will revisit these items in the following pages with some specific scenarios First, let's look at the east to west traffic We call traffic between private networks as a east to west traffic and in the east to west traffic If the private networks are with the same address scope the traffic between them will be allowed this is just a normal Neutron router behavior and that's what you have already known about Neutron router before address scope But for the private networks with different address scopes The east to west traffic will be broken at Neutron router This is a different behavior with address scope and that's what we called go in a long scope in last page Let's see the IP table rules for this scenario first in the mango table of IP tables every network packet will be associated with a mark according to the originating interface and If the network packet wants to go into an interface and the mark does not match The packet will be dropped The mark here has a 1-1 relationship with address scope So if network packet comes from address scope a it will have mark a and When it tries to go into address scope B, it will be broken Because the mark does not match So the traffic cross scope is not allowed Let's look at north to south traffic We call traffic between private network and external network as a north to south traffic and in the north to south traffic If the private network and external network are not in the same address scope Neutron router will do network address translation to the traffic That's what you have already known about Neutron router before address scope But if the private network and the external network are in the same address scope Neutron router will do stress routing That's to say the addresses in the private network can go directly to the external network and the last words Because they are in the same address scope the addresses of them must be unique and legitimate So stress routing is feasible between them. Let's see the IP table rules for this scenario Also in the mango table every connection that will go out of router gateway will record the mark to connection mark Connection mark can be persisted along with the connection So when the returning packet comes from router gateway Each mark can be set according to the connection mark and As a result the returning packet can go back to the source address through the IP tables Also in the net table of IP tables The set will not be used if it is a connection in scope As you can see in the screenshot We have a accept a row before the SNAT row to prevent the SNAT If the connection is just in scope As a result the direct route will be performed Okay, let's see the last scenario floating IP with the address scope This is a picture that combines the two previous scenarios That is across scopes east to west traffic is not allowed and north to south traffic Will do network adjust translation For the floating IP it will still serve as the access point for fixed IP in the external network Not to say there will always be network adjust translation between fixed IP and the floating IP No matter if they are in the same adjust scope or not When a port is associated with the floating IP The port will be given the access to the scope of floating IP So After associating a floating IP no VM can assess the privacy networks in the scope of external network Even if it is across scope traffic This is because the port now have two addresses in two scopes So it can assess these two scopes Let's see the IP table rows for this scenario First all network packages Whose destination are floating IP will be marked according to the fixed IP so that these packages can go through the IP tables and the reach the fixed IP and If the network package comes from fixed IP and will go to a scope of external network This mark will be changed to make it go through the filter table So after associating a floating IP The port can assess the whole adjust scope of external network Okay, that's all about the Detail of implementation Car you will introduce the relationship of adjust scope and other features All right, thank you that was good Two things occurred to me actually that I don't have in my slides But that I thought might not be totally obvious The first one is I want to make sure that we understand that if you don't do anything and you upgrade to meet Haka nothing changes All your routing all your NAT everything if you use tenant networking L3 agent routers If you when you upgrade nothing changes The this these are all opt-in things Where you've got to want to use it and configure it To get it the other thing that occurred to me was there's a bit of history behind this implementation We actually considered at least three different completely different ways Of implementing this and if you consider all the different Possibilities within those three completely different ways we considered probably a dozen different ways to implement this The one I was going after first was actually to use multiple routing tables and Policy-based routing in in Linux within the router namespace And I I actually tried really hard to get that to work for liberty and I I just ran into a Lot of problems with doing that mostly mostly that things just aren't Multiple routing tables are are are kind of Imature I guess in in Linux a Lot of utilities that you run you you might run within the within the namespace Can't be Directed toward any of the one of the routing tables you can't choose your routing table and and there was there were some Really kind of thorny issues around Getting SNAT to work the way that it works in neutron When when that is applied in the router for north-south traffic getting that to work ended up being really thorny and We ended up Doing this IP tables based in implementation Mostly because it was possible But it does and What's not obvious in in here is it there are some limitations to what you can do You might expect well now I have multiple address scopes. I can connect my routers to To all these different routing Routing domains and not worry about IP address overlap and you know these these are like These are like verse. Well, they're not quite they're not quite that good and and actually There is a there is kind of a new verve implementation for Linux that we also considered Using as the basis for this implementation But that was just way too young to even to even really consider very seriously so if if you think you've got a router now that can That can connect to different address scopes without regard to address overlap. You might be a little disappointed The router will still refuse to connect to overlapping address spaces Just like it does today because the IP tables implementation can't distinguish between Different address scopes in in quite that way But given the limitations that you In in how you can connect your routers. It does behave the way that you expect Blocking routing and applying that where where you would you would expect it to be applied So you might still be thinking I've been talking up here for half an hour Or we've been talking up here for half an hour. You might still be thinking. Well, what am I gonna use this for? The use case that we've that we have in in Mitaka and I think Ryan Tidwell is here somewhere He's led most of the implementation of BGP with a neutron And he gave a talk here about it, but I'm gonna go into it for just a couple of minutes because it's It's really the the one relevant use case that we have in in Mitaka for address scopes In Mitaka we've we've made neutron capable of being a BGP speaker We did that by adding a new BGP dynamic routing agent to neutron And we have we have all the code and queries and things that that generate the The routes and next hops that we need to Talk to the external world any routers that that are That are just excurned external to neutron so the way we did that initially is We associate BGP with a a network in neutron an external network and When BGP is associated with an external network We take that external network and we gather up all the routers that have their external gateways connected to the external network and for all of those routers we look through them and And we say okay, what private networks do you have behind you and For all those private net tenant networks that are behind them we look at their address scopes If the address scope matches the one for the external network Then we grab that and we say hey, that's an interesting network. We can we can advertise that with BGP If the address scope doesn't match Or if it's in the no scope scope We ignore it So the BGP feature that we've added in Mitaka depends on address scopes. I Should have had the picture up the whole time. Sorry. I forgot that was next So that this is this is a an illustration of an external network The big router on top. That's the external router. That's a physical router The little routers at the bottom. Those are neutron routers. So those are virtual routers and You can see on the external network. We have we have an address space In this case, it doesn't have to be public. So I stuck an RFC 1918 subnet on the external network and The the upstream router has the 0.1 address and and the neutron routers each have their own addresses out of that subnet and Then behind the neutron routers each one of them has tenant networks And I've colored them the same as Honghui colored them red for For the external Address scope and yellow for an internal one in this case Projects have created the routers and they've create most of them have created subnets That belong to the address scope that that can be advertised outward. Oh, you know what? That's an old slide. I had I actually had a table of all of the I forgot to replace the slide up in the corner. I had made a table of Of all the routes and next tops that BGP would advertise Well, so in this case The BGP speaker would be off off to the right of the slide and the BGP Speaker would create a period session with the external router the The physical router and when after it does that and it scoops up all of the The tenant networks that belong to the same address scope It will advertise three routes in this case Each of the subnets with with the routers IP address external IP address on as the next hop for that route and so with that the The ten the tenant networks become routable externally and It completes the whole routing loop This slide is the same thing but with IPv6 addressing and the other thing I plan to use Address scopes in is routed networks routed networks is is something we don't have yet. It's not in metaka. We're planning it for Newton a Routed network or a routed provider network is a provider network. That's not It's not the typical large L2 that you need for a provider network today It's you can route it in any way you want You can have any kind of routing infrastructure down to The typical use case is a a router at the top of rack so each of your racks can have a router at the top and They all belong to the same neutron provider network Address scopes are an integral part of this and on such a network You've got L2 segments and You've also got the ability to have floating IPs that That float around the L2 segments and those are implemented there well are will be implemented using a routing protocol and the reference implementation will use BGP as the routing protocol to be able to route those IPs anywhere within within the routed provider network and Actually might I have to run up to another talk where we're going to go into depth about routed networks up on level four In ballroom D just after this so that that's all I have I'll open up to questions for a few minutes And if you have a question, please come to the microphone so that we can get the question on the recording I was just wondering if The router is created automatically when you are defining a address cop between two subnets No, the router will be created Still by the project and that the project will still do the wiring They'll set the external gateway to the external network. They'll set their internal ports to the internal project network and It's it's when that happens. That's when the magic happens. That's when neutron looks and Says okay, do they match? if they if they do match in address if the address scopes match then the Nat is automatically turned off and and if you have BGP configured Then the the routes are announced my use case Was where you had multiple pools out of a single scope. You said that was a legitimate use case Uh-huh, and you said that the site admin could create a global Scope so a tenant could then allocate a pool out of that global scope, right? Right that the tenant would allocate the pool and then request that that be Added to the global scope So my question the only quotas I'm aware of in neutron are number of networks number of Subnets not their size. Is there anything that keeps one tenant from? gobbling up All the address space. Yeah, some subnet pools do have a quota system Quotas are set It works slightly differently for IPv4 and IPv6 for IPv4 you set the quota in in absolute number of addresses In with IPv6 you set the quota in number of slash 64s That can be allocated. Thank you. Yeah Hi, Carl. I had a closely related question actually about the case where you who's using both an address scope and Sub subnet pools, so is the the non overlapping this of IP addresses enforced at the address scope level? Yes, okay But so there's I think that's possibly interesting corollary of that then which is that if you have a shared address scope But tenant private subnet pools it may there may be some allocations that the tenant can't do but they can't see why not Right that's very insightful. That's actually perfectly true. That's probably just how it has to be there there is overlap is managed across the address scope which does create some interesting scenarios when when you have tenants owning the pools and and and a shared address scope Thank you. Yeah, that's a very good point Neil All right. Thank you very much for coming. I Probably won't have a lot of time to stick around so because I got to run upstairs, but thank you very much. It was a pleasure