 All righty, good afternoon. My name is Mark McClain. I am a senior developer at Dreamhost. I am also the program technical lead for the OpenStack networking project, Neutron. So I promise if I say quantum, I apologize in advance, it's hard to not say that at times, even after the name change. So a little bit what we're going to talk about is IPv6 in Neutron. The IPv6 story is a little bit of an interesting one in OpenStack, but kind of backing up if you're not familiar with IPv6. You know, the joke is it's always the future, but that's been going on for 20 years. The truth is, is IPv6 is here today. Currently, there are service providers who are actively deploying significant deployments, even from a business to business users, as well as customer users. I've had non-technical friends of mine remark that also they start seeing these funny addresses start showing up on their home router, and they're like, what is this thing? Why is it so long? And why do I get to some websites and can't get to others? So v6 is here, we need to be prepared for it. And when we start talking with v6 in OpenStack, it provides a couple of interesting things. You now get a globally, depending on how you configure it, you can get a globally unique routable address to every VM. You don't have to deal with floating IPs, you don't have to deal with NAT. That in and of itself has a couple of interesting use cases. And also for some of the providers, whether it's businesses or public providers, you're not constrained with the v4 address space. You don't have to do crazy things with NAT. You can go get a block, and you can get tenants full blocks. So when we talk about IPv6 with OpenStack, the initial development focus of OpenStack, if you go all the way back to Nova Network, was v4. A lot of the constructs in OpenStack surround v4, so the routing story, the IP allocation in v4 would go through and take, if you put a slash 24, has 256 addresses, would go create 256 entries in the database. If you put a slash 64 in v6, your database is going to churn a while, creating a bunch of entries. So one of the challenges is how do you design an allocator that works for v6? There's also infrastructure configuration changes. v6, really, you have to split it into two different... when you're working on it, considering you have to consider the management side of the v6, as well as the tenant side. One of the things we're focusing on, and at least in Neutron right now, is getting the tenant side of v6 working appropriately, and then the management side will come. The management, it's a little trickier because some of the underlying web frameworks that we use to build up the API endpoints in OpenStack aren't very v6-friendly right now. There are teams that work cross-project working on trying to make those a lot more friendlier. So in Havana, IPv6 support is one of the things we worked on. We spent some time on it, but like anything, sometimes building community consensus, also getting the appropriate amount of support and code merged, it's really difficult to do because six months is a very short timeframe. So some of the things we're going to be talking about are in Havana today. Some of the things are a little bit... are items that we're still working on because unfortunately, when the feature freeze came, we could only get so much done. In terms of IPv6, Internet data v6 works today. If you're using a stock deployment of OpenStack with distros, you can do v6 VM to VM on private networking. Fine, if you're using provider networks, you can use v6, but you have to configure the provider route or upstream. Security groups work with v6 if you're using the Neutron CLI tools to manage those security groups. Horizon will do it properly. Nova proxies are a little bit weird and some people will hit or miss. We made a few changes internally. And then subnets, the v6 in terms of subnets is you can create a v6 cider, you can create gateways, you can create all the resources that you would normally expect to find in a Neutron. Yes? Metadata with v6 is a completely unanswered question because when you consider metadata in a pure v6 network, most of what OpenStack does is recreates the Amazon models. So 192, I was not in 192, 169254, 169254, one of the things we have to come out of consensus is how does cloud and network in a v6 network? Do we take a well-known address? Do we use some kind of discovery mechanism to discover v6 metadata? Most of the metadata services now, if you want that, you have to run a dual-stack network or you have to use something such as convig drive or injection to get the metadata injected into the instance. There's still a few gaps. One of the problems today, if you look at it, is host configuration. DHCP v6 in Havana, most, it does not work. It's one of the gaps that we ran into. Also, if you're interested in doing RA or router advertisements, that also, there's no ability to configure that currently. RA is kind of important because some of the default images that you will get from some of the distributions do not properly implement DHCP v6. And so you have to use a combination of RA or some folks have built images that require both RA and DHCP v6 for providing additional options. And so really, you can think of sometimes you do both, sometimes you do one or the other. It all depends on what you're interested in your deployment. Another one of the gaps is we talked about having publicly routable addresses. The Neutron component right there, there's no way for the provider to say, I have, say, a slash 40 that I'd now want to carve up and pass out to each of my tenants. So if you're using RA, one convenient way of doing configing a host is to use stateless autoconfig and to give each tenant a slash 64. So one of the things the Neutron team's trying to figure out is how do we carve up a provider block? How do we automatically distribute that? Because whether it's a public cloud or a private cloud, as a deployer, you want to be able to lock that in and not have a self-service model for some folks to provide their own. In other circumstances, you may want to provide a self-service model where you can allow application developers in-house to use a previously assigned address and provide it themselves. Another one of the gaps is the current L3 agent doesn't properly route v6 traffic. It handles v4, but doesn't actually set up any IP tables or rules properly. So like I said, there's some gaps to kind of give you a little bit of hope. And v6, looking ahead to Icehouse, the team, one of the things we've created is a standing IPv6 team. That team is focused on a number of things, including closing functionality gaps of IP address management. When it comes to v6, folks want to have the ability to choose whether or not to using stateless autoconfig or whether or not to using a sequential allocation method or using even an external device to handle the allocation of the ciders for a particular subnet or even host addresses, as well as host configuration in RA. Fortunately for us, we can leverage DNSKMask, which is the current back-end in the open-source implementation because it provides both RA and DACP v6. One of the issues as a community is that's going to require us to bump the minimum version, which some of the distros do not ship. A v6 compliant version of DNSKMask. And the last thing is improving routing. Like I said, the v4 routing is in the network node is set up properly, but what we need to do in v6 is set it up to allow traffic to flow both in and out. That also creates a couple other issues when we talk about whether or not traffic should a tenant hypervisor should all traffic be allowed in or should tenants have to... Do we need to pair that with the firewall service so that you can selectively enable which traffic enters the tenant network? So those kind of talk a little bit about challenges of what we're facing in terms of how we're going to solve... We're looking ahead to Icehouse. What I want to talk a little bit about is Dream Compute and how we address and provide IPv6 now. So just a little background on Dream Compute. Dream Compute runs the latest Havana release. We were able to install it and have it running using our CI systems within less than 10 days after the final release of Havana. One of the ways we were able to provide v6 to our tenants is we've customized L3 via a neutron plug-in router. We have a customized router appliance and a customized L3 agent. If you... And one of the questions I know a lot of you are thinking is we have three customized pieces of code. Why were they not upstreamed in time in Havana to enable the entire community to take advantage of it? One of the things when doing this is we needed a little bit of time to explore the space, figure out how to solve these issues and also socialize and learn and then have consensus with the community. So when we talk about our neutron plug-in, internally we've called this a conda as the code name. It's code which we've published in our repository. And basically it's a module layer 3 plug-in. It allows you to coexist with standard L2 layer 2 plug-in. In our particular deployment we use in this area is MVP. A little bit older release is our layer 2 provider. And one of the things we've also modified it to is on network creation it auto-creates a v6 subnet. So in terms of a public cloud when you're trying to, our particular use case for our customers, when they sign up for our service we don't want them to have to think about whether or not they're using v6, go configure it and try to explain it to them. What we're trying to do is out of the box provide v6 connectivity to the hypervisor. One of the benefits is for those who understand it they don't have to allocate a floating IP. They can have direct access. What we're also pairing that with is we run our network in a dual stack. So we run both v4 and IPv6 because for the near term if you're providing a web service you need to have both because some clients can only connect with v4. The question was does every VM get a whole subnet? No, every tenet we assign a slash 64 to and we use stateless auto-config. We use stateless auto-config which basically if those aren't familiar with it stateless auto-config takes the MAC address uses a well-known algorithm and takes the first 64 bits of the sider and then basically builds the other 64 bits from the MAC address and now you have your full 128-bit IPv6 address. It provides a guaranteed uniqueness without having to go lock the database and go scan through a number of entries. Also you don't have to worry about using allocation pools so in a v6 deployment where you can have a very wide space you don't have to worry about doing searching through it. It's only unique with MAC addresses unique. Yes, you are correct. And one of the things we do within our deployment is ensure that MAC addresses are unique because that assists with migration and some other things. And then the one last thing our plugin does is we do extend the router attribute the router resource attributes so that mainly what that does is provides a little extra information in terms of on the admin side for interfaces because what we do is we run all of our L3 is via a routing appliance. Our appliance is based on open BSD. We're big fans of the PF firewall. We find it easier to configure so we like BSD, we use it. The instances we run are basically they're very small. They're 256-meg instances that we basically run within the cracks of our deployment. They're easily mobile. It's also interesting because when you do this model you don't have a network node to your failure domain is a lot smaller if you were to lose a hypervisor host. The traffic, you only lose the tenant's router that was associated on that hypervisor. The one nice thing is you can also live migrate those or recreate those. In addition, the routing appliance handles IPv4 DHCP which is standard. It's very similar to how the traditional DHCP agent works. In addition, we also provide floating IP NAT. So it provides DNAT and SNAT. Also, our router appliance handles metadata services local. We run the agent. What it does is it answers locally on the local network and proxies that request out through our management network and up to a metadata service. What that ensures is that the metadata request is still secured. In terms of v6, we use RA. RA plus Slack, I talked a little bit about that earlier. And then we use internally to publish out the IPv6 ciders for each subnet. We use BIRD and BGP internally. Yes, one at OpenBGP, mainly for orchestration of just the configuration. BIRD allows us to do both RA and BGP in the same process in the same config file. It was considered as... And then the last thing is we have an L3 agent which we nicknamed the rug. If any of y'all are big fans of the big Lebowski, the rug is the joke goes. It ties the room together. In our case, the rug is what ties our deployment together. It takes the logical configurations that are in the Neutron database and takes those and then manages a pool of services for the routers. Basically, it's customized L3 agent. It uses the existing L3 APIs. So from the Neutron's perspective, we were able to just take the L3 agent, remove the open source one, drop bars in without really modifying anything. What it does is this agent also is responsible, has multiple workers for creating the instances and manages the pool, calls out to Nova to spawn up the instances, ensures that the instances are running and healthy, and does all the standard management tools, management activities you expect to ensure the instances are alive. And then the one kind of caveat to that is the instances are run within a special tenant. We wanted to do that to ensure that tenants could not stop or start their instances because that causes support headaches. We also didn't want them tampering with the instances. That does provide a little bit of a challenge because cross-tenant plugging in Neutron is kind of a bit of a challenge because Neutron, even our tenant runs as an admin, but what we have to do is we, underneath the hood, we had to create special policy rules that allowed the service tenant to plug in to other networks and use port from another tenant. So it's kind of a quick overview. One of the tricks with doing V6 as far as making these changes is, you know, I could show you a demo, but this is logging into a V6 and, you know, those aren't very exciting. So really it's questions, if anybody has any. Yes? Yeah, most of the current Tempest testing is focused around V4. Like I said earlier, one of the legacies of OpenStack is most of the infrastructure, most of the use cases originally were around V4. So in terms of increasing Tempest testing for functionality, we're going to have to add V6 test cases. We also, you know, then evolved. I mean, most likely we need to add more V6 cases. It's probably not isolated to you. Any questions? And I'll get you next. Yes? So the question is, is can we get away from tunnels by using native addressing everywhere? One of the challenges is that if you want to provide tenant isolation in terms of broadcast traffic or even if you're doing local V6 discovery, you still need the tunnels to isolate, to tenant traffic in a public crowd. Yeah, there are other alternatives depending on what underlying technology you're using for multi-tenancy that don't necessarily involve tunnels. You can restrict and create flow rules, but you're ultimately still creating tunnels eventually. So yes, so the question is, could you use multiple L3 agents and have in... I guess I'm so capable of that. And getting some kind of failover. So internally, one of the things we're experimenting, one of the things is, because we're currently in beta and entering compute, and one of the things we're doing before GA is we're going to provide some high availability. Because we're using BSD, if we're going to be using Carp to do a failover and do state sharing. Right, so the question is, is we assign a slash 64 to every tenant, and then we create a router for them so that when they boot an instance, it's already there. It's one of those... If you talk to people in terms of usability, since we're deploying a public cloud, we want to have the networking just on and ready for our tenants. So we made the decision to go ahead and pre-create some resources at tenant creation time. There's also the option that when tenants are signing up, they can choose not to have auto-creator resources and manage the resources themselves. Did I answer your question properly? The reason we use PF is mainly when we were designing and architecting Dream Compute, a bunch of us on the team like PF firewall rules, mainly because we needed a NAT for v4. One of the things also, if you take a look at some initiatives in Havana, and we decided not to activate it because the API itself is still experimental, but the Havana team... The neutron team worked in Havana to provide edge firewalling as a service. And so PF enabled us to... We were looking to leverage that to provide an edge firewall. Currently, right now, we're just relying on standard security groups for our deployment. Yes. How we're allowing the tenant to prevent killing the... So from the tenant's perspective, the current operation of the APIs is exactly the same as today. So when a tenant goes through and disassociates their subnet from a router, we remove it. And then the state changes. The rug handles propagating the state change all the way back through to the routing instance. And when the tenant goes through and deletes the router, the same state machine will fire up again, and we go through and delete the router, remove the instance. And so from the tenant's perspective, the logical constructs behave the same, and then the real-world implementation behaves the same as you would expect. Yeah, that was one of the things we kind of spent a lot of time working on is to make sure we ensured. And one of the other things is a lot of you are wondering how do we configure those routers. What we do is we have a special management port which runs on a well-known ULA network that we can use to access the management. We also use PF rules within that to prevent traffic from exiting on the management network. Yes? Sorry, I'm having a hard time hearing you. Sorry, what was that? I'm sorry, I'm having a hard time hearing you. Right. So, oh, RADDB, no. We're using BGP. And then to provide the router... We're using BGP to provide the routes statically from our core all the way into the instance. We're using Bird, not Quagga. So the question is, while working on high availability, what have we done with Nova Scheduler to ensure they don't appear in the same host? We actually have a team working on scheduling and how we place the routers within the pods. One of the exercises as part of our beta is ensuring that we're getting proper placements. So, what we initially want to do is track what's in the stock open stack distribution and also track a little bit with what the ICE house distribution's going to allow. So one of the things we're looking to leverage in the service instance is, for instance, we'll probably add IPsec VPN first and more likely SSL VPN as soon as that interface is finalized and the APIs finalize because for our use case and our typical user, they're going to want the SSL side because of remote warrior. It's a little bit easier. Yeah, it's easier. Any other questions? All right. Thank you all.