 I'm Brian Haley. This is Sean Collins. Welcome, everyone. We're here this afternoon to talk about deploying and operating an IPv6-only OpenStack cloud. So first, a quick agenda. First, we want to talk about what do we mean when we talk about an IPv6-only cloud? There's maybe a couple of interpretations of that. And then second, why did we do this work? Some of the early work with DevStack that Sean and I did with some others. We'll cover some of the issues that we found during this work and then also talk about some of the future things that we're working on now and going to work on hopefully in the rest of the Mataka cycle. But before I do that, I'm going to break the rules and Sean's not happy, but I was going to do a live demo of what we've done. So this is just a VM running Ubuntu. And if you see the highlighted lines up there, basically I'm turning on service version 6 and setting an IPv6 address, and I'm just going to run a stack. And while that's running, we can go back to our presentation. Yeah, it's a live presentation, too. So we'll probably have a blank slide stuffed in there. All right. So I'm sure most of you, when you saw the title and you think of OpenStack and IPv6, you always think of Neutron. Neutron Network's what you're going to present to the tenant, whether it's v4, v6, or dual stack. But there's another side to IPv6 only in OpenStack Cloud, and that is the operation side. Can we deploy an OpenStack Cloud in an IPv6-only environment? And by this, I mean that all our demons are running over IPv6, all the RPC message queues are running over IPv6, all the endpoints in Keystone, or all IPv6 addresses, as well as all the tunnels. So if we're running with OBS and VXLAN, all those tunnels are growing over IPv6, as well. So why would we want to do this? I don't think you're allowed to have an IPv6 presentation without an Aaron slide on it about how long it's going to be before all the IPv4 addresses in North America run out. But on September 24, which is just about a month ago, Aaron ran out of IPv4 addresses for handing out with the caveat that they will hand them out to you if you use them to transition to IPv6. So we're at a critical point in time, at least in North America, I know that most of the rest of the world has already passed this point in time. Of course, there are other reasons, too. There are deployments that don't have any IPv4 addresses. This was actually raised at the last summit by Cisco that they had customers in pure IPv6-only environments. It could be that you maybe only have IPv4 on your external-facing VLANs for floating IP, and you only want to run your core at IPv6. You don't want to consume any addresses. One of the second reasons is people are getting a lot more comfortable deploying and operating with IPv6. For example, there's even customers running IPv6-only VMs. They're using IPv6-only VMs. There's no need to have any IPv4 for floating IPs. Why do we need to run the services over IPv4, either? The third reason is that you might have IPv4 addresses because you're using 1918 space, but you might be consuming it at too big of a rate, and you might want to maybe move that to be running over IPv6 and save it for other things. Of course, one of the most obvious ones is you better find the issues now and not later. For this work, we picked a target configuration that we tried to stick to, and it basically had the normal open stack components, Nova, Neutron, Glance, Keystone, and Horizon. For Neutron, we chose open VSwitch with VXLAN because that's an extremely common configuration, and we also turned on DVR because that's a common configuration for most of our customers. Swift unfortunately dropped down to the like to have components, so if there's any Swift people, I apologize, but we'll go into that later. I'll hand it over to Sean now for a little bit. One of the first things that a lot of people that are in the community are probably chuckling a little bit about is even our demo is using DevStack. DevStack is a set of scripts that will deploy an open stack software onto either a virtual machine or a physical device. What we need to do is in order to actually test IPv6 functionality, we need to add this functionality into DevStack because it is used to drive a lot of the CI systems within the open stack gate, and it is used to deploy all of the infrastructure that then tests a developer's patch. The nice thing about DevStack is that prior to the work that Brian, myself and others did to get V6 working within DevStack is that we already had the prerequisites of installing a cluster, configuring all the services, and everything like that, so the amount of work that we actually had to do was very small. We also wanted to contribute back to the community. It's been really important throughout the IPv6 efforts within Neutron and everywhere else is that it's done in the community and shared with the community to have buy-in from all the developers within the open stack ecosystem that, yes, IPv6 is an important feature to have. There were a couple patches that have been floating around within the community for configuring the services and then also the data plane. Within that, there were these two configuration knobs that were used to set both the IPv6 address that the host will use, and then also an IP version configuration knob for what version of IP will be used for all the services layer. Prior to this work, most of the configuration was it was just the 0.0 v4 address, and then host IP was referenced in other locations. So what the work did was to standardize on the service host, a service local host, and service listen address that would then be referenced throughout the rest of DevStack so that it could be set in one place and then deployed everywhere, so that Keystone, all the components would then correctly have the IP address put in, be it either a IPv4 address or an IPv6 address. So one of the main things that we had difficulty with is that in some configuration files, you can give an IPv6 literal, while other ones, you have to wrap it within brackets. So there was some code to handle those different accepted formats within each daemon and service. There was a really good cross-collaboration between all of the companies that are involved in OpenStack, and it really was through this process that something like this was achievable. So if you're interested in doing this, how do you get to the point where we are, where we're deploying IPv6 within a DevStack context? So within this, you just set the service IP version equal to 6 into your local .comp file, and then you run your actual DevStack stack script, which will then create the cluster, install all the services, and then the source OpenRC line is just to load in your credentials so that when you use the command line clients and everything like this, you have your credential set and you can do further operations like booting a virtual machine onto an IPv6 only network. One other thing for convenience is that we have some logic that we'll try and set the IPv6 address. It will try and auto discover it so that you don't have to go and copy that in, because you may be on a laptop that is getting an address dynamically, and it's a pretty long address and you just would be copying and pasting it in, and you're just repeating yourself. And for the developers in here, you don't repeat yourself. If you know where the source of truth is, go and fetch it from there rather than forcing you to copy it. Still may have? Okay. Well, one of the big things that we ran into first is the fact that we were a little ahead of the curve when it came to using OpenVSwitch as our driver. The local IP address, which is used by OpenVSwitch to link hypervisors together for the tunneling protocol, was not able to and accept a IPv6 address. So for a while, you could have the majority of your OpenStack deployment, V6 only, but then the hypervisors and the networks that built the underlay, unfortunately, had to use IPv4. However, there have been patches, and you'll see it in some of the new slides where this limitation has been lifted. So for the time being, you had to fall back to an IPv4 address. And this is actually, so I linked to the mailing list for development of the OVS project where they're actually undergoing the review for this patch. So things are looking really great on the OpenVSwitch side. Thanks, Sean. So after we got multi-node working using virtual machines, we decided to a bare metal deploy. So as I say here, the first thing to do and worry about in a bare metal deploy is actually acquiring all the hardware you need. Even though I work for HP Cloud and we have plenty of systems in our data center, you still have to justify why you need them and how many and what type. So I'm sure everyone will go through that. And then the second point is don't be disappointed when you don't get what you want. So it's kind of like Christmas, right? You ask for three SO390s and a few of those and they say, well, you can have two. Is that enough? And take what you get, reconfigure the roles that you intended for the systems and move on. I put the big red arrow at network configuration because for this, since we were running IPv6 networking, we had to make sure that the infrastructure was configured correctly. And not every network administrator has dealt with a lot of IPv6. So it might be a little bit of a learning curve there. And we were also deploying some advanced services such as prefix delegation. Some of those things might not be supported in your hardware. They might. So it's kind of a big step. And also we were running multiple VLANs in order to isolate traffic. Again, it's just there can be an extended period of time here. In addition, we actually had to have people go in the data center and plug some cables in. That takes time. It's not like you can file a ticket and have it serviced immediately. Installation is typically pretty painless. Your OS of choice for us, we chose Ubuntu. But then the next step, which was just as important, was to verify that everything was configured correctly. And for this, I don't just mean the systems themselves, such that you could log in as the administrator, as certain accounts, but verifying that all your VLANs were configured correctly. We actually went and configured all the VLANs with addresses, made sure all the nodes in the data center in our little cluster could communicate with each other correctly over all those VLANs. Because trying to chase down one machine, not having a VLAN plumb to it later on is just extremely painful. And then, of course, security. Since we were doing bare metal and this was being hosted in our cloud data center, we were going to be having at least one interface sitting on the live internet. So we had to go and disable services, firewall off ports, and things like that so that the security team wouldn't come running for us. And then finally, we could start doing our deploys. So we took the multi-node deployment just one step further and we turned it into a multi-node, multi-interface dev stack. So in our first round of testing, we decided on three VLANs. And this was the first one, so that we could divide these things up a little bit. So first, we had the management and API, VLAN. And we put those two together because we wanted to have them both behind our firewall so that they were only accessible by employees. So only people with credentials could log into the machine. And then also because our entire control plane was running on that VLAN. Second, we had a separate VLAN purely for the tenant traffic, which was for VXLAN traffic only. And as Sean mentioned before, we had to go to IPv4 for those tunnels. So this was actually the only interface that had IPv4 running on it. And then finally, for our public VLAN, which had direct access to the internet, that was all for floating IPs and for all of our external facing IPv6 traffic. So I have a little diagram kind of explaining what I just talked about. So here, if you look in the yellow, we had our management and API network. And we used a single slash 64 for that configuration. And all of the nodes in this cluster were on that one or using that slash 64. And if you look on the green, that was our tenant network. We had a single IPv4 address on each node. And that was a tagged VLAN. And then finally in red here was our external network. And we had a slash 24 available for floating IP space. And we had a slash 52 prefix for IPv6. So that gave us 4,000 IPv6 prefixes to use for tenants. So first attempt and the first failure. So the first thing we ran into was that since we were using DevStack was that it wants to have an IP configured on your external interface. In our case, we weren't going to be configuring an IP on the interface because all those are living in the neutron routers inside of the namespaces. We didn't want to consume any of those floating IP addresses. And also, we didn't want to expose the system on the internet anyways. So we had to manually add those interfaces to the external bridge. So that was step one. So it should work now, right? Well, it didn't quite get there. We were having problems getting in and out of the external network. And we finally traced the problem to the fact that when the default network was being created by DevStack that it wasn't adding our VLAN ID to it. So none of the packs were being tagged properly. And so we also had to manually do that. So we had to run, for example, these three commands to create the network and then create our two subnets, one for v4 and one for v6 on top of it. And this is actually pretty easy in DevStack to do now that I know the correct setting. There's always a knob for whatever you're looking for in DevStack. Yes. So there's this create initial networks. If you don't want it to create your subnets and networks where they might not be correct, you can turn this set to false in your local .conf. And you should be good. So finally, the third time, and maybe it wasn't the third time or the fourth or fifth time, but the third time it actually worked. So in this case you can see we have an IPv6 address for the host. Horizon is showing it's available over an IPv6 address as is Keystone. So from there, we knew we could crawl. So we decided to walk. So we started adding nodes. And over the next few days that week, we added six more compute nodes to it, all with DVR enabled. And we employed two additional network nodes. And the one thing you should know if you haven't done a DevStack multi-node install is the compute nodes require both the L3 agent and the metadata agent in order to operate properly. If you don't enable those in your local .conf, your VMs will boot and nothing will work. Either DHCP will fail or a metadata will fail. So it's kind of a thing you need to know for DVR. And we're actually currently using this for upstream development. I'm working on both the DVR and IPv6 and I will sometimes take upstream patches and put them in this system to see whether they solve the problems. So it's running close to Liberty Final. So most patches apply cleanly. Eventually we will have to upgrade this to a later version. So let me take a quick look at the demo. Big fail. Oh, it's always the way it works, isn't it? Sean warned me. We'll check in with it later. Let me restart it and we can talk through some slides. I do have screen shots. I know what the problem is, though. Again, we can fail fast with DevStack, right? It takes about five minutes for that to restack itself. Okay, these are still mine. So where are we going from here? One of the things we noticed in this deployment when we were doing the DevStack work is that the knob in DevStack that we added lets the systems change from IPv6 only mode, for example, for all your service endpoints in Keystone, to IPv4 only mode. What about that place in the middle where you might have both V4 clients and V6 clients trying to access this thing? I talked to one of the Keystone people internally and also brought it up in the design summit session around the service catalog. And I think one of the recommended solutions was, well, don't use IP addresses at all in Keystone. Use host names. So in this case, it could purely be a documentation task to say, well, if we return a host name, the client should be able to resolve that to the correct address for what call it's making, and then it should just work. So again, we're hopefully talk about that. I don't know if that session is tomorrow or the next day, but that's kind of the way we're headed because it's going to be the simplest thing to do. And then of course, since we did have a problem running all V6 everywhere, and we now know that there's an upstream patch for the Linux kernel and in the latest, I think 4.2 kernel, we want to go and pull that code even into just a test system and see what happens. So we really want to get that working because we're also going to need to update the OpenV, the Neutron OVS agent code because right now it's hard coded to V4 for that local IP check. It actually passes version equal 4 to Oslo for verifying that it's an IPv4 address. So we'll need, there's a little bit of work there, so the quicker we can get to this, the quicker we can have a patch up to Neutron and then just say, well with this version of kernel, this really should work. One of the other things, we chose VxLan and OVS purely because of our customers, but there's no reason that someone can't take the Linux bridge or some other driver and try that and get this working. So I can throw it out to you guys, if someone has an interest in trying to get this to work with Linux bridge or some other technology, feel free to contact us or try it and let us know on the mailing list or on IRC because we'd be interested in hearing how you did. And then Swift, I mentioned earlier on about Swift and getting back to the disappointment in the hardware is we didn't, Swift really requires I think three disks minimum to set up kind of a very small configuration. Since we didn't get it, we really kind of pushed it aside as well as the fact that DevStack won't configure Swift by itself. DevStack wants to configure everything and you can't tell DevStack to just do Swift. So we just kind of punted that one down the road. And then finally, there's vendor distros out there, both myself and Sean work for vendors that have open stack distros and working on getting support for this into our distros is obviously on future work for both of us. And back to Sean. Sure. So the reason why we've been doing most of this work in DevStack is as again I said it's mostly the CI infrastructure. This is where we need to do the work to uncover any problems. And the last couple of cycles I've been involved with the infra program to make sure that when we do DevStack it would then create all the V4 networking and then also then do the dual stack networking so we could shake out bugs in all the services so that dual stack would work correctly. So this is what we did probably last cycle and to ensure that the work that we had done within the V6 working group would continue and it wouldn't bit rot and that new features if they did affect IPv6 it would be identified quickly and patches could be done to rectify that. So the goal is eventually to have a CI system, I'm sorry, a CI job within the open stack gate that deploys everything with IPv6 only to ensure that all of the components within open stack can operate in these types of environments. Do it quickly now so that we can identify the issues far before any deployers are actually considering such an aggressive change over to IPv6 only. Most of them currently are just evaluating dual stack which is an additive configuration and then obviously the more radical one is taking away V4 only. One of the main challenges of this is in the infra code base where service host has been hard coded to local host. There are some patches that we're doing to remove that and then one of the other difficulties is that only one of the clouds that the infra program runs upon has real IPv6 networking so that makes it difficult to handle that where sometimes it's the V6 that we can rely on from the provider, sometimes we'll just have to use like a unique local address prefix. A lot of the scenario tests assume that the floating IPs are the way that you actually do connectivity into an instance and when you're doing a V6 only tempest run for all the tests there is no floating IPs in IPv6 because that implies NAT currently within the API and the behaviors. So the other thing is that tunnels are then used because a cloud that's running these tests could actually be spawned on one provider's cloud perhaps Rackspace and the other one on HP's cloud or another one they use tunneling to link those so that they look like they're on the same actual L2 broadcast segment so it looks something similar to this. But as you can see this was pulled from the infra documentation and it is primarily IPv4. There is code that already does the dual stack case but it's a lot of just work on our side to just teach the rest of infra about some of the configuration that we've done within DevStack and update the shared knowledge that everybody has. So I think the last thing we want to do is thank some of the other participants within the community that were crucial to getting this work done. If you don't see your name up here please stand up and be acknowledged and actually if you're on this slide as well please do. Yeah Jeremy and Adam were actually my two interns this past summer that did a lot of this heavy lifting here so I told them I'd embarrass them by mentioning them and having them on YouTube and then also because it was done in our data center there were a lot of people from our knock to our dev people and ops people that helped out here getting all the systems together and configured that there's probably more than they bargained for I think when I first asked for them. I think that's it. And that's it so thank you. And if there are any questions we were asked to have people come to the mic so everyone can hear you. I think we're the only thing standing between everybody and their party. Yeah we don't want to stand between you and the beer festival downstairs. Yes. Oh you're right. It's still running. It's still cooking away. I can show you the slide shots while it's working. So Sean was very smart and he had me make two screen shots. So if I ran the demo correctly and I did an open stack endpoint list when it was all done this is the output and I realize it's a little bit teeny tiny but you can see over here that every one of these service endpoints is an IPv6 address neutron at the bottom and even here this is our horizon snapshot looking at the server that was booted and you can see up here at the top that it's it's accessed over the IPv6 address of the system. So it is still it's still stuck so good job Sean for making making me take those snapshots and I did that in my hotel room right before I came down and it worked so that's the any questions you need to come to the mic please. Did you share somehow your local files for this DevStack installation? I'm sorry it was a little difficult to hear you. Your local configuration for DevStack did you put it somewhere that I can download it or what we'll do that's actually a very good point what we'll do is we'll put in the docs.openstack.org or if you go to devstack.org there's already a neutron guide what we'll do is we'll go ahead and start putting in our configurations to help you yes thank you. Yeah and it really was just that one if you put in that one line and you had an IPv6 address on eth0 it's automatic it's all done if you don't then you need an IPv6 address somewhere on the system and then have to specify it in that host IPv6 value so yeah it was that was part of why I think we got good traction was because we simplified that down to that option that you could go back and forth between v4 and v6 very quickly and and not have a lot of impact on anything else so thanks. Anyone else? Thank you very much.