 Good afternoon folks. Hey, so we are from Comcast. My name is John Brzezowski. This is Sean Collins, Anthony Vega My role at Comcast is basically to for the past almost 10 years I've been running the v6 program Comcast and over the past many years with Anthony and recently with Sean Sean is a senior engineer and a neutron developer. Anthony is a network engineer and he's also contributing to OpenStack We've been working very hard to make IPv6 support available for our internal use within OpenStack and our our goal here today is to share what we've done with you And also help to kind of raise level awareness because we think we really do need a lot more support within the OpenStack community as far as v6 is concerned So a little bit of background if you don't mind from kind of the bigger picture point of view Not sure how many people in the audience here are really familiar with this, but you may have heard before is running out Maybe you don't care, but I can tell you we do It's largely what I've been You know spending a lot of time for the past decade almost working on back in 2011 IANA and APNIC both ran out IANA is the global body that that kind of governs address addressing for the world APNIC serves Asia Pacific ripe subsequently 2012 You know met a similar fate and just a few weeks ago here in the Aaron region in North America They too found themselves in a situation where they they dipped into their final slash eight That is the current definition of depletion Which basically means that when you go back to these registries you can no longer request Or expect to get the same size blocks that you got you know previously So from our point of view we saw this a long time ago Like I said about ten you know about ten years ago in 2005 is when I started a Comcast And from our point of view v6 is truly you know We see that as kind of the future for us from up from a growth Really allowing us to kind of develop kind of services and products that we want to take to our customers in a way That's going to give them the best experience As you may have taken note of the past couple of years or event There's been a lot of industry activity from a v6 point of view We've been part of that and you'll notice that v6 deployment has certainly picked up I remember looking back a year from a year ago when we were in Portland There's there's just a heck of a lot more people who are talking to us about p6 support Which we think is really great thing, but at the same time We're also finding that the absence of v4 and still the kind of the aggressive growth on the v6 front is still Making it so that some folks have to deploy transition technologies as you'll see here at the bottom We make reference to category not just one of a host of many that we think people kind of should pay close attention to you Because we do believe that it will affect the way that people access the things that you build on the open stack infrastructure Next slide So again our goal is ultimately to relieve our dependency on IPv4 and we've spent You know a long time really chipping away at that the most recent chapter for us is really what we're doing here in in Open stack we can tell you from very Long and sometimes painful experience that the path of v6 is not overnight We've had many conversations with people over the years and tell them you know when you decide to play v6 You know, maybe it's not right now. It is not something that you can just do You know you go to bed on a Friday night wake up on a Monday morning and decide that you're gonna flip it on It just doesn't work that way And I think I'm preaching to the choir when it comes to kind of how technology has to be deployed in incremental fashion One of the things that you've probably seen from our perspective over the years is that one of the key aspects of our deployment at Comcast particularly from a broadband perspective and Has been to enable dual-stack Connectivity to all of our customers and a few slides will kind of give you some tidbits and some facts around where we are and where we're going But ultimately our goal is to really get to a place where we can run the network and offer the products and services in a v6 only Configuration that's the goal. All right So where we are and where we're continuing to move towards is moving to a place where we only use v4 where we have to And we want to we want to ultimately leverage IPv6 to simplify your thing, you know So you may throw a rotten fruit at me, maybe not but you know when we met within some of the meetings Yesterday some of the operator forums, you know, you know and a lot of conversation that we had conversations We have here in the design meetings talked about that and and all the things that kind of We're doing to kind of work around some of the challenges in that That's exactly the some of the space that we're trying to avoid from our perspective And we feel v6 really is kind of a key network technology that allows us to restore the end to end We are absolutely trying to avoid transition technologies We wreck you know, I think everybody recognizes that there are side effects To a lot of that a lot of that technology And ultimately like everything that we've done our goal is really to start early lay the groundwork so that we can grow into We can grow with technology and the solutions that are available to us to ultimately to deliver Well to deliver a proper customer experience To our to our customers next slide so so it is May I think and In just this past month April we completed the rollout of our deployment across the broadband network So what does that mean? That means 100% of our broadband access network now is I'd be six You know enabled with the capabilities for for customers to walk up plug in their own You know retail device or use one of ours and have v6 on by default huge milestone for us and Something that we're very proud of and you know, I know that you'll hear about some more in the weeks to come We are in fact the world's largest native IPv6 deployment 30% of our customers are actively provisioned with native dual stack huge absolutely huge if you go to World IPv6 launch.org. It's an internet. It's a website. That's run by the Internet Society Every month they update and tabulate updates from all the participating service providers and show you, you know What the deployment has looked is looking like our goal by the end of this year is to be at a place where we have 50% of our broadband customers being provisioned with native dual stack So I want to make sure everybody's clear on that, right? So 100% of the footprint is available I mean it doesn't mean that 100% of the customers are actually using it but what it does mean is that 30% of my customers have been provisioned v6 and Are are using it on a on a daily basis? Many of our products and many of the things that we're doing from an infrastructure point of view already support v6 Right, so broadband residential broadband. You're hearing about right now Commercial broadband services in the queue, you know, you know higher-end commercial services like Metro ethernet all support v6 And there's a lot more to come, you know, where from you know from where we are right now All of our infrastructure supports v6 If it didn't then we couldn't really take the steps that we're doing now to continue to enable products and services Would be sick support and of course our customer-facing infrastructure must support as well And and this is obviously where things like open stack coming to play. Next slide Sean So little little plug for us here. Hopefully you guys don't mind the kind of the the shameless plug here But this is basically a picture that the internet society has put out for all the participating Service providers and this is a graph of our growth from May of last year until till now and you'll notice that the growth is pronounced and there's more to come on this front Next slide so a little bit I've given you a little bit of introduction around what the You know our deployment looks like right now I went handed over to Anthony he'll enumerate some of the additional use cases that that we're leveraging v6 for today Anthony so all this talk that John's Been telling you about getting us on to dual stack and making sure that our entire deployment is dual stack There's a reason for this if we didn't go dual stacked. We'd have a lot of issues around You know getting access to systems and like John said we're trying to avoid transition technologies wherever possible a couple He's really use cases The X1 video set top box as I'm sure you saw in the Portland Summit Mark mule gave a nice demonstration of the X1 set top box which runs off the cloud if We didn't have v6 It would be incredibly difficult for us to be able to deploy the large number of devices that are needed to get The entire footprint to use something like this They consume lots of IPs and in order for this to work We need to be able to talk to them to manage them and for them to interact with the rest of our network over a dual stack CDN is a good one if if you want to get things on all kinds of devices your iPad your PC Or the set top boxes that you use in your home These need IP access they need to be able to get this data from our core network and for that to work We've got to have our CDN to be dual stack enabled in order for the data to get to you Messaging systems things like email or voicemail or text message systems Things that you can go to like Xfinity connects All the products that we offer our customers need to be dual stacked for the vision of a v4 minimal network to be possible Voice services is a good one if you've got Xfinity voice then the MTA is that you connect your phones to are going to be dual stacked in order to receive service And lastly but not least the DNS platform All of this talk about getting broadband enabled in the home on v6 could technically work with v4 DNS, but it wouldn't be ideal and We really mean it when we say we want everything to work in v6 We mean the whole stack from the bottom up and that includes services like DNS which are a very large part of the network So what's our approach been so far? Initially it was just getting end-to-end functioning We worked on the core make sure the core passes packets properly then we work on the edge devices and make sure that We can get access from one end of the network to another because really that's the basis of the foundation You can't do the management tools. You can't do the access tools unless your core works properly v4 Largely unchanged. We didn't really touch much of it. It's still operating the same way. It's been operating for the past 20 odd years things have You know continue to stay pretty steady But we've added v6 and I'm pretty sure most of you haven't noticed any changes That's a good thing if we do our jobs right you won't notice because there won't be any issues and so far That's been the case Initially you might need v4 right some of your tools might not do v6 yet But maybe the access works maybe you want to get access to your customers Maybe you want them to go on v4 or v6 transport, but you can't really deal with the devices that they're on To be able to maintain them without v4. It's okay. It's gonna be part of the process Getting to a dual stack world isn't something that you just dive into it's something you have to go in incrementally into But we're working to define the requirements for the cloud at this point so What do we need? What do we say when we need dual stack right? There's a couple different ways you can approach this you can say slack. We're done. You can get a v6 address and we're out That's okay for some people and that might actually be a good target for a first step Next step. Well, maybe we want something more important. Maybe we want something a little more fine-grained We want to put extra options in DHCP v6. Maybe that might be your next step There's other options, too We're we realize that we're not the only cloud operator that wants IPv6 and that there are other ways to do this So maybe you want to use cloud in it. Maybe you want to use made of data That might be another option and down the line or prefix delegation So what we're doing at concast is trying to help with the new trying v6 sub team to define all the Requirements that are necessary to get dual stack to work in your environment so the real reason for this and and You know the primary purpose isn't really to get all of the tools in open stack to be speaking v6 day one We know that's not really You know achievable goal is to get the endpoints to talk to each other when you get a v6 address on your hosts on Your VMs on a neutron port on an LBAS port whatever you have set up when those things can talk to each other And to things outside of the cloud Step one's done right. That's the idea The problem with this though is that you can't have anything in v6 Not be on parity with v4. So what we need is performance We need the tune ability We need you to be able to pass packets at the same or similar rate that we do for v4. Otherwise, nobody will want to use it The other thing is you know, it's good to say we've got slack And we can send packets up the network and an L2 flat network works Cool, that's step one it again step two here would be getting away from just a flat L2 bridge domain And to go into things like overlay networks GRE tunnels getting IGMP or getting IGPs and BGP to operate in the cloud so Anthony and John have talked to you about where we came from and where we want to go in the future and As a neutron developer, I'm going to bring you into Where we are currently and how we're going to go into the process to see to realize the vision That has been set down. So back in October Shridhar Basam John Brizowski myself and Anthony Vega Sat down and did a hackathon For OpenStack to determine what changes would need to be made in order to get IPv6 Provisioned on an instance that was booted through Nova using Nova net using neutron So the result of this hackathon was that we've identified a number of changes And bugs that were filed and the fixes In order to achieve this goal. So we had a small change into Nova We had some changes that we identified to the security group API and then also some difficulties with DNS mask So we took all of these changes and we realized that we can't be the only one That is interested in IPv6 in OpenStack So one of the very first things that we did was to publicly announce If there would be interest in forming a working group within the neutron community To gather up all of the people that are most likely have found these issues on their own and try and coordinate all of this development effort towards a common goal We also proposed a blueprint to create a new attribute for subnets to help Drive the provisioning process of an actual IPv6 subnet within neutron because the current Attributes that were part of the neutron API were not sufficient So as part of that that original blueprint print idea transformed Through the community process into a new proposal of two separate Attributes one that would deal with the addressing mode for neutron And then also for the RA mode in neutron and the reason we split that up was that there are going to be some cases Where a router that is actually providing routes for an instance is not orchestrated by OpenStack It could be a hardware device the point is is that there's probably many different types of implementations and Neutron should attempt to support as many as possible and not make decisions for you So all of this work was done in the community through the IRC meeting that we started on a weekly basis The mailing list discussions that we had and then also the blueprints So this is a consolidated list of all of the possible values that we determined for an IPv6 subnet And what you'll see is on the left hand side The attribute values for both the RA mode and the address mode If it is actually a valid combination and then what it actually turns into on the network The flags that are set in an ICMP v6 packet that's transmitted on the network So I'm also going to demo to you today through some screenshots some of the work that we currently have in progress that has been put into The community and is currently under review But is running in a lab environment to demonstrate that the patches we have are viable and that it accomplishes the work that needs to be done So as you can see we've created a v6 subnet that is going to be driven By an external device an external router because you see the IPv6 RA attribute is blank and Then neutron is going to use Slack and EUI 64 to determine the address So that when you do API calls and horizon the address will show up properly so that the user has that expectation of I'm going to launch an instance And I'm going to know its IP address is carried on So as you can see here's a screenshot of Horizon are in our lab where we've launched a couple of instances Demonstrating the v4 and v6 address that they are all of these are dual stacked and Then if you have the next screenshot You'll see that in the side of the console the instance correctly came up with the IP IP prefix and the address that it would be assigned and routes are ready As an additional step you can see the properties of the network that we've created that is a dual stack network With both the IPv4 and IPv6 subnets Being child objects of the network in the neutron API And I'm going to turn it over to Anthony for some of the caveats we have of our current design So the idea was to support as many configurations as we could because we know that everybody's going to have a different network There's no one single silver bullet to fix IPv6 and neutron That being said Some of the definitions in the rfc's go a little bit beyond what we can immediately calculate and Predict with neutron so privacy extensions is kind of a big problem right if you're going to use slack some systems want to use the privacy extensions mode to calculate their address and Because of the nature of the way privacy extensions operates It's a little difficult to calculate the address ahead of time that you're going to use so Therefore we don't technically support privacy extensions. You can use them But we won't be able to enumerate your address in horizon or an API calls Link local security also came up so one of the things about IPv6 and the way it operates as many of you probably know is that a Router has to issue a router advertising packet or at least issue after being solicited This causes a couple of small concerns around security things like rogue routers or rogue RA issue. It's become an issue We have patches in They haven't been accepted yet, but they're running in our lab demonstration that actually restrict down What you can and can't set as a value for an RA? So security groups will actually filter out RAs that aren't listed as a neutron router port Multiple addresses so Unlike v4 v6 can actually add as many addresses as you want to an interface without breaking the stack While that's possible with node It's a little difficult to do in horizon right now because neutron doesn't technically support multiple addresses from the same subnet on a port So while it's possible, we won't be able to handle it with the API currently And then there's the issue of floating IPs So we know this is a sore point and it's something that needs to be fixed in the future The current implementations don't actually have a method of handling Floating IP distribution there's no way to assign them to a VM right now unless you continually poll made a data service So right now there's no floating IP support. So what are the next steps? How do we move forward from here? Right now there's a couple of patches in flight. There's plenty of reviews up in Garrett I would encourage everybody to go take a look at the filter look for the IPv6 flag to find all the patches that need reviews If you can please go take a look plus one or minus one at least its progress, right? There are patches in right now for DNS mask one of the big problems with all of the ways that neutron currently handles non-provider network is DNS mask is pretty much the end-all be-all for configuring a system That's great in the v4 world where everything assumes DHCP at boot time with v6 unless you've gotten an RA DHCP Isn't even going to trigger so we need DNS mask to be able to issue router advertisements DNS mask actually supports it implicitly in code But the patches are in right now for us to be able to program that through the neutron API's There's a couple of different patches going through for different aspects of slack in DHCP capabilities from a couple different teams We've had some great participation from the IBM team NIFO 6 and Cisco And there's a couple other people on the distribution that have been involved There's the patches that we just demoed to you They're up in review, but like we just showed you they actually work We're using them today So there's other things that need to be addressed right the the plug-in networks for L3 things like ML 2 Mechanism drivers that need to be able to handle things that may not be the simple Nova Network style configurations All those are going to need to be addressed. We need to be able to provide a v6 prefix v6 advertisements and eventually things more advanced services like prefix delegation or being able to route DHCP v6 has to be implemented Like I said, we've got DNS mask supporting code, but the orchestration mechanism is there in the API so if you've got You know some time to sit down in the code and want to deal with the neutron API's God bless you But we could use some support in getting stateful and stateless into neutron Routing is a big one. We know it's it's also an issue for v4 But in the v6 world, that's actually a little more of an issue if you want to do things like prefix delegation You actually have to have static route injection at least and preferably a dynamic routing protocol enabled And ML 2 options like I said the the orchestration between all the different Mechanism drivers isn't there yet and if you own one and if you're contributing to one I'd encourage you to make sure it's still stack enabled prefix delegation Like I said, it's a pretty big beast. There's there's a lot of work to be done around it With how we want to divide up networks the allocation algorithms how we want to sign them how you want to handle hinting through router router solicits and DHCP and how you want to handle routing down to those prefixes or if you want to build them as Subneutron networks or not and like I said, there's other advanced v6 features things like Segment-based routing that could be used for advanced network functions that are native to v6, but again need to be enabled in Neutron Alrighty, so back to me. So I'm sure Anthony. We would prefer plus ones instead of minus ones and You know, so we certainly encourage you to plus one things The the other thing that we'll take note of here or at least I make mention of to you Is we're doing a lot of work with the with the developer of DNS mask So a lot of the items that that Anthony talked about from DNS mask point of view as far as prefix delegation are concerned You're gonna find we'll make its way into You know probably the summer into the mainstream DNS mask as far as prefix delegation is concerned and even some prefixes delegation for that matter, you know because a lot of that functionality We're looking to use in in other areas as well. Not just open stack So so as it was alluded to a minute ago, I mean there there are a number of things that we're also looking at from a from an open stack point of view and Some of them intersect with the work that we're doing from a v6 perspective Anything talked about segment routing. I mean, you know, we're also looking at NFV and service function chaining So there's a lot of work going on right now in the ITF I don't know if there's a lot of people who kind of you know, hang out at ITF's Wouldn't surprise me if there was some intersection here But but that's definitely an area of interest for us and truthfully the bulk of that from our perspective is very much v6 only Right, so a lot of where we're taking our strategy for for the network moving forward is very much v6 only You know, you know, I don't know how many of you feel about this But from from some perspectives for us, you know, there's there's no more life in v4, right? So I mean we're we have to continue to support it But if the day is done, you know, we're looking to build You know the network for the next decade and and from our perspective v6 is a huge part of that And a lot of the work that we're doing here is is going to be it's going to be a cornerstone for that Some of the things that we've seen over the years anecdotally and we're also looking to do some some research and papers here We've also seen that in some cases and it's actually as a matter of fact It was Anthony who kind of brought this to my attention years ago, and I didn't believe him at first But we anecdotally have seen that v6 performs better than v4 in some cases We don't have all the raw data kind of analyzed just yet, but but you know, I basically told him I didn't believe him at first and and that was a pretty Was a pretty bold statement from from from a person like me who's been doing this for a long time at Comcast So that was that that's work that we're continuing to kind of explore And of course, you know as we mentioned in the front end of our talk here, you know from and you know Open stacks is a critical part of all the things that we're going to continue to do from a Infrastructure and a product services point of view so from our perspective making sure that it meets every six requirements and hopefully Yours as well, you know what we're trying to continue to advance support there and Possibly even look at you know new services that we could that we could create that yeah, maybe would not be necessarily available You know either without kind of infrastructure or without Over that v6 next last one So I think we are actually ahead of schedule kind of so Like I said before there's a lot of stuff that's either in flight or not implemented yet. So how can you help? We can't do it all on our own. We recognize that we also recognize that there's There's scenarios that you guys want to implement that I might not have thought of or that Shawn hasn't coded yet. So help Requirements if you've got them Please bring them to our attention write a blueprint show up on the mailing list discuss it Show up at IRC have a chat with us Join the sub team speaking of IRC There's the link slides to go up later, but there's a sub team in neutron specifically dedicated to IPv6 So we've got some pretty dedicated people in and they're pretty knowledgeable about v6 You've got a question even if it's just about how v6 works in general show up Please ask your questions. We'll be glad to help out But if you want I'd encourage you to join the sessions there and bring blueprints Bring bring code reviews bring code patches. We welcome everything code neutron Neutron itself needs some fixes right the API needs a little cleanup But in order for it to work properly for v6 we need to extend it as well And please I'm sure you've heard it from every presentation You've been to for the past two days, but review it helps a lot and elsewhere so The cloud is is a big thing for everybody and open stack is a great tool for orchestrating it But there's also applications that run on it and if you've got an application that's v4 only or uses what I'll call IPv4 Isms that would prevent it from operating properly in v6. I'd encourage you to take a look at those code bases as well So last but not least wanted to kind of definitely, you know reinforced a message here You know it's free PR from our perspective for the people you know who we list here But basically, you know, we're not the only ones right and we know just over the past couple of days There are many other people who are as you're working on v6. You'll note that there are some very There's very, you know recognized names on our list here Facebook Google cloud fair cloud flair And of course you get a world v6 launch. You'll see a whole host of other people who are actively deploying v6 you know and I have to say Over the past couple of years, there's just been an enormous amount of movement and We certainly feel like there's you know good company that we're keeping on this front So I think what we'd like to do now is turn it over for some Q&A We have about 10 minutes for that. So by all means, so Well, yeah Your broadband connection. Yeah The question was is it working on your Xfinity yet? And it is Or at least it should be depending on what kind of device you have you have a Comcast right device It absolutely should be and the fact that you're asking me tells me that we did our job, right? That you didn't even notice Assuming that the state of Tempest for IPv6 tests is non-existent Have you guys looked at contributing as anyone in the community been looking at adding to Tempest? so We're working with Eno Vance from Shridhar Gaddam To do some additional patches to Tempest along with Henry Gasal from Cisco To start exercising portions of the API and then we are also investigating building more scenario tests to actively spawn instances Determine the IPv6 address that they have and do all of the network verification steps that the gate currently does on the V4 side As well as the V6 side. So yes, we are very interested in getting that As part of improving the quality of neutron and ensuring that Patches will not break that functionality once it is landed and the same kind of question for rally as well for benchmarking at scale Unfortunately, I don't know very much about rally. So we would certainly love to have somebody who's an expert in rally Inform us as to what we can help with Yeah, I'd say for that Please show up to a V6 subteam meeting or shoot it to the mailing list and start a discussion about it Hi there My name is Patrick and I must admit that I'm completely new to open stack, but I do run a productive dual stack network and I thought it was common wisdom that you do not run router advertisements in a hosting environment. My question is why don't you simply use link local addresses for the volt gateways? So that yeah, that's a good question it it is common wisdom in a Static network style environment where you don't have an elastic compute setup that you don't want to matter advertisements because it's a security vulnerability We're kind of breaking that common wisdom because it may not be wisdom in a cloud system, right? So the concept of elasticity in something like open stack means that I need to be able to provision new systems incredibly rapidly I may not actually be able to provide the base OS its configuration information up front What you're saying what you're suggesting would actually require running something like a service VM to do a local configuration Inside the neutron network before a cloud VM can even instantiate or you're suggesting that would actually have to have some intelligence in the base image To query a metadata service, which is one way of doing it and we're completely amenable to that solution as well We just think it there's extra possibilities because we control the network layer That we can also filter out our a's that shouldn't be coming from a device That's not actually a router because we can do that We can actually prevent some of the vulnerabilities that you would normally see in the kind of static environment where there's no Host layer controls that the host doesn't control so to speak Things like security groups enable us to prevent the normal types of attacks. You'd see with our ace boofing as a part of that we also have patches that landed as part of the ice house Cycle where we control What ports and what IP addresses are allowed to transmit our a's that them are broadcast to instances as well And that was given to us by the people at IBM Zuhan Pank So we were aware of the problem and we are attempting to address it as well Yeah, we've been on top of it was actually followed as a bug as soon as we implemented v6 slack And Patrick just for you know to for technical clarity you always need router I mean you should have router advertisements to get to at least learn the default route was your question around Also, was it really oriented around slack or was it actually the suppression of router advertisements overall? I suppress router advertisements in my network completely. We don't at all. I use some link local high availability protocol like hsrp version 2 or glbpp to provide a default gateway with a link local address Yeah, we basically in all of my broadcast domains. Yeah, we use our a's As and and the link local address is really the only one But we should talk afterwards because that's been something that we because one of the pieces feedback You know when I when I first you know when we all first are talking here One of the biggest things that we want to try to do is accelerate the way people could get their nodes online with v6 support not kind of Continue to have them jump through hoops and that sort of thing. So it was it was it was a conscious decision I'd actually say that that's that's a really good example of what I was talking about earlier and that We at the sub team can't Dictate any one method, right? We can't know what everyone's requirements are so bring your requirements come to the sub team send a message to the mailing list I would be glad to include your provisioning method in the neutron v6 capabilities Talk to you later. Sure. Thanks. Are there any anybody actively using open stack with v6 question The only change that we've had for Nova was in the Libvert driver To disable hair pinning when you are using Nova with neutron and that was just so that if you're deploying a Network and you're using slack You won't have a problem with duplicate address detection where the packet goes back into the instance and causes IPv6 to fail. So I think you might also have been asking about Nova network. Yes, right? So Little political thing there were intentionally not backporting the Nova network We think it might end up causing more pain in trying to get the seamless upgrade to work and That the cores would prefer it to be neutron only So quick question for you guys anybody actually using v6 and open stack out there other than us really It's cool. I mean we'd love to hear from you. I mean I think We met before yesterday, but if there's anybody else in please approach us and we we'd love to hear more about what you're doing One of the things that we can share with you is just last Friday for anybody who's kind of familiar with v6 There's a there's a fella that we know Jason Fessler who is the basically the developer of test-ip6.com Years ago probably about four years ago now Comcast deployed a version of test v6.com so that so that we're test-ip6.com And we basically gave it the FQDN Test-ip6.com cast net because we really felt it was a valuable tool for not only our customers But it was also a valuable tool for for my you know our co-workers to really kind of diagnose and and troubleshoot things v6 Just this past weekend You know we have we have an open stack v6 enabled node that is now in the GSLB rotation for that service and believe it or not It's a it's a pretty you know I mean it's not you're not talking huge amount of traffic, but but for a service that You know for a service that is you know basically Used for testing and troubleshooting it gets it gets used pretty heavily so people use it a lot for validation They want to know what their v6 address is they want to make sure that their DNS Resolver and their servers are working well, so we recently put one of the one of the the nodes in that GSLB rotation on an open stack cluster and To everybody out there that raised their hand If you get some some free time, I'd say Show up to either the dev mailing list or even the operators mailing list because if you're using it you're an operator so if you've got experience running it if you've had any issues in the past and Had workarounds or no problems It's good to talk about it. It's good to hear from other people that are using it and what they're doing with it All right, any other questions comments rotten fruit All right, thank you all. Thank you so much. Thank you You