 Hello, everyone. My name is Ciprian Hackman. I'm a DevOps engineer working for Poly Poly, and today we'll be talking about the ROTE IPv6 support in K-Opps. What is K-Opps? K-Opps is a cloud-aware Kubernetes installer. Provision is the underware cloud infrastructure and also installs the Kubernetes 4 components, including add-ons. Since K-Opps owns the life cycle of both the cloud resources and Kubernetes, it can handle all aspects of creating an IPv6 Kubernetes cluster. Focus so far has been on AWS and GCE, but adding other clouds should not be an issue. Why IPv6? Larger address space allows for simple flat network architecture. No need for NAT, no need for complex calculations and doing magic with subnets. Easier to avoid IPv6 conflicts between VPCs accounts companies. Native cloud route, native routing for pods allows direct targeting behind load balancers without going through kube proxy or node port. It's both performance and debugging improvement. CNIs can also be much simpler if they are designed for IPv6. This is not to say that current CNIs are not IPv6 aware, but they are designed with IPv4 in mind and have to deal with all the limitations of it. We decided early on that we would support single stack addressing for pod networking and do all stack at the edge. This allowed to reduce the complexity of the development and very easy to assess the current status of IPv6 in Kubernetes. Also support for dual stack is quite complex and negates many of the benefits of IPv6. Not many would be able to migrate IPv4 single stack clusters to dual stack anyway, again due to complexity and risks and downtime. As at the time instances were only allowed to get single IPv6 addresses, we decided to use unique local addresses. This does not unlock the real benefits of IPv6 as NAT is still required, but the assessment of Kubernetes readiness can be done. It also requires overlay networking and tunnels between nodes, but we were ready for that. Well, ready, but not quite. IPv6 must be configured in lots of places and I really mean lots of places. So both control plane and CNI must be configured and they must actually agree on that configuration. After we found out the right settings with the help of the Kubernetes developers, we hit a challenge with AWS cloud provider. It was not aware of IPv6 addresses at all, which was not really good for us as we needed it to assign the IPv6 addresses to node objects. We fixed that and again another challenge IP addresses were not prioritized as we expected. After many debates, the decision was that cloud controller should send the IP addresses in the order that it desired to be used. So it basically decided the priority. We fixed that and fixed that and tests were possible now. We created periodic tests and we tried them for some time. During this time together with other developers, we fixed all the needs that we found mostly testing issues, but there were some minor issues in Kubernetes too. And we thought that this was it for now. What a surprise. My name means so much, yes. And I work at Sport River and at Sport River, we are really interested in those benefits that we mentioned earlier. And last fall, AWS released a number of IPv6 features that allowed us to implement just that. So we decided that local unique addressing isn't really going to give us any of those benefits. So we decided to drop the port for that and focused on global addressing instead. So global addressing required us to do quite a few changes in how we provision our networking. So we had to enable not 64, but 64 so that the cluster can reach things without an IPv6 address. We had to do some changes to the round tables. We had to do some changes to the north templates. On instances, we also had to do things like disabling last grade and overlays, since they would be getting in the way when we are using global routing. Launch templates also doesn't allow you to configure both an IPv6 address and a prefix at instance creation time. So in the launch templates, we said that instances are assigned automatically a single IPv6 address. While during the bootstrapping of the instance, we had to create our own object for provisioning IPv6 prefix for the instance. We also had to do something that would take the assigned IPv6 prefix and give it to the corresponding node object so that the cluster knows what IP addresses on schedule for the given instance can receive. When all this was put together, we had something that looked like this. As you can see, there is a mixture between IPv4 and IPv6 addresses. And the reason for that is not queuing up. It was a little bit inconsistent with what address family it used for host network. So all the points that you see here with IPv4 addresses are those running with point network enabled. And this cluster lasted for about five minutes. We were primarily testing on Ubuntu. And on Ubuntu, there was a bug in system D network B that when they tried to renew the DSP at least, it would get a reply from AWS containing both the single address and the prefix. And it would fail to properly parse those and just decide to remove everything. So all the instances after about five minutes no longer had any addresses assigned. On other distros that we were testing this with, we had a few other minor issues. Like, for example, it was a bit tricky to configure the kernel to properly accept the router announcements. Since when you're using prefixes, router announcements are disabled by default. But we actually managed to either fix the merchant bugs or work around them. And we now support the VN10 and new Ubuntu LTS where it was just released. As mentioned, we had to create something that was aware of the prefixes that AWS would assign to the instances. By default, there's a controller that looks at the whole cluster-wide sider and takes the subnet from that big sider and assigns it to each of the new nodes. And that wouldn't work since the node sider has to match the prefix assigned to the instance. So we had to disable the controller that counts this logic by default and create our own controller that was aware of the prefix that AWS had assigned and that patched the corresponding node object, giving it that prefix as its pod sider. The really cool thing about this is that CNIs are completely unaware of what's going on in them. So all the cloud-specific stuff with prefixes and all that is hidden behind that controller. So that means that any CNI can support global routing with every six out of the box without doing any changes as long as it's configured to look at the node object's pod sider. The controller currently runs as part of the X controller, but theoretically, it could be a final standalone or it can run as part of the control manager. Hello, my name is John Myers and I work at Proofpoint. At Proofpoint, we are particularly interested in how IPv6 can solve problems with local IP address allocation. CAOPS initially implemented IPv6 using public topology because it is simpler. In public topology, all nodes are dual stack and are accessible from the public internet. In order to allow non-host network pods to connect to IPv4 targets, we enabled DNS64 and core DNS and the local DNS. In order to do the latter, we submitted a PR to Kubernetes DNS adding the DNS64 plugin. CAOPS creates a NAT gateway in each zone in order to perform the NAT64. After AWS released support for IPv6-only subnets, we implemented private topology. In private topology, we have three types of subnets. First, at the edge, we have dual stack utility subnets. This is where CAOPS put NAT gateways and public load balancers. It does not put in any nodes in these subnets. Worker nodes are placed in IPv6-only private subnets. In order for the nodes to connect to IPv4 services, CAOPS enables DNS64 in the subnets AWS DNS servers. It routes the NAT64 range to the NAT gateway in the zone's utility subnet and the rest of IPv6 to an egress-only internet gateway. For the control plane nodes, CAOPS creates a new type of subnet called dual stack. Control plane nodes need to be dual stack because CAOPS uses instance targets for the API server load balancer and AWS load balancers only support IPv4 for instance targets. This type of subnet is like public subnets, except nodes don't get public IPv4 addresses and it uses an egress-only internet gateway. CAOPS needs to make some arbitrary assignments for in-cluster resources. As these addresses are not routable from outside the cluster, they are made from the unique local address space, though they are not technically unique. For the Kubernetes service network, I attempted to spell service in x-decimal. You kind of have to squint a lot. For node local DNS, I attempted to spell node DNS. There is additional work needed in upstream projects in order to get a practically functioning cluster. At the time this talk was recorded, we don't yet have the AWS load balancer controller working. This is probably because CAOPS wasn't giving it sufficient IAM permissions. The AWS Cloud Controller Manager service controller doesn't work with IPv6, so we will need to disable that and use LBC for services of type load balancer. A lot of LBC's defaults are only appropriate for IPv4. We hope to make it configurable to use a set of defaults appropriate for IPv6 when running on IPv6 clusters. External DNS currently doesn't support QoT A records. This is a work in progress. Hi, so I'm Justin, I work at Google, and so I thought it would be good to tackle the question of how specific to AWS this work is. Now that all this great work has been done by the Kubernetes community for IPv6 and by the CAOPS and AWS team specifically for running a full cluster on AWS, can we just run this on another cloud? Well, the short answer is that it's pretty straightforward. We have to enable IPv6 on the cloud resources like subnets and VMs, but then most of it carries across to GCP and presumably to other clouds also. There are some minor differences like on GCE, VMs automatically get a slash 96 IPv6 range, and so we don't need to write code to request it for each VM. But in general, the sequence is pretty cloud agnostic. The VM is assigned an IPv6 range, our IPAM controller writes that into the nodes pod sider, the CNI provider then assigns pods IPs from that pod sider. IPv6 siders are fully routable, and so we don't need to configure anything. If anything, we now have the inverse problem. We need to be very sure that we have firewalled off any access that we don't want. And generally, because this sequence is cloud agnostic, the good news is that by building on the AWS work, it almost just works on GCE. Here's a quick screenshot, showing the classic guestbook example running with IPv6 pod IPs across a few different nodes. These are in the 2600 range, so these are fully routable IPv6 IP addresses. The gotcha, though, is that not everything quite works when it comes to networking. The IPv4 Kubernetes networking stack is pretty complicated, but for IPv6, we need much less. We don't need an overlay. We get a sider with enough IPs. We don't need routing logic. The siders we get are fully routable. IPAM is pretty simple now. There are plenty of IP addresses, we don't have IP reuse problems, and we generally don't need to worry about running out of address space. But as we've seen with AWS, some of the assumptions made by the stack simply don't apply anymore. Right now, we have a few snafus. As with AWS, the classic way of exposing services with type load balancer doesn't really work. The load balancer is exposed on an IPv4 address, and cube proxy and IP tables can't nat that to our pods. Because we're turning off a lot of layers, like setting up cloud routing, sometimes we now miss steps. For example, the network unavailable taint sticks around, and we now need new code to remove it. All of these are surmountable problems, but they are new problems. Maybe, hopefully, by the time you're watching this at KubeCon, we'll affix all of these. But I think what's most interesting for the future direction of CNI in IPv6 is that these aren't the problems that CNI providers addressed in IPv4. CNI providers in IPv6 are going to be primarily about a small bit of configuration, and generally getting out of the way. What's far more important in IPv6 is firewalling and generally preventing access. And those are pretty different requirements to IPv4.