 Hi everybody, thank you. Thank you for coming. We are tired to talk and to cover more of the aspects of IPv6 and Kubernetes that are not clear. But first of all, let me introduce us. I'm Antonio Hea. I'm Kubernetes contributor since 2019, I think. I started working on IPv6 and Kubernetes and this talk is because we started in Twitter complaining about IPv6, about Kubernetes, and then we met and say, well, there are a lot of things that we don't know about IPv6 and people don't know about Kubernetes and it's not clear how this is working. So why don't we meet and we talk and explain to people what are the problems? Let me introduce myself. My name is Fernando Gont. I'm an active participant at the ITF and I have done a lot of work on IPv6. So that's how we ended up getting together and joining forces about this. Go ahead. Yeah, so the key thing here is I work in Kubernetes and he works in ITF. So they define the standards that we have to use and in Kubernetes, we have an open source software project. We define the standard that we want to use and we need to use the standard that they define. That is not clear, but I think that you get what they're trying to mean. So in Kubernetes... Okay. There you go. Okay. So here we go. So in any software project there are two important concepts that we need to clarify. One is the reference architecture and other is the reference implementation. The reference architecture in Kubernetes is what we define with the APIs. We define APIs and we define behaviors. And then Kubernetes is a software project. So we deliver software to implement these APIs. The project is a project that is evolving continually. So you can see now some of the components of the cluster, but in the past there were more components. The project keeps evolving, keeps modularizing the different components and then defining new APIs. And then it's when in a Kubernetes cluster you have a reference architecture and a reference implementation. And this is the first myth. You don't have to use Qproxy, for example. We, Kubernetes, define a service API and Qproxy is an implementation. You don't have to use IP tables. You can use EVPF. You can use other technologies. You don't need to have to use a lot of things. So first myth of all that we need to clarify is Kubernetes defines APIs, defines behaviors. How Kubernetes ensures these behaviors are compliant. It defines E2E tests. There is a whole conformance project that gives users the security that they are clusters of conformance with what Kubernetes defines. Okay. So we talked about APIs. This is just, we want to touch a bit on this because this is important to understand a bit better what is next in the talk. The API represents an object on the cluster and the API represents the content that we have with the components and with the clients and with the users. An API, I'm not going to go deep on this, is defined by a group and a version and a kind. And then you have some metadata to define some properties of the objects. And you have two important things here. You have the spec and you have the status. The spec is what the user or the components expect to happen. It's declarative. And the status is what the component on the cluster report to the user. So it's informative. And this is going to be important later for the presentation to understand better what are the myths on IPv6. So we can see here, the spec is decided state and the status is observed state. So more or less this is not, was not about networking, was more about general aspect of software development and Kubernetes in general. So let's go a bit deeper. So what is Kubernetes? Kubernetes is a platform that orchestrates containers, but the container is not the minimum application of Kubernetes. It's the pod. A pod is, we can say it's different containers that are running in the same networking space. And to run these pods, we need a compute. In Kubernetes, the object that represents the compute is the node. So what has a compute? We need network. We're talking about network in this talk. We need an old network. The compute has to be connected somewhere. And we need a pod network. And this is something that Kubernetes already defines. This is what Kubernetes defines is that this pod network has to have some special properties to meet the Kubernetes reference architecture. So it says that every pod should be able to reach every other pod without not in the pod network. It doesn't matter how do you create these networks, where are these networks living? You just need to meet these requirements. Okay. So if we see the API, when you have a node, you are going to have an object like this. Okay. We have different fields. And we can see that in status, you have the IP. So what does this mean is that you are not telling Kubernetes, I want a node with this IP. You are saying, the Kubernetes is saying, I have this node and this node has this IP. It is reporting to us. Who is doing that is doing the QLED. That's some component that Kubernetes clusters have. Okay. We can see clearly how the IPs are reflected. There is another option that you can use or know that you can specify the pod network. But this is optional and it's an implementation detail that you don't need to use what this is flipping. Okay. Okay. So, okay. We have a node. We have pod network. Now let's create a pod. You can start creating containers, right? We say the container has to live in the same network space. But then these containers need to be able to talk. They need a network. How the network is created? Well, the network is created is an implementation detail. The way that this work is the user create a pod. Say Kubernetes, I want to create this pod. I have this application. I have these containers. Just please go ahead and create a pod. Just go to the API server. The API server does its magic and go to the scheduler. The scheduler say, okay, you have this node that is free. And the scheduler points the modified the object to bind it to a node. Then the node has a component that's called the QLED and say, okay, I have a pod. I need to start to create a pod. And then there is a protocol that QLED use with the container runtime that can be container D or cryo that says, I want to create a pod. That's what Kubernetes says. Kubernetes doesn't say, I want to create this pod with this IP in this network. Kubernetes say to the container runtime, I want to create a pod. When it finish, the pod has an IP. The container runtime tells the IP to Kubernetes. And Kubernetes says, okay, you have your pod and this is the IP. So another important thing is that Kubernetes is not declarative about the IPs of the pod. It's an implementation detail. You can assign the IP whenever you want. You can use CNI. It's a problem on the container runtime. But, okay, we have pods. Okay. That's so far. It's so good. We have the pods. What are we going to do with the pods? Pods are ephemeral. Pods are not living. You cannot rely on the IP of one pod. So like everything in software, we create another abstracting ledger that is the service. We need to discover it. We need something consistent. We need to run applications. We don't want to run pods. Okay. So we create the service. The service for network people is just we can model it as a ledger for load balancer, right? Services are complex. We are going to have different types, but we're going to focus on the services with IPs here, okay? So think of the service like a ledger for load balancer. It gives you an abstraction. It gives you DNS discovery. So you can say, okay. And the service has another property. It's able to forward all the traffic to a set of pods. So you can aggregate the pods and in the service and say, I want to send my traffic to, okay? We have the definition of the service, right? We have the service. And we can say, service, use this IP and expose this pod. And with the selector, say, choose this pod. So all the traffic that goes to the, in this case, the IP of the pod is 10, 96, 0, 13. What Kubernetes does is implements the API. Then there are different components that implements the service capability that is usually not, okay? But you don't have to do that. The only thing that you need to do is to obey the properties of the pod network. You say, every pod should be able to reach every other pod without that. And now that I'm saying is the net is in the pod to pod communication, not in the pod to service communication, okay? Don't get confused. So when the pod tries to read the service, the magic happens. You proceed whatever implementation that you have and say, okay, you can go to any of these pods because I know this service has these vacants. Once you choose one backend, the packet goes to the backend. And can you see the IPs? The original IP is the original IP of the pod and the destination IP is another pod. That means that we have guarantees because of the network model that each pod is going to be able to talk with another pod, okay? So the other pod will see the IP. We do the magic reply. And before sending the packet back, because the original pod was asking the service, we need to switch again the original IP to the pod IP, okay? So far, this is how service work. Intra cluster, okay? But we have another necessity. We need these things to be published. We don't develop just for pods. We need people. Kubernetes uses a lot of companies and you need to publish your application. So this is the tricky thing. And Kubernetes was born with IPv4. You have all IPv4 private addresses there because who has the money to spend all these pods and knows IPs using public IPs, okay? It's not easy. So we need to solve a typical network problem. We need to be able to spot a private IP that is inside a network with another private IP to the internet. How do we do that? There are different ways of solving this problem. The most common solution is to use a load balancer, okay? In this case, we have the service as a new type that is called load balancer type, right? And you can see, like in the previous example, you can define this is going to be my front end. These are going to be my back ends. And in this case, something happened. You see? You can see the status. There is some magic, some controller externally that say, okay, this service is going to have a public IP and this is going to be the public IP. You have a client, send this IP to the client because I'm going to make that your packet goes to my load balancer. And from the load balancer, I'm going to send you to the node. And from the node, I'm going to be able to send you to the pod, okay? So far, so good. We know, like, what are the networking problems that Kubernetes solved? And now we are going to understand better. We know that all these limitations are IPv4 limitations. So we are going to understand now better how can we improve this. So before actually getting into the specifics about IPv6 in Kubernetes, let me provide like a super brief introduction or overview about IPv6. I mean, IPv6 is rather complex, so we could be talking about IPv6 for quite a while. I will try to summarize in a couple, two or three slides, the most important aspects of IPv6 when it comes to Kubernetes. Okay? So first of all, in order to understand some of the features or the motivations for IPv6 or the idea behind IPv6, you have to understand the concept, the context in which it was developed. Many, many years ago, like over 20 years ago, it was like evident at the time that we would run out of IPv4 addresses. The Internet was a research project that was too successful, if you wish. So it was clear that we would run out of IPv4 addresses. So the question was like, okay, what do we do about it? So essentially, there were two different paths to actually solve the problem of IPv4 address exhaustion. Okay? One of them were like stop gaps, like short-term solutions, things to be able to extend the lifetime of IPv4. And then there was like, you know, more complicated, if you wish, longer-term solution. So what are the stop gaps? So this is SIDER and BLSM. Like essentially what we did was we got rid of, you know, classes in IP addresses. Even, you know, when still in a lot of places people still teach TCP IPs, if there were like classes in addresses, that has been long gone. Okay? With SIDER, BLSM, we got rid of classes. So we just have like, we are in a position to better use, you know, IP addresses, you know, oops, better performance when it comes to routing, and also be able to, you know, assign, you know, addresses in a more efficient way. That is, we were back one, I think. It was back a couple of slides. More, yeah, there. So that was the, you know, the stop gaps. And another feature or another, let's say, technology or mechanism that was like super useful, yet controversial, to extend the lifetime of IPv4 was the use of nuts. Okay? That is something that, you know, the IETF never liked, still doesn't like. But at the end of the day, ironically, one of the most successful technologies, right? So everyone is using, you know, not at home or even here anywhere. So those were the stop gaps, which, you know, they were super successful in extending the lifetime of IPv4. And we still have, you know, we still have like, you know, IPv4 for quite a long time, probably. So what was the longer-term solution? Well, the longer-term solution was to engineer a replacement protocol for IPv4. Just, you know, try to produce a protocol that, among other things, would provide like a larger address space. And this is IPv6. So main goal of IPv6, you will hear a lot of marketing claims about all of the things that IPv6 does, which is, for the most part, not true. The only thing that IPv6 does is provide, you know, more addresses, period. Okay? If you, you know, anybody that makes all their kind of claims, if you ask further, you'll realize that not true. It just provides more addresses. And, you know, an important aspect which, you know, has, you know, affected the, you know, the deployment, et cetera, et cetera, is that IPv6 is backwards incompatible with IPv4. So you might expect, okay, same protocol, different versions. So they should be able to interoperate. Well, not true. It's a completely different protocol. It could be named anything else, and it would be the same thing. Incompatible protocol, backwards incompatible protocol. So without a flag day, normally you'd have like a situation such as this one, okay? You are going to have like systems that only support IPv4, so they are IPv4 only. They will be part of the IPv4 Internet, so to speak. Systems that only support IPv6 will be part of the IPv6-only Internet, and you will see systems that support both protocols, the so-called dual stack systems. And of course, since they can speak both languages, both protocols, they can, you know, interoperate with, you know, or they can participate in the IPv4 Internet, so to speak, and the IPv6 Internet, if you wish. So that's one of the important aspects of IPv6. Now, the other important aspect is that I mentioned that, you know, the whole point about IPv6 is to provide a larger address space, okay? So that's probably the most important feature when it comes to IPv6. What can we say about IPv6 addresses? Well, they are 128-bit long addresses, so they are super long, maybe even too long, if you wish. But then when it comes to other aspects, they are similar to IPv4 addresses. For example, we will aggregate addresses for the purpose of routing, so you will see prefixes in the same way as you see for IPv4. We have multiple address types, same as in IPv4. There's unicast addresses, multicast addresses, et cetera. And we also have different address scopes. This is similar to IPv4. In IPv4, you have private address space, you have global address space, so this is similar in the IPv6 world, too. One thing that you might be surprised at when you first encounter IPv6 is that IPv6 hosts normally configure multiple addresses with different properties. So you will see the same interface that we will have, for example, a link local address, a global address, a unique local address, et cetera. So it's like the common case is for a network interface to configure multiple IPv6 addresses. And one final comment summarizing the whole thing is that the typical size for an IPv6 subnet is a slash 64. So 64 bits to actually configure like a host ID or an interface ID, more properly speaking. Now, there's another aspect that is important to highlight, which is the topic of unique local addresses. So essentially, unique local addresses provide private address space in IPv6. So you could say that it's the equivalent of the private address space from the IPv4 world, like the so-called RFC 18 addresses. There's probably one important difference in that when you configure ULA addresses, part of the prefix that you use for the addresses should be randomized. So let's say the specification says that when you configure a ULA prefix, you should randomize part of the prefix. And the goal of this is that if you were to use ULA space for one network in one company, for example, ULA space for another network in another company, and these two companies eventually were to merge, then you can avoid address overlaps, which in IPv4 we find all the time. They have one company using 10.0, blah, blah, another company using the same other space, and it's okay. But then one company acquires the other, and Houston, we have a problem there. So ULAs are supposed to help with this. Another thing to add is that ULAs are a controversial feature, controversial topic at the ITF. And the reason being, like the more hardcore IPv6 proponents say, well, we did this whole IPv6 thing to actually provide global addresses. So we are still proposing private addresses for IPv6. But the thing is the feature is there, and it's useful in many different scenarios. One thing to keep in mind is that the expected use case expected from the perspective of the ITF is that you use ULA space for internal communications, but the idea is that you don't use private address space in combination with NAT. And obviously the reason is that if we were to do private address space plus NAT, then why are we moving to IPv6 in the first place? Right? So it was here. Okay. There we go. So what's the deployment model for IPv4 and IPv6 if we compare the two? Well, if you think about the current architecture for IPv4, essentially you use private address space for your local network, and you connect your local network with the NAT to the public internet. Obviously, the thing is we don't have enough public IPv4 addresses, so we have to share them. But for IPv6, that's different because since we have enough public address space, the idea is that we can provide a public address, a global address, to each system that we have in our network. But normally what you want to do is not just simply connect your network with the router to the public internet because normally you don't want to expose all of your systems to the public internet. Even if quite a lot of people argue that you should do that, I'm 100% against that. Okay? So that's the deployment model for IPv6. Okay. So now let's, that's kind of like the overview of IPv6. So let's talk about how Kubernetes implements IPv6. So we talked before about Kubernetes and reference architecture, about APIs, about reference implementation, about implementation. So when Kubernetes defines the IPv4, you can see in the APIs, you can see the IPv4 address. So this is what Kubernetes does. And one important thing in Kubernetes, maybe some people disagree, is the users. So we want to provide a backwards compatibility. We don't want people to have, to change the thing. So you have one manifest and you apply the same manifest and then you got IPv6. That's how Kubernetes implemented IPv6. Transferring to the user, right? But then you have to say, well, but I have IPv4. I need to move to IPv6 and I need to deal with dual stack use cases, as Fernando said before. Okay. Don't worry. We got you. So we have another field. Instead of having singular fields, we add plural fields. And that's how Kubernetes reference architecture, the API, implemented IPv6 and dual stack. They just added the necessary fields so you can apply your manifest in one or the other environment. What is important here is what the implementation did. Yeah. So essentially this is like a dual stack, of course, a version of Kubernetes. You have IPv4 and IPv6 addresses. Now, if you want to think about how Kubernetes or what's a typical implementation for incorporating IPv6 supporting Kubernetes, essentially it's the architecture that we have in this slide. So essentially, we use private address space, ULA's, and connect whatever is using private address space with an app to whatever next network, like the Internet. So there are two things to think about this. What's the pro? What's the good thing about this? Well, it's that the IPv6 support mimics the IPv4 support. So at the end of the day, if you look at this, it's like, well, it's like IPv4 just with IPv6 addresses. So it's easier to understand what to expect, et cetera, et cetera. And that's the reason for which you will see this kind of implementation. Now, so far so good, this way of implementing IPv6 support is good. We have IPv6 supporting Kubernetes, but then this also begs the question, we are mimicking the IPv4 supporting IPv6. And the question is, are we missing the chance to actually leverage the addressing features of IPv6? The whole point of IPv6 is having more global address space. So if we just do private addresses plus not, we are not adding any kind of value to Kubernetes. So then what we did is try to ask ourselves, what could we do different here, of course, like instead of just implementing with IPv6 the same way that we do with IPv4, what could we do differently? And obviously, things that we could do is at least two things. One of them would be to provide global unicast addresses to services. And what could we do with that? Well, if the services themselves, they have global addresses, then we can get rid of the load balancer because the service itself is publicly available. You get rid of the load balancer, you save money in a way, and there's also like one, the architecture gets simpler, but we could go like even more hardcore. What if we were to give the pods themselves like a global address? So now we don't even need the service to have a global address because the pod themselves, like we have plenty of global address space, so we don't need to save addresses. So one thing that could be done is assign global addresses to the pods themselves. And if we could leverage the DNS to map a name to each of the pods that we want to connect to, then we could get rid of two layers of indirection. We could get rid of the load balancer, and we could get rid of the other layer of nuts. Of course, that's part of the story. You might wonder things like, okay, but if we give all the pods like global addresses, so who's going to protect the pods? What do we do about that? So obviously, there's other things to be analyzed and questions to be answered. So this would be the case of services with GUAs. Essentially, what we do is we get rid of the load balancer, okay? And we just can contact directly the service. We don't need the load balancer again. And then there's the other option, which is pods with GUAs. And what we could do is, essentially, we resolve the DNS name. We get, like, short live quad-day records. The idea, of course, you don't want to cache those records for a long time. And then we directly connect to the pods themselves. No load balancer, no service layer in there. Again, lots of details to be analyzed, like, for example, what do we do with the pods? How do we protect the pods if they have global addresses? And this is why it's to be continued, more analysis to be done. I think we have some couple of minutes for Q&A.