 Today, I'm going to talk about DualStack Kubernetes with a Direct Server Editor. This is a little bit background about me. I'm Kannan. I'm just working as a Senior Technical Lead at the Tata Communications Limiter. I have almost 13 plus years of experience in tech industry. Just I have provided my Twitter plus LinkedIn link. Let us talk about the actual speak. The agenda is like what is DSR, and how DSR data routing will happen, and what is the business use case for DSR, and how DualStack Kubernetes with the DSR setup looks like, and then I'll show the demo. Then the questioner talks. Basically, what is DSR? DSR is nothing but direct server return, which means whenever you are requesting any data to the server from a client, from our mobile application or anything, is our client. Then whenever we are trying to reach out to the server, first it will just reach out to the load balancer for that particular server, then it will reach to the target server, where we need to fetch the data. For the security purpose, basically what it is by default, they are just applying the source NAT, which means our source IP would be completely netted. I mean, it would be completely translated to the different IP address, and then it will send out to the application. Then when the application replies back, the same flow, I mean, whatever the flow the data comes from the application, from the client, the same data flow it will go back. That is the traditional way. But in case of DSR, the moment data received on server, so it will just reply back to the client directly. So in between, there might be a switch, which will just route the data towards the client which is requested. Basically, director server return is a method of load balancing that allows traffic returning from a load balancer server to be routed asymmetrically, skipping the load balancer and traveling back to clients through the server's default gateway. In a traditional way of routing, first of all, from the internet, just I mean, as a client, we are just requesting any data to the server. Then first it will reach out to the load balancer. Then load balancer will do the source netting. It will just hide the source IP, and then it will just send out to the particular server. Once we got the response back to the load balancer, then load balancer will detect, and then it will just route back to the corresponding client. Here, the problem is like, whatever the, I mean, from the client to server, whatever the data path it is sending, but the same data path it has to reached back. But this, we cannot do it for all the application. Say for example, for highly data streamed applications and all, we cannot send the same, I mean, we can send it via load balancer, we can request it via load balancer, but the video data cannot be transformed via load balancer, so which will heavily affect the load balancer. So that's the only reason we are going to DSR. But yeah, I mean, with respect to DSR, what will happen is like from the internet, first of all, the data, I mean, whatever the data we are requesting, so that request will reach to the load balancer, and then load balancer will send it to the corresponding web server, and then web server will directly reply back to the client through the switch. Basically, here we are avoiding the so many hops. You can see like, from the client, it has to reach load balancer and then from load balancer to the particular node corrects, and then from the node to the application. But here in this case, the application will directly reply back to that client, which were originated by, which originated the traffic. Yeah, so I mean, basically this is the DSR. So what is the business use case for this DSR? The number one, suppose if you are the client and you want to see how much server you are utilizing it. I mean, if you want to see it, we can see it through this DSR method. And you wanted to build based on the source IP. I mean, then we can go with DSR method. Then as a server, you wanted to allow particular, I mean, in a server, you wanted to allow particular source IP. Then also we can go ahead with DSR method because we never masqueraded the incoming source IP. So since we have complete privilege to see the source IP, we can just secure our application based on the client source IP. And then the latest, the trending one like video games, multimedia, content delivery network. Actually, these are all the things were completely based on the data server, direct server return. The reason is like, I mean, the moment you are requesting for a video, I mean, if you are requesting YouTube to play the video, that request would be just only one, just some one or two packets of IP packet, right? But the replay might be such a very huge data packet, correct? And whenever you are requesting, your request will go via the load balancer because your request is like, hey, I want this particular video. That's the only request you will be having it. But the video content would be such a huge data, correct? So that never come back with the same load balancer. So I mean, if we allow the data to be transferred via load balancer, I mean, it would be like, almost the load balancer will get overloaded, right? So just to avoid such kind of conditions, we are going with direct server return, okay? Basically the streaming services need a huge amount of video data from the server to the mobile client application. But actually, whenever you are requesting that request would be very, very, very less number of packet, but response would be huge number of packet, correct? So yeah, I mean, this is the only reason. And like the emails and all is one of the best use, best business use case for direct server return. I mean, basically the email, if you are attaching such a very huge content and if all the mails go via load balancer, the load balancer cannot even withstand it, right? So that's the whole idea for direct server return. So here, in our case, yes, we have brought up the dual stack Kubernetes with direct server return. Basically, this is our whole architecture. First of all, for dual stack Kubernetes, what is required? The pod IP should support both IPv4 as well as IPv6. I mean, both IP address, it should support, correct? And then, I mean, that is the main criteria. So now, with the pod retworking, we have supported both IPv4 as well as IPv6. So in the CNI, we would have clearly mentioned what is the pod IPv4 subnet IP and IPv6 subnet IP. So based on that, your pod will be allocated with both IPv4 as well as IPv6 for IPs, okay? And then, now, yes, we need to have a router, this physical router. Basically, this physical router should be having ECMP and PGP capability. And we will be having our Kubernetes cluster node, correct? And we might be running a pod on both the nodes. I mean, there might be a case like in a single deployment can have multiple replica and which might run across all the nodes. So I mean, this is one of the use case, like it is running on both the nodes. This is, we can consider like, you know, this is one of the deployment. Now, whenever the data is being requested, first it will reach to internet, then it will reach to the router. Now, router should send it to the corresponding Kubernetes cluster node. Okay, prior to that, there is something called Sirvis YAML, correct? I mean, you can see there is something called Sirvis YAML. So in the Sirvis YAML, there is a parameter called external IP. So the moment you have mentioned the IPv4 as well as IPv6 external IP, then your Kubernetes cluster node should advertise those external IP to the physical router, okay? Basically, the first step would be your Kubernetes cluster should have both IPv4 and IPv6 capability. I mean, your pod should have allocated with both IPv4 as well as IPv6 IP, number one. Number two, on your service YAML, there is a parameter called external IP. So where you can mention the public IP, public virtual IP. So that public virtual IP will be advertised from the Kubernetes node to the actual physical router. Okay, then your Kubernetes node will be having IPvS ADM. IPvS ADM is a Linux utility which is basically used to route the data in an effective way, okay? So the moment, okay. So now we have just allocated some of the IPv4 and IPv6 external IP to the pod. And the moment the traffic is originated from the client, first it will reach out to the internet, then it will reach out to the router. Now, since our Kubernetes cluster, Kubernetes node have advertised those IPs, now router knows which is the node it should forward. Okay, so and then it should have the ECMP capability, like who is the shortest path to reach out the, reach out the incoming packet to the Kubernetes cluster node. And then the router will forward that data to the shortest path Kubernetes cluster node, okay? Then once that data is reached with that, once the data is reached on the Kubernetes cluster node, then how it will be routing to the pod and how it will reply back. I mean, that's what we are going to see in depth actually in a further slide. Prior to that, how we achieved it. I mean, that's what I'm going to explain now. There is a open source Qbrouter which supports IPv4 with DSR. Yes, actually we have taken the Qbrouter open source IPv4 supported with DSR package. And then on top of it, we have, we have just enabled the dual stack external IP, okay? And yes, the first thing what we did is like for the pod IP, we were, we should be able to allocate both IPv4 as well as IPv6 IP. And then in an external IP, in the external IP, we were supporting both IPv4 as well as IPv6 IP, okay? I mean, so that both in, I mean, for a same application for both IPv4 and IPv6 can send and receive the traffic at the same time. Now, the moment you have mentioned this external IP, the Qbrouter will start advertise the external, I mean, will start advertise the IPv4 and IPv6 external IP to the gateway, I mean, to the router basically. So this router should have the ECMP and the PGP capability. So here the moment you have ordered the external IP, I mean, it will just to start advertising the IPs. Actually, how we are achieving this? See, there is something called the Mangle IP table, okay? So whenever the service has been, whenever the service has been applied, first we will create a Mangle table, okay? And in the Mangle table, there is a parameter called firewall mark, okay? Just we will set the firewall mark. And whenever the IP packet comes with a destination IP as a external IP, which is mentioned on this service, this Mangle, I mean, this Mangle table, just it will, it will just mark the incoming packet with some hexadecimal value. This hexadecimal value is a unique number, which is derived from both external IP and the port number. So that, I mean, so that we are all, we will always get the unique firewall mark, okay? And then once we have the firewall mark, we will add the entry to the IPvS ADM table, okay? Basically, I mean, in the demo, I will show you each and every IPvS ADM table contains its port IP, okay? And the moment the data comes with the external, I mean, destination IP as external IP, then through Mangle table, we are just marking the packet with the firewall mark. Then it will be routed to the IPvS ADM table. Once it is routed to the IPvS ADM table, the IPvS ADM table, we will be having a entry for this particular firewall mark. These are all the, I mean, following parts would be supported. I mean, such kind of entry would be there. I mean, I'm showing it in the demo. Like there, I mean, for in a deployment, there might be a multiple replica, correct? So there might be a chance that like single, I mean, for the entire deployment, we'll be having one IPv4 and one IPv6 external IP. So in that case, for IPv4 firewall mark, we'll be having the corresponding IPv4 port IP. And then IPv6 firewall mark, we'll be having the corresponding IPv6 port IP. Now, since we were creating the IPvS ADM table with the firewall mark mode, the incoming data never be masked code. Rather, on top of the incoming packet, the IPvS ADM will add another 20 bytes of data. The source IP would be the node IP and then the destination IP would be the port IP. And then it will just route the data. So that's how the packets, which is belongs to the external IP, when it reaches to the node, it is just reaching out to the pod. So on the pod, we were creating the IP IP tunneling so that the, I mean, once it reaches to the pod, it just removes the first 20 bytes of data. And then the further, I mean, with the further data, it will be able to detect that is belongs to that node and then it will send it to the application. Once the application is processed, it will just reply back to the reply back as a response directly to the client. That's how the direct server written with the DSR is working. Oh, just I'll show you the demo. Yeah. There is a, I mean, in my simple keyboard cluster, just I'm running only one, just one of the pod, which is a simple web server, okay? So I've seen inside the pod, there is, there are two IPs would be created. Two IPs were created for this pod. One for IPv4 and another one for IPv6. I mean, just we have enabled the dual stack functionality for the entire keyboard cluster. Also with respect to the CNI, just we have assigned both IPv4 as well as IPv6 address. So with this way, for a single pod, we are allocating both IPv4 as well as IPv6 IP address. So you can see, I mean, every deployment or every state full set will have its corresponding service correct. So likewise, yes, our application also having a service. So here you can see there are two IP address where mentioned. In fact, these two IP address are the public IPs, okay? One is for IPv4, another one is for IPv6. In fact, here you can give n number of IPv4 and IPv6 address, which means for a single deployment, you can have n number of public IPs in both IPv4 as well as IPv6. So now apply this. In fact, just now we have seen it. There is no tunnels were created only for a particular pod, we have both IPv4 as well as IPv6 IP. In fact, this is spingable and the, you can see that, okay? It is spingable within the Kubernetes cluster. You can see for both IPv4 and IPv6 address in what is responding back. So now we are creating the service with the external IP. So let me just apply this and see like the moment I have applied the external IP, you can see with the pod that there are two tunnels. See by default, the pod had one IPv4 address and then one IPv6 address. So now the moment we created service with external IP, it is creating, there are two tunnels it is creating. One with IPv6, another one with IPv4, okay? And this is IP IP tunnel, this is IPv6 to IPv6 tunnel. Okay, and yes, actually this particular tunnel has been created within the pod. Okay, let us understand what is creating on the node. And see here, see here, there is an entry for this, there is an entry on the manual table. So for this particular public IP, if any packets were received, just we are marking that packet with this ID. I mean, any TCP packet is coming, just we are marking that TCP packet with this ID. Then we are routing this packet to IPv8 ADM. Yes, actually we are setting MSS as well. In fact, this is just to avoid, in case we have set an empty U and we should, I mean, this is just auto-call with the entire MTU and then it will just start accordingly. I mean, that is the functionality, by default functionality of Kuber Auto, but yeah, we have extended with IPv6 also. So if you have seen it, so here with this, we are setting the, in case if this destination IP comes, we are just to set, we are just to marking that particular TCP packet with this ID. And when you see this IPv8 ADM command, you can see like, there are two firewall marks were created. I mean, the highlighted things were two firewall marks. So each firewall mark, the corresponding IP table were created so that when the packet comes, we are just marking it. So under it is routing to the IPv8 ADM. So basically the IPv8 ADM is a Linux utility, which just forward the packet to the destination. For our case, that destination is a pod IP, correct? So when the external, when the data with external IP comes, external destination IP comes, we are just a routing to the IPv8 ADM table by marking that packet. And then with the same firewall mark, we have the corresponding pod IP. So yeah, I mean, we have individual mark for the individual IPv4 and the pod combination. So here I have one IPv4 and one IPv6 external IP, I mean, public IP, so that's the reason that corresponding firewall mark has been calculated and then it's corresponding pod IP is being assigned, okay, on the IPv8 ADM table. So now, I mean, due to this IPv8 ADM table, the packet, the incoming packets were directly routed to the pod. So the pod might be situated, I mean, it might be placed on the same node or on the different node. If it is on the different node, the packet would be routed. So before routing it, what is happening is like, like, you know, it is just encapsulating the, I mean, it is just encapsulating the IP packet with this pod IP, and then it is simply it is routing it. So when it reaches to the distinct pod, basically this is the pod IP, right? So when it reaches to the distinct pod, okay, we have this, we have this tunnel, right? So we have the tunnel, since we have the tunnel and the pod IPs. So once it reaches here, it just removed the first IP header, so which belongs to the pod. Then when it see the IP address, so it is belongs to the external IP, it is belongs to the public IP, and that same public IP is being assigned in our pod, and then it will allow the application to consume the data. Okay, application to allow the consume the data with the preserved source IP. Actually, we just preserved the source IP by putting another IP packet on top of it. Basically, we have preserved it, right? So I mean, we have just a tunnel there. So when it reaches here, just we remove the first 20 bytes and then we have the further packets. So with this, we are, I mean, we are able to consume with our application. Then the application will directly reply back to the destination. The source IP will be the state of public IP. The destination will be the, destination will be the traffic which is originated by the traffic which originated by the user. So, I mean, with this, the packets were directly reaching towards the source IP. I mean, in between, there might be some default gateway to reach out the packet to the user. I mean, it never follow the same ideal way. Like, you know, first it did not follow to come through firewall rule and then the actual physical router, then from the physical router to the firewall rule, then load balancer, then Kubernetes node. Then when it reaches to the Kubernetes node, again, we might have some, we might keep some ingress. So through that, it will reach out to the part. But in this, but with this solution, with this direct dual stack with the direct server return solution, in a single deployment, we can have a number of public IPs which we can assign it on both IPv4 as well as IPv6. And then at the same time, the application can serve both the traffic. So basically, mostly the video streamings and all generally working with this kind of manner. Say for example, okay, now if I just call it, see it is reachable. And if I call with this external IP, again, it is reachable. Again, it will be reachable. So in fact, this pod would be reachable from outside my network also. Because I mean, we were advertising this IP towards, towards the router, correct? So that's how the complete dual stack with DSR is being established. So thank you, thank you so much.