 Hey, everyone. Thank you so much for coming to my talk today. My name is Abdul Basit, and I work as a product architect with Rakutan Symphony. I work with our customer mostly around Kubernetes networking, service mesh, multi-cluster. And today, I will be talking about one of the multi-cluster out there, which is based of EBPF. It's from a product called Cilium. So for those who are not much aware of Rakutan, Rakutan have multiple businesses. And one of the business is Rakutan Symphony. And in the Rakutan Symphony, we have multiple product we use. And one of the product BU is SimCloud. And I work for that specific BU. And we have three verticals. One is the cloud native platform, which is Kubernetes distribution optimized for high performance networking and storage use cases. And there is a special focus around zero touch deployment. We have a highest performance cloud native storage available as well that can run on any Kubernetes distribution or on any cloud. It doesn't have to be packaged together with our cloud native platform. And then we have SimCloud orchestrator, which is an orchestrator for bare metal and service automation provisioning across thousands of clusters, data centers, and servers. So yeah, let's go to the Cilium. So I believe most of you are aware of this Cilium, at least as a CNI. In a brief, Cilium is a CNCF graduated project. It offers features like CNI, load balancing, cluster mesh, security features, observability, service mesh. There are so many things. I wish I could talk more about that. But at the foundation, Cilium utilizes an EBPF, which there was an early mention of that as well in the Kipler, I believe the Kipler project, which enables somehow a Linux kernel to add enhanced features for security, visibility, and networking. So yeah, I'm just to show you the amount of features that Cilium have if I put them all together in one graph. And today we are just going to talk about this one small circuit, which is the multi-cluster networking feature of the Cilium. So in a nutshell, EBPF provide a mechanism to load a programmable hook into the Linux kernel, which makes the kernel dynamic to act on the events that happen in the kernel. For example, the program here is waiting for an exact call from the process, and then it returns the PID of that specific process line. And specifically, we saw earlier in the morning that EBPF is used to collect the utilization matrix from the servers as well, the energy utilization matrix. So the best example for EBPF is to compare it with the JavaScript, how JavaScript enable us to do or to interact with the browser, how we do the click, for example, and then it will invoke dynamically certain amount of scripts. And that's how today EBPF is powering the Linux kernel to bring many new enhanced features in the kernels that were not possible previously without doing a complete rebuild of the kernel itself. When I started working or when my first interaction with the Kubernetes when I was working in a company, usually when people start Kubernetes, I mean, I believe most of us start with a single cluster. But for us, even back then, there was a strict requirement that we have to start multi-cluster because the workload has to be physically separated from each other. And we end up something like that. Initially, I named this slide as ingress-based multi-cluster. But in my opinion, it was deceiving, so I changed it to ingress-based cross-cluster connect, give it a different name because it's not really a multi-cluster. But it is simple that how it works is that to do multi-cluster, it works the same way as today we use Kubernetes to expose our services out to the internet or the terminology that we use as a North-South traffic. There is no relationship between the cluster, completely independent, no service discovery. So if the source in cluster one want to talk to the destination in cluster two, source goes out, makes a service discovery outside the cluster, and then comes in to the second cluster via the ingress. I hope we are not using the same ingress for everything, so usually we will be having a private ingress for a cross-cluster as compared to the one that is exposed outside the internet, to the internet run. And this is usually the starting point that I see with most of our customer rights. So this is what sort of make them start thinking it's a time to look into something better, to do a much better multi-cluster topologies. And one of them was discussed earlier by Brian as well, with a very good context of what we can do with multi-cluster. So moving forward in today's talk, we'll be talking about a service mesh-based multi-cluster. So it is like creating a single logical clusters of multiple single clusters. And these clusters can be spread across multiple clouds, multiple data centers. So even if you look in the diagram, the services are global, and the endpoints are global as well. So for example, a source in cluster 1 wants to do the service discovery. It does the discovery in a cluster 1. And similarly, if a source in cluster 2 wants to do the service discovery for the app 1, which is a destination in this case, it does the service discovery within a cluster 2 as well. And the endpoints are global. And that makes us to do very, very interesting routing policies on it. Some of which I will try to explain in a bit as well. So it supports complex use cases. For example, federated identity, global routing, and multi-cloud deployment. So some of the use cases of the service mesh-based multi-cluster, we have global services across multiple cloud providers. And we can have a very good availability for those services. For example, a service can fail from one cloud to another cloud. We can have a better utilization in failovers. In fact, the better utilization use case is what I'm discussing with one of the customer at the moment. So today, the way they have done the architecture is that the whole app stack is deployed in one data center. And that is the active app. So they don't have the component or service level failover. So all the request is coming to one data center and being served from that data center. And the other data center is completely idle, doing nothing. We can do locality aware routing. That was discussed as well earlier in the brand talk ride. So when we have a global endpoint, what we can do is that we can do a smart routing ride. So if the request is originating from within a cluster one, we can have a policy that the request goes to the cluster one, to the endpoints in cluster one, or maybe cloud one. But if it is coming originating from a different cloud or different cluster, it goes to the being served from the other cluster. We can use security and global policies in a much better ways, because we know the identity of all the endpoints. So we can enforce policies. And also, we can have a much better observabilities based on that. So let's just take an example. For example, if I want to do the service mesh based multi-cluster, what do I need to do? This is, by the way, not specific to CDM cluster mesh. It is a generic that I have seen across any other service mesh base. Multi-cluster topologies as well. So at minimum, we need at least two requirements that has to be there, for example, the cross cluster service discovery and the network connectivity and load balancing ride. So if I have a source in cluster one and a destination in cluster two, if source talk to the local QBPI server, for example, and try to ask about the destination, it will not get any answer for that ride. So in that case, what we have is that we have an additional controller that sits in the cluster and it goes and talk to the QBPI of a second cluster, get the endpoints of that cluster, and then inform the source somehow. Let's not talk about the detail. That, hey, OK, the global endpoints of the second clusters are accessible. And once we have the service discovery, we need the networking part to connect the source and destination. And that's how then we are able to make the service multi-cluster work. These two requirements, at minimum, are good enough for some people. But I know for a majority of us, we need more than just cross cluster service discovery and network connectivity. So there are important add-ons that are required. For example, we need monitoring and observability across multiple clusters. We need encryption. In many cases, access control is mandatory. And we also need advanced traffic management for multi-cluster use cases. So one of the best things that Cilium cluster mesh offer is that if the previous slide, whatever requirement that I discussed, is all available within one product right. We don't really need multiple product to make it work. Cilium being a CNI, we know Kubernetes networking doesn't work without CNI. Anyway, CNI provide us the connectivity. And since Cilium cluster mesh is, Cilium is aware of multi-cluster with the cluster mesh feature. It is sort of aware of the routing across multiple clusters as well. The identity information is shared between multiple cluster. There is a service discovery and load balancing built in within a Cilium cluster mesh. And now Cilium is also adding up a lot of service mesh features as well within a product. So let's look at an architecture which is very specific to the Cilium. So if you look at a cluster one on the left and cluster two on the right, we can see that a few components. So there is an agent that runs in every node of the cluster. The job of the agent is to talk to the local Kubernetes API, discover the endpoints, network policies, and it also manages the EBPF maps on the node itself and also attach those EBPF maps to the specific sources in destination. In addition to that, we also have a cluster mesh API server. You can consider this as an agent, which is per cluster. And the job of this agent cluster mesh API server is that it is that middleman to share information from one cluster to another cluster. So for example, the agent in cluster one reads all the information, for example, deployment, identities, network policies of the cluster mesh of a second cluster. And since the cluster mesh in a second cluster is local in a second cluster, it can talk to the local API server and get all the information and then share it over to the first cluster. So because of all this information that is shared between that is available to the agent, we can see that it makes it possible for the service discovery and also network policies and the identity information that is shared that can be used to enforce. And similarly, the observability data is well-ride that is collected based on that identity sharing as well between multiple clusters. So to make cluster mesh work, there are specific requirements. So all the clusters have to be in the same data path mode. So CDM supports two types of data path mode. One is the native routed data path mode. Another one is the tunnel mode. So all the clusters either have to be in the tunnel mode or it has to be in the native mode. I believe those who are using the cloud-based deployment are most likely going to work with the native routed mode. All the nodes must be reachable. Podsiders in all cluster must be non-conflicting. That's why in the demo I'm using IPv6 in fact because I know the pain of getting dedicated podsiders for every cluster, especially if you also have to make it routed, it causes a lot of problems. I mean, not really a problem, but it's very, very difficult to achieve that, especially if you have a lot of clusters. So yeah, something that you guys can consider to opt in for IPv6. It works pretty well. So yeah, the firewall ports that are required for intercluster traffic has to be allowed. The cluster's names must be unique. There is a specific requirement for the native data path mode is that all ports must have a native connectivity between each other. So today if you're using VMs, it's pretty much similar to like that that if you look into your VPC, you should be able to see all pod IPs same as your VMs IP addresses as well. So in demo, I have two kind clusters. And there are IPv6 unique side arranges for cluster 1 and cluster 2. And for native routing, I'm using an open source FRR router. Consider this as maybe your VPC that provide you BGP capabilities to do the peering. So the reason of picking FRR is for me so that I can peer each cluster with the FRR to make the native routing work. And then I have a cluster mesh deployed in cluster 1 and cluster 2. And these cluster mesh API servers are exposed via Metal LB. And we will look into the first use case of doing the multi-cluster east-west traffic route. So for example, the sleep, I have a Hello World deployment in cluster 1, which is having a version V1. And a Hello World deployment in cluster 2 with a version V2. When I make a call from a sleep pod in cluster 1, it should be able to talk to this Hello World V1 and Hello World V2 in both clusters. Let me try to. So if I have Iliasis as K1 for cluster 1 and K2 for cluster 2, I get nodes, I believe. Oh, so much delay. So I have two clusters, and maybe we should have used. And then within, let's look into the cluster 1. We get kubesystem, everything done already in advance. You can see the cilium. You can see a cilium mesh API server. That is the cluster mesh API server. You can see the Hubble for observability. And then in the default names, we have V1 pod in cluster 1. And OK, so V2 in cluster 2. Sorry for the lag. This demo is I'm accessing over the VPN. So let's try to verify the east-west traffic. So I have a script for that. What I will do is just go to that script. It's in the app folder. And OK, so it is verify east-west OK. So we can see in the beginning some of the information. For example, just the node IP addresses for each cluster. And then to show you the pod-siders from within the local cluster that are known to every node in a cluster. So this is just from one of the node. And then it also knows about the pod-siders of the second cluster, cluster node. That's mandatory. And then we have a code from cluster 1. We can see that it can access the local service. And it does the load balancing between cluster 1 and cluster 2. So the important thing to remember here is that if I do, for example, the endpoint, we can see that hello world has only a local endpoint. So the cluster 1 by itself, QVPI of the local server is not aware of the second endpoint. It happens actually in the ABPF, right? So the request comes, and within the ABPF data path mode, this load balancing is happening. If I have time later, I'll try to show you that as well, to look into the CDM agent. And then we can get the information out about the endpoint that it is aware of. So that was the traffic flow between the cluster. I will also try to show you guys the traffic flow for the north-south traffic. So let's see what happens when the traffic is coming from the internet, right? So what we can see when the traffic is coming from the outside, in fact, let me see. I can show you guys the slides again. OK. For the north-south traffic, I have an STU ingress that I'm using for the north-south traffic. And expose the hello world service on the STU ingress. So the request comes and hit the STU ingress, and then it goes to the hello world, right? What I'm seeing here is that we can only see the version one, right? So we don't see the version two. And the reason it's happening is if I look into the STU endpoint, I can only see the local endpoint, right? So that's how it works. The STU uses a client-side load balancing. It does its own service discovery and asks the Kubernetes server for the local endpoint. And then it uses that to send the traffic. So the STU ingress is not aware. And it is a common issue. Even if you use IngenX, you will see more or less the same behavior. I will just show you the trick, how to make it work with the STU. I believe there are other mechanisms for other specific load balancers. So what I'm going to do is I will just try to tell STU to use the cluster IP instead of going directly straight to the endpoint. But I hope it works because the response is pretty slow. So I have deployed a service entry in a local cluster. And I call it a helloworld.global. And that service entry spawns to the cluster IP in a, I'm sorry, respond to the helloworld cluster IP service. And what I will do is I'll just go and edit the helloworld vS instead of going directly to the helloworld service. I will tell it to go to the helloworld global. And then let's try to verify again. Hopefully it works. No, it does work. So we can see the v2. And also this endpoint IP has changed. So OK, I'm going to get a helloworld p6. Anyway, so that's it from the demo. Thank you. If you have any questions, we still have three last minutes. Thanks.