 Thank you very much for having me today. So my name is Dan and today we're going to talk about a little bit of our Kubernetes I've got 20 minutes the original talk was 40 to an hour We have plenty of demos and and stuff. So I had to cram everything into into 20 minutes, which means Yeah, we I'm good. I have a demo hopefully it works But it's it's a much shorter version that what I usually run. Okay. Who am I? My name is Daniela very hard to pronounce so people call me down Daniela whatever and people make fun of me because I do quite a Lot of Kubernetes, so they shorten my name to D5e That's me. I spoke at KubeCon twice I'm a certified oops That's very quick. I'm not sure why what's going on. I'm a certified community administrator So I spent I spent quite a lot of time Just researching and playing with with Kubernetes. That's that's also part of my job. I work for a company called learn Kubernetes That's it. They basically teach Kubernetes and that's why I spend I spend a lot of time with it and today the talk of today is basically We have these Kubernetes clusters. How do we connect them together? If they are in different regions in different cloud providers Now before we tackle this this challenge, it's worth doing a very short recap of how Kubernetes works So generally we Kubernetes what we do as developers or just you know somewhat interesting in deploying We have like a single unit and we say hey Kubernetes, please deploy some deploy some workloads for me That's basically how it works. We don't really care. What's happening under the hood? I mean most of the time we don't but in reality what happens Is Kubernetes will basically figure out where these deployments needs to be placed inside your infrastructure? So we don't care but human is does because we still have to rely on service and we still have to rely on machines deploying these workloads Okay, oh good so far. What's next so? Kubernetes the way the way Decides how to place these workloads is decided by what we call the control plane and control plane is this basically a collection of Different controllers that work in tandem that work Work together and most of the time that I'm for big blocks. I mean that I'm way more than four but we're interested in four of them which are the API server which ingests your requests and then we've got a database called at CD Then we've got a controller manager, which is brain of the operation and then we've got the scheduler Which basically assigns workloads to nodes. That's that's basic So what happens when you deploy an application in Kubernetes? Well that that request you say I want to deploy something that request goes inside the API server goes through several steps of authorization authentication and just changing also The request on the fly and then eventually it gets stored in in database in at CD Once is inside at CD then it's picked up by what we call the controller manager the controller manager is one of these systems It basically just syncs the state So if you ask for something it's going to check that you got what you asked for in this case Let's ask for the deployment and the manager notices that there are four miss three missing pods It's going to create them. Oops. They're very quick. I'm not sure what's going on So these parts are created and creating in pending inside the database Those are you're reaching down to disk and then since they're pending the scheduler It's notified of the change and it will pick them up and then try to allocate those workloads to particular nodes that's basically the end of Of the scheduling of port is worthless What's important to notice that at this point in time all you've done is basically just changing values inside the database There is nothing actually going on inside your infrastructure Right, so who creates this workflow the inside inside your servers? Well, we need something which is going to do the work and that's something is called the cubelet So the cubelet is basically like an agent that lays inside your nodes and they basically what it does. It just pulls down all sort of information about the node right and tries to reconcile the state of the node with the control plane and If there is a workload, then it's going to create it All right, or if there is a request to delete that workload that is also also going to delete it Whoa Okay, so we got we got this cube we got we got a cubelet. We know how the control plane works another things that it's important to mention oops is Is that the traffic does not flow Through the control plane when you deploy this application these applications are serving web traffic like production traffic Then this is not the kind of scenario. We are looking at I'm not sure what's going on. So I'm gonna stick to The seriousness of my clicker. Yeah, sure sure. That's a good idea so what's going on is is that The traffic doesn't doesn't flow like this instead of what we see is Most of the time the traffic flows directly through the nodes and these nodes are fronted by a load balancer And then what basically means is that this this control plane could go down and can still serve traffic, right? Which is quite quite convenient for us, right? And this nobody know the basic let's let's have a look at how the basics apply when we when we scale this cluster from One single region. So one single cluster to multiple clusters, right? So the first idea is okay I've got I've got an application I've got an e-commerce website or a financial application that needs to be deployed into different regions for availability, for example So one, you know, the first idea you might have is okay I'm gonna have a single control plane and then several nodes spread across regions That's perfectly fine. There is You know all the all the traffic will of course will be ingested by the nodes, which is great But reality is that some of the communication that you have between the control plane and the rest of the nodes will be slow Right because of course getting different regions You might say okay, and that's you know, I can I can bear the cost I can bear the trade-offs Let's do this Then the same sort of problem you will sort of face when you deploy your applications, right? Because Kubernetes trees those nodes as equals, right? So it's gonna deploy you're gonna deploy these microservices and then they start talking to each other, right? So you might have cases where some of the traffic which is distributed inside the cluster is a little bit slower Because cuban is essentially doesn't doesn't know that this traffic is going to be the redirector somewhere else Now there are changes in the community Kubernetes API to make it that a little bit better But but the default this is what you get which is you know, sometimes everything works fine and then suddenly everything is super slow Okay, so how do we fix this? Well, what if Right, we use one of these cuban 80s features to fix it So instead of having a single control plane, right? Kubernetes can be designed can be deployed to have multiple control planes So the idea is if I can have multiple control planes and each control plane has got a database and all these database synchronized between each other Who is stopping me from deploying? Multiple control planes and then eventually one for each region that I'm that I'm that I'm using in my infrastructure No one is stopping me. I can do that. Does it solve the problem that we have before well sort of yes, because These nodes they can talk directly to to to the control plane in the same region. So that sort of communication should be solved However, this introduces another issue, which is which is replication, right? So we talked about how this database needs to be replicated between between instances and The issue is that the database will write to disk only three squarer, right? So if the majority of the nodes agree on a value in this particular case It will take ages to a green value because they are in different region and latest is very high So this design is even worse than what we had before So we know we can't we can't really do a single cluster with multiple with multiple nodes in multiple regions We know it's very tricky to do a single a single cluster with multiple control planes and multiple nodes in multiple regions So what can we do? Well, the reality is that we can have a single cluster per region and that generally the better approach That that you might take when designing this sort of stuff However, when what we do this there are a few issues the first issue is how do we decide? How to place these workloads in which cluster we're going to place these workloads? This is issue number one second issue is if I route the traffic to the u.s Cluster or the u cluster of the South Asia Pacific Pacific cluster, and there is no app or it's overloaded How do I route the traffic to the other cluster? All right And third one is if I've got storage in one cluster and that application needs to move Do I need to move the storage on the other cluster as well? I don't know so what I played with this sort of Examples then that I found a fun a quite interesting tool called Kermada and basically the way it works is it's an orchestrator of clusters and you install a Cluster you install this the software in one cluster which becomes the manager and then the rest of your cluster will become workers And the way Kermada works is basically we just send the request to the manager and the manager will basically Distribute these workloads across across the entire fleet and why was it is it good? It is quite good I think it's quite clever in a way it basically creates a mirror of your control plane But this control plane is basically cluster aware so it's aware that there are several other cluster that needs to be orchestrated and and What happens is you have an agent which is similar to the cuba in a way and then it's the agent Which is basically going to send the commands to the individual cluster So you have a single cluster which is basically sends all of these commands to the agent and the agent will send the commands to the cluster themselves So that's basically how it works, but the interesting things is this Kermada, which is basically a glorified control plane for clusters can have something called policies So you can basically say I want to create a deployment and I want to go to just Deployment us cluster. So if you do that and you submit the policy will basically have a deployment just in the u.s But go back to my policy and I change it to the yield so Europe then this part will be moved to Europe, right and There are also many other ways to Describe how you want the state of the cluster to be in this particular case I say I want to deploy two pods and if I say Duplicated then Kermada will deploy two for each cluster who got in my fleet Or I could say something like Divided, you know, I could I could put different weights or I can say aggregated So filling one cluster first and then move on to the next one and Then there are way more ways to configure however you want this to be to look like This is a good and when you deploy Kamada, the networking for Kamada is isolated to this to the cluster itself Which basically means that if the traffic is rooted inside Surface your precipice or in Singapore then it stays in Singapore. There is no way to share this traffic with the rest of the clusters So what do we do? I Want I want to be able to share this data. I want to be able to share this traffic with the rest of the fleet I can do that if if somehow I find a way to share this IP address and this information about workloads that have been deployed in my infrastructure and And the way we do this is basically by intercepting the traffic Proxying it to the other cluster and then rewriting it when we are on the other side And to do that we use something called service matches. So one one way to solve it It's not the only way but one way to solve it is to use something like service matches So you might have this name So Istio, which is what I use in the tool as well. So what are these service matches and how they work? So essentially you have Services you have deployments inside your cluster and then they all talk to each other So when you install the service match what happens is Each of these applications will be fronted by a proxy. So all incoming and outgoing traffic will be filtered by the proxy But this proxy is not just like a regular NGDX proxy. It's actually a programmable proxy That can be reconfigured on the fly. So the way it works is we have got another control plane Which is the Istio control plane which on the fly will basically reconfigure the routes and the end points For this for these proxies which gives us quite quite a lot of flexibility for saying how I change my mind I don't want these two services talking to each other anymore Or I could say something like hey when you do the log balancing Can you please do a different split than 50-50 or run-robbing right? All these information is something we ask we send the control plane the control plane will reconfigure this proxies on the fly So that's that's how it works What can we do with this? Well if we can reconfigure on the fly we could also say hey this traffic We also have multiple pods in multiple clusters. So route this traffic somewhere else and That somewhere else is is basically The service match so this is basically how the service match multi cluster works. So we've got cluster one and cluster two and And what when we store Istio you see the proxy there the programmable proxy becomes part of your application is a single unit and Then when you install what we call the gateway Which is basically a way to connect this to serve these two clusters together and then if you will basically start Sharing and discovering endpoints from the other clusters, right? So this is basically where we share. This is my workload. This is yours, you know, and we keep this information In in the cluster now when the traffic comes in and it goes to the proxy Then we have the ability to say actually, you know what I will send this through the gateway to the other side And that's basically how we share We share the traffic So what does it look like? So we've got a deployment We know we can spread this pod across the cluster We know we have policy we can decide how to how to spread it in in our fleet and now we have These service mesh which is also able to decide where the traffic should be placed now We've done we've done something basic. We just basically Install the service machine connect to the cluster, but we also do clever things such as always send a traffic the local cluster And if you're overloaded go somewhere else, right? So in this particular case, it's just just a basic configuration But at the end we are basically able to ingest traffic in one cluster and then if we want to forward it to the others Do you believe me? Someone say no No Anyone else? Okay, I've got four minutes to make it work Let's see, okay Okay, so what I've got Hopefully it works. So what I have is a small small script and what it does is basically just Send the request of the cluster and sees and basically check the response now hopefully the Wi-Fi works. I have no clue What's going on? Or how slow it is? Maybe maybe we'll wait a little bit longer, okay And so this is a dashboard that I built so this is I deployed three clusters today and before I came this is why I was late Just deploy the cluster, but what this dashboard is is basically just pulling the data from the free cluster I've done and then there What happens is I've got here Some some radio buttons at the top where I can select where the traffic is going So I make a request to the London cluster or I make a request to the US cluster And then I basically have a small application that replies with Let me see if the script works. No, of course not that replies with a flag with with the region where the cluster is deployed So this you can see here that when I send the traffic to the US takes ages But most of the time it goes to the US and now it's going to London All right, so I send the request to the cluster in the US It travels all the way to London and then comes back All right, and if I switch to to Singapore Then it goes to Singapore and then it goes back to London and then it comes back So basically this is Istio deciding there's some of the endpoints that we've got inside a cluster needs to be rooted elsewhere Before they go back as a response Here we go Okay, got it so two minutes. Let me wrap up So I think so what we what we discussed today, so We had a look at how to keep an egg's work. So that was important So so we don't understand how scheduling works in Kamada. So how do we? Manage to have policies that can spread cluster can spread where it goes across We had a look at how you know different options that we have to deploy cubanates So we could have a single control plane multiple nodes in multiple regions But we sort of discover that the better way to do it sort of easier way to do it Is to have a single cluster per region We had a look at Kamada. So these two. I mean, it's not the only tool that does this You so some of you might have used I go CD To do deployments to the multiple classes perfectly valid option. I was just interested in and this one for for my my research We sort of have a look what how Istio shares IP addresses between clusters and And and how this traffic is forwarded in in the network using using gateways We also look at them when he actually worked this time, which is excellent news. So I just Everyone for for joining me today This is me. I Don't know how much time I've got Maybe 30 seconds for questions One is link ID, which is the second most popular service match and then the third one is called kuma from from the Kong Kong guys Yeah, I think yeah, those are those are the most popular I guess but there are many more options So the question is why do I need to multiple multiple control planes? So the reason why you need multiple control planes is because cubanates Stores what we called endpoints so endpoint is basically the IP address of the pod and that IP address of the pod is used by Several several components in the cluster. So the components are core DNS So the DNS system the ingress and what they do is that we basically make an API request to the control plane and say Can I have back a list of nodes, please with this list of list of IP addresses, please with this list of IP addresses? They basically reconfigure on the fly the DNS the ingress and several other components to make the cluster work So when when you grab a control plane and you shut it down if you don't do anything at all Your cluster will work But as soon as you have the new workload then first of all, I don't know how you're gonna schedule that workload But assuming that one of your workloads goes down and that IP address is not available anymore There is no way to update the control plane saying hey this IP address is gone So your DNS your ingress will still have a stale list of IP addresses It will still route traffic to those even if you don't want to now. This is not great This is but your cluster still works. So you don't have a hundred percent on time. You will have Depending on what's going on. You will have the graded service But but it's not like fully percent, you know a hundred percent down. So yes We won't have we want to have multiple control planes because we want to make sure that these endpoints are propagated correctly But again, it's it's trade-offs It goes back to how at CD works and how replication and databases work So the more the more a control plane you ask the slower it gets And the last consequences to that Oh, this is a last talk before lunch Yeah, okay, yeah, I think Google answers is a collection collection of tools come out It's just like open source tool focused on on scheduling and you know multiple classes. I think I'm not I'm not an expert in answers by anyway But I think answers is is a color is way More unified collection of tools that help you to deploy Kubernetes on on-prem or in the cloud. So I think you know, you can have Istio in answers. That's for sure I don't know how you would do multi-cluster. I'm pretty sure Google has got like a Lot balancer that can distribute traffic across multiple clusters. That says for sure So I think they are quite different isn't you know different different sort of Aim for for the project Cool any other question I Guess the question is where is lunch? It's basically just a collection of tools like even in the same way you would install Prometheus So you stay the same way you would install Istio Then you install this in one cluster and that cluster become becomes cluster manager will basically become like This this entity that can manage other clusters and then it's a little bit weird because then your Qube CTL Then the same cluster will have two two Qube CTLs like two Qube companies one for the normal cluster Then if you do Qube CTL get pods you will see Kermada and then there is the Kermada Qube CTL Same Qube CTL different Qube config and if you do Qube CTL get deployments They show you that it shows you the deployment across clusters, which is like a control plane for The entire fleet you've got and and basically Kermada what it does it basically aggregates all of it all of the Qube config from all of the other Cluster, which is quite cool. Are you see a unified view of whatever is going on in your fleet? Yeah, it's it's on top. It's turtles all the way down. Oh, yeah All the code for the demo is open source and it's all line So I can share I can share the the Github repo not not an issue at all Yeah, but this this sort of stuff breaks quite easily, so I spent this morning As you can imagine, there are so many moving moving parts I was like, okay, can I do it? No, maybe and then eventually I figure out how to do I Don't I don't think that's that's the case because the the service match will basically do that work for you So I think there are like several ways you can connect multiple clusters together And then some of those will be like similar to what we do in in AWS like VPC peering where you need to be Careful about how you assign IP addresses like an example of that. I think I'm not I never use it But I think it's called it's a product called seal use selium service match So this this will be like a lower level. So we're talking about probably L2 connecting clusters, right and then if you say so the very top right most of the time layers seven So so in that in that particular case, we don't need any any precaution when it comes to networking but There are consequences on running a lot of proxies in your cluster when it comes to Yeah, I'm managing them resources. It is not three. It is not free for sure But but it is it is an option if you if you've got these kind of problems cool No one is asking where lunch is Excellent. Cool. Thank you. Thank you very much