 We'll get started. This is a seamless multi-cloud Kubernetes, a practical guide. I am Justin Sanabarbara. I've been a contributor to Kubernetes basically since the start, and I work at Google. Hi, everyone. My name is Ciprian. I'm a software engineer at Microsoft. I'm a contributor for Kubernetes and various project link to it since a long time ago also. We're happy to have you here. Awesome. So one of the things that we're celebrating this year at KubeCon and the various KubeCons is 10 years of Kubernetes. And from the very start, Kubernetes set out to give application portability. And that means you should be able to take a working application that works on a Kubernetes cluster in Cloud A, and you should be able to take that same YAML basically and move it over and run it on Cloud B. And 10 years in, we've basically achieved that. So of course, we're still aiming to expand the set of applications that do work well in Kubernetes at all. And there are still some rough edges, but broadly it works. But there's an important detail in that. Like, what we're talking about is one application running on one cluster, and so one Cloud and one region. It was not really part of the original mandate of Kubernetes to run one application on multiple clusters or on multiple clouds. But what we are going to talk about today is how to effectively climb that pyramid of complexity so that you can get to real multi-cloud. We'll talk about why you might want to, and just as importantly, why you might not want to do this. I also have a confession. I, at least, I don't know how you feel, thought it was going to be easier to climb this mountain. I think we all in the Kubernetes project thought it would be easier. The idea was we'd start with one cluster, and we would get to multi-cluster, and everything would be joy and happiness and simplicity, and it's proven tougher. So we have recorded a few videos today to get through all this faster, but it's all live. If we have more time, we're in questions afterwards. We can sort of show you some more stuff. But as we go through this, I think bearing in mind the complexity, think about where that complexity line is for you. Like, at what point do you pull the ejection handle and say, you know what, this is just too complicated and I want to stop. For me, there's an obvious sweet spot where I think it makes a lot of sense, but everyone will have their own requirements and their own place to draw that line. Okay. Okay, so let's start at the beginning. Why would we even want to step outside of the happy place, a single cluster, that we've all spent 10 years optimizing? Well, the biggest restriction in Kubernetes is that it's really designed to run in a single region. In the early days, we were going to have just run in one zone, but if we're going the path of multi-region, we may have, well, tried to solve multi-cluster. But today, one cluster runs in one zone, one region. Many businesses need to run in multiple regions, maybe for compliance issues or because you want to tolerate regional outages or because your customers are spread around the world and latency sensitive. Another big reason is that you probably shouldn't be adding nodes indefinitely into a single cluster. The sweet spot depends on your workload, but there is a point where you're hitting problems simply because of your scale. And if you split your clusters in two, you just cross these problems off your list. And along the way, you may solve some compliance issue or some DR issues. You may have to do this for one reason, but still get the wins for all the other reasons. Subtle reason that we'll keep touching on is having two cluster means you can avoid a lot of shared fate. Put your hand up if you had a Kubernetes upgrade, break some of your workloads or cluster-wide component that went down and reduced functionality. So yeah, I guess, okay. If you change one cluster at a time, even if it breaks, you don't have all your pods in one cluster. So ideally it would survive. Of course, it's not all easy. Most of the tooling is based around the idea of clusters. In SIG multi-cluster, we are trying to address this to be more user-centric or application-centric, but you may need updated tooling that understand the idea of aggregating multiple cube API servers. For application developers, depending on how you structure it, the traffic is now going to cross clusters. And some apps will start misbehaving if that also translates to more latency or a higher error rate. Some of the convenience when running with in a single cluster like service account identity also start breaking down. And if you start running clusters across regions or even across zones, you'll also start to see higher traffic costs, quite a lot I would say. I would say looking at this list, these problems are not the bad, but if you really need multi-region, you have to pay for the multi-region traffic. But we'll show a pattern to minimize the inter-region traffic. Traffic slowdowns and errors can happen even in a single cluster. So it's not a bad thing to find these problems. And as a community, we're building that multi-cluster where tooling like cluster inventory in SIG multi-cluster. But that was multi-cluster. Let's quickly cover off why you may want to go to multi-cloud. Often it's accidental. You combine two companies or two teams and now you have two clouds. You may say no problem, we'll just pick one cloud but sometimes you just can't consolidate because of something keeping you in one cloud whether that's hardware or software or you may want to reduce cost both today and in the future when you negotiate your contracts. And finally, the clouds are not evenly distributed. For example, availability in regions like China, I don't know, India, but multi-cloud is not free. And these problems are even harder to solve. Take all the problems of multi-cluster and add some extra just for you. In terms of network traffic, multi-cloud looks a lot like multi-region even if the two clouds are in the same region. So you lose a lot of the fast paths, you definitely lose the cheap path and you should not assume that the network is secure anymore. You should assume that failures happen a lot more often and if you're using some of the convenient features like automatic authentication to cloud services, that is likely broken also. You're basically stuck with the lowest common denominator functionality. The big problem is that you're off the beaten path. There's not as many tools to help you with. There's not as many docs to help you and there's not one vendor that understands everything you're doing. But it's not all bad news. We can architect to minimize this. So let's start digging into how to make sense of all this. So there are a lot of challenges, but if there is an attractive payoff, if we can minimize those problems. So let's look at where we are. So as we started, we pointed out that together we have built a Kubernetes that works well for application consistency. But we've also, along the way, been building tools for consistent management of clusters. In fact, we've built lots of it. We've built Kops along with others. Cluster API, CubeSpray, there are some commercial services. So this is the first step, the tooling that understands the idea that you want to manage a cluster, that you want to manage multiple clusters, and that those multiple clusters might be in multiple regions or multiple clouds. So let's start with the cluster API. Like most of the tooling, like this tooling has sort of all followed the same pattern. There's a model. Cluster API uses CRDs. Everyone else is sort of converging on CRDs, including Kops. And that model describes both Kubernetes, sort of the common functionality across clouds, and a model that describes the infrastructure specifics. So those vary across clouds. But generally, there's a tool, in this case, Cluster CTL, that helps generate that model, like largely independent of the clouds. And so what you can see is, whether you type cluster, CTL generate cluster, infrastructure AWS or Azure, behind the scenes, there's a lot of details, but when you're working on, when you're working at the CLI level, the interface you get is consistent. And a lot of those details are abstracted away from you. They're not hidden, they're still there if and when you need them, and you probably will need them, but you don't have to sort of know everything about AWS or Azure or GCP right off the bat. And similarly, Kops is another OSS tool that Cyprian and I are two of the maintainers of. Again, look, it's very similar. There's a declarative model for infrastructure and for the Kubernetes installation. And again, you create clusters very simply using a CLI tool with very similar commands across the clouds. And sort of the details of the load balancer or whatever it is, is largely abstracted away from you. So if Kubernetes is solving for applications and application portability, and we've also started working on creating multiple clusters, what is the story for those application developers that want to deploy those applications to multiple clusters? In other words, are we done? And we are by no means done. The original effort here was the Kube Federation project or KubeFed. I think what KubeFed did really right was optimizing for that application developer experience. The idea was essentially from the application developer's perspective that multiple clusters would look and feel like one massive Kubernetes cluster that was sort of magically auto-scaling in all the regions. It was a very ambitious goal. And I think 10 years in, I don't think we're not all using KubeFed today. It has been slow going and it has had a few reboots. And in my opinion, that's because of two problems. The first is that architecturally, it's difficult to avoid introducing a new single point of failure. So KubeFed has sort of a, that control plane is now a potentially single point of failure. And that's something we'll deal with again and again. And the second is that KubeFed was very early. It was building a lot of this before CRDs existed, the core Kubernetes APIs. I'm not even sure deployment existed when KubeFed started. Those core APIs were still evolving. So that made everything harder. The efforts are still continuing here, but it certainly isn't ubiquitous. And I don't think we've reached a conclusion as a community that is necessarily the right and the perfect model. One of the things that has come along though, over the past few years, is a different approach, which is the idea of using GitOps to deploy applications. And so what we're really saying is that the new interface, GitOps is that new interface that replaces KubeCuttleApply. So you probably today already are using GitOps. We could do the show of hands. Who is using GitOps? There we go. Excellent, ubiquitous. So you probably have a GitOps deployment mechanism so that when developers push to Git, that deployment mechanism, whatever it might be, rolls it out to at least the development cluster automatically. And so the argument here is that extending that deployment tooling so that it can cover deployment to multiple clusters is both natural and will be mostly invisible to developers. So it fits the way developers are already working with the system. So it's a very natural point to introduce multi-cluster. It doesn't solve all the problems. It doesn't solve the observability problem. It doesn't solve some of the plumbing problems that we're gonna tackle next. But in many cases, if you aren't giving your application developers KubeCuttle access to prod, which you probably aren't, it's a great solution that fits with what you're doing today. Already changed? I'm unchanged. Okay, so if you have your application deployed to multiple clusters, let's talk how the application actually runs and talk to each other. The first step is to get traffic into the cluster. So when user visit the www.example.com, we want the traffic to go to one cluster. Ideally, we want an even distribution or geo-aware distribution. Okay, so we have three approaches. We have IP level. So anycast, DNS, and external load balancer CDNs like Cloudflare. Each one has its own complexities. I guess the most common ones are DNS and external load balancers. We were preparing a demo, but the video is not ready. So this is a very simple way of getting traffic into multiple clusters easily. You could use external DNS. Basically, you're declaring the same DNS name for each of the services in, for the same service in separate clusters. And you have to set a different identifier and everything is taken care of by external DNS. You get to have various ways of splitting traffic like weighted, latency-based, failover, geolocation-based. So it's pretty flexible if for almost any usual workload. That's not stateless. In this simple case, I chose weighted. You can see that when you're doing DNS lookups, you're getting IPs from both clusters. And well, one important aspect here is the TTL. Considering that you have multiple clusters, in case of one goes down, you would like your application to request very often to be able to switch to one that works. So I think this is a wonderful sweet spot. I was talking about where my personal line might be. I think you can probably guess. But in this, what we've got here is create multiple clusters. Every cluster runs all of your applications. You use DNS to ingress the traffic and basically spray it across multiple clusters. You get up to do the deployment. Apps only talk to apps within the cluster. So there's no cross-cluster traffic and you avoid a whole bunch of cost, a whole bunch of complexity and a whole bunch of problems by doing this. And so for me, this is a great place and perhaps a very natural place to either stop or start depending on how you want to think about it. But there are times or reasons when you can't stop here. So the most obvious one, which I'm sure you're all thinking about is when you have a database somewhere. You can't not trivially put a database into all your clusters and expect your East customers to just have a completely different world from your West customers. You have some singleton or stateful service and you want all your clusters to have the same view of that service. And so the cheat mode here is just not to put these into the cluster. These are going to be hard services to run in any case. And when you're talking about multi-cluster or multi-region or multi-cloud, they become very hard services to run. So this is something where it's certainly worth thinking about whether you should consume this as a service, whether that is from your cloud or another vendor. But keeping it out of the cluster often works out better from a TCO perspective. I think we see this even with a single cluster where a lot of people don't run their SQL database in their cluster. They find it easier to get it from their cloud vendor or service. There are shortcomings with that. You do have to be aware of latency. You probably have to deal with authentication in a way that was more complicated than if it was running in the cluster. You might have to distribute cloud credentials. If you can use OIDC, that's better. You may have noticed we're sort of pushing back or thinking about cheat modes here. And it's because we're trying to avoid this east-west traffic. It's a big can of worms where pods talk to pods in other clusters. If we can avoid that, it's worth considering. So stand up those parallel universes, parallel services, all the clusters are homogenous and they don't talk to each other. They're isolated. If we can consume services from outside the cluster, it helps us keep us in that happy case. And one of the cheat modes, I guess, if you feel like you still can't do that, is we can publish internal services using the DNS that Cyprian just showed for our own internal consumption, which sort of transforms internal services and what might be east-west traffic into north-south traffic, which we know how to deal with. But in general, when we're dealing with east-west, it's gonna be really easy to accidentally get a single point of failure. It's easy to mess up on security and it's gonna get that cross-cluster traffic is gonna get much more expensive. But sometimes we have to do hard things. There are plenty of use cases. I think I've given you enough caveats, so let's move on and look how it can be done. Let's have some fun, maybe a demo. One more slide before the demo. So there are three problems we have to overcome for east-west traffic, in my opinion. They are discovery, connectivity, and security. Service meshes like Istio tackle all three and give you, they solve all three at the same time. I think one of the reasons that they have this reputation of being complicated is because they are doing that. They are solving all three problems together and it's not really the Unix philosophy of composable tooling anymore. Another problem is that they're replacing all of Kubernetes networking, but the good news is that they do solve all three. They do work once you get them going. There are other efforts. Multi-cluster services in SIG Multi-cluster is tackling the discovery problem. IPv6 support basically works in most clouds today and can solve the connectivity problem and SPIFI continues to tackle the identity and the security problem. Each of those solves only one piece of the puzzle, but I think what's interesting is that together they solve exactly the three problems that we're trying to solve. So as we continue to develop these three pieces, east-west will hopefully get easier and those same solutions will also apply to single cluster, so we won't have this big disconnect when you suddenly wanna go multi-cluster where now I need a service mesh. You'll just be using them all the time. The video didn't show on my screen, that was terrifying. Okay, so here's the video. I will try to find where it is. There it is, I will get there. So, okay, we have two small clusters. The first is in US East and the second is in US West. I cutely named them East and West for East-West traffic. They are independent clusters broadly and the problem is how do we get that traffic going between East and West? So we have the Kubernetes guestbook example and you will notice that we're only running the Redis leader pod. Just pause here for a second, I think. We're running the Redis leader pod in West and so the problem is how do we get the East pods able to talk to the Redis leader when they want to? They have replicas, they both have the replica pod so for reading they will hopefully go to the replicas but they want to go to the leader for writing and that requires essentially track crossing the continent from East to West. In the West cluster where we are running the leader we do have a normal Kubernetes service and what we would like is we want that same service in the East cluster also so that we don't have to change our application so that guestbook will still talk to the, oh, sorry, guestbook will still talk to the service named Redis leader and so you can see this one, the top one is West and has Redis leader and we do also have a cluster service in East called Redis leader as well, it has the same name so it will still resolve the same and we don't have to change guestbook. What's fun is we actually have the same endpoints in them so let's look at the endpoints. For those who don't know the endpoints are the pod IPs that sit behind the services so when we look at the endpoints in West you can see them, they're IPv6 so I apologize, I will pause to give you a chance to read and I will call attention to the important thing. So if you look at guestbook for example there are three pods, those IPs are all different, they better be but if you look at the replicas there are two pods so they each have two IPs and they're different, if you look at the Redis leader there is only one pod, it is only across both clusters, it's only in West so the IP address of it is that first one, the one that ends in B4 C4 and if you look at East it has the exact same IP address so we have used Kubernetes services to solve the discovery problem, we are using IPv6 to solve the connectivity problem, one of the nice things about IPv6, if you were running in IPv4 these would have virtual IP address, they would likely have virtual IP addresses that weren't valid outside the cluster, some sort of overlay, because they're IPv6 we can give them real routable IP addresses, security comes in, don't worry but that solves the discovery and the connectivity problems and the way this is working is using the service import from SIG multi-cluster, so you can see there's a service import in that East cluster, the service import export API is only an implementation and implementations are pluggable so what we're actually doing here is we are syncing basically IP addresses or sets of IP addresses across the clusters, oh that is unfortunate, this is really gonna test my ability, let's see if we can do this, I can't remember what we said here. And so the other thing we're also doing at the same time is we have basic firewalling being driven by syncing IP addresses of the nodes so that's how we get some of the security, it's not enough but it tackles some of the problems like denial of service type attacks, how do I advance, okay, I think, and so that is East-West, if I'm the cursor, I think. Okay, over to you, you want to close this out? Okay, I'll do this, okay. So that's a pretty rapid run through but we have shown how you can run multiple clusters across multiple clouds fairly easily, now for me the natural stopping point came before we came to East-West traffic where it got a lot harder where there was a lot more sort of implementation work that had to be done but we've shown how to make East-West or one way to make East-West work, you can also use service mesh but we've shown a couple of strategies also to avoid even needing the East-West traffic and why you might want to. I think it's also just as important to think about what are the implications of this, where are we, what does that mean, how should we think about clusters? For me the big change is about thinking about clusters as commodities, clusters as cattle sometimes, so rather than updating clusters in place, let's think about a world where we are able to bring up a new cluster with that new configuration, we add it to the pool, we remove the oldest clusters when the new clusters are basically happily serving traffic. If you're on the cloud, you might as well use that supply elasticity but there are I think a ton of benefits even if you're on-prem and have to sort of really think carefully about capacity. We avoid a ton of risk and a ton of complexity, that is really the goal, that for me is why East-West is crossing that line but that's what we're trying to do. The big change I think, the other big change for me is thinking about what is a cluster, like once pods in one cluster start talking to pods in another cluster, a cluster isn't really a set of pods anymore. For me it's about thinking about a cluster as a particular version of the Kubernetes control plane, the cluster software, the location, you bring up resources, clusters where you want them and you update the versions of software in a managed and safe way using blue-green strategies. Okay, so let's summarize what we've learned. There's a great sweet spot for multi-cluster, every cluster runs everything, GitOps deploys everything, avoid East-West traffic, use ingress to bring traffic into multiple clusters and DNS is good enough. If you need to do East-West or stateful, it's possible. Service mesh is the most complete option today but it has a reputation for complexity. That's something that you cannot avoid in this space. In the Kubernetes project we're somewhat accidentally working on breaking up service mesh into more modular components. That will also apply to single clusters but those are not what we would consider seamless today. If you have to do it, please be mindful of the fact that it isn't seamless and come join the efforts to make sure that at least your use case is heard and taken into discussion. And finally, the next time you're thinking about upgrades of clusters or software on those clusters, think about whether you can mitigate a lot of the pain by embracing the seamless version of multi-cluster GitOps homogeneous clusters with ingress as the only shared point. Thank you. I think we've saved a couple of minutes for questions but I acknowledge that lunch is also now so I don't think anyone will be offended if you try to satisfy that hunger first but if you are hungry for answers, please either put your hand up and we can do a public question or we face government find us afterwards.