 Hello. This is the SIG multi-cluster intro and deep dive. I'm Paul Mori. I'm Jeremy Olmset-Thompson and I work on GKE at Google. And I'm Laura Lorenz. I also work on GKE at Google. So we're going to cover a lot today. We're going to cover what is this SIG about. We're going to talk about areas of current activity like cluster set, name space, same as cluster ID and cluster set membership. The multi-cluster services API and multi-cluster DNS. And more. We're also going to have a deep-ish dive that covers multi-cluster services API, cluster ID and multi-cluster DNS. And finally, we'll talk about how you can contribute and be involved with our SIG. So let me share first what this SIG is all about. The whole idea is we're asking ourselves what should be the Kubernetes native way to do things like expose workloads from multiple clusters to each other, or replicate workloads across clusters or target deployments to specific clusters if you have multiple clusters in a group. So these pressing questions and more are the purview of SIG multi-cluster. And it turns out that the work that we do touches many different functional areas, all sorts of different problems. And we also find ourselves collaborating with many other SIGs. But where we're at right now is working to identify the best, most durable primitives that are going to help us achieve all of these multi-cluster use cases. And we love input and really need your input. Where it comes to multi-cluster setups, real user stories and use cases are extremely valuable, you know, designing any standard that is extremely valuable. And many of the projects that we have are in AlphaStage and are still malleable. And we need new tools to expose new needs. So we'll talk about it more when we get to how you can contribute, but we would love to see everybody and hear your multi-cluster use cases for the projects we're working on. So let's talk a little bit about our approach. This is really important to us as a SIG, I think we've seen great progress because of this kind of shift in mindset. But we try to avoid premature standardization or solving any optional problems. I think in the past we've maybe tried to reach too far into what could be for multi-cluster instead of focusing on problems that we knew existed and specific problems that we're seeing in the community that need to be solved now. So that's basically the changes. We're trying to instead focus on the specific functionality that we want to build, pay less attention to what could be but making sure that we're not closing any doors, and then kind of work backwards from those problems and see how things evolve into bigger and bigger problems. So let's talk about the current activity right now in the SIG. So the first thing I want to talk about is the cluster set. And you may have heard us talk about this before. This is your first time coming to SIG multi-cluster. Let's dig in here. Cluster set is not an API. It's a concept. We don't really expose it as any resource, but that's changing. We're starting to have some resources that really kind of revolve around the cluster set. But the cluster set basically represents a pattern of use from the field. So historically in Kubernetes, resources within a cluster are not aware of the cluster itself. There's no real concept of a cluster from within the resource model. We're changing that. And so a group of clusters in a cluster set are clusters that are governed by a single authority and have a high degree of trust within a set. If you wouldn't want things in these clusters to talk to each other at all or even be aware of their existence, it's probably best to keep those clusters separate. This is if you want to have some coordination between your clusters and communication between your clusters. And if you're okay with kind of exposing what goes into the services and whatnot that make up the workloads that run within each cluster, then cluster sets are probably very interesting to you. So with cluster set, we introduce this concept of namespace sameness. And this is that permissions characteristics and identity are consistent across clusters in a cluster set for each namespace. So foo in the bar namespace in in one cluster is the same general workload or service in another cluster. Namespaces don't have to exist in every cluster, but for clusters in which they do exist they behave the same so that maybe that you have a service that exists in multiple clusters in the same namespace the same name as an extension of that service. Or it may be that you have a namespace in each cluster that is always cluster local and and never shares any information outside of that cluster and that's okay too but the idea is that you don't have this kind of mix and match so it's very predictable. You know how namespaces are used, and generally someone with access to a namespace in one cluster should have similar access to that namespace in another cluster. So with the cluster idea this is kind of the first time we're trying to expose that that cluster concept as a resource. So really what we've done is we've introduced this new cluster property CRD, which is basically a name value pair that the idea is that this is a, a resource that you can use to expose cluster level properties. And the big one that we're pushing to start with is ID dot kates.io so this will be a well known named cluster property that has the, the discoverable identity of that cluster. So services within that cluster workloads in that cluster can can use this resource to figure out, you know, where am I running. We're also introducing the cluster set dot kates dot io cluster property which will be the identity of the cluster set, or at least the way for a cluster to identify the cluster set to which it belongs so this may be a membership ID for that cluster within the cluster set and maybe the name of the cluster set itself, you know, going back to not solving problems we don't need to solve we're kind of leaving that up to implementations. But the idea is that now a workload will be able to discover which cluster is running in and, and which cluster set to which it belongs. These need to be unique, unique to cluster within a cluster set for the lifetime of that membership so basically as long as you have a cluster belong to a cluster set, it will have a consistent cluster set ID and it will have a consistent cluster ID. So you'll always be able to identify your, your clusters in that cluster set. And, you know, these are really to build that reference for multi cluster tooling that we're going to talk about here in a second. So the first big project that we've been working on for a while and some of you may have heard us talk about before is, is the multi cluster services API. And this is basically an extension of the service concept across multiple clusters I mean services are the basic way that workloads communicate with each other and in Kubernetes, and we've extended that across clusters. And so we've just kind of been focusing on just the API in common behaviors that we want to see in services so we've, we've left room for various implementations and we have some out there we've got Submariner, GKE has one and STO has been developing one as well. We have not built a standard implementation, but we've built an API that we, we want you to be able to rely on in no matter which implementation you're using. And this builds on the concept of namespace same as so basically Laura's going to dig into this a little bit more later, but with the multi cluster services API, we let you stitch services together across clusters simply by using the same name. We've left room for control planes to be centralized or decentralized depending on your implementation where things are running what makes sense. But cluster IP and headless services just work the way they do across cluster the way they do in a single cluster now across clusters. And there are some DNS changes as well involved with this that Laura will talk about as well but again all this is additive your existing services. And we've been working with other SIGs to figure out how this can integrate as well. There's been some work with SIG networking in the gateway API, which plays night nicely with multi cluster services as well so this is really becoming kind of your standard service building block for multi cluster communication. Now, what everyone's been waiting for the more. Alright, here's the more we got cube fed which is preparing for beta. We've got work API which is about spreading groups of resources to different clusters, and also recently we've had some conversations about SIG multi cluster in the context of election in the context of services that are running across multiple clusters, considering how this, you know, might might interact with existing work to enhance leader election and wondering, you know, what SIG multi cluster should or could recommend or implement as a reference in this area. And I'll just sneak a little call for you to join us if you're interested in that topic as well. So I wanted to take an opportunity to do a little bit of a deep dive though honestly might feel a little shallow in the sense that I only have so much time. But I've pulled some slides from our past conversations in SIG multi cluster about MCS API about cluster property and about multi cluster DNS I'm going to try and show some of those to you. So you can get even more of a feel for them beyond the updates that we just gave. So about the MCS API we talked about it Jeremy talked about it a bit before and he mentioned that the idea is that we're broadcasting a service out into other clusters so of course we've got two boxes here that are two clusters right. So, say you have a service over here in this cluster on the left. If you wanted to get accessibility of that service available over in this cluster on the right. The idea is that this service export object that the MCS API describes will be placed in this left cluster this service import resource that the MCS API describes will be generated in this right cluster. And that's going to connect these two core Kubernetes API objects the service that you've had in your producing cluster and endpoints or endpoints slices you have in your consuming cluster. So the way that the way these actually look right from the service export perspective this is the resource that you as a administrator or needing to configure your service to be shared across clusters would actually create either yourself or through some sort of tooling. And so this is a the, you know, direct point that you'll experience as a user of multi cluster services. And it's a very simple resource that basically just has a name and a namespace and the whole point of this is that this name and namespace matches exactly with what this services name and namespace is. And that marks that this service should be exported from this cluster. So very simple shape here right schema here, just to connect these two points. If that has been marked, then it's up to the MCS controller to generate the service imports in the consuming clusters. So the service import schema here is designed to include all the information that's needed for the consuming clusters actually consume this as if it was a service that was present there. So it needs to know some information about from the originating service, like what ports it's exposing and other metadata from there. And then it also needs to know how to get to that service from our current location right from our right cluster we need to get to. We need to get to our left cluster. So it's also going to need to know what the IP is to get there and in the case of, if I was exporting a cluster IP service then this would be our cluster set IP here. So similar to how cluster IP services work today, you get one IP that you can load balance in front of all the back ends from other clusters. So just backing up two slides just to touch one more time on something that Jeremy mentioned, I'm showing the left to right direction here. So you could also have the service in the right cluster, create a service export in the right cluster, and then those back ends will also get shared across our cluster set hero to clusters into the left cluster. If you had like a north cluster and a south cluster on this diagram to they could also all have the same service, all have a service export marking it for export and then this name and namespace service the back ends and all of these clusters are all considered safe to respond to any request for that service with any of the back ends and any of the clusters because of namespace sameness because we consider those to be functionally identical. So just one other add on from what Jeremy mentioned earlier. So another exciting thing that's going on is the MCS API is being integrated into other projects in the Kubernetes ecosystem. So Istio is actually taking on a multi phase approach of links to RFC here if you want to give it a read to integrate the MCS API, and this includes in sort of the later phases to implement a full MCS controller in Istio D. So you could use Istio to run your the controller that understands the MCS API standard and standardizing on this MCS API. It benefits the MCS users because you could upgrade into Istio service mesh just naturally if you already were using a MCS API controller that you had an implementation for or you used one of the implementations that exists but then someday you're like oh actually I want to use this other thing that's an Istio. It could be easy. It is easy for you to upgrade directly into Istio in that case because you're already using an API that it's built on top of. But it's actually also helping the Istio users to so it simplifies their DNS infrastructure the way that their Istio DNS infrastructure today is has kind of a couple different options or optional things that can run alongside. But if it can use what is described in the MCS API it kind of simplifies it and unifies it onto this one standard that's already Kubernetes native. And it also allows some flexibility between full service mesh everything knows everything about everywhere type situation and a more cluster local type service discovery. So the service export concept that we're using in the MCS API for people to opt in to exposing a service from one cluster to the rest is also actually useful in Istio to kind of take a step back for users who do want to have some at least some of that cluster experience. And they can use the service export resource that we've described to do that same marking type of activity so that Istio can choose between full service mesh, expose ability or just the service export cluster local style. And the other thing I want to highlight here for MCS API integrations gateway API, which is the new ingress API. This now I've got an example over here from GKE, but supports referencing different types of kinds as the background reference for ingress traffic and it does support using service imports so you could have your ingress traffic get routed to a multi cluster service. If you wanted to using this type of format here so pretty exciting that you can connect your gateway API with your MCS API in the wild right now. So, diving in a little bit more into the cluster property CRD Jeremy mentioned before that it's about storing these properties about a cluster, the cluster property CRD itself is very general purpose it's just a name and a value so you can store you know anything you want in here. But the important part for the MCS API is that we want these two to definitely be able to exist this ID kates.io which is what we want to represent the clusters ID or name. And then this cluster set dot kates.io which we want to represent something that describes the membership of that cluster in a cluster set so the exact details of what that value is can be implementation dependent we have a couple of recommendations in the cap, for example that you might use the cube system namespace new ID as your clusters names been discussed by some other folks in sick architecture and around before as well. But as it come where it really comes as a cap is just to make sure that this CRD exists that the schema looks like this and that if you do use this name ID case I oh or this name cluster set that case I oh that certain properties of this resource instance can be assumed by users of the MCS API. So, looks simple but is very necessary for us to be able to track this type of membership and also cluster names which is useful for DNS. I have myself because here is some diagrams actually got four slides here that are pulled from a deck that we've been using at say multi clusters to discuss multi cluster DNS. And I won't go super into detail but I just want to highlight basically the different types of records that we're supporting with the MCS with the multi cluster DNS specification. So the goal, the idea is that we make this DNS look very similar to how cluster local works just make whatever minimal changes we need to that make this work in a multi cluster case. So you'll see this blue dot test at SPC dot cluster set dot local instead of cluster dot local if you're used to using our cluster dot local DNS. This is the big change here for cluster set IPA and or quad a records right we're just changing out this zone. So this will still get you to the IP that can route to any of the matching back ends. Just like you would expect for a cluster IP type service, but across the cluster set in the case of a multi cluster service. And then we also support SRV records for cluster set IP as well. If you want to get this DNS name back but are targeting a specific named port for headless services. Also, trying to keep it very similar when it comes to the aggregate pod IPs DNS so if you want to get back the pod IPs for every back end for a service name and namespace that has been exported across your cluster set. You can use the same looking thing right that you're used to blue dot test at SPC but again just changing cluster set instead of cluster dot local. The place where this cluster ID I keep talking about shows up is here for pod DNS for headless services so if you just want to get to this one pod over here or you just want to get to this one pod over here in cluster B. You need to know its host name, which is already normal for headless services in the single cluster case, but you also need to know the cluster name because it's possible that the host name for this blue pod over here blue one is the same host name as this blue pod over here in cluster B blue one right so if it's possible for them both to be the same we need some other way to disambiguate them and that is where this cluster ID that we've been working on actually shows up. And finally, SRV records also exists for multi cluster headless. This is a more useful use of SRV records if you want to know all the pod DNS names for all of the back ends for HTTPS you can use this SRV record to get all of these pod DNS names back so you can target all your back ends individually. I know that was a bunch of deep ish shallow ish dive I just wanted to show you that there's a lot more going on behind the scenes. If you do want to see even more there is another talk, including demos that myself and Stephen kit who's another member of SIG multi cluster produced for cube con and a 2021 so it's called here be services beyond the cluster boundary with multi cluster services should be published in a couple days here. And it includes demos of the MCS API and multi cluster DNS on GKE and open shift so you can see it in action. All right, let's talk about how you can get involved in SIG multi cluster which we would, the three of us would love to see here why don't we get that next slide. We really need your input. This is basically you can think of your input as one of our most precious natural resources in SIG multi cluster. Please let us know your use cases your problems and your ideas. You can see the the homepage here under the under the community repo in SIG multi cluster, the slack channel also unsurprisingly called SIG multi cluster and finally the list. SIG multi cluster. Our meetings are bi weekly on Tuesdays at 1230 Eastern 930 Pacific and 1600 hours and 30 minutes UTC. We hope to see you there. Thank you. Thank you very much for watching our talk. Thanks a lot. Hope to see you soon.