 Hi everybody, welcome to the SIG multi-cluster intro and deep dive. My name is Laura Lorenz. I am a software engineer at Google working on GKE. I'm Paul Maury. I'm a software engineer at Apple. And I'm Jeremy, I'm Sid Thompson, and I also work on GKE at Google. Welcome everybody. Today we're going to cover what the SIG is about. We're going to look at current activity like the deprecation of QVED, cluster set and namespace sameness, the about API, multi-cluster services, and staple set slices. We're also going to see a deep dive and demo on the about API and multi-cluster services API in action using the AWS cloud map implementation. And finally we're going to talk about how to contribute and get involved. Let's talk about what the SIG is about. There are a number of problems for which there's a question of what should the Kubernetes native solution be for them, like exposing workloads in different clusters to one another, sharing cluster metadata, and a cluster's place within a fleet with its users. And how do we generally break down the walls between clusters? This work touches a number of different functional areas and we're still working to identify the best, most durable primitives. And that is a good opportunity for you in the sense that we need your input. Your real user stories and use cases are extremely valuable for us to know about. Please tell us what you're working on. We're going to talk about how you can get involved later in this presentation. Let's talk about the approach that we want to take as a SIG. We really want to avoid premature standardization. We want to focus on APIs and leave room for multiple implementations of the same APIs. We also want to avoid solving any optional problems and be laser focused on keeping multi-cluster consistent with the single cluster experience within Kubernetes. And we want to work backwards from very specific problems into bigger things where that is applicable rather than starting with big ideas that don't have a specific focus. With that in mind, let's talk about the current activity within the SIG. First, we wanted to communicate to you all today that we've arrived at a decision to archive the CubeFed project, which you've probably heard about from previous intro and deep dive sessions. There are a lot of great learnings that came out of this project, but it's no longer under active development. Fortunately, archival does not mean deletion. You will still be able to access the CubeFed code. You'll be able to fork it and change it as you please, similar to how you can change and fork and work with any code in the Kubernetes project. And finally, we wanted to say thank you to everyone who contributed to the CubeFed and Federation projects. Thank you. All right. I want to talk a little bit about cluster set, which is a word that we use a lot in SIG multi-cluster and has been talked about some in our previous intros and deep dives, but to reiterate this very core concept. This represents a pattern of use that we've seen from the field where folks who are using multi-cluster deployments have a group of clusters that are governed by a single authority, have a high degree of trust within the set. And importantly to note for this concept of cluster set is that they abide by this concept of namespace sameness. So there's a link here for some more details on namespace sameness, but in short, it's that permissions and characteristics are consistent across clusters for a given namespace. So it's not that the namespace has to exist in every cluster that's in the cluster set, but it's that they should behave the same across those in which they do. And this concept of a cluster sets membership is something that is stored in this cluster property called cluster set.kates.io, which I'm going to talk about on the next slide actually. So for this cluster set membership concept and other cluster metadata that is important to multi-cluster problems and also just generally across the Kubernetes ecosystem. So one of the projects of SIG multi-cluster is the about API. So there's a cap 2149 linked here. You can also go to SIGs.kates.io slash about API for the code. But in general, this is a cluster scoped cluster property CRD. So it's a CRD called cluster property that stores a name and a value. It's very simple in terms of schema, but the point is to make a dedicated place for folks to store about the cluster. So as described in a lot of detail in that cap, this is used to uniquely identify clusters using this yellow background format on the right. Also to identify their membership in a cluster set using this form that you see in the blue bubble on the right for the lifetime of their membership in that cluster set. And overall, this is a reference point for multi-cluster tooling to build on within a cluster set. So for valid DNS labels or to help with logging and tracing. And then in general, this is now a well known place to store these and also any other cluster properties that you might need that might otherwise have been implemented as ad hoc annotations on semantically adjacent objects. So we have an example over here of this green bubble where there's plenty of room in this about API for your cool implementation of anything to reserve a name and be able to store some value that is cluster metadata that you need. Let's talk about multi-cluster services or MCS. So we've developed an API, you can find it at sigs.kates.io slash MCS API to help spread services across clusters. When I think about multi-cluster, the first thing that comes to mind is services. They are core building block. I think about how can I consume services from another cluster? How can I move services between cluster? How can I separate my producers and my consumers so that they don't necessarily have to care about where my service lives? And MCS is our solution here. MCS allows a single service to span and or be consumed by multiple clusters easily without having to think about where the service actually is without producers having to think necessarily about who and where the consumers are and vice versa. We focused only on the API and common behavior. Our approach, as we talked about earlier, is to focus only on the problems that really need to be solved and to leave room for implementations. That's what we've done with the MCS API. We define the service export, which is a resource that lets you export a service. And that leads to the creation via your various implementation to import a service via a service import that is created in receiving clusters. And that's it. We left all the room for implementations that we could. And this has been really great. So far, we've seen a few pop up. We've got Submariner, GKE has one, the SEO service mesh has one as well. And there's a new implementation for AWS. Consumers only ever have to rely on local data, just like when you consume a traditional in cluster service. And this means that, you know, implementations are free to have distributed control planes, centralized control planes, whatever is needed. You know, we recognize that in various different platforms, if it's on-prem, if it's in the cloud, if it's whatever it may be, you know, you have different needs. And the best way to get endpoints between clusters and exposure service might be different. So we wanted to leave as much room as possible for implementations to decide what's best. And the most important thing with MCS is that your cluster IP and headland services just work across clusters like they do in a single cluster. It's the same experience you're used to. It uses the same, you know, cube proxy networking components. It's exactly like consuming a service within your cluster, except the endpoints can be somewhere else and you just don't have to care. The next project we want to talk about is stateful set slices. This is brand new, but really, really exciting. And what this aims to solve is spreading a stateful set across clusters, specifically in the service of migrating stateful sets. So, you know, MCS has solved some problems in this space. You know, you can spread a service across clusters. But when you get to stateful, things get a lot more difficult. And that's the problem that we're solving here. So with stateful set slices, you'll be able to have an implementation that moves one by one your replicas from, you know, cluster A to cluster B without downtime. Again, kind of following the patterns that you're used to in a single cluster. This is this is really exciting fresh work. And we recommend that you check out the talk from our friends, Peter and Matt, which was a couple days ago. So hopefully the recording has been posted now. Please check it out. This is exciting new work. And we're very happy to see where it goes. We have a lot more going on. We are still working on adding more sophistication to multi cluster networking in partnership with say networking they were looking at network policy multi network support. How do these things fit in the multi cluster world. We're investigating what it means to build a multi cluster controller potentially distributed around the world with things like multi cluster leader election. So we're trying to figure out what what the reference implementation should look like or what APIs are really needed to support a, you know, scalable, basic standard multi cluster controller experience. That's something we progress now. And we've got the work API, a newer project that's focused on figuring out how to get groups of resources into different clusters. So if you're coming up with new ideas, if you have new ideas, please come talk to us if you have problems, you know, we want to hear about them. This is the direction that the city is heading right now. All right, now I want to take us into a demo. I want to showcase those that about API and MCS API that was just talked about in action and I'm going to use the AWS implementation of the MCS controller. It's built on top of AWS cloud map. And I want to do a shout out to Akash and Dustin, GitHub handles run Akash and a static void for all of their work on this and also helping me out with this demo. So I'm just the messenger they did all the great work. So I'm going to show you right now a demo using the AWS cloud map MCS controller for Kubernetes. So this is available on GitHub here. And there's read me here with a lot of directions. There's also a blog on the AWS site about how to go through the installation as well. But I'm going to show you everything kind of after everything is set up and how it interacts with both the about API and of course the MCS API. So coming over here, I have these two terminals to kind of showcase that I have two EKS clusters running right now, I guess my red and my blue cluster or my dev one and my dev two cluster. And what I want to highlight first is that both of these clusters have the about API CRD installed. So let me actually show you the CRD that are installed. You'll see that it has this cluster properties at about kates.io. This is the about API. And it also has the MCS API in the form of the service exports and the service imports and same thing on my other cluster. So the first thing I did to set up here was actually to set in both my dev one cluster and my dev two cluster, their cluster names using the about API so I named my dev one cluster mega cool cluster. And I named my dev two cluster unbelievably cool cluster. This is going to show up later when I show you headless services. So overall, just as an example, this is what those CRDs look like. So again, using the about API, they're of type cluster property. And we have one that has this well known name cluster cluster set dot kates.io. This is the name of the cluster for my dev one cluster is called mega cool cluster. And then about kates.io for sorry, cluster set dot kates.io is the name of the cluster set and I've set that to make a cool cluster set in actually both cases if you are interested. So we've got unbelievably cool cluster over here but still mega cool cluster set. So get out of both of these real quick. All right, so I've already preset up my dev one cluster to have some services running. So I have two examples here one is the one from the AWS cloud map blogs example so this is a cluster IP type service. That's backed by three engine x pods. And then here I have another service called where am I this is a headless service that is backed by three pods running this where am I deployment. So right now I don't have any service exports running in here so I'm going to actually create them. So first off, no service exports right now in my example space, but I can go ahead and create them. Applying from this exports.yml file that I have. So this is creating a service export where for the where am I service and a service export for the my service service. Let me just show you this really quick here. So this is what service exports look like for those who are unfamiliar, very simple. They're just the namespace and the name of the service that should be exported from this cluster. So it basically acts as a flag to say yes, the service should be exported into other clusters of the cluster set. So we can actually go head over to cluster to here and take a look at the service imports as they're being created. So right now I'm grabbing the service import specifically for the my service named service, which again is the cluster set IP service. So here's what this looks like on this side. Importantly here you can see it knows its name and its namespace. It also has some other properties of the underlying service forwarded over here to my dev2 cluster such as the cluster set IP, the port, the protocol, and again what type it is. And it also has some metadata about the cluster that this service is backed by. So the actual pods are in mega cool cluster, right? That was the name of my red cluster whereas my blue cluster was unbelievably cool cluster. So that's also included here in the service export and we can fully experience it all by I'm going to drop myself into a busy box that I've already got running here on my dev2 cluster. And I can go dig for the A records for this cluster set.local DNS, right? MyServiceExample.svc, this ending zone here cluster set.local means that I want to query for the multi cluster service and you'll see that it responds with this IP. And actually we can scroll up a little bit and see this IP and the service import for the cluster set IP service 10, 100, 112, 152, 10, 100, 112, 152. So that's the case for the cluster set IP type service. So I want to showcase something over here about the headless services. So if I start actually inside a busy box I have running in my dev1 cluster. I just want to showcase to you that this is a cluster local service here of course. So it has these three replicas behind the headless service. And if for example I curl this you'll see that I get this output here that indicates a couple things. But one nice cute thing is this pod name emoji so I can recognize it as different from for example this IP responding, this pod IP responding. So I want to showcase that that's what's going on over here again with the cluster.local DNS zone just regularly in my dev1 cluster. Now if I go over to my dev2 cluster I can take a look at the service imports again this time for my where am I service. And again it has the name, the namespace and it has some metadata about this service including that it is of type cluster. And again of course it knows what cluster the backends are originating from. And this is all important because if I again exec into my busy box I have running over here. If I do a dig for the A records for again the cluster set.local zone you'll see that I get all of these endpoints back again. These are the backends that are running in this red cluster up here in my mega cool cluster up here. So I can also grab the serve records and this is going to showcase that each of these also has a pod DNS that is disambiguating based on the cluster ID. So this cluster ID again got pulled from the about API that was set against the cluster dev1 I called it mega cool cluster. The service was exported from here imported into my second cluster here. And then now it has these DNS names that target the pod DNS individually for each of these backends back in the mega cool cluster by their IP and their clusters name. So we can again use this if we want to do a little curl in here and we can get our you know our cowboy back or we can do this one and we can get our flag back. So we can target each of these pods individually with our pod DNS. So all of this again is back here again it's AWS cloud map MCS controller for Kate's and the thing that it's built on top of is this AWS cloud map product in AWS. So we can actually kind of see in here the actual cloud map services that are created for the namespace my service and where am I for example for where am I have these three service instances in here. For my my service I have a entry in here as well. Thanks again to everybody who has been working on the AWS cloud map MCS controller for Kubernetes really excited to be able to demo the hot new 0.3.0 version for everyone today. As we said earlier we really need your input to know what the right things to be spending time on are. You can check out our homepage slack channel and mailing list that are linked below. Please join us for our bio weekly meetings which are on Tuesdays 1230 Eastern 930 Pacific. Time those are US time zones. If you'd like to bring something to the SIG please feel free to throw it on to the agenda it's world writable so you can just go in there and make some edits and we'll see you I hope at one of those meetings. All right thank you so much for joining us for the SIG multi cluster intro and deep dive we're going to see you at the Q&A now but hopefully you had a great time at cube con and we look forward to seeing you at the SIG. Thank you everybody have a great cube con. Thank you very much and hopefully we see you at the bi weekly meetings.