 Hi everyone, thanks for joining us for hereby services. My name is Stephen. And my name is Laura. And we're going to talk to you about the uncharted unexplored world of multi cluster services. Let's start with the map, shall we? This is the Linux globe, famous for anachronistically representing an unknown region with the words hereby dragons. They're in the corner here, where it says Hickson Traconis. So this is a good setup for our conversation here about conversation about services across multiple clusters. We can imagine it like our Kubernetes clusters are islands on this map separated by dangerous and uncharted seas. You want to open trade routes, but that means having open ports and sending ships which risk being attacked by pirates across those uncharted seas. Even if port taxes is controlled, which is good because you can then collect taxes, there's a lot of effort in each port. And since the seas are uncharted, how do other islands know about your ports? Now, there are some other ways that people have been trying to do this. We are at the service mesh track of this conference after all. So Istio is an option. This is an open source service mesh that you can deploy across your Kubernetes clusters and VMs in any provider. It's super powerful, but not everyone chooses to implement a whole service mesh or it might have a smaller Kubernetes deployment for which it feels too heavy. Other folks choose to roll their own solution. So sometimes people will stand up a reverse proxy. For example, engine X between multiple clusters and handle the configuration to route request multiple backends themselves. The downside is that there's no general configuration pattern for this. So you basically have to figure it out yourself. But there is another way. So CIG multi-cluster, which Stephen and I are both part of, have been working on a standard that describes how to extend the existing Kubernetes service concept to multiple clusters, leading to the multi-cluster services or MCS API. The whole idea CIG multi-cluster is going for is to provide a multi-cluster solution that feels really natural to how people already use Kubernetes services today. Everything is modeled around the existing service primitive and MCS defines the minimal amount of extra stuff needed to make them available across the clusters. The goal is that client application should have barely any changes in order to use a multi-cluster service instead of a cluster local service. This API is based around the idea of a cluster set, capital C, capital S, which is some number of clusters that are working together. The standard describes certain concepts that must hold true for clusters participating in a cluster set for them to safely deploy service objects across multiple clusters. So let's talk about what those concepts are, Charlie. First up is namespace sameness. This is the driving principle for MCS that shared namespaces across a cluster set should have the same contents everywhere, especially services. This is similar to the idea of extending services to a cluster set or extending namespaces to cluster sets. Think of it like when the same type of terrain or climate is shared across a certain latitude on the map, like we can see in our little map here with the colored regions existing across multiple islands. When it comes to climate on islands, this means that the same type of plants can all grow in the same colored regions. Plants that need the red regions climate could grow on the big island or on the smaller islands to the south. When it comes to clusters, this is like the fact that if your namespace exists across both of those clusters, you would expect the same services to be able to grow in both of those namespaces too. So in real practical terms, this means that services in a given namespace must be the same in any cluster in the cluster set where they are deployed. They will be considered functionally identical and the back ends of any matching namespace name are safe to respond to any request for service of that name and namespace, no matter what cluster in the cluster set they come from. The second concept is the MCS controller. This is the agent that actually implements the behavioral requirements of the specification. Sigma the cluster was very careful to define this broadly. The MCS controller could be a human type in kubectl commands all day, as long as they kept to the spec. The important point is that this MCS controller implements the cross cluster synchronization needed by the MCS API. This includes synchronizing a number of Kubernetes API objects across the cluster set, some of which are unique to the MCS API such as service exports and service imports, and some of which are in the core Kubernetes API, like endpoints and endpoint slices. So let's talk about some of those API objects, especially the ones that are new in the MCS API. A service export is a really simple API object, pretty much just name and namespace. But the point of it is to mark a service for export. Basically, by creating a service export with the name and namespace of a service, you tell the controller that you want that service to be accessible in that same namespace across the cluster set. This is the main interaction point for a user or administrator with the MCS API. You create these service exports. The MCS controller just consumes them. So what does the MCS controller do with these service exports? To synchronize the information, the MCS controller makes service import objects, transporting the metadata of the exported service and to each namespace same cluster in the cluster set. It includes information like the type of service and the ports and protocols of the originating service. This information is used to create endpoint slices, which is part of the Kubernetes API. So we are back in core API territory which we left behind at the service stage. These are all resources that the MCS controller makes for you as part of its role synchronizing the information across the cluster set. So all together it's like this. A service in one cluster can be marked for export by creating a matching service exports in that cluster. The MCS controller will watch those service exports and when it sees one will create the corresponding service import in all clusters in the cluster set that have the same namespace. The MCS controller will also create the endpoint slices in each consuming cluster as well. As you can see the MCS controller is acting like a cargo ship, transporting the information from a service export into new service imports in all ports of call. We hope you'll agree the MCS doesn't add much to achieve its goal from Kubernetes native services on the left. Back to Kubernetes native endpoint slices on the right. So we mentioned earlier that another goal of Sigma Cluster was to minimize the amount of change required in client applications. This leads us to the magic ingredient, which as you might have guessed is DNS. Multi-cluster services come with a multi-cluster DNS. This resolves A and SRV queries for the cluster set.local domain so that consuming applications can determine what IP address is to use to connect to multi-cluster services simply by changing your classic.cluster.local to.clusterset.local. There is a detailed spec with all the different DNS forms required but for MCS consumers there's two main kinds we expect people to interact with which we put on the slide here. So for cluster set IP services, there's one main name, the service name followed by the name space and then that.svc.clusterset.local. This is a variant on the cluster local service name continuing the practice of extending the standard core behaviors. For headless services, each pod hosting the service in each cluster exposing the service gets a DNS entry using its pod name and a cluster identifier which we'll revisit at the end of this presentation. The service name format is also supported and resolves to all active IP addresses on which the service is available and on top of that SRV records on the service name allow consumers to enumerate the available pods and you'll actually see that in action later in the Submariner demo. Let's see how all this works out in practice. Look at the architecture of Submariner, an open source project which happens to implement most of the MCS API. I'll also demo it later in the presentation so you'll get to see in practice. The first component in Submariner is its data repository where Submariner calls the broker. This is just a bunch of CRDs which are used to store the list of connected clusters, how to connect to them, and any other information which needs to be synchronized. The CRs using these CRDs are accessed using a Kubernetes API endpoint. This is typically provided by a full blown Kubernetes cluster, but it doesn't have to be. It can also be hosted in one of the connected clusters instead of a dedicated broker cluster. Now we know how to store information about clusters, how do we go about connecting them. In the simple case, each cluster has distinct siders so every pod and service has an IP address which is unique across the cluster set. To connect clusters, Submariner expects the administrator or an infrastructure support tool to provide at least one gateway node in each cluster. Submariner then opens tunnels between each gateway pair and sets up networking roles in each node to route non-local traffic through the gateway nodes to the other clusters. The situation is a bit more complex if any of the clusters have overlapping siders. In this case, Submariner sets up an overlay network to end up in a scenario close to the simple case. Each pod and service ends up with a unique address port combination in the cluster set. We add ports to the mix to reduce the IP address based requirements. Now that we have connectivity, we can start sharing services following the model we described earlier. In Submariner, this is handled by a component called Lighthouse. Inside the cluster where the shared service is hosted, the Lighthouse agent watches for service exports. When a service export is created, it produces the corresponding service import and stores it in the broker. In other clusters, the Lighthouse agent watches for service imports in the broker and copies them locally. It also creates the corresponding endpoint slices, which we'll see in the demo later on. Once a service import is present in a cluster, the corresponding service can be resolved using Lighthouse's DNS server. Submariner configures each cluster's DNS resolver to forward MCS queries in the cluster set.local domain to the Lighthouse DNS resolver. This checks for service imports and returns the appropriate IP addresses. Those addresses are accessible directly thanks to the connectivity provided by Submariner. All right, we're going to switch over to some demos now. I'm going to let Laura share her screen because she's going to show us how this works on GKE first. All right, thank you, Steven. So what I want to show you here is how this works on GKE. So first, let me mention some of the setup. So I have a terminal running over here. And I'm going to run this command to show you that I have these two clusters, GKE 1 and GKE 2. They're there, GKE 1, GKE 2. And they're also both registered in the same cluster set. So how we do that in GKE is by being members of the same GKE hub. So that's our version of a cluster registry. We can check this out with this G-Cloud command G-Cloud container hub memberships list. So there they both are. This is what the app looks like, but this is that they're both, they both have membership in the same cluster set. And then the last thing I want to point out here for the setup is that they both have the same namespace kube. So the setup is two clusters. They're in the same cluster set and they have the same namespace kube. So let's look over here to the CML that I've got. This is the CML we're working with. So on the left is the service object. I already deployed this in my cluster along with the deployment back in it. And the important part here is that we have this namespace kube con and this name where am I? And later in this demo, I'm going to deploy the service export. And this is the one with this exact same namespace kube con and name where am I? So that's the whole opt-in process for you as a user which I'm going to demonstrate. If you want to make sure that the service gets exported across your cluster set, then you create a service export that matches its name and namespace exactly. So let's start here with my two clusters. I've labeled my islands here, GK1 and GK2. And GK1 is one where we have the service deployed on it. But I haven't made that service export yet. So I cannot access anything from GK1. The service from GK1 from GK2. Nothing has been connected yet. And I'm going to prove that over here. I have another pod running in GK2, which I have a shell open in this terminal here. And I'm going to try and curl the multi-cluster DNS name for the service over in GK1. But we're going to get no response here. In fact, it doesn't even know what hosts we're talking about because no multi-cluster DNS has been set up right. So there's no connectivity setup between the two of them yet. But what we're going to do here back over in this view is we are going to create the service export now. And this is going to kick off the whole process that's going to make the service import end point slices and everything else happen in GK2. So I'll just cut that out real quick. So we can just double check. We're talking about this KubeCon RAMI1. That's what we're going to do to mark our service for export. And then I'm going to go ahead and apply this in GK1. So I'm making the service export here in GK1. So once we have that service export, the magic starts to begin. So the controller here, it's going to take all the metadata it needs from GK1 about the service and any back ends behind it. And it's going to carry that over and replicate that in GK2 in the form of the service import. So let's check out what that service import looks like. So over in GK2, got a command here where I'm going to look in GK2 for service imports in this namespace. So this can take a second to get set up, right? Our ferry is still checking along across the ocean here. So we're going to just check a couple of times until it shows up. And if necessary, use some moving magic to speed this up. Okay, great. There it is, right? Let me clear this out so we can see it beautifully. All right. So here is what the service import looks like, right? So we can see there's some information here from the original service. So like what ports it's exposing, what type it is, et cetera. And then we also have this, the spec.ip. So this is the VIP assigned to this multi cluster service from GK2. So this is how GK2 knows how to get to back ends elsewhere in this case to GK1. So we can check to make sure that DNS has gotten programmed to and just generally approve the connectivity between the two of these. Over in our DNS window section, I'm going to take this VIP with me. And first we're going to curl this VIP, right? So we get a response back and the response for this application that I have deployed in GK1 conveniently tells us the cluster name that it's responding from, right? So we are successfully getting this response back from GK1, even though, again, this pod that I'm curling from is in GK2. And then to see that DNS is working as well. We can curl this, where am I? We're getting the back end from GK1, even though I'm making this request from GK2. So this is the fun part, right? Everything connected together. The DNS is really easy to switch out. We just put this like set at the end, cluster set.local instead of cluster. So that's kind of the basic setup. But I'm also going to export the service from GK2 as well, so that we can see a little bit more going on. So I'm going to go ahead and apply some YAML that has my actual application in here into GK2. So I'm creating the service and this deployment, right? The same one that I have put into GK1. And if I just left it as that, then it's just a cluster local service. But if I also create a service export that matches it, right? So the same one with the same, we can double check it one more time. That has this namespace cube con and this name, where am I? Which is going to match our service that we just made GK2. Then this is going to take information the other way, right? So we're going to be taking, getting our ferry to take some metadata from GK2. Bring it over to GK1, make service imports over there, make endpoints over there. Technically, it will actually make this information available in GK2 as well. So we'll actually take a look at that in a second. But the point here is that because of namespace sameness, we're going to consider every service that has this name and namespace, if it's been exported to be safe to respond to requests for that service, regardless of where the request originates from in the cluster set. So this means that any cluster can access any backend for the service, regardless of cluster origin, as long as it's exported everywhere. So I'm going to run some acute cuddle command over here and we're going to watch it for a while to see when the endpoints actually get associated fully from GK2 with GK1 all in one, the multi-cluster service. It takes a little second to come up. So I'm going to kind of explain the details and maybe talk a little slower too. But what we're looking at over here in the terminal is the endpoints slices inside GK2. And you'll see for the skew command namespace. And you'll see that there's two instances here. This first one here is the cluster local service. I just deployed the service in GK2, right? So this is the endpoint slices for that. But then this, 77 seconds ago, right? But this one here is our multi-cluster service. So this is more of an implementation detail on GKE, but the way we actually hold multi-cluster service endpoints is by making a new service object. And then that also creates new endpoints or endpoint slices in the consuming cluster. So that's what this sort of prefix here, GK-MCS, this other name is here because that's how we implemented that in GKE. But the important part here is that right now this multi-cluster service has the endpoint for GK1, and I'm expecting the endpoint for GK2 to also get added in here. And that's going to be evidence that all the connections kind of connected together. And that over in GK1 and in GKE2, which is what I'm watching right now, the endpoint slices are representing all the back ends of the multi-cluster service. So the reason this takes a second, oh, it already happened. Here we go. Sorry. I talked too slow, maybe. So now we have the back ends for both of them in this endpoint object, right? I'll make it a little bit more clear, clear this out and take off my watch just so we can see the full current state, right? So we have the cluster local service that's always going to exist here as long as I deploy the service in this cluster. And then here is the multi-cluster service that now knows both endpoints. So practically what this means is if I go back over here to my DNS world, if I curl this DNS name, I'm actually going to load balance in front of both GK1 and GK2. So you'll see sometimes I'm getting GK2 back, sometimes I'm getting GK1 back, even though I'm using the same DNS name, right? We can do this a couple more times. More GK1, now GK2, now GK1, right? So because these services are considered the same to us, they have the same name and namespace deployed across the cluster, then they can be load balanced across just like the cluster IP services, but in this case we call it cluster 7IP. So there you have it. That's your basic cluster IP type service in a multi-cluster world. And I'm going to hand it over to Steven to show us how this works in Submariner and also with a headless service instead. Yep. Thank you, Laura. So as Laura said, I'll go pretty much over the same grounds but on Submariner. So I've set up two clusters, cluster 1 and cluster 2. I'm using only Submariner on Kubernetes with no fancy management tool, so there's no gear to show. But this guarantees that you'll be able to reproduce this in your own environment should you wish to do so. And assuming that your environment is supported by Submariner. I've already set up a broker on cluster 1. So cluster 1 effectively represents both islands on the left. And both clusters are joined in a cluster set. The same where am I service that Laura used is set up on cluster 2, but this time as a headless service, as you can see here from the cluster IP setting. So let's start by taking a look at the services in the KubeCon namespace on cluster 2. And there we can see where am I and it is a headless service. We can look up its IP address from inside the cluster. And there we have two addresses for it. And we can contact one either one to see its output. And so if I repeat that eventually there, I've got the two different IP addresses, 10.2.96 and 10.2.128. So this is on cluster 2 and from cluster 1, the service is unknown, whether I use the cluster domain or the cluster set domain as you might expect. But thanks to the connectivity that's provided by Submariner, we can access it from cluster 1 using one of the pod IPs. So I'll get one of the pod addresses and contact it from cluster 1 directly. And we can see we get a response from 10.2. So let's actually export the service properly. With Submariner, we could do this using a Submariner tool called SubCuttle, but I'll show you how it's done using a service export CR. So that's it created. And now the service can be discovered from cluster 1. There we have both addresses again. And this was indeed from cluster 1. And we can contact it using the cluster set address. And same again, eventually I'll get a response from a different IP address. But you might just have to trust me on that one. And I remember this is ahead of the service. And without cheating by looking at the cluster 2 pods and the cluster 2 cluster directly, we can find the endpoints by looking up the SRV records. So this is from cluster 1 again. I'm going to look for the SRV records and we get back the two different host names here for the two pods. And we also get back the port number. So this way we can figure out how to contact the endpoints directly. And so I'll do that right now. So here we're just querying the SRV records again to get one of the host names. So that's it here. Where am I? 5C, 5D, et cetera. And we're contacting it on port 8 to 80. And we get the response back. So let's take a look at the underlying data structures. First, the service imports on both clusters. So I'll start by cluster 2. So we have here one service import for where am I? And on cluster 1, we have 2. This shows how Subrainer shares the data. The entries in the Subrainer operator namespace are the cluster local copies. And the entry in the Subrainer case broker namespace is the broker copy. So that's the shared copy that's copied everywhere. And the imports are used to produce endpoint sizes. So let's take a look at those. On cluster 1 this time. So we have just the one endpoint slice corresponding to the remote endpoints for the cluster sets. And on cluster 2, we get both, we get two back, one that's the cluster set endpoint slice and the other one that's the cluster local. And to finish off, let's take a look at service availability. I'll set up where am I on cluster 1. Here we go. And I'll export it. So this is the same as previously, but on cluster 1. And now if we take a look at the SRV records, we'll eventually see four pods. So there's just two for now. And that's it for. And they're on the different clusters. Anyone of those can answer. So if I query from cluster 1, I'm still getting back and a response from cluster 2. And now I've got one back from cluster 1, cluster 2 again. And if I drop the cluster 2 deployment, the endpoints are immediately updated. And so I'll do a combined command that deletes and queries the SRV records again, just to show you that it is actually immediate. That's it dropped. And we're back down to just two entries for the SRV records. And if I redeploy where am I on cluster 2, the endpoints will be restored. So I'll deploy it again. And now if I query again from cluster 1, that's it. We've got four responses back. And so that concludes the summer in your demo. And I'll hand back to Laura for the rest of the presentation. So thanks. So hopefully everybody got a good taste of how this all works in practice today. Let's talk a little bit about what the future is for the MCS API. So as we've alluded to in this presentation, there's parts of the MCS API that each individual implementation is not fully supporting yet. So as the standard becomes closer and closer to its final form GA, each implementation is working to converge towards the specification so that it can be considered fully MCS API compliant. For GK, for example, our implementation of multi cluster headless services doesn't yet create endpoint slices in the consuming clusters and doesn't provide SRV records yet either. For Submariner, cluster set IP services aren't allocated to stable virtual IP that clients could actually use. Everything is handled through DNS instead. But I point this out because this is one of the interesting things about working on the MCS API right now. The spec and the real world implementations are developing together. So as a multi cluster debates points to include in the spec, we can try them out in the real world and come back and provide insights. This has been particularly helpful to distinguish between things that must belong in the standards and things that are just implementation details. This has been a really successful way to create a standard retail component is useful to everyone and is agnostic to different technical stacks. Great segue because the best future for MCS would include even more implementations of the MCS API. As mentioned, this is the best time to try more approaches and get more feedback while the standard is still in development. That being said, there are a few other pieces still moving as well. That cluster ID that we need to make DNS records for the MCS API is a cluster ID that is implemented in the multi cluster headless services that is represented. That is represented in a separate cap and the CRD is just recently an offer and is called cluster property. Getting that cluster property CRD implemented in each of the current implementations and moving it to beta and then GA is part of the process of getting the MCS API to GA as well. Besides just pure MCS API implementations like sub and sub, we also wanted to see other community projects standardized on top of the MCS API to make it easier to build more complex solutions using the same MCS API we all now know and love. The gateway API, the new ingress solution from SIG network, supports using service imports as back end references for ingress routes. Istio itself is also implementing parts of the MCS API to make it easy for you to use MCS API first, and then you can use it as an upgrade. Upgrade with very little changes to your application configuration. So are you ready to try it out? We invite you to install this yourself either using OpenShift or GKE as then with today, or installing the CRDs in Subrainer.io in your own clusters. Subrainer has a series of quick start guides on how to deploy this MCS controller yourself using clusters running on your own. Finally, SIG multi-cluster would love for you all to get involved in the development of the MCS API and all the other good multi-cluster projects we have going on. We meet bi-weekly on Tuesdays over Zoom, and there was also a maintainer track talk published a few days ago with even more details about all the projects we're working on. So hopefully we can see you there. Thank you, everyone, and we will see you at the Q&A.