 All right. Hello and welcome everybody to this open source summit EU virtual session. Thank you very much for joining us today, especially at the Friday afternoon. My name is last opens and I'm a software engineer at Cisco. And I'm here with my colleague. We'll be a copers co presenter today. Would you mind introducing yourself? Yes. Hello guys. I just put a camera to you. Hello everyone. I'm sure it was a software engineer at Cisco and are you happy. All right. So how we are going to do this mostly is that basically I'm going to do the blah blah blah talking parts and then we'll show us the actual and meaningful demo contents. So today we would like to introduce to you to an open source tool called cluster registry. So cluster registry is a tool that we designed and implemented at Cisco internally. And we fairly recently open source it. You can find the source code that we have. And there you will see that this is a Kubernetes operator. The business logic is written in a Kubernetes operator and it is written in Golan. So we prepared this build this tool originally for our for a very specific use case of ours. But from day one, we had in mind to create a fully generic tool to support numerous multi cluster use cases. At least this is what we hope. And that is why we are giving this presentations and doing similar things to show around this open source tool to the community. And we are very much looking for feedback to understand what kind of use cases you might have. So as you are listening to this presentation today and it rings about to you that it might be useful for this or that kind of use case. Or you have ideas how we could improve this please and any feedbacks are more than welcome. So cluster registry. If I would have to summarize the most the two most important things that it does is that first of all it helps to form a group of Kubernetes clusters to be able to do some kind of operations besides those clusters. So as the first thing, and probably more interesting interestingly, it also helps to actually synchronize any kind of Kubernetes resources across these Kubernetes clusters. So you will be able to synchronize not only like pods or deployments, but any kind of other Kubernetes resources as well on demand configurable with the depth of the API using this this cluster registry tool. So these are the two most valuable things of this tool. And we will show in detail what we mean by this. So now I will just mention that it is also fully distributed. So it has no central component. It is no single point of failure. If it makes sense for your use case, then then it might be useful. And also it uses a gossip like protocol, meaning that it's not like if, if some tenant has some kind of information, it's not like it needs to be talked to directly to all other participants in the cluster group. It's enough if it's shared funds, and then it will be spread across the participating clusters. Yeah, we can we can move to the next slide, which is showing us most of the architecture, let's say, or how the cluster registry controller works in multiple clusters. So it is in a multi cluster case it is the controller is installed in each and every cluster. And the important thing to note here is that from a security perspective what we did is that the right operations only happen on local Kubernetes clusters. And the remote side, the remote operations are always read operations only. So then when a controller needs to do some kind of synchronization between these clusters, it only reads from the, from the remote clusters and it will do the right locally. Let's talk about the API. So basically, we have C free custom resource definitions CRD is defined for the cluster registry. And what we are planning to do is go step by step for each of these three, and basically explain these steps. The first one is the cluster CRD, which is exactly as you might expect, it describes the Kubernetes cluster itself. It has a name, of course, and it has cluster ID, which, which aims to be a unique identifier of that cluster. The implementation detail is that it's actually the you ID of the cube system namespace, because that's not that namespace cannot be deleted so it will be the same as long as the cluster is live. And there are just two other important field sections in this in this year. The first one is secret reference or some kind of authentication information for that cluster, which basically in this case holds a cube config for this cluster for index test. So basically, again, this year is present in this cluster. And this is the authentication information for this cluster itself. And other clusters. See this have this information them based on this information they can talk to this cluster. And another thing, which is needed is the Kubernetes API endpoints field, where basically you can override the Kubernetes API server address defined from which exact network, on which except IP, you can teach that Kubernetes API server. And also if needed, you can provide a Cable and Cable as well. Yeah, and this was the spec of the clusters here and the status is basically filled by the cluster registry controller. So basically it collects all kind of information from the API server from multiple places and it collects these two one single place, where you will be able to see all this kind of distribution when this version of the regions etc. All this kind of information at one place about that single cluster. The only other field that I would mention is the type field which as you can see in this example it's local. And another option it can be spear and when they show us when that is the case. Okay, so let's turn into demo mode. Okay, so so we prepared three clusters for this presentation. Basically, we all just have very small pre configuration here so the first cluster is almost completely empty. The other ones just only contains the cluster registry controller for speed of the process but for the first one I can show you how easy it is to install that. So, so basically, we have a ham chart for the for the cluster registry controller that will manage the cluster here so everything else that was he was talking about. So, what you basically see here is just a simple ham install the two variables that we set is one is the cluster name and the other one is the the API server endpoint address that was also mentioned earlier. So if you hit enter, if you just brought it easy to install the cluster registry controller. It's a pretty small from resource and size perspective so it should be up and running relatively quickly. And you will see that if you create a cluster resource as well and this is a this is a convenience of the of the controller. So basically, you don't have to create the local cluster resource but based on the provided information it will create it for you. So, right now you can see that it's already it is already created and and if we if you check the the content. So basically we get the spec part is is configured with the endpoint everything else and also the status is also filled with with those useful information that you have. Okay, so, yeah, we have a cluster defined. This is a local cluster and if you check the other clusters right now, you can see that they all have the similar local cluster resource that will show what you have locally. Now, okay, so that's pretty good but not good enough. So the next step is to somehow collect those clusters together. Now, this is pretty easy, because all you have to do basically is just to copy the cluster and a circuit for that particular cluster. Now, I just copy it and I just mentioned quickly so when the control when the controller starts it basically creates a local secret that contains a queue config for that particular cluster. And if we only have just provide free access for that cluster. So basically this command we just copy those resources resources and we can just paste it on any other cluster. So if you paste it on the second cluster. That's right. Yeah, so if you paste it on the second cluster, then what will happen if you check the clusters, then we have to. We have the first cluster set here. If you have some more information, then you can see that it's not seeing yet, because obviously the other side is just not configure them in the first one. So we have to basically repeat this step and copy that information from the second cluster and paste it on the first one. And if we are making it right, then it's now seeing. So basically those two clusters are known, know about each other. Okay, so what about the third one. Basically, the step is kind of similar. So we copy the information from the first one and paste it on the third one. Now, that that is very interesting from what comes in. So basically, as you can see, we've already had have just the local cluster previously now we are the first one, but there is a second one as well. Why is that is because we have the synchronization subsystem we did the controller and okay we only just have connection for the first one and is as you can see it's not fully seen yet because we are not on the third here, but it will be still able to see basically the cluster information. So just to finish the synchronization, we just copy the third cluster's information and paste it on the first one just to finish the cluster for group forming. And as you can see, if we check in any of those clusters, the representation of the whole cluster group should be exactly the same. And so, and as you probably get is that we don't have to add, as I mentioned is like a gossip protocol so they don't have to add, if you have, I don't know, 10 plus test, obviously don't have to do this. So we have a connection manually for, for, you know, a lot of times, but with the synchronization it's, it's just pretty easy to to form even larger groups. And, yeah, and so let's get back to us to continue with the presentation. So also the question is how is that possible, because obviously they just copied to one cluster this cluster, CR and the secret, and it was already on the second on the other one as well. And how is this possible. And also, it's great that we connected these clusters to a group. I mentioned that this tool is able to synchronize resources between these clusters. How does that work. And that's exactly where the resources include CRD is coming into the picture. So basically, with this CR, this is the most important CR of this of this cluster registry tool. You can, you can define on demand, but kind of resources you would like to sync synchronize across these clusters and how exactly. So basically what you see here is that in this example, we are syncing a secret type resource. And basically, we can provide an exact match for an exact object. Right here we are syncing this test secret from the cluster registry namespace. And we are also providing different measures for example for the annotations. And this, this API is pretty, pretty broad. So you can define a lot of, a lot of other stuff, a lot of other measures. So you can define measures for namespaces for any, any particular key value pairs that this resource holds, or label measures with the does not exist or you know, other similar label based operators. All this kind of stuff you can define, and also besides the measures, there is another tool you can define which is the mutations. So you can not only synchronize simply resources across these clusters, but you can actually apply mutations on those. So you can modify the original resource and to, for example, in this example add a label to this, to this other resource. So this is, this is what it does and being able to be kind enough to show it to us. Okay, so get back to the demo mode again. So basically, just to show you guys real big so so when a controller starts it creates these resource sync rules by default. And this is why people are able to, to synchronize or the tool will be able to synchronize the clusters because there are that specific resource sync rule to move the, the secrets and the clusters and the singles that says over. Okay, so let's give you that, that example that lots of compassion. So, so we have a resource sync rule here. And we get back to those later stage. So, the reason simply is just, it's a bit different from what's on the slide but basically be matching for, for secrets in the default namespace that matches that annotation. And when it synchronized over the, we would like to mutate it and so on this particular label to the synchronize pieces. See what happens if I apply that particular resource. As you can see, it's added to the, to the cluster. And if you check on the other clusters, there is this synchronization rule as well. So by default, those synchronization rules are themselves synchronized in the whole group. If you check in the default namespace what kind of secrets we have there. You see just the, just the default service upon token secret there on the second cluster. This is kind of the same on third one as I said those are empty clusters this. Let's see what happens if I add these secrets on the first clusters for example if we can check there. The state is, oops sorry, there is a bit more gap than I need. Okay, so there is no secret there what if I add one. We check this. It's created. Check the other signs. Second one, third one, boom, there are those secrets. And also the label that the other annotation is also there. So if I issue the exact same command, then you can see if I can select this. You can see here that that particular label is not set on the original resource but is set as a mutation on these resources as well. Okay, what happens if I if I remove a sync resource. Okay, so from the second cluster, I remove the test secret. Okay, is there again. So you cannot remove why because the original is already exists. So if I also, but if I delete from the original location, you have what happens, then it finishes. Okay, so it gives the whole thing just fully synchronized eventually. Okay, now, basically, as I mentioned, the resource sync rules are synchronized, you know, in every other cluster now, but there could be a case where you don't really want that. So you don't want to have the same synchronization effect maybe on every participant cluster. So what you can do there to solve this, I just removed this. And it's now removed from the other side as well. So what you can do is that you can use a special annotation to basically modify the, the, the resource thing, the synchronization for the resource, and you can specify that that particular resource sync rule should not be synced over. So let's just apply it on the on the second cluster. Let's say I just want to synchronize those particular secrets on to the second cluster only. Then we can apply this resource here. We can check. It's there. If you check the other sites, it should not be there since we said that annotation. So again, with the same secret is obviously it's not here. Let's see what happens is if I apply it again on the on the first cluster. If I apply it, then it should be synchronized over here and it's there. But it's not on the other cluster because the resource sync rule is just basically just created here and it didn't get synchronized. So this is just a very basic example about how you can synchronize over resources, how you can control this is basically a cool model. So the resource sync rule is kind of configures what to pull. And with that annotation, you can, you can find way into this, you know, based on that you put it. And with that, I let's get back to last to continue the presentation. So basically, the resource sync rule, you can synchronize resources across all your clusters. And with this annotation, you are able to define which clusters to synchronize resources as well. And maybe you are wondering, is there a feature that you can define which clusters from you would like to synchronize only. And yes, there is. And this brings us to our third CRD of the cluster registry, which is called cluster feature. So basically, how this works is that in the resource sync rules back, there is a cluster feature match field where you can provide a feature name. And basically what happens is that on that cluster, rather this resource simply defined, it will only pull from those clusters, then there is a cluster feature CR defined this exact feature name. So on that is that other CR on this example, the above. So the cluster features here with that test secret feature feature name set on those clusters that it is defined. This secret, these resources will only be seen from those clusters. Okay, let's just do that them on it as well. So let's get back here. I just removed the secret again. And I also, it should be removed. I also removed the resource include as well, just to be a bit. So go back here. Comment this annotation out but have discussed that picture. So what it changes basically what I'm going to show you is that we reapply this resource include it will be synchronized to all the clusters is that's the intention because we don't have this special annotation, but we are the initial match here. So what will happen is that when I apply it on the first cluster, it will be synchronized. That's fine. Now see what happens if I create the secret again on the first cluster. I create the secret. See what we got here. Why, because we configure that we, although we have the resource in cool on the third and the second and third cluster. There is basically where the cost that is created on the first one that the cluster feature is not exist. So basically, it's never matches so it's never get synchronized. So to fix that, we can apply the cluster feature on the first cluster that that that says basically that it provides that feature. And if you check the secret again, nice gets synchronized. And we didn't have to, you know, we didn't change the resource includes there. So it's a pretty big coupled way. So basically, the resource includes already there let's just say some some applications on some resources then that application basically to a resource in cool and intent to synchronize something over without any knowledge about which other process can provide within the cluster loop. And basically that can change, you know, at any time when somebody installs another application that is, you know, provides that particular resource that the intent is for the other process, then it can install the cluster feature. And so basically, this is it one, one other thing that would like to mention here is that why we have this cluster feature because, you know, generally we could say that we can select clusters for for, you know, for the same labor, so the clusters by labels, for example, but why we choose not to do that is because of ownership. So if you want to, let's say you have a cluster resource and if you and if you would base this feature on to on labels, then somebody has to accept different kinds of labels on that particular cluster resource on on a cluster, you know, based on several factors, and you have, you know, 10 cluster features provided by 10 teams in your organization then who has the ownership for the cluster features. It's a much more clear, you know, connection and architecture with the different cluster features see how the because then those teams can define their own cluster features, you know, without stepping on each other stores. So just just this small addition to that. And let's get back to let's continue the presentation. All right. Thank you very much for this nice demos and explanations. Sure. So last but not least, I would like to touch upon other importance topic of cluster registry just real quickly, which is just the permission question. So basically, by default, those remote queue conflicts have very limited access to those other remote clusters. That's totally intentional, just to be able to read, but we absolutely have to. But what happens if we want to synchronize some other type of resources, let's say config maps or it could be even custom resources, which are by default not allowed for the cluster registry controller to read from the remote clusters or to write on the on the cluster. So for that, we are actually utilizing built in Kubernetes feature called cluster role aggregation, or something like this, probably. And with that, basically, you don't need to modify the actual original sources service accounts with the cluster roles rather with the other rules defined rather you can define your own cluster roles. So for example, let's say you would like to synchronize config maps, you can, you can specify on those roles with these labels, and auto auto magically, these will be applied to the original cluster registry cluster roles, and this way, very easily, you can define that you would like to read or write certain resources. And with that, I would I would just summarize that, again, this cluster registry tool is able to form Kubernetes cluster group, which is which is already a great benefit, you can have a consistent you will be a clusters. You can see the other clusters as well. And with the very extensive API as you saw, you are able to synchronize any kind of Kubernetes resources in a decorative way, close to clusters, and you can even define how exactly. And again, I would highlight again that how this system works, it is fully distributed. There is no central components, out of those three clusters that ran them out, none of them were special, none of them were central or anything like that. This everything works in a fully distributed manner. If you are interested in more, check out the repo, the source code reach out to us. And again, we deliberately prepared this demo in a way that we showed the CRs, the Lego steps that we have for this and not, we were not showing an exact use case, because this way, I hope it resonates, hope it resonates with some of you. You are seeing maybe some use cases that you might use this for. You have some ideas how we could improve this. So, again, I would encourage everyone that feedbacks are very welcome and we appreciate that. So you can find us at this email address is check out the source code. And thank you very much for listening today.