 Okay, hello everybody. My name is Jaluka Marventa. I'm a principal engineer at Cisco and my talk today is about how to manage Kubernetes add-ons for multiple cluster using cluster runtime state. We started this project because we were using cluster API on prem to bring up a bunch of Kubernetes clusters on demand. So our goal was to create a cluster, upgrade the cluster, delete the cluster when we were done. When you bring up Kubernetes, you know that after you have a Kubernetes cluster now, you have to start installing like a bunch of add-ons. You need like a CNI. If you want to have network policies that dictates who can talk to whom, you need to have like Prometheus. If you want to collect metrics, you need to deploy your own applications. And there are other tools today already. Flux, Argos, CDE, Ranger Fleet, if you're using like Ranger, to deploy add-ons. And Zveltoz was just to be, wants to be like a solution to easily manage and deliver cluster add-ons to tons of clusters. As I mentioned, we started this project. We're using cluster API. So there is a management cluster and from this cluster all the managed clusters are reachable. And the idea behind Zveltoz is that you install Zveltoz in the management cluster, and then you use a Kubernetes label selector to select a set of clusters. And then you simply list all the add-ons that you want to be deployed, and then you let Zveltoz do the work. And Zveltoz is going to deploy like all those add-ons. And on top of worrying about like taking care of scaling the number of clusters they can manage, Zveltoz also wants to give you like a tiny point of time, an exact view of what add-ons is deployed on which clusters and where. So those are the two main goals. The cluster profile is the CRD that is introduced to instructs Zveltoz on what to do. And it has like three main fields in the spec section. The cluster selector, which is a simple Kubernetes label selector. So in this case, in this example, it is selecting all the clusters which have labels environment equals to FE, which stands for functional verification. And then it has an ElmChart section where you can list all the ElmCharts that you want to deploy. And then it has like a policy ref. Policy ref, it's simply pointing to config map and secrets. And each one of those reference config map and secrets in its data section can contain like Kubernetes YAML resources. So Zveltoz can take any resources contained in any config map data section or in secret data section and deploy like in the cluster along with the ElmCharts. So I have an example here. We have a management cluster. And inside the management cluster, we have cluster API because we use cluster API to bring up new clusters, new Kubernetes cluster, and we have Zveltoz. And then I simply post to the management cluster this cluster profile, which is selecting all the cluster with environment production. And it's simply asking to deploy caverno ElmCharts version 2.5.0. As soon as we add a label environment production on one of the cluster, Zveltoz detects that this cluster is now a match for the cluster profile. It takes all the resources ElmCharts, in this case, it's just caverno, that are listed in this cluster profile and deploys in that cluster. Likewise, if we add the same label on the second managed cluster, Zveltoz is going to do the same cycle again. It detects this cluster is a match for this cluster profile, and it deploys caverno in this other cluster. And same way you deploy, you ask Zveltoz to deploy resources like add-ons in a cluster, you can ask Zveltoz to remove resources from a cluster. So what happens if we remove like the labels environment production from the second cluster? Now this cluster is not a match for the cluster profile anymore. And so Zveltoz simply goes and removes all the add-ons that have deployed in this cluster because of this cluster profile. This is a little more complex example. When you deploy Calico in a cluster, not sure if that is still a requirement today, but at least it was a requirement in this version, you had to tell Calico which is the pod sider for that cluster. Which means that we need some sort of template because we want to create a simple cluster profile that says like install in all the clusters with label environment production, install Calico. But Calico requires like to know the pod sider, so we need some sort of templatization. If you are familiar with cluster API, when you create a cluster with cluster API, there is a cluster instance in the management cluster. And that cluster instance contains in the spec cluster network pod sider blocks. It contains the sider blocks allocated for the cluster. And so with Zveltoz you pretty much what we are saying like in this example, we are asking Zveltoz to deploy Calico. But at the time of deployment before deploy, take the information from the cluster instance in the management cluster. It takes the pod sider which is present at management cluster source. Use it to templateize this template and then deploy. And this applies to bot like Elm charts and Calico, sorry, and any type of resources which is contained in Compting Map or Secret that are referenced by Zveltoz. Some resources are fetched by default from the management cluster by Zveltoz and those are all the resources that represents a cluster. So if you are using cluster API, there is a cluster instance, there is an infrastructure provider instance, there is a QBDM control plane which contains some other information. Zveltoz is not limited to manage cluster API power cluster. Essentially if you have a GK cluster and your GK cluster is for instance reachable from your management cluster, you can register this cluster to be managed by Zveltoz. Pretty much like there's going to be like a Zveltoz cluster instance representing this cluster in the management cluster. This is also fetched by Zveltoz when deploying add-ons in the managed cluster. But we don't know all the possible use cases. So Zveltoz can also be instructed to fetch any resources which is present in the managed cluster and use those value to instantiate the add-ons before deploying it. So in this example what we are saying, we are asking to get a secret which is present in the default namespace and it's called autoscaler and identify this resource with the key autoscaler secret. So if the config map contains a template, this template simply using this autoscaler secret can fetch any value which is present like in this secret. And this is true like for any resource that is present in the management cluster. Now because you are dealing with a lot of cluster and a lot of add-ons, sometimes like it is nice to preview what the outcome of a change will be. If I'm deploying a Helm charts and I'm going to touch like a cluster profile, I would like to know what cluster is going to be affected by the change and how. So Zveltoz has this sync mode where you can set like sync mode in dry run. When the sync mode is set in dry run what it means is we are asking Zveltoz to run isn't logic. So find all the clusters that are going to match the cluster profiles, pretend that it's going to deploy all the add-ons like Helm charts or the content of the config map and secrets, but do nothing to the managed cluster. Simply let us know what the effect is going to be. So for instance like if I change the cluster profile that I listed before, no change is going to happen like to the managed cluster, but then I can ask Zveltoz if I were to commit this change, what will change? And the answer is going to be that like in this cluster here in the Zveltoz management workload managed cluster, the caverno release will change from 2.5.0 to 2.6.0. Which means like no changes really take an effect like in the managed cluster and simply previewing what the change would be if I were to commit this change. Now Zveltoz is another cluster but as other features but like the main topic of this presentation was how to use the cluster runtime state to decide what to which add-ons deploy on which clusters. And so I'll go back like to the sample that we had before. We had the management cluster with Zveltoz and we had like two managed clusters and those two managed clusters are running Kubernetes version 124.2. My, what I want to, the intent and type here is to ask Zveltoz to deploy, I want to deploy gatekeeper version 390. But I want to do, I want to deploy this version only if a cluster is running a Kubernetes version which is greater than 124.0 and less than 125.0. So here there is a second CRD which is used like to configure Zveltoz. This CRD is called a classifier and the classifier has a constraint section. In this case, the constraint section is simply based on the Kubernetes version. It's saying that if the Kubernetes version is greater than or equal to 124.0 or is less than 125.0, it's a match for this classifier. And if it's a match for this classifier, we are asking Zveltoz to add this label here, gatekeeper v3.9 on the cluster. So, so far we were managing the cluster labels ourselves. Now we don't want to do that anymore. So we want to delegate this job to Zveltoz because we want to have Zveltoz change the labels using the cluster runtime state. So in this example here, as soon as I create this classifier instance, Zveltoz is going to deploy a Zveltoz agent running in each of the managed cluster. This Zveltoz agent is going to detect that those clusters here run a running version 124.2, so they are a match for this classifier. Because they are a match for this classifier, Zveltoz is going to add a label gatekeeper v3.9 to both the clusters. Because the label, this label has been added to the cluster, Zveltoz is going to detect that this cluster is now a match for this cluster profile. And because it's a match for this cluster profile, we won't get gatekeeper v3.9 to be deployed in both the clusters. And so this is what happens. But now, let's say that I know that gatekeeper v3.9.0 works just fine if the Kubernetes version is less than 125. But if the Kubernetes version is greater or equal than 125, I want to automatically upgrade gatekeeper to version 3.10.0. So what I can do, I can create another cluster profile that says if the label is gatekeeper 3.10, deploy gatekeeper version 3.10.0. And then I can create another classifier that says if you see any cluster whose Kubernetes version is greater or equal than 125, I want you to add this label, gatekeeper v3.10. So if now I'm going to upgrade this cluster here, so from 124.2 to 124.2. Because this cluster is going to be upgraded, Svelto's agent automatically detects that this cluster is not a match for the old classifier, but is a match for the new classifier. Because it is a match for the new classifier, the label is supposed to change. Now the label is not supposed to be gatekeeper 3.9, but it's supposed to be gatekeeper v3.10. So Svelto's changed the label on this cluster because automatically, because the label on the cluster has been changed. Now this cluster becomes a match for this cluster profile, and it's not a match for the old cluster profile anymore. And because it's a match for the new cluster profile, we want to deploy gatekeeper v3.10. And so gatekeeper gets automatically upgraded, and there is no configuration that is done by us. So we simply express the intent, and then the system takes care of deploying everything. There are a bunch of other features which are listed on those two slides. Essentially the main one, some of those like already touched, the main one I want to mention why we deploy like another Kubernetes add-ons manager. It's because we wanted to use the runtime state of a cluster to decide which add-ons to deploy. But also we need like an event-driven framework to deploy add-ons in the spawns of events happening in a cluster. And we want to have like a system to a way to say, even a cross-cluster configuration, a way to say is like if you see an event in this cluster, this is the set of add-ons that you want to deploy like in this other cluster. It has pretty much configuration drift detection and other features, but like I think I'm almost done with my time. So I want to leave a few minutes if there is any question. Okay, no question. Thank you so much. I appreciate you being here. Thanks a lot.