 Hello everybody, welcome to time to migrate to application set session. My name is Vicky. I'm DevOps engineer for several years. I started as a Java developer, but then I learned about Github's automation and cloud. Today, I'm part of cloud platform team at Bank of Berlin. In this session, we'll start with intro about our city. We'll talk about application set and cluster generator. We'll describe the steps we made for migrating to application set, the consideration, the implementation, and finally, the summary. In Bank of Berlin, we'll use Github suite AlgoCV. AlgoCV is a declarative Github's continuous delivery tool for Kubernetes. AlgoCV is responsible for monitoring our running applications and comparing their life state to their desired state. We use it to manage and deploy applications to multiple clusters. The desired states are specified as a manifest and custom resources in our Github repositories. One of the main custom resources is the application CR, which is Kubernetes resource object representing a deployed application instance. AlgoCV application is an atomic unit that represents a collection of Kubernetes native manifests. As the none of our cluster rule, we found ourselves configuring a great number of application CRs that were almost identical, except for their target cluster or their environment Github. Our application was spread in several Github repositories, which made the maintenance even more difficult. Realize this, automation was necessary for efficient management of our applications. Maintaining applications separate across multiple Github repositories is very difficult, so we searched for a way to automate our application creation and modification across multiple clusters. The solution was just in front of us. AlgoCV application set. Starting with AlgoCV version 203, the application set controller is part of the AlgoCV installation, so we don't need to install it separately. It gave us the way to automate the creation and the modification of application CR across many clusters. With one AlgoCV application set manifest, we can target many applications to many Kubernetes clusters. It's a sort of an applications factory, so how it works. When we create, update or delete an application set resource, the application set controller responds by creating, updating or deleting one or more corresponding AlgoCV application resources. AlgoCV itself continue to be responsible for the actual deployment of the generate child application resources, such as deployment, service and configments. The application set customer resource CR, as you can see here, is very similar to the application customer resource. But as a template section, which contains all fields defined in the spec part of the application CR, change is made to the application set template field will automatically be applied to every generated application of the application set. The application set CR is based on the generator concept. Generators are responsible for generating parameters that will be rendered in the template section of the application set CR. Parameters are key value pairs that are rendered into the template section. The application set offers some generators and ability to combine generators. I will focus on the most used generator in our migration process, the cluster generator. In AlgoCV, managed clusters are stored within secrets in the AlgoCV workspace. The application set controller uses those secrets to generate parameters to identify and target the generated application to the target clusters. Suppose we have two cluster registers with AlgoCV, cluster01 and cluster02. And we define the application set to the product cross plan with cluster generator. The controller will create two cross plan applications, one in cluster01 and the second in cluster02. We also use label selector which give us the ability to narrow the scope of the targeted cluster to only those who have the matching spatial label. We use this ability to separate the application configuration according to the cluster environments. The example here, app will be created only in cluster that contain the label in vdev in their secret. Won't create in cluster that don't have this label. We use label selector to automatically generate specific product application or a set of product application in every new cluster according to its environment label. Just f2a, the relevant environment label in the cluster secret. It's also possible to use match expression for more powerful selector and even combine both. Match expression gives the ability to select the target clusters according to their expression value. For example, for the expression above, it's any cluster that has in its server one of the various HTTPS address one or HTTPS address two. The app will automatically be deployed. Valid operators can be in, not in, exist and does not exist. And now for the second part Alina will continue with the presentation. Hello. And before I continue with the presentation, let me introduce myself. My name is Alina. I used to be a C-shop programmer, a teacher and now after finishing up school, I became a DevOps engineer. Today I'm part of Cloud Platform team at Bank of Pauline. In my free time, I enjoy petting my cat and learning Japanese. Steps for migration. Identify commonalities among applications. We should find what the same between versions, consider using overlays and customization. Plan and strategize migration. We should think what products are better migrated first and on which clusters. Evaluate and choose the appropriate application set. We should check from the available upsets and choose one that is suitable for your needs like which is cluster generation. Considerations. Ensuring backward compatibility. We need to make sure our product works in all versions we need no matter how old or new. Testing across different environments. We should also test that our product works correctly in all relevant environments. Simple case test. Let imagine we have a repo where each brand represents in the cluster and we have the following products. Product 1, Product 2, Product 3, Product 4 and Helm Product 1. Product 1 has a different configuration by cluster. Product 2 has a different configuration by environment. Product 3 has a different configuration by environment and some clusters have a different configuration. Product 4 is only used by some clusters. Helm Product 1 has a different values file for each cluster. Let's begin with our first example, product by label. In this example we are using a label to determine in which cluster this product will be deployed. All clusters will have the same version of the product deployed. As you can see every cluster secret that has this label product for on will have the same version of the product. And into our second example, Helm with values file per cluster name. In this example we are using a label to determine in which cluster this product will be deployed. Since this is a Helm product we are using the template part to adjust the values file name to be different and adjusted to each different clusters. We can see the template part is highlighted. And every cluster that has the label will be deployed with its specific values files. And the third example overlay per cluster name. In this example we are using a label to determine in which cluster this product will be deployed. Each cluster will have a different version adjusted to the cluster. We can see in the template part in the path part we are adjusting the path by the name of the cluster. Once again it's highlighted so we can see we are using here the template to get the cluster name so each cluster will have a different version per cluster. And into another example overlay per cluster environment. In this example once again we are using a label to determine in which clusters this product will be deployed. Every cluster will get a different configuration depending on its environment label. We actually added a variable with a different value to each environment option and we are using it in the template part in the path value. We can see in the highlighted part the use of values dot overlay path the variable we declared in the cluster generator part. And into our last and most complicated example overlay per cluster environment or name. In this example we are using three labels to determine in which clusters this product will be deployed and in which version. We are using the end label and the in existence of the label exclude product 3 with the value on. If the cluster secret fulfills these two conditions the product will be deployed in the chosen end version. For the last option we are using the existence of the exclude product free label with on value and also the product free by name label. And then the overlay will be chosen by the cluster name. Since we can only access the cluster secret values such as the cluster name we must build the template inside the clusters generator part. This example is relatively rare. We only use it in about 20% of our upsets. Although Upset work great by himself Monorepo can only make it better. One place to store them all. Monorepo simplified managing product version finding all product in one place no more need to search where a specific product version is located. It work well with Upset. As I said before just using Upset it's good but Monorepo makes it easier and more comfortable to use Upset. Normal searching where that specific version is located. As I said before it's a great place to manage all product version everything is organized everything is one place in one branch. Benefits of application set. Simple maintenance of Argo CD apps across clusters since the Upset manages them. One Argo CD Upset actually manage a lot of Argo apps. Reduce redundancy and improve consistency between all Argo CD apps. No more copies and copies of probably the same Argo app. One place to manage them all. If we need to change something it's all in one place. No need to change many copies of almost the same Argo app. Limitations and inconveniences. Changes in an Upset can affect many clusters at one. Users need to be careful with changes. As we all know with great power counts great responsibility. Since we are mostly using label to determine in which clusters the apps will run cluster secrets begin to have too many labels. Think if you're going to have many product and each one will have its own label you're going to have a really big cluster secret in the end. By using Upset we are losing flexibility for ease of use. Do you remember our last example? Future considerations. I would like to think of ways to reduce the amount of labels being used inside the cluster secret. Maybe decide on some common labels to use for our products. Consider using one Upset to manage more than one product. We actually had the thought in the beginning but it didn't work quite well but now maybe we are more mature to try to actually make one Upset to manage more than one product. Integrate the use of Go template and sprig function library inside our Upset. The Go template were in existence even in the start of application set and we actually tried to use them but it also didn't work quite well but maybe also now it's the time to use them and the sprig function library actually gives us the ability to even actually code a bit inside our application set to use conditions and so on. Conclusion. As far as managing Argo apps using Upset it's a correct path in minimizing duplication and easing the management of Argo apps in many different clusters. Me, Vicky and my cat would like to thank you all for listening to our lecture. Hopefully this lecture helped you understand a bit more about application set and maybe even give it a chance. Thank you.