 Welcome everyone. My name is Jose Gato. I work in Red Hat as a software engineer and I'm going to talk in this presentation about going to ZeroTouch Lifecycle Management for Delco Edge Deployments for the ones that attended to the previous talk. We can see that this is complementary because here we are going to see some more details about how we do the cycle management in the Delco deployments. This presentation is so in part of the work that we do inside the Delco integration team where we again help two of our Delco partners to run the different workloads and deployment in the open SIF platform. In this case, more in concrete, what we are going to see is how we create a whole management platform for not only deploying but also for doing the whole life cycle management and also following the ZeroTouch provisioning ideas. About the agenda, the different topics that we are going to see. I will start with some background and context about why Delco needs this kind of ZeroTouch provisioning and life cycle management and what we are proposing. The proposal is to build a management cluster with ACM and CTP GitOps and how we build this management cluster and the different tools that compose it. And then we are going to see two GitOps, CTP GitOps plugins that allow us to do the deployments and configuration using ZeroTouch provisioning and GitOps. Finally, I will do a demo. So the background, why? Because, well, our Delco partners and customers have to scale, especially when we go to the Farage. In the Farage is when we are closer to these radio units and antennas, and you can imagine that if you need one cluster per each antenna, we are talking in numbers about 100,000. It's not acceptable to manage all these clusters individually, especially when mostly most of them are going to be the same hardware, the same workloads or very similar. We are using a common platform to manage all these sites from the cloud up to the Farage and not only to manage the deployment, but also to do the configuration, the monitoring and the different upgrades, so the whole life cycle management. Because we are going to use a GitOps methodology to do everything, we will have a Git repository where we will store our different manifest for our infrastructure and for the configuration. And we can say that the Git repo is going to be the unique point of truth for all your infrastructure. Also, with this Git repository, it's easy to scale and easy to repeat the things because you can just copy some files and do some push and commit. With this idea, you do a commit, you push it, and automatically, zero touch, your infrastructure is deployed and it's configured. After this push, there are a lot of components, services, operators, interactions that are going to happen behind in order to orchestrate everything without your intervention. Okay, so let's see more in detail how to build this management cluster. This is more in detail that we have seen in the previous presentation. Again, we have the architecture of one half or one management cluster that is going to manage other clusters. The management cluster is actually an open-siv cluster with a base of CoreOS and open-siv, together with Red Hat Advanced Cluster Management, ACM, that is going to provide the different services, functionalities, etc., to do your cluster management, together with the assisted service installer that is the newer installer for open-siv cluster. Also, we will use Argo CD to do the synchronization between your Git repository, the objects that you have there, into the ACM. Also, this Argo CD is specially configured because it contains two new CTP GitOps plugins. And finally, the topology, our lifecycle manager, that is a new operator that allows you to do the lifecycle management, but more in concrete about the configuration that you need to apply, and we will see later why we need that. Okay, this is very quickly a screen shot of Argo CD, where we can see one deployment of one of the clusters. The information is stored in Git, Argo CD will synchronize this, and ACM and assisted installer and other components we will do the installation. This is the screen shot of ACM. That is okay. It's the central platform tool that allows you to do all the management of your different cluster and the different versions. More details about how we implemented that, okay? Of course, we start with Git repository. You can have a Git repository with whatever system that you want. It can be GitLab, GitLab, or just a simple Git repository. There, in the Git repository, what we are going to do is to create our manifest. These manifest are two new manifest that are provided by the CTP GitOps plugins that previously we have installed inside Argo CD. And these two templates are called Site Config and PgT Policy Gen Template. Site Config to define your infrastructure and Policy Gen Template to define your configurations. So the files go to Argo CD, and Argo CD takes this into ACM where the assisted installer is leaving. It will take all the information and will start the installation. In parallel, the topology of our life cycle management will take the configuration, and it will allow you to decide how do you want to apply all the different configurations. So what is the scope of the Serotoge provisioning? We have the day zero. In the day zero, we are going to define our infrastructure with one of these CTP GitOps plugins. So you have this new template or custom resource definition that is called Site Config where you will define the different classes that you want to have. Then when the clusters are deployed, we move to day one, and some operators are going to be installed, configured, and some tuning is done to make this cluster a special cluster to later run Telco workloads or CNPs. Some of these configurations are coming by default in all these clusters when we are using CTP. Some others are customization that you will add as policy gen template that also are going to be applied in day one. Why? Because the point in between day one and day two is important because it is what said that this cluster is ready for running CNF workloads. So when everything is done, this cluster is labeled as done, and the day two, the different Telcos can deploy their workloads. And of course, after that, you can continue doing more day two operations with new upgrades or new configurations. And these are the two new templates that are provided by the CTP GitOps tools that are, well, the plugins are installed in Argo CD and provide you with these new templates or custom resource definition. One to define your infrastructure, the other to define configuration. So let's go into detail about these new plugins and the new templates. The first one is the site config that allows you to define your infrastructure and allows to define your infrastructure. And this is important with an unique custom resource because before having this plugin, you could do something similar in ACM. But in ACM, to make a deployment, you have to create a lot of different manifest that make the thing a little bit more complicated here. All these manifest are rubbed into a unique custom resource that only contains the fields that are needed for this telco deployment. So how does it work? Okay, you are in day zero, you have your Git repo, you define your site configs, you push the changes. These go to the Git repo, these go through Argo CD with these plugins that have been implemented as customized plugins. And this site config is going to be transformed into different resources that are the ones that are really needed by the ACM. This is a site config example, you define the name, you define the open-source cluster version, sunnet working for the cluster, sunnet working for the node, and maybe the disks that are going to be used for the installation. Now we go to the policy and template. The policy and template helps you to do the configuration. It's like a set of pre-configurations for concrete tasks that are very related to telco. And, okay, very similar, you are in day one or day two, you have your policy and templates that are applied through Argo CD with this plugin, and the plugin converts this into something that ACM and teloperator understand that are actually policies. This is an example of a policy and template. And the idea of these, let's say, helpers is that they help you to, for example, to deploy one operator, one operator requires different manifests, a lot of lines, and here it's easier to use, because you don't need to know how to do this manifest. You only say that you want to install the storage operator. Maybe also you customize a little about the name that you want to do to the storage class, or, for example, this helper that allows you to set which offensive version you want to have, and you only have to decide the decided version. And to just ending with the different, with the new custom resources, we have a third one that is clustered group upgrade, and this is provided by the top of your lifecycle operator. So it's not 100% true that for a configuration, once that you push this to the git, it's automatically applied to the cluster. Why? Because if we think again that we have 20,000 antennas, maybe you don't want to do one change and to apply this immediately to the 2000 antennas. Maybe you want to do it in different groups. You will upgrade to three antennas at the same time. There were one question before about what happened if something goes wrong. Maybe you want to upgrade only one antenna per region or things like that. Also, because what happens if the policy fails in the antenna 100? Do you want to continue upgrading the other ones, or do you prefer to stop here and to see what happened? What happened if you have long bad width, maybe you want to use one feature here that is attached to download everything before starting the configuration, another different feature that I provided by this new operator. Also, it's very easy to use because you just decide which cluster you want to apply one policy and template and how many clusters you want to upgrade at the same time. So now we will go for the demo. In the first, I don't know how I'm going about time. It's okay. So, okay, what we are going to start is one existing cluster in our infrastructure with three masters. So the site config already exists in our Git repository and we are going to do is to scale it up to add it to more workers. In order to do that, we are going to take this worker definition in the site config. We use worker zero, the disk, the networking, etc. And we will add this to the existing site config to have the final cluster with five nodes. Let me go to my local version of the video that I think is going to see better. Okay, so here in the left side I have my Git repository. The customization Jamel file contains the different site configs that you have in your infrastructure. In this case, I only have one. And this one contains the information from these three masters, okay? We have three masters with its networking, network configuration, etc. What we are going to do now is to add two new workers. The definition is going to be pretty similar, okay? For both of them. And following the GitOps methodology, this site config has been modified. We make a commit. We push it. And Argo CD and the other components will do the next step for you without intervention. Here now it will start the installation. As you know, there are two new nodes. So in the bottom right here we have only three masters. And right up here are some objects that are provided by ACM and other operators that are pretty important. These three ones that are here are agents that were in charge of installing the previous masters. Now by the moment we are focused here in this resort that is a PMH that is going to do the provisioning of the two workers that are, as usual, Intel Core bare metal servers. So what it's going to do is to switch off the servers. It's going to mode virtual media with an installation ISO. The installation ISO is going to happen when you say that here is provisioned. While that is provisioning, the server is booting with this life ISO in some seconds in the video, but some minutes in the reality. The new agents will appear here as part of this installation ISO. Here they are. You have these two worker agents running on the new servers that will start communicating with ACM and Assistant installer to get the instructions to do the installation. So when the agent appears, we can move to watch in other resources in order to see how this installation is progressing. So you only did the git push and git commit and everything was automatically orchestrated. We continue. Well, this is just the last screenshot because the last frame goes very fast, but okay we can see in the last frame we can see how the now cluster have the two new workers ready and while this installation reaches to 100%. So the second demo, I think I'm more than okay. Okay, so now what we are doing is to upgrade, no to do an upgrade of the different clusters versions. Okay, we have clusters that are running 4.12.3, this 4.12.1 and we are going to upgrade them. The interesting thing that we are going to see here is that we are going to do it in groups. We only are going to upgrade or to apply the configuration in groups of two clusters at the same time. So in the cluster there is already one policy that keeps the clusters in the version 4.12.8 and well, or maybe we change it this policy in order to make the cluster to be in 4.12.8. We push this change it, this creates some, the plugin will create ACM policies that are needed for doing the configuration and then we will have to create this new cluster group of great resource that is going to say, this policy that is called upgrade 4.12, I want to apply it to this list of clusters and I want to upgrade only two clusters at the same time. So again for the video I will go to my local copy. Can you read correctly the text in the video less, okay? Can not be bigger? Well I can make zoom in the desktop if needed. I think it's more or less okay. We have, as we have seen this policy that in this case for this demo is saying that now we have all the clusters in 4.12.10. We are going to modify this policy to say that we want to go to 4.12.11 for example. And we are going to push and commit this, I think, wait, sorry. So okay, we go to our GitOps, we push the changes and everything will start working. Second, we have to make to say to the TALM operator to apply this configuration. We have seen in the previous sample, we take the 4 cluster, we say to apply this policy and in groups of two at the same time. So one that we had the CU, we apply, I think it's wrong with the video. Demo effect also affect to the videos, which is nice. Okay, so we change the version, we commit and we push the changes. So here we go. So here in this part we have the 4 cluster that we have selected here, SNO 5, 6, 7, and the L8K. We see that now all of them are in 4.12.10 and we want to upgrade them, okay? Because we have saved to only upgrade to cluster at the same time. Now when the installation... The video is running but it does not refresh here. Okay, I don't know what happens with the video. But okay, I can pause it here and we can see that the installation is toilet. It is moving for 4.12.10 to 11, but it's only happening in the two first clusters. The video wants to work, but I don't know why not. Okay, now in this part of the video we can see how the first two clusters are already in 4.12.11, that is the decided status and have started to upgrade the other two clusters. Sorry, because I don't know why women might pause the video. Then it does not... Okay, so very quickly this is the same from a graphical user interface where you see that only SNL 6 and 5 are upgraded at the same time. Okay, so well, conclusions, just finishing. We have Therotouch provisioning and GitOps that seems to work very well together for the objectives that we want to have with these partners. Red Hat ECM can manage all the clusters from the cloud to the far edge and well, it seems that it's a pretty good idea to have your infrastructure defined in a declarative way in your Git repository and allows you to have scalability and replicability. So now I have finished my presentation. We can move to questions in case that you have. Okay, the question is about if this management cluster is operated by us or by the partner. Okay, usually in the cases that we are working, they are running their own management cluster to manage their spoke clusters. The question is how many clusters you can manage with a management cluster. Sorry, because the answer is very... But it depends on how is your hub cluster, how big is your hub cluster. But in the previous presentation, we have seen that one hub cluster that was a compact cluster. It was a compact cluster in bare metal with hundreds of CPUs. Okay, it's not a small one. It can manage, for example, and deploy more than 3,000 single nodes offensive. More questions? Yeah, sure. It can be whatever. The question is if when we are doing the Git push is to a Git repository if it is inside the management cluster or not. In principle, the management cluster is not composed by a Git repository. It could be. This is maybe up to you. In principle, this can be whatever you want. But by default, ACM or Argo CD don't come with a Git repository. When you configure Argo CD to work with these CTP GitHub tools, you have to configure the applications to say where is your Git repository, the Git repository that you want to have in to sync. By other application, for example, because this is Argo CD... Okay, it's other application. But I didn't understand currently the question. Okay, the question is if they are going to... If our customers are using Git or other tool to prepare these files. In principle, I'm not sure if Argo CD can synchronize with other application. What we are doing is always we are using Git repositories. How the manifest reach to the Git repository, this is true that depends on different partners or customers. Do you mean physically? Okay, so part of the configuration... Okay, great. Part of the configuration that is in the site, the question story is about how you define where the cluster, the managed cluster is going to be physically. And the rest of the clusters. So for the spoke clusters, it's because in the site config is where you say, in this case, it's most of the time we are working with bare metal. So you have to know what this bare metal already exists, of course, and you have the bare metal interface with the IP user and password that you define in the site config. So of course the management cluster have to have connectivity to these bare metal servers. And in the site config is where you define where these are in the way of what is the IP of this interface in the bare metal that do the installation. I think these are different methodologies. But the question is how we solve that because this edge cluster could have close connection to the hub cluster from time to time. Okay, so what it will happen there is that if there are not connectivity between the hub and the spoke, in ACM you will detect that because there is one agent running there that is connecting to that. If this happens, the spoke cluster is autonomous. So it will continue working as it is and the workloads will be running no matter what happens. If you don't recover this connection, you will lose the possibility of doing day two operation. Okay, that's okay. Okay, no more questions, thank you.