 Good morning, good afternoon. Welcome to Cube on the Edge. My name is Ian Jolliff, and I'm here with my colleague Yolanda, and we're really excited to be talking to you today about a GitOps approach to zero-touch provisioning for the 5G VRan Edge. Well, let's maybe start off with introducing each other. Maybe you go first, Yolanda, and then I'll do the same. Hello, I'm Yolanda Robla. I'm from Spain. I've been working in Red Hat since five years, and I've always been dedicated to open source and cloud. In the latest times, I've been working in Kubernetes and OpenShift, like, specialized in Telcos, and well, I'm working in, like, using OpenShift for 5G and RAM deployment. Thanks, Yolanda. Again, my name is Ian Jolliff. I recently joined Red Hat. I'm located in Ottawa, Canada. I've been working in open source projects for the last 10 years, and most of my career in telecom. I'm really interested in virtualization, OpenStack, Kubernetes, and other open source projects. So, let's dive in. Today, what we'll discuss is some of the 5G RAN topologies and challenges, really trying to take those concepts and apply them to infrastructure as code at the Edge, go through some declarative frameworks and workflows for these deployments, and then wrap with a demo. So, one of the technologies that we're leveraging is OpenShift. It's really a distribution of Kubernetes that comes with things like registry and many other very important components, like networking overlays. So, let's start by looking at some of the challenging topologies at the edge of the network. Let's start at the bottom and work our way up. So, starting with a three-node cluster where we have supervisor functions and worker functions on a set of three nodes. This would probably be located in a regional data center, potentially running CU workloads. Up next, we've got where the supervisor nodes are located in the regional data center again, but then leveraging remote worker nodes at the edge of the network. And this is the configuration that we're leveraging in our demo. And the third topology is really where supervisor and worker nodes are both at the edge of the network. And this is really our single-node Edge server configuration. This really helps with low bandwidth or completely remote sites. And that's some work that's going on right now. Another key technology that we're leveraging is open cluster manager. And it really allows you to manage many clusters, as if you were operating in the core of the network, helps with life cycle management, policy governance, and advanced life cycle management concepts, like deployment and updates and upgrades. So, some of the key considerations at the edge of the network are really about latency from the DU to the radio unit. Extremely tight tolerances are required for latency in this modality. And so, you really need to make sure you're getting the maximum performance out of the servers that are located at the edge of the network. Things like real-time kernel, multis, and SROV are key components that we need to think of and deploy at these edge sites. So, we're really providing a common automated infrastructure that allows you to manage all these nodes, as if they were in the data center. We've also looked at how to deploy with small ISO, so it's easy to bring up and deploy your servers at the edge of the network. So, it's really all about making sure that these servers that are in very disparate geographies can be configured and managed as if they were part of a normal Kubernetes cluster. So, let's shift the focus of the conversation today to GitOps. And this is the deployment topology that we'll be following. And I thought we'd start with an industry definition for GitOps. I won't read it all, but the keywords that really resonated for me are really it's a set of practices. Traditionally focused more on application deployment, but we're using GitOps in the context of infrastructure deployment. The other major concept and word for me that jumped out was really declarative. So, we'll be using YAML in a set of well-known defined patterns to provide a declarative framework for all the infrastructure that we need to deploy. And it'll all be stored in Git and be able to be leveraged by the open cluster manager technologies that we've been talking about earlier. So, how do you break this very complex ran topology down into a GitOps and have it all defined as code? Well, it really starts at a high level from a site plan that a Telcom operator would have for their mobile network. We're going to break that down into a set of manifests that can be stored in Git. And then we're going to deploy with ZeroTouch using the open cluster management technology. So, it's really important to understand some of the players in this workflow and some of the key personas that we identified were in the Telcom operator side, the planning organization who's responsible for network design, vendor selection and capital planning. And then on to the installation and deployment teams where they're building out the sites, hiring contractors for the installations, making sure power and physical plant infrastructure is available to these new and interesting servers that are now living further at the edge than they ever have. Then lastly, it's really the maintenance and operations team that's really responsible for the day-to-day operations, updates and upgrades and the deployments. So, really, those are the three personas on the Telcom side. So, let's look at the overall ecosystem that is enabling this new paradigm for getting compute to the edge. We have the system integrator really pulls together a set of partners and the overall ecosystem, integrates orchestration and brings together the lower layers of the stack together with the application layers. And they can help the operator consume the site plan and represent the infrastructure as code. The infrastructure provider persona is really the people providing the platform technology such as OpenShift and OCM, Open Cluster Manager, and also deploying the high performance profiles to make sure that the workloads can operate these strict VRAN workloads. And then lastly, we have the application provider that's really providing a 5G RAN application or a 5G core application. So, a very diverse ecosystem is now coming to play to enable these new 5G RAN applications at the edge. So, that I'll hand it over to you, Yolanda. Okay. So, I will explain how the workflow for this GitOps approach is working. Our source of truth is just a set of Git repos where we are containing what we call a site planning. And a site planning is just a definition, a set of YAML files that are containing all the different settings for the clusters that we want to deploy. So, we will include things related to the cluster like the cluster name, domain, like IP ranges. And we also have definitions for the hardware, like for the for the bare metal host. We can just provide the details on the VMC, starting network definitions as well. So, we have this data with the site plan. And then we are combining that with other manifest that are defining the Kubernetes manifest operators that we want to deploy in all the clusters. Like, we give the cluster a set of rows. We could tell which manifest we went to use. Everything with that is combined with customized and we are producing the final output. Like, it will be a YAML definition of all the settings and all the things that needs to be applied in the clusters. When we have this YAML, we are just applying, passing that to OCM, to Advanced Cluster Management. This is consuming it and as soon as the clusters are joining and rolling into ACM, it will be just reapplying all the different operators, the different configurations that we have just told in our site plan. So, we end with clusters, finally configured and deployed with our settings. Okay. So, I'm going to explain a bit the operators and all the layers that we are doing for our cluster provisioning and definition. So, we start with the base components. We assume that we are starting with servers that we don't have any operating system installed that just are there from the factory. So, we start by provisioning them with the operating system that is CoreOS. On top of that, we're installing the OpenShift Container Platform. So, when that is completed, the nodes, the worker node, will join into an existing cluster. Once it happens, we start applying the 5G RAM profile that is composed by a set of operators. So, we start with machine configs that are just for enabling like system definitions or low-level settings at system level, like we enable NTP, SCTP, like that. Second, as the 5G is very important to have good performance, we are using the performance add-on operator that is taking care of that. So, it will start real-time kernel on the nodes that need it. It will just define the number of CPUs, the cost we reserve, the number of huge pages as well, depending on our workloads. After that, we also need to just interact with low-level network functions. So, we want to consume directly the network functions from the card. So, we are using the salo operator. This will allow us to just use the physical function and the virtual functions for every NIC. And also, on the DUs, where we need very good time synchronization. So, we are using PTP, that is precision time protocol. So, open cluster management will be installing that as well. Okay. So, finally, I'm going to explain a bit how the deployment process looks like. So, we start as we were telling with everything in Git, like we have the site plan in Git, what we call the regional data center. We have a cluster with open cluster management installed there. And then, we can apply our GTP. And it will be used to deploy different types of clusters. So, we can just deploy like clusters with a control plane and remote worker nodes. So, that will be one use case. Another use case will be single node, where we can just use it for the use or it's just a server with masters and workers at the same host. And also, we can have a CU pool that we are using three masters that have also the roles of workers. So, this GTP is able to just deploy clusters with these different topologies depending on our site definition. That's great, Yolanda. Thanks very much for walking us through that. I really like how we can start with the infrastructure site plan, apply the customized overlays, then wrap that overall picture in policies that can be then leveraged by the open cluster manager. So, let's dive in and see the demo that we have here. Hello. So, let's start with the demo. As I was explaining before, everything starts with a site plan. A site plan is just a set of YAML files that are stored into a GTP and that contains the information needed for deploying a massive number of clusters. You can see here what I'm talking about for each cluster. We have the settings, the name of the cluster, the domain, the version, some network settings as well as the English and the API bit. Well, then we can define the control plane. And just for the control plane here, where we explain the number of masters, the MAC addresses of the masters as well. And then we define the workers as well. Workers are the remote worker nodes that are going to join after the control plane is deployed. So, for each worker, we define the VMC details, like the address, username, password. So, with that, everything is translated to just some massive inventories. And from there, we start the deployment of the clusters. Okay. So, next thing, I want to show how to configure static networking for the workers. What you need to provide is just a file, a YAML file that will contain all the network configuration in NMS state format for all your worker nodes. So, you can see here that we have a list of interfaces. Each interface is belonging to a worker node. As it's consuming NMS state, you are free to define the kind of network that you want, like BLAN, bonding. So, next thing is creating a cluster. You can see that simply we're just going to run the next stack in the playbook. It will just create a cluster using the OpenShift assisted service. Okay. You can see here, this is the assisted installer web user interface. And you can see that we have just created a new cluster with the settings that I pass it on my memory. It is using 4.7. You can see it matching the cluster name, the cluster domain. Okay. So, the step finally just finished. And you can see that we have three nodes just enrolled into assisted installer. And with once we have just three nodes, we can start with installation of the OpenShift cluster. It will just set up some settings and just the order of deploying the cluster. And it will just wait. Okay. After some time, the cluster has been installed. You can see here that you have the three masters shown as installed with all the details. And now it's time to just add the worker nodes. I'm starting to show you how the server is reacting. It's okay. So, you can see that it has been powered on. And we're telling them instructions to boot from virtual media. It starts to use the, to mount the image that it has downloaded. And it will just complete the provisioning step now. Okay. You can see that the nodes start to report up to assisted installer. We will have two worker nodes that we'll be joining. As you have seen, the nodes are already enrolled in assisted installer. So, next thing is to deploy the cluster. So, you have seen that the workers are starting the installation. Okay. So, you have seen that finally the worker nodes are enrolled into the cluster. So, we have a working Kubernetes cluster with master and to remote worker nodes. So, we'll have this repo that is called GTP ACM manifest. And it contains all the objects that are going to be applied into advanced cluster management. Everything is, here is working in a GitHub way. And everything is controlled by just Kubernetes manifest and CRDs. Here is how our basic layer is composed. So, we have common. Common is just the base, base manifest that are going to be applied to all clusters. Then we have profile that we are going to use it to configure clusters depending on the function they have. Like we can have an additional layer that is hardware type, that we can configure clusters depending on the type of hardware. And finally, we have the sites that is specific configuration at site level. Everything in ACM is working via subscriptions. A subscription is just an object that is telling ACM that it needs to listen to a specific repository or to a specific folder. Okay. So, here, for example, you can see that we have the subscriptions for common base. Everything is composed mostly of namespace, channels, subscriptions and applications. And you can see that the subscriptions are just pointing to another repo. Our RAM manifest that is actually having the source of truth of all the configurations. And we have the same. You can see that we are matching the structures that we had in ACM subscriptions with the CTP RAM manifest. Okay. You can see here that everything is based on a policy. A policy is just an object definition of ACM that is informing how it needs to configure which object that it needs to create into the cluster. And it is just simply creating the operator. Okay. I'm telling, I want to create this object in all clusters. And I'm just going to create an object that installs as a operator. After a policy, there is also what it calls placement bindings and placement rules. So, the placement rule allows you to define to where do you want to apply this policy. So, in our case, we are applying the policy in all types of clusters. In this case, for example, we are just applying all the configurations to all the clusters that are due. And then we can have the site policies as well that are specific things that needs to be configured at site level. Like, for example, bilans, name of nix. So, things that needs to be configured at site level, for example, are the nix electors. Like, you need to exactly manage the name of the interface here. So, it needs to be at site level. And everything in ACM is just configured into this, in this way. This is just all ACM installation. And on this installation, I have applied all the manifest that you have shown in the ZTP ACM manifest repo. You remember that I have been showing that we have different layers. And we have created subscriptions based on that. So, you can see here the run subscription, hardware type, the profile, and the sites. When a subscription is applied, it's actually creating all the objects that we are pointing to into the folder of the Git repo. So, you can see here the final outcome of that. We have the common run subscription that is the main object that we created into ACM. This was pointing to the ZTP run manifest matching folder that contained some policies, placement bindings, and placement rules. So, you can see here how ACM is just applying all the different objects. What I'm showing now is how a cluster is enrolling into ACM, this is progressing. You can also see that the cluster has some labels on this profile view. It means that it will just apply all the workloads that are matching with the view profile, and this has been defined for the placement rules that I created. And you can see that everything is installed properly. ACM is monitoring the status of the cluster. It shows the version, name of the cluster, the profile, the number of nodes. You can see that it doesn't have any application. And this is anything you can see that there are policies that have not yet been applied. When a cluster joins ACM, ACM automatically recognizes that and tries to start reconciling all the changes that are needed to configure the cluster automatically. This is, you remember that this is managed via policies. So, we have several policies that are defining the kind of objects manifest that needs to be applied on the cluster. And then ACM is matching those policies with the rules that we defined and decides where to apply. This is prone to change, and we are just working on a final version, where everything will be integrated. So, no need to run manual ansible playbooks. Everything will just be defined on the grid repo, including the cluster definition, worker, hardware, the hardware settings for all the clusters. And the procedure will be just automatically writing all the information from it and applying the procedure that you have seen here with Assistant Installer and ACM to configure all the needed clusters. Thanks, Yolanda. That was a great demo. Really enjoyed that great job. Really had a great time doing this with you, and let's open it up to questions. Thanks very much, everybody. Thank you.