 All right. Hi, everyone. Today we're going to deliver a talk, a lightning talk on how we leverage the talks to deliver managed services on Kubernetes reliably at Red Hat. But before that, let's introduce ourselves. So about me, hi, hi folks. I'm Yash Vardhan and I'm currently working as a site reliability engineer at Red Hat, working on all the things around open shifting Kubernetes and I love to spend my time fiddling with Kubernetes, and anything around distributed systems. And to shed off some programmatic monotony, I love to spend my free time outdoors, running long distances. So that's pretty much about me. Over to Ashish. Hello, everyone. I'm Ashish. I'm also a site reliability engineer at Red Hat. Like, yes, I also love Kubernetes. And in my free time, I love learning new programming languages. When I'm not working, I love to go out hiking in the mountains. So enough about us. Let's talk about manage services at Red Hat. So the core of manage services at Red Hat revolves around our platform, which is called the OpenShift Dedicated. So OpenShift Dedicated is vanilla Kubernetes, plus a bunch of operators and webbooks, which makes this version of Kubernetes called as OpenShift. Right. So now the unique USB of this platform is that customers can install additional software on top of their clusters. So we call these additional software add-ons. And we, the MTSRI team is on the hook for delivering and supporting these add-ons. Under the hood, these add-ons are nothing but an opinionated way of packaging Kubernetes controllers. Yeah. Managing the add-ons really. So as an SRI team responsible for add-ons, we use kitops to smoothly deliver our add-ons end-to-end. That is from the product developers right to the customers. So how do we do that? We have a couple of repositories, which include the manage tenance bundle repository and the manage tenance repository. Together, these two repositories store the entire state of all the add-ons. Talking about the manage tenance bundle repository, this repository houses all the artifacts of all the add-ons. So what I mean by artifacts is that the deployment of the add-ons operator, the CRDs that come with the operator, the stateful sets and other Kubernetes resources. It can also have dependencies, which the add-on depends on this operator. It can also specify that here. One other thing about this repository is that each add-on artifact is versioned in that a particular version of the add-on has its own directory and its artifacts sit inside that directory. Now talking about the manage tenance repository, this repository houses all the configuration and metadata of an add-on. So what do I mean by metadata? Things like the name of the add-on, the image of the add-on, the namespace where the add-on has to be installed, and also it can have constraints, the runtime constraints like this add-on can only be installed on clusters where this amount of RAM is present and things like that. Finally, we have pipelines which do the plumbing and communication between these two repositories. Over to the next slide. Yeah. Now let's take a look at the UI, which the customer sees when they have to go and install an add-on in the cluster. Thanks, Ashish. So this screenshot which folks are seeing in front of you, this is how a cluster looks to a customer when they install an OpenShift dedicated cluster on the Red Hat Cloud. So this UI offers a lot of capabilities to fiddle with the cluster without even opening up your terminal. And as you can see as a part of this UI, there is an add-on tab which is selected and this tab essentially shows the entire catalog of all the add-ons which a customer can install. Finally, installing an add-on is as simple as just clicking on one of these thumbnails and clicking on the install button and you are done. You can install an add-on, a fully-fledged add-on. And if there's a complicated add-on to install, which requires a bunch of input parameters, well, the UI would prompt you to provide those parameters through an interactive form. So very simple process to install an add-on. But the main stuff happens behind the scenes. So let's dive right into it. So don't get feed-out with this diagram. This is much more simpler than it looks. So you might be noticing a lot of colors flying around here in this diagram, but they are essentially meant to emphasize on how abstracted and automated things are from the eyes of both the customer and the core add-on developer with the power of the tops. So first of all, let's dive into the bottom part of this diagram, which represents the add-on release lifecycle. And obviously the customer has no visibility into this thing. Only the add-on developers, the internal Red Hat add-on developers, have visibility into the add-on release lifecycle. So the only step they take, the only manual step they take, the add-on developers, they only raise one full request containing the add-on artifacts like the deployment animals, the custom resource definitions and whatnot to the managed tenants bundle's repository which Ashley shocked about. And we have continuous integration jobs to ensure that these bundles, this pull request is absolutely valid and compliant with our ecosystem before it's merged. So as soon as a pull request is merged in this managed tenants bundle's repository, all these artifacts, all these animals are wrapped into a docker image and this docker image is pushed to one of our internal registry. Then the next step is to propagate the link to this newly pushed docker image to the managed tenants repository. So just a quick revision, the managed tenants repository as Ashley explained, it houses the configuration and metadata of all the add-ons which includes the name of the add-on, the display pick of the add-on, parameters of the add-on, etc. So as a part of all this configuration there is also supposed to be a reference to the artifact image of the add-on and that's exactly what's going to happen in this step. The link of the newly pushed image will be propagated to this managed tenants repository. And once this propagation is successful, again our automation, our GitLab bots, they synchronize the entire state of the managed tenants repository with the OCM. Now what is OCM? OCM is the OpenShift Cluster Manager and what does it do? Well, it serves our customer and well, how does it serve our customer? Well, the UI, you saw it, this is OCM. Well, this UI plus the backend which powers it, that is OCM. So yeah, so whenever a customer clicks on the install add-on button in the UI, OCM captures that event and it's like, okay, cool, I'll install the add-on on your cluster, on your cluster. But it doesn't do that directly. It actually offloads this job of add-on installation to one of our another internal component called Hive. And Hive is basically the component which is like a grandparent of all the customer clusters in the world. And it essentially orchestrates the add-on installation for all of our customers. So Hive is essentially Cube CTL applying all those artifacts, all those add-on manifests on the customer cluster. And not only that, it also ensures that if the add-on is fiddled around with the customer cluster, it's reconciled back to the desired states, matching exactly what is present in our repositories and what OCM is aware of. So that's the beauty of this entire ecosystem powered by GitOps. I mean, things are abstracted as much as possible. Most of the things are automated. I mean, you can look in the diagram, most of the text is green-colored and well green-colored in our case represents automated steps. And the only manual steps are like one step per person. So a customer takes the one manual step of clicking on the install add-on button and the add-on developers, they only take one manual step of raising the pull request. That's it. Everything else is automated and abstracted. So I believe that's the power of GitOps and shedding more light upon how GitOps compliance is achieved with this model. I guess Ashish is going to continue. So over to you, Ashish. Yeah. Let's talk about how we achieve GitHub compliance with our add-ons territory flow. As we've talked about earlier, the entire declarative state of all the add-ons is kept in the managed tenants and managed tenants bundle request. And as again, we talked about this earlier. All the add-on artifacts are versioned and it's immutable. Add-on developers are forced to have new patch releases or major releases if they want to push new releases. Right? So it's immutable there. Unfortunately, it's not pulled automatically at the moment. We need a manual trigger which in turn invokes our bots which would push all our add-on state into OCM. And talking about continuous reconciliation, what we have right now is sort of like a lazy reconciliation. Whenever a change happens in the repository, we take the entire state in the managed tenants and managed tenants bundle to the repository and push it into a platform. Instead of just isolating that change to a particular add-on. So in that way, we have continuous reconciliation. There is already work in progress to make this a truly continuous reconciliation with periodic reconciliation against the class. That's all folks. Thanks for coming to our talk and feel free to reach us over at our social media handles which you can see here in the slide. Thank you so much everyone. Have a nice day.