 Yeah, good morning. My name is Eric Hemelreich. I work for Red Hat as a principal software engineer and team lead in the OCM group. Today I'm going to talk about how we use GitOps in the OCM organization to help us scale. I'd like to start with a story, a bit of background. Before I worked for Red Hat, I was a software consultant for a few years, and then oftentimes I would be on a three or six month assignment where I was set to join another team, be productive, and then leave. I've kind of lost count of the number of times where I show up on site at a client on a Monday morning with my computer, ready to get started, and I don't have my access to whatever systems. So I'm stuck waiting for a couple of days to get access for their IT to set things up. I guess I can talk about definitions. What is GitOps? Basically what we're talking about is putting a lot of BAML files in a Git repository as the source of truth, and then having a reconciliation process that runs often to go apply those files. I'd like to maybe start by looking through some examples. This OCM resources repository, we've been using for about three or four years to help onboard and off-board people into the OpenShift cluster manager group. So talk a little bit about the structure. There's a ton of files in here. As I mentioned, there's a reconciliation process. It's a Go program, but I think what's more interesting to look at here is the data directory. We have a ton of BAML files in here defining users and organizations and what information they should have access to in OCM. So you see we've got three environments production and a couple pre-prod environments stage and integration, and I think maybe the easiest place to start is look at an onboarding task. How does that work? You show up to a new job and you need access to certain systems. At most places, you file a ticket with IT and wait, or it's a manual process that somebody does and said what we've got here in OCM resources is a merge request. So this person's being onboarded. There is a JIRA ticket linked with even more details. They opened a merge request and we can look at the changes here. So I'm going to go kind of fast and I'll try to leave some time for questions at the end, but there's a lot going on here in this file. So we've got a schema defined. So we validate the structure of all the YAML files. There's a user ID identifying the person, as well as the second user ID, and then the most important part here is the roles defined. So this person needs SREP roles and SREP developer permissions in the staging environment, and then we've got metadata about user ID, what country they're working from. And as I mentioned, we have the JIRA link there for even more context. And if you just kind of look at the merge request, you can see who opened it, who it was approved by, and who it was merged by. And you can see down here at the bottom, every time someone supplies some YAML to get onboarded, we have a validation process to make sure the YAML will apply cleanly in the environment, and then we can see who approved it and who merged it. So what's good about this versus the traditional IT process is this is auditable, it's self-service, it's very quick to reconcile, versus kind of the other way of doing things, filing a ticket with IT and kind of waiting. I'd like to go through another example. So that's basically users. We also model organizations in here. And so just thinking about the normal flow of developing a product, maybe you have some feature that's not ready for production yet, or should only be exposed to a few users, that's the merge request we have here. It's adding a capability to a specific organization. And I'm going to go quick here, but we have a schema where we validate the YAML structure. We have an organization which is just a group of users. We've got a number of things like access to quota, which is OpenShift cluster manager, create and manager clusters. Part of that is getting quota to do so. And the interesting part here is this last line here, which says we're going to define this key and value, install config override and set it to true. So that's a way through GitOps to enable some optional functionality for a subset of your users. You could think of if you're building features and they're pre-production, not quite ready for all customers, you could enable that through GitOps. Okay, so I've gone through kind of two basic examples here about how to define the YAML. Self-service, users can submit it, can see who approved it and merged it from an auditing perspective. I'd like to briefly go through the reconciliation process. And there's a lot here, so I'm just going to kind of go quick through it. The reconciliation process is just a go program that parses the YAML and runs many times a day. So we're looking at the YAML data and using it as a source of truth. So it's going to make API calls to the system to make sure that user had SREP roles or that organization had that specific capability. And so there's a lot here, but basically it validates the YAML. And then this is a lot of install and setup steps. But you can see here we're sort of starting to get to the output. We're going to validate all of the YAML and then move on to the reconciliation phase. So it's checking that everything in the YAML is still in the system via API calls to OCM. So make sure that organization has quota capabilities. Make sure that user has those two roles assigned. And what's nice as well is we have kind of an audit log here of every action it's taking. So you get time stamps. Who requested this change? When did the change go into place? So all of that information is quite useful to have. You think about like security and compliance certifications. Having a formal process like this is required. So GitOps is one way to do that. And I'd like to also mention that we've been using this process for about three years. It's grown over that time. And so we're starting to automate more and more tasks, which would be manual, which has come in handy as we've hired more and more engineers. So I'd like to look at another example. Here we've got a person in support needs access to support OSD and Rosa. And we can look at the change here. Again, it's that user file. Here's a user ID. Needs access to these roles in this environment. And that's about it. So here the interesting part is if we look at the activity on the MR, we've got that validation job that runs to make sure the YAML will apply cleanly. So there's a couple pushes there just getting the structure quite right. And then the interesting part here is this comment by the OCM Resources bot that says, hey, the YAML looks fine, but you need an approval comment from your manager. So this used to be a manual process where we'd ask people to get that approval. We're starting to automate it. So it's very flexible. You can kind of start small and then add more automation as you need it. Another interesting thing that the spot user takes care of is I talked a little bit about onboarding, hiring new engineers and so on. Another process is somebody leaves for another job or maybe they transfer to a different department at Red Hat. So in terms of like security and compliance, you need to make sure when somebody leaves, you lock down their access. It's very important. So as part of this reconciliation process, we've got it set up to run three times a day and it checks for any users that have left and revokes their permissions in OCM. Let's see. So I've got a few more examples and details I can talk about. Somewhere else, this GitOps style of project came in handy was I got a request to do quarterly access audits from some of the compliance folks. So here I've added an auditing command. You can configure a list of permissions that are considered sensitive or elevated and then this audit command will produce a list of users who have those permissions. And then the compliance folks can take a look at that, make sure it's the right list of users and remove anybody who shouldn't be there and so on. So that's the audit command. Another benefit of taking this GitOps approach would be the fact that we had a situation where I talked about those three environments, production, stage, integration. We had to move our integration environment to new infrastructure. In terms of GitOps, it was very simple. Just point this repository at a different integration environment and run the reconciliation loop rather than having to think about how do we migrate the data or having somebody do that manually. I mean, it was quite straightforward to change the URL for integration right there. The last sort of benefit I'd like to talk about, you know, taking this GitOps approach, putting your data in YAML in a repository is that if you follow this process, you get some other benefits like think about, I mentioned quota for clusters. You know, if you have a cluster and you want to install an add-on, you can request quota here through YAML. So instead of having to build another program to get analytics about who's using these add-ons, you can do a lot by searching the human readable YAML in the Git repository. So that was a lot and those are some of the benefits. Anybody have any questions? Sure, I think I saw your hand first. Yeah, so the question was about like, is this inventory management in YAML? I guess the question is about what does the number mean? Sure, so I think, and this is defined in the schema files, but those are quantities rather than inventory management. So it's somebody saying, I need to install 100 copies of this add-on. Does that make sense? Oh, yes, so the file name is the ID for the organization. It's internal ID. Yeah, any other questions? Yeah, go ahead. The question about how to manage user data like user names of the users, because they will be public. Sure, yeah, so we've gone with username for these files. I'm sorry, repeat the question. The question is about how to, looking at the user files, there's emails, other maybe sensitive information. Maybe users don't want to have that public. We've gone with usernames here, which, yeah, it was a decision, a choice. It's what uniquely identifies the user in our system. So you could identify them with another, an ID, an anonymous ID that would be fine too. Like we've done with organizations, we're just using an ID in the file name there. And it defeats the purpose because you see the ID of the user ADCD and you don't know who that person is and to prove you're approving the access for instance. Yeah, so the question is about using ID or username. And I think what you're saying there is, yeah, we're using the username for that reason. Like in this repository, we don't have a reason to keep anything secret, really. Sure, go ahead. Yeah, so the question is about like, how do you teach users to use this? How do they know that, you know, they have to specify this add-on in this format. What I haven't shown here is a lot of teams have, as part of their onboarding documentation, go to OCM resources, open merge requests with the ML file like this. So it's in the other teams onboarding documents. And the other thing you can do is, you know, it's a Git repository. So you can, you're looking for examples, you can just browse and see what other people have done. All right. And thanks so much. I think we're out of time.