 Hi, I'm Michael Crenshaw. I work as a software engineer at Intuit on our Argo CD team, which means I help maintain the GitOps component of our internal development platform. Besides supporting basically 45 instances of Argo CD at Intuit, supporting like 20,000 applications and about 200 clusters, I work on the open source side as a maintainer of Argo CD. I'm an approver, which means that I can review and merge pull requests. So I help get bug fixes and features in. So this talk is going to be about how GitOps I should be, which is maybe a bit of an ambitious topic for the GitOps conference. But basically it's just tales from what I'm going to call our kind of GitOps environment, places where we've strayed from the dogmatic principles and made some pragmatic choices where necessary to get things done. So to understand how GitOps we should be, obviously we got to remember what GitOps is. So you all know the definition, but I'll make it concrete in terms of what Intuit does to make these things happen. So first, declarative, reads and Kubernetes, so that's easy. Everything is YAML manifests, Kubernetes operators keep things in sync. It's declarative for free. Versioned and immutable, Intuit uses GitHub Enterprise. So our developers for each service have an individual deployment repository. So every change that our developers make to their deployment is obviously versioned and it's in a commit. That those sources are pulled automatically by Argo CD. So we just pull the Git repository to get the latest sources. And then it's continuously reconciled. And here's where Intuit starts to stray a little bit. So in terms of Argo CD continuously reconciled means you have two settings enabled. Automatic sync and self-heal. So automatic sync means that when a developer makes a change in the deployment repo and they push, Argo CD picks it up automatically and then applies that to the cluster. Self-heal looks at the other side of the equation, which is things can change on the Kubernetes cluster. Operators can change resources, individual users may have cluster QCTL access. Self-heal means that Argo CD will see that someone has made a change and it will overwrite it with a desired state. I think that point four is where people tend to move a little bit away from GitOps because it's very difficult to get to that point where you have automatic syncs and self-healing enabled. So I'm gonna give you four stories about stepping away from GitOps. And the first example I'll give is my team's region evacuation setup. So at Intuit, all 45 of those Argo CD instances are deployed primarily in US West. And we have a passive copy of every single one of those instances mirrored over to US East, but it's all scaled down. Today, the way we do a region evacuation is we have a command line interface that opens a pull request against our Argo CD manifest. And it basically just increases the replica counts and spins up Argo CD in the East. And it also brings it down in the West so that there aren't conflicting actions between the two sets of controllers. That works okay, but the problem is the SLA for GitHub itself is 20 minutes. So if something goes wrong in the West that affects GitHub Enterprise, we have to wait 20 minutes for our GitHub team to get up and running for us to even merge our pull request to start the failover process for Argo CD. It takes us about 20 minutes to get Argo CD from pull request merged to up and running in the East. So we can't wait 40 minutes from the first deployment failures to actually let our developers start deploying again. So in order to get things up and running faster, we just kick, get out of the equation. Instead of running a CLI to open a pull request, we're actively working on developing this new system. It's just gonna run Kube CTL in a cron job, which will trigger as a job in the East region. It'll just manually do the scale down of the West on a best effort basis, obviously, because maybe West is so bored that you can't even confirm that it's down. But it'll definitely scale up the East, and this means that we can recover in parallel to GitHub. So about the same time that GitHub is coming online, Argo CD is gonna come online. Our developers are gonna be able to start working again after about 20 minutes. So this is us moving away from GitOps a little bit in the service of actually getting back our GitOps system when it crashes. I think we'll eventually move back to GitOps, because our GitHub team is working on getting their SLA down to five minutes. I think there's some low hanging fruit in Argo CD's recovery time. We can get that time down. Eventually, we'll go back to the old way, open a PR, do everything via GitOps. But strategically, for now, we want fast recoveries. So we're not doing GitOps in this instance. So that's story one. Story two is a little bit of history of Argo projects. So I think you heard at the keynote, Dan mentioned that Intuit bought Applatics, which was developing Argo projects, sort of about the same time Intuit's going full tilt, developing its internal development platform. And that platform is very, very heavily Jenkins-based and imperative-based. So we had a lot of pipelines already set up. We have a massive Jenkins function library that our developers are used to, to do things like pushing jars to Artifactory, pushing CodeCov information, and doing quality gates like Linting YAML. Our developers want all those tools. And all those tools are part of an imperative system that is a little bit of a deviation from sort of the, let's do everything before the PR has merged GitOps system that Argo CD tends to thrive better with. So rather than just rewrite everything in a way that jives more of the GitOps Argo CD pull system, we made them coexist. The last step in these imperative pipelines is the Argo CD CLI, Argo CD App Sync. So you're not getting quite that pull system. You're still pushing the information, okay, Argo CD, I want you to sync. But you're still getting the benefits of all your source of truth that's still in Git. You still have this long audit trail and users are interacting with tools that they're familiar with. So it is Jenkins kind of pushing sync and doing a bunch of other stuff. But people still get the GitOps side of Argo CD. So that's story two. Story three is bootstrapping clusters. And this one's really fun. This isn't one that Intuit is doing. I saw someone asking in CNCF Slack about a problem they were trying to solve. I'm not exactly sure what service they were trying to provide for their customers. I just know that they wanted to spin up a cluster and have everything available for the users of that cluster, including a GitOps system to run from day one. It sounded as if they basically wanted to spin something up for a whole new team or potentially a separate company. Maybe they were software as a service. At any rate, they had a problem. They needed Argo CD because that was gonna be the GitOps controller for other users, but they were also gonna host GitLab in the new cluster. Obviously, if all your sources are gonna be in GitLab that is hosted on the new cluster that you're bootstrapping with Argo CD, you have a chicken and egg problem. You gotta have GitLab to get Argo CD and Argo CD to have GitLab. So the way we solved this problem was actually using a feature called Argo CD CLI that is anti-GitOps by design. We have something called local sync, which means a developer uses the CLI. They run Argo CD AppSync with a local flag and it builds their manifest locally on the developer's machine and pushes it directly to Argo CD. So completely bypasses Git and the idea behind it is if you're an admin, you're working in a dev environment, you don't wanna have to open a pull request, get it merged just to see what your changes are gonna look like on the cluster, you just wanna push straight out to the cluster. So we've got this tool called local sync, intentionally a little anti-GitOps. The way it works behind the scenes is we have a field in the Argo CD application spec. If you're not familiar with that spec, application is just a thing that says, here's where we're pulling the sources from, here's where we're writing it to, the Kubernetes cluster. That spec has a field called operation. And inside of that field, you can just dump manifests, just plain text fields. And the Argo CD controller will pick those up precisely as it sees them and applies them to the destination. It completely ignores the spec.source.repourl field. So this customer, what they ended up doing was writing all the GitLab manifests directly into the application spec for the GitLab application. So when they applied it to the cluster, they just needed Argo CD running. Didn't need GitLab. GitLab would then spin up based on these hard-coded manifests. And from that moment on, you're done with the whole lower part of that manifest. You're back to just source spec.source.repourl, your internal GitLab instance, and you're off and running with GitOps. So again, maybe moving a little bit away from pure 100% GitOps and the interest of getting your GitOps system up and running. I thought that was a pretty sweet use case. Okay, fourth and final story of being anti-GitOps, rendered manifests. The diagram you see here is Intuit's typical GitOps flow. You got your developer. We use all customized manifests. They check that into their deployment repo. Argo CDCs that picks it up, runs customized build, and then applies that to the cluster. Intuit is a financial technology company with really stringent compliance and audit requirements. So the fact that we have sort of this pre-final version of the manifests in Git is insufficient for our audits. Particularly because we make use of customized remote bases. So we're pulling sources from things that aren't in the deployment Git repo. And that all needs to be in the final audit, because we need to know precisely what was applied to the cluster. So instead of having Argo CD build the manifests, we build a system that sort of hijacks the manifest build part. There's a Jenkins pipeline that watches the deployment repositories main branch. When a developer checks in a change that they want, Jenkins sees it, picks it up, runs customized build, we actually do a couple of other small transformations, things that the user shouldn't have to worry about, that we can just hide from them in Jenkins. And then we push that back to a branch on the same Git repository. So if they made a change in the prod environment, we check it back to a branch called prod. And that ends up being the branch that Argo CD watches. Now Argo CD's job is easy. It's just got plain manifests. Plain manifests no need to run customized build. It just deploys it straight out to Kubernetes. I would love to see this re-implemented as a native Argo CD feature. I think that Argo CD should have the ability to, after it finishes building your manifests, write it back to Git, write it back to Postgres, wherever you want to keep your audit log. For the moment, this works. But if anyone's interested in that kind of feature, I'd love to hear your thoughts on it. So that's the final tale of us sort of pragmatically stepping away from GitOps. The answer to the question, how GitOps should I be, please hold the tomatoes, it depends. The answer is be strategic, be pragmatic, do what works for your environment. These are just four examples of places I've seen where it's okay to maybe ignore one principle in the interest of actually getting things done. So I think the final answer is GitOps most of the planet. That's it. Thank you all. Thank you.