 Hi, my name is Krisha Hernandez. The head of community over at Acuity. I'm also an open GitOps maintainer. And today my presentation is about rendered manifests. It's a pattern that I've discovered while working with enterprise customers over at Acuity while adopting the GitOps journey. Before I get into rendered manifest pattern, I am gonna go over the GitOps principles, right? I'm gonna start here because it's important to get a little bit of context. So as some of you may or may not know, these are the GitOps principles. And if you're not familiar, go ahead and go to opengitOps.dev to find out more information. The GitOps principles themselves exist to address the genuine problems of visibility and collaboration with working with complex systems like Kubernetes. The principles themselves stress the importance of declared desired state and continuous reconciliation. However, these leave considerable room for interpretations. Unironically, the GitOps principle tells the end state but not how to get there, right? The principles themselves are declarative. And I often get questions and see posts and videos about GitOps best practices, right? About how GitOps repo should be structured, what config management tools to use, how branches should or should not be incorporated but things like that. There's a lot of commonly accepted best practices out there and I'm actually here to challenge one of them a little bit but it's all in good fun. But before I do that, I wanna talk about and hone in a little bit on config management tools in GitOps. So config management tools. Config management tools are great. And in general, they seemingly play a big role in your GitOps journey. But what's great about it is that there's so many tools out there and really you can use whatever you want and whatever works for you and your organization. Config management tools actually won't significantly impact your ability to implement GitOps because they all attack the same issue. So this is why we say, hey, just kind of use what works for you. But they all attack the same issue of kind of abstracting the configuration for you and how it gets applied to a Kubernetes cluster. For example, Helm and Customize both address the need for a standardized set of manifest and the ability to alter them based on the specific environment they will be deployed into. This abstraction is extremely convenient for keeping your manifest dry. Don't repeat yourself, right, DRY. This is why config management tools have gained wide adoption in the GitOps community. You can have kind of like a basic set of configurations and then kind of rubber stamp them out and alter them depending on their destination. But there lies the problem, right? These abstractions create a problem. They are typically referenced directly by the GitOps tooling to determine the desired state. A change to a Helm chart or a customization base is a change to an abstraction. The true impact of manifest deployed into environments is actually unclear. So if you take a look at the configuration here on the right hand side, here's an example of really what goes on in a GitOps workflow, especially when using tools like Argo, CDN, Flux. So you push a change to your trunk, right? And when I say trunk and I'll say main, and I'll use them interchangeably in this talk. So you make a change into your main branch via PR, via Directly Commit, whatever. And the GitOps controller will take those, right? And for example, Customize, right? You make a change in your base. GitOps controller will take that change, will run, customize, and then apply that manifest, those rendered manifests onto the Kubernetes cluster. And so really what's happening is that the GitOps controller is kind of doing that at deploy time. So the desired state of Kubernetes cluster is not a Helm chart or a customization, actually. It's the manifest that is rendered by these config management tools. That is your desired state. So, and a lot of folks, they use Customize and Helm is, and they use those manifests in Git. And that's what typically what most organizations will store and get when practicing GitOps. But keep in mind that any abstraction between Git and applying those, right? Can have undesired effects. For example, if the version of Customize changes in your GitOps controller, those, and you're using something like a deprecated feature or something, you actually won't know about that feature until the deployment phase because the rendering happens at deploy time. So what am I getting at exactly? I'm saying that you shouldn't mutate your source of truth right before it gets applied. The desired state stored and approved in Git or wherever your desired state a store is should contain no abstractions from what will be applied during your reconciliation. Your declaration should be treated similarly to like a container image where it's immutable and applies as is to the cluster. Having your GitOps tooling run Helm or Customize right before applying the manifest to the cluster is like saying, it's like your container running app get installed on startup, right? You're mutating it right before it gets applied, right? And so what's the solution, right? What is the pattern, right? And this is where the rendered manifest pattern comes in. So solution is called the rendered manifest pattern. So each change to the trunk or the main branch, right? Of your GitOps repo will result in the manifest getting rendered by CI workflow and stored as is in Git. So meaning that this is kind of a different shift left, right? You're shifting all the rendered manifests, all the rendering, excuse me, all the rendering to the CI system. And it kind of creates this artifact and this artifact represents the cluster's desired state without any abstractions. So if we take a look here at this, the rendered manifest changes the picture just a little bit, right? You will still interact with your trunk the same way you do now, right? You push a change into your main branch, but then your CI engine, your CI actually runs something like Helm template or it'll run something like customized build. And then it'll store those rendered manifests in a branch in your Git repo. And then on an environment-specific branch. And this environment-specific branch is what GitOps controller then deploys onto the Kubernetes cluster. So you actually now get a one-to-one mapping of what you see in your desired state, in your state store is exactly what gets applied, right? Without any abstractions there, right? So then these environment-specific branches, the diff between what goes where is now is completely transparent. And you'll be able to see plainly the effect of a change in main, for example, on each environment. And I'd like to actually address an elephant in the room. So the common misconception, there's this quote that we say and a lot of GitOps practitioners say, don't use branches for environments. And that's actually a misconception. So the misconception is people take that literally, don't use branches for environments. People take that literally. And it's actually, it's not, we should probably stop saying that as GitOps practitioners because it's not what it's meant. What we meant was by don't, what we meant by don't use branches for environments is what we mean by that is don't use Gitflow. This is not Gitflow. On the surface, rendered manifest pattern might sound like Gitflow, but it's not. You can continue to use short-lived feature branches or trunk-based development. I'm a trunk-based development guy and a guy for your GitOps repo. These branches, these environment-specific branches are not used for promotion between environments. This is where Gitflow is. You assume that you're merging a dev branch into the stage branch that you're not doing that with rendered manifest at all. They're not used that way at all. Changes from one environment branch will not be merged into another branch. Instead, the contents of these branches are maintained by an automated workflow, like your CI workflow, for example, and are generated based on what's in the main branch. So your workflow actually isn't changing. You're still working with your trunk. You're still working on main. You're still PR-ing against main. Actually, your workflow does not change at all. What changes is where it gets rendered and how it gets stored. It gets stored in environment-specific branches. Consider the contents of these branches akin to a release bundle. It's kind of similar to a release bundle, and it just contains plainly rendered manifest with a desired state of an environment. So it's similar to kind of like, you wouldn't pull an image and you wouldn't be able to do that. Do Docker run, make some changes, do Docker commit, and then push it, but that workflow seems terrible. And we're not telling you to do that either here. These are specifically storing what amounts to us artifacts. So let's take a look at some of the advantages. Is that a glance, the advantages of read or manifest are that you get improved visibility into the desired state. You eliminate obfuscation, introduced by config management tooling, and you actually see, you improve that visibility into your desired state. You reduce the risk of built-in tooling with truly immutable desired state. You find out what goes wrong earlier in the process. You find that out in CI versus right at deploy time. Improve performance for your get-offs controller. It doesn't have to render anything anymore. This is especially true with Argo CD. You get a significant performance boost. And you can set like deployment protection policies based on each environment. You can actually get pretty granular. So actually I want to drop out of here and show an example. So this example here, you have a change happening to an umbrella chart in Git. And let me make this a little bigger. So here there's this change that happened in the umbrella chart. We went from version this version to the next version. And just looking at this here, I actually have no context of what is going to happen in my environment, right? I just know the version is going to change, but I actually don't know that impact. The only way to find out is to run customized build, dash-enable helm and be able to see that locally. But that's still, right? It's not the version of a helm or customized locally maybe different than in my environment. And I actually don't, significantly I don't get a diff or anything even building it locally, right? I don't even get that diff. So even building locally, I don't really see that. So like this change here doesn't really tell me anything especially like as a release manager or DevOps engineer and like I don't know what's gonna happen. I don't know if this change is good or bad. Now switching to, let's say my dev environment branch, right for rendered manifests that I can actually see now plainly see what that change is gonna do to my environment. And this is our significant changes, right? There's changes like removing and adding of service accounts, updating a configuration, deprecation of certain things, configuration changes, the list goes on and on. Now here, at least now I have contacts. I'm like, oh, okay. This is a significant change to my environment. And I actually plainly see what's gonna happen if I approve that change into my environment. So this is kind of like what it gets you for rendered manifest at least from a process standpoint. So let's jump back to the presentation. So let's go to some of the disadvantages, right? So it's not all unicorns and rebos, right? Shifting manifests, rendering to CI actually adds complexity to your CI system. And also, I can't emphasize enough that a lot of work has been put into Argo CD and Flux and their automation and control systems where I don't, as an end user, I don't have to think about customize or helm that has all done for me. So I'm trying not to undervalue that. There's a lot of value that there. And so using that, I kind of lose some of that functionality that's built into these GitOps tools. Also, it doesn't work well with tooling that renders plain text secrets, right? So like sealed secrets is a big example or like customized plus stops, right? If you're using rendered manifest, it actually doesn't work out well because now your secrets are in plain text in a deployment branch. And so, for users that are using sealed secrets and those type of things that rendered I actually recommend moving on to something like the external secret operator, which is one of my favorite things to use to kind of move away from that to be able to use that. But if you're still using sealed secrets, that's a consideration you need to take into play, right? Clux users, right? There's a disadvantage. You lose all the features that the Helm controller provides you. There is, if you're actually really into Helm, you're basically switching from using Helm to plain manifests. Essentially, you're going from Helm to Helm template and you lose things like, for example, Helm lookup, right? And which a lot of people rely on as well. And Argo CD, at the same time, right? By the same vein, you lose kind of like runtime injection along a lot of Argo CD users like to inject the application's name into the configuration. So, to differentiate it from one environment to another, you lose that as well. So there's just kind of some of the things you need to keep in mind about using rendered manifests. So being from acuity, we saw these issues and new tools are emerging, including one by us, right? Something under, something that we called cargo render, right? Cargo render is a part of the larger cargo ecosystem, right? Cargo is a open source project, completely open source, created by the same creators as the Argo project who then formed acuity. And now cargo is part of acuity's open source project portfolio. It's still an alpha. So I don't exactly recommend using it in production, but I do recommend kicking the tires because cargo render can be used standalone without adopting cargo. There's a GitHub action available. So you can use the GitHub action and actually don't worry about complicating your CI. There's now a tool that does rendered manifests for you and you don't have to worry about rendering things in CI. You can, you know, your CI actually doesn't have to change that much. You can use, you know, this GitHub action, for example, or the CLI tool for cargo render to be able to doing your rendered manifests, right? So learn more cargo renderer.acuity.io or actually join the acuity community where they're, you know, doing a bunch of awesome stuff around open source and cargo. So what's next? What are my hot takes out of this, right? So kind of hot takes. So I know I've stated this before, but it's worth restating. Manifests should be treated like artifacts, really. Like we've, you know, we've, I think we're to the point where it is now we need to treat manifests the same way we treat container images, right? They need to be immutable. They need to be treated like artifacts and which, you know, kind of brings to my next point. I actually think GitOps has outgrown Git to be honest with you, you know, working with customers here at acuity, we rent, you know, you know, our customers have ran into issues with like rate limiting from Git or overload from GitHub overloading their internal Git system. You know, it all obviously depends on how many changes you're using and how you're scaling your Git repo and, you know, how you're tuning Argo CD, but really I think we're at the point where we have to start considering that we've outgrown Git, honestly. And I believe OCI is the path forward, right? OCI as more integration comes into play with things like Flux and OCI, which there's a great integration there. Argo CD is on the road to adopting OCI as well. And OCI is gonna be a big component in secure supply chain. And storing your manifest there just kind of makes the difference. And it's more a depth to handling artifacts versus using something like Git. Now you're using kind of like an artifact storage system to store that, right? And that makes complete sense. So learn more about rendered manifest. I'll keep this slide up for a few seconds here, but go ahead and scan that. It'll take you to a blog that goes very deep into this pattern and with more information about rendered manifests and acuity. With that, I'd like to thank you very much for my presentation. And if you have any questions, I'll be around in the chat. Thank you.