 Hello, everyone. Welcome. Bonjour, tous. As you may have noticed already, ArgoCity is not an opinionated project. What I mean by that is that it doesn't dictate how you use the tool. You can do things like using the generated manifest like it was mentioned yesterday at ArgoCon in different talks, or you can do something a little bit more esoteric like using Knicks to generate your manifest and hook that up in ArgoCity during runtime. Those are use cases that are available and it's up to you to decide how to use it. How to use it too. This talk is about those use cases where you want to generate manifests in a specific way. That is specific to your company. Let me quickly introduce myself. My name is Leonardo. I usually go by Leo. I'm a staff software developer that into it. And I am a ArgoCity and ArgoRollouts maintainer. Hello, everyone. My name is Alexander Matushensev. I prefer to go by Alex. And I'm a long-time maintainer of Argo project. I started working on it from day one. And currently I'm a chief architect and co-founder of Acuity, a company that tries to make Argo even better. And I'm going to start the presentation. So this is the plan and agenda for today's talk. So as Leo mentioned, we are going to talk about config management. And first I will describe what do we mean by saying config management tools in ArgoCD. Next I will jump into the feature that we're going to describe in details today. It's config management plugins or CMP for shortness. So I will describe first iteration, second iteration. And then I will do a demo where I will try to teach you how to create your own config management plugin. And then I will give it back to Leo. And he will do a deep dive and describe security enhancements, high-level architecture of feature, some challenges and improvements that are coming soon. And with that, let's go ahead and start the presentation. So as I mentioned, I want to give a brief overview of what config management tools means in ArgoCD. And so from day one, I'm sorry, I have to stay away from my mic. From day one, we decided that ArgoCD is not going to force engineers to store YAML files in Git repositories. Obviously, because it requires a lot of repetition, no developers wants to do it. And everyone prefer to use tools like Helm or Customize to generate YAML on the fly. And so as a maintainers, we realized it's a huge requirement. And so we decided to introduce first-class support of those tools in ArgoCD itself. And first-class support means those tools are baked into ArgoCD image and available with no configuration out of the box. And the next big decision we've made is, ArgoCD is going to be smart about those tools, which means it will decide automatically there is no input from a user, which tool supposed to be used for manifest generation and to describe it even simpler. So if ArgoCD sees char.yaml file in the Git directory, it uses Helm. If it's customization, it uses Customize and so on. And then after some few iterations, we decided to go even one step ahead and ArgoCD got a user interface support and CLI support. And I'm kind of trying to show it on the screen right now. So ArgoCD allows end user to interact with the Config Management tool and quickly change, influence the manifest generation through parameters. And so in this particular example, developer can override image tags on the fly using Customize and that enables use cases such as dynamically generate applications for pull requests and provide so called preview applications or simply modify your QA environment to test the changes that you want to test and without committing code to Git. And so looking back, I can say it was really great decision to support those tools out of the box. It really helped with adoption, but after a year or two, we learn about some disadvantages. And the list of problems is on the screen right now. The first biggest issue was our opinion about the most popular tools, not necessarily matches with everyone else's opinion. So one mistake we learned very quickly is for example, Casonet, the very first tool by the way that was supported by ArgoCD. It just died a year after release of ArgoCD. So obviously not the most popular tool anymore. And on top of that, we've got a lot of requests to support more tools such as TANCA, HEMFI, CDK and those tools are very obviously great tools and deserves to be supported. However, we learn our lesson from supporting Casonet and then we realize that it's a lot of maintenance burden to support a new tool in ArgoCD. And so finally, we've got a lot of complaints from end users, developers that ArgoCD release cycle not necessarily matches with the release cycle of all those tools. And so the caveat is let's say HEM release a new awesome feature that's available for you today. But if you want to use it in ArgoCD, you have to wait for the next ArgoCD release which is probably happening two months in the future which is obviously not optimal. And so it was clear that change is required. And that's why we come up with an improvement. And improvement was a new functionality called Config Management Plugins or CMP. And this slide represents the first iteration of this feature. And to quickly describe it, the main feature that it provides is ability to take the CLI that you want to use for manifest generation and then just dynamically configure it in ArgoCD. And so as an administrator, you would have to provide pretty much two pieces of information, first the name of this tool and second just the CLI command that invoke the binary of your tool and produce JSON or YAML. And this way you can pretty much take anything and convert it into Config Management tool in ArgoCD. And we even supported limited version of parameters. So end user have ability to provide a list of environment variables that will be passed to the environment while executing the shell command to generate manifests. And so the result was pretty good. The YAML snippet that you are seeing right now it's the real time snippet that a lot of people used in production. It allows to take helm and customize and combine it into one tool customized helm. So helm will produce a manifest from a helm chart and customize provide some patches on top. And as usual, we solved like 80% of use cases and didn't solve remaining 20% that required a lot of work. Here is a list of things that we didn't do well or just didn't plan even to solving on solving. So first of all, parameter support was not ideal. Like end users had to know the list of environment variable names with no suggestion from user interface which environment variables even make sense. Second, we did not implement auto detection. So let's say if you have a plugin that's meant for helm file and if you point ArgoCD to a directory which has a helm file configuration, ArgoCD would not know what to do and end user would have to explicitly specify a team application spec which is not ideal. Again, not the perfect user experience. And finally, ArgoCD administrators, we were not super happy because we did not really provide any help for them to deliver binaries. And they had to do tricks that you see on the screen right now. The YAML snippet on the right side of the screen shows a patch that they need to apply to a particular ArgoCD component and use tricks like init containers to download binaries on the fly and then volume mount copied into the main container which is not ideal. And so finally, we're getting to a feature that I'm going to demonstrate soon, config management plugins V2. So we kind of worked on the feedback of our first iteration and implemented the new version of config management plugins functionality. It had some, it basically had a, it's like a completely right, so it had breaking changes but it provided a set of improvements. So the biggest improvement was ability to automatically detect the tool that's supposed to be used for generation, which means if you, for the Helm file that as you will see soon, you can literally configure it once in that centralized managed configuration and then ArgoCD is going to know that the plugin is supposed to be used to generate, to use Helm file for Helm file based application manifest generation. Next big improvement was parameter support. So pretty much as a developer of a config management plugin, you can provide first class like an experience in ArgoCD UI. So ArgoCD is going to know which parameters you can possibly provide to this application and default values of those application, which is a big improvement. And finally, we improved security and Leo will cover security improvements in details once I'm done. And so that's enough of theory. Now I'm going to, I will try to do a demo with those microphone issues and show you how that feature works in real life. So for this demo, I created a Git repository so you can just basically repeat your demo, repeat the same demo on your laptop. The only requirement is you need to have some kind of Kubernetes cluster, MiniCube, K3S, whatever works for you. So now I'm going to switch off the presentation and jump into the Git repository. Let me make screen a little bigger. And so basically the Git repository has all the information you need to try the demo yourself. It's going to teach us how to use CMP V2 and create a plugin that enables Helm file as a manifest generation tool in Argo CDM. In case you don't know, here is the link to the Helm file home directory. A short summary, it's the CLI tool built on top of Helm that adds a lot of awesome features. It simplifies secret management, environment variables usage and more. So I encourage you to try it. I really like it and use it. And next, I want to switch to content of this repo. Just briefly explain what do we have here. And we literally have three files in addition to readme. The most boring file is customization.yaml. I hope you know what customizes, but basically it's the tool that brings together a bunch of YAML files. And so first thing I'm doing here, I just take open source Argo CD manifests that install plain Argo CD with no modifications. And then next, we slightly modify it by applying a patch to Argo CD repo server, the component that's responsible for manifest generation. And next, I want to show you the changes that patch applies. And so as a short summary, it introduces a sidecar container that represents our plugin. So the name of plugin is a Helm file because it supports Helm file usage in Argo CD. And the main difference of the first version, comparing to second, is that here you're able to use any image that has binaries of your tool. And so you are no longer limited to Argo CD image. You can use your own image here. And so for this demo, I'm using community maintained image that supports integration between Argo CD and Helm. And then next, I have volume mounts that just must be there for every config management plugin. So you can find those volume mounts in documentation or just copy it from this demo. And last bit is I need to provide a YAML file that has configuration detail of a plugin. And in this case, I'm using just a config map, customize config map generator, and config map volume mount. So basically the content of this, the file that will be delivered using volume mount is defined here in the plugin.yaml file. And it explains to Argo CD first how to generate manifests. So I just selected two commands, init command, behind the scene, download Helm chart dependencies, and generate command, eventually execute Helm template to produce a YAML output into standard out. Next, more interesting part is, this is a part of configuration that explains Argo CD, how to detect that application in the git is managed by Helm file. So, and as you can guess, it looks at the content of the git folder in the git repository. And if it sees Helm file.yaml, it's going to use this particular plugin. And last bit, the most complex one that you can learn in Argo CD documentation, it's parameter detection. So basically it's a shell command that produces adjacent list of elements where each element is a parameter that has a name, type, and default value. And that's pretty much it. So we have this git repository and it's ready to be used to use it, just execute this YAML snippet which will create a namespace and then apply the manifest generated by this repo. It will install Argo CD, configure it to use the plugin. And so for the sake of time, I did it before the demo. So now I just have a Argo CD running locally. I can open it, it's running on my local host. And as you can see, it basically has no applications. And sorry, let me open it again. I had just a bookmark that creates an application using Helmfile. So I hope you are familiar with Argo CD user interface, but long story short, it creates an application based on content of this repository, Helmfile. I'm using it because it has the examples that I need to use and I'm going to use it for the demo. And the first enhancement I want to demonstrate is detection. So as you can see, Argo CD inspected the content of this repo and it detected all the Helmfile based applications. So as an end user, I don't have to guess like what my path is. So it's a nice convenience. I'm going to use this sample deployment based on Helmfile. And next improvement is, as you can see, Argo CD knows that it's based on a plugin and it detected all the parameters that as an end user I can specify and kind of influence how manifest are being generated. So I'm very happy it worked. Basically demo already successful, but just to finish it, I need to finally create application. It will take a few seconds. And as you can see, application was created. If you are familiar with Argo CD then great, you already know what happened. If not, then basically Argo CD knows that this directory has two Kubernetes resources. One of them is deployment and it's supposed to be deployed to namespace that I chose. And that's it. So pretty much you have enough information to create your own plugin. It can be a Helmfile or whatever tool you like, like Tanka. And now I want to pass it back to Leo so he can explain what happened in background and how it works. Leo, please continue. Okay, thank you, Alex, for the demo. And I want to start this section of the presentation just opening a really short parenthesis, just to say that by the time the Argo CD project was about to be graduated by CNCF, one of the prerequisites that was brought to us is that we had to step up in our security posture. At that point in time, we were having CVEs happening and being presented to us on a weekly basis, sometimes even three CVEs at a week. And as a matter of fact, CMP was one of the main targets. CMP V1 and even V2 when it first came out was one of the main targets of those CVEs. And at the end, we got graduated but if you analyze all the issues that we faced, it boils down to those two main categories that I'm showing you here. And it has to do with the fact that the CMP process was running in the same process as the repo server. And they were sharing the git environment. So with that, we could have some really bad things happening like from a CMP, we could access another git repository, nothing not related with the application itself, which is even private repos and sometimes, which is a pretty bad thing to have. The other problem that we had, the other category that we had of issues is the possibility to have that directed traversal attacks. So again, the CMP process was mounting the same volume where the repo server has cloned all repos that it needs, that is configured with that particular ArgoCity instance. So from a CMP project, a repo server, I could have something like a symbolic link there defined that go one level up and have access to other repos. Maybe we find a secret committed to a repo and, well, I'm kidding, nobody commits secrets in git, right? Anyhow, and what is the solution to the problem? It's quite simple, actually. We don't need, we can't share the git environment with the CMP process and we can't share the file system. The challenge here is how we can achieve this, how can we improve this security issue avoiding breaking the existing CMP plugins that existed at the point in time, right? Unfortunately for the git environment, we had no choice. It was just too much power to provide to the CMP process to have. So we had to cut that. It was that that was a conscious decision that we made. Unfortunately, we had to break the compatibility there. But for the file system, we found a solution to the problem. But before jumping on how we address the problem, I want to provide you a brief overview of how CMP works behind the scenes, okay? So when Alex explained about the sidecar, the CMP V2 that runs in the sidecar, basically that sidecar, what it does is it runs an implementation of this interface here. So this is the interface that repo server uses to communicate with CMP plugins. So it's a GRPC interface, and we provide the default implementation already for CMP developers. So it's a binary that ships. It's in those config maps and configurations that Alex showed before. But let's say the Argo CD needs to generate manifest. So how that works right behind the scenes, how actually Argo CD generate manifests, leveraging CMP plugin, and that's how it works. Basically, on this diagram here on your right side, I'm simplifying an Argo CD instance, right? So we have the application controller. So the yellow boxes are representing pods. The blue box inside those yellow boxes are containers, okay? So from the application controller pod, we have the controller service running there. And whenever a manifest needs to be generated by Argo CD, that is done invoking a GRPC service to the repo server. That request goes over HTTP2, and repo server is the component responsible for generating manifest in Argo CD. When that request reaches the repo server, if there is a config management plugin register to that repo server, the repo server will automatically get notified by that. And that communication also happens over GRPC, but at this time using a unique socket. So it doesn't go out of the network, and we can apply additional security because it's file system-based. And if you notice, this interface that I mentioned here on the left is, aside from this init attribute here, is a one-to-one match to the other attributes. So this yellow square that you see here at this part, this represents exactly the config map Alex was showing you, okay? So it's a one-to-one match. So basically repo server will invoke this generate manifest method, and the method will basically just execute whatever the CMP developer defined in the generate attribute of the config map. So that's basically how you define and create new CMPs to attach in Argo CD. And I'm representing here that we can have multiple containers, multiple CMP plugins running at the same time, right? Okay, but let's go back to the problem that I was referring to, right? So how we address the issue with the shared file system. As I said, we don't share the same file system anymore. So from the CMP process that runs in this container here, there is no notion of the file system running in the repo server container. What we do is when this request reaches the repo server service, the service will identify the directory that is related to that particular application. It will compress that directory in a TAR-GZ file, send it over the GRPC to the repo server process. That repo server automatically extract that file and just those files are available to the process. So there's no possibility to inspect other files that are belonging to the repo server. Okay, so known issues. We know how the current, this is the current state. This is where we are today with CMP. And between the maintainers, we all agree that the CMP installation process is complex, right? The thing that you have to patch a repo server and add an additional sidecar container, maybe there's a better way of doing this, right? And we all agree that there must be. It's a little bit hard to debug because it runs as an additional container. So you have to remember if you want to inspect the log to really filter the container that you want. It's a little bit annoying. It's not a super big deal. But the fact is that you have to deploy all those components locally in order to develop a test properly, a CMP plugin. There is a known issue. So there's a GitHub issue opened already and we are aware of that. Related to Mono Repos. So basically the thing is if the repo has more than 200 megabytes, there are situations where users are facing timeouts because this TARGZ process and sending it over to the other process over to your PC, sometimes it takes longer. There's a way to address this problem by filtering the files in the Git. So like excluding the .git folder before compressing it is one alternative. The feature is there. It was implemented by one contributor. And yeah, one more issue that we're aware is that the CMP as it is today, it doesn't provide you out-of-the-box access to specific features of customize and helm. And that's why we implemented the parameterization in ArgoCity, in the CMP implementation in ArgoCity to allow having those use cases possible. So basically, a CMP developer needs to implement those features in a specific CMP, leveraging those parameters. Okay, and where do we wanna go? And before jumping here, I want to clarify that there's no real ongoing work on any of those topics here. Basically, this, between the maintainers, this is kind of an agreement where we think the CMP should go. So we all agree that we could have a better installation process, right? A streamline, maybe not only for CMPs, but also for the other extension points that exists in ArgoCity. So this is something that could be very well improved. The other topic that we would love to see happening, and this is the first direction that we went when we wanted to address the problem with traversal attacks, is doing in the repo server something similar to what Docker does. And what I mean by that is leveraging Linux username space. So the idea here, what we try to do, right, was that from the repo server, we wanted to open a new process in an isolated username space that has only the specific directory mounted for that specific application. I ran a POC at that point in time, I called it Limbo, and I got this concept working locally in EKS clusters. Unfortunately, ArgoCity needs, not unfortunately, the fact is ArgoCity needs to run in Red Hat clusters. So in that environment, we ran this POC together with our Red Hat friends contributors to ArgoCity. And for this functionality to work, we needed to have a Linux specific capability enabled at the node level, if I remember correctly, and that wasn't the case in OpenShift clusters. At that point in time, maybe that changed, I don't know, but that would be a much better way to address that problem because we wouldn't have to, you know, compress that file over, send over your PC to the CMP process. Things will be way, way faster and still safe. Last but not least, one thing, I was discussing with Alex yesterday about the timeout issue. Maybe there is an easy way to fix that problem. We can make any promises at this point. We need to make sure that this is not bringing the security issues that we had to fix in the past. But maybe in the next version, we're gonna have a solution for that problem. And that's about it. This QR code here will drive you to link three. It has links to every content presented today by us and as well as the presentation itself if you want to have the link to it. It also has a link for you to provide feedback to us if you want. And thanks everybody. We still have five minutes for any questions. And I just wanted to add that maybe last slide, consider it a call for action. So a list of improvements kind of were agreed by maintainers. If you're curious about contributing to ARGO, this is the awesome list that you won't have to prove that those improvements are necessary because maintainers like really looking forward for help to implementing them.