 Thank you everybody for showing up. This is an absurdly big crowd. I was not expecting it, so thank you all for coming. So let's get started. My talk is NextGen Argo. It's all about config management plugins and some recent improvements we've made to them. So before I even get started, I kind of want to get a feel for how many people here have used a config management plugin in Argo CD. How many people use one that is not like open source? You use one just for your company. OK, so a few people. All right, because I want y'all's feedback on what I'm about to show you. So first, I'm Michael Crenshaw. I work at Intuit. I'm on our Argo CD team. Intuit runs about 40 Argo CD instances and about 20,000 applications. So my team is responsible for keeping that up and running for all of Intuit. My co-worker, Sindhu, couldn't be here to present in person. But she develops the stuff that I'm about to show you to a great extent. And I'm going to have a demo video from her after I do some introductory work. So first, most people probably know this. But just to contextualize the component that I'm going to talk about in Argo CD, I want to give a quick overview of what it is, what it does. So it has three basic jobs, pull sources from something like Git or a Helm repository, to transform those files into manifests. And then three, apply those manifests to Kubernetes. The part that I'm going to focus on is called repo server. And that's a pod that runs with Argo CD. It does those first two jobs, pull sources, and then transforms them. And it has some built-in functionality, like Costas was just talking about to use Customizer Helm to transform sources. But that's not always enough for everybody. So it has a system to talk to a sidecar. And that sidecar presents a service which can transform manifests that can then go to Kubernetes. Why would anyone want to use a CMP? So that's config management plugin. And I'll probably just refer to them as plugins most of the time. The first main reason people will use them is to use an unsupported tool. So out of the box, Argo CD will let you use Helm, customize, JSON it, or just plain old manifest files. If you want to use something like Grafana Tonka or YTT and Carvel, then you might want to use a plugin for that, that's how you'd add support. Second reason is maybe one of the built-in tools doesn't support the feature that you want within Argo CD. So things like Helm plugins, or like Customize 5.0 just came out and I know there's some things in that that we don't yet integrate with. Maybe you need that now, so you write a plugin to enable that feature for your organization. And then finally, a lot of folks find it useful to do post-processing on manifests. So common one is injecting secrets. I've seen people actually trigger side effects. So whenever manifests are rendered, they want to trigger some other job for whatever reason. Plugins are a good place where you can hook in and do that. So the problem that we wanted to solve with plugins is that the UI isn't super rich. So if you've used Argo CD with Helm, what you're gonna be used to is not this, this is the plugin interface, but what's on the right. You get rich integrations, you can set values files, set parameters, they're all laid out for you and the user doesn't have to go hunt things down in order to know how to configure Helm. On the other hand, the CMP UI, up until version 2.7, you could set environment variables, which we get passed to the plugin, but that's it. It's not communicated up to the user what environment variables the plugin expects or anything like that. You just have to know. So what we're trying to get to is what's on the left. So before I explain how we've gone about parameterizing plugins, I wanna tell you basically how you install one and it's pretty simple. On the left-hand side, you've got a config file and it sets the plugin's name to my plugin. It sets up a shell script and this one's really simple. It just spits out a config map and you can see it's got some environment variables interpolated in it. And then on the right side, after you've configured your plugin, you install that on a side car with an Argo CD binary and it fires up and runs that config. On the right side, you actually use the plugin. You can set the plugin name and then pass it an environment variable and we prefix annotation there with Argo CD ENV for security purposes, keep people from just setting arbitrary environment variables but that's how the communication occurs. So this talk is all about the bottom block of this new config on the left. So same deal, it just spits out a config map but now we've created what we call a parameter announcement and this is just telling the user we have a parameter called myparamname and we can give it a more human friendly title. We're telling the user and the user interface this is a parameter that we accept and you can see it's a slightly different environment variable that gets generated to actually use that in the plugin but same basic concept. And on the right is what your user will now see. They get the nice long human friendly explanation of here's what this parameter is for and then the user can edit it. You see the little puzzle piece icon that's just telling the user this is something that the plugin advertises that it accepts. If a user manually edits the manifest they can add whatever parameter that they want but this one is explicitly supported by the plugin. And that's what it looks like when you edit in the interface and then click to the manifest tab what actually gets persisted to the application resource is that parameters block and it sets the name, the more machine friendly name and then the string value that the user wants. So what I've shown you is just like the read only view and what Sindhu wrote up was the write support and I know it's a little harder to focus in a video but I think the edit support for the UI is pretty impressive so hang with me and I think you'll really like this. This is Sindhu. Today I will be discussing on the topic UI changes related to CNP parameters. So right now on the screen you see the plugin.tml file which is a P type config management plugin. So the main point of interest would be the parameter section. So you can see a list of parameters present in the parameters tab, the string param, array param and the map param and generally CNP accept the collection type string array and map. So I have an example for the three of the collection types. So I already have applied this manifest file. So now let's get into the Orgo CD UI to see the CNP parameters. So now I have selected the application plugin. So now I'm going to app details and under the app details you have the parameters. So right now all the parameters that are part of the CNP manifest or displayed here. So you can see the string param of type string has value value array param of type array has a list of values and map param of type map has a list of key value pairs and the icon, the puzzle icon says that this parameter has been populated by the CNP plugin. So this parameter is coming, this parameter and its values coming from the CNP plugin. You can see that for all the parameters it's the same description that it provides. So the values and the parameters are coming from the CNP manifest. Okay, so now let's see what all actions we can do on the CNP parameters. So right now when I click on the edit you can see the parameters in the edit mode. So you can override the values, you can add new values, you can delete values. So for example, for string param I can override the value to values for array param I can delete the value for map param might be I can add a new key value pair. Okay, so right now you can see that the values of these three parameters are overridden at the application level. So our application is the plugin. So we are overriding these parameters at the application level. So now these overridden parameters are stored as part of the application manifest. So right now, even before saving this I will cancel this and now you can see this is the application manifest and you no longer see any parameters here. So now if I go into the edit mode, okay, and now if I make some changes here, value three and for example, if I delete this key to value two, okay, and now I say so these three parameters all have got overridden at the application level. So these three parameters are stored as part of the application manifest. Okay, you can see that the changes being reflected in the parameters section. So all the new values showing up overridden value showing up and the deleted value being removed. So now if you go to the manifest, you can see all the parameters which have got overridden are displayed as part of the parameters in application manifest. You can see the string param, the value for array param, what are the values and for the map param you can see this. So now if you go to the same icon, it gives us info saying that this parameter has been provided by the plugin. So the parameter name is coming from the CMP plugin, but its value has been overridden in the application manifest. You can see that but it is overridden in the application manifest. The same goes here, the same goes here. And the other option that we are providing here is, for example, the user doesn't want to override the values but he wants to use the values coming from the CMP. So he can just do the reset. It goes back to the CMP value. For example, even for map param if I do, you can see those two keep value pairs being popped up. So now since this string param and map param have been resetted to the values in CMP manifest, so these two parameters are no longer part of the application manifest. Only array param is part of the application manifest. So when I save this one, you can see that in the manifest only array param, no longer the string param and map param are value or parameters of application manifest. So another one, what I want to show you is to add new parameters in the application manifest. So now let's take this, I will take the same example here. And now I go and I add the value. Okay, let's rename to array param one and let's delete this value three. Okay, now I save this. Now you can see the same parameter being shown in the parameter section. So array param one, which has value one and value two. So when I go into the edit mode, this is the same type of actions can be done on this parameter as well. You can delete this value, you can overwrite the values, you can add new values. Okay, and the other option that is present for this parameter. So this parameter array param one is not part of the CMP manifest. It has been introduced at the application manifest. So it doesn't have anything to reset back to. So the only option that is available for this, for the user is he can, like the user can delete this parameter. So when I click on delete, this entire parameter gets removed. So and it no longer is seen on the parameters UI or in the manifest UI. So if you go here and now if you check this under the manifest, you don't see array param one anymore. So that is a one. And so since the string param and map param have been reset back, you can see the this puzzle icon saying that this parameter is provided by the plugin and this parameter is provided by the plugin. But for the array param, it says that it has been overridden at the application manifest. Okay. So yeah, these are all the, so these are all the changes that I wanted to show from the CMP, from the CMP side. And this concludes my demo. And thank you all. In case of any questions, Michael would be happy to help. Thank you. So hopefully it's clear that the user interface is pretty powerful. Like the examples here are just examples, but I think you can see that if you're trying to present a config management plugin to your users, this is going to give you the tools you need to let them give you the information you need and present to them what they can give you. Not to play. I need to go to next. Cool. So what we've seen so far are all static parameter announcements. So say your plugin, you know you need three things that the plugin is always going to accept. Sometimes you're not going to know until you're looking at a specific repository what parameters that plugin is going to need. So this is kind of dense. This is a lot of codes, so let me summarize it. This code looks at a Helmfile repository. Helmfile just being another tool that's sort of built on top of Helm. And it finds out what parameters does the top level Helmfile accept. And then it loops over the dependencies of that Helmfile and says, okay, what parameters do those Helmfiles accept? And it spits all that out in a JSON document. So a JSON form of a parameter announcement. And inside of your config management plugin, you just change the parameters block to instead of having a static list, you call a dynamic command and you just call this script. So what that script is expected to write is what you see on the right side. A JSON blob with objects. And each object describes a parameter. And on the left side, I just have the struct for the object, so you can see what kind of fields it accepts. It accepts a lot more things than we've looked at here, so there are a few more features that we aren't yet taking advantage of. But that's the basic structure that you need to send. And now, so the user interface that we're trying to get to this sort of Helm look, so everything we're doing here is kind of informed about the idea that someday we want to be able to build our Helm support as a CMP, whether we switch over to that completely or not as a future question. But there are reasons why you might want to use a CMP from security isolation or just dog-fooding our own plugin feature to support Helm. So on the left side is what I would sort of envision a Helm CMP looking like. And you can compare it to the right side. It's basically the same experience. You can accomplish all the same tasks. And in the manifest on the left, that's how you configure your Helm chart using a CMP. Compare it to the right, which is how people do it with built-in Helm support today. Functionally the same. I honestly kind of like the map better than the list of key value pairs. So we've talked about how you get a user to set a parameter, but we haven't talked much about how your plugin actually uses it. So we wanted to make this as easy as possible for someone who's just writing, say, a bash script to use these parameters. So there's an algorithm that converts the name of your parameter, and suppose if it's a map parameter, the names of the sub-keys, into environment variables in a predictable way. And the algorithm's simple. We just uppercase everything, replace everything that's not alphanumeric with an underscore. So Helm parameter.autoscaling.enabled becomes Helm parameters.autoscaling.enabled. That's one way to use it. And that's the best way to use it. If you've got a simple CMP, it just accepts a few parameters. That's great. If you need a little bit more power, we also just have an environment variable that passes you the whole JSON object of all the parameters that the user has set. And you might loop over that to construct a list of, say, Helm flags that you use in the shell, or just however you want to use those parameters. Just as sort of a proof of concept, a super popular Argos CD plugin is the Vault plugin. I went and looked at their documentation, all the parameters that they accept, and then I wrote a quick script to present that in a user interface, because they accept about 20 parameters. And the only way to use them now is to have their docs on the right and Argos CD on the left and copy and paste the environment variable name. That's not great if your user just wants to get their app up and running. So I'm going to open a PR. I've got the code written up to add this to Argos CD Vault plugin. And I hope other plugins kind of follow suit and add support for this as well. And to wrap it up, so for the future of config management plugins, I envision a few things changing. The UI is already really, really good. I want to add support for things like hard fields or checkboxes and number fields to make it even more intuitive for the users. I'd really love, like I say, to have Helm or JSON support as a CMP. I think that plugins are going to be the future of how Argos CD generates manifests, because they do bring so many advantages. And also, I just want to see more plugins. On the right side, by far, the three most popular plugins add Vault support, add Helm file support. This plugin is something kind of funny and cool. It's focused on combining Helm and customized support. It also provides a way to chain plugins. So people, long story short, people are doing really cool things. So give those a check and talk to me if you've started writing another plugin or thinking about open sourcing one. I'd love to hear about it and about how parameters can serve those use cases. And finally, this is right now to better understand how users use all the Argo products, not just CD. We got to the parameterization project because we heard that users needed it. So this is the best way to communicate to us, besides just talking to us in the hall what you need in these products so that we can prioritize them and work on them. But that's my talk. Thank you again all for coming. Raymond, of course. I know you use the Vault plugin. Hopefully you like my PR. Hey, Michael. So I am using the Argo CD Vault plugin with Helm and other things. One of the things I've seen is I have a Helmargs parameter, which becomes a very large string. Is there like a way to render that in the UI a little nicer? Do I just add like new lines or do I need to put in a PR? You might try to use an array parameter. That way your user can just hit a plus button and then add hyphen hyphen whatever you need. It may be a little bit awkward because like you may need hyphen hyphen flag name and then another box under it to have the actual value for that flag, but it's still a little bit better than just a really long string. You could also use a map for key parameters. Thanks. More questions? Yeah, we went really generic with the UI on purpose so you can accomplish those things. It may not be perfectly pretty for each use case, but hopefully it at least covers it. Hey, never use CMPs. So I have a more general question about CMPs themselves. Maybe it's interesting for you. If CMPs actually extend or change in any way the manifest that I have in Git, doesn't that bring up the problem that now I have something in Git that's actually very different than what ends up being deployed? Yes, so I think that doing mine in costuses talk in the opposite direction may have actually really helped because yes, plugins often pull in additional sources of truth like vault. You have heavy, heavy transformations on the files in order to get something that you then produce as manifests. But I think that's okay if you do kind of what Costis was talking about, which is as part of your PR process, call the Argo CD API, ask for a diff of just the source files. It'll call the plugin for you. It'll generate the manifest for you and then you can comment that back on the PR as a diff and say what is the plugin actually going to do and what is actually going to get applied to my cluster. I actually wrote an Argo Workflows plugin to do diffs because I really like Argo Workflows for CI. I think it's a CI tool, please use it as a CI. But I wrote a plugin that does exactly that. Thank you. One question for you that I actually have thinking about this is we've been talking about adding OCI support in Argo CD to have like OCI as a source of truth. I think you could potentially check out OCI packages with this. But do you think that that's the right place for these to take place or do you think that there's going to be some separate service changes? I think other work has to happen. I was a little bit misleading on this slide. So repo server does pull sources and transform sources. Plugins have nothing to do with pulling sources. Technically, you can add credentials on your plugin sidecar and pull from wherever you want. But it's not really designed as a fetching tool. It's designed as a rendering tool. I would love to build a system for Argo CD to support arbitrary sources as sort of plugins. And you could hack it in right now. But that's not the direction I would go first pass at that feature. Excellent, thank you. Other questions? We have time for one or two more. Sorry, I'm not into... Sorry, my name is Belinda. I'm not into the CMPs or plugins, but we use Argo CD. My question is, is the plugin architecture easy to write plugins in any language? Or is it Golang? Or what's the interface for the plugins to be written and tested and working? Yeah, it's completely language agnostic. You pass in a command, a lot of people use shell scripts. Technically, we have a thin wrapper around those shell scripts, which is a GRPC surface. Technically, you could write your own GRPC service and put it on the sidecar in whatever language you want, just as long as it implements the API that Argo CD expects. Most folks are just happy with the command. But yeah, whatever language you want, just as long as it spits out YAML or JSON Kubernetes manifests. We've got more. Looks like we have two more here. Are we doing on time? We're good, we can do these. Hello. My question is how the CMP plugins work with multi-source applications? They don't. So, when we wrote the multi-source apps, we really wanted to solve one very specific use case, which is people want to put their Helm values file in a Git repo and use a public Helm repository for the chart itself. We got support in for that because we knew we needed it. Someone, I don't remember his last name, but someone has put up a PR to basically allow different sources to access files from other sources. I think it's a good PR. It would enable you to use say a plugin that calls Helm to use an external values file. It's a big PR and it's going to take a while to review and I'm going to need some help from the other maintainers. But we do want to move that direction and support that use case, yeah. Hi, I'm Felix. Thanks. Quick question. Is there a good place to go to online where I can see all the kinds of things that I want to add? We don't have a list right now. I've thought about adding one to Terry's from Acuity's awesome Argo list, which is pretty sweet. There are four that I really know about and three of them were listed on that last slide. The extra one is someone named Luke Juggery wrote another Helm file plugin and there's sort of that one and then Travis's Helm file plugin. There just aren't a lot of features like this are going to change that. And if you know of others, please tell me where they are because I want to create a list. Yep. Another question about the plugin. What's the best way of handing errors in the plugin and show them in a good way for the end user? Yeah. So do you need to shield them for security purposes or just like give something friendly to the user? Currently we have a make file plugin that's fail. It's very crazy output that doesn't save the end user anything. So we are trying to improve it. So if the command exits with a non-zero exit code, we send them the standard error and we also log the standard error. So something saved to send error should work then? Yeah. If you just pipe something to standard error, whatever you want the user to see and you could also potential like write logs to disk if there's something sensitive that you don't want to go back to the user. Yeah, but those are removed in the sidecar when the next one comes. So it's hard to get. Sure, yeah. It's hard to debug it that way. But it would have been super if it was a way to do it. Probably, I've got to think about how we could enable. Basically you want your debug logs and you want something that actually goes to the user. We could definitely build a mechanism for that. Nothing really occurs to me at the moment. I don't think it would be super easy. But that makes a lot of sense and I don't think it would be that hard to implement once we had the idea. Excellent. Thank you, Michael. Thank you all.