 Hello, welcome everyone today we're gonna be talking about Argo CD and Argo rollouts extensions. My name is Leonardo I go by Leo and I am a staff software developer at Intuit and I am one of the maintainers in the Argo CD and Argo rollouts projects. Want to introduce yourself? Yep, my name is Zach Aller I'm a senior software engineer at Intuit as well and I am one of the lead maintainers for Argo rollouts. All right, thank you Zach. Zach and I we work in the core Argo team at Intuit and we share our time between maintaining the open source projects and improving the Intuit infrastructure. In this slide you can have a perspective on the size of our infrastructure and the complexity behind it. In our team we're constantly asked to keep improving this architecture, this infrastructure and add new features improve the user experience. However, some of those initiatives are really Intuit is specific and the answer to that that not only the two of us but our entire team is related to is how to build an extension points in the two projects that we deal with on a daily basis so Argo CD and Argo rollouts. And with that I'm gonna pass over to Zach to provide you the agenda for the day. So as Leo said a lot of times a year trying to extend Argo rollouts and Argo CD for Intuit specific use cases. Today we're just gonna go over basically six extension points, three of them from Argo rollouts and three of them from Argo CD. You can see what they are here. We'll kind of go through them as we go. I'm gonna start off with Argo rollouts in the beginning. We'll start off kind of a high-level overview of what the Argo rollouts plugin architecture looks like. Argo rollouts as a Kubernetes controller when it starts up it knows how to read it to take their config map and in that config map it has information about where to download these plugins. So it goes and downloads or copies those plugins to a well-known location within the rollouts controller and it starts those plugins up as a child process of the controller. At the heart of Argo rollouts plugins is they they're basically just a RPC server that the controller then is able to make function calls to. That config map that I talked about looks like this. It has a couple of the same fields in them. All of Argo rollouts plugins are configured the exact same way. You'll see we have a name which is the name of the plugin, a location and an optional SHA-256 hash. We support both downloading from HTTPS as well as using file locations. If you do use a file location it's up to the Argo rollouts admin to basically mount that executable up inside the container. The first extension point that I'm going to talk about is traffic routers. So for those that aren't familiar Argo rollouts traffic routers are basically the tool that gets used to actually shift the traffic from the stable to the canary pods. It can be anything from it's EO mesh to Amazon ALB to Gateway API etc. These plugins are pretty tightly correlated with in your rollout manifest you can call like a set weight step which means send set 20% of the traffic to the canary pods. As a plugin author to implement one of these plugins you basically have to implement a go-link interface. There's a couple function calls that we'll go through here and look at these real quickly. For brevity of the slide I left out a lot of the arguments and the return signatures just to clean things up. But one of the very first things you have to implement is the knit plugin function and this basically gives you as the plugin author the ability to instantiate clients, long-lived, cube client, any type of long-lived process that you want in your plugin that you only want one instance of can get initialized there. We have update hash. The argorollouts controller will call this when the replica set pod hashes change. Basically a deployment has just been triggered etc. You have kind of the gist of what the traffic router plugin system is all about and that's the set weight function. This directly correlates to when argorollouts is configuring a particular weight to your canary. It calls this which then the plugin's job is to go configure whatever external system is out there to configure the correct weight. Rollouts also supports header routing and mirror routing. Those are other similar functions to set weight just with different feature sets. We have verify weight. This is a pretty interesting function to implement. This generally gets used today for like an amazon load balancer. When we set the weight of the ALB to 20% let's say, verify weight periodically gets called to make sure that the ALB is actually configured to that weight before moving on to the next step. There's some interesting use cases that we'll talk about with verify weight a little bit later. And then just some management stuff. Some remove managed routes which is for mirroring and header and then type which just returns a string. How you configure these plugins within your rollout object is you have your traffic routing instead of doing some of the built-in stuff like it'sio etc. You can just use the plugins field and you see the name there and then basically any config that your plugin would require. So why would you want to create a traffic router? Well there's obviously the very simplistic cases you want to add support for Azure load balancer or some other type of service out there that we currently don't support internally. There is also a whole handful of kind of calling them untraditional uses. Argo rollout supports configuring multiple traffic routers at the same time. Yeah the weight is off. Okay we're back. Yeah okay so supports multiple traffic routers at the same time so you can basically have your normal one that's actually doing traffic and then piggyback and only implement the verify weight function. Not all the functions have to be implemented. And in that you can do things like log-based verification where you query some logging system to make sure that the traffic percentages have been met before going on. You can sync multiple rollouts to make sure that multiple rollout steps are both at the same spot calling third-party systems and APIs etc. So that was traffic routers. One of the other extension points would be analysis runs. So for those that haven't used rollout analysis runs are the feature within rollouts that lets you that tells the rollout controller when to abort a rollout. Today that's generally used. It creates a various metric system. You only look for error rates etc. and if you're a canary you have the higher error rate that will import it. To implement one of these is very similar to the traffic routers just maybe a tiny bit simpler. So we have a net. We have our run function which is where you would put you know the meat of what your plugin is doing. You have your resume terminate garbage collect and type. Those are kind of managementy type plugins that get used today for the job provider which is a feature in rollouts that lets an analysis run spin up a Kubernetes job. So you can imagine like garbage collect basically cleans up the old jobs and things like that. And various providers you know as a plugin author you can basically use these these hook points as you need. One of the more interesting hook points would be the get metadata. So a lot of times in your analysis run you'll want to return some form of metadata back to the analysis run Kubernetes object. Rollouts will call this function and then you as the plugin author are allowed to return any metadata that you want to save in the analysis run template or the analysis run resource. Today this gets used is in Prometheus we render out the Prometheus query. We substitute all the arguments that you have passed into your query and give you a nice copy and pasteable version of that. But as a plugin author obviously you're free to do what you want. To use a plugin provider in your analysis template you basically do the same thing. You have the name of the plugin and then any particular configuration that your plugin would require. So why would you want to do this? A couple lightweight things you know the true stuff you have some database storage that your metrics live in that aren't currently supported and you want to add support for that. An interesting one that I kind of like would be the Argo workflows plugin. So today the rollout support spinning up jobs. Jobs are pretty simple it would be kind of need to have an Argo workflows plugin as an analysis provider that would allow you to spin up like a cluster templated workflow to do either load testing or checking to make sure your canary is okay etc. And there's a whole slew of other uses that you know people will get created with. And the last but not the least this is currently in development so it's not set in stone yet but it would be the ability to have a step plugin so within rollouts you know you have your rollout and you define how you know setting 20% of your traffic pausing setting 15% of your traffic continuing you have all these steps that you define in a rollout. So step plugins give you the ability to have a hookpoint to run your particular code as one of your steps in your deployment process. So the easiest one yet is you basically and this could change of course is you have your init plugin your run what runs whatever code you want and you know is the step done and that's kind of what the interface is looking like today. And that's basically how you'd use it within your rollout step definition you you know call your particular plugin pass in any type of config that you would need for your plugin and then our rollouts would run that as at that point in time. So there is a whole bunch of interesting use cases for step plugins. These are just some of the kind of top of the head ones that haven't really been fully thought out but ideas that we've talked about like multi-region traffic control a lot of traffic providers support you know multi-region traffic control and things like that. So you could have to you know start of your rollout you could shift all your traffic over to another region continue your rollout you know and then shift traffic do your validation chest and then shift traffic back and then kind of roll out on the other side. You know you could gate rollouts you could make sure that certain conditions or your change management system said that it was okay to you know do this deployment. One of the interesting features that kind of coincides with traffic routers is the traffic router plugins is traffic routers have a lot of diverse features set and so you might have a plugin that implements like Gateway API or some other mesh provider and you have a very particular step or a feature of that traffic router that you want to use as one of your steps. Now you can actually create a co-plugin for that particular feature whether it's you know doing header routing or controlling error rates or retry policies a feature of that particular traffic router plugin that you implemented before so they kind of go hand in hand. You have feature flight canarying you know you could call out third-party services to enable features instead of carrying traffic you know you can carry features and control right there infrastructure validations check to make sure certain things are in place before you know proceeding to the next steps etc and there's also this GitHub issue 2685 people have been contributing ideas and feature requests for step plugins so if you are interested or have ideas please go and add your two cents. I'm now going to that was a whirlwind tour by the way but I'm not going to pass it off to my colleague Leo and he'll talk about Argo CD plugins. Thanks Zach all right so today like Zach I selected three extension points in Argo CD to present to you today and the first one I want to talk about is what we call a config management plugin or a cmp for short so config management plugin is so config first of all config management in Argo CD terminology and this wasn't really defined by Argo CD but config management is really the task of generating the manifest for your application right so in Argo CD we have the feature to allow users to define their own method to generate manifests right so Argo CD ships natively with customize and helm so why would someone want to write a cmp cmp plugin so customize and helm are not the only two tools to generate manifest there are plenty of others in the industry so maybe your company is using one of those two tools so cmp plugin can be an answer to that if you want to use Argo CD internally so if you have specific manifest configs you can also leverage cmp plugins so for example you can configure a docker container and have helm inside that container and have some additional logic that the native helm doesn't provide and you can implement your own it really it's really really use case specific okay so maybe moving forward I'd like to present to you the the Argo CD component architecture and I like to show this diagram here because it gives you an understanding of where those those plugins live so in the case of the cmp it lives in the repo server so repo server is the Argo CD component responsible for generating manifests and you would install cmp by adding a sidecar container to the repo server and configuring it in a way that when Argo CD needs to generate manifests for a given application it will delegate to this additional container to generate the manifests all right some interesting ideas and this is something I wanted to bring up today Argo CD has a different different feature called multi-source so today the way multi-source works it allows a user Argo CD user to define different sources for generating manifests for a given application the problem is not the problem the reality of multi-source today is that those two sources are independent so there is an ongoing progress process it's an open PR to make the multi-source work in a chain manner and what is the what are the use cases that this would unlock is the fact that for example you're using a helm chart in your in your in your company in your infrastructure that doesn't provide specific configurations a real helm chart it's really about the helm chart author to specify how that what are the possible configurations that can be defined but if if that's the case though if the helm chart doesn't provide you that you can write a cmp plugin to do this less mile manifest manipulation all right so moving forward the next plugin I wanted to speak about today is the app set generator plugin so different from cmp app set is all about generating our Argo CD applications right and app set will automate how Argo CD applications are created so it already ships today with a big collection of generators but it's if in your case for some example one of the provided generators does not address your use case you can implement your own generator so that's what we call app set generator plugin and why would someone want to implement an app set generator is basically to address specific companies needs right so if you if you for example to generate Argo CD applications you need to call an API that is only internal to your company in order to get some data some labels or maybe the the server URL you can do so by leveraging the application set plugins okay so going back to the same component diagram where does the app set generator plugin lives it lives in the upset controller so the way to configure it is by defining a config map that you can reference in your Argo CD up sorry in your application set uh crd oops let's do the dance again and hopefully it's gonna come back space all right okay we're back and yeah and the way to configure app set plugin is as I was saying right so you define a config map and you reference that config map in your application sets your d and the controller will handle that generator all right so the last plugin I wanted to extension point I wanted to talk today is UI and proxy extension this particularly I've been involved lately and specifically on the proxy extensions but that's not good sorry about that all right so UI and proxy extension so what it is so it's all about improving the Kubernetes resource visualization okay so this is all about the UI extension so why would someone wants to do that well one of the most powerful features of our city is the is the UI that we believe it's the UI so it allows users to have a much more intuitive view on how their applications are behaving in Kubernetes it's a lot more user friendly and it provides better insights on hidden info in YAML so if you know Kubernetes a little bit you know that it has that status field and oftentimes it has valuable information in that particular attribute so someone can write UI extensions to read those status fields and provide a visual inside our city so it will have an additional tab and I'm going to show a few screenshots that we run in production today all right so going back again on the component diagram where the proxy extension lives so proxy extension is all about the API server so API server is what serves the UI and the way to configure UI extensions is specifically mounting a react component on the article CD API server but you don't have to do that manually we ship some docker images that can be used to to simplify that installation process so let's move on to some screenshots I brought up to you today so I'm going to present two two extensions that we're running in production today add into it so the first one is the rollout extension so when users go in our city why and they when they click the Argo sorry the rollouts resource they will be able to see a new tab called more to see more information about that particular rollout so this is what I was mentioning before of providing a more intuitive view instead of looking at the YAML and try to understand what that particular rollout is behaving you can have a visualization right so right now we know right the way that this is a kind of a rollout it has 23 revisions it has a few steps and so forth this is a pure UI extension and I didn't mention exactly what is the difference because I wanted to reach this slide first so this is a view of the matrix extension that we are also using in production and into it in all our clusters in all Argos all into it applications and this this is a special type of extension because it leverages this this functionality that we call proxy extension so imagine that for for UI extensions it has full full access of the native Argo CD API but if you want to build an extension that doesn't doesn't necessarily can be addressed just by reaching the Argo CD API you can use proxy extension and the matrix extensions is a very good example on that so here we have a backend service running in every single cluster add into it that serves prometheus matrix extracted from prometheus to through Argo CD API okay so I don't have a lot of time today to there there's a lot into this and today we don't we don't have much time left and if you want to know more about UI and proxy extensions you are invited to join our talk at KubeCon where where Remington and I will do a dip dive on specifically on UI and proxy extensions and show you how that looks behind the scenes all right and that's it for our talk thank you very much this QR code has the all the links related to this talk not only one specific link so it will open a page where you can find everything related and the last one is to provide feedback if you want to provide feedback feel free to do so and we can we have a few one minute for Q&A if someone wants to ask a question if not we're we're going to be available for chatting person as well thanks everybody