 My name is Aleksandar Mitushansov and I'm a Principal Software Engineer at Intuit and I'm a long-time maintainer of Argo. So, as Henrik said, I've worked on pretty much all of the projects and my focus right now is Argo CD. And my co-speaker is Jesse. He will talk about himself very soon. First few words about company I'm working for. So I work for Intuit, which is a US-based tech fintech company founded in 1983 and we are proud builders of products like TurboTax, QuickBooks and Mint. And so we serve 10,000 million customers from all the segments and Intuit is a CNCF Gold member. Next, I'll let Jesse continue. Hi everyone. My name is Jesse Sun and I am an Argo project lead along with Aleks. I'm also co-founder of a company called Acuity, which we just launched this week. Acuity provides vendor supported distribution of Argo as well as support and services around the Argo project. So before we get into it, it's always useful to start and explain what the Argo project is at a high level. And Argo is a collection of Kubernetes native tools and all focused on the area of application delivery. And it's comprised of four main projects, CD, rollouts, workflow and events. And each project is focused on a specific use case. And they're all designed to be used standalone with loose integrations between the projects. So you might be wondering how did we end up with four loosely coupled projects and then a collection of ecosystem projects. And it all comes down to some design philosophies that we share about the project. And so we believe that each project should stay focused on a specific use case and solving that use case very well. For example, Argo CD is all about delivering manifest from a Git repo to Kubernetes clusters. And a few years back we had this need to do the blue, green and canary deployments on Kubernetes. And there was this push to have that functionally baked into Argo CD. But the more we thought about it, it felt like it was out of scope from what Argo CD was originally aimed to do. And the approach could be considered too opinionated, maybe make it Argo CD too complex. And so we decided to develop it as a separate project. And that became Argo rollouts. And we felt like that was a great decision because today we have a lot of people who like to use Argo CD but they don't need Argo rollouts. And then we also have users who use rollouts but they don't use Argo CD. And so you kind of get to pick and choose the tools that make sense for you. And our second point is that even though we are focusing on single use cases, we still want this great but complete user experience. And this means adding those extra features that make our users' lives easier. And for example, that's why we have a pretty heavy focus on like UI because it just adds so much more to complete that user experience. And finally, as we're adding this new functionality to these projects, we still want to maintain that these projects be lightweight and as possible and avoid bloat. So a lot of the times these principles are kind of at odds with each other. Like we're always getting requests for new features, for valid use cases. But at the same time, we want to do it in a way that doesn't keep creeping and bloating up the project beyond its original scope. That adds to project complexity and fragility. So the question is how can we make improvements to the project but keeping the core projects lightweight and stable? So it turns out that Kubernetes has this pretty good mechanism for adding features to resources in an indirect way, which is annotations. And annotations allow us to effectively expand the spec of a resource but without implementing the functionality into the core controller logic. The whole idea is that you can write your own controller which monitors those same resources and then adds functionality based on those annotations. But we think one of the biggest benefits of adding features this way is that it allows us to implement features in completely independent projects. And why is this a good thing? Well, the first is you get a much higher development velocity. By having a separate project, you can iterate faster on that idea. You can have a completely separate set of maintainers. You're freer to experiment and make breaking changes without impacting the core users. You can give earlier access to those features since you're not tied to the release cadence of the main projects. And if the idea doesn't pan out, then it's okay to abandon it. Compare that to if we decided to implement that feature in the core project and if that didn't pan out, it's much harder to remove features once it reaches the core. This all helps to keep the main projects lightweight. As a separate project, the component is optional and so users who don't need or want that feature, they don't have to install it. And finally, as these projects mature and become more stable and gather momentum, there's always that option. We can always formalize that and bring it into the main project down the road. So we formed this sister organization. It's called Argo Proj Labs. And this is a place where we can host ecosystem projects from the community that complement the core projects. This is a place where the community can collaborate on new ideas, new features, but it also provides a way where others can discover these useful add-ons. And labs act like this incubator of these experimental features that we want to explore, but maybe we're not quite sure of. So as of today, we've actually accumulated over three dozen ecosystem projects. Some didn't pan out, but others are quite popular. And we're now at a point where we're considering including some of these popular ones as part of the core projects. And today, we'll be going over a handful of some of these projects that enhance the Argo CD experience. Alex will be giving a demo of how you can get more out of Argo CD with these add-ons. The first project we'll be demoing is Argo CD image updater. And this is a tool which monitors your container registry for new image tags that get pushed. And when a new image tag is pushed, when it detects that it's newer than what's running, the image updater can actually write back those changes into your Git repo so that Argo CD can deploy them. And the benefit of this is that your CI pipeline, it can remain completely disconnected from either your Kubernetes cluster or your Argo CD server, with all updates being facilitated by image pushes. Next, we have application sets. And the use case here is if you've been using Argo CD for some time, you might have felt the need to automate the creation of many applications. For example, maybe you run hundreds of clusters and you need that same application installed in each of those clusters. Or another reason could be maybe you have this Mono repo with all the things you need to install, and you want to create an application for every directory in that Mono repo. And application set is a controller which helps automate this process and create what we call sets of applications. And there was actually a talk on this yesterday by Jonathan Wesson and Shama. And if you missed that, I encourage you to watch that when it's uploaded. Notifications, this is a project that allows you to become notified about state changes of your applications. For example, with notifications, you can get a Slack, email, pager duty, notification, anytime say one of your application degrades, has configuration drift, or maybe every time just someone clicks a sync button. And we also have a deep dive on this service tomorrow that Alex will be talking about. And I think you'll want to attend this because this project is actually useful not just for Argo resources, but you can actually incorporate this into other custom resources as well. And the last one we'll be going over today is Argo CD extensions. And this is something we're really excited about. So Argo CD extensions, it provides a way for users to add new visualizations to Argo CD UI, and it loads it at runtime. So the backstory around this feature is that as heavy rollout users, we wanted a way to bring the rollout-specific visualizations into the Argo CD interface because if end users, if they're just looking at the spec, it's really hard to understand what's going on. It's not really user-friendly. And originally, we were going to give special treatment for Argo rollouts and bake that directly into the Argo CD code base. But the more we thought about it, we felt like we could do this in a more generic fashion and support custom visualizations for any type of custom resource. So we came up with this new architecture that allows users to build and package their own UI components for any Kubernetes resource and for that visualization to be presented into the Argo CD UI. And so now Alex will be demoing all these add-ons and you'll be able to see these things in combination in action. Thank you, Jesse, for awesome presentation. Let me stop sharing the slides. Cool. And I want to validate you see what I see. All right. And little preparation before the demo. Okay, so as Jesse mentioned, we are going to go through Argo CD-specific add-ons. And so we thought that one demo was the southern words and instead of explaining all the details and showing you slides, we can just show you all the these add-ons in action. So before I start the demo, I just wanted to highlight that this URL is a public GitHub repository URL that has all the materials you need to do the demo yourself. So if you're interested, just clone it and use the readme file. It has all the descriptions of all the steps. And one thing you would have to do is to go through some preparations. Of course, you're going to need some Kubernetes cluster and pretty much any cluster works. This is what I did. I just configured a K3S cluster, mini-cube would work or any other cluster. Next, you are going to need some credentials. So Argo CD is going to communicate with Slack, with GitHub. And so you will have to create tokens. And I wrote some detailed instructions about how to get those tokens. One for Slack, one for GitHub. And a small snippet to upload these secrets into the Kubernetes. And finally, you will need Argo CD because it's kind of a tool to drive the demo. And so it's just easier to do the demo and see what's happening using Argo CD. And just for the sake of time, I did it. I already have a training on my mini-cube. And so we are here right now. So we are ready to take a look at Argo CD UI and start using it. Let me go ahead open the URL. And so if you don't know, this is how Argo CD user interface looks like as soon as you install Argo CD. So there is an empty page, no applications, and we are about to create few. And before I create, I kind of want to explain like what am I trying to achieve. And so my goal is to use this GitHub repository to represent the target state of my cluster and manage it. And there are different ways you can do it. I chose to just create a folder for a namespace. And each folder here, so it represents a namespace in the target cluster. And it has some manifests. And so this is one example. So we have Argo CD folder. This represents Argo CD namespace. And as you can guess, it has Argo CD yamls. I didn't want to copy a bunch of yaml files, so I just use customize config management tool to pull a lot of manifests together. And this is it. This is the customize yaml file. It includes Argo CD stable manifests. It includes one installation manifest for application set, one for image updater, and for all other add-ons. And this is pretty much, you know, part of a demo. Like this explains to you how you can get a done installed in your Argo CD. It's like we're trying to make it as easy as possible, and in a perfect case it's just one yaml file. Okay, so next I think we're kind of ready to start using Argo CD and create applications and start, you know, pull these manifests into the target cluster. If you are a user of Argo CD, you kind of know one way to do it already. Typically you can switch to Argo CD user interface and create new application by clicking new application button. You would have to fill some information here, specify the repo URL, and so on. And this is maybe not the best possible way. You can as well automate this process using scripting, using Argo CD IPICLI, but I am going to demonstrate you how you can use application set to do it. And so I will just go ahead and use app set to create applications. I'm going to run a kubectl command just for the sake of time, and then now I'm going to switch back and explain what I just did. So I just kubectl applied a file, and that file is in the same repo, and it has application set spec. And so this application set is supposed to generate applications in Argo CD for each and every folder it finds in the Git repository. And so if you remember, I had four such folders, and it's supposed to create those applications. The folder name goes into the application name, as well as into the target namespace. And plus I configured these apps to self-sync. Like as soon as applications created, they should just create resources. If namespace is missing, the namespace will be created. And hopefully if all went well, yeah, we've got four applications as expected. So application set created these apps for us. And so we have one for Argo CD. We have Argo rollouts. We have Ingress, because I needed it for my demo. And we have one application that I want to focus on. It's a demo app. It's still syncing. I know the reason is basically my Ingress not yet up, and Argo CD keep basically retrying to create Ingresses. So that eventually, basically as soon as Ingress controller mutating hook is up, Ingresses will be created. Done. So basically, yeah, I should step back, I think, and explain what you are seeing. If you are not a user of Argo CD, you might not know, but this is Argo CD application details page. And the goal of this page is to explain to you which resources are part of your application. And so you might see here that we have two Kubernetes services. We have a couple of Ingresses. Nothing fancy. One object is interesting. It's an Argo rollout object that manages one replica set and few pods in the cluster. And you can use Argo CD UI to learn more about these resources. So if you click on the icon for that resource, you out of the box get some information about your resource, including the real manifest. And this is pretty much enough for you to understand what's happening with the rollout, except there is a caveat. You must know the YAML structure. You have to scroll down and maybe find status of this rollout and understand what it's doing now. You will have to go through the spec to understand what will happen when update happens. And so that requires, it's not the best possible user experience. And we can definitely do better. And thanks to Argo CD extension addon, we have this new tab called more. If you click on it, you will get a rollout dashboard that visualizes the state of a rollout. And I guess I really think this is like way better because without looking at YAML, I immediately know that this rollout is supposed to use canary deployment in case of a change. It has eight steps defined in the canary strategy. Right now, we have just a single revision with five pods. And I have all of them, all these pods uses this particular image. Argo approach rollouts demo and tag is blue. And it's important to remember it because we will be changing it soon. And so we also have visualization of steps. It's kind of not very interesting now because rollout is not running any canary. And to demonstrate your real power of that UI, I want to change this rollout and start the canary upgrade. And you will see how it's really useful to use this UI to understand what's happening with rollout. And of course, there is a simple way to make this change. Maybe use kubectl edit. The GitOps way is to go into the Git repository and make a change in a YAML file. But we are here to see the power of addons. That's why I will be using Argo CD image updater. And just as a reminder, the goal of the project is to automatically make the changes in Git for you. And I will have to run a couple more SLA commands to do it. I will do it now and explain what I did. So the first command is just a kubectl patch that disables autosync. I do it for the sake of demo so that you can see what is happening so it would not be instantly. And the next command, it's another patch to the application set that injects a set of annotations. And as Jesse mentioned, we use annotations to enable the features provided by the addons. And let's take a look at the annotations. Annotations are here. So I have a JSON patch that was used in the kubectl patch command. And so we have not so much stuff here. We have two groups of annotations. One explain image updater, how exactly images should be managed. And next another annotation for notifications that will enable slack notifications. And let me go through it a little bit in more details. So this particular annotation explains to image updater that we want to manage this image. And the next annotation explains that the changes supposed to be committed back to Git. And I chose to commit these changes into a separate branch just so that you can see it before it gets merged. I also specified a pretty simple update strategy called name. And what it does, it just lists all the tags, compare them as strings, and pick the latest. So it's not real life. And basically I would encourage you to use something else. For example, Senvure. So you can, if your tags are following Senvure notation, I would use that. And finally, notifications. Even simpler than image updater. So what that annotation is saying is that we want to get notified about the syncing process on an application. And message should be sent to Slack into that channel. So it's pretty simple. And the goal is basically these annotations and users supposed to apply. And we're hoping that it's a simple enough experience from a user perspective. Okay, so next check. So because we applied these annotations already, the work should be done already for us. And we should have get a change in the branch. And luckily it happened. So we GitHub detected a new commit in a branch. If I click create pull request button, you can preview what is about to be changed. And you will see a Argos CD application specific file that has a set of settings that explain to customize that image should be, the image tag should be changed to yellow. And I'm just going, I like it. Basically this is what I expected to see. So I'm going to approve and merge this PR. Nice. And we are ready to go back to Argos CD. So Argos CD checks Git every three minutes by default. I don't want to wait three minutes. So I'm just going to click refresh button, give it couple seconds to detect a change. So we can use UI to preview what has changed. And basically as expected, the image is no longer matching to what we have in Git. And Argos CD did nothing so far because I disabled automatic syncing. And so I will just sync it manually. And hopefully in within few seconds, it will actually apply all the changes. And rollout should notice the change. Yes. And it should start, you know, executing the Canary strategy. And this is the moment where you finally can see the power of this UI. So it's now self-explanatory that there is Canary upgrade happening live. And so this panel now make more sense. So you can see that we are at the step two, actually. First is already completed. And so the first step was supposed to roll out. It was supposed to create a new version of application, but it should have only 20% of all the ports we have. And you can see that we did get secondary vision, which has only one port. The previous revision has four. And right now we are at the stage where rollout waiting for human approval to approve and progress the rest of the Canary upgrade process. And I promised notifications. And we've got them. So a minute ago we got two notifications. One about the start of the rollout process and second about successful completion of rollout. Canary step number one. That's it. That's the end of the demo. Before I switch it to YouTube for questions, I wanted to show you a couple more slides. Yeah, so demo is done. And I wanted to share with you some information about what it takes to create an extension and how you can work on it to add visualization of custom resource you care about in Argo CD. So of course you will have to install the addon itself. And this is the first URL. It's just a pointer to a readme file which has installation instructions. The second URL is a sample extension that you can use as a quick way to get started. So you can just fork the TRIPO and start writing code. You would have to write some JavaScript and if you know React, it should be straightforward. It shouldn't be very difficult. If you want really fancy extension, we can help you as well. So the last URL is a GitHub repository that has a set of reusable React components that we already used to build Argo workflows, Argo rollout and Argo CD UI as well as Argo rollout extension UI uses the same library. And so what that library is, as I said, it's a set of React components and all of them look like Argo. That's why you don't have to repeat the same CSS styles if you want your extension to be kind of fit nicely into Argo UI. And this is not the end. We are really excited about that feature and we plan to keep working on it. We're hoping to get more extension types, the ones that you can embed into a sidebar, application level extensions and we are already working on another way to extend Argo CD. We call this feature Config Management Plugins V2 an idea that we want end users to be able to add support for Config Management tools such as CDK or TANCA by Grafana in Argo CD UI and get kind of almost first-class user experience. And this is it. This is the end of the demo and the presentations. Thank you for listening. Thanks, guys. I have a microphone here. Questions raise your hand. I'll try to run around. We'll get the first one over here. So thank you. I just wanted to ask what is the difference between Argo CD notifications and Argo events? Actually, Argo events are to consume events and create a Kubernetes resource. That's how I understand it. So idea that it can connect to a lot of sources and eventually create a resource and I guess the most frequently created resource is Argo workflow. And so Argo CD notifications kind of only support one source which is CID that it watches and, you know, react on changes. But it kind of... So you cannot select a source but you have a variety of destinations where the notification can be delivered. It could be Slack, Telegram. Actually, there is more than a dozen already including, I don't know, up to the JIRA and GitHub integration. And we actually have a talk about this tomorrow and encourage you to visit it. You showed us earlier the Canary percentage in the dashboard. How do you do that with Argo CD? Do you play around with the number of replicas in Kubernetes or do you maintain some traffic routing or how do you do that in Argo CD? Yeah. So this is actually powered by Argo rollouts which you don't need Argo CD to do it. But the way rollouts works is it does integrate with, like, Ingress controllers or service meshes. But for what I call the basic Canary, you don't need that in the way it works. If you want a certain percentage of traffic to reach the new version, you can think about it as we just deploy the percentage of overall pods of that new version and then we pause. So I like to describe it as a way that a slower controlled rolling update. And that's how we... That's a basic Canary for rollout. Yeah. A question. So if we use Argo CD extension, is it possible to utilize Argo CD RBEC as well because Argo CD already have RBEC mechanism, right? I guess that was one of the benefits we wanted to have. So you don't have to, you know, reimplement your own RBEC. You can rely on Argo CD and it won't let users who are not supposed to, you know, observe applications that they won't see and by extension. Hi. Thanks for the talk. So we're just starting to look at Argo CD and one of the things that, from security perspective, you wanted to kind of see if we can build our own extension or like what we can do is basically we don't want our Kubernetes clusters or anything running on our cluster be able to talk to our code repositories, right? Because if the Kubernetes cluster is compromised, you know, the surface here is huge at that point. So what we wanted to do was, you know, have our repositories push our manifest, right? We could use customize or whatever it is. At the end of the day, it's a bunch of YAML. Push that to S3 and then Argo source of truth now instead of GitHub or code repository becomes S3 and it does everything that it does just like replacing GitHub or whatever it is with S3 and I was just wondering what are your thoughts on that? Yeah, so Argo CD already supports at least two type of sources. One is Git and second HelmChart and the criteria like we wanted in order to support the source, we need to make sure this source has some notion of versioning. Basically, and S3 I guess it's like, it's a hard requirement for S3. I think like in your case, I would recommend to run a sidecar in Argo CD repo server. That sidecar can, you know, continuously download S3 bucket content and if it is, let's say, a terrible of like Git repository you can unzip it into, you know, local file system and point Argo CD to that file system. And I just know that trick because I think I spoke with one of the users and they did that. So Argo CD can, it doesn't care where your repo is. It can be just on a local file system. First of all, thank you for building Argo CD. It's a great tool. We're just curious in terms of extensions, have you given some thoughts to secret management? I know there are already some tools out there but it feels like Argo CD is quite comprehensive but secret management is missing. Yeah, I can take that. Yeah, secret is a common issue. I think I would say overall with just GitOps it's not necessarily Argo CD. And because every organization has their different postures and tools and stuff that they prefer to use for managing secrets and it's possibly easy to get down on like a rabbit hole of adding integrations, right, to like followed or a secret manager or all these things and thus far we wanted to avoid that integration rabbit hole. That's also the reason why notifications spun out as a separate project too because there's no end to notification services that you could end up supporting. So I do think there's a future where there is something like a Argo notifications but for secrets, right, like an extensible way where organizations can bring in what they want to use for their secret management and plug it into Argo CD. We just didn't want to build that direct integration into Argo CD core because we actually learned our lesson with config management where we first support a case on it and then JSON and then we can start getting requests for tanka and all these things. So the idea is that we want to do this in an extensible way where we don't have a strong built-in vendor support like for specific vendors. There was also a session yesterday called It's a Secret, Managing Your Secrets to Get Ups Away talking about an Argo CD vault plugin. So if you missed that, that might be worth looking into as well. And with that, I think we're out of time. So if you have any more questions, please take them outside, clear the room, and we'll continue the discussion outside. Thank you, everyone.