 Hello again. Um, this is Dan Garfield. We should, we should tell everybody, Hey, we're going to start the talk now, right? Yeah, there we go. Start off. Nice. Okay. So yeah, just to introduce ourselves, Luke. Yep, we felt some New York times. And my name is Dan Garfield. I'm the co-founder chief officer of code fresh and now running open source at octopus deploy as we've just been acquired. A little bit about ourselves. Again, I'm with the New York Times, 170 year old company that is deploying the Kubernetes today. At first, I thought that was impressive coming from America, but here in Europe, that's probably not as impressive. But yes, the New York Times is a building a digital first experience leaning into technology to reduce a comprehensive news coverage. There's many Kubernetes clusters throughout our organization, but at our core platform, we manage a few really large multi tenant clusters using Argo city at the core of all that as well. We've made some contributions back to Argo CD. We heavily utilize application sets, helping us scale our platform needs using all sorts of the generators within less matrix poll, etc. And just recently, the other day, we published the 1000th word puzzle. Does anyone have a 1000th wordle streak? If not, please go play the puzzle. And there's plenty of other new games out there as well. And then coming to introduce code fresh. So we're basically an enterprise platform for Argo. We do have Argo workflows as well as Argo CD and Argo rollouts. But we focus quite a bit on the software delivery use case. That's really our our bread and butter. And like I mentioned, we were recently acquired by Octopus deploy. And they're luckily they're doubling the investment that we're doing in Argo. So that's gonna be really cool. We're Argo maintainers. I'm an Argo maintainer myself. And I think in terms of this talk, obviously, we're going to talk about application sets. Through the lens of everything everywhere all at once. And within code fresh, I think our largest application set is probably about 5000 applications, isn't, isn't like crazy, but It's kind of crazy. Each one of those is a cluster also. And then it has application sets that are fired off underneath that, which we'll talk about a little bit. So anyway, it'll be interesting. I'm happy to be here and talk about it. This kind of started funnily enough on Twitter. Yeah, be careful what you tweet at. You might end up giving a talk. But yeah, we just kind of were joking about appsets, but we wanted to explore. And that's kind of how we got here. So just a refresher for those that, you know, let's just build up the paradigm here within Argo CD. You have the concept of an application and an application. You have a desired state, which is usually represented in Git. It may also be represented in Helm. If you were in the last talk in the other room, maybe soon represented in OCI repository. And then you have your actual state that lives on a Kubernetes cluster and Argo CD's job is to reconcile those things. And it does that with an application manifest, which is basically a policy for how you want that reconciliation to happen. Do you want things to be auto healed? Do you want them to sync automatically? Do you want to have manual syncing? So really an application is just a spec for the policy that you want for how your desired state is needed. And then with an application set, and how many people are already using application sets? So most people, you're familiar with this. Okay, so I can go a little bit faster. So an application, no, so an application set is just a way of generating those applications. And it's literally a Kubernetes resource. You can go and just look at it as an object after it's generated. Okay, so obviously with application sets, we've got tons of generators. I know you guys are all experienced, so we don't necessarily have to spend too much time on it, but list generators, cluster generators are obviously very popular. But we have these sort of meta generators too. We have matrix generator where you can combine multiple generators and say I want to take a list of resources. And then for every cluster that exists, I want to combine those into making an application for each one. And utilizing generators, utilizing application sets, there's many benefits. Simplifying your experience, automation, reducing manual effort rather than crafting every application definition. There's some automation here. Just a lot of benefits to kind of scale out your Argo experience. And so if application sets are so straightforward, all the benefits, there's nothing to fear, right? So let's just dive into maybe some accidents you could have with an application set. I wanted to share an accident I actually just had recently with an application set using the pull request generator. You never really want to see in your log somewhere that you've hit the GitHub API rate limit. But it is easily possible. We ourselves, in our use of application sets, we create an application set definition per tenant, per application. And we load those into our IDP and that gets distributed out to all of our services and teams. And so we created a simple template for a pull request generator. We wanted our ephemeral environments to be managed through pull request generator through Argo. And we feared no big deal, but we deployed this across some monorepo services. And those repos had hundreds of pull requests in them. And through just some manipulation of configs, we quickly were able to discover that GitHub has a rate limit of 15,000 requests per hour, 250 requests per second. And you can make the applications that controller, it can make all of those calls. You shouldn't let it do that, but it's possible. And a lot of times people overlook this because in Argo CD and the config map, you can set the reconciliation time for an application and that has nothing to do with reconciliation time for application sets. So if you set one, it doesn't mean the other one is set and you can find yourself in trouble. If you enable pull request generator, now the application set controller is making calls to GitHub, not just the repo server component. But this is kind of your simple pull request generator config. This looks innocuous, but say if you don't create a label strategy for all of your services, if you just kind of generically apply one label to all pull requests, you run into a problem. Anywhere you see like re-queue or queuing seconds, never touch those, never try to make Argo go faster is one of our lessons learned. If you ever need to increase responsiveness, always switch to webhooks so you can have the application controller respond to webhooks for pull request events instead. And so yeah, there was kind of a little accident, but beyond accidents, you can have some success. Yeah, one of the ways that we use application sets in CodeFresh that's pretty well documented, we give a talk on it at KubeCon in Amsterdam. So you can look that up. Just if you Google CodeFresh vCluster, KubeCon, you'll find it. But basically one of the, in our platform, we have the ability for people to deploy Argo instances, what we call hybrid instances, we call it CodeFresh GitOps. They deploy it into their clusters and those are managed individually. And that's probably the most popular way to do it, but we also have a hosted Argo offering. So if people can come in and do a hook, just click a button and they get Argo instance that they can point at their cluster and they don't have to install anything to go in. So for that to work, we actually use Argo CD, managing vClusters, managing nested Argo CDs that are then given to each user. And so that's an application set where we have thousands and thousands of them. And like anytime somebody creates an account and hits like start hosted cluster, it makes it get commit on the back end. And then it fires off the application set generator that fires off generation of the cluster and so on and so forth. So there's essentially nested application sets beneath that, if that makes sense. And so we've had some accidents and we've also had some great success with application sets. How else can we explore application sets? What potential can we unravel? We created a GitHub organization you can see linked here. It's called the Multiverse Labs. You can follow along with some of our experiments, findings and proposals up there. We've dumped out various logs and configs. There's a lot more experiments than what we're just going to show today. So starting off with like the application set template. Again, many of you are probably familiar with this. A standard application set has a template object. They utilize go templating within their template object as well as spreek function libraries are available next to the go text template functions. An early experiment of ours was to see if we could do some kind of app sets of apps of app sets, some kind of endless or recursive creation of applications of application sets. As the application generated by an app set must conform to standard Kubernetes rules, like a name being unique for resource within a namespace. And so creating, looking at the spreek functions, we're like, oh, let's grab the random alpha function to generate a randomly unique name for every app generated by an application set. This seems pretty straightforward, pretty quick test. But let's just see for those, because there's so many applications that users in here looking at this, raise your hand if you think this would be a good idea. Okay, thanks for being brave. Raise your hand if you think it'd be a bad idea. Okay, so most people just aren't paying attention then. All right, fair enough. So the lesson here is to consider your templates about determinism, what might happen. Well, within a few seconds this launch and I went from zero apps to 200 applications. And it just kept going. I had to kill the controller to stop it from endlessly generating. It turns out that the applications that controller will continuously evaluate the templates. And it's always a unique template with this random alpha being generated. So they never match every time it runs. And it will just spin off creating applications until you stop it to do that. So those that picked bad idea, you were correct. Spoiler alert, that's probably going to be the case for most of the things in this talk. So, oh, well, no, you go ahead. Yeah, so those kind of just are, as we're starting our experiments, we're just trying to see what some of the functions in the template do. And so, you know, from there we end up with all sorts of applications generating in unexpected ways. So the next thing that we thought of was, okay, we want to generate applications recursively, but we know we can't use random names. So we did come up with a format for that. So basically the way that this works is you've got an application set help chart. So this is a help chart that just contains an application set. And it's essentially blank. It's just a template for an application set, right? And then we can use that to generate an app that installs a help chart. And that help chart will be our application set template, which will generate an app to install a help chart, which will then install... You guys see where this is going? Okay. So we do need to use deterministic names. And so the technique we came up with for that was to use a shot. So what we would do is we would take the release name and we can actually just dump that into a shot and then we would truncate it to a length that's reasonable because we don't want to have infinite length on a name. And this meant that we had a deterministic way of passing a name to the next generation of application sets. And if you go to the next one, here's how that looks when it's loaded. We basically passed in that value shot and we use that in the name for the next set. So this way the application set names are always deterministic. And you can generate thousands of applications this way and they will always get the same name because you're always giving the same inputs. It's almost like Bitcoin mining in your application set in a way because you're just running a bunch of hashes. But it wouldn't work very well for Bitcoin mining. But at any rate, so here's what that looks like in a quick demo. There it goes. So yeah, we're going to create an application, right? And I'm just going to dump the YAML in here pointing at the multiverse repo. And when we sync this, it's going to generate a new application. And when we sync this, you can see that one just started off with a zero and now it's going to come in with the hashes. And if we sync that, it's going to generate another application. Now guess what's going to happen if we sync that one? Yeah, you figured it out. Good. So of course, manually, this will just go on forever. It'll just keep on generating applications. Now, you can actually use this technique of passing values to another generation to set limits so you could pass a counter and you could have an if in your home template that would basically stop the generation of applications once you reach the desired amount or a desired outcome. So you can actually build some logic into these in potentially in a very abusive way. Again, I'm not saying you should do this. But the fact that you can, it's good to know. So this, of course, if you just turn this one, this demo has manual syncing on, so we just have to click the button. So what happens if we turn auto syncing on? Auto syncs are best practice. Yeah, everybody should use an auto sync, right? So we turn that on and then it looks like that. And do you want to explain the behavior here actually? Yeah, you'll find in our Git repo just an example, kind cluster that bootstraps Argo and runs our experiments and so we drop this in, turn in auto sync and off and running it went. Within a couple of hours I had 2,000 instances of an application, one application after application set generating the next step for a while. The behavior was kind of weird as far as the application set generator just slowed down. It took like a few seconds to generate the next application and a few more seconds to a minute to generate the next one. But it would keep going. Probably if I had more power beyond just a little kind cluster it would probably go a lot faster in more exciting ways. Again, on our repo you can see all the dumps of the Grafana logs and things like that if you want to see what's going on. Next time we could hook up a GPU so that we could generate the hashes more efficiently and that would allow us to run a lot more application sets. Alternatively here we used SHA-256 but you could use SHA-1, right? And that would be faster. So there are kind of different ways to slice it. We can give more resources to the application set controller. Obviously our kind cluster, not the one, not super beefy, but a GPU maybe that will be next year's talk is using GPUs to create application sets. Well it was at least one of the nice features of Argo with auto syncing and also turning on pruning where it will clean up after itself ideally. So it was neat to be able to just delete the first parent application and that just deleted the whole chain down. So luckily it didn't totally crash out your machine but it worked out. But not just breaking things in the Argo application set. I'm going to show a simple example of perhaps mastery in an application set. How could you combine a variety of powerful generators to say search your whole enterprise for helm charts and deploy all of them? So as an example here, utilizing many generators, a matrix that joins clusters with a merged generator that joins together to get generators, it would be easy to extend this as well to lists and pull request generators. This is sort of a contrived example here but pretend this is your organization that has an extensive helm library of your applications and rather than defining applications for every chart or application sets for every chart itself, have an application set scan your entire helm library. We'll use the Bitnami chart repo as an example. I'm reading this right. Is this just looking at the entire Bitnami repo and it's going to generate an application for literally every chart that it finds? Beyond one. But then combine that with every cluster? Right, every cluster you have because make sure every cluster has every copy of the chart. Let's see what happens. I mean that's 20 applications top, tops, right? I don't know how many images there are in the Bitnami repo. I think there's 100 charts out there and one cool feature to show off here of the git generators is doing a merge on two git generators, one that just finds a path of every chart and one that finds the actual path of charts looking for the chart.yaml and the applications that generator will pull in that chart yaml as values you can use in your template. And so this is kind of a neat way to utilize app sets. You could instead say in your enterprise you can have say like an Argo.json file and set the git generator to go look for Argo.json files as an example and then generate the paths. But yes, putting together all this on a little kind cluster, let's see what happens. You get a fun little low test of your Kubernetes cluster again this being local things got exciting really quickly. This is kind of one of the first times I've seen an out of pods error in Kubernetes. I think you're limited to a thousand pods on a node by default. You shouldn't hit that, that's not a good thing. But this is things that can happen as well as just taking every chart in Bitnami it's all sorts of features of Kubernetes all sorts of databases and a variety of things. It will make interesting things happen to your system. So it's kind of a fun low test but beyond that it's kind of a really fun proof of concept though how you can in a very simple application spec deploy every chart you can find or use that in some other ways to manage your whole enterprise. Obviously you can put selectors on those things right so you can have selectors on those charts that you'd be looking for specific values and you can make decisions about what would automatically be deployed to different environments. So in that way it's a much more useful real world application of that. And of course you can extend these as well with one of the generators as a plugin generator so you can actually run your own container and just do arbitrary like whatever you want. The matrix of matrix anybody ever tried to do a matrix of matrix? One person? Did it work? A matrix of matrix? I think two is the limit. Yeah two is the limit. You can do two but you can't do more. There's a hard check in Argos CD that prevents you from going beyond that. We try. We want it to. They're catching on. They're putting in the safety blocks here. No for sure. But again if you check out our Git repo there's a variety of different combinations of generators we were trying out to generate some cool experiments here but some lessons learned here. For sure always test your application sets. Hopefully you have different Argo environments. Your dev test staging environments where you can run application sets. It's really easy to see how you can get into trouble with them quickly. So be sure to test with those as well as rely on good monitoring and observation of your Argo platform. In my earlier issue of hitting GitHub rate limits be able to alert that you've set something off that you shouldn't have and that you don't bring down your whole Argo platform. Definitely iterate on it. This brings up the value of doing application sets into a staging cluster first. That's obviously very important. With applications it's very easy to preview what's going to happen when you modify that application. You can see how the policy would change and generate the diff of your manifest and so you have a very clear picture of this is how this is going to change my environment. With application sets though you don't have that and even if you set an application set to do manual syncing for the child applications you may accidentally delete a child application that you wanted to preview. You can't preview that you've deleted it. Definitely testing locally is good testing in staging is good. I think that there's a lot of room in the community to have better preview different tools for application sets and that that's a real problem that over the next year I would love to see better solutions for. And finally don't take anything we did and put it into production. Or do. YOLO. Yeah. We have about a minute left so thank you. I think you can find us on Twitter or X. I'm at Today Was Awesome to just think how the day went and then also at CodeFresh obviously you can follow us there. Please review our talk and check out our examples here and again thank you for attending our talk. Thank you.