 and we're live. Hello and welcome back. My name is Robert. I'm part of the SUSE and Rancher community here at SUSE. You're here for today's master class, and I think we're talking about Rancher because we have Dan Garfield on. Dan is from CodeFresh. I don't even know your exact title. Actually, we have an introduction slide. We can go to that, but Dan, welcome. Thank you. Yeah. I'm glad to be here. All right. Today's class is GitOps with Rancher and Argo CD. And Dan, again, graces for this time. We've actually talked a couple of times before. It's usually do partner interviews with him that you guys have seen before at KubeCon. We've done in the last two KubeCons, both in Valencia and in Los Angeles. So this would be nothing new for our community, but with no delay, we'll just get started. Dan, you want to start sharing? Go through all of the... There we go. Yeah, you should be able to see my screen now. We're all set, and I've got my Argo mug here. Yeah, and I got my Argo t-shirt. We're representing properly. Even I got an Argo t-shirt on today, and I know it's blasphemous. Everyone knows that I only own Rancher shirts, so this is the first shirt I have. It's not Rancher, so I'm proud. So I'll be watching the chat as we go through and feel free to make comments and ask questions. And I always think that these things are way better when they're conversational, when people are engaging. So as you're looking at stuff, as you're seeing stuff, as you're having questions, put it in here because it makes it a lot more interesting. I see we've got people joining from Egypt. I think we've got people from all over the world joining, which is always lovely to see. So to introduce my... Well, I guess we have all of these... Okay, a few... Yeah. So a few slides here, so yeah. Here's some housekeeping things. The platform we are using is called Crowdcast, and below you can see the chat. Some people are using it. If you feel comfortable telling us where you're from, please do put it in the chat. We'd love to... I love telling my kids where I talk to people around the world and not just other Ohioans. There is a poll. You'll see those come up, I'll be asked. Or if you have a question, please ask a question and upvote and downvote it. I will put it in polls, so if you see those, I will put it in chat to say, hey, go answer these things. It helps us keep the conversation going. So next slide. This is a master class. This is 45 to 75 minutes, depending on the questions. We welcome questions as much as you got. Please keep them on topic. So if it's a little off topic, we'll just have you route those questions over to the community. We'll get them answered over there. We try to get them all, but if we don't, and we'll get to you within the community, we always answer these things. This is being recorded. You will see this on YouTube here when I get the video cut and cleaned and put up there. So this will be up there. So if you're asking those questions, it's already there. I think that's it from my side. I think it's your slide, so I'll do it. Yeah, okay, perfect. Let's get into it. So, yeah, session's gonna be recorded on YouTube. So let me introduce myself. My name is Dan Garfield. I am a co-founder and chief open source officer at CodeFresh. We are an enterprise Argo company, and I am an Argo project maintainer. Truthfully, I don't do a ton of code contributions these days, but do help out just a little bit on some of that stuff, more on the community side. So less interesting probably. We're gonna be talking about avoiding configuration drip. So just to back up on this talk title. We're gonna be avoiding configuration drip with Argo CD, and a lot of you folks might be thinking, what is configuration drip? Don't worry, we're gonna cover it. And this is really about GitOps at the end of the day. So we're gonna even up level it even a little bit more and get deeper as well. So we don't need to go through the... We're gonna cover just a brief intro to Rancher, explain some configuration drip. We'll talk about GitOps. We'll talk about bidirectional sync and what we're really talking about. It's more unirectional sync now that I think about it. Self-healing Kubernetes clusters, we're gonna get into a demo and we'll do Q&A and we'll be rocking and rolling from there. From a Rancher introduction slide, and I don't know if you prefer to talk to this, but Rancher has like a ton of tools and most of you are probably already aware of this. We've got Rancher Kubernetes Engine. We've got the Rancher service, which allows you to manage many instances of Kubernetes and many destination clusters. You've got Fleet, which is part of that and provides not only authentication services and proxy services, but allows you to go pretty deep in managing really Kubernetes at scale. So if you have a ton of Kubernetes clusters, this is where it becomes really interesting. The first place that I saw this, so there are a couple of places actually that I see this. First off, if you can see behind me this stack of machines over here, this is actually running Rancher K3s. So this is my home lab cluster and I got a couple of HP Elite Desk, I think G3s or something, and I got them all off of a university surplus. I don't know if you all have that where you're from, but if you go to a local university that you usually have surplus and you can buy machines on the cheap. So I upgraded my Kubernetes cluster recently with a bunch of surplus machines. So now I have a lot more power than I used to. Before that, it was all atomic pie clusters. So I've got Rancher K3s running as my Kubernetes distribution, and then you have these other services from Rancher that allow you to manage scale. Where I first saw Rancher really being used at scale, when we were organizing GitOpsCon, we invited Chick-fil-A to come and give a talk, and I think you've all probably seen Chick-fil-A give a talk. If somebody in the chat wants to throw their GitOpsCon talk into the chat, they also recently presented at the Argo Meetup. You could throw a link in there if somebody else wants to grab it. But what they're doing is they're using Rancher with Fleet with Argo CD, and that's really the basis of where I came out for this talk was looking at what they did and the success that they've had and saying, okay, well, how do we introduce Argo CD to more people and help them understand GitOps and make this more accessible? So what they essentially do is they use Argo CD, which is a tool we're gonna introduce. Many of you are probably familiar with it. It's the world's most popular and most used GitOps tool today. And they use it with Fleet and what they do is they basically have in every storefront. So for many of you I see, you're from Germany, you're from Brazil, you're from South Africa, you're probably not familiar with Chick-fil-A. Think McDonald's, except better chicken, more focused on chicken. But it's a restaurant, it's a fast food restaurant. And they have a Kubernetes cluster in every store. And what they do is they have, all of those clusters are added through Rancher Fleet and they can basically manage the rollout across all those clusters by grouping them into regions and things. So they can say, okay, let's update the version of our software in this region and then we're gonna do it in this region and we're gonna do it in this region. And then each of those clusters actually is using Argo CD to do reconciliation for itself. So Fleet basically says, okay, update the desired version of applications that we have in Argo CD and then Argo CD goes and actually manages the update within that cluster. So it's very self-reliant, each one is very independent and you can basically rely on each one to come and grab the software that needs itself. So if that explanation that I gave you doesn't make sense, it will by the end of the talk. So let's talk for a moment about configuration drift and why this happens. So configuration drift is poison within an organization. Basically it means that you have environments that are supposed to be similar or an environment is supposed to be updated, but for some reason it is out of date with its definition. When we talk about its definition, this is where we go into get-offs for a second. So how do we know what's supposed to be deployed? Well, if we're doing get-offs, we have it all defined in get and whatever the definition is in get, that is what should be deployed. And if you think that's, you might be looking at your organization, you're saying that's not the way that we work. We actually, we update get and then we talk to Joe and Joe's our release manager and Joe goes in and he does a bunch of work. I don't know what he does but he figures it out and then he updates stuff when we're ready. And maybe we stuff have stuff in get that's not ready to be deployed, but Joe knows not to deploy it. Or we have stuff that we've updated in get or maybe actually there's some hotfix and I don't know how he makes it work, but Joe goes in and he figures out how to make it work. And this is like when I started first building software, this'll take me. And maybe some of you still do this kind of thing, but I had our IT guy that ran the servers and I would make an update. I would build a patch and I would email him the patch and say, please go apply this. And he would go and apply the patch to the server. And then if it didn't work, he would yell at me and I would yell at him. And then we would maybe switch out servers or something like that. Cause we usually kept in, we kept one server that was out of, that was basically a backup of live that we could switch to so we could hot swap it if we needed to. And he would go into like the windows UI and he would specify which server should be serving content. And that's how you do it. I think all of us have been there. That was part of my career. I did the same thing. So you told that story and you could have put me there. And I'm like, yeah, been there. I was on both sides of the house too. I was the guy who was like, this doesn't work. What's going on? So we've both been there. Yeah. So it's not a great way to work because I mean, at this point, we're not even using containers, right? This is back of the day. So some of you might still be in this situation. You're saying, I want to improve the way that we're running software. Well, if I am running GitOps and I basically said, look, how do we know what's supposed to be deployed in production? I'm going to commit it to Git. And if it's committed into this repo and in this branch, it's supposed to be deployed. That's the rule. That actually simplifies a lot of things. And a lot of the grease between releasing software, you start to get into, if you say as a hard and fast rule, once it's committed into this Git repo, it should be deployed. Well, the other area of where configuration drift happens is really in that grease. And a lot of downtime that's caused within organizations, and I would say my estimate, I didn't take a poll on this, but I would estimate 90% of downtime is caused by these mistaken deployments, 90% of big downtime. Let's narrow it a little bit. So examples that I'm thinking of, in 2017, yeah, 2017, 2018, AWS had an outage where somebody was making a change. They were connected directly against their production and they fat fingered the fat, do you guys know what fat finger means? For the international audience, it means you're typing, but your finger was so big that you accidentally hyped an extra character. So it just means you mistyped, it's a joke. It's a way of saying my fingers were just a little too big for the keyboard. So AWS engineer accidentally changed the number of desired replicas by a factor of 10, and this ended up causing all of AWS to go down. I don't know if many of you can probably remember this outage. There were a lot of jokes about people's dishwashers not working because they were internet connected and that kind of thing. So that was a bummer. Costco had a similar kind of change where somebody was connected against production on Black Friday and made a change, as I recall. And I may be getting some of these details wrong and Costco engineers are gonna reach out and say, that never happened, you're libeling us. And I'll say, my mistake, my mistake. But this is very common. People go in and make changes, cowboy style. They just connect directly to production and they tweak something, they get it working. That's a common source of configuration drift. I also added security breaches. If somebody edits your live production, they've hacked into your system or something and they're mining Bitcoin or they've injected code or whatever. This is another case where you get configuration drift happening. And if you're not doing get-offs, you probably don't know about it. Another source is failed automation. So if we go into, if we look at like configuration drift, we maybe, we start off, we've got these three servers over here on the left. Somebody makes some kind of ad hoc change in server two. And then we go to do a deployment and server two fails, the rollout fails because there's been some sort of ad hoc change. And I think about this most often in Kubernetes clusters. So if I had a number of different Kubernetes clusters, maybe I'm rolling out or maybe I have these different things. So let's pause for a second because there's actually a good question here. Is a mutable environment is a part of reducing configuration drift at info level? Like we won't make any manual updates patches on servers, rather we'll recreate them with new golden image. Yeah, so immutable environments, I think is a really great part of this story because if you're using something like Kubernetes, you don't update servers. You make a change and it destroys the pod and it creates a new one with your changes. So the each piece should essentially be immutable once it's deployed, right? And this doesn't apply just to Kubernetes but it would apply to other services as well. If you have good discipline, good infrastructure and good management, you won't have to update virtual servers. You'll recreate virtual, you'll create new virtual servers and then reroute the traffic to the new one. And that's a better situation to be in. So this configuration story, when I was thinking about this, it reminded me of a story that I was familiar with that comes to us from a major credit card company. And they had a situation where they had a team that was working on some components of the service and they ended up laying off the team. This was a number of years ago that there was an economic downturn. So they laid off the team and they expected that they were gonna move over the management of the service to another team that was gonna be taken on more responsibility. And as they headed into Christmas, they had an outage on that service. And unfortunately, there was no one from the team that had been managing that service to look at it and tell them what was going on. And so when the new team that had taken on this service went in and looked at it, they said, okay, hang on, I'm trying to figure out what's going on here, but it looks like none of the changes that have been made to this server for the last year and a half have been getting checked in to get. So there's been a bunch of changes and we have no idea what they are. And this engineer ended up having to, this was a number of years ago, but they basically printed out all of the running code and then went through it line by line to figure out what was going on and fix the outage. Now, this represents really a big organizational failure because it means that people are basically making ad hoc changes on top of ad hoc changes. You don't know what's running. So this drift becomes a really, really big problem. And so you'll notice when this is happening because you have things like it succeeds in staging, but it fails in production. If that's happening often, that probably means there's some drift failure happening. There could be some architectural failure happening too. If you have a view and you can just ask yourself, do you have any systems that you view as like, oh, you don't touch that system. That's, you know, that's Joe's system. We don't mess with that because it's a whole thing. It's complicated, you know, da, da, da, da. Well, it's probably not complicated. It's probably poorly implemented. That's probably what it means, right? If you have a system you're afraid to touch, it means that there's probably configuration drift problems. Rip that bandaid off and start. If you can't deploy updates to a system, it means, you know, the old saying is, if you want to get good at something, do it repeatedly, right? So if you want to get good at updating that system, start doing it repeatedly because if you can't update a system, if it's off limit, it's already fragile. It's gonna break. You just don't know when it's gonna break. So you may as well get into the habit of starting to update that system now, right? Lots of hacks, quick fixes in specific environments. Those things become very common where people are like, oh, you have to add a little overlay. You have to add a little tweak to this when you deploy it because this environment, it's special. It's a special snowflake. And people are trying to figure out what the difference is between these environments. This is a good symptoms that you have a lot of configuration to problems and that you're probably not following and getting the full value out of GitOps. And some people like, this goes back to, I'm gonna say, I don't know if your name is rendering correctly on my screen, but Cajetay's question about immutable infrastructure, immutable environment. A lot of people will say, look, we will use something like Terraform and we'll do a Terraform apply. So that gets me my desired state. And so I'm actually working from Git. I actually check that in and it's okay. The problem with that is that the configuration drift that we're talking about usually happens, in fact, it always happens after the infrastructure has created. People go in and they make a cowboy change. There is some kind of tweak that people make and maybe they don't document. They, a hacker gains access. Something happens where that desired state, maybe it wasn't even applied correctly in the first place. That desired state isn't actually being met. And so now you have essentially a black box. You don't know what's happening there and it's terrifying to touch it because you don't have predictability about what's gonna happen. So this happens, like I said, with Kubernetes as well. Somebody Kube CTLs in to production. They apply some change and now you go to do a deployment and it fails because it's different than what's expected in staging. This is a big problem. And Kelsey Hightower had a really good tweet about this. He said, Kube CTL is the new SSH limit access and only use it for deployments when better tooling is not available. You should really only be using Kube CTL for local deployment. If you have to break glass and use it against staging, that's a problem, let alone production. If you're doing it in production, oh boy, we're in trouble, right? If you're doing it against staging, you really, you should be in the position where you're not even having to ever do it for staging because if you don't ever have to do it for staging, you know you're not gonna ever have to do it for production, right? So you don't wanna be having people making ad hoc changes like this. So what's the better way to be making this? So there are a lot of strategies that people employ. They say we have really great documentation for how changes should be made. We have audits on all manual changes. We enforce these best practices, we have great training. These by themselves are doomed to fail because this is ultimately a question of tooling and organizational structure. So these things, they might be helpful but they're not gonna be enough to get the job done. You need to be using GitOps tooling that is gonna enforce this stuff and you need to be doing it in an enforced GitOps way. So this is when we get into Argo CD and GitOps. So for those of you that are unaware, I'll just gonna shoot this to you really quick. This is OpenGitOps.dev. This is an open standard that we helped author. We worked with, I think we had over 120 interested parties. No, it's 90 interested parties were involved in the creation of the standard and over 120 individual contributors. I'll throw it in the chat here for you. This goes into the standard of what GitOps, the principles that you really need to follow at a very basic level to be following GitOps. And I'm not gonna go through those exactly just yet. We're gonna, yeah, go ahead. Real quick, do we wanna kind of plug what the slight differences between GitOps and DevOps might be for those who might not understand that there's a conceptual difference? Yeah, great, great question. Okay, so from DevOps from a definition standpoint is debated quite a bit. We don't really know what it is. It's like, we know when we see it, that's what it is. It's like DevOps is like developers working with operations to improve and deploy stuff. We've got the unicorn tale, that book, that classic book. We've got all these books about DevOps and how to do it. GitOps I think is best thought of as an implementation of DevOps best practice as a subset of DevOps. And DevOps, typically we're like, okay, we'd like to have things defined in code, infrastructure is code that feels like DevOps. We're gonna have good separation of responsibility. We're gonna have good communication between teams. And when you get into what DevOps really is, it ends up meaning a thousand different things at a thousand different organizations. GitOps is a pretty well-defined implementation of DevOps that basically says four things. First of all, yes, you need to be using declarative. Your desired state needs to be entirely declaratively defined. And so that's, if you're coming from DevOps world, you're probably thinking, yeah, okay, we agree. Infrastructure is code, that's important. Declared is as opposed to doing something imperatively. So like, if you are, oh, I wanna have a server. So I follow this procedure, which is I click create or I run this command that creates a server, I SSH into it and I make some changes or whatever. Well, that's an imperative, those are a bunch of imperative operations. Declaredively saying, I want a server with this profile. Now there's gonna be a bunch of imperative operations that go on in some sort of automation to create that desired declarative state. But you've defined it declaratively, right? So you're saying, look, I just wanna server with these parameters. I don't care how it happens. Go and do whatever operations automatically that need to happen. But at the end of the day, I know that I'm gonna be able to recreate this over and over again the same way every time because I've created the desired, my desired state has been done declaratively. And we talk about that desired state. We're not just talking about server creation, we're talking about all the software that runs on those things. Kubernetes is very, very good for GitOps because it's really good at this declarative side. So this is why you see so much discussion about GitOps within the Kubernetes world because it's so good at it. The second thing is having it be versioned and immutable. And so this is something that I see people mess up on all the time. I took a hobby project this weekend. I don't know if any of you guys play games, but I was setting up a game server and a lot of people that do containers in the home lab, they just use Docker compose and they just run it on a single machine. And it's so common like Steam servers, they don't allow you to specify a version when you install a Steam server. You can install a branch and then it just gives you whatever the latest one. It's horrifying to me because as a GitOps professional, I want it to be versioned and immutable. I wanna be able to say deploy this version and I'm always gonna get the same thing. And when I'm ready, I'll say deploy this new version. And so you see a lot of people deploy things where the container that they've specified in their deployment is using the latest tech. Well, you're using a declarative system for sure, but it's not actually versioned and immutable. And they'll do the same thing with Git where they'll say, I'm relying on a Helm chart that is always just whatever the latest is. Well, that's actually you've taken versioning out of your deployment equation. And then next we have these software agents that deploy things automatically. They pull that state automatically and then they continually reconcile, which means they're aware of the actual state of what's happening and the desired state as defined in Git. So when we talk about GitOps, it's doing at least these four things and there is more to kind of solve above and beyond this. But if this is confusing, it should make sense within the context of this demo because we'll show you how this works. So we'll introduce you to Argo, but let's just look at a practical example, okay? So you are making a commit to your source code, right? You probably have a repository where your application source code is. That's gonna make a build. It's gonna make a new container image. It's gonna push that image to a registry and we're gonna open up a pull request onto a different repo probably. Generally I like a two repo approach for GitOps and that second repo is going to be your kind of infrastructure repo and I'll show you what this looks like when we get into the demo. This updates to manifest the charts. There's some kind of pull request that happens and then the cluster looks and says, oh, the desired state has changed. I'm going to go and make an update. So this is how a deployment would be happening. Now you compare this with a classic CICD approach where you just say, oh, I'm not opening a pull request instead I'm just running automation to deploy the new version. Well, what happens when that new version fails? How do I know what's supposed to be deployed? How does that work? Well, you've created a expected desired outcome with imperative operations rather than creating your desired outcome and then letting imperative operations take care of it behind the scenes. So that might be a confusing point. As we get into this, I think it'll all start to make sense, okay? So from a high level standpoint, you basically have Argo CD. You have your desired state in Git. It's pulling those changes from Git and then it's looking at the actual state in Kubernetes and it's saying, oh, something about what you have in your definition is actually not correct. So I'm going to update it and Argo CD is reconciling here. Now, when we say reconciling, I'm not making a change against Kubernetes and then Argo CD sees the change and says, okay, I'll go write that to Git. That's not what we're talking about because the source of truth is in Git. Whatever's happening in Kubernetes is not the desired state necessarily. Whatever's happening in Git is the desired state and Argo CD is going to make sure that desired state is going to be applied based on policies that we set. To introduce you to this, so this is a subset of the Argo UI and we'll jump into that UI in a second. Yeah, I've got it over here. This is a view of an application within Argo CD and this is the guestbook application. We can see that this is made up of a service, a deployment which has a pod and a couple of other resources and we can see that not only are all of these healthy but they're all synced. So this is telling me that this Git revision is what's been defined in Git and this sync has occurred and it's working properly. And it's always gonna guarantee that these things are in sync. And if someone goes and changes the Kube CTL onto the cluster and make a change, that would show up as drift and it would be automatically detected by Argo. So in order to be doing GitOps, we need to have a component that is aware of the desired state in Git and the actual state wherever that is. So this is why in the Terraform example that we had earlier where I said, hey, I did a Terraform apply. That bootstrapped my infrastructure, that bootstrapped my components. Okay, now what if someone goes in and changes it? How do you know? Well, Terraform does have some tooling for this but it's actually, and I don't mean to pick on Terraform but Terraform is not very good at being aware of state. It's not very good at being aware of state changing. And so if someone makes a manual change, Terraform is very often not aware of it. And so you can't detect that drift and you can't therefore correct it. So you don't actually know what's happening in production. You are assuming that the things that you put into Terraform when you hit Terraform apply have been made into reality and you are assuming that they are staying that way. My dad said to me one time when I was growing up, assuming makes an ass out of you and me, which is how you spell assume. We don't want to assume. We want to know. There's a, was it Reagan that said trust but verified. So we trust that the systems are gonna work but we need to have some verification in place for it. And I heard someone tell me the other day, they said, I don't know if I need to get out because we just lock our environments. So we just have a CI CD process that updates it and it's like, okay, that's great. What's happening in production? And they're like, well, what do you mean? I mean, it's locked. So it's gotta be whatever was last applied, right? And I'm like, maybe you tell me, you don't know. You don't know. You don't know what's happening in production. You don't know. You're assuming. Good luck with that. That's a recipe for, it's like saying, look, I deployed the service. I don't need monitoring on it. I deployed it. I assume it's working. Well, okay, that's nice, but what if it broke? Well, I assume it didn't. Okay, well, good luck with that. You're assuming that you're monitoring just because the monitoring isn't down that the configuration is correct. Well, even if you have monitoring in place, hey, your servers might be running, they're serving spam that somebody injected on there, but yo, the metrics are all there, so they're fine. You don't know. So you wanna know what's happening in production. That's where this stuff comes in. And it's gonna help you avoid downtime. It's gonna make things easier. So when I make a change to my application, in this case, we've detected manual changes have been made and we find that they are out of sync. The service has been updated. And this could be because the deployment was changed, in which case this sync would show that there was, it would show up out over here, or because somebody made a manual change, which is what happened, which is why this is showing up is out of sync. Now, in Argo CD, once it detects this, you can have it either automatically correct these things with auto healing, or you can have it allow the issue to exist until you do a manual intervention. That's a policy that you set at the application level. And it allows me to view a diff. So it will actually show me what's changed. And in this case, the actual state, the desired state is it wants port, this port 80 and target port to be set, but it's not been set. So that's been removed for some reason. So at that point, we can hit sync, it will sync and that state will be set. Monica asks, how do you connect Argo CD to the image registry so that the new image I deployed to the Kubernetes cluster? So I'm assuming that your question is, there's two meanings of your question. One is how does Argo CD actually connect to a private repository and have authentication and have that work, which that's something that is pretty easily accomplished in the docs when you go to add a registry, you can add them to authentication information and you can pull stuff. I think the meaning of your question is really, if I update the image, how does Argo CD become aware of it? And this is actually through your configuration update. So you should be updating the configuration. So when we had this process flow back here earlier, where we say, oh, an image has been pushed, we need to open a pull request to get, that could be done by a person or it could be done by automation. The way that I typically do this is I have my application repo and I'll show you, let's just show you an example here really quick. You know, I was joking about my, this game server that I was setting up, I'll show you on here. This is an embarrassing repo by the way. I've just been flinking around in here over the weekend. So it's nonsense. But basically I have my application repo here where I have my Docker file that's building things and I've just been fiddling around with this weekend. It's a hobby project, but I've got my repo where my application stuff is stored. And I actually have inside of here some deployment information. I have a customization that I'm using and just plain Kubernetes manifests because I just wanna add the additional option if people wanted to deploy this manually or something. So this is my application repo. Now, when I make changes to this, basically what I'll have this do is this will automatically open up a pull request onto my infrastructure repo. So let's go to that one and I'll show you what that looks like. So this is just for my home lab. Oops, project. Now, this is the repo that I'm using to run my Argos CD instance. And I'll just show you really quick my Argos CD instance. Here we go. So here's my Argos CD instance. I think this is like, we were just about to go into the demo anyway, right? Yeah, we were right there. Okay, so this is my Argos CD instance. And you can see for those of you that aren't aware of Argos CD, each one of these tiles represents an application and an application is an arbitrary definition of resources that need to be synced along with the destination for where they need to be synced. In this case, if I'm looking at, let's look at this demo app here. I've got this demo app. It's this demo app. Here's the application definition. It's to be deployed at my default cluster. This is the repo where it's pulling from. The target revision I'm pulling from right here is head, which is a little bit of a no-no, like I said earlier, because I want it actually to be version. So I need to update that. And then this is the path of where it's pulling from. And I have some policy in here where it's automated sync. So anytime I make changes, it will make changes here. It automatically will prune resources. So if I delete a resource from my manifest, it'll get deleted, and it has self-healing, which means if I go in and edit this resource in Kubernetes, Argos CD will pick up on that drift and automatically destroy it, which I'll demonstrate for you in a second. I've got my service. I've got my deployment. I've got my other components, my resource group, my pod. These are all the elements that make up this service. And the way that this repo is structured is I have an apps folder for each of my apps that are deployed. So this is the hobby project I was telling you about. And this has a base reference, which if we look at the customization here, is reference. The Kubernetes branch that I have, just because it's all alpha while I'm fiddling with it. And once I've played this, this will be referencing a version instead. So you're actually applying getups to your P Rising server. And you've got this one lab. Yeah, yeah. This is awesome. So not many times we get this guy to dig out that someone is going to do this. Not many times we get this guy to dig out that someone is going to do a demo on the game server that he worked on this weekend. So this is actually good. I'm enthralled right now. I'm like, wow, this is actually a real use case. So you're not getting a demo. You're actually getting a real world use case. Yeah, it's more like a sprint review. And I'm just sharing my progress. Yeah. So once this is done, this will actually reference a release, a specific release. And then, you know, if you think about the way that this is structured, I have my application repo and I have my kind of what I call my deployment repo, right? My application repo has everything in it that I need to run the application, but my deployment repo specifies what is actually getting deployed. And so I have my base. I'm using customized here. And if you're not familiar with customized, it's a Kubernetes resource package manager similar to Helm. But what I like about it is I can reference the resource from my base repo. And then I also have overlays for each of my servers. Now, in this case, I only have one and I actually just removed it. So I stuck it into a backup folder. So this isn't getting deployed anywhere at this moment. When it's ready, I'm going to move it into the elite cluster folder, which will then automatically deploy all these elements. And as my customization, it references that base, which I remember was referencing my original resource in my application repo. And then it's applying a patch for the load balancer. So specific to this environment, I have a IP address. In this case, not an IP address actually, I just specify that it's node port, which is not the default. So I just make my change. So going back over to our demo app here, let's just show you like, let's say that we have some drift that's going to occur. And I'm going to share, that's the best way to share my screen here. I'm going to share my whole screen. Let me just move some stuff out of the way. Yeah. Okay. So let me change my share. Sorry. Just now getting this. Okay. So you should be able to see my whole desktop. So everything's going to be a little bit smaller now. But, um, so you, we've got our Argo instance here. This is monitoring both my Kubernetes cluster and my, uh, and my, uh, github repo. Right. So I'm going to switch over to my, uh, my atomic cluster. Okay. And we're going to go over, get rid of these logs. We're going to be looking at those right now. And let's say that I'm looking at pods on my default. Super deployments. On my default. And, um, this one is elite cluster demo app, which is in the default namespace. And it has simple service and simple deployment. So this is the resource we're looking at simple deployment. And if I were to edit this. So, right, I'm, I'm going around the process. I'm, I'm going to be modifying this thing. And I'm just going to change, uh, the replicas to zero. Uh, why wait, I don't do it in the status. Hang on. Should do it under the spec. Here we go. Haha. Okay. So I changed this replica to zero. I can do this. Uh, and I saved that. Now you'll notice that it actually happened really fast. This actually saw that there was a divergence right away and fixed it already. Did you see that happened really quick? Let's do it in the terminal because, uh, maybe it'll be a little bit more noticeable. Make this nice and really big for you to see it. Okay. Um, Make sure I'm on the right. Oh, I'm on the wrong cluster right here. Let me change my cube context. Okay. Now I'm on the right, right cluster. And I get my deployments. I've got my simple deployment and I'm going to do K. Uh, deploy scale scale deploy simple. That's deployment. Replicas equals. Zero. I've destroyed it and our, our goal seed is picking up like too fast, but you can see it's just, it's picking it up immediately. And it's saying, Hey, it's not supposed to be scaling down. It's supposed to be set. It's supposed to be set and running. Uh, it's not supposed to be have that many replicas. It's supposed to have zero. It's supposed to have one. So if we look at it again, uh, you know, we can see that it's already, it's already put that replica back. So it's not going to allow those pods to get destroyed. It's going to, it's going to be recreating. I'm seeing they're getting recreated immediately as soon as I try to destroy them. So any configuration drift that's happening is automatically getting corrected here. Um, Now, if I were to go and change this and get, so let's go over and look at this simple deployment. This demo app again. So we have this specified here. Let's look at the overlay. I think I said that the, um, I think I said that my, I wasn't referencing a specific version. Let's reference a specific version. So this is coming from this Argos CD auto pilot. Example. Over here. And they actually have releases. So I could say, let's, let's pull a specific release. We should, we should be doing. So we're going to edit this and we're going to say. F equals. I actually want to be building always off of this version. And set demo. What's the other, what's the other tool that you're using there? You're, you're flipping back. Is it? Oh, this. Yeah. This is lens. I don't know if any of you are aware of lens lens is a cool tool. It's got a, just a nice UI for monitoring and working with Kubernetes. Um, and, uh, basically you can do all the stuff that you do with coupe CTL with lens, but it just gives you a nice visual look at it. And, um, I've started using it more and more of a kind of come around to it. So I changed the specified version. And this is going to pick up. See, it just picked up on the sync right here. It's so fast. Three E 94. Yeah. So it's picked up on that sync and it's looking at it saying, uh, yeah, actually there's no difference. Between what was deployed and what is deployed. So I was in luck. That's, that's what you want to see. Um, and if I look, you know, if I was looking at this, uh, this summary of applications, you're not going to see any differences here because I have this automated sync policy set. If I didn't have the policy set, it wouldn't be happening. Now let's say I was going to deploy a new application. I'm going to show you kind of a no, no. I'm going to hit new app. And let's do a demo app too. And we're going to put this in our elite cluster project. It doesn't really matter. Um, and I'm going to leave sync policy on manual. And, uh, the deletion finalizer, um, what this means is that this is wholly managed by Argo CD. And so if you deleted Argo CD, it would delete the resource. So I actually don't mind that that's fine, but it is potentially destructive. So be aware of it. Um, I'm going to leave schema validation in place. I am going to auto create the name split namespace. And I'm not going to do a replay. So this is important if your, um, Manifest are getting too long. Uh, and then we're going to specify our repo, which we're going to go back to this autopilot example. And this demo app. Okay. And, uh, I'm going to pull from. Um, the specific. You know, release version that's currently out. Just like we did a second ago. So I'm going to do V zero, three, eight. And we'll go to examples. Demo app. I did all this right. I don't ever do it this way. So that we, and I'll tell you why in a second. And, um, I'm going to create a new namespace for this that I'm going to call, uh, master class. Um, not going to be a recursive directory. I'm only adding the one directory and, uh, this, this works with helm charts. Uh, in this case, it's going to be customized and it will pick it up automatically. I don't need to do anything special that you're going to do that. Um, and I'll hit create. Let's see. Unable to create applications and valid repository, not accessible. So let's see. Did I specify something wrong in my tag? V zero dot three, eight. Well, rather than messing with it right now, I'm just going to switch it to head for a minute. I just showed you how we could switch it. Um, otherwise let's see. Does it pick up that way? Or do I need a path on here? What am I doing wrong? Why do I need repository? Not accessible, but it is accessible. Unknown error testing repository. I don't see why that would be wrong. That looks right to me. We just switched it to head. Hmm. What am I getting wrong here? Uh, it's funny. I don't ever do it this way. I always do it through, um, through my, uh, hold on. Let's choose a tag. It says that it can't connect to the repository, which doesn't make sense. Why would it say authentication is required? It doesn't like it. Well, at any rate, I mean, we don't have to go through it right now. Um, but the point is if I were to do it this way, and I'll tell you why I don't ever do it this way. Um, I don't ever do it this way because if I just create this app, it doesn't create the definition of the application. It doesn't create it and commit it to get. So while I'm managing this application with GitOps, I'm not managing the definition of the application with Git. So, um, this is, this is, this is why I don't like the, uh, this is why I don't ever use this UI to do this is because I actually always just do it, um, in Git. So the way that I've structured this repo is with something called an application set. So under my projects repo, I have this elite cluster YAML and in here I have something called an application set. An application set is a way of generating applications. And in this case, uh, the generators are looking for any time I have a config JSON that's under apps, any path elite cluster and I have a config JSON, it will take that and it will automatically assume that it needs to be, uh, uh, it needs to be deployed and it will generate the application based off of this and we'll fill in the destination, the project, the source, the repo URL, all of that stuff is going to be set in by what I have sitting inside of that config JSON. So when I look at this config JSON, you can see the path it's taking arbitrary. Um, and this config JSON just specifies the app name, the user given name, the default name destination name space, the destination server, which is going to be with my local Kubernetes cluster. Now Argo CD can connect and manage many, many Kubernetes clusters so you could have 50 or 100 in here. Um, the source path, uh, I have specified to where it's coming from the source repo that it's coming from the source target revision and, uh, any labels that I want to have. So this is all just specified automatically in here. And then I use a tool called Argo CD autopilot, which is, um, this will, if you use this to install, let's go down here. Uh, if you use this to install Argo CD on your repo, it will write the entire configuration of Argo to get. So when I look at this instance of Argo that I'm using, you can see Argo CD is an application that's on here. So I'm using Argo CD to manage Argo CD. Um, I see there's some discussion going on about the tools that I'm using. So this up here is a Rancher desktop. And it, it just keeps track of your, I mean, we could just do it in a command line too, but it's, you know, it's, it's faster just to do this way. So Rancher desktop does this, um, Docker desktop does this. I like Rancher desktop a lot because I don't have to have Docker installed. And, uh, it's better that way. We like the fact that you like the Rancher desktop too. So we're, uh, I'm personally happy seeing that you're using it. So that is the dropdown. I think he was asking for. Yeah. I think, I think, Coup CTA, uh, Coup CTX is great too for switching. Um, and then yeah, uh, uh, I do like lens, um, for, for fiddling with clusters. I mean, it's just Dyson visual versus, uh, I mean, we can do it in command line too. It's fine. Um, but yeah, like I was saying, Argos CD is actually managing itself here. So if I wanted to update the version of Argos CD, then I'm running. Let's go and look at our apps here. Uh, Argos CD is not listed under my apps folder because I have this sitting in a special folder under bootstrap, which you don't have to do, but you can. Um, and if we look at Argos CD, uh, we have the definition of my application. This is a CRD. Uh, this is Kubernetes custom resource. And that specifies what an application is. So once I sync that to the cluster, it's done. So this gives me my definition of what the application for Argos CD is. And if we go back and look under Argos CD at our customization, uh, you can see that I actually have, um, my base reference, uh, uh, in here. And then if I wanted to have, uh, I have the service exposed. So I have a custom, um, overlay for that under my customization. That's a patch that's getting set to expose it. And then if I wanted to change what version that I'm running, I would just change the reference here and commit it and be done. So you can see how you can use this to manage the definition of application across many environments. Um, and this is why I don't actually ever use the UI for creating the applications because I'm always just creating them and get directly. And yeah. So pretty hard question here. How does it, how does it create an Argos CD application based on the application manifest? Do you have to have a pipeline? Ah, okay. So this is, yeah, yeah, yeah. Good question. Good question. Okay. So when I, the way that, the way that, um, Argos CD autopilot works and I'll, I'll drop a link, uh, in the chat. The way that Argos CD, uh, autopilot works is this will, um, this bootstraps Argos CD. So basically installs Argos CD. Manually under the cluster, right? The, the, the, the, this binary is actually taking care of that operation. And it's specifying its definition from this get repo. And then from there, Argos CD sinks everything. So only the only thing that needs to be synced, uh, the only thing that needs to be done is Argos CD is created. It's bootstrapped. It tells it to manage itself. And then it automatically will go and pick up the rest. So this is why I'm using this, um, application set with application sets. I don't, when I add an application, I don't actually even talk to the Argos CD server. The only thing I do is I update get. So if I add, uh, something that matches this generator, so I add that folder, that file, it will, it will do that. And then, uh, Argos CD autopilot actually has a command line tool that does this. So this doesn't change. This doesn't even talk to Argos CD. It only talks to get and updates it from there. Um, duck asks, do we need to create root app in Argos CD UI or Argos CD CLI first? Otherwise how can we register our application set repo to Argos CD? Well, that's what Argos CD autopilot is doing when you do a repo bootstrap. Uh, let's say that you weren't using this tool. How would you do it? Okay. Well, um, if I were just going to go install Argos CD, I would just let's say I'm going to do it manually, right? Now there is a Terraform module for it. Uh, there's a crossplane race resource for it. So there are ways of doing it just entirely declaratively, but let's just say we're doing Argos CD quick start. And we're just following this, um, this path here. Uh, so I would create, I'm doing manual operations, right? I'm creating a namespace. I'm deploying the resource. And at this point, uh, Argos CD is not managing itself. All I've done is that I've installed Argos CD. Well, now I create an application for Argos CD. And I apply that using CUBE CTL. And at that point, Argos CD is now self managing. So the resource, if the resource already exists and you create an application for it in Argos CD, Argos CD will detect that it already exists. And it will say, okay, I don't need to sync it. I'm, I'm good to go. So it will, it will just take over the management of any resource that exists. So in, in the example of my home cluster that I was showing you, I actually have a few resources that aren't under management yet, that I had applied manually before I had Argos CD installed. And I will, I am moving them under management by just creating an application for them, um, under my directory structure, which is picking it up. You do need to, you do need to apply the application set. It's a custom resource. So this needs to be applied. So you do have a few little, you have a few things that you bootstrap on to the cluster, um, in order to do that. And this is actually where Rancher, uh, becomes really nice as well. Um, because I can, let's, let's, let's look at Rancher really quick. Let's see if I can, um, we're getting, we're getting into like, we're getting into other things. And, and we'll probably, we're getting close, close on time. So I don't want to take too much time, but, um, since people are asking about it, I can show you how it should work. Um, so, and open this. Hopefully I don't have to re log in. Okay. So with Rancher, which is a similar thing. I mean, you have to install the agents, but you can add any number of clusters and you can specify, and I'm not going to show you this deeply here, but you can specify repositories that you want to have automatically synced onto those clusters. So you can basically say, sync my Argo CD components onto these clusters to bootstrap them that way. That'd be another way to do it. But yeah, I mean, you do have to get something connected to the cluster at some point to start the whole process. But once you start the whole process of like, if we look at my elite cluster, for example, if I wanted to, um, let's say a disaster has happened in my cluster.wiped, if I wanted to run this, I would, uh, go get my repo out and I would make sure that, um, I was connected to the right, uh, cluster. And I could actually do a, uh, customize, apply on my bootstrap resource. And this would trigger everything else to happen. Um, so everything else would just be bootstrapped, including all the applications. And I would, I, that way I have disaster recovery. So, um, anyway, we're a little bit into the weeds. I know I've, I've taken you on a little bit of a tour of, of a couple of different things. Um, let's, let's move into a couple of questions that I think people want to ask. Oh, before, well, let's, let's do, yeah, before we do Q and A, I just want to just call this out really quick. So the reason that, um, where we're coming from, I mentioned that it's very top of the hour that code fresh is the enterprise version of Argo. So we, there's the community version, which is the Argo open source project. And then there is code fresh, which is the enterprise, mostly open source, um, enterprise version. And what we do is we basically allow you to do this at scale. So if you have many instances of Argo, you have a thousand instances of Argo and they're all behind the firewall and whatever. Uh, we provide this universal dashboard where you can search across all the applications deployed, no matter where. Um, and we automatically give you door metrics. We automatically calculate how quickly your deployments are happening, how many times you're having failover of deployment. We integrate with CI. So this is worth checking out. If you haven't, um, you just go to code fresh.io and you can get a demo and you can, you can sign up and use it for free. And we just announced that we actually are, uh, are launching a hosted Argo CD version that'll be launching. Um, planned as next month. So you can sign up and you'll get a free hosted instance of Argo CD. And the other thing I'll call out before we get into questions. Um, Is that there is a really fantastic get ops certification at code fresh.io slash get certified. I'll throw this in here. This is currently free. And, um, this actually provides you with a free environment where you can bootstrap Argo CD. It shows you how to do it. It shows you how to create your applications and it will go through the details of how to do all this stuff with get ops. It shows you how to do secrets. It shows you how to do progressive delivery, canary and blue, green deployments. Um, all of those things using get ops and using Argo CD. So I definitely recommend you check this out if you haven't. Um, and with that, let's, let's move in to some of the questions. Cause I think that people are asking a few questions. So, um, Yeah. Oh, yeah. Sure. Go ahead. I think you got the ducks. We have, um, Kajate has kind of a question and a follow up and then we have one in the question. Well, it's Kajate again. He's asked the three questions. So when we grab these two that you, that you see and I'll ask the last one. Okay. Uh, not sure if it's relevant right now, but what is the best practice for managing multiple environments with Argo CD single Argo CD instances for multiple environments or Argo CD instance per environment? Okay. This is actually a really, really important question that, um, I'm planning to do a talk around and there are, there, there is, uh, some implication about this, um, in a blog post that I'll share on running Argo CDs securely. So the reason this guy, so I'll throw this blog post in the chat, which is, is highly relevant to this, but, um, why would you, we mentioned that right in this, in this case, I have, uh, Argo CD is managing a single cluster, but I could have it managing lots of clusters, right? So when should I have Argo CD managing lots of clusters or should I have Argo CD per cluster? What would determine why I would do one versus the other? Okay. There are a couple of things. First off is performance. So once you get up in the, you know, you've got 2000 applications running on Argo CD. Uh, it's going to, even though they're all deployed potentially to different, um, clusters, uh, it's going to start to have some performance issues. And now Argo CD does have a, uh, an HA version that does allow some scalability, but even if you're using the HA version, if you're hitting around 2000 applications and, and this, this is super caveated because if those, if those applications have, you know, a thousand resources, then you're not going to get to 2000 applications. So it's, it's a combination of basically how many resources under management, but, um, it'll start just taking a really long time to sync. So for performance reasons, you're going to want to start to split up, uh, Argo CD instances. So that's one reason. The other is security. So Argo CD, we didn't talk about this, but Argo CD has role-based access control. Uh, so I can create teams and I have single sign on too. So I can have teams. I can have it syncing to my single sign on and I can have it, uh, you know, you can deploy to these namespaces. You can deploy to those namespaces and that's working great. There's sort of a caveat to that, which is that, um, the trust that you have, uh, should be at least somewhat tempered. And that's why I shared this, um, scaling Argo CD securely, because if you think about the operation that's happening, when you create an application, you're allowing somebody to execute a Helm chart or a customization, uh, or arbitrary, you know, files that could potentially try to reference things that are outside of their scope. And there's some mitigation that's done, but you should be aware from a security standpoint that, um, there's, you should trust it a monitor, a moderate amount. And so I wouldn't want to have, you know, 5,000 people on one instance. Um, I'm going to have it, you know, be a couple of teams. So that's another reason to split it up. Um, and this is where that control plane from code fresh becomes really valuable because this control plane lets you manage many instances. And I mentioned single sign on. So for example, I can create my single sign on once with code fresh and then every instance of Argo that I deploy is automatically associated with it versus having to set up single sign on every time I set up a new instance of Argo CD. So, um, those are some of the considerations that you would go into for why you would want to split up Argo CD and you'd want to have multiple instances. Um, the other question you asked is, plus if we're trying not to do any imperative changes on environments, what is the use cases for features at Argo CD, like a maintenance window? So yeah, Argo CD has a feature called, uh, so first of all, they have, uh, uh, sink windows. So you can basically say only sink during these time periods. Um, and you also have the idea of like pausing synchronization. Why would I want to, why would I ever want to pause synchronization? Well, uh, there, if, if the situation arises where things have broken in some spectacular way and, um, you know, ideally you just, all you do is you just revert a change and get, and you let it roll back to the previous version. But, uh, for the scenarios that you can't imagine, people do want the ability to say, look, something's going wrong and I might, maybe I shouldn't be, maybe I shouldn't be connecting directly to production using coupe CTL or using lands or whatever, but I'm going to, because I'm going to go jump into, jump into logs and stuff and whatever. And actually in the new version of, um, I haven't, I haven't turned it on, but, um, you can actually access logs and in the, uh, I haven't turned on the RBAC for it, but you can actually even exec into containers. If you have the permission set, I don't have them set here, but, um, but that's why you would want to have that maintenance window of some kind is, if you want to do some sort of manual changes. And sometimes this goes because like maybe your secret provider is not functioning properly and you need to go and work on that. Um, or you don't want to be fighting with Argo CD because you're making a change and Argo is breaking it or whatever. You shouldn't be doing that in production anyway, but, um, you know, people are used to having that as an option. So it is available as an option. So real quick, you can show us some things that were bad practice and you were like, don't do it or don't do it this way. This is really bad. But you also showed us, um, you have your application repository and kind of like your infrastructure repository. These are some best practices around Argo CD, I assume. Uh, where would someone want to learn more about some of these best practices? Oh, that's a great question because we've got a great blog post about this. So Argo CD best practices is a great blog post that, uh, Hannah, um, put together where she talked to the community and, um, got a ton of things and separating get repositories is the number one. So that's one of the things that we talked about. Um, creating the directory structure to enable multiple application system. I actually showed you that as well. And this, this explains it more in depth. Uh, that shows the promotion strategy application sets is actually covers a lot of the stuff that we've talked about. There goes a little bit more in depth. So it's worth checking out. And then, um, obviously the certification, which is free, as I mentioned, uh, is really good for teaching this stuff. And level two is almost done. We're going to have that out soon. So level one is out. Um, and I think we have almost 8,000 people engaged on that. It's the, it's the world's most popular get off certification. Um, hands down, it's pretty amazing. So yeah, there we go. Um, let's see. And then yeah, any other questions before we wrap up? I mean, I think that, uh, what we did here, I hope this is interesting. I mean, we covered a lot of the principles and then I kind of gave you a tour of things and showed you how some of these different things work in practice. Um, gave you some general best practices and general tips. Uh, and then gave you some resources where you can go into learn more of the stuff is kind of more of an introduction. Um, to some of the stuff. Um, oh yeah, check out, uh, uh, Luke's, um, talk on a multi-tenancy and get ups with, with a rancher and Kubernetes. That's great. Also, you know, while we're, while we're, well, I got you, Argo con is just around the corner. It is. Uh, it's going to be September 19th. The schedule is going to be announced this week. Um, we just finished the program last week. Everybody's just accepting and we're, we're just making sure that everybody can make it and stuff, but there's going to be workshops here. There's going to be a bunch of talks and keynotes. Um, first in person, get, uh, Argo con, we did a virtual one last year. We had about 6,000 people, uh, at this conference. It was just absolutely insane. It was the first one we did. So definitely check out Argo con. I'll throw the link in the chat. Um, any other questions before we, uh, come to a close. No, no, I, I threw two polls out there. Um, the first one was who uses Argo CD. So over, over half of the people who are on this webinar, they're using it, which is good. Um, and then. Sorry, Dan, but I asked who explained the ranch better. You were me. Um, and apparently I do a better job at that, which is good. Cause if it didn't, then I'd be probably looking for a new job, but, uh, I, I just voted for you as well. So you. Um, there's anything else. Um, I'm actually, I'm following you now on GitHub because I'm curious to see how you get this. Um, my friends, um, play be rising. So I, the idea that you're kid, you're containerizing this. I'm just like, I'm going to follow this. I want to see if he gets it done and he gets it running up there. Cause that would be kind of a, a cool example. I love anytime you can put a real world example down and you're like, yeah, I'm doing it for a game server. I'm like, I was all in. I'm impressed. Yeah. It's actually, so I'm actually running it right now. And it's running successfully. I have some cleanup to do on the project to make it, you know, nice and consumable. And it's, it's actually a windows server application. So my container is actually running wine to run the server. Which is gross. It's gross. It is what it is. Uh, but, um, you know, it's working. I'm using it and I've been playing on it. So it's working. It's working. Okay. Yeah. You couldn't get a windows container. Maybe. I could, but then I'd have to add a windows node. I like having all my nodes. I like having all my nodes just be. They're all the same. I don't care about them. I can throw one away. I can plug in another one. If I have to have a windows node, then I'm going to have to have like this is my windows node. And I actually already have. Arm nodes in my cluster as well as, uh, x86 nodes. So. I don't know. It would probably be wise, but I don't want to. Maybe I'll go see what they have available at the sort, sort surplus and do that. Yeah. The wine is gross. I mean, I don't know. It's not, it's not, I don't, I don't recommend it. It's not a, it's not a good situation to be in, but it's running. It's working. So if any game developers are listening to this, we really don't want to have to use windows to run a server, to run our games. So yeah, make a containerize that and Linux and then we will take it. We'll help you. We can help you scale that out too. We're expert at it. Also, if you're, if you're a steam developer, add the option in store. So if you're a steam developer, add the option in store. In steam server commands to specify a version. What the heck, man? Why do I, I just have to only get the, the currently available version, even if it's broken, I want to be able to specify whatever version I want. Yeah. I might tweet that later. Just, just see what happens. See if, see if they have a response. Thank you. All right. Well, Dan, thank you very much. We do appreciate it. You're always welcome back to, to speak to the community. And I think I'll be seeing you here in a few weeks. And I think I'll be seeing you here in a few months in October. Are you going to be in Detroit for your con? Okay. Wait, we have got GitOps con happening in Detroit. And there's going to be some really amazing talks. Actually that CFP just open. So you have the people here already, Argo users. If you have a CFP you want to get in a talk, you want to get in for GitOps con that just opened up. GitOps con. I think we're going to have here GitOps con. Let's put the link in there for everybody. I want to make sure I post it so people know it's up. It's actually coming from the next foundation. I did not know that. Yeah. It's a CNCF event. Yeah. I open up to the CNCF project. Okay. Okay. I threw my Twitter in the chat. Feel free to hit me up directly or DM me if you, if you're struggling, especially if you're working on like scaling Argos CD and you, you want advice or thoughts on that. Happy to, happy to help you on that. That's something that I do quite a bit. So feel free to, you know, my DMs are open. So reach out anytime. Awesome. All right. Well, thank you Dan once again and everybody else. We'll see you in the community.