 All right, we are officially at 350. I think we're gonna get started. Thank you everybody for joining This is gonna be a fun session. We have 90 minutes now Let's let's do this. So first of all introduce myself my name is Dan Garfield I am the co-founder and chief open source officer of a wonderful company called code fresh I'm an Argo maintainer. I've been working on Argo for Maybe five years now We've We launched a commercial version of Argo from code fresh about four years at three and a half years ago. We helped bring Argo into the CNCF and then to Sponsored it as a project and helped it get into incubation and then into graduation finally So we've been doing this for a long time. We've helped I don't know countless companies adopt Argo and get ops and these best practices and Joining me today are two amazing people Who will raise their hands. We've got costus capillones who's right here and next to him is Laurent Rochette and Laurent is a experienced Solutions architect and health people adopt this stuff all day long and debug and all those kinds of things And then costus is actually the author of the training that we're going to be doing today and if you don't follow him He does a lot of blog posts and trainings and things he gave a talk earlier today That was very good about doing get ops with databases But with all of that preamble Let me ask you a question. How many of you have done the code fresh get ops certification? Perfect, okay, so this is actually really good So if you didn't know this it is the most popular get-op certification in the world and the fastest growing We have over 20,000 students who have done the certification All of you because you are here are going to get free access to the entire Certification even though we're only going to be doing a portion of it today How many of people have deployed an app using Argo before? Perfect, okay So we're because we're not really going to cover that too much in depth if you haven't figured that out now as we Go through feel free to raise your hand if you get stuck on anything Costus and Laurent are going to be roaming around and if you get stuck on something something's not working Something's not making sense Feel free to jump in and then we're also a small enough group that if you have questions shout them out They've got microphones we can run around and we can we can have the conversation and figure it out Does that sound good? All right, awesome We also have for all of you Don't leave without this but for everybody who's doing the certification do we brought you a free Argo tumbler? so Put hot beverages in it or cold ones. I'm not the boss to you. Look. I'm not saying look you do you Okay, you put popcorn in there if you want it's not my business All right So to get started We have a code that's set up for you. That's code fresh dash FTW all Upper case this is going to give you free access to the entire certification We're going to be working like I said on a subsection of me So I'm going to show you how to do this in a second, but it's learning dot code fresh.io. That's the code I'll leave it up there for another 30 seconds I'm going to demonstrate how to redeem it and then I'll put it back on the screen So it's going to go away, but it's going to come back. It's coming back. It's going away It's coming back if you don't redeem this today It's going to expire so if if your plan is to take a picture and dart out of the room You have to redeem the code then you can dart out of the room. All right So if I go to learning dot Code fresh dot.io It's going to have this certification here and The easiest way to do this is if you scroll down and you see these two courses keep going until you get to the bundle click review bundle and by now and Then put in you won't need to put in a credit card or anything put in the code fresh dash FTW Click redeem and it will show $0 cost Okay. All right, so I'm going to put that back on the screen So you can all get that completed Now today, we're going to be working on promotion with get ops and There are a lot of gosh tricks to this and ins and outs of it Like I said, we have 90 minutes if you do the entire certification if you just if you're like, hey, man This is I feel like I don't have the basis of what you're talking about Feel free to just go start at the beginning of the certification put on your headphones that ignore me But what we're going to do is we have some presentation where we explain the principles and then all of the certifications come with labs So in your on your computer in your browser, you're going to be able to run these labs. It runs remotely Hopefully the internet's okay. We just spun up like 50 VM. Actually, we spun up 150 VMs come to think of it So you shouldn't have very much wait time as we do these labs And then the other thing is when you sign up for the get off certification, you'll get an invite It's in the very first section of the certification to a discord called get ops champions and you Chosen few can join and become a get ops champion and join that discord And so even after today even in a week or in a month you can pop into get ops Champion certification ask questions and get ideas and there's a lot of other like-minded folks who are doing get ops with Argo Now it's possible to do Argo CD and not do get ops a lot of people do that actually it's very common for people to deploy applications Using Argo CD and never set up those applications and get and so they have no way of doing disaster recovery That's shockingly common, but we're going to be doing better than that today because we got it All right, so I'm going to take this code off the screen Oh Okay, take a picture of it and raise your hand if you can't and you've got you've got two people behind you can help Anybody yeah, raise your hand and call it out and again if you're just joining in your in the back If there's tables up here that you can set your laptops on While you're working you don't have to come up if you don't want to but it makes makes life easier at least for me All right, so let me jump into The certification here I can see the VMs have started so that's nice Let's go into Oh There's no expiration date on the certification so you can go back and do it over and over again I Guess we don't plan necessarily to give you indefinite forever access, but for right now you have indefinite access So certainly if you get it done in the next You know month, you know, it'll be there if you get it done in the next six months, it'll it'll be there All right, so let me Jump in it's actually weird to get into the course when you're an admin of it. It's like Sorry, give me just a second While you're setting that up So the code that you saw on the screen is actually giving you access to the bundle which includes the first one with Github's basics and then Github's scale is what Dan is going to show today So even if you're just starting out, you can completely ignore Dan and go into Github's basics on your own today Or the day after tomorrow He was good. He's going to start from the middle But you get access to everything so don't worry about you know if you want to choose another course So once you've logged in you've signed up You'll see two courses that you have available. The first one is Github's fundamentals That one covers installing Argo installing applications setting up progressive delivery blue green deployments canary It covers a bunch of different really interesting patterns And so it really serves as a foundation and I definitely recommend you do go do that because even experienced Argo users Learned quite a bit in that first course on the fundamentals because it is so oriented around being get first you're going to want to navigate to get opposite scale and On the left hand side, you'll see it's got like introduction and About Github's champions and there'll be a link to join the Github's champions discord Which doing today is actually a good idea. We'll monitor we'll be monitoring that discord Then it's got handling multiple applications, which we'll talk about app of apps And then it will also talk about application sets I'm going to start by Presenting and discussing a little bit about Folder management and how you should structure that and then we'll go into the exercise on environment promotion Now we do have a full up a full exercise on application sets multi cluster management all that stuff but I think Well, let me let me actually ask I'm going to give you the choice we can start with application sets or we can start with the Folder management environment promotion modeling So raise your hand if you want me to start with application sets Raise me raise your hand if you want me to start with the folders environments Okay folders environments is winning so does that mean most people have used application sets at this point raise your hand if you've used application sets It's about it's about the same people who said to start with something else If we have time we can come back and do application sets I'm going to be starting with promoting releases with get ops and Like I said, I'm going to present some material and then we'll start the lab We'll give you some time to complete the lab and then I'll also do the lab So you can follow along and see if there's anything you missed. Okay, so The first thing that you want to know is that there are a number of different strategies for managing Your get ops repositories so when we talk about a get ops repository what we're talking about is Where you store not only your Kubernetes manifests or helm charts or Customizations, but also where you store the applications and their definitions themselves. So typically You have a repo for an application where you're building a binary, right? You're making a change. It's got a docker file. You're building an image and then you have a repo for get ops that is The source of truth for your applications. They should be separate repos right off the bat Okay. Now once you've got your get ops repo, how do you organize that? There are basically three different approaches that we've seen the first one is an environment per branch So each branch in the get repository is its own environment with all of its manifests The second one that we see is an environment per folder and when I talk about an environment You know what I'm talking about. I'm talking about like essentially a Kubernetes cluster, right? It's or or an Argo instance, right that you want to have watching An environment per folder means all those manifests fit into multiple folders So you've got like data center one data center to East Coast, you know, maybe you name your your environments Frodo you know Baggins other Lord of the Rings characters Gandalf and then The third one that we see as it is an environment per repository. So they create a new repo for each For each environment, I said there were three, but the fourth one, of course is a combination of any of those okay, so the first big takeaway that I want you to have is Like hella don't use environments per branch We really recommend against this Because if you think about it, you've got a branch for QA a branch for staging a branch for production, right now That's really simple if you have three environments and they're named QA staging and production But I just said what if you have an environment named Frodo or US West or US East or what if someone forks that repo to make Changes and they have a new branch. Does that represent an environment? You don't know and so just by looking at git It's actually hard to figure out what's deployed because you don't know which of those branches actually map to environments So that's problem number one problem number two with this is that doing diffs is actually very difficult The reason The reason that diffs is difficult is because if you think about multiple environments, so if you have a dev app a staging app and a production app There are values and things that are meant to be promoted and there are values that are not meant to be promoted, right? So it's like oh, which database does the dev one connect to you know Is that value saying it's connecting to the prod database? Well, I hope not Right, so you know that there's going to be long-running differences between these environments branches aren't set up for managing long-running Differences that means that every time that you want to promote a change you get into a git cherry-pick situation I think there was a survey recently that showed that like 90% of devs Basically don't know how to use git They know how to do a commit they know how to a push and then that's it Like that. I got a workflow. That's it So give it doing like git cherry-pick picks or getting into like a git bisect raise your hand if you've ever done a git bisect Yeah, yeah, so ask them have for help later everybody else, but That's what you get into if you're doing an environment per branch Is because you need to now start thinking about git cherry picks and then doing diffs between branches means Is is more difficult to you can't just do it off of your hard drive You actually have to run against to get to do diffs Diffing operations. So promoting is harder Diffing is harder knowing the truth is harder. So we really recommend against this approach The reason that people like to do this very often is because it grew out of application development, right? So trunk-based development is like great for working on applications. Oh, I create a feature branch When I'm ready I merge it and devs kind of looked at that and they're like, you know feature branches Those are those are that's kind of like an environment like it's like a long-running thing, you know but That that doesn't really translate into the GitOps world because as a source of truth that Joe Schmoe has a branch that's been alive for for two years That is a feature that he's abandoned. You don't know what the hell's going on there, right? So trunk-based development is why people have done this It's not because they made a proactive decision about the right way to manage their repository. So number one takeaway From this section is basically don't use environments. I mentioned all of these things I talked about merging and that's a that's a problem. The other issue is like if you look at customize or helm Typically you have like different values files, right? And I mentioned that now you need to have You need to have these multiple values files that you're managing across these different branches and then like how do you do promotion again? It goes back to the promotion element. It's it's tricky So customize actually has kind of given us a hint here with how that tool works. So let's look at Using folders for environment and this is typically what we recommend The really cool about thing about using Let's see. I think I have a diagram here. Okay, so check this out Here's an example where I have a number of environments. I've got integration GPU integration non-GPU low-GPU ProdEU prod us staging Asia whatever if I want to run a diff between those environments I can literally just run diff folder one folder two And I get the full diff between the environments dead simple really easy and many organizations are not used to working this way and so there is some discomfort because Typically with git you're not setting up permissions based on folders, right? You're typically doing it on a repo level That exists though. You can do Folder level permissions with a coder's file That's supported and get so you can create a coder's file and say this team has access to this folder This team has access to that folder and you're done. So that that technology exists most Engineering teams I've seen haven't used it has everybody signed up for the certification raise your hand if you have not Sorry, I didn't mean to trick you Okay The other thing is that promoting with this is a lot easier So if I have two files, and if I'm using something like customize I can have a patch called staging Well, I'm just never going to copy that one over right that's easy And so those elements of what's get what moves between environments suddenly it's down to specific files So that makes it a lot easier also as far as a source of truth goes if I Look at my folder structure. I know Exactly what's deployed. I don't even have to go to Argo. I don't have to go like Yes, I need to go to Argo to make sure stuff is sinking, right? But I don't have to know what the to know what the desired configuration is I literally just have it all sitting in my group repo already easy peasy done. Okay, so using an environment per folder that is that is what we recommend and And we do go into a little bit more here And it's worth reading through this, but we'll get into some of it with the exercise But like doing multiple changes at once is very easy to do this way because you can copy Multiple things between files right in different folders So that's very easy to do and then using helm, you know You can have your Different values that you bring into your you can have like sub sub charts sub values and things that you bring in and copy over And separate just like you would with customize Okay, so that makes that makes life a lot easier. All right the third option was an environment per get repo approach and This is basically the same as an environment per branch. Sorry per folder Except you have moved up a level of directory structure, right? Because each like you can copy on your machine multiple repos and you can do diff between them and typically The reason that people do this is because they don't know what a code owner's file is Right, it's just because they don't have that experience of doing it like I did a I did a survey earlier And I was asking people like how many of you have ever cut a release Like actually created a release object like in github, you know and almost nobody had ever done it Even though we're all deploying new software all the time It's just not something that like internal organizations are used to doing and it's the same thing with a coder's file Whereas in the open source world We use coder's every day So if it works for open source to use coder's where like you're giving access to people that you've never even met before It's going to work for your organization like it's it's it's just a generally better option the only reason that you might not want to go with something like a coder's file is because You don't want people to have read access to something in that configuration To which I would say what are you putting in that configuration that you don't want people to have read access to and you're like Oh, I got secrets in there and then I'll say what? Don't do that, right? So typically environment forget repos because there's some other Essentially problem or best practice that's being ignored and I'm not saying there aren't situations where that needs to take place Somebody came up to me earlier today, and I said I like your presentation because it acted like not everybody's perfect in the world I was like, oh good. Well, I need to do more of that. So This there might be situations where you'd go with this option. Okay so Go ahead and fire up your environment promotion exercise and what we'll do here is we're going to examine our Applications and then we'll show how promotion Would work with our different get repositories Now for this one you do need a get repo and because it is get ops you need to fork This repository to work now. I'm showing you the instructions. This is what you're going to have on your screen at the same time So if you navigate to this exercise You can open up this Get repo Go ahead and make you sure you fork it. I actually probably already have a fork I've got plenty of forks Yeah, so in this case, I'm going to go to sync fork I don't have any new contribution contributions, but you can see I'm I'm actually 14 Ahead so I could go back and like revert my commits and get rid of these to to reset Not a bad idea. Honestly Okay so All right, so I What we're going to cover in this one is how to deploy three different to three different environments a QA staging and production and How to model your environments using get folders and then how to promote releases from one environment to the next and then I'll also Give you a few extra bits of info So from here Why don't you go ahead and Let's try this if unless there's an rejection. Why don't you all go ahead and start working on this lab? Raise your hand if you get stuck We'll give you about 15 minutes to work on this lab And then I'll run through it relatively quickly and make a few comments. Does that sound good? Okay. All right So the environment that you are looking in your browser. It's completely online. It's just for you It's personal. You don't not see what other students are doing and you can experiment as much as you want after the exercise has finished It will be destroyed and then if you start the exercise another one will Start brand new But this is a real Kubernetes cluster and a real Argos the instance running In the clouds, so don't use it for any Production purposes as all data will be lost at the end just to make sure And if you guys are interested in the technology we use to do that or partner instruct is there That's where the people we use to basically generate those coaster on demand and stuff So tomorrow you can go you guys on the floor tomorrow. So go look at them and it's a great technology So we're very happy with them. Okay. I was gonna start running through the exercise Unless people start shouting and say, please don't do that Nobody is shouting Okay All right, so we'll run through this. I've got my fork up and running and While you were doing the exercise I got it all reset so I can see that it's currently up to date I've got my fork there. Good to go. All right So the first thing we're going to do is we're going to look at the Kubernetes manifests that we're going to deploy And they're under the environment promotion section you get out of me full screen. It's confusing me So if I look under the environment promotion subfolder, you can say I have a base. I have environments and then I have variants So let's look at the base folder. I've got a deployment customization and a service the deployment has some simple app in it the service has a trivial go web app service and If I look at my customization, it's literally just bringing in these two YAML files If I look at my environments, I've got a product QA and staging If I look at prod I've got my deployment my customization my replicas if I look at my customization It's got a couple of different strategic merges that it's doing to update that base Okay, so that's pretty simple. I know some people haven't used customized before so The customization you can think of all this like a helm chart, but it's referencing the specific Manifests that need to be updated and if we look at these Updates if I look at like this deployment for example, it's going to find a deployment That's named simple deployment and it's going to update Whatever container is named web server simple with these environmental values, right? and then I've got these variants That are basically matching non variants. Okay, so let's um, go ahead and start it Next The nice thing about these labs is it does kind of install and manage our go CD for you in the later Experiences, but there is an exercise you do at the very beginning to install our go CD So I've got to create three Argos CD applications one for each folder One for environments prod environment staging one for environments QA and These should be installed to the prod staging and QA namespaces respectively These namespaces do not exist and they need to be created either manually using create namespace or With the application, which is that I'm probably gonna do with the application You can use any valid method to create the apps such as the Argos CD CLI or the UI or your own custom application resource now if you're doing Earlier in the get-offs training. So if you go back and do other Exercises it will go through each of those and show you how to do them So if you haven't done it before And then if you're having trouble creating the Argos CD applications by yourself Be sure to look at the get-offs fundamentals course. So I everybody said that they had created applications before so I'm gonna go over and look at my environments prod QA Staging so I can create these applications any way I want now the way that I would do this in the real world Which is you know generally what the training is meant to be for Is I would actually create these using either an application set or a I would more realistically I'd probably create an app of apps and then pointed at a folder and then pointed at these three applications, but for For today, I think I'll just create them in the UI to be lazy but because it's more interesting than watching me like Type a bunch for applications sets or something. So let's go to our Go create our apps and My first one is going to be called Prod you need to make sure you follow the naming convention because that's how it's going to check You got it done. I can use my default Naming strategy and I can use a manual sync policy in this case I'll be using auto create namespace and I'll create the deletion finalizer the deletion finalizer for people that haven't used it before is Can be Tricky for new users if you set the deletion finalizer the way that kubernetes works is Any resource you'll look it'll always have a finalizer on it And basically once that finalizer is released a resource will automatically be deleted So when you set the resource finalizer as Argo CD if Argo CD Releases that it'll automatically be deleted. So what happens? I don't know if anybody saw the talk by Michael goodness earlier today But he one time had an experience where he accidentally deleted Argo CD Does anybody know what kubernetes does if you delete the thing that would be holding your finalizer? It deletes the thing so he deleted Argo CD. So all of his resources automatically deleted Yeah, now you're paying attention Yeah, I didn't see like a big deal. So that's what this this option does and so a lot of people don't use the deletion finalizer It just depends on how you're using Argo CD now if you're using it in a fully get ops way I Generally use this for my stuff And I know that you know Argo CD is not going to get deleted, right? But that is a potential failure condition to be aware of I Don't really need to set any of these other options auto create name space. I'm going to use for my source repository I'm going to use my fork so I've got my fork and Then my path is going to be Environment I can copy it Okay Then local cluster my name space is going to be prod name space, which is going to get created automatically and I should be ready to go. I'm gonna hit create. Let's see what happens Because I've got manual sync turned on. I'll need to hit sync manually for the other ones I'm going to use Auto sync, which is what I usually use and we'll talk about those options Doesn't look like I screwed anything up right now Awesome, so it's created Okay, so now I'm going to create a new one for staging I'll call this one staging default automatic In this case, I'm going to use pre resources in self-heal. What does prune do prune means that if a resource is deleted from get It will automatically be deleted from the cluster Earlier today. I talked about using version to config maps because config maps are like kind of Poorly used by operations by most organizations because they'll update a config map and Then there'll be something that breaks and they'll try to do a rollback And so the old pods will start up and bring in The new config map and they'll still be broken so you like can't roll back just by rolling back your previous pods So you have to roll back the config map to but if you do a version then it's a lot easier So anyway, that's a whole different topic. I'm sorry for bringing it up And then self-heal means that if someone edits it Live on the cluster like if I went and did a coupe CTL patch or coupe CTL edit Or someone just pushed another change Argo would notice that and it would smack it out of the way and say nope Whatever's in get goes get out of the way. All right I'll leave the deletion finalizer. It doesn't really matter in this case Auto create my name space my same Repository I've been working off of and Then my path would be This staging path, right easy Same cluster URL same name space Just a staging and I'll go ahead and hit create. Okay, so this one's gonna automatically get synced Now I haven't actually modified anything in my repository yet, right? So now I'll show you how I do this with app of apps really quick for funsies so To do this with app of apps What I would do is I would go into my environment promotion section and I Would create a folder called apps And this is going to contain all of my references to applications. Okay Now Inside of here, I'm going to create a file Called What's the one I haven't deployed yet? I Haven't deployed QA yet. So I'm going to create a file called QA dot yaml Okay And then under this QA yaml. I'm going to cheat And this is a kind of an extra little tip for you so this might be too simple for you you guys you're gonna think this is boring Dance too easy. I should have done the harder stuff and you can put on your headphones and just skip on to the next exercise move on I'm gonna set the sync policy to automatic. I'm gonna do prune resources Resources self-heal. I'll leave the deletion finalizer all out of precedent Let's let's not do the deletion finalizer because I told you how dangerous it can be Let's set up my repository URL Make sure it's coming from my fork and then I'm going to point it at QA and my local cluster and Something to be aware of with cluster URL versus name if you're deploying to the same cluster always just use the URL You don't have to worry about it But if you have multiple environments, I would always use a name Because then I can always just change what it's pointing at to bring up a new environment, right? So in this case, it's the same cluster so I'll just leave it as URL, but something to be aware of All right, so namespace QA now you're all thinking wait. Why did you do that? Because I just wanted to generate the yaml for me Then I didn't have to remember how to do it from scratch right Somebody somebody out there is just thinking. Oh That's really nice. Yeah, it is really nice. All right, so now I got my QA in here and I can go ahead and push this Check my git status git add apps git status git commit I'll sign this off and parent app for app of apps and QA app Okay, so I just pushed back to to my own version now nothing's going to be deployed yet because I haven't actually haven't created the app of apps So I'll show you I'll create a Parent app I'm going to do an automatic on this one and definitely I'm going to do prune and self-heal So if I delete the application yaml, it'll automatically get deleted from the cluster I don't need it. I don't need to auto create the namespace Same repository URL path is going to be Environment promotion apps, right? That's what I created Destination is local, but this one I need to send to Argo CD so that Argo CD will pick up the applications in there Yeah, so I'm going to deploy to the Argo CD namespace because I'm deploying the parent app That will then pick up the application that I just created and get and then sync that to the cluster Which will then create the next app Makes sense. So it's an app of apps and there's a whole exercise on app of apps So if this is if you're looking at this and you're like wait, there's like you can do that You're gonna want to definitely do that exercise so you can see it automatically picked up this QA Right and it's going to the QA namespace automatically. So and Then I've got my parent app here, right? And if I look at my parent app The only resource that it's going to show is this QA app and as soon as this QA app finishes deploying It'll show up as in sync. I can jump directly over to that QA Yeah, until a recent version of Argo CD You could not create the apps itself in a different namespace than the one where Argo CD was installed So yeah, we still kind of in the habit of do that. I think in your version It's not a requirement anymore. But yeah, now you can configure it and you can add additional to additional namespaces Okay, so now I should have three Applications deployed and it sees that they're all there and it's happy with me. Okay, cool Now Here's a question For you What if so I created those two applications in my cluster, right? What if I then created them in my get repository? What would happen any guesses? Nothing would happen That's right. Nothing would happen. It would automatically just take over management They already exist in Kubernetes so this is this is true as well of Existing Kubernetes resources if you have and some people will say hey I've got some resources that are not under management of Argo CD How do I get them under management of Argo CD literally just create an app that points at them and as long as The source of truth is the same as what's deployed literally nothing will change Except that now they're under management Right, so you can just migrate stuff over it'll add an annotation. That's what we'll do It'll add an annotation to let let you know that let our girl know that it's happening All right, so each of these is using a different selection of manifests all managed by customized overlays And so we're going to promote releases by copying files between the folders that hold the overlays Okay, so if you look at the manifest located in each folder each application is defined by the following files Version dot yaml that defines the container image settings dot yaml that defines the application level setting for the application So those are all just variables right replicas yaml that defines the number of replicas and service yaml that points to each application All these files are managed by a customized and are defined according to the promotion scenario of your organization in our example We'll mostly deal with the first two files. You can do a different split depending on your needs For example in our case we can move settings between different environments by copying the settings dot yaml file While we have different replicas per environment We can also have we have also included two customized modules the prod one for the production and the non-prod one that is used for both QA and staging environments you in the applications. We have created three separate tabs So you can see these these three tabs QA tab Starting staging tab and production tab and you can see that staging currently has version three Sorry QA has version three staging has version two and production has version one So now we can actually do a promotion by just updating these Just copying them right now if I'm using the structure. So if I go back and look in here I can do let's go into these Let's go into the ends here folder and if I wanted to do a diff between prod and QA I Can literally just run that diff right here and it will show me all the differences in these files, right? And so what I'm going to see is I've got things like my service node port that I don't want to promote But then I've got things like My values and stuff and this isn't a very pretty diff view, but you can do this with an ID Right, so give you the same thing. Just to show you how it works Okay So the way that we do our updates is we just copy File Right and we we make sure that the files are Representative of what's supposed to get updated. So if I'm going to update And you can actually label these two like you could create an overlay called staging And it's like I never update the staging one on production if that gets copied over I made a mistake, you know And you can automate it, right But let's go ahead and promote the current application version from staging to production By literally just copying these two files Okay, so Let's look at my let's just do this in the ui And I'm going to look at my environments and I'm going from staging to prod And I'm looking at the version dot yaml. So I'm going to go to my version yaml And I could run this Right by doing like a cp command or I can go to prod and version And paste right and you can see the only thing that's changing is the version number And let's go ahead and commit that to Oh, I need to add it Okay, I'll push that And this is going to automatically get picked up Uh, and I think my prod I had set up on a manual sync, right If I go to edit my application You'll see I can no longer get a preview of the manifest by doing that. I have to actually go to the manifest here Click edit directly And then that's how I would copy my application over to to move it back into management But let's go ahead and just sync it All right, so now it's picked up the change. It's going to spin up those new pods And I've done it. I've uh, I've done my promotion from one environment to the next Easy peasy Okay Let's go ahead and check that And of course we can automate this using ci now in the example we do this with github actions Um, you could do this with code fresh ci as well Um, I don't know if I should go through and set this up During this I think we should move to the next section, but let me let's just do a poll Raise your hand if you want me to demonstrate setting up the workflow Like one person. Okay. Raise your hand if you'd prefer that. I just move on to the next section That's most people. Okay. Uh, the eyes have it. So here, um, the these sections where it shows There's just there's I mean, I'll just show you very quickly If I look at github actions here You can see I've got a promote application Action workflow that runs and um Basically all this does and it'll they exist under the github actions workflows folder Basically all this does is Copy information from files. So if I look at like So here it's running like a copy file action between two files. It's not doing anything fancy. You don't have to do yq You know if you're if you're structuring stuff in the files this way You don't have to get into those more complex operations. You're literally just copying files Um, and that could just be automated, right? So that's that's nice and easy. Okay, so, um That gives you the baseline of like copying the files Showing you how that structure works and showing you how a promotion would happen and we can look at our production app We can see it's now using staging So let's go into the next section um Okay, so let's let's talk for a minute about image updater Then before you start make them start the next lab so they won't have to wait for them Yeah, go ahead and and go to the live exercise our go image updater and hit launch and I'll Talk at you briefly While it's launching Okay, so now we've shown Uh, how that folder structure works and I know it's like so simple. It's like, hey man You're just showing me like how to organize a folder and it's like yeah, that's all I'm showing you That's how simple that is but when you do that, right? You literally You literally avoid so many complex problems that come from using Uh, like branches for environments Um, just organize like good hygiene on your your config files makes your life a lot easier And before I even talk about image updater Um, I want to mention a tool called argocd autopilot Uh, I know I'm asking you to raise your hand a lot. How many people have heard of argocd autopilot? Wow, oh my gosh. Okay. So check this out. Here's here's several things. We didn't talk about remember when I created that parent app And I created in the ui Well now that parent app isn't in git, right? So if my cluster went down right now and I wanted to get back to where I was What who was paying attention? What would be deployed if I just deployed what's in my git repo? Did anybody remember it would deploy my parent app and what? And qa and nothing else right because all the other stuff I just created in the ui Okay, now Think about your argocd instances that you have in your organization How many of you have any of those apps were created in the ui? Okay, are you starting to get a sense of the problem? Right, so if I needed to bootstrap this it wouldn't come back. So what argocd autopilot does This is a tool that we developed for argo and it's a command line tool And it has The most remarkable command so you add a git token You add a repo and you run argocd autopilot repo bootstrap And what it does is it installs argocd And it sets up an application set That will and App of apps that will make it automatically self managing. So argocd is under self management And it automatically knows what folders to look at and once you run this command You literally never have to touch the argocd ui At all everything else is git So, um, I use this for my I have a homelab cluster. I've got like a three. Well, it's like five nodes I've got like a five node homelab cluster that I run everything off of and I came in one day and something had like Broken on the cluster. I think it was like I had let uh k3s Not be updated for too long or not restart for too long. So like the certificates expired So it was like a crashed so I had to reboot strap the cluster I literally ran argocd autopilot repo bootstrap dash dash recover Reboot strapped argocd reboot strapped all my applications. I was back up in like two minutes. Like it was so fast um, and I have like stateful workloads and stuff. I mean, it's like If you can't do that if you can't if you can't delete your your stuff and then just run reboot strapped then Like this you definitely should finish the whole certification and then start implementing it, right? So Oh, yeah. Yeah. Yeah, totally. I totally view it as disaster recovery, but it's an it's an it's an ops problem Right, but like if you solve your ops problem You also solve your disaster recovery problem Now you do have to think about stateful stuff and like how that gets reboot strapped and so you need to have like a solution for that and uh, costus grave gave a great talk today about um, Updating databases with get ops that would be very applicable to that kind of setup, right? So definitely check out argocd autopilot Go start this repo and um, and try it out, but I'll show you like My home lab cluster um, you can see I've got It created my bootstrap projects and user and then under projects It has a application set that generates everything for that project, which is pointing at a specific cluster Which is just the local one And then I created an apps folder For my parent apps, right? And then I just have If I want to add stuff specific, you know, then I do that that way, right? If I can just add it directly and get Okay, so image updater Is solving a different problem What image updater allows us to do is monitor how many people have used image updater Okay, so this is new for most of you. Good. All right, so image updater is a really cool tool We have within argo. We have a whole section of tools that are called argo project labs They're technically not cncf projects They're just community projects and anybody can make them if you want to build one at some point You know and add something to argo project labs, you know Just getting contact and what we can help set you up to do that but Image updater started is that and what it can do is it will monitor an image registry For tags and you can set this by semver You can set it by specific, you know patterns or whatever and then it will automatically Update your git repo with the new tag So if you're just Promoting new images image updater like takes the headache out, right? It's very easy because you're just saying hey when a new image gets tagged for staging Automatically deployed to staging, right? But it's versioned and it can handle all that so image updater is very powerful and nice to use And So the way that the way that The way that image updater works Is it has by default it will just update the workload in argo So it'll just specify a new manifest for argo cd deploy So if I don't have anything in git And then it has an option that's called get right back and get right back will actually update git So you have to enable that as an additional option. So let's go through and and do that lab There's not really more for me to explain off of that. Yeah, that's good So let's do this. We've got We don't have a ton of time We've got till We've got about 15 minutes So go ahead and fire up the live exercise if you haven't already done it um And let's do this exercise and then uh, we'll have one other tool we'll introduce before the end That's not good. Um, and then raise your hand if you didn't get an argo tumbler Oh, it's like most of you. So, uh, maybe we should do laraunt. Let's just open the box by the door So as people go out, they can grab a tumbler. We've got more than enough for this room. So we should be good Yeah, just take one. Yeah. Yeah So actually let's let's do this. So, um As you start working on that all I just want to introduce kind of like the final like two little topics Um to be aware of so after you finish this the next, uh labs are on sink hooks sink windows um And uh, and then of course if you haven't done the app of apps or the application sets earlier Then you should do those exercises and then you can take the test and you'll be level two certified The level one if it looks too simple, just go do the test But it'll show you how to do progressive delivery and things like that now, um with the if you think about this problem of like updating manifests and updating these resources Doing it with a ci pipeline works Image updater is going to cover binaries for us, right just images um, and probably like a good I don't know 60 of the time all you're updating your images So for handling things that are out of phase from just updating images there is uh A million dollar question, which is like how do you update the configuration as well? How do you do it across environments? and When I say it's a million dollar question, uh code fresh is betting that it's more like a 500 million dollar question um We announced this week A new tool a new feature for modeling environment promotion That allows you to model not just images, but also configuration And you can also model specific lines of configuration that go through promotion So you can do like staging to production promotion and it knows Which sections are supposed to be promoted and which ones aren't supposed to be promoted so We're not going to demo that today But definitely stop by the code fresh booth. We can give you a demo of it and show you how that works There's a really cool spec that goes into it. I don't want to spoil any surprises But ai's part of the story because that's fun like you have you have to have ai You know, nobody's going to care if you don't but ai's part of the story um modeling environments across multiple argo instances Multiple clusters. That's part of the story. So definitely check that out. Um That's the final word on it. And then uh, I think what we'll do is we'll let you work on the Image updater lab if you have any questions. We're here to help answer those and then It just thumbs up like recommend so far so good or did you feel like hey, that wasn't enough time Like I need six hours to do this Which is true thumbs up people are feeling good. And then you'll be able to work on this on your own You can do the entire rest of the course and the level one and the level two and of course level three Is in production. So that'll be out later