 Welcome to our talk on multi-stage deployment pipelines, the GitOps way. I'm Kent Rankort from Acuity, and I'm joined today by one of our co-founders and our CTO, Jesse Sewin. So what exactly do we mean by multi-stage deployment pipeline anyway? When you hear that, most of you probably have a pretty good idea of what we mean, but it doesn't hurt to take a few minutes to make sure that we're all starting from the same place. So let's imagine for a moment that you're just setting out on your GitOps journey, or as we like to say, so you've decided to GitOps. So we'll start with the Kubernetes cluster. That seems like a good starting point. And this is GitOps, so of course, we have a Git repository that's going to be the source of truth for our application. And now we need some way to sync the contents of this repository to our cluster. The founders of Acuity are also the creators of the Argo project, so we're a little bit biased. We'll use Argo CD for this. So we set up an Argo CD application resource pointed at our repository, and it syncs everything and continues to keep everything in sync. And everything is great. But now what? We've got one instance of our application, continuously syncing to the contents of our repository. Let's call this instance dev. Well, it's pretty unlikely that dev is our only environment. So let's be a little bit more realistic while not overcomplicating things. We'll say we also have an instance of the same application running in a QA environment and instances in each of two production environments. So here it is, an example of the titular multi-stage deployment pipeline. And in our experience, this is where most people encounter their biggest pain point with GitOps. If we like the state of the instance running in dev, how exactly can we promote that state to QA? Now, I know what you're probably thinking. You're thinking, I use customize, so it's just a change to the QA overlay. Or you're thinking, I use helm, so it's just a change to a QA specific values.yaml file. And you're not wrong. But how can that change itself be made automatically or at least with minimal manual intervention? Well, it turns out that GitOps doesn't really have an answer for that. And this is where GitOps practitioners start to get creative. Now, the implementation details tend to vary, but broadly speaking, there's a common pattern that we observe people using to solve this problem. And we call this CI kung fu. And here's an example of what it might look like. So starting from the top, we see Argo CD has just synced the dev instance of our application. And that's basically where we left off in the previous slide. We got clever and we used a post sync hook to signal our CI system, Jenkins in this case, to execute some automated verification process so that we can assert that the application is working as expected. Because just because the application deployed and Argo CD says it's healthy, doesn't really guarantee that everything is okay, right? If those tests complete successfully, and this entire diagram is the happy path, by the way, then Jenkins will run some scripts to take whatever change had been applied to dev and apply it to QA. Asynchronously, the Argo CD application that deploys the QA instance of the application will notice that change and it will sync everything. Then part of what we just discussed repeats. A post sync hook signals Jenkins to run some automated verification. This time if it passes, notification is sent to a human user, maybe the product owner. And if they want to progress the changes into production, they can approve it through some means. To keep things simple, here we just have them kicking off a job in Jenkins manually. That job will run some script to take whatever change had been applied to QA and apply it to either one or both production instances of the application. Here I chose for it to update London only. Most of the process we already discussed repeats and the prover may choose to promote the change to the second production instance in Zurich. Frankly, just explaining this makes me tired. Imagine how little I would care to be stuck having to implement this. But this is what people tend to do. So apart from my personal aversion to this, what's objectively wrong with it? For one thing, it's obviously pretty complex. And this was a simple implementation that really only accounted for the happy path. If you want to also account for the various failure cases, it gets a lot more complex. Another problem is that it's linear and it's inflexible. It can take the latest change from one stage to the next, but how does it handle rollbacks, for instance? Or what if you want to return one stage to some arbitrary state? Let's say you want to make Dev look like it looked last Monday, for instance. How do you do that? I wouldn't claim that it isn't possible, but it's just another thing that makes this already complex process even more complex. Worse still, this is incredibly tedious to replicate if you have dozens, hundreds, or even thousands of applications that need to follow this same process, perhaps with small variations from AppTab. So last and certainly not least, there's no mission control center for this. There is no single place to go see what is running where. And then remember that we deliberately started out with a pipeline that wasn't too complex. What if your pipeline looks like this one? Here, we've not only added additional stages, but we've also added two parallel stages where we conduct some AB testing. So when that testing is complete, whatever process we use needs to account for a user manually selecting one of two candidates to progress to the next stage. So maybe it's time for a little bit of a reality check. CI systems tend to be very flexible and therefore useful for automating things that aren't really CI concerns. So for far too long, engineers have fallen back on their CI systems to solve the problems that their CD systems do not. So much is this the case that we've mostly taken to saying CI CD in one breath as if they are somehow inseparable. But looking back on the complexity of the sequence diagram that we picked apart earlier, we start to see that this codependent relationship is not necessarily a good thing. So let's decide here and now that we're going to stop over leveraging our CI systems. We'll use them as intended for producing artifacts synchronously and quickly. And everything else is in the realm of CD. And it's time for CD to step up and become more responsible for itself and stop leaning on CI for help all the time. And this is where cargo comes in. Cargo is a new open source project from Acuity and it aims to provide a more elegant solution for orchestrating the kind of multi-stage deployment pipelines that we've been discussing. Cargo allows you to model those pipelines declaratively and it provides you with the means not only to progress changes from one stage to the next with minimal intervention, if any at all. It also allows you to roll back changes or restore a stage to any previous state for that matter. It supports three kinds of artifacts, container images, helm charts, and Kubernetes manifests read directly from a Git repository. And in that case, it also has built-in support for customize and helm. And it optionally integrates to the sub-project called Cargo Render that can use configuration management tools like customize or helm to render Kubernetes manifests as plain YAML that it stores in a stage specific branch. There's some really interesting advantages to expanding your YAML before Argo CD or Flux or whatever you're using to manage deployments gets its hands on it. I won't go too deep into that today because my colleague Christian Hernandez is doing an entire session on just that topic and I really suggest that everybody check that out as well. At this point, I'm going to hand it over to Jesse who's going to show you Cargo in action. So I'll be doing a short demo of how you can use Cargo to promote changes between environments using GitOps. Okay, so the main view of Cargo is the pipeline view. And so what you're seeing here is a multi-stage deployment pipeline with a number of different environments ranging from dev all the way to production. So in Cargo, you will typically model your deployment environments as stages. Stages are independently defined and loosely coupled with other stages. And we found the finding stages in this way let's allow allows you to have a lot of flexibility and achieve some pretty complex and powerful deploy pipelines. So on the far left is our dev stage and you can see that dev is subscribed to two things. First, it subscribes to image container repository. And then second, it subscribes to a Git repository where it watches for new Git commit pushes with manifest changes. And Git subscriptions is a unique feature of Cargo. Cargo lets you promote YAML changes in Git from stage to stage just like you might promote image tag. And we'll see how that works in a little bit. So whenever Cargo detects new changes to these repositories, it produces what we call freight. And freight is basically an artifact that you wanna promote and deploy as a unit. In a common case, freight will be a new image tag but it could also be in this case a change in your YAML. And so at the top is what we call the freight line. The freight line is an ordered list of freight that Cargo has detected in those repositories. And as new images or commits are pushed they start appearing on the left and the older ones on the right. And we also have these color indicators these map to the stages that you see in the pipeline. And it's there to let you know what was last promoted in each stage. So looking at this picture, you can get an idea of like how far behind certain stages might be from others. Here we see that dev is four changes ahead of staging and then prod is another four changes behind staging. So going back to the pipeline view we see that the dev is connected to a downstream stage called staging. And what this means that anything that successfully deploys in dev is a candidate for promotion into staging. Following that is where things get a little more interesting. So after staging, we can see there's two parallel stages AB test A and AB test B. And here we're demonstrating the ability to do AB test so I can promote one freight to the A group and a different freight to my B group. And then seeing how they perform I can make a decision about which freight should ultimately go to production. And we have these finally we have these three production environments London, Paris and Zurich. And these are all subscribed to this what we call control flow stage called EU. And this is not actually modeling an environment this is what we call control flow. And we found in cargo we found it's useful to have a stage to use as an interaction point. So in this case, prod EU is used as a join point so I can coordinate the promotion of the three downstream stages at the same time rather than promote these things individually. Okay, so now let's actually get to promoting something. cargo does support fully automated promotion as well as manual. And so in this demo I'll just be doing manual promotions. So to promote something you click on this target icon to the left of the stage. And one thing to note is when I click promote on different stages my freight line becomes filtered just to a subset of freight that I can promote. So when I promote to dev you'll see everything available when I try to promote the stage is a subset and then going down further to prod is even a smaller subset. And cargo only lets you promote a freight that is qualified for that stage. So what do we mean by qualified? As of today, something's qualified if it has been verified to be successfully deployed in the upstream. So what that means is like if you're using Argo CD cargo will also make sure that the Argo CD app underlying application reach the healthy state. And in the future cargo will have the ability to execute your own tests or scripts or even query metric providers like Prometheus to perform the verification. All right, so now let's actually do a promotion. So earlier I introduced a new environment variable to my manifest and I just added this GitOps con EU environment variable to my customized base. So to promote I click on this target icon and I'm gonna promote the dev. I select just the most recent freight and then I can promote click yes to promote. And so what will happen is it will make a commit to Git and then follow up by Argo CD sync if you're using Argo CD with cargo. And if we look back at the commit that it made this is a commit that was actually made by cargo and we can see that it actually made that commit with the new environment variable that I introduced as well as this new image that I introduced because going back at the freight line we can see that there was two changes. There was the new image tag as well as this new environment variable. Okay, so we just saw how to promote how cargo does a GitOps based promotion. I also wanna show how you can use cargo to promote something downstream to multiple subscribers. So in my pipeline, I'm actually currently in the middle of an AB test. So my A group is running 041 and my B group is running 040. So 041 and 040. So let's pretend that looking at logs or metrics I decided 041 was the better of the two and I wanna promote this to my prod environments. So to promote something downstream you can select this little truck icon on the right. And when we promote this way downstream we actually will kick off multiple promotion jobs one to each downstream stage. So you can see even the UI I can select from these qualified freight and I decided that this one was better of the two AB tests and I can promote this one. So this will go ahead and make three Git commits to my three environments. And going back to my GitOps repo you can see I just recently made three different commits to my different environments. And we'll just look at one of these. And we can see that the 041 image was just promoted to one of my environments. Okay, so that was a quick demo of cargo and how you can use it to orchestrate your environment promotions using GitOps. Hopefully that gives you a taste of what's possible with cargo. And we have other interesting examples and use cases that cargo is solving including canarying and orchestrating multiple microservices and we're working on a lot of other features like pull requests and automated testing to be included as part of your promotion process. Thank you, Jesse. Now cargo really is in its early stages still. So I want to share some of the highlights from our roadmap with you just to give you an idea of where the project is currently headed. In the interest of time, we're not going to unpack each one of these but I invite everybody to re-watch the video later and take a closer look at this slide. Once again, we are in the early stages of the project so it's also a really good time for you to get involved and give us your feedback. Maybe even open a PR. You have a really good opportunity right now to influence the direction that this project takes and we'd really love to hear from all of you. So I'll wrap up with a list of resources to get you started. Here we have links to our documentation, our Discord server and of course our GitHub repository. We're looking forward to hearing from you. Thank you for your attention and enjoy the rest of the conference.