 So my name is Costis, I'm a developer advocate working at CodeFresh. CodeFresh is an enterprise platform for Argo. I'm also a maintainer for ArgoLouds, and I'm actually contributing to documentation, so not that features. So if you wanted to contribute to Argo CD, you don't want to let go long. There are ways to do it, come find me after the talk. I'm also the co-author of the first ever Argo CD slash GitHub certification, which is something different than what was announced in the keynote today, and I'm going to say some things about this in the end. So what are we going to talk about today? So you install the Argo CD, or Flux. It works, it's stable, and then you start using it, and your job is to approve pull requests. So this is a pull request that some of the else had created it, and your job is to look at it and say if you want to approve it or not. So you look at it and you see, okay, it's changing the number of replicas, it looks good, maybe it's Black Friday, and you want to increase the replicas in production, you approve it, you merge it, and nothing happens. And nothing happens because, let's say you're using Customize, you change this file in a Customize overlay, but there was another Customize overlay downstream that had an override in the replicas field, and your change did absolutely nothing. The same thing can happen with Helm values. So this is a scenario we are talking about. You have a structure of Customize overlays, this is your base overlay, and then you have overlays for staging and production, and they have downstream overlays for different regions, or you can imagine the same example with Helm values, and they have hierarchies. So by the time you look at a pull request and you need to approve it, your context is only the overlay or the Helm value for the change, which is not enough, this is not what you want. Ideally, you would like to have the full context, know the whole hierarchy, and know the actual change, because then you can make an informed decision and say, yes, this is the change that I want to approve or not. So this is a problem we are going to talk about today, and I'm going to show four solutions, three wrong ones and the correct one. Especially for Helm users, you also have an extra problem that Helm is using templates. So if your Helm templates are super complex, sometimes you need to approve a change like this, and one of my favorite things is to mentally run Helm in my mind and understand what is going to change, said no one ever. So it's also the same problem or similar problem that ideally you would like to see the full manifest and not the Helm template, because this is what will go to ARGO CD. So let's see the choices, what you can do. The first choice, which is obvious, if you have seen ARGO CD, has the capability to show you a diff. So if there is an out-of-sync resource, either something has changed in Git or in the cluster, ARGO CD can mark it for you, and it tells you exactly what is the resource that changed, and then you can click on it, drill down and see what is the change. And in this case, you are looking at final manifests. So all the overlays are not importantly more, the Helm templates are rendered, you are looking at the final change. This is what you want and it looks good, but it also has some issues. First of all, in order for this to work, you need to disable auto-sync and self-heal, and you shouldn't do this because then you are not truly following GitOps. Both these options should be enabled. So by the time you change something, it's applied right away and you don't see any diff. The other problem is that this happens very late in the process, like ARGO CD is looking at the final manifest that is committed and pushed to the branch that is defined by the application. So this is super late. If you disagree with that change, it's not very difficult to do anything. But on the other hand, it's built into ARGO CD, and there is nothing to set up or create there. So can you use this? Yes, I think you can do it. I would only do it in production environments where many people don't want to have auto-sync enabled. They want an actual human to be able to approve the pull request, and then I would use it only for sanity checking. So I have already done my, let's say, official checks and my official approvals of the pull request, and I'm just having a last set of eyes to do it. But only for last minute, let's say, verification. The other thing you can do, and this is also another good question, how many people have used the diff command in the ARGO CD-CLI? How many people have used the minus-local argument? There, okay, some people. So there is a similar thing to do, a similar capability with the ARGO CD-CLI that allows you to diff the application, like what is in the cluster against local manifest. And again, it has full support for everything that ARGO CD supports. So if you're using Helm, it runs the templates. If you're using Customize, it applies the customizations. So this is how it looks. I run it here and I see some changes. If you ask me, this is one of the features of ARGO CD that is not publicized very well. But it's really important because you can diff against files that are in your laptop, like they are not committed anywhere, they are not on a Git repo. And you can imagine that if we have this, let's say, building block, the next obvious thing to do is to have your CI system run this as a command, as part of the CI process, and then attach the result in the pull request. So by the time you look at the pull request, you not only have what your diff provider is giving you, but you also have the capabilities of this ARGO diff command. So it works. It solves the original problem. You have the full context of everything. Again, it's built into ARGO CD. There's nothing that you need to do there. But it has some issues in regards to security and how it needs access to the ARGO CD cluster. In order for this comparison to work, the CLI needs access to the Kubernetes cluster and the ARGO CD instance. So you need to allow your CI system to have access to your ARGO CD cluster and maybe you don't want to do this. Or maybe you are not even allowed to do this. Maybe let's say your CI server is in the USA and your ARGO CD instance that you're deploying is in China and maybe it's not even possible to do this. So your security people might not be very happy about this and you also need to do some networking setup. So my recommendation is great if you are experimenting locally. If you want to try something quickly on your laptop and you create a manifest and you don't want to commit it somewhere at this point in time, you run the diff and you see, okay, yes, this is what I need. But I wouldn't use it for any production, instars or anything else. So here I have an intermission for Terraform because I think this is really interesting. This problem that we are talking about is not really new. Terraform people have the exact same issue. Normally you don't want to run Terraform Applying. Terraform also has a plan command where it tells you what it's going to do with infrastructure and by default it presents a nice report where you can see, okay, my change is really what I need. So the obvious thing to do again is to take this report and have your CI system run it for you and then attach that report to the pull request. So by the time you look at a Terraform pull request, not only you have the diff in the Terraform files, but also you see what is actually changing in your infrastructure. And this is a very common scenario. There is actually an open source project called Atlantis that you should look if you're not familiar with this but it has exactly that. It attaches Terraform plans to the pull request. So in a way it's similar to the Argos in this CLI workflow that we've seen before. But again, there are some issues here. In order for this to run, Atlantis needs the Terraform CLI, the exact same version that you use, access to the Terraform state. And if you're using Terraform, you know what that means and its implications and access to your cloud provider. In a sense, it needs everything. It has the keys to the kingdom. So, yes, it works, but again your, let's say, security people and networking people might not be very happy about it. And I'm mentioning this because this is something that was already present for Terraform people and it doesn't mean that we need to do the same thing and the same workflow. Maybe there is a better solution in the case of Argos CD. You're pre-rendering the manifest. That's another solution and I'm happy that we have a guy from Intuit here because I love this. So another approach is essentially you pre-render the manifest and save them to an external Git repository. And this is a third Git repository. Remember we said that one of the good Argos CD practices is to have a Git repository for the source code and a Git repository for the manifest. Now we have a third Git repository that has manifest. Argos CD is looking at that second Git repository. Argos CD is only looking at final rendered manifest. It doesn't know about anything else. So the way it works is that you as a developer work on one Git repository that has your, let's say, Helm templates or your customized files and then you need an automated process that renders the manifest when you do a change and commits them to the second Git repository. And here I'm mentioning a second Git repository but some people might prefer a different branch on the Git repository or a different folder. The end result is exactly the same. So this is how it works. One Git repository is the one that humans interact which is at the top and it has my, let's say, original files and then the second Git repository is at the bottom which is the one that Argos CD is looking at and it has only final rendered manifest. And you need to set up an automated way that takes the source, the source manifest and then renders them whenever you do any change. Now this might look simple in theory but in practice it needs also some other stuff. I as a human should only interact with the first Git repo so I'm going to create a pull request there but the diff I want to see if you remember the previous slide is on the final rendered manifest. So I also need a second pull request on the other Git repository. So how do you create this manually or maybe you have your CI system does this for you and then once I approve my pull request if I approve the pull request on the first Git repo I also need the pull request on the other Git repo to be approved or if I reject it. So I'm just saying this that this needs some setup and you need to account of all the scenarios and workflows that come with this approach. But the end result is great. So this is the same example that I've shown in the beginning with Helm. Here I have a diff and because I'm doing it on the second Git repo that has the final rendered manifest I actually see a manifest and not a Helm template. So as a user this is very easy for me to say okay this is what I need and I don't need to run Helm template in my mind. So the problem is solved with this approach. So you could use it. There are some companies that are using it and into it is one of them but for others it solves the original problem. You're looking at a full context especially if you use Helm it's super useful but everything else is a disadvantage. You need to set up this copying slash rendering process. There are more moving parts right now in your deployment process. Your CI system also now becomes a point of failure because if this rendering process doesn't run you cannot deploy it. So if your CI system is down and your ARGO CD is up there's nothing that you can do. And also if you're working in a big company you need to pay attention to onboarding people and to explain what is the process because a very common mistake is that people will commit to the wrong repository. So maybe I'm starting a new position in this company and I go to the ARGO CD instance I look at the applications I look at the repositories that are used and I find the second Git repository the one that has the final rendered manifest but you don't understand that this is not the way I should do changes. So you need to pay some attention there and play some guardrails on what people can do and they cannot do. And if you are using a separate Git repository now you have a double number of Git repositories. You have essentially three for each application the source code, the final rendered manifest and the Helm charts. So it works but if you ask me it's complicated then you need to have let's say a well-disciplined team to make it work. So if you're already using it that's fine but I think there are better solutions for this. So yeah, that's the result. You can use it but make sure that you know what you're doing and you can explain to people what they are doing and what they should not be doing. So we have seen the three let's say approaches that I don't like and this is the one that I personally recommend rendering the manifest on the fly. So here remember I said that we should not look at what Terraform is doing and re-evaluate our stance according to what Kubernetes and ArgosID is doing. One of the problems that we had with ArgosID CLI is that we needed to ask the cluster about the current state and then perform a diff. But if we are following GitOps we already know what is in the cluster. It's the same thing as in the Git repo. So we can cheat here and do a comparison just between Git. So say, see what is the pull request that is coming and this is the proposed change and assume that what is in the cluster is in the main branch or the master branch. So we perform a diff between two Git states to get different branches and not the cluster. So this is how it works. We still have a single Git repository that you can see at the bottom. The developers of DevOps still interact with a single Git repository and whenever I do a change an automated process just renders the manifest on the fly by doing this Git diff between the pull request and the main branch and then attaches the result as a comment. And ArgosID is looking at the exact same Git repository. So it's a single Git repo. The CI system does something but it's not a single point of failure anymore and I get a nice diff. And I integrate that to my pull request process. I'm also going to show an example but in any CI system it's super easy to get the files of the pull request the files that are in the main branch or the target of the pull request and then perform a diff and attach this to the issue. So this is a live example. I'm looking at a pull request and this is a comment on the pull request. And unlike the first example I've shown this comment has final rendered manifest and I can see again exactly what has changed. It's not helmet and blades. It's not customized files. So as an end user I get what I want and the problem is solved. So is it perfect? No, it's not perfect but in my opinion it has many advantages. First of all we're working only with Git. We don't need access to ArgosID anymore or the Kubernetes cluster. We don't need about the topologies. There are no problems with networking. We still use the support that ArgosID is using for helmet customized. There is no confusion over multiple Git repositories and it also works for Flux. So all the previous solutions maybe they were ArgosID specific. This is not so even if you are using Flux you can do the exact same thing but you know what is going to change. So if you move from a company that has ArgosID and go to a company that has Flux you can do it. There are some disadvantages. First of all you need to set up this. Again you need to set up this rendering plus commenting process but as I said your CI system is not now a single point of failure so if your CI system is down you can still deploy but you don't get access to this nice div and specifically with Helm you might miss our corner cases but you see with this div with what actually goes into the cluster. So maybe in some cases it's misleading but I think the advantages outperform the disadvantages. So my recommendation is that this is a very simple process. You should use it. It's scalable and it can be used for both production and testing environments. And as a bonus I also want to mention another scenario. We have been talking about looking at the changes and understanding what will change when you merge a pull request but another important scenario is understanding what will not change so the opposite. So in a previous presentation I was talking about organizing your GitOps repositories and how to model your applications in a Git repo and there were also some other presentations talking about this very same topic. I'm not going to go into this right now but I'm going to talk about a very specific scenario. So this is a folder hierarchy. The purple boxes are folders and the blue boxes are let's say customized overlays or Helm values, whatever you prefer. So on day one somebody comes to you and says we need to have this change in staging. So you make a change in the customized overlay that is in staging. You approve it, you merge it, you apply it and it works. One week later they come to you and they say same change in production. So you go to the production overlay you do the change, you apply it you merge it and it works. And then you look at this and you say okay, I have this duplication if you're using customized the obvious thing to say is okay, I'm going to remove this change from the individual overlays and since it's common I will add it to the base overlay one level above or if you're using Helm values maybe you want to move that value one level above. So that final step is mentioned in that article about the GitHub repositories and several people said oh this is too complex I'm not going to do this refactoring because I don't know what will change I don't want to touch too many environments at the same time this is super risky and so on and so on. And when I look at the comments of those people I realize that they didn't have this presentation in mind. So this is the plain diff that the Git provider will show to you and you can see the changes are not really important but you can see a change is removed from two environments and then you add another change in the base overlay. So you look at this change and you say oh this is touching three environments it's super risky I'm not going to do anything forget about it but if you adopt the approach I'm talking about about creating a smart diff this is what you would see So this is my own smart diff that compares the main branch against the pull request I have three environments and you can see it's absolutely empty So I can look at this pull request with my own smart diff ok smart diff plain diff and say ok I can merge this with 100% certainty that nothing will change I will merge this and Argos CD will do absolutely nothing I touched I touched three environments So I did a refactoring and I know that my refactoring is correct there is no risk and if you have this information of course you can do as many refactoring as you want without any fear So I'm saying this is again super important because now with this system I know what will not change not just the changes So think about this scenario as well Now many people ask how do I get this you are at the bottom and there is an example with github actions but you can do the exact same thing with CircleCI, codefresh, Jenkins whatever CI system you use it's not that complex So we have seen five solutions about previewing the changes that will appear in your application first we have the naive gith diff that doesn't have any context so don't use it but we have the CLI diff that argosid is offering this is a great building choice but it's far too late in the process so I wouldn't recommend it we have the CLI local diff which is great for experimentation but if you want to put in a production instance you need to give access to your argosid instance from your CI system which is something that maybe you don't want to do you can pre-render manifest and place them on a second git repo this is a solution that it works it's complex if you ask me and you need to explain to people what is happening behind the scenes and how they should use it and my personal recommendation you render the manifest on the fly and you attach them as a comment on the pull request so some resources the first one is a link to the blog post that has this presentation plus more details the second one is the Atlantis project which if you are a terraform user you should look at if you haven't seen it a certification which is a training not just a certification where you get an argosid instance on your browser and you can work through various scenarios and it also includes the thing I talked about with multiple environments and promotions between environments so in this case there is a scenario where in your browser you get three real Kubernetes clusters plus an argosid instance that controls these Kubernetes clusters and you can do promotions and there is a cost to this certification because of this cloud provider but with this code you get 100% of this price there are maybe less codes than people in the room so that's it we have five minutes for questions you can do a help lookup in Argo you have the values potentially in the application and plugins which can decorate based on the cluster the last thing thank you is exactly that sorry I think I was asking the same question in terms of how do you account for configuration on the application resource with regards to helm or customise in that way but is there version differences as well between what you're running in CI versus what's actually running on the Argo CD server this problem can be solved let's say easily you can query Argo CD and ask what kind of versions it uses one question that you didn't ask is what you do with application sets so if I'm using let's say application sets ideally I would also like to preview what applications are created by application sets so it's not that simple but it's a good let's say starting point of getting into this mentality I will use the GitOps principles and assume that I have something in Git and it's a trade-off if you have something that works 90% of the cases maybe this is enough for you and for 10% you do something else rather than doing something else for 100% of the cases and making your life difficult but if this could be a real setup especially essentially you would need to replicate everything that Argo CD does dynamically for an application including application sets and Helmer Customize or any plug-in that you might have in Argo CD so all these might miss some corner cases at the end Hi, I just want to I'm not sure if this is also a corner case where like so right now for this method you're assuming that it doesn't ever need to connect to the Kubernetes cluster it doesn't ever need to connect to the Argo CD API and so like if that was a resource that got edited by something else basically I think what my developers would say like well I don't have 100% confidence that this diff is going to show me the real what's actually going to happen if when they compare it to tar from plan that they were used to that actually connected to the Amazon API and that gave me real time this is what's going to actually happen but I don't understand here like what's of confidence can I give to my developers that this method would be give will give you a reliable way to say like hey wait about the change the helm chart is going to be there I think that the people that are supposed to approve a Helm request they should have the knowledge in the first place of you know the corner cases and they should be able to say ok for this change the smart if I'm looking is really smart or what I'm looking is not that smart so I won't say that this is something you offer to developers I'm going to say this is something you offer to the people that approve pull request for Kubernetes manifest if in your case they are developers and you assume that they know what they are doing that's fine but if in your case it's only a very small subset of people and they are the experts on the Kubernetes manifest they should also know what they are doing thank you another one there's no API in the Argo API to render the output where you could pass it a call and say I want you to render this for me as far as I know ok thank you because the idea I was thinking is you could just get Argo to render this for some arbitrary PR right ok thank you so for people that don't know him we are lucky enough that we have Michael the Argo CD guy here so if you have any advanced Argo CD questions you can ask him ok any more ok thank you very much