 Thanks for joining. Welcome to GitOpscon, North America 2021. I'm Ryo Tasawada. This is your declarative configurations human-friendly. Close the look-out, code reviews with GitOps. Let's get right into the agenda. I'll start with a brief introduction myself and the company upside-down, and then we'll touch on the target audience and the goal of this talk. After a quick brush-up on helm and customized, we will look at some example pull requests with them. We will then take a look at this brand new project I have been working on, Importer. I'll briefly talk about the importer details, and we'll go straight into demos. Finally, I'll touch on other tools and considerations for great code review experience with GitOps. So a little bit about me. I'm the CTO at AppSider, leading platform team, and also responsible for backend development. I have worked in finance space for about a decade now. You can find me at usual places with RIT, SWD, that's my full name without vowels. I'm based in London and have two cats, as you can see in the picture to the right. You can find that picture used as my profile picture in most places. Also, I'd like to briefly touch on the company I work for, AppSider. It is a fast-growing fintech startup providing V2V's payment services. We are CNCF end-user supporter and use GitOps extensively in production. The headquarters is in Tokyo, Japan, but the remote team is around the world, as am myself, I'm based in London. Before moving on to the actual talk, I have to touch on this must-have topic. We are hiring. If you're interested or curious to know more, please feel free to reach me at RIT SWD. There are mainly two materials, this slide, which is made available via tiny URL shown here, and the repository we used for demos under my GitHub profile, RIT SWD, slash GitOpscon-NA-2021. This talk specifically focuses on code review with GitOps. As a prerequisite, it is expected that you're familiar with the GitOps solutions and also use of helm and or customize with GitOps. For the target audience, if you review code for GitOps setup day-to-day, are looking for better code review practice or just had enough of helm-chart syntax, hopefully this is the talk for you. This is probably not for you if you have no issues or whatsoever with code review with helm, customize, and or other tools for cluster setup at the moment. By the end of this talk, hopefully you will familiarize yourself with some of the items mentioned here, such as understanding how GitOps requires extra care for code reviewers, tooling to help helm and customize setup, very first look at importer and more. I know we all love code review and this is probably clear to everyone, but let's touch on this briefly. Code review is done to ensure code correctness and get explicit sign off from reviewers. In case for GitOps, this would be the sign off to actually deploy something to clusters. There are several tools and ways to review code, but for this talk, I assume we are reviewing code from the web UI and we are only talking about GitOps code. I think it is safe to say that reviewers should have a good understanding of Kubernetes resources, but what about the other tools? Are they truly required? If a new configuration management tool emerges, does that mean reviewers need to learn the new syntax, best practices, quirks and everything else by the tool? The more tools you have, the more difficulty it gets to review correctly. We will check out some example podcasts in a bit, but let's have a quick brush up on Helm and customize first. So Helm is about its chart and the key files are as shown here. Helm charts can be quite complex and from the code reviewer perspective, go template language for templates can be difficult to understand and review. Customize is about customization.yaml files to define the list of files you need and use those customization.yaml files with base and overlays to generate a complete spec. Thanks to its patching capability, environmental differences can be handled easily like the structure shown here, but because of the nature of the patching, it can be problematic for code reviewers as well. So now it's the first demo time. Let's go right into my repository and check out some example pull requests I have created. Each pull request has detailed description from overview and challenges to potential solutions. I won't be touching on every point in this talk as this is more for your reference later on. So let's look at this first one. It's about upgrading Grafana from 6.3.0 to 6.16.2. These versions are arbitrarily chosen, but let's have a look at the file changes. So we can see that the versions do match up. So we are updating from 6.3.0 to 6.16.2. But that's as much as we get. There is no clarity about what kind of resources are being created, removed, or updated from this poor question file changes. So for this, in order to actually see the resource changes, you need some toolings to help. And this particular example, I have set up this repository with slash run diff automation. So this automation uses this DYFF CLI tool to have the diff between two YAML file inputs. Those two YAML file inputs come from this Helm template commands being run behind the scenes. So Helm template is run against 6.3.0 Helm chart and also 6.16.2 Helm chart. So if you look at the result slash run diff comment like this, you can see the diff of 50 differences before corresponds to 6.3.0 and after corresponds to 6.16.2. And you can see 50 differences, how exactly each change would result in the cluster, the resource differences. And you get another one. This is for staging. And you can see how the differences are being made for each environments differently. You may have values differently. For that reason, you need two differences or multiple differences based on environments that you need to deploy the Helm charts against to. So the next example is updating some Boolean flags for Helm chart and this one uses the flux Helm chart. I have copied the fluxes Helm chart here in this repository. You shouldn't be doing this. This is just for demonstration purpose. If you are to really use fluxes Helm chart, you should be using directly or by dependency. If you look at the file changes, these are some changes of enabled flag to false to true, secret, config, so on and so forth. Again, this time you don't really see what actual resources are being created other than you are updating something related to Prometheus, something related to the secrets, et cetera. So if you run the run diff command once again, it does give me that it's only updating staging. So it was originally meant to be only for staging and we didn't see only one file change, so production is not updated. But for staging, we don't see any output. This is a limitation with DYFF. DYFF can only handle if the two files, two YAML files have the same number of documents. If you update some of the enabled flag, it is likely that some new resources are being created or some resources are being deleted. And for that reason, the document number don't match up with the two files. That means if you are to have some automation and tooling around Helm charts, this DYFF automation might be too simple to really help you code review more efficiently. Let's look at another example, this time with Customize. So this is about updating some version for one deployment. So if you look at the file change, you can see that Toolkit Alpine, this particular image is updated from 0.2.x to 0.2.y. Just one line change, one letter change, and this is very straightforward base update. So this is to update all the environments, production and staging, and if there is any other environments, it would probably update all of them. And if you run the run DYFF once again for this particular change slash run DYFF, it actually gives me that for production, there is no difference. For staging, we can see that the 0.2.x is now being updated to 0.2.y. So that's expected. Staging is working as expected, but production, there is some overlay or some patching happening, or maybe resources is not pulled in for production. So for Customize, it might be simple enough, but from the DYFF point of view, from the code review point of view, actually it doesn't tell me everything unless you have some automation or you go through all the code or the customization.yaml files. The next one with Customize is about adding some missing service account. So if I look at the file change, some service accounts are being added. Customization.yaml is listing another file service-account.yaml, which corresponds to this new file. It's adding this color service service account. But there is another one that's created, service-account-derm.yaml. Is this an okay file? Is this something that was missed in the customization.yaml or is this just for some later on usages? This is unclear. From the code review perspective, customization does give you this flexibility, but it does give you less clarity why that file is needed. If I run the slash run diff once again, this is another limitation that we hit with Helm. The number of resources change between before and after the change. And for that reason, diff cannot be shown. And for this kind of diff to be shown, you need to invest in some other automation or some tooling to help you do the diff. Maybe UnixDiff or Sdiff might be able to help, but this is something that you need to invest in. So we looked at code review with Helm. Helm is a package manager, so working with a third party tool is very nice. But as we saw in the simple examples just now, Helm template commands may be required for reviewers to understand actual impact. We didn't have an example of template files being updated, but I think it's evident how that can be also tricky to review. The examples with customized looked straightforward, but it also has some unfriendly behavior to reviewers. The code change didn't explain how one change affects another one environment, but not the other. In order to get to the bottom of the spec, one would need to check all the relevant customization.yaml files. If the diff cannot be shown, such as the case we saw at the end, it gets even more difficult. So we want better code review experience after all. It needs to be easy to review. Files should be declarative, just like simple Kubernetes YAML spec, but still provide some sort of code reuse. And the syntax must be simple. Also, tooling needs to be simple, easy to install, no complicated logic when used, and can be easily integrated with CI and any automation setup. Now it's the time to introduce importer. Importer is a CLI tool to import any lines from anywhere. It works by passing the provided file and look for special markers, which are mainly provided as language comment. These markers can define where to import the code from. As you can see on the right hand side here, the imported lines appear in the file. This is an example of generating a new file, and we will look at other examples in the demo later. Importer is a brand new project I have started. You can find some details here and also full details and background in importer repository. Importer is designed to be simple and in a way, dumb. It doesn't know anything about filed validity. It just looks for matching markers and import some data when necessary. This design was the key to ensure it can work with helm and customize. Importer is not to replace the existing tools but to augment them. When you find code review getting more and more difficult, importer may be an option worth looking into. So now it's another demo time. So now let's see importer in action. First of all, how do you install importer? It's really straightforward through install Upsider tab importer. If you want other installation option or the details are provided in the repository. Because I have importer installed already, let's see that in action. I have some files prepared here. Markdown demo.markdown. So this markdown has some data and this comment is the special importer marker syntax. And there's another one down here. Importer marker is always a pair. There is one for begin and one for end. And this one is named as a short description. And it tries to pull in the data from this markdown dash snippet.md with a symbol of full demo. Let's see what this file contains. So this is another markdown with a description, some other content, and you can see another comment export for demo begin, export for demo end. This is another marker importer provides importer uses. And this is an exporter marker. So this full demo is the symbol that matches up with the importer marker. And the data down here will be replacing the data up here within this importer marker pair. So let's see that in action. Importer update markdown demo. Now see the file. So you can see that the file is now updated. It imported the file and the content from markdownsnipet.md. Let's see another one. YAML, demo, YAML. So this time it has some data description and it has some data outside of it where we have this comment, another comment with the importer marker. Importer marker is a pair again. It's begin and it's got end. And this time I'm looking at YAML snippet YAML and metadata only symbol. There is another option for indentation making sure that it will be aligned to the two spaces here. If I look at YAML snippet. YAML, it has some other data name, description and within the test data stanza, you can see that exporter marker appear twice. One is for sample nested, which is begin here and end here. And there is another one that is nested metadata only exporter marker, which is ended here. And it has metadata name and namespace. Let's see how that gets updated. Importer marker update YAML, demo, YAML and see the file. You can see that now metadata name and namespace, this data is pulled in from YAML snippet and make sure that the indentation is correctly two spaces for this case. If you just validate by looking at snippet, you can see that the indentation was different from the original snippet file. Now let's look at a pull quest based on importer. So this one is updating Argo CD from version 2.1.0 to 2.1.2. So if you go directly into file changes, we can see an important marker at the very top. It says it's got the marker name of Argo install. It begins here and from file of github.com, Argo approach, Argo CD 2.1.0. This is an official installation spec Argo provides and I'm updating that to 2.1.2. Because importer update has taken place already for this file, you can see that 2.1.0 is now being updated to 2.1.2 for some of the image tags. And also what's interesting is that you can see additional M variable defined for 2.1.2. This is something that's very clear and because importer has updated all the files, this makes the file completely declarative. If we go back to description, we can see details and benefits, but those are covered in the slides, so let's get back to the slides. So with importer, you can clearly see the resource diff. It also means that any resource additions or deletions can be fully tracked by commit history. There is no hidden dependencies or external factors with importer and all the files become truly declarative for each file. Customize helps declarative setup but it doesn't guarantee to have each file to be fully declarative. There could be some patching on top of them. Helm uses heavy templating, so although Helm values files are declarative, Helm template files are not declarative in most cases. So let's quickly summarize the importer takeaways. First of all, importer does not aim to solve everything. Importer takes code generation approach and for that reason, the code can get stale easily. You can fix that by adding some automation, which you can find example in the importer repository. Importer also does not have any template, repeat, et cetera, things listed here that these may change in future, but it's kept as simple as it can be for now in version 0.1.0. There are plans to have IDE support, nested importer marker, handling and more in the future, but it's a brand new project after all. I'd love to hear your feedback and ideas. Before wrapping up, I need to mention about validation. Validation is beyond this talk because you can end up with valid but unintentional definitions. We looked at some examples, example requests with Helm and Customize and they were all valid from Kubernetes resource point of view. Even if you had some custom validation for your use case in the CI pipeline, some unintentional change can still creep in. From the code review perspective, valid does not mean that all the changes are good to go. As to the validation itself, there are great tools out there, such as OPA, code file, et cetera, which are definitely worth investing in for better pro requests and merge request management. Here's a couple more tools that could provide better code review experience. HCL, hash code configuration language and Q are both great tools, but as mentioned earlier in this talk, it is important to take code review experience into account before introducing new tool. Finally, here is the list of tools and items I use in this talk. Although I touched on DYFF diffs limitation, it's a great tool and definitely worth checking out if you haven't done so already. There are some other tools such as chat ops automation setup, which you can find references here. With that, thanks very much for listening. Feel free to reach me on Twitter if you have any questions or comments. Thank you.