 secure your GitOps and implement a robust security strategy. So first a little bit about me, I'm an engineer at Intuit on a platform team that's building secure multi-tenant Kubernetes infrastructure. I'm also a co-author of GitOps and Kubernetes. And actually this talk covers much of chapter six of that book. If you're interested, please check out the book. So Intuit's using Kubernetes at a pretty large scale. We have 350 plus clusters and in fact if you filed your taxes last year with TurboTax, you filed your taxes with Kubernetes. And for all those clusters we're using GitOps kind of as a standard mechanism to deploy apps to Kubernetes. So first I want to kind of take a high level view, look at the GitOps operator security pattern. What this diagram is showing is who are the subjects who are making changes and what are the things that need to be protected and what are some of the points in between that need to be secured. And the rest of the talk is really gonna be drilling down into each of these, looking at specific things that you can do. But first let's go through a typical process where you have an engineer pushing code changes to the code repo. You would then have a CI pipeline that's watching that code repo and would check out the code, run the unit tests, run other kinds of scans depending on the CI pipeline that's defined. And then ultimately it's gonna push a new image to your image registry. Then you're gonna update your Kubernetes deployment repo, this is your GitOps repo. And you're gonna update that either with the new image or maybe some other infrastructure changes to your application, some other Kubernetes changes that you want to be pushed out. And then the operator is watching that deployment repo and any merges to master will be applied to your Kubernetes cluster. But before that happens, you have the opportunity to have code reviews on those changes, approvals, and that's one of the key kind of inherent security benefits of GitOps is to allow you to have a second pair of IDES, that audit trail that people have talked about this afternoon as part of your normal process. And so once that is approved and merged to master, then the GitOps operator can then deploy that out to your Kubernetes cluster. So given all that, we have some inherent benefits for security with GitOps, but what are some specific things that we need to look at to secure it even more? So what do you need to secure? Really the answer is pretty much everything. So there's a long list. So on the left hand side are all the different components that really need to be looked at closely. The last two application and the cloud provider, those are a little bit outside the scope of GitOps per se. So we're not gonna be covering it, but the other six we'll be covering in this talk. So here's a typical CI CD pipeline. And really the CI CD pipeline is defining your policies and your standards for deploying to production. This is how you enforce your engineering process. And so as part of that, you really want to protect it. You don't want that pipeline to be bypassed. You don't want it to be compromised. It's really codifying what you think requires quality software to be deployed to production. So the first step is securing that pipeline. Obviously you wanna restrict who has access to that pipeline, who can make changes. And really that's because that pipeline is driving all of the new software to production to your different environments. And if that pipeline is compromised, then it can deliver harmful software, whether buggy software or otherwise. For your CI pipeline in particular, you don't wanna use long-lived sort of machine user credentials in the CI pipeline for either the Kubernetes clusters or for the cloud providers. You wanna use short-lived credentials. Next is the container registry. And most enterprises will use a private image registry to store their container images, not only for security, but also for reliability performance and privacy. If you're writing proprietary code, you don't really wanna be pushing that to Docker Hub or something like that. So at least I didn't do it. We have our own private image registry. And once you start managing your own private image, excuse me, private image registry, you can have some more configuration and control over that registry. And one thing is you'd want to make it so that image tags are immutable. You don't want to change the meaning of an image tag once it's set. If you've deployed an image to production, you don't want that image tag to be changed out from underneath that running system. You also wanna use tools like Twistlock, Prisma Cloud to scan those images for vulnerabilities. And we'll come back a little bit in a couple of slides to talk more about the container registry. Next is securing the Git repository itself. So the Git commit log and the revision history of your GitOps deployment repository is the audit log. It tells you everything that's going on, everything that's being deployed to your Kubernetes cluster. And so it tells you what changes were made, when they were made by whom, and hopefully if you have a descriptive commit message, it'll tell you why, right? And so you really want to use pull requests and code reviews for all the changes to the deployment repo so that you have a good audibility, the ability to roll back and you know exactly what's going into your cluster. It's also desirable to use multiple deployment repos for each team or application. This allows you to have granular security access controls on those repositories. Different teams can be restricted from accessing certain GitOps repositories. Or if you're using a single repository, you can take advantage of the code owners file that's featured in GitOps to, or a GitHub that allows you to control which teams can access and approve merges for certain branches of code. Also you want to closely control master branch, merge and write access to the GitOps deployment repos. Because really in effect, whenever you merge something to those repositories that those changes are being propagated to your cluster. So it's the same as giving those developers cluster access. So you want to closely control who can approve merges and affect those changes. Next for the Git repositories, you need to enable branch protection rules for the master branch. And the three that I list here require pull requests for before merging. That is basically means that you need code reviews of those PRs before they're merged to master. Now keep in mind, these are not, this is your GitOps deployment repo. So it's not traditional programmatic code, but it is your desired state of your cluster. You want to have a second pair of eyes on all those changes. So you definitely want to enable request reviews before merging. Also you want to require review from code owners like mentioned previously, if you are using the code owner's file to define who has merge authority. And then you also want to enforce automated checks. This is where you can run unit tests, effectively unit tests on your configuration before it's even deployed to the cluster. And at a minimum you can do some kind of linting or some kind of syntax change checks of your configuration but then you can even go deeper and do some static code analysis to really implement deep policy enforcement. And then again, this is even before being deployed to the cluster itself. Next is securing the Kubernetes cluster. Ideally you would require that only images that have successfully passed through your CI CD pipeline be allowed to be deployed into your cluster. You really don't want developers to be able to side load images, right? You want to ensure that it's gone through your process and gone through all the steps in your pipeline. You also don't want to allow images with known vulnerabilities to be deployed, which is hopefully part of that pipeline is to identify those. So this is difficult. This is not necessarily easy but you need to do some type of signature of those images when it's going through the CI pipeline. And you also need to use some type of admission web book like OPA to require that all images being deployed into the cluster have that kind of signature and validate that signature. Next, don't allow the latest image tag, that's just a bad idea. And you want to verify compliance with the CIS Kubernetes benchmark. This is a whole topic, a whole talk in itself. There's probably additional talks over the next few days on this. But the CIS Kubernetes benchmark is a good place to start to make sure that your cluster itself is secure. Then as we talked about earlier, you really want to have your own private image registry. But as part of your Kubernetes cluster, you want to block image pulls from any untrusted registries. In other words, you don't want to allow pulls from Docker Hub or any other public internet registries. And how do you do that? Well, add into it, we use OPA. And this is a little code snippet which is really one of the first tutorials for OPA on how to block untrusted registries. It's a very common pattern. If you're not already doing this, strongly recommend checking out OPA. We use it not only for this, but for a lot of other policy enforcement. Next is you need to secure your secrets. As was just mentioned in the previous talk, you can't just commit sensitive information to get. That's a bad idea. Kubernetes secrets are not encrypted. You need to think of some other solution for doing that. You also don't want to bake sensitive information into the container images themselves. You might think, okay, maybe that's a different way we could do it, but that's also problematic. So it really comes down to three different ways you could do this. One is you could use an out of band process. In other words, not GitOps for deploying the secrets to a cluster. So you would maybe use GitOps for all your other deployments and everything else, but then secrets may need to be manually provisioned. But that's kind of a pain and doesn't really follow the whole benefits of GitOps that we're trying to achieve in the first place. So the next one is perhaps you could store encrypted secrets in Git. So if you encrypt the secret and then commit it to Git, then that's okay. But then how do you get the key to unencrypt it? So there's always some secret that you need at some point. But luckily there's some patterns that allow you to do that. So sealed secrets is one, there's other approaches that allow you to do that as well. The last is to use some type of external secret management system. This is where the application containers retrieve the secrets at runtime. And example of this is Vault and also add into it, we have our own proprietary system that does this. Then next, obviously you need to secure your GitOps operator itself and how you do this might depend a little bit on which operator you're using, whether it's Argo CD or Flux. But there's some common themes. At the end of the day, you want to use the principle of least privilege so that the GitOps operator only has the permissions that it needs. If you're not deploying a certain type of object through GitOps, don't let the GitOps operator have privileges to write that type of object. You definitely want to limit the GitOps operator cluster role write privileges. So anything that's a cluster wide access, you want to be very careful about it, especially if you have sort of end user or developers accessing that your GitOps and managing their own GitOps pipelines. You want to only give the GitOps operator the privileges that the dev teams themselves would have to the cluster. So don't give GitOps operator cluster admin privileges. Then you want to also enforce team specific security checks in the GitOps operator. And this is where the multi-tenant aspect comes in. If you have multiple teams, you don't want one team to deploy to another team's name space. You don't want other teams to be able to deploy another team's software. You want to have them strictly controlled to only the teams that are allowed to do those operations. You also want to maintain the audit log of the GitOps operator. Because again, this is telling you everything that's happening in your cluster, everything that's being deployed. And if you need to go back for forensics and see what happened, you can understand what the flow was that caused a certain operation to happen. And lastly, you need to have a disaster recovery plan for your GitOps operator or your GitOps infrastructure because likely in the event of a disaster or some other type of incident, other teams are going to need to use that infrastructure to restore their services. So really the GitOps infrastructure is a critical foundational piece. I quickly want to talk about the GitOps multi-tenant management. So add into it, we have our cluster platform team that provides GitOps infrastructure to each tenant. And then each tenant has one or more namespaces that they're allowed to deploy into. And the GitOps is configured to only allow those tenants to deploy to their namespaces. And we've specifically prohibited cluster-scoped resources such as demonsets and cluster roles and other things like that. We also use limit range, resource quota, OPPA and other mechanisms to enforce policy. And again, you don't want those to be manipulated incorrectly and you want to limit that access in the GitOps infrastructure. Now at the same time, the platform team itself would like to use GitOps to manage the clusters. So we have a separate set of GitOps infrastructure that's used by the platform team that has elevated privileges that lets you do those types of things. Deploy cluster-scoped add-ons, manage the tenant namespaces and do the policy provisioning. That's all I have for this afternoon. Are there any questions?