 DevOps is big by design. When we talk about DevOps, it's really critically important that we talk about context because the term is so big and it covers so much ground. Is DevOps a cultural movement? Is it a mindset? Is it a collection of process and tools? These are the things we really need to scope down when we're having a discussion so we can all be on the same page. Because again, DevOps is big and it's open to interpretation. GitOps on the other hand is very much more narrowly scoped. It is about taking DevOps best practices and applying them to infrastructure automation. But that can still get pretty confusing at times. Now to shed some light on the subject, one of our sponsors, Adaptivist, will be coming to the stage to talk about a very specific definition of GitOps and continuous delivery on Kubernetes. And when you have a moment, please check out Adaptivist demo on the demo stage as well. Now let's watch. Hello, hello, hello, and welcome to GitLab Commits Virtual Session on GitOps. What's in a name? My name is Vlasmus Plaston. I am a Principal DevOps Consultant at Adaptivist. I am at Servitor pretty much all over the web. Today's session is going to be on GitOps, which is clearly the new hotness based on all the sessions and tools things coming out of it. But what is it becoming? What are different things called GitOps? Are they really and so on and so forth? I am going to cover a few flavors of GitOps out there, which can be sort of confusing to get a hold of. So I am covering something I think of as true GitOps, which is the sort of original things that we works came out with with Flux and it covers auto CD and so on. These generally serve as extensions to DevOps, as if it's a small additional piece of small narrowly focused process that you follow to do a specific thing. It relies on continuous delivery, Kubernetes uses an operator and is a pull effect. GitLab also has a series of things called GitOps, which we will detail a little later. In their case, it's really more of an evolution of DevOps. It's sort of everything is becoming GitOps because there's a bigger focus on using merge requests and Git and putting everything in there, especially for your inference. So in their case, it covers CI CD. It uses some agents for Kubernetes in some cases, and it kind of mixes both push and pull when it comes to how changes flow and what sort of connection. I have a third one that's a little bit more goofy, which is Hubo GitOps, and I will get into that name a little later. But in short, it is any sort of approximation of GitOps, mainly the true form of GitOps, the one that's the closest to the real intended original thing, which means that there's some sort of operator-ish thing. It's not necessarily one of these nice things like you would get with Flux or Agro, but the idea is that you need to be following some sort of operator pattern, which involves pulling your changes from Git into a cluster of some sort of restrictive environments. Finally, there's also a category I think of as might as well be GitOps. These are things that are not explicitly listed as GitOps, but they might as well be because they're really following the spirit and pretty much could be branded at, but for a reason they're not. One of these options is something that Google kind of generically calls application delivery, which is a push-only type thing if there's a CLI that pushes changes into an environment, but since it's a CLI that can serve as an operator, you could put it into an automation system of some sort and really use it as a push instead. So let's look at these in a little bit more detail. True GitOps is what I consider the original that was coined in 2017 by WeWorks as operations by pull requests. The focus is to extend the stuff that you're already used to through DevOps, which includes CI and particularly the supporting culture of getting developers and operations professionals closer together and get them to just work better together with tools and APIs and process as opposed to the good old just throw things over the wall. Specifically, True GitOps is an opinionated approach for how to do CD, which is something that lives inside a Kubernetes cluster and pulls changes into it as they emerge into Git. It does not accept pushing things into a cluster because that means you have more access than you need in that cluster. You really kind of rather want to have a declarative Git repo that you just go update and then when it's updated, well, something in the cluster pulls changes into it. So tools like WeFlux or AutoCD provide this with a lightweight operator of some sort inside the cluster for security so that after it's set up, you don't need to have anything outside the cluster actually able to reach into it. There are some secondary topics then. So for instance, AutoCD has a nice operation QI, which allows you to do a few extra things. And there is also that piece about how you actually set up the cluster in the first place, which does require having access to the cluster. It requires some extra bits of pieces. But in this context, it is not part of GitOps. It is part of a setup that enables it. So to summarize, true GitOps definitely uses pull with an operator inside the cluster. It is Kubernetes and that's it. You can approximate other things with other options, but this one uses Kubernetes and it only does CD. There are some discussion about whether you can do some operations as well. So there are some newer options floating around where you also have some level of continuous operation floating around in part of this. But that's true GitOps and there's a little link down here to the original description of it. And WeWorks has a ton of articles on GitOps and such. So on GitLab, there's a bunch of different kinds of GitOps flavored things. So kind of in the wild world right now, GitOps is really being introduced as a possible at all over the place, except on anything, every product you could imagine with a rocket actually related to true GitOps. I have actually been in a session on GitOps that I thought was going to be technical, but got into talking about human psychology as related to GitOps. Okay, sure, right, all right, cool. So GitLab tries to take a balanced approach. While true GitOps takes a relatively small thing and just attaches it to GitOps, GitLab really infuses the spread of GitOps all over the place in GitOps. So as noted earlier, it's really more of an evolution. So things that are GitOps are now GitOps. Defined by GitLab, GitOps really just is infrastructure scope plus merge requests, along with CICD. Those three pieces, you have GitOps. Okay, you can sort of imagine these things infused into those possible. Particularly, this is a natural fit because GitLab tries to do all the things with the pipeline front and center to do all the things and run them. So generally, the GitLab pipeline is the master of all the things, has access to do all the things and triggers and activates all the things. You can run other things as part of a GitLab setup, such as simply setting up AgroCD inside a cluster and work them separately. But that's your choice and part of that was still GitOps to a degree with GitLab. But at that point, you're doing two different things that are just kind of bolted together, which may or may not work depending on your setup. More detail on this later, but in short, it is sort of a mix of push and pull. In some cases, you have a specialized agent for Kubernetes. In other cases, you're doing pretty traditional CI CD as part of standard DevOps that will use Terraform and things to update the targets. But while having absolutely everything stored in Git, managed with a merge request process and so on and so forth. It generally covers both CI and CD, since it's the central pipeline doing everything. And oftentimes it's Kubernetes, but it can be applied elsewhere in sort of following the same spirit. And there are a whole bunch of articles as well on GitLab, on the GitOps topics for various topics, lots of nice presentations and things too, if you're curious about more detail. Hubo GitOps is sort of a funny thing. There is, as I said, a lot of weirdness out there. These days being called GitOps. And I came up with a simple test of simply replace GitOps in a sentence with DevOps or ISE, did it change? If not, you're probably not looking at GitOps or at least not something that is trying to embody the original spirit of true GitOps. However, if you kind of still do want to do GitOps, but you're not quite able to, you know, adopt all the things and you're not necessarily in a working environment that supports it, you can still approximate it. So there's a way to do all kinds of tools and just set it up to follow the patterns that approximate true GitOps. And you can do plenty of benefit like that and it's a good way of taking one step towards formally adopting GitOps more clearly. So I have done this and I use Jenkins and bestowed the term Hubo GitOps on it. It was really just a set of Jenkins controllers working in different tiers that were isolated enough to where that Jenkins could kind of function as a pull effect in one simple CD dot. So ultimately in this case, the main objective is always going to be auto triggered CD from pull across the merge requests with some sort of makeshift operator ideally isolated within the environment that is having CD applied to it pulling changes in as they get merged with Git. So it's still pull, but it can kind of be a makeshift operator that doesn't always look as clean as you might like. It is meant to be CD only, but it can be co-located with CIO operations. In the whole GitOps I've done with Jenkins Masters for a particular client, it ends up using dozens of Jenkins Masters, which might be a lot depending on your company setup. So you can shrink that down quite substantially and you really just have CI jobs in your Jenkins, you can have CD jobs in Jenkins, you can have operations, utility jobs in Jenkins, so on, and you can still say, well, the CD part still is using GitOps in spirit, even if it isn't quite as isolated as it could be. So it can work. It can work in Kubernetes, or really whatever you manage to hook up to some form or another and call it GitOps with the right spirit behind it. Finally, as I said, there is something I consider might as well be GitOps, because, you know, while there is a lot of modeling buzzwords out there stacking GitOps on everything, but there are also times where the opposite happens, where something really kind of is GitOps, but doesn't call itself that, it doesn't feel like it needs to, but it just kind of happens to be close enough to where you might as well think of it as GitOps. I found Google's allocation delivery to be a really cool example of this. It is an add-on for Google Kubernetes Engine, or GKE, and it does actually work with GitLab, which is cool. It does a lot of these opinionated config file layouts in Git. That's where it can work in GitLab, Git Hub, and their CLI is able to automatically create pull-agress and things like that in the Git provider, which is an aware of how it integrates with GitLab or Git Hub. Finally, it adds a few extra UI bits on GKE, which is typically have a lot of syntactical sugar and spiffy UIs on Google Cloud. What it is, is again, essentially IAC with Git, not called GitOps, and the CLI operator is just a new push base. You actually go in there and type, you know, da-da-da, apply my changes, but really you could have that step installed into an agent and automation tool like Jenkins, and you just have the automation tool do da-da-da, apply the changes when it sees that Git has changed. So you can hobo it into a pull approach that's even more GitOps-y than plenty of things that are fully advertised as being GitOps. So, in this case, it does start as push, but it can be impressed to pull. It is CD with some enriched UI work, but it only works in communities on Google Cloud because it's a GKE network. And there's a bunch more details in here for this new phone, Google's Cloud site. So, with all that mind, do you want to adopt GitOps? I would say, not explicitly. You want to look for something that really kind of follows the spirit, the patterns of GitOps, not something just because the name doesn't match the background. So, look at the tools, look at what they actually do, not what they call themselves, which is probably a good general advice for most of anything. I have a few more technical details for the remainder of the presentation, so let's look a little bit deeper at some of these examples. So, Agocd is kind of probably the best example of true GitOps, because it's fairly straightforward. It's well understood. There are plenty of good guides. It has a narrow purpose. Agocd does GitOps in communities. That's it. It has a nice little UI where you can enter in your new apps and see what kind of status they're in, refresh things, delete things, organize things. This UI runs out of the operator that you put into your cluster, so you can go in and configure it, but you're still actually having Agoc pull in the changes whenever you change stuff in Git. This is a straightforward setup if you can make a good fit out of the opinionation that Agocd requires, which again is definitely Kubernetes. It's individual apps. Insofar that you make one app and it points at one video pro, and that's it. It takes care of itself. If you're doing tons and tons of independent things and so on and so forth, you might need something a little bit more. There is really cool things going on in Weaverx's GitOps setup where they have some scaling things that kind of give you another layer to this that they call profiles. There are usually going to be put materials and demonstrations for that floating around if you're curious about it. And that's like speaking of Weaverx. And there was some talk about Weaverx flux in Agocd actually verging at some point, but I recently found out that that segment was an independent effort. So we're still going to have two of them that kind of sort have been doing most of the same thing. But you might find that some things in one of the other fits with your environment a little bit better. Finally to be clear, it has both this UI as well as the CLI where you can do things like I want to register a notification. This is the Git repo it uses. Here are some extra things to think about. You can do that either in the app or in the UI. It is really easy to get running. There are really good tutorials. You can play with it and it really helps visually highlight what is the core of GitOps, what's the point, what is it that you're trying to do with it and how it's different from the things. With GitOps, it's really more of a wide purpose and range of things that it helps you do. It's really kind of taken a lot of traditional things and then it's kind of just put a layer of GitOps on top of it all to just keep up with the latest, greatest process and patterns and so on and so forth. So in this case there are three or four approaches depending on the timeline. I think this final one was in development when some of the material I was looking at had been written. I'm not quite sure if that's out yet or not. But the first three out for sure which is essentially the standard old-fashioned push in the CBCD pipeline which just as long as it uses Git as your declarative source of interest or precipice and so on counts as GitOps. There is one specifically pointed out that uses Terraform in a particular way that's what this diagram here is which really gets into making sure that Terraform is setting up all your things correctly whenever your pipeline runs. Both of these are push. Both of these take full access because GitLab wants to have the ability to fully control the time clusters and do what needs to do to it. That has traditionally been a bid much for some companies to swallow just because that's a lot of power to leave in a tool and we know everything can be compromised in days and if it is well then you have access to everything that you do. One of the kind of fundamental tenets of GitOps was to flip that around to where if you compromise something you can't go and completely mess with and backdoor into the production environments. Although of course if you did really compromise and people start merging things then you can still introduce stuff but you can't completely overhaul the process and start just quietly supple and putting things in. You still have to go through the front door you just can get to the front door if you compromise a system like that. But there's still a door, it's not like free for all. The next two that are specific to Kubernetes uses a new GitLab Kubernetes agent setup which is another thing that is detailed on their website in the link down here. Specifically this does kind of a two phase approach in that there's both a Meteor agent that's sitting on GitLab that's doing most of the work and then there's a smaller agent that sits inside the cluster that does kind of the poll type thing but the GitLab agent is the original thing that sits all these things up and handles some sort of communication and so forth. So it's a little bit more complicated and involved than simply having one of the true GitLab's operators floating around in your cluster. And as I said this sort of supports two different setups. One is the standard kind of poll because you have an agent sitting in the cluster that looks at your Git posting things when you want it. And there is one that is meant to use push but with the advantage that it's using this established setup where you don't quite need as much access or permissions assigned to GitLab for it to be able to use it. So this is a more comforting option than old fashioned straight up CD with full lab and access to all the things. Between these four methods you can type in both Kubernetes and other things. So I said like Terraform we just pointed that role AWS it doesn't have to be using Kubernetes it still is AAC plus merge requests and do your things. And again the first two has full access and the second two are more limited in what they need. All this stuff really works well with the overall enterprise strategy that GitLab promotes because you can consider all this part of the whole SDLC and just add GitOps where it's possible. So that's the deeper dive on GitLab GitOps but for the really deep stuff they have tons of video out there for you to look at and play with in examples and so on and so forth. This still motion does it. Sanity check on what different kinds of GitOps actually mean. For a little bit more on the hobo GitOps which is really anything you can make work that to some point approximates GitOps much as you can within whatever constraint you're working with. That means it can be very arbitrary. This setup with different tiers in a Jenkins master curve tier adds up a lot if you have dozens of teams and each team has you know three CD controllers as well as the CI Jenkins and maybe a lab or some sort of with dozens of even hundreds of Jenkins masters and sometimes that works if that's just the size you're working at and it works for you awesome great if it works do it. This leaves with a lot of flexibility but it's a lot less guided and opinionated so it will take some discipline in actually coming up with a setup and not being tempted to say things like well yeah I prepared this pull request from the QA branch of repo inter production but while I did get it approved I need to make this one other change and I have access to this thing so I'm just going to go like squeeze this change in just real quick because we aren't really used to GitOps so we can just improvise a little bit like resist the temptation of being a good hobo GitOps and do it the right way I said this example was using tiered accounts in the OBS that is each team has a production account which is heavily limited for access a QA account which is a little less limited but you still need to use the regular process in there you can probably see some more things and then a dev account which is where you have more full access as developers and you probably will take over things and make sure that whatever you set up for your Jenkins is actually working because in this setup you get these Jenkins managers as operators and it reads states from Git but it also requires you to actually code in the deployment job logic in for instance job DSL so that when you have an application has a CD to all the Jenkins which knows how to deploy the application into something on AWS so finally, as noted earlier while this ends up probably causing Jenkins sprawl on yesterday's really good at automating Jenkins or use really lightweight throwable Jenkinsers that are just completely like hit the button magic happens and it's up and running you can do less and say you could combine two of these or all of them will include CI and sort of just have multiple things inside the controller as you see fit if you have small teams and their stuff isn't very sensitive maybe they don't need a whole bunch of different accounts and Jenkins managers and so on maybe they can do more with less so it's really up to you that's the hobo part just it works finally, here is application delivery on GKE this is going to I mean you can sort of kind of imagine being a little bit like Igo at least on this one applications tab which normally if you're used to GKE but you haven't used application delivery and all the things this thing tends to be empty because it's really a place for add-on some extra things and with application delivery in place you start seeing it being populated so if you have individual applications in here which has your usual level of information and components and particularly to this setup is a very strong environment identity so you're really getting into using your app-configure repo to define these different environments and carry things between environments so that when you're in the UI you can go look and see my staging environment is in here and prod is in here and the good tag of this one here which is in staging is this thing so it's very nice kind of overview of the whole environment between A to B and C and so on they have a really nice CLI it seems like everybody in the law has known CLI these days but what I like about it is that it goes in and makes the updates for you and stages them in Git which starts helping a little bit when you're trying to scale if you're doing tons and tons of micro-services and you need to make all of them work within Git if you can just type in and upgrade this dependency for all these things and magic happens and you have 20 ps and then you can say yep, that roughly looks right merge all of them, magic happens cool, you're still using Git it's just kind of automated a meta layer of just getting crazier with how nice it is but ultimately at the foundation level it's still GitOps even though they never call it GitOps anywhere it just uses the same principles and it's pretty cool as noted it has a really nice new integration on here on GKE and it has that really extra focus in modifying environments and promoting code in these environments step by step by step so, it might as well be GitOps and that was really it thank you very much for coming to this talk I hope it has helped a little bit in making hits and tails of what is GitOps what isn't and what might as well be and hopefully by next year we will be able to meet in person and hand up good bags GitOps enabled toasters because you don't know it could happen alright, thanks so much we would do some Q&A or something like that if we are around and if not, if you watch this recording well, cool take care