 So, hello everybody, and thank you for joining to our talk, build your own GitHub operator for Kubernetes and beyond. My name is Boris Trivok-Cholo. I am Senior Software Engineer at NIMACOM. And my name is Cecile Scheidt. I'm Cloud Automation Engineer at NIMACOM. Our focus is to help our customers and their software development teams to achieve operational experience. So, we consult our clients from the beginning where requirements are defined during the whole process of the development and also in the operational phase of the software. And we are a Swiss-based company here with a branch here in Germany and we consult a lot of customers. So, we got bigger customers such as big car manufacturers and we got smaller customers. Some do not have Kubernetes. Some do not have huge clusters. Some don't even know what Kubernetes is about. So, but we are really convinced of the numerous of the various benefits of GitOps and we try to implement it wherever it's possible and appropriate. And in this talk, we want to present two cases where we've precisely done that. So, we implemented GitOps outside of the established paths of Flux and Argos CD. But first of all, I want to just clarify one thing. Operator and agent are interchangeably in this talk. So, what I mean is that if we are using the operator term, we don't mean the Kubernetes operator according to the Kubernetes, according to the operator framework. What we mean in this case is the piece of software or the component running the GitOps operator control loop. And the contents of this talk is about, first of all, the evaluation of the four principles. So, what are the four principles about? What gives, what freedoms it gives to us? And what are the moves of the four principles? Then we show to you guys the first story with Ansible agent and Docker swarm. Then we share our story about a help file used as a GitOps agent. And in the end, we show you our lessons learned. Josia, the stage is yours. Thanks. So, let's look at the four principles. Actually, we're not going to look at them in detail. We won't repeat them here verbatim. You're probably going to hear them a lot throughout the day. But let's focus on one thing. Nothing here mentions Kubernetes. No big surprise. That's why you're here in this talk. But also nothing here explicitly mentions containers either. So, whenever you have a declarative format for expressing it, then you could even be running wasm or jar files with GitOps, as long as you can express it declaratively. And one other thing is also that the principles never require you to run the GitOps operator in a target system, for instance, in your Kubernetes cluster. Instead, you could, for instance, have a GitOps operator running as part of your scheduled CI pipeline, unless you're doing CI ops, of course. Now, at the end of the day, there are very good reasons why we have grown used to expect these points from our everyday operators because that has a lot of benefits. But it's important to understand that these points do not automatically follow from the principles themselves. The face of GitOps may look remarkably different depending on the context in which you are implementing it. Which is said, as long as you have a declarative format for describing it, you can do GitOps with it. So what file formats are there currently? We did some research, and so far, we only found the following formats, and we called it the P1 market because of principle 1. Some of these formats are mostly good for managing virtual machines. Some of them can orchestrate containers as well. And for some of them, some tools even offer GitOps functionality outside of the Kubernetes world. See the rightmost column. And this list is probably not really exhaustive or focused certainly on declarative formats for managing VMs or containers. But for any specific use case you can come up with, you might be able to find other declarative formats for that use case. But let's boil it down. Let's imagine we are in a situation where we can use at least one of these declarative formats. Now, what features would we expect from an operator supporting our desired file format? This list is certainly opinionated. It's what we think is important. Of course, we want to pull the Git repo, and we want to continuously reconcile these contents. We want to deploy these declarative contents from the repo. That's the bread and butter of GitOps operator, principle 3 and 4. We also want Git-based pruning. That is, if I delete some piece of content or even an entire file in my repo and I commit that deletion, I want that object that resorts to also be deleted in my target system. Actually, this is implied by principle 4, but few tools support this out of the box that we listed as a separate kind of feature. Secrets management is also very important, because we don't want plain text secrets in our Git repos. And if things go wrong, during the reconciliation process, we can alert on that. So, looking at these five, from our perspective, most wanted features, what does the market offer? Again, we did some research on the P3 P4 market for principle 3 and 4 regarding GitOps operators for not Kubernetes or not only Kubernetes. We say not only because we also have pipe CD in this list. It's an interesting edge case, because it has a control plane and agents. The control plane has to run in the Kubernetes cluster, but the agents can run anywhere, basically, when on bare metal. So, we decided to keep it in this list. What we see right away is that only very few tools really support a big set of features here. So, that's a pretty clear sign that at least in the current state of our industry, some custom development will always be necessary to get good GitOps in non-Kubernetes environments. And we will now turn to a story of how we developed a custom GitOps operator for a Docker swarm scenario. Out of the tools in this list, we could have used Portainer. And we did use it at first, but it lacked a few very relevant GitOps features. So, we decided to implement our own operator. Let's turn to the story. What was the background of that? We were in a digital transformation phase, together with a customer, helping them with app development and also local no-code consulting. And as part of these efforts, we introduced the ICD and even containers for the very first time for the customer. For certain reasons, the customer decided to use Docker swarm for container orchestration, and we did not only do app development for them, but also platform engineering to help them with an internal developer platform running also on Docker swarm. We had multiple tenants, and we introduced secret management with SOPs in age here. Now, given these circumstances, what kind of GitOps setup did we in the end implement? In short, it's just an Ansible playbook that runs every 15 minutes from within GitLab CI. And I can also already imagine some of you screaming at the screen, what a scam! This is CIOps, not GitOps. But remember from the principle slide, these principles don't prescribe where the operator should be running, and only says what it should be doing. And this is definitely not CIOps. Our playbook is never triggered by single commits, for instance, from the build pipeline, and all it ever does is run the config ripples and running a rollout. Now, before we dive into the actual Ansible code, you might ask why did we even choose Ansible? And the technology decision was basically based mostly on this. We had experience with Ansible Python with Yamla inside our team, and we had SSH access to the hosts. And Ansible just excelled at SSH. Plus, we have a huge ecosystem of mostly item-portive tasks at hand that we can just plug together. We did find one existing project, but it was not complete enough for our use cases, didn't only support a Docker compose and a single host, plus we didn't want to maintain actual Python code. So, we went module shopping in Ansible Galaxy and built our operator. These are, again, our five most wanted features to define how we implemented our operator. For principle three, there was a built-in module for cloning git ripples. For principle four, there was a community module for deploying Docker stacks. And pruning got interesting. There are two important distinctions here. Docker services are something like Kubernetes deployments. You run your workloads, and Docker stacks are something like Kubernetes namespaces or customizations, if you will. And for Docker services, it was easy. We just used a single flag. During the Docker stack deploy, roll out for stacks, it was more complicated. We would have needed to implement something like git-based pruning there ourselves. And we decided to save on that and just offer manual partner attendance to delete their stacks when they deleted files. The secrets management, we were able to use a ginger filter to interpolate secrets into the final manifest. And for alerting, we used the built-in capabilities in GitLab, which is sending an email on a failing pipeline. Now let's look at the playbook itself. This is like an extreme abbreviation of the playbook, but it really captures the essence of the actual control that is going on. We cloned the ripples for principle three. We searched for the Docker stack manifests. We decrypt secrets and interpolate them into the manifest and write these final manifests out to the host, also to have a bit of a debugging capability without rendering secrets in CI logs. And interestingly, this rendering first, then rolling out, is very similar to how Oracle CD works behind the scenes. And yeah, after creating the manifest files, we deploy them, and that also includes pruning of services. Now, what were the outcomes from this? On the positive side, although we do not have exact measurements, we still think we get these. Better stability. We can more confidently rely on our Docker swole manifests and just roll it out on a Git basis. Also, our developers, tenants, can continuously deploy to dev environments and can deliver faster. As a platform team, we can now also more easily deploy new applications as part of our platform offering. But, of course, we had to teach the developers a lot about the asynchronous nature of GitOps and how they can debug or manually trigger the reconciliation process itself. You can find the anonymized version of the playbook on GitLab. Please note that we will not keep maintaining it. It's just a one-time copy paste with anonymization from what worked on this project. But there is another project, which we probably will keep maintaining. And that's on our second story. It's a GitOps operator for Kubernetes. And I'm going to take your part. Thank you, Josiah. First of all, I want to share you guys the project context. So we came to the customer and we talked with the guys. And we found out that they are using Kubernetes cluster and help charts to deploying their applications to the Kubernetes cluster. And we were pretty happy about that. But we found out that we are only on a tenant road. So we were not able to install Flux or AlgoCD into the cluster because we got not the privileges to do that. But Josiah and me, we wanted to bring GitOps principles into the project. And we found a solution with Helmfile. And Helmfile is a declarative format for managing Helm releases. On the right side, you can see a sample configuration. So here, a Helm release is defined. And furthermore, you can add additional tags or configurations to it and store them in a Git repository. But the main challenge was for us to implement the operator, which occurs the reconciliation on the cluster. And we made that really simple. So we created just a simple cron job here. And this cron job only has a crontab. And this crontab will be executed every five minutes. And it only clones the Git repository. And then it starts the container with the Helmfile in it and applies the changes from the Git repository. If we compare this with our requirements, we fulfill principle 3. It pulled automatically with our init container. Then principle 4 continuously reconciled. We fulfill that as well with that Helmfile apply. For pruning, this occurs in two phases. So if you want to delete your Helm release, you just have to set the configuration installed to false in your Helmfile and you just have to commit that or push that into your Git repository. Then you have to wait till the reconciliation get executed. And then you can delete the Helm release from your Git repository. If you want to delete the complete Helmfile, you just have to do it manually. For us, it was not the priority and we didn't implement a finalizer logic for that. For secret management, you can use your known Helm plugins for that, such as SOPS and for alerting we use Prometheus for alerting when the cronchup failed. In the end we implemented a pretty decent GitOps operator for Kubernetes with very minimal efforts and even without any cluster admin privileges. So, we brought GitOps principles into our custom project even without Flux or August CD or even without complicated operator frameworks. If you are in the same situation we released our Helmfile GitOps agent as Helm chart. You can use that, you can contribute to that feel free. And we want to share you guys also some lessons learned and all these success stories were only possible because we had a declarative format. Without such a format we cannot do GitOps. So, GitOps was always possible or is always possible if you have a declarative format. So, if you are using whatever you want, whatever format you want, there is always a GitOps operator for that or you can just build up your own custom GitOps operator. So, that's a fantastic news for the future spread of GitOps. Then for writing and maintaining your own GitOps operator you have to invest some time, so that comes not for free. But we think the benefits of GitOps outweigh the cost by far. So, you of course, you need some time, you just have to invest some time on it, but you get really the benefits of it. And lastly Kubernetes it's really easy, so Kubernetes is really the breeding ground for that operator for the for the operator framework. So, you can just do it really simple or really complicated and Kubernetes is really a good substrate for that. And this makes perfect sense given how Kubernetes is designed as one giant control loop. So, thank you guys for listening to our stories and I hope this inspired you to implement GitOps principles even in your projects where you cannot use where you cannot use Kubernetes or limited by using Kubernetes. And this I want to just use it also as a small announcement. So, we created a GitOps book in German and it's a really comprehensive book with more than 300 pages and this book will be translated in English as well probably. So, fingers crossed and so keep an eye on it if you liked this talk. Thank you very much. Thanks a lot.