 Regulated environments have their own unique requirements when it comes to compliance, separation of duties, quality as well as access controls. This requires specialized treatment throughout the software delivery process. In this talk, let's hear Christian and Andreas talk about the evolution and growth of GitLab in their environment at ERT to close to 1,000 pipelines, challenges that they've faced with scaling, as well as how they're integrating GitLab with Captain to overcome scaling challenges and integrate monitoring and quality gates SLOs into their software delivery process. Like we've mentioned before, GitLab is open source and plays really well with others by providing APIs for almost everything that you can do within GitLab. We truly believe in the theme of today's event, let's innovate together to bring the best solutions to our users and customers and that's what you will see in today's talk. Over to you, Christian and Andreas. Hello and welcome to our session, GitOps in a Regulated Environment using GitLab and Captain. My name is Christian Heckelman, I'm a DevOps engineer at ERT. ERT is basically from a technical company who is a technology company that helps to minimize risks in clinical trials or in other words, we are a vendor for clinical trials. And with me today is the one and only Andy Grabner. Hi Andy, how are you doing? Hey Christian, thank you so much. Yeah, I'm doing pretty well. It's been a while since we saw each other last, just a couple of days I guess, but only virtually, unfortunately. I'm very happy that you brought me on the stage with you. Typically I'm in the driving seat, but this is where it's great that you are taking the driving seat because it's really your story that you've been driven from the GitLab site and then integrating it with Captain. So about me, I'm a death rail for the open source project Captain at Day and the DevOps at Dynatrace. And thanks to Christian, he really showed how easy it is to integrate GitLab with an open source product like Captain and he also helped us to drive Captain forward. But now Christian, back to you because I will come back later and talk a little bit about Captain, but you are the main star here. Right, thank you. So let's take a little bit, look back, so scaling GitOps in a regulated environment, how we started using GitLab actually in our company, how we started Kubernetes in our company. And it started more or less quite as an accident. So back in the days, a couple of years ago, we had a lot of different source control systems. We had TFS, we had Bitbucket, we had Subversion. We had different build systems like Jenkins, TeamCity, also TFS, GoCD, and so on. And we also had a Git server, which was basically a big box with an Apache mod Git on it and every time a developer needs a new repository, he needs to request a new repository, then the administrator was creating the repository and so on. We had a complex or difficult process in our case. And also my colleague and I, we need to provide a new pre-receive hook, which was not able to execute on the old box because it was an old CentOS version, which does not support the new Python interpreter, we needed for the pre-receive hook. So my colleague and I, we were installing just for our use case a GitLab server. And all of a sudden, all the different development teams were looking and saying, hey, cool, you're using GitLab, can we use it as well? And we said, yeah, sure, hop on. And so it started that people are using GitLab more and more and more in our company and then our company decided to do something like what we called one ERT DevOps Toolchain, which means we were consolidating all our different source controls and moving to GitLab. And as well, at the same time, we were using or started using Kubernetes for our microservice deployments because till then we were still deploying to VMs using Puppet as a configuration management and rolling out there our, for instance, a Java application, right? So as I said in the last two and a half years, we were migrating over to GitLab with one, yeah, from move from GitLab CE to GitLab Ultimate, but I will come into this topic later. And as you can see, GitLab was really adopted very well. So users were using it because it's a self-service driven approach. It's easy for them to use the integrated CI-CD stuff. So around about 3,000 projects state now, it's on our instance, and around about 1,000 CI-CD configurations. And as I said, we were also starting using Kubernetes for our deployments. And so we started to scale our CI-CD pipelines. So Kubernetes aren't the only services we are providing, but a bunch of them. We only have around about 60 services running in our Kubernetes environment at the moment, but we have a lot of different environments. So we don't have only a pre-prote and prod, but we have a lot of development environments, integration environments. In our case, pre-prote is in validation environment and production environments, which ended up in a lot of different deployments. When you think about 60 services in 14 stages or environments, we ended up in around about 840 deployments for all of the services in the different environments. And to scale the delivery pipeline, we started using captain to orchestrate all the tools around our delivery pipeline, right? But let's take a look back and how we started using GitLab, GitLab CI-CD in our Kubernetes deployments. So starting with the generation one, we only had GitLab community addition. I think it was around about version nine. So, and as I said, we are working in regulated environment, which means that a developer shouldn't be able to deploy to a higher environment like in validation or production environment by its own. It's regulated in our case. So what we did here, we had built here the separation of duties with our capabilities of the CE edition. What means that the developer was building the software, it's running a unit test and packaging his software in his repository and he's also able to deploy to a development and integration cluster. But then development was triggering IT operations and said, hey, look, we have here a new artifact. We have here a new image of our software, of our application. Please deploy it to the new environment. And we had separate repository where we then are taking a new image tag or new deployment information and deployed it to our validation or production environment. We also had a lack of automation around it for test automation, for instance, or for monitoring automation. This wasn't done. We did it manually, like creating synthetic checks in our monitoring system and so on. And also for our Q&A department, we were triggering them so that they can run their performance or load tests or end-to-end tests against the application after the deployment. So it wasn't integrated well, right? So we need to work on that. And with new capabilities of the CE edition, like multi-project pipelines, we had more and more this end-to-end visibility within our workflow. And we also started using GitLab's ICD templates, for instance, for our Dynatrace automation. So creation of automatic synthetic checks, pushing deployment information to Dynatrace and also adding a little bit of resilience to check our monitoring system after deployment if there are any defects detected on the deployment itself. We also, or our Q&A department, also contributed some CI-CD templates like for Jmeter and Catalon tests. And we were starting using Helm instead of plan out YAML files to deploy it to our Kubernetes cluster. But the end of it was we had a big monolithic pipeline. And what the big monolithic pipeline looks like is here on this screen a little bit visualized. That's only a small part of the pipeline itself. So as you can see, we have a lot of different stages. We have a lot of jobs, upstream and downstream repositories. It was quite hard. And the main CI-YAML file, as you can see, is around about 1,000 lines of code. And if you think about when you're writing in microservice and the microservice only is about 300 lines of code or let it be 500 lines. And you have a deployment pipeline with over 1,000 lines of code without any included templates or any other tools around it, like Docker images you're using within your pipeline bash scripts, that's quite hard and quite complex and quite hard to maintain for a small operations team. So what we can say with the conclusion of our Generation 2 pipeline that with GitLab, we were able to use a single platform for our development lifecycle. We were able to consolidate our different tools we had to now and enable the developer in a self-service driven approach to create the projects to use our Kubernetes cluster for easy prototyping of new services and so on. And that's all triggered from GitLab. But the pipeline complexity is really significantly increasing and troubleshooting the pipelines is also a little bit challenging if you only have a small team of DevOps engineers or operators who needs to support the developers with their pipelines. And also one thing, our remediation use cases later on when something new is deployed, how you make sure that your service is running and what are you doing to, yeah, to remediate error rate increases and so on. So then we were stumbled across the project captain at one of the Dynatrace summits and we thought, hey, that would be a great benefit in our pipeline, our tool chain. So, and also in the meantime, we were able to use GitLab ultimate. So we were now able to use multi-project protected environments in our upstream repositories. We have the ability to use multi-cluster support for Kubernetes for different environments. We were able to use feature flags to continuously deploy something on production environment and then later on on a date X and able just the feature. We were able to put security features in our CI CD pipeline and so on. But for us, the great part was introduced some captain in our pipeline. So we were able to replace all our Dynatrace CI template with captain services, which are working out of the box. We contributed a synthetic service, for instance, with the captain community as well, for a grading automatic synthetic test in our Dynatrace template. We contributed a GitLab service. So captain couldn't be only triggered by GitLab, but captain can trigger GitLab by itself as well. And we could replace our low testing stages with a Jameda service. But the biggest benefit here is introduction of quality gates. So SLI and SLO based evaluations of your deployment every time somebody is deploying something in every environment, right? So as CD part of the pipeline will look like this. So we have this pipeline will also be triggered by the actual source code repository. So we still have the separation of duties here, but it's fully automated. So in the first step, we will download our artifacts, which are basically the captain configuration in the CI repository. So you have kind of, not a kind of, but you have this GitOps approach that the developers can define what he's expecting and we will automatically assign it and then check it. Then the first beginning step or second step will be the captain onboarding, where we are creating the project and captain automatically or updating the resources. When you will create a project or update resource in captain, captain can also put all the stuff into a dedicated upstream repository. So you have a different GitOps approach here as well. So, but we are doing it slightly different like the recommended way, but it is because we want to have all the configuration next to our actual source code. Then with a great and a captain project, captain will start to automatically configure our monitoring in our Dynatris send, but this could be also be a Prometheus service, right? Then after the deployment, we'll send the deployment finished event to captain and that's all the developer needs to do. He sends the development finished event and captain will orchestrate all the tools around it based on the configuration. So it will, for instance, push deployment information to our monitoring system where it says, hey, this version was right now deployed on the service by this pipeline, et cetera. Captain will then trigger all the other tests like Catalon tests, which are other GitLab pipelines which are being triggered and the results are coming back to captain and will be evaluated as well, as well as running our low test in Jmeter. And all we need to do now is to wait for captain for its evaluation of the SLI and SLOs. So we will ask captain on the quality gate stage, hey, is your evaluation done already? And if so, is the evaluation result positive? So is your service acting as expected? And if so, it will automatically promote the artifact to the next higher stage, or you can do what we are doing here with approval stage to have this in GitLab the environment separation here, that one of IT operations needs manual approves a step and then we can deploy to production environment where we can then enable the feature flag whenever we want, right? So we deploy your software and then can use the GitLab feature flagging to enable the new feature on your production environment. This is status quo in our company with captain zero seven, but moving on with captain zero eight and captain zero eight is now able to deploy to multiple Kubernetes clusters. We can completely get rid of the Cd part here and just triggering all the stuff directly within the CI part from our source code repository. So it's a developer will build this image, all the tests will being covered. And then we are onboarding the captain project and captain will start running completely, yeah, separate, triggering all the other tools, deploy the software to deaf cluster, triggering here the evaluations, automatically promotes the artifact to an integration cluster. And also here it can be then approved by a GitLab pipeline or within the captain bridge. It's only an API call and then promote the artifact to our production environment. So that's the forecast GitLab or captain zero eight was released a couple of weeks ago and I will need to take some time to adapt it for our GitLab pipelines. So in conclusion, what I can say, what we had in the past, we had big monolithic pipelines, which were hard to debug. And now we have much more smaller pipelines with a declarative approach, how to use the tools. We using captain to orchestrate all the stuff around it, around our deployment. And in the future, we can use captain entirely without writing a single line of the ICD configuration or batch script to deploy our software faster to the different environments, having dependency management with manual approval stages and having more resilient in our deployment when we are using captain automated operations like disable a feature of like in GitLab when the error rate increases on your service. So with this being said, I want to hand over to Andy, right? So he can tell you a little bit more into detail about captain. Thank you. Yeah, thank you so much, Christian. It's just an amazing story that you just told us. And also along the way, you just helped us to really, you know, make captain better so that the both worlds what you are using on the GitLab site and captain helps you in the end to build better automation that is easier to manage. So I want to give you a quick overview for those that were captains completely knew what captain really is all about. You see here a couple of links that in case you want to follow up with us like the website, follow us on Twitter or join the Slack channel. So the question is why captain, right? Why would you use captain? They think that we have seen from Christian already you need as a deaf also an SRE engineer put a lot of automation efforts into your pipelines. You need to automate starting from deployment through testing monitoring configuration quality gates. And I'm pretty sure there's more things here that you need to put into pipelines. And what we've seen from Christian it's all possible but it can get, you know a little complex and hard to maintain and hard to troubleshoot. Also what we, at the day job I work at Dynatrace and I got to admit we're still using Jenkins but I'm pretty sure we're looking to GitLab soon to replace that. But what we are currently also facing is a challenge where we started with Jenkins pipelines which were great for us. But as we started with the first pipelines for microservice environments and then switched it over and rolled it out to more services we ended up copying, pasting and then in the end across all the different services and projects we ended up with this so-called snowflake effect of different permutations of our pipelines. We then did an analysis and also saw with our work we have a lot of code duplication in our pipeline code which makes it very hard to maintain. So here obviously GitLab will already help if we move over. So now the next thing though is what does captain do? So captain's approach and architecture was heavily driven by the new requirements we saw as more and more people are trying to automate their individual microservices or smaller type of applications. Let me explain what we did. So the first thing we said is as we analyze these pipelines it seems everything was kind of baked into one thing. The process, the tool integration sometimes also the configuration was everything in one pipeline or automation script. So what captain is doing we are breaking apart these barriers when we're removing these hard dependencies and we're then separating process from tooling and configuration which means as you can see here we are basically on the left side looking at what type of processes do we really want to automate? What type of tasks do we have in automation sequences? Then we have capabilities. So these tools, we don't call them tools anymore. These are capabilities that can do a certain job. They've moved to the right. And on the very far right we have moved configuration files because certain tools that will need to execute a test that needs to reach out to the monitoring tool to reach to get to query some data. They also, they need configuration files like the test file, the SLI and SLO definition the run book for automation. So we moved that into Git. And what we then did well, we're using event in to really orchestrate that process because we know what process we wanna orchestrate but instead of hard coding which tool is doing a certain job captain's orchestrator is simply sending an event when it is going through the process chain the tasks along a sequence and says, hey, I need some capability that can deploy container number one in depth with blue-green and then you may call a GitLab pipeline that can do it you may call another tool you may just have a helm apply whatever it is this capability will pick it up has access to the Git repository does its job and then responds back a good or no good and based on that captain makes a decision on whether to move on in the sequence yes or no. So really in the end what we have with a process definition so which tasks and events should be sent the tool definition is at the event consumers and then the configuration that is stored per tool. So in the end what we wanna achieve with captain we're not going to replace any existing of your tools we wanna make them less complex we wanna bring additional automation like what Christian said, monitoring configuration automated testing, automated quality gate evaluations we wanna bring these to your pipeline instead of you having to build all this logic in your pipeline code but you can still do it if you feel comfortable do it but we believe we have a great approach to reduce the complexity in case you wanna build all of this yourself. The best way to get started has exactly what Christian said it is and this is also how captain started where they're integrating with monitoring tools like Prometheus, Dynatrace we are really good in test execution so we have integrations with different testing tools already and at the core of captain is really this capability to do service level objective evaluation so SLO evaluation where captain can reach out to your monitoring tools pull in different SLIs metrics and then compare them against your objectives and the nice thing will be built here you can define objectives not only on static thresholds but also based on baselines. So the great thing is you choose with whether you wanna get started and then you trigger that automation sequence from your GitLab pipeline for instance and bring this type of automation into your pipeline with very little effort. Now to sum it up because captain is a lot of things also that Christian mentioned there's some other things they wanna do in the future. Captain at the way I see it is a data and an event driven cloud automation platform almost depending on who you are whether you're deaf and ops and SRE you pick the use case that you want to automate whether it's quality gates, test automation or the remediation you pick your use case then you bring your configuration for this use case every use case needs certain conflicts like I said earlier, the SLIs, the SLOs the workload that should be executed by a look testing tool maybe some chaos experiments if you also do that or some run books for automation. Most importantly, you then connect your tools to captain, captain is the orchestrator. If you have a great CI pipeline or a GitLab pipeline that already like I'm thinking Christian in your case cattle on are something you have you're triggering from GitLab then captain can call this particular pipeline. So take the best of both worlds and let captain automate everything around it the configuration and provide these use cases as a self-service underneath the hood. We are trying as good as we can to follow the GitOps approach as the lows are at the core captain itself is event driven and we're basing the events on open standards we are using the cloud native stand the cloud events standard from the CNCF community kept itself as I said is also an open source project. Hopefully this gave you an overview and Christian, I wanna say thank you so much for all of your work that you've done and thanks for coming to us and reaching out to us with your challenges. Yeah, but I wanna basically hand it over to you maybe for the final words. Yeah, thank you. So if you're facing similar issues like we did in the past. So just stay in touch with us. Follow us on Twitter, find for instance my captain demo in my GitLab repository on gitlab.com slash c-hegment slash captain demo there is also in video recording on one of the everyone can contributes cafes where we are getting into details on this demo and also showing how captain will integrate in GitLab or GitLab will integrate captain and yeah, follow the captain project at Twitter, at GitHub, join select channel. Just reach out to us and have a great day.