 Hello everybody. My name is Tomasz Seng. I'm a Cloud engineer at VirtusLab. I work with AWS and Azure on my daily basics. I'm a maintainer of Kubernetes distribution for one of our clients. These days I mostly write software in GoLang and infrastructure as code in Terraform. Let me tell you a story. There will be a dragon and the epic fights. Once upon a time there was a Jenkins snowflake and no one knew how it worked. The Jenkins has been deployed manually on a random EC2 instance, but someone, that was no longer working at the company. We didn't have full control over it. There was no Jenkins admin credentials and no SSH keys to the instance. Everyone just crossed their fingers for continued work. In the meantime, a team of four heroes was sent to kill this beast. There were various problems to solve. The Jenkins version was very old and as a consequence, not secure, it was running on a top of infrastructure built by hand. It was nearly impossible to recover. The Jenkins home directory and the result, its backup was huge. The agents didn't automatically scale to the number of builds, so it was either wasting our resources or not having enough of them. Our team of four went to the first battle with the improvised weapons. It was a massacre. Nothing went according to the plan. They fell victim to the silver bullet anti-pattern and the hype during development. Unknown tools and technologies were used and they didn't scale to the complexity of the problem. This solution had less problems. The main gain was that we could recover from an easy-to-instance failure. Also, restarts of the Jenkins were less scary and the version was up to date. Fortunately, there was still no auto-scaling and too many things were done by hand. As a bonus, we gained new problems with terraform state inconsistency and unsealed scripts that were hard to write. Enter the Kubernetes. The team of four met a wise druid. He told them about a magical artifact called Kubernetes, so they went on a quest. The challenge was to assemble all the pieces together. In the era of the Kubernetes 1.4, there was no production ready distribution or installers. Every cluster had to be built and hardened from scratch. I will leave you to your imagination how hard that was. The main advantages of using container orchestrator with the Jenkins are auto-scaling and self-healing. The simplest way to deploy anything to Kubernetes is to apply that bunch of manifest. As you can see, there is eight of them. Not the end of the world, but not convenient either. High-driven developments tried again with Helm 2.0. It was supposed to make things easier. Spoiler alert, it didn't. There is a whole article written by my colleague about it. The title is thinking twice before using Helm 2.0. This solution was more complex, but solved more problems. Auto-scaling and self-healing were now available. Recovery was almost automatic. Yet, there was a ton of accidental complexity and errors were not clearly visible to the end user. This made it fragile. Gaining a powerful weapon as a Kubernetes was not enough. It required learning how to use it properly before it could deal the maximum damage. Managing Jenkins is complex, and automating it makes things 10 times harder. To properly express the domain, it felt necessary to use real programming language. Golang with the operator button were the answer. Operator is a program that controls whole life-side of another program. In this case, we have bots running in Kubernetes managed through various APIs. The operator is Kubernetes native. It runs a separate software, has application-specific knowledge, and extends Kubernetes API with custom resource definition mechanism. Here we have an example of a custom resource definition. You can think of it as a class definition from the object-orienting programming. Here, we use the previous definition just as we would use another Kubernetes resource. You can think of it as an object instance from the object-oriented programming. The control loop of an operator uses Kubernetes resources to check if the observed system is in desired state. The loop is very simple. Observe, analyze, and act. Let me give you an example. We have a Kubernetes cluster with Kafka operator running on it. The user creates a custom resource and the operator control loops gets triggered. The operator compares the actual state of the resources with the desired state from the custom resource. It notices a stateful set is missing. The stateful set gets deployed and then the Kubernetes stateful set controller creates three Kafka pods. Later on, the user updates the Kafka custom resource with replica set to two. Then the operator control loops gets triggered again. The operator compares the actual state with the desired state. The difference is the number of replicas. Operator updates the stateful set to adjust the replicas number. Then the Kubernetes stateful set controller deletes one Kafka pod. Operator framework helps to build operators, especially the SDK. With operator SDK, you can bootstrap a new operator even if five minutes. Then you can go straight into design your custom resource definition. Next steps would be to implement the control loop and start watching some resources. Operator SDK encourages a specific deployment flow. After building the foundation, we start the infinity loop of local test, end-to-end test, and publishing. The end result is a set of manifests and the container image. Operator framework also provides operator lifecycle manager. It can be used to deploy operator to clusters at scale. Operator lifecycle manager uses cluster catalogs as a source of its operators. The biggest cluster catalog is the operator hub.io. If you want to run an application tab of Kubernetes, I would check this site for an operator. Jenkins operator is also available on the operator hub.io. Now is the time for the final battle. On the one side, we have a dragon. On the other side, we have the fellowship of the operator. As you can see, logo of the Jenkins operator is combined from four different logos. The blue guy is the mascot of the Golang programming language. The suit comes from the Jenkins. The Kubernetes logo can be seen on the plate. The green V is like VirtusLab, my company. Olex showed us before. We have a night t-shirt with the Jenkins operator as well. I have the same with me. There are many aspects of operating Jenkins. We want our Jenkins to be stable and secure. We believe that Kubernetes and operator SDK give us the right tools to properly describe the domain of operating Jenkins automatically. With the power of full programming language, we can express all of its complexity. Similar to the Kafka example, we have an operator extending Kubernetes API to manage full lifecycle of Jenkins. Jenkins operator watches custom and native Kubernetes resources necessary to run Jenkins. Based on those, the operator manages Jenkins master pod and configures the Jenkins itself. Then Jenkins uses Kubernetes plug-in to run builds on top of Kubernetes. The Jenkins operator control loop is divided into two phases of operation. The base phase is responsible for creating Jenkins with same defaults. The user phase is responsible for applying all of the user's configuration. The base phase ends when we successfully run all base configuration Groovy scripts using Jenkins API. Also, we do a lot of more stuff, for example, we install plugins and so on. The user phase ends when all of the user configuration has been applied and backup was restored for the Jenkins job history and artifacts only. Let's go through some Jenkins custom resource examples. Here we have a Jenkins custom resource definition. Here we have a simple Jenkins custom resource example. The underlying part is where the Jenkins master pod definition starts. Here we have the official upstream LTS Jenkins container image set. Custom resource is not the only manifest used by the operator. Operator uses also native Kubernetes secret and config maps resources to allow for user configuration. We can use them to customize Jenkins configuration with the operator. Here, a user adds a Groovy script to the config map. In this case, we configure the number of executors on the Jenkins master. Actually, this example could be simple, but I wanted to showcase the secret management. The operator makes Kubernetes secrets available inside Groovy scripts. Users can also use configuration as code plugin that is installed by default by the operator. Similar to the Groovy scripts example, here we have a secret and a config map. In this case, we set a Jenkins system message that shows up on the Jenkins main UI. As you can see here, we can also reference Kubernetes secrets values from the configuration as code YAML. Jenkins operator by default installs plugins necessary to make it more useful on Kubernetes. Those plugins are required by the operator to be fully functional. In this case, versions can be adjusted, and only explicit versions are allowed, no latest. The dependent plugins are installed automatically as well. Users can also add more plugins in a similar way to the base plugins configured. All the plugins configuration can be adjusted using Groovy scripts and configuration as code YAML showed earlier. One of the default plugins that is available in the JobDSL plugin is JobDSL plugin. This example shows how to add a seed job that will add Jenkins jobs from a Git repository. There must be a repository URL. In this case, it's a Git repository, but it can be a different version control system there. We have to add a path to the JobDSL scripts called targets. Here is an example of a JobDSL script. It creates a pipeline which is stored in Git. Again, a Git repository URL is provided here. The path to the pipeline file inside this repository. Here is an example of a Jenkins pipeline with Kubernetes plugin extensions. Here you can reference a container by name in the pipelines. Another feature of the Jenkins operator is the backup and register of jobs, history and artifacts. Currently, a local disk backup is supported, but you can create your own provider. The Jenkins home directory is mounted to the backup container. We attached a persistent volume to the backup container to store the backups. It can be some kind of network attached storage or network attached file system. Kubernetes PVC API is extendable and many providers are available. Here is an example of PVC that in AWS can be backed by an ABS volume. The backup and restore feature provides two extension points. One for the backup command and the other for the restore command. Then can be run on the same container or two separate containers depending on the use case you have. The default image uses just simple bash scripts. It's enough for them. Another feature is notifications integration with multiple providers starting with Kubernetes events, email, Slack, Mailgun and Microsoft Teams. It will allow for better visibility of the Jenkins state to the users. The most important events will be broadcasted to the configured providers with the configured logging levers. Kubernetes events are standard and cannot be deactivated. So the available logging levers are info and warning, similar to the Jenkins events. I mean Kubernetes events. For even more information like stack traces, you can enable the variables mode. Okay. So now it's time for a demo. So Jenkins operator acts really fast. So I've recorded the video for you. In this demo, I would like to show all the features that I discussed so far. So first of all, we have to create the Kubernetes namespace called Jenkins. Then we apply the Jenkins custom resource. And as you can see on the screen, after applying, the operator created a bunch of Kubernetes resources, including pod with Jenkins masters, two services, one for HTTP, one for agent, service account, secrets, config maps. Now the Jenkins is starting up. And I'm going to copy the node port because I run this example locally in Minicube. So the operator creates Kubernetes secrets and stored their username and password for the Jenkins operator account. And you can extract this password from the secret with this command. Everything is documented in our website. So now I'm going to copy this password, just to use it later. And here we have logs from the operator. As you can see, the operator starting, which is some resources, pod, secret, config map, something control loops. And now it's creating the Jenkins master pod. Generating Jenkins API token for the authorization. And now it's applying the groovy scripts with the same defaults I have said before. Okay. So as you can see, the base configuration phase is complete. And the user configuration phase is complete because I didn't specify any custom things. So it took the same time as the base. So when you see this in logs, that means your Jenkins is fully operational and you can look to the Jenkins. So I'm using this password extracted from the secrets area. And as you can see, there is no warnings. There is no warnings on the plugins. So our plugins are up to date. And now I'm going to configure something Manawali because the Jenkins created by the operator is immutable. What does it mean? For example, if I change something Manawali like this, so now I'm changing the master executors to one. If I will lead the spot, this change will vanish. I will show you in a second. So for example, we can also create a job. And also this job will vanish after restart the Jenkins pod. So now I'm going to delete this Jenkins master pod. And as you can see, another pod is running. So I cut the video to speed up the presentation. And as you can see, again, the operator applies the base configuration with groovy scripts. And user phase is complete. So Jenkins, you can log into the Jenkins. So let's refresh the website. And as you can see, there is no job which I created previously. And the build executors is set to zero. So as I said before, those changes were vanish because we don't persist this configuration. Everything should be put in the Jenkins CR, basically. So now I'm going to apply the... I will modify the Jenkins custom resource with all the stuff I mentioned previously. So as you can see, I configured the notification for the Slack. And operator gives me information that something has changed in the configuration. So it's going to restart the Jenkins master pod. So now it's creating the Jenkins master pod. As you can see, base configuration is complete. I'm waiting for the user configuration to complete. Okay. So it's complete. We can log into the Jenkins. We use the same prediction as before. And as you can see, the scheme has changed. We have... We set the system message from the configuration as code plugin. We set the master executors from the groovy scripts. And we set the seed job which creates three different jobs. So to showcase the backup resource feature, I'm going to run this build, the Kubernetes plugin. So it runs the... another pod in the same namespace as the Jenkins. And it connects to the Jenkins as an agent. That's how the Jenkins Kubernetes plugin works. So the plugin waiting for the pod. We can test this running on that. As you can see, there is KAD as agent pod. And now it's terminating because I set only echo hello world there. So we run this edge app successfully and we have one build in this job. So now I'm going to delete the Jenkins master pod. And after we create the Jenkins master pod, this job history should be persisted because we configured the backup and restore feature. Now we're waiting for the user phase to complete. It's complete. And we're going to see if there is really one build in this job. And again, seed jobs get triggered to create the jobs. And as you can see, we have one build stored successfully in this job. And now I'm going to add the config map with the cruvy scripts. And I want to change the amount of master executors. For example, I can change to something which should be not done. And as you can see, we get notification from the Jenkins operator in the Slack that tried to run this cruvy script. And it encourages some error. And you can see stack trace because we set the verbose mode for the notification feature. So I can see what is wrong with my cruvy scripts which configured the Jenkins. So this is very nice because taking the change almost instantly. Now I'm editing the Jenkins custom resource. And this character is not allowed. So when I save this, I get another notification that the validation failed. And this plugin is not... The version is not good for the Jenkins. Okay. So now I'm going to delete the Jenkins custom resource. And because we set owner for all resources created by this Jenkins custom resource, after deleting the Jenkins custom resource, all the resources will be deleted as well. And that's all for my demo. Now I want to show you the the GitHub repository. As I said earlier, we have community. So you can join our Slack. Feel free to just come and say hi. Okay. This project is bad or great. So it's up to you. So also we have a community call of every Thursday on Google Meet. So you can talk to us in person, ask some questions. Maybe you have... Basically, we can gather some feedback from you to make this better for the other users. We also in the GitHub have a project. So you can see what's going on in the project, what's the project, in progress, ready to release, and what is close. Maybe we can change the priority of the issues. So if you have some issues, please fill one. So we can have discussion for it. Also, we have a website for the Jenkins operator. We have nice documentation there. If you see some gaps, you can just fill the issue. We'll try to make this better. And nice thing is the schema for the Jenkins CR. So if you have problems with the validation, you don't know which is possible or not. You can just click on the link and see. Okay. So the backup looks like this and what the fields I need to field, basically. Okay. So that's all for me. And I'm waiting for the questions from you guys. Thanks so much for the presentation. For participants, again, please share your feedback in the Google forum. We will really appreciate that because it will help us to improve the meetups, and we will also share feedback with Tomas. We will also appreciate that. And let's go through some questions. We answered some more synchronously, but we still have some on the list. So the first question is, why do you use a single CRD type of these different fields in the specification instead of different CRDs like Jenkins master, Jenkins plugin, etc.? Yes. So I forgot about this. One slide there. Roadmap. Yes. So in our roadmap, we want to, first of all, focus on the refactoring and improvements. Refactoring means that we want to split the user face into a few custom resource definition as the user asked this. So this will be done in the near future because we face that the Jenkins custom resource now is huge. We want to make it light, basically. Yeah. So this is the answer. We will do that. Thank you. Another question. So do you consider using Istio and Jenkins Kubernetes separator? Yes. So because we use the pod for deploying Jenkins master, so we act like the controller for it. And this is how to put it. So it don't work as we expect. So we want to move this to the Kubernetes deployment. And with deployment, we won't have problems with Kubernetes admission controllers which modifies the pods. For example, inject some site container like Istio do. So after refactoring and moving from pod deployment, you will be able to use the Jenkins with the operator and Istio. Thank you. And we're coming forward to this change. So the next question. You currently rate the capability maturity level at basic according to operator, do you plan to target level through your hire? Yes. So on my list, I have to deploy the new version 1.4.0 which have more this capabilities. For example, I don't remember everything from the list, but this one from seamless upgrades. So it's in 0.4.0 is available. So we will update this when we will update the operator hub.io catalog for the Jenkins operator. Thank you. So the next question is, could the operator configure Jenkins authorization and authentication, for example, Dapp SSO and other methods? Yes. It's possible because you can configure the Jenkins through the groovy scripts. And in these groovy scripts, you can configure this authentication authorization, also with the configuration as code. But there is one thing you have to be careful because the Jenkins operator also needs to apply this configuration for the Jenkins API. So you have to give to the Jenkins operator account admin permission in this, for example, LDAP or OAuth. It is up to you what do you want to use to be able to apply those configurations for you. Without that, the operator won't be able to configure your Jenkins. In principle, there is a way to configure that, even if you use LDAP account. So one of the topics we consider maybe for the next week is to have a meeting between Kubernetes Jenkins operator developers and Jenkins developers so that we could see problems and discuss how we could address them. Stay tuned for that. I want to add one to this because we also talk about the authorization method for the Jenkins operator to the Jenkins. And I think we have to extend this model because now we have two strategies and we have to add more to basically user can define all this authorization and authentication mechanism delivered by the plugins in the Jenkins. So it will be improved in the future as well. Thank you. So the next question, why is it better to use operator and not Helm chart with backups and some other things you have in the operator? What are the real advantages? Okay, so with the Helm chart, you have no guarantee what the status of your Jenkins is because you just deploy and you don't know what's going on. If the backup has been restored successfully or there is some problems with scaling or scaling down. So you don't know if your Jenkins is running or not. But with the operator, it keeps every configuration the Jenkins running all the time. So it's checking. Okay, this is okay. Check and move to another list. You have to have to check. So with the Jenkins, you have more you can be more so you can you can knew that your Jenkins is running basically. So if something is going wrong, the Jenkins operator can send a notification to you that, okay, something is wrong. I can do something. So you have to check this. But with the Helm chart, you don't know what the status of the Jenkins is. Thank you. So we have a few questions left. And if somebody wants to ask more questions, please submit them. So there is always a limit on the agent so that it can be used with a Jenkins master. Does it help in scaling? So we set the Jenkins master executors to zero for the security reasons. So everything should be run as the Kubernetes plugin as a separate pod under the need. So that's the answer. Plus we thought about something like managing agents by the operator and connect them to the Jenkins which is configured by the operator. So for example, we have some kind of use case. So we have for Scala SBT and it has to be run for some part of time to be to fast compile things. And with the Kubernetes plugin, you run the agent as you run the build. And you don't have this opportunity. So we thought about some kind of static agent, but this is not easy. But it's doable in operator. So for example, we can extend and create separate custom resource definition for managing the Jenkins agents, basically. Thank you. And the second similar question is, is it possible to have multiple Jenkins masters deployed in the cluster? It's possible. But it's not possible to run two Jenkins master from the one Jenkins custom resource. So there is how the Jenkins works. So you have one master. Everything is done by inside and making this high availability. For example, spinning two of them, it's not easy. So we have talks about it, but basically it's not easy to do that, basically. If we will know how to do that, we for sure will implement this to make the Jenkins be more available for the users. Thank you. So there is another question which is rather related to governance. I'm just looking to find it. So do you plan to join CNCF and make this operator official? To be honest, I didn't thought about it. So this is my answer. I don't know how the process is and what we can gain, what we have to do to join the CNCF. I know there is some, for example, there is some checklist and you have to make all these things from the list before you can join with this project. So just to clarify the current state from the Jenkins governance standpoint, Jenkins project itself is a part of continuous delivery foundation. And continuous delivery foundation is also a part of Linux foundation. So basically we have a lot of communications with CNCF, but Jenkins itself, it's in another foundation. Regarding Jenkins Kubernetes operator, right now it's formally donated to the Jenkins project. But in the future, it's evolution may really depend on how project goes forward. For example, Jenkins X also started as a part of Jenkins project, but then it evolved to a separate project. Something like that may also happen with Jenkins Kubernetes operator. Let's see. Okay. And the last question, which is rather related to general use in Kubernetes, is how do you build Docker images using Jenkins and Kubernetes? So as you can see in the presentation, I used the official Jenkins image, LTS. So we don't build some custom Docker image. I know there is some practice with bake some plugins in the Docker image. So it's possible from in the operator, but it's not... So basically it's up to the user how to want to do that basically. So from our perspective, we have to do everything that we put in the Jenkins CR. So for instance, put install plugins and whatnot. So it's up to you what do you want to use. So for the OpenShift, they have separate Docker image, which some baked plugins, and it's also run with the Jenkins operator. Thank you. The next question is, is it possible to run two completely different masters using Jenkins Kubernetes operator? Yes, of course. You can have multiple instances of the Jenkins as you have multiple instances of the Jenkins CR. So one good practice we learned that is good to have one operator for one Jenkins because in the operator you have a lot of information in the logs or the notifications. So it's make the work easier when you have one operator and one Jenkins. But for example, if you have another nice space, you can deploy another operator with another Jenkins. So it would work basically. Thank you. And the last question we have in the list. So the manifest file says epiversion.jnksio slash v1alpha2. Does it mean that the Jenkins operator is an alpha station and not recommended for production use? This v1alpha2, I know it can be confusing for the users that changing version for the customer service definition is painful in the Kubernetes because you have to somehow translate one version to another. So the version one supports v1alpha2 and version two supports v1alpha3 or whatnot. And you somehow you have to or manually change the the Jenkins custom resource. And we for now don't have a tool which will automatically detect that. For example, we run the Jenkins operator version two and translate from v1alpha2 to valpha3. But if we will get this kind of tool, for sure we will change. So we will change the valpha2 when we change the Jenkins custom resource API. And this occurs only one in the past. So we have to change some map from the list and this is the breaking chance. So we have to introduce the new version. And it doesn't mean that it's not production ready. Our customer runs this in the production successfully. So don't look at it. Thank you for the answers. It was the last question in our queue. So Tomos, would you like to add something before we close? So I'm really happy if you want to write some goal line code and help us to build the Jenkins Kubernetes operator. Have so fun. New language programming. So if you have some time, you can raise issues, make some pull requests. We have calls. So I want this project to be more driven by the user. So I don't want that that's this project from one person, but I want to be driven by the users, basically. So if you want, you have some spare times and please make some pull requests. So it will be fun. I really like the code language programming. It's very nice to me. So that's all from me. Okay. Thank you. Thanks so much for the presentation and thanks to all of the questions during this meetup. And looking forward to seeing the Jenkins community. Okay. Just a quick heads up. We have our next meetup on Friday. So Marky will be talking about securing your Jenkins CIC container pipeline with Unchall. So Marky, would you like to quickly introduce it before we close? Yes. Can everybody hear me okay? Yes. Hello, everybody. We will be demoing the Jenkins pipeline plug, excuse me. We will be demoing the Anchor scanning Jenkins plugin, which will allow you to scan items within your job, items being Docker containers, and then get alerting for that. And it's very in line. And it's a really awesome tool to use for DevSecOps. And that's what I'll be presenting. Look forward to seeing everybody. Thank you for the introduction. And again, thanks everybody who attended this meetup. We will process the video and publish it tonight. And if you have any questions, please join the Slack channel shared by Tomos. And we will be happy to answer questions and consider the discussion there. So that's all. Thank you. Thank you.