 Hi everyone. Welcome to the Jenkins online meetup. Today we have a presentation about moving from local installations to Scalable Jenkins and Kubernetes. We will talk about the Kubernetes Jenkins operator. Thanks to Mateusz and Peter for joining us. They represent Kirtoszlaw, one of the main contributors to the Jenkins operator on Kubernetes. And yeah, today I hope you will have an interesting presentation. And if you haven't used the Jenkins operator before, if you haven't used the Kubernetes operator before, I believe there will be a lot of good topics to study. Quickly about the Jenkins online meetups. So Jenkins online meetups, anything about Jenkins, including Jenkins for users, Jenkins for developers, Jenkins for the community. Basically, these meetups are quite informal. So we for discussing topics we have tried to have live Qoomi live demos. We focus on the interaction with users. We will have Qoomi and we will have Afterparty. Well, a kind of Afterparty after discussion. And yeah, you're welcome to participate during the presentation, after the presentation we will again share chats where you can join and ask the questions. As you may have heard, we do a series of Jenkins and Kubernetes meetups. We started the session last year, we had something like six meetups during 2020. And we continue this year with this is our second meetup. This is a meetup about Qoops and Jenkins. And we will be talking about Kubernetes a lot during CDCon and during the contributor summit on Jenkins 25th, sorry on June 25. So feel free to join. And also there is DevOps world, which is done in September 28th 30s. And you're welcome to join. So big line for call for papers is actually today. And the program community there. And if you're interested please play a few weeks ago we did presentations about how to apply and what will be the community agenda DevOps world. So just consider that. And yeah, shout out to the sponsors of the meetup. So it's continuously different foundation which sponsors the platform and called this which sponsors time for horse and also shwack, etc. So, let's move on. And yeah, for Jenkins and Kubernetes we always look for speakers. So any experiences about Jenkins and Kubernetes relations are welcome so how you use Jenkins and Kubernetes how you do automation how you develop your pipelines. And if you develop plugins or if you develop things like hand charts to Jenkins operator you're more than welcome to join and present. If you just, just doing integration is Kubernetes tools you're also welcome to join so basically be interested in the entire cloud native system. So you may have seen this huge diagram on the CNCF. So basically whatever they're not just Kubernetes is a subject of interest for us, and you're welcome to present. So there is a link below I will share later in the chat. So you can just submit a talk and we will be happy to host this meetup. And of course, today we have two hosts. So it's me. My name is Alek Timnash. I'm one of Jenkins contributors. I work on many topics and I also have a nice online meetups. And we also have Rahram who joined us today. Please welcome him. Would you like to do a quick introduction as a host. No. Okay. Okay, so let's keep working through the data and yeah. For Kone, for Kone, we agreed that the questions will be answered after the main part of the presentation. If you want to ask questions please use zoom Kone or the chat. If you want to ask to use zoom Kone, you can see the controller in the bottom there is Kone section. Ask any question and we will either ask unsighted synchronously in the chat or ask speakers after the presentation. So the main part of the presentation is stay online because we will have an open discussion. So we will stop the recording. So any questions you would like to ask related on related you can ask them of the record and we invite speakers to stay as well. Yeah, it will be very informal and you are welcome to participate in this part as well. After the meetup, there is Jenkins operator on the virtual slot, I will share the link in the chat. So you can join this channel and ask any questions about the operator there and the entire contributor community is in this chat so there will be many people who are interested. So during the meetup, please keep in mind that we have a code of conduct in Jenkins. This code of conduct is very simple. It's been nice. And yeah, there is more text here but it's quite short. So I won't stop much on that. Please welcome Mateusz and Peter. They will do the main part of the presentation. Thank you. Thank you as well for the introduction. Thank you very much. And let me just share my screen. Okay. Does it work? Can you see it? Yep. Yeah, thanks. Great. Thank you. Okay, so welcome everybody to our presentation about a little bit of a journey from a local Jenkins installation to a Kubernetes cluster deployment. In this presentation, we would like to show you how different size organizations might deploy Jenkins and the challenges they might be facing on the way. But first, let's talk a little bit about ourselves. Okay, so my name is Piotr Reba. You can call me Peter. I'm a DevOps engineer at the virtual slot and together with me is Mateusz Korus. Matt, would you like to say a few words about yourself? Yes, hello everyone. My name is Mateusz Korus and I'm very pleased to be here and share the presentation with all of you and as well as Piotr as the co-presenter. So we have some interesting stuff prepared for you and we hope you will like it. Okay, thank you very much. Yeah, so this presentation is split into parts. We'll be talking about the challenges with different ways of deploying Jenkins. And Matt will talk about how Jenkins operator can help you when using Jenkins on Kubernetes and about some plans for the future. So let's start our journey with a sample project. Okay, so say hello to Alice. She has a business idea. It's a mobile app that helps you keep a healthy diet, but to be honest for our purposes, any project works. And so she's written some code, has some tests, and she would like to integrate a CI into her repository to build tests and upload the staging build. So naturally she looks at Jenkins. This is an option that will get the job done and what is more, it's free. So yeah, great. That's the app. Okay, so Alice would like to keep things simple. So she decides to use Docker. She runs this command and she opened her browser. It worked. This is the initial configuration. And yeah, basically after setting an admin password. She saw this and empty Jenkins instance. And so she did some more configuration on her jobs and pipelines. For example, she might have used the Jenkins file. And then she finished her day and went to some pierogies. So basically she went home. So the next day she comes to work. She puts up her computer, realizes that the Jenkins is down. And so she runs the same command again and opens Jenkins and more or less sees this. So again, empty Jenkins instance. It's gone. So, well, it turns out you can't really lie on the same container to be always up. And you have to treat containers as short lived and make sure your data survives through container recreations. So Alice does a little bit of more reading on the subject and decides to change her Docker run command a little bit. So there are two changes here. First of all, the minus the option, which will start Jenkins in detached mode. So it won't block terminal window. And the second change, she will mount Jenkins home. So this way, her data will be safely kept intact on her local file system. And to be honest, you might call it good enough at this point. You might want to tweak it a little, especially if you're a fan of declarative configuration, you might transform that into a Docker compose file and apply that. But we're not done yet. So Alice realizes she has a lot to do in order to get her product market and she doesn't want to miss her opportunity. So she asks her friend Bob and to help her with the app. And at this point, it starts to become a little bit problematic for them to share the same CI. She might be working from different places, either, well, the same city or even around the world. And also how this notices that when jobs are run, her machine slows down so they figure out that it will be best to move their CI CD tool away from developer machines. So local installation suddenly it's not very practical at this point. So would you move it? Well, one option would be a thing called the machine. So basically, you take what you've been running on your own computer and move it somewhere, somewhere else, wherever it's handy. And by doing that, you can allow more people to use that software since it's not tied to your computer being on and around. So this is an old PC and mini computer like Raspberry Pi, just bear in mind that Raspberry uses different architecture so you have to use different docker images as well. But whatever that has enough power works at this point. And I'm sure many of you have seen this kind of deployment in their careers. It's a little bit of an improvement than we had before. It works especially if one team basically works from the same place, same room. And what's more, you don't have to think about exposing your instance anywhere outside or figure out any security issues. But this approach still has some flaws and well to quickly grow go through them and not to go into too much details. So first of all, you have to do system maintenance, keep your OS in good shape. You have to take care of hardware. When it fails, it will fail eventually. So prepare for that. And do some upgrades in case you run out of processing power. Some environment environment issues might happen that could range from broken AC and your machine overheating or to somebody unplugging your machine from the electrical socket. Yeah, and you have to deal with it networking and security risks. You have to figure out how or do you even have to expose it outside use a DMZ VPN or whatever you have to figure that out. And lastly, it depends on the SLA of the electrical power and the internet supplier. So in case your internet is down. You might not be able to access your CI. So the question would be to use a dedicated server. So question is, should at least now spend a lot of money and order some servers and start looking for a collocation options. Well, it turns out that probably not. Unless she wants to pivot her business. Or that she just wants to build her up. So she'll probably pick something from any cloud provider. And this way she can get rid of a lot of environment problems. And she doesn't have to worry about hardware anymore. It will be kept power to connect to the internet by the service provider. The hardware will also live in a restricted area so somebody stealing your hard drive is much less likely on display. But if we were to revisit our drawbacks from before. So system maintenance stays we still have to do this. Hardware failures will be handled by the service provider. You still need to upgrade at some point in case your Jenkins has to run more jobs. Environment issues get cut down by a lot. Still networking security topics. And lastly, you will suddenly not really depend on the SLI of the electrical power and internet supplier, rather the service provider so the server itself. And yeah. So, if we are to identify a single point of failure at this point, it will most likely most likely be the server itself so in case it's down the whole CICD services down. And so the next step would be to try to address this. We'd like to run our Jenkins in some sort of a cluster and well, normally, this would add another complexity of managing more servers than before because well, we had only one and now we are thinking about several servers so it would be really nice if we got the benefits of running a cluster while building being abstracted away from the OS level. And if you even could get this as a managed service would be a huge plus where all you do is say I want to run this application or to be more precise I want to run this container image. So let's imagine a world where if a container fails gets restarted automatically if an underlying server dies, for example, its work gets rescheduled on other servers automatically. And a world where you can scale your application automatically based on your resources. So, as you probably know, we live in this world. Yeah, that's Kubernetes. That's the description of it. And so I think we can all agree that Kubernetes gives a lot to the cloud computing world. It changes the perspective. That's a little bit of complexity, but you can't really avoid that, that because it abstracts away the underlying OS and hardware and software. And what's more, it's available as a managed service from all the major cloud, public cloud providers. And now we are not really thinking about servers but see Kubernetes cluster as a whole. We are no longer doing with OS and services running processes. We're just thinking about Kubernetes resources like namespaces, spots, services, persistent volumes. So, yeah. And that's how a simple manifest file of a Jenkins deployment on Kubernetes might look like. There are some apparent problems with that. First of all, Jenkins will run in the default namespace and really you should, well, it's advised to have a separate namespace for Jenkins to do some separation of resources. Secondly, and probably the most important aspect, you won't be able to connect to the Jenkins instance because it's not exposed outside of the Kubernetes cluster. So you would have to work a little bit more on that, either your support for wording, set up a service, configure ingress, these kind of things. And lastly, it has the same flow as our initial Docker run command, which is the Jenkins home is not mounted anywhere. So your data will be lost. Well, most likely will be lost. You can get to it, but it's not easy when people gets restarted. Yeah. So, so far, we've been mainly focusing on availability issues. And there is more. There is whole maintenance and operations aspect to that. So we have to configure your instance, you have to find some plugins installed and keep them in good shape. You probably want to run some scripts. And there is a whole topic of observability and monitoring in basically if something bad happens, you have to know where to look for issues and know how to address them. Maybe set up a monitoring and other things. So you'll be the first to know that your services down. And the security hardening. And I would like you to refer to the Jenkins documentation on that subject because it's quite extensive and that's, well, that's that's an important part of documentation to it in my opinion. Yeah. So, I think we can all agree that using Kubernetes will put us in very great good place to begin with. But there is more, more to deal with. And we have to find a way to glue things together so Kubernetes configuration and maintenance of Jenkins and security aspects, all of that. I might ask a question. What is one of the sources of our problems when it comes to running Jenkins and Kubernetes. Well, it has state. It's not a stateless application. So it's not like a pure mathematical function where the output depends only on the input. But Jenkins has its states. So it in the west. That state influences the way Jenkins behaves. And I'm sure many people have experienced at least some problems with Jenkins that are caused by this. For example, one plugin update my lead to a thing we call dependency hell where you have to figure out how to update your plugins and fix their configuration in order to for Jenkins to work. Yeah. So how can we solve this issue. So when it comes to configuration, we can use groovy scripts, which is basically the main entry point to the Jenkins API besides rest API. And this allows us to execute groovy scripts directly on Jenkins. And there are a few ways to do this. One of them is to put some scripts in Jenkins home in the special directory, and they will get executed on each Jenkins start. And I would like to refer you to there is a github repository open source when where there are some very useful groovy scripts that are just waiting for you to go and grab them. So, yeah, feel free. And as a side note. Yes, you can extend Jenkins. And that's a really powerful way. You can use one of the 1500 plugins that are available to download on the site presented on the slide. And just you have to be careful to quote with great power comes great responsibility. So you wouldn't want to end up in the plugin dependency hell. You have to find a balance with the plugins you want to install. So let's mention some plugins that were mentioned for our case. So first of all, Kubernetes plugin. And that's a great way to integrate Jenkins with Kubernetes in order to auto scale your Jenkins agents on demand. So you basically define a pod template and keep your Jenkins main instance running. And when the job, one of the jobs needs to run with the triggered somehow. And a new pod will be started using the pod template. And after the job is done. It will be cleaned and the resources will be. Well, received back pretty much. Yeah. And another topic that will help us with configuring our Jenkins. Well, basically those two plugins configuration as code and the job DSL. So configuration as code. Let us configure Jenkins. And it's plugins using YAML files so human readable files. And the job DSL uses a specific domain specific language that lets you define jobs using again a file. And so it would look a little bit like this. So you will probably have a git repository where you would keep all your configuration files. And by doing this, you get well bashing branches, you know, all those things for free to be honest. And yeah, so we would like to have a recipe to run Jenkins using job DSL and configuration as code kept somewhere. And point Jenkins in that direction. So it uses that to configure itself. And there's another subject that we haven't mentioned so far. So it's the authentication and authorization. Well, as a starting point Jenkins comes with a username and password. Authentication, which is good. It works. But if your organization starts to grow, you might start looking at different options. So for example, you can use a directory service like LDAP or Active Directory or use an external identity and the identity provider to set up a single sign on. You can integrate with external service using all of OpenID Connect. And when it comes to authorization, the builds in way is, well, enough to start with, but if you find that you need time grain access control, you might want to look at the Matrix authorization strategy plugin, which which is really cool can be configured using a configuration as code file. Okay, so a little bit of a summary. So we came a long way from local to Kubernetes deployment. You might be hopping on the same journey at different places. You can start with Kubernetes, which will give you a lot. But I think we can all agree there are still many things to start out to have Jenkins runs mostly on Kubernetes. And it may be important to mention that we haven't touched the subject of backups so basically anything that is not configured using files will have to be backed up somewhere and reverted. And the goal, the goal is to be able to start Jenkins in a predictable and consistent way. And that really concludes my part of the presentation. So Matthew, it's your turn, the stage is yours. Thank you. Great presentation on some of the real problems that people trying to deploy Jenkins and Kubernetes phase. And here comes the Jenkins operator project. The Jenkins operator project is all about automating the lifecycle of Jenkins, which itself is an automation server so you can see that it gets pretty meta. But since we all here love automation, I believe that everyone here is excited about the possibility to have a reputable and predictable deployment of Jenkins possible to them on any cloud infrastructure really that supports Kubernetes which is like almost all of them. Next slide please. So a sneak peek at the Kubernetes operator architecture, the Jenkins operator architecture. You can see here that within the Kubernetes cluster there are two parts to namespaces. The first one on the left is the namespace of the operator itself that works by managing the lifecycle of the Jenkins, which is surrounding the separate namespace on the right. The Jenkins in the right in the right on the right part is just a complete deployment. It has the Jenkins master pod Jenkins agents as well as the jobs. And it's fully functional by itself, but all of the upgrades scaling or scaling is actually done by the Kubernetes plugin that operator uses and provides to you by default. And besides that there are two parts which are the external storage where you can backup your artifacts and your job history, and also the ingress to Jenkins, where you might want to set up VPN or a bastion host. And with Jenkins operator it's very easy to set up identity provider, such as GitHub of. All of that uses all of that within the Kubernetes cluster also takes advantage of the airbag model, which is the model that admits privileges to certain users and group of users. It is very well built it's very rigid and it's great to use. Next slide please. So with Jenkins operator we are really trying to push for the configuration as code mindset, because we do believe that it's great to have your configuration as code. It allows your infrastructure it allows your configuration to be versioned and to be repeatable. It allows you to use the GitOps model which is the new hard work in the, in the industry. And it is great for debugging for understanding the whole infrastructure it's not just, you know, someone some somewhere, someday clicked a few buttons in the UI, and now nobody know how things work. So this is clear in the code that you can see in your git repository, as well as the history of how that state came to be. And next slide please. So as Peter mentioned earlier, one of the greatest things about Jenkins is its accessibility by plugins and Jenkins operator of course allows you to use plugins that you want to use. So if you have an example of a manifest, you can see that the kind is now not not a pod with just the Jenkins image but it's kind Jenkins, which is custom resource definition provided by the operator. And you can specify as many plugins as you want. So you have to specify the plugins that the Kubernetes operator uses by by itself such as the DSL and Kubernetes plugin. They come bundled with the operator cover can extend it however much alike. And the command to apply it is just as easy as it is for the deployment using a pot without using a Kubernetes operator. Next slide please. So as Peter mentioned earlier, one of the big subject for Jenkins is security Jenkins is by software and software standards, a pretty technology with pretty long history. So it has some legacy mechanisms with it built in it that even Jenkins themselves, the Jenkins itself advises users to disable such as old JNLP protocols, CLIs and other ones. So Jenkins Kubernetes with its initial configuration of Jenkins does it by default so we don't have to even worry about it. And you will, you will have a secure Jenkins deployment without having to manually tweak every setting that needs to be tweaked. So the top of that, as I mentioned before, it leverages the airbag model from Kubernetes, which is really good and the, like the new industry standard really. So that's, that's a great thing about the operator it, it allows you to have peace of mind without actually doing very much work in the department of security. And next slide please. So the thing that Piotr mentioned before also is backup and restore with our mindset with our configuration as code mindset. The only thing that you need to backup are jobs history and artifacts, because the whole configuration you have in your Git repository. And whenever Jenkins starts up, it can just pull all this configuration and have all your jobs ready. So you can have a restore script that runs and pulls your, pulls your jobs history and artifacts, but the job definitions themselves will be handled by, by the operator using the job diesel plugin. And next slide please. So, if you stick to, to some of the good practices that we, that we presented, such as having, you know, an fMRL state of Jenkins, the repeatable deployment, basically having a recipe for the Jenkins that you want to get. You can use the Jenkins operator to work behind the curtain to actually to actually deploy the Jenkins in the state that you specify. And it handles all the updates, all the lifecycle that needs to be handled because running Jenkins in Kubernetes is not a trivial task but with the help of the Jenkins operator we hope to help it become at least less of a hustle and more of a pleasant thing to do. Next slide please. So what's on our roadmap. And that actually relates to a question that was asked by Thomas Love about the new release of the Jenkins operator. We are working on a new schema for the Jenkins operator. And it will introduce more granular, granular elements for the components of Jenkins. So, we are migrating from pod to deployment as, as Thomas Love, as per Thomas Love's question. We want to refactor the end to end tests and we also want to be more engaged with the community so we need to establish some processes within the Jenkins operator to help you basically work with us. And also on our horizon is, is there integration with third party systems. And next slide please. So, as I mentioned before, the biggest problem with current version of Jenkins operator is that it only uses one custom resource Jenkins custom resource. And we are actively working on a new API that will allow for it to be multiple custom resources such as Jenkins Jenkins agents. And seed jobs. And because of that, it will, it will unlock many possibilities for future features and, and upgrades to the operator itself. And so, because not every element is dependent on every other element. The updates will be less of a hassle, not everything will have to restart at the same time, but it is also very difficult subject to tackle. Maybe on the right we have been working very hard to actually figure out what the dependencies between between elements are and how, how we need to go about, you know, there are interactions to make it all work seamlessly and have as much as much as possible. And also we are actively getting involved with the community as of this presentation and also actively looking for feedback you can see the link on this slide. If you are using the Jenkins operator or if you want to try it you can send us our feedback, it will be very much appreciated and it will help us shape the future of the Jenkins operator, hopefully in the right way, thanks to your suggestions. Next slide please. A big part of how we are proud to be working in the community is that we have become an official sub project of Jenkins. You can see the also the link on the slide with the announcement on the Jenkins page and we hope to be engaged with the community and hopefully have a very good and beneficial relationship in the foreseeable future. Another thing that we're doing for the community is that we are participating in the Google summer of code which is, if you haven't heard of it, a program from Google that takes some issues from open source projects such as Jenkins or Jenkins operator and issues or, you know, features that need to be added. And it gives some students opportunity to work on those features. And it's, you know, you probably remember how difficult it is to get your first experiences as software engineer so we think that this initiative is great for for students to have their first, you know, real life experience and also help the open source as a part of this project we are implementing a security validator. That will help to for the end user to see security problems with the plugins that they are installed because, you know, if you have, if you have some so many plugins there are so many things that can have some vulnerabilities and it's important for the for the user to be at least aware of them. And you can read more about this project on the link at the bottom of the slide. And next slide please. That's the last slide so thank you very much for being here, everyone. Thanks for co presenting with me thanks to all I can background for hosting and questions that we haven't answered it. Yeah, just getting started. There is a lot of questions. There's one thing I would like to mention. And there will be some t shirts as prizes for the feedback form. And one of them, one of these, basically, I'm not sure if you can see them. Jenkins operator t shirts. So, yeah, yeah, we'll report the link because yeah I'm spamming everyone is links at the moment sorry about that. But yeah, repeating the link to teachers is important enough, that's for sure. The first matter that I tell them but I don't feel like I've answered fully is the Thomas loves question about the new release of the operator. The new daily release that new like mainline release we don't have an official term for it yet will be when we're finished with the new schema when we have tested it and it's good for for you to try out as well. But we have also introduced a nightly builds, which are built daily from the actual state of the repository so any current fixes that we are implementing in the old API are available through the nightly builds and you can check out documentation on how to take advantage of them. Okay, so next question is about the using a single body set of deployment that's what we're working on. And hopefully soon it will be a deployment. Okay, Kubernetes Jenkins operator replace a Jenkins server external. I'm not sure about it. What's the question. I mean, a question from corporate training. I hope I pronounce it more or less right list. Oh, sorry. It was me who kicked that. So, why is Jenkins run a single port instead of a deployment in the operator. I just said that currently, well I don't know why it's because it's, it's been done this way while we were implementing the first version of the operator but we acknowledge there are some some issues some inconveniences that come with it and we're working on having Jenkins as a deployment and not just as a boat so I hope that answers the question well it doesn't answer the why but I hope it's less relevant. If you want to proceed to the results. I did have issues for the operator where you can raise any questions feedback and the race feature request if you have them. So I can share the link later but yeah this is the best way to actually post any request concerns because that's how open source works. Of course. And yeah another note, I shared the link to the feedback form in the chat. So we would appreciate your feedback so that we could make our meetups better. So that is feedback to organizers and also feedback to speakers. And we will appreciate all kinds of feedback and process that and maybe share it with everyone. So please send your feedback from because we really appreciate the feedback and we really process the feedback. Yes feedback is also very much appreciated all the time. It helps to shape the the future and see what can be improved. So maybe Piotr you want to answer some questions from the Q&A. Which one. Well, your choice. Have you tried to run this using github approach from Thomas I don't think we answered this one. But yeah, we basically are trying to enforce github approach with using the operator so that's the advice way to do this. So, which questions haven't been answered yet because it looks like I might be behind on questions. So I have a question about Covernet Jinx operator replace a Jinx external service I believe be answered, right? Yes, I think yeah. Okay. Can configurations code be used on an already running environment. You didn't have the plugin before to migrate over configuration. That's an interesting question. Of course you can install a new plugin. Well basically configuration as code. You can add it to your Jenkins. But I'm not sure if there is a way to export your current configuration as a YAML file that will be compatible with configuration as code. And also, well, what I would advise you to do to maybe try to spin up a test Jenkins instance. You can start with this plugin and gradually try to well in a safe environment and gradually try to migrate your configuration. And well, basically, if something goes wrong, you can always start from the scratch because you won't be touching your main Jenkins instance. And one thing is that you want to keep a recipe on how to get to the desired state of Jenkins. And unfortunately if you have a Jenkins that you just clicked some options in UI it might be a challenge to to migrate it to configuration as code or if you do that, and you spend some some time on it. We do believe that in future it will be worth and it will save you many headaches in the long run. So the next question is about OpenShift. So the question is from Andres, will the creator be available on OpenShift? Well, it will. The idea is, I was with, but with the changes we are currently doing the big API schema change that will be probably released without OpenShift support, but I'm not sure at this point. The OpenShift support will be added eventually. So currently, as it stands, it should be working on OpenShift. But we are mainly focusing on the new API and trying to get on Kubernetes and OpenShift will come later. One thing to mention that actually in Jenkins we have two official operators, because at some point the Red Heart team started implementing their own operator. And basically we ended up this creating a hard fork for that. So currently if you go to the Jenkins CA you can also find the Jenkins automation operator, which is basically being maintained by Red Heart and they have implementation specifically for OpenShift. So as a user you have a choice which project to take, you can take the Jenkins operator, you can take this Jenkins automation operator. And hopefully at some point today we will still converge into a single project, but it depends on many factors. But I just wanted to say that such option is also available. Okay, great. Okay, any other comments or more? Okay, so the next question is Jenkins operator production ready or like for 500 users? Well, I would answer this, maybe like this. It's as production ready as Jenkins itself. I guess that will be a big change. So it's always a good idea to test this first and maybe gradually move your users and basically test it out to see if it really fits your requirements. So, yeah, but, well, I don't see why it wouldn't be. Just bear in mind there will be some API changes. So maybe hold on with the production migration, just a little bit more. And about the scale of usage, well, I don't know if you have actually 500 users working on a single project. And with Jenkins operator you can easily spin up multiple instances of Jenkins. So you can have like one Jenkins per team or however you like to split it. And I think it will solve, if you have some issues with its scalability, I think it will help you to solve it by running just multiple instances of Jenkins and also Jenkins operator allows you to have the agents being scaled automatically using the Kubernetes plugin. So I think it is feasible to have 500 users using Jenkins operator but maybe not like one Jenkins operator. I would advise you to maybe consider splitting your workflows into several smaller Jenkinses. So that has the advantage of, if something actually goes wrong, it only affects like one team that actually broke something instead of all of your 500 users. Yeah, that's actually a very interesting point. Using operator makes it easy to basically spin up multiple Jenkins instances. So it may make your life easier but by well and easier, easier to scale with users. So it's easier to probably manage one small or few small smaller Jenkins instances that one done one huge Jenkins. So it's very horizontal not vertical. That's our approach. Next one is one. Sorry. Actually I had a question about production readiness and the scalability as well. So what I know, VirtusLAB created a product managed service based on the Jenkins operator. Do you have some metrics or the insights from these products in terms of scalability, etc. So what's your experience there? Wow, that's an interesting question. I don't have this data, but we can dig a little bit, maybe ask the right people and we can get back to you later. If you want to quickly speak about this product, it's also perfect to find. Or should we just move this coin? It is then. So, yeah, there is long question about testing the operator. So my suggestion would be to pass on this question until the live Q&A part so later on the record so that you can discuss it. And yeah, the next question is, could you please send us the link to the sample groovy code repository. Okay. I will. I just have to look it up. So maybe you can go to the next question and in the meantime. Thank you so you can just put it to the chat and share with a lot of this. Okay, so that is another question. Can I replace my actual Jenkins server on my visual machine with Jenkins operator in Kubernetes today. Yes, with Jenkins operator you get a fully functional Jenkins alongside with some plugins that we are using for it to run smoothly within the Kubernetes. And there's no reason why you wouldn't. Thank you. And have you ever looked at integrations with Qwirt. So since we are talking about visual machines, maybe it's something that was considered. I don't know anything about it. Hmm. What is the question. Yes, so Qwirt is a project. Well, one of my CNCF projects in incubating. So basically what it does the users Kubernetes device plugin API, and it allows to provide a specific virtual machines as Kubernetes resources, so that you can acquire them. So I had an experiment when Qwirt was used to provision agent of the virtual machine and share between controller instances. So something like that. So I guess the answer is no you haven't looked in that. But yeah, general it's a cool feature it's a. Yeah, we're certainly going to write down and look into it in the future so thanks for the suggestion. So, yeah, I actually have seen one implementation so maybe there will be a presentation later about that. Let's see. So, I am moving on. So, yeah, there is a question I didn't quite get it. So let's turn it to you. Yeah, the next question is from jk if it's a single port and not a deployment the operator, make sure it's running. If I know to this. This part is running colon gold dance for example. So basically what is failover policy for the current operator implementation. Yeah, that's what we'll get we started to put it shortly. So that's right. Yeah, remind it to everyone. We will appreciate your feedback and the feedback form. So, yeah, I'll share the link again. We will appreciate your feedback. Okay, so we still have some time. So the meetup was announced for 60 minutes but we can continue a bit I think is it fine with everyone. Okay, so let's continue a bit and if you have to drop at this point. Thank you and we will publish a recording later today. So, yeah. So, the next question is about notes so can one at custom Jenkins notes. For example, notes which are not containers. I guess it's a bit aligned with what I said about cube dirt. But yeah, what's your opinion on that. Currently, I don't think you can. But it's something that we could think about at some point in the future. The problem is that the agents are contacting Jenkins currently through to Kubernetes services and the services may not be exposed outside. So, as it stands, it won't allow you to connect your agent from the outside. So, yeah, actually, in Jenkins, there are two types of agents that are involved agents and about engines. So in bond is those ones which initiate connection on their own, for example, classic general P agents, though they don't really use general P is standard. And if you are the types of agents like Kubernetes agent, but at the same time, they're also outbound agents. So for example, SSH built agents, my agents and other plugins, for example, CC to plug in allows to connect the engines in the in bound mode over open SSH. So for these agents, you can actually use Jenkins operator because whatever only thing you need is to put configuration for this agent in the configuration is called YAML. And once you put it there when your agent starts up, these agents will be connected and then Jenkins will manage them as outbound. And the agent in this case Jenkins will be connected to them. So even if they're not within containers, but it doesn't cover all use cases for agents and Jenkins. It's still quite, quite handy. Yeah, but the terms of operator there may be a manual workaround possible. Yeah, so you can try to do this and look it up. It may be possible to do this is just not supported by the operator by default. So there may be some manual work involved there. Maybe another example for the future request. So outbound engines. So there are some examples in the Jenkins file runner documentation. And in the Jenkins configuration as code plugin documentation. So basically all these cases can be reused in the Jenkins operator. So of course you wouldn't be running Jenkins file runner. But yeah, the same configuration can be used to and if the plugin is present, you will be able to connect an external agent. Okay, moving on then. And yeah, thanks. So another one. I believe you answered that. Yeah. So, yeah, the question from India does the operator include actions like requesting storage from your cloud provider, the volume from API. And if so, how is it restricted to certain quite product providers. So that's something that's supported natively by the we are basically just making persistent blank claims. So there's there's this handle by Kubernetes. We are just interacting with the Kubernetes API to make those claims and the underlying you know, actual claiming happens by Kubernetes itself so we don't. We just try to leverage as much as much power of Kubernetes as possible and you know not to reinvent the wheel. That's a totally right approach. Another approach which is technically possible in Jenkins and in the Jenkins operator there is a plugable storage set of stories. So I've seen some of them. There is a link on Jenkins. So for example, there is a plugable storage available for artifacts for test results. And again, it's just a configuration and the number of plugins installed so you can use them along with Jenkins operator. So for example, there is actually for managers for S3, which integrates with API's and even some data from agents without passing through the controller. So there are implementations like that and again, they can be used in Jenkins operator. But yeah, I wouldn't recommend that if you can use a Kubernetes native way. Yeah, it's probably important to mention that we want to be portable so we basically want to have a Kubernetes as a requirement and we want to run on different types of Kubernetes clusters. So we have to keep this in mind that, well, other usages in mind also, yeah. And speaking of external storage, from what I know Jenkins operator includes external credential storage by default through Kubernetes secrets. Is it correct. Yes, we are using Kubernetes secrets. And there is a plugin that basically ports Kubernetes secrets into the Jenkins credentials. So yes. Thank you. So it's quite important part and nice that it's supported natively. I'll post a link to this plugin later. So the next question. Is it possible to spin up several Jenkins instances using different configurations of Jenkins. Yeah, I mean, okay. Okay, so I can answer this question. One operator only can control and provision or one instant of Jenkins, however, you can spin up as many operators as you want so you can have multiple instances of Jenkins, but that means multiple instances of the same operator itself, which have an advantage that you that you can use, you know, different namespaces for it and have that level of isolation as well. So, yeah, I hope that answers your questions. Yeah, and it might be something to consider for the future to support more Jenkins instances for one operator. So, yeah, what do we have next. So the next question is about, you mentioned that the one Jenkins controller per team setup, do you have any suggestion on how to manage common resources, most available licenses in our case, and local resource plugin be used and think that cross multiple Jenkins controllers. I can actually provide the answer, but you may not like it. So, no, you cannot use a local resources plugin in such a setup. So we provide BCI which is basically a multi controller instance. And then you may have such issues with provision resource. But there is actually native Kubernetes way for that. I've already mentioned the device plugins and actually device is actually something feasible, a physical. And, for example, when I was working for my previous company, there was no Kubernetes we were actually using another platform, some retention, but the principle was the same. So we're providing a unique resource device in this case license. And when you request an agent, for example, using Kubernetes plugin, you can specify resources you want to get one provision agent, and then you can defer this license as a resource, and you will get a Kubernetes native way to manage resources. It will be the source and Kubernetes will orchestrate access between many Jenkins instances. So it's the approach you could use basically, well, in any Kubernetes based setup, whether it's a Jenkins, separated class Jenkins called BCI or whatever else this approach will work because in this case you will use Kubernetes for resource scheduling. This is something really good for licenses because license quotas are really tough to manage. And if you want to know more about licenses and Kubernetes, I can do a separate presentation about that because I spent quite some time on this one. Great. Thank you. Yeah, and speaking of that, please submit feedback because if you want to know more about such topics, submitting feedback form is the way to do that because then we know about these questions and play the backlog. Okay. So what else do we have. Yeah, any Kubernetes sharing one workspace per job. So I guess support for shared workspaces and other things. Mateusz, would you like to answer this. I believe it is possible to share workspaces you can configure it in your seed job definition and if that's not enough for you. There are some things that you can do with configuration as code to hopefully fit your needs. So that's up to you, but it's possible. Yeah, also, there is a plugin called external workspace manager. This plugin was created during Google Summer of code 2016. The first part of it was to create the APIs and to create an extension of points. And we did that use examples as an FS and other ways to share storages, which is mapping the workspaces. But there was also a full project to actually have native support for Kubernetes and for Docker. This project hasn't been implemented per se, but if someone is interested we have all the APIs so small matter of programming and you can have a Kubernetes native implementation for workspace sharing with an agent. And I guess in this case, there are also solutions based on Kubernetes resources. I'm not 100% sure, but I think it's technically possible. So for example, Tecton does so that it's, it's, it should be possible to implement it in Jenkins as well. Okay, yeah, I'll share the link to workspace manager. So feel free to try it out but yeah for disclaimer it's not fully ready for Kubernetes use cases yet. Okay. Yeah, how will the resource allocation of POTS happen? Well POTS are provisioned by using the Kubernetes plugin. So, well, first of all, you can define POTS templates there. And for example, use labels to decide which job should run where. And of course you can define a resource limits for those POTS. So yeah, and the actual allocation is handled by Kubernetes schedule. Okay, so we have two questions left. One question is, well, both questions are from Guillaume. So I guess it's a single question. So we can take it to the next part of the presentation. So that we can just discuss it just as important as the record. So there are no questions left. I would like to thank the speakers. Thank you for hosting and thanks to everyone who participated. We'll publish the recording later. If you have any questions again there is a operator chat, you will share the link along with the recording you can generate and ask any questions. And if you're interested to stay offline, we'll stop the recording and have an open discussion. Thank you so much everyone. Thank you as well. Okay, thank you.