 Thanks for coming to the Jenkins online meetup. Today we have a special guest, Marty Jackson. He will be presenting the first talk about the Jenkins and Kubernetes and he will show how to get started with it. If you have never participated in Jenkins online meetup, it's an online platform we use in the Jenkins community. We have several dozen meetups around the world, but we also try to organize online events, especially these months. For that, we have a special meetup. This meetup is basically organized by contributors. Everybody there volunteers their own time and we try to do our best. We talk about developer tools, about best practices, about using Jenkins everywhere. The main objective is to show and tell. We want to have presentations. We want to have marketing features. Instead of that, we want to focus on real experience and we invite everyone to contribute by comments or by joining next meetups. Again, the topic today is about Jenkins and Kubernetes. I'm not sure how many of you already use Kubernetes, but it definitely gets more and more popular. In the Jenkins project, of course, we want to ensure that Jenkins runs mostly on Kubernetes and that Jenkins is a part of Kubernetes ecosystem. There is a lot of ongoing work including road map, where you want to include more topics. Of course, we also want to talk about what happens and how to use Jenkins and Kubernetes. Basically, we're interested to talk about anything. If you're interested to share your work stories, key studies or show how to automate your flow service, Jenkins and Kubernetes, we are welcome such discussions. I use the automation world, but it may also mean CI, CD, DevOps, whatever. Jenkins can help everywhere. If you want to do some specific talks or if you want to get more information, for example, for plugins, for account charts, for Jenkins Kubernetes separator, it's a separate project which is available or just for integration with different Kubernetes tools or cloud providers which use Kubernetes please do so. For all these topics, we are going to have meetups. We are actually planning a series of these meetups. Right now, there is around five and we are looking for more. If you're interested to present your experience, just let us know. There is adoption. We share a channel and there is also Jenkins CI channel in the Kubernetes slide. You can use one of these channels to contact us and you can talk about presentations. Or if you want to request a particular topic, we are also happy to help and work together for options. That's it from me. Today we have Marky who will present how to get started on Kubernetes. Marky is a negative contributor in the Jenkins community. He is also a really active contributor in the Kubernetes community. Of course, it's a pleasure to have him at this meetup. I believe Marky will present himself later with more details. If you have any questions during the presentation, there is a Zoom chat. Please ask questions there. After the presentation, we will ask Marky these questions. If you want to ask anything offline, if you want to see this video on YouTube, also please use a YouTube channel or Kubernetes slide. We will be monitoring all these channels as well. That's it from me. Thanks a lot for being with me. Marky, all the photos yours. All right, everybody, good morning, good afternoon, good evening, wherever you are. I hope everybody can hear me okay. I am going to start for anyone that give me just a second here. Hopefully everybody can see the presentation. Today we are going to be discussing Jenkins on Kubernetes. The way I do my talks is I try to make them as fun as possible. Mainly because I am nervous and cracking a joke when you are nervous is always a good thing. Without further ado, first and foremost, I want to thank everybody for taking the time to listen to me ramble. If you could just hold your questions until the end, I will do my best to answer the questions. I am going to go ahead and get started. For those that don't know me, my name is Marky Jackson. I am a software engineer working on open source integrations with Anchor. I am a Jenkins core maintainer as well as a plugin maintainer for many plugins, the Kubernetes plugin, the Prometheus plugin and many more. I am a Kubernetes org member. I work with the Contrabex and most recently I was made a release engineer associate. You can find me on GitHub at Marky Jackson dash Talia or on Twitter at Marky Jackson five. Also at LinkedIn, I am not going to go ahead and read that all. For those that don't know me, one of the things I would like to make very known is I love to answer people's questions. It makes me feel good to help people. If you have a question that you don't want to ask here or don't feel comfortable asking in a public setting and you want to do it privately, my messages are open on Twitter and you are more than welcome to message me. I try not to do a lot of the help desk things. I expect that everybody has done a level of research before you ask a question. But if you want to just chat and say hi, I am a super friendly person. For those that don't know me, we will say I am a friendly person. What are we going to talk about today? The presentation is going to cover a light. I specify a very light overview of Kubernetes and the Kubernetes plugin. The reason I say a very light, I won't be going into the real deep technical weeds of what Kubernetes is or what Jenkins is. I will cover how to set up and configure Jenkins on Kubernetes. I will show you how to use the Kubernetes plugin. I will do a code walkthrough that will show you some of the YAMLs that I will be using for this. I will do a demo of that and then we will close up with some questions and answers. So let's go ahead and get started. What is Kubernetes? Everybody knows what I would hope what Kubernetes is. It is a pretty hot topic in the tech sectors. But Kubernetes or K8S, as some people call it, it is a project that was spun out of Google. It was an open source next generation container schedule that was designed with lessons learned from developing and managing a project internally at Google called Borg and Omega. Essentially what that does is it abstracts away the layers of hardware of the nodes and provides a uniform interface for application and scheduling of resources in a shared pool. Why would you want to put Jenkins on Kubernetes or use some of the agent ability for scalability in Kubernetes? Like I just said, scalability, containerization, and also infrastructure as code. One of the things that I didn't mention in here that's really critical to understand is when you think about Kubernetes and Jenkins, one of the ideas here is a cost saving. So putting Jenkins in Kubernetes also is a benefit and a cost savings. So the Kubernetes plugin, I am one of the maintainers. The gentleman named Carlos Sanchez wrote that plugin, which basically allows the running of dynamic agents in a Kubernetes cluster. By doing that, what you do is you lower the cost of having to spin up agents everywhere. It helps you with the ability to set that up as infrastructure as code. And then I've also included a link to where that repo for that plugin is. I will also say that I won't go into the real deep technical ways that you can use this plugin. I will keep it at a very high level just for running dynamic agents in Kubernetes. But there's all kinds of different ways that you have the ability to use the Kubernetes plugin, such as setting up POD YAML templates and all kinds of things. If you look at the read me on the plugin, there's all kinds of examples. So before we get into the demo, which is what the crux of this is going to be, I want to talk about some assumptions in this demo. Number one, in order to run this demo, you have to have a Kubernetes cluster. You have to have the cube CTL, which is the command line interface that allows you to communicate with that Kubernetes cluster. You have to have the code. I've included a code repo here where I have made it public, the code that I will be using here. For this particular demo, I'm going to be using mini cube and it'll be running on the latest 1.18 version. You do have the ability to run this on any type of cluster. An example in the Kubernetes community, when we do a lot of our testing, we use kind, which is Kubernetes and Docker. I've included, if you install the kind platform, there's a command on how you can create a 1.18 cluster. And then you can run through this demo. So I am going to get started into the demo. I will go ahead and give me just a second as I move through some of these screens here. So I want to talk about, before we get into the code walkthrough, which is right here, I'd like to talk about a few things that we'll be using in here and I want to give them a little bit of attention before we start. The first thing is the Docker hub. I've included in my Docker repo the image that we'll be using for the deployment. Yeah. I'll also... Okay, it starts. Sorry. Yeah, I'm here. Yeah, just a second. I believe that I needed everyone else. Did we lock the meeting room? Yes, lock it please. Yeah, yeah, that would be great if you could lock the meeting room. Okay, just a second. Okay, locked. Thank you. One of the things we'll be talking about is the Jenkins image that we'll be using for our agents. Now, I wanted to talk a little bit about this in depth because there's a lot of work that's going on around this. As you know, there has been a lot of commentary, a lot of pool requests in removing some of the terminology that's used in Jenkins. This is the repo that we'll be using for our template. I'd like to make a note in this particular section. The terminology that was previously used is now been deprecated. The new terminology we're using is... Okay, so yeah, that's why we shouldn't post meetings next time. Yes, we're getting zoom bombed. I will have a lot of work to clean up this video. Yeah, sorry all, but you posted the zoom link somewhere in public and I guess we're getting it now. Okay, so one of the things that has been done here is changing of the terminology. So a lot of the images that have been previously used have been deprecated in terms of naming and the new naming is now the inbound agent. Across the Jenkins ecosystem, there's a lot of work that's also where people are changing the terminology. I'm really proud of the work that the entire community has done around this and I've noticed it's happening a lot in other open source projects. So one of the things that I will strongly ask people to do is when you're posting comments in chats, try to be mindful and empathetic of the language that you're using and the language that the Jenkins community is using. So with that, let's go ahead and get into a little bit of the code. I am going to share my editor, which is a little bit easier to do. The base image is a very simple image. This is not meant to be an image that you use in production. I've added some comments into the image here that things to be cleaned up. These would be great first issues that I'd love to help mentor people. If you wanted to get involved and try to understand something, I've created some comments in here for cleaning some of this up. I'm going to spend a little bit of time on the R-backy ammo and the reason being is because I want to talk about some of the pieces that are involved here. So we're doing three things in this chunk of code. Number one, we're creating a service account. Very, very simple. That service account is named Jenkins. The second thing we're doing is we're creating a cluster role. And in that cluster role, we're defining what resources we'd like this service account to have access to. And finally, we're saying in the cluster role, finally, we're saying what verbs are allowed for the given resources. We then create a cluster role binding, which binds that service account to this cluster role. Now, the reason I wanted to spend some time in discussing this is because a lot of people may be confused about how R-back is set up. So there's two types of R-back. There is what you would call a role or a cluster role. When you define something at the cluster role, this is at the root level of the cluster. For development work, this may be good to do because it helps you get around things quicker. For production, it is not advised to set things at a cluster role. The second way you have to do that is to set a role and a role binding, which allow you to do something at the namespace level. But again, for this particular demo, I'm setting everything at a cluster role. We then have our service and there's our service. Our service is pretty straightforward. I will cover this one part and that is for this demo, I am using a node port. A node port is good for quick and easy development. A node port is not what you should use in production. So please note that this demo is for development only. You should not use this in production. Finally, we're covering the deployment. The deployment is a very straightforward piece of work here. It basically will call our image. It will set some Java arguments and then we're going to set a volume, which is locally an empty directory. Again, this is only for development. I would not use this in production. So could you please increase the font size a bit? Oh, I'm sorry. Let's see here. I can't increase the font size. If you can't see this in the recording, the code is available to walk through. And if you have any questions about it, actually, let me bring it up this way. Is that a little bit better? Yeah, that's great, Marky. Thank you. You're welcome. I apologize for the small text. So a very simple straightforward deployment. I will stress that again, this is only, I know you keep hearing me say this is only for development and not production. The reason that I'm saying that is there's a sometimes a tendency to say, oh, this works great in my local testing. I'm just going to move this straight to production for my company. I have seen that happen. This is not the code you should be putting in production. So with that, I am going to get started. So I have a mini cube cluster that I've already launched. What I'm going to do is I'm going to get started. The first thing that I want to do is I want to just go ahead and create a namespace. I want to call that namespace Jenkins. Again, it will be great to increase the fun from it. Is that better? Let me just clear my screen and it should come back up. Is that bigger? Can everybody see that? It's fine. So what we're going to do is we're going to create a namespace and we're going to call that namespace Jenkins. The next thing that we want to do is we want to go ahead and deploy our RBAC configuration to the namespace that we created called Jenkins. And we'll see what this has done for us is it's created the service account, it's created our cluster role, and it's created our cluster role binding. The next thing we're going to go ahead and do is we're going to go ahead and deploy our service. We can see that the service was created. And finally, what we're going to do is we're going to create the deployment. Again, we're always doing this in the namespace. I'd like to point that out that it's always good to ensure that you use the namespace command. So by default, if you don't use the namespace command, what this will do is it'll put it in the default namespace and you don't want your applications running in the default namespace. And we can see that that has been created. So let's go ahead and look at those pods get created. And the reason that I'm leaving this up on the screen is because I cannot go to the UI until I see that the pod has been created. And now we see that it's running. So we will go ahead and move over to the UI. Missed one little piece here. One of the things that you're going to need to do is you're going to need to get the mini cube IP. And this will tell you the IP that your local mini cube is running on. You'll need that to be able to get into the cluster. I actually missed another piece as well. I apologize. One of the things I showed you in that deployment was we were using a node port. So what do you have to do here is you're going to need to get the actual node port. And you can do that by running a cube CTL get service. It'll show you the Jenkins service that we created. The type is node port. And you'll see this actual node port. This is the port you will use to connect to your Jenkins instance. And just like that, we now have Jenkins running in Kubernetes. So we're going to move to the next part of the demo. One of the things that I wanted to show you in the code that we've created in the Docker file, I have already pre-installed some plugins. And one of those plugins you'll see I've installed is I've already installed the Kubernetes plugin. So we're going to go ahead and get Kubernetes plugin configured. Recently that there was a change made to the plugin. It is no longer in the global configuration. So for this plugin, you'll come into the build executor status. And you'll see configure cloud. This is where you will start that configuration. So we're going to go ahead and add a new cloud, which will be Kubernetes. We're going to configure this. The first thing that we need to do is we need to add the Kubernetes URL. For the Kubernetes URL, we're going to go ahead and run a command, which is kubectl, cluster-info. We're going to grep for master. This will tell us the port that the IP and port that the master is running on. And we're just going to drop that right there. And then we want to test the connection. We see that the connection is successful. The next thing we need to do is for the communication in the Kubernetes cluster to be able for the Jenkins control plane to connect to the Jenkins URL, we're going to need to go ahead and get the actual IP of the pod that is running Jenkins. So that will just be a few commands. One, we're going to get the pod. We'll copy the pod name. We will describe the pod. And in here, you'll see there's the IP. So we will copy this IP going back and 8080. So we're awesome there. We now have our Jenkins set to communicate within the Kubernetes cluster. I'll just go ahead and apply that. And we'll move to the next thing, which will be configuring a pod template. We'll add the pod template. For this particular one, I am going to give it a name of inbound agent. We'll set up the details. So we know we created this in a namespace called Jenkins. We're going to need to give the agent an actual label. In this particular demo, we're just going to call this inbound agent. We need to create the container. So this is the part where I explained in the Jenkins Docker Hub where we will be taking the inbound agent. And that will be for, we'll be using that image. So I'm going to go ahead and give this a name. We know that the Docker image is coming from Docker Hub. And it's from Jenkins forward slash inbound agent. We're going to just do a little bit of configuration here to make sure we have some things set as we want them. So you have the ability with this Kubernetes plugin to set request and limits. You can have liveliness probes. You can have this set to run as privilege mode or to run as a specific user. You also have a lot more flexibility. You can set the YAML for the pod. Something that I'm going to go ahead and do is we know we created a service account called Jenkins. So I want to make sure that this runs with that service account. I finally want to just make sure that I set every agent to terminate and give me just a second, which is right here. And what this allows you to do is when the Kubernetes plugin launches an agent, you have the ability to say, keep that agent or to terminate the agent once the job is done. For cost savings, I want to make sure that I'm terminating these things so they're never running. And for that, we are done making our configuration. So I will go ahead and apply. I will save this. And I'm going to go ahead and now build up a new job. So I'm going to do a very generic job here. I'm going to call this demo. I'm going to use a freestyle project. You do have the ability to use pipelines for this. For this particular demo, I am not going to use pipeline. For future demos, we do have some coming up in May, one where I will be doing. And for subsequent demos, everything will be in a pipeline. For this demo, I just wanted to keep it as generic as possible. So we come in. The first thing that we're going to want to do is we want to make sure we're restricting where we actually have this job labeled. And we know we created this called inbound agent. So we'll move there. And I'm just going to make a very basic job. And I'm going to just say echo hello world as soon as I can spell. And it's very straightforward. We will save this. I'm going to turn auto refresh on. And I am going to build this job. The first thing you'll notice when I built this job that it's sitting and it's waiting. So let's come back here and let's watch that agent get created. So you can see we have now created the inbound agent. This is launched in Kubernetes in the same namespace. The pod is creating. We'll give that a few more minutes. We can now see that the pod is running. And if we go back to our job, we will see shortly that the job is now running. We can see it has run the hello world. It used a default YAML for the template. And we can see that the job was a success. Now, if we go back to our terminal, we can see that that same pod that was spun up for the job has now terminated. The beauty of that is is you can now paralyze builds. You can save costs. So I'm going to go back to my slides because that sort of ends the demo. So what did we learn here? Kubernetes allows for a robust way to provide Jenkins scalability. It allows the ability to run many more build clients in parallel. It also allows the ability to automatically replace or replace corrupted Jenkins instances and to spin up and remove agents based on need, which in turn saves cost. So that concludes the demo. I would like to move to the question and answer question. I'm going to stop sharing my screen so I can see the chat because I do see a lot of notes in chat. Yeah, while you were speaking, I was also capturing questions. So I will share Google Doc with everybody in the chat. The way we have questions which haven't been answered. If it's more convenient for your market. Yeah, that's totally fine. Okay. So if so, let's start from the beginning. Oh, it's so nice to see anonymous Goose, the Kubernetes related meetup. Who said that? They get a honk for that. Okay. So what's next for meetups? It's a question I got in private and I'll briefly answer that if you don't mind. So we from a series of meetups right now we are talking with speakers. We have some reviews. In general, we want to have one meetup maybe every two weeks. We have one meetup announced for next Tuesday. It will be about this configuration as code, this job design for jobs management, and this health brain, which was mentioned in the chat multiple times. So stay tuned for the next meetups and we will be increasing complexity. We will be doing deep dives during the next meetups. So there will be a lot of interesting content there, we hope. Okay. I would like to extend my apologies for the Zoom bombs there. I apologize deeply for that. For the video, we will edit that out. Again, my apologies to everybody for that. The world is an interesting place currently and you shouldn't have been subjected to that. Right. We've been using Zoom for two years or more and this is the first time we actually got the Zoom bomb. Yeah. Okay. So let's go next to the questions. How is it different from Jenkins X? So this is Jenkins X allows for a very get ops nature. When I say get ops, you can actually create your pipeline directly from a get repo. This is more native, very different. Jenkins X also offers the ability to spin up the cluster and in some ways that's a super awesome. But what I meant for this demo was to be a very native, no tools underlying other than really many cube being used. But you can do this with Jenkins X. There is that ability. Hopefully that answers the individual's question. Many of those like from the ministry and governance point. So Jenkins X right now is a separate project. It has been built around Kubernetes ecosystem to continue delivery in Kubernetes environment. So he, this project targets this specific use case. Jenkins is a modular automation server. It can do a lot of different things. And you can also run it in Kubernetes, but the use cases at the moment are different. So you can probably also do continuous delivery with Jenkins and Kubernetes. But at the same time, you will have to configure a lot on your own while Jenkins X provides many features out of the box. That's correct. The purpose of this was to be more native in our delivery of the underlying platform as well as the code that we were deploying. We didn't want to use any special tools to do that. Yeah, right. The next question is about documentation. So will these steps be documented somewhere? So unfortunately, I did not put these steps in the readme. However, I promised to have that out to you in the next 24 hours. The repo will readme will be updated with the steps for what I actually did. Okay. What do you define if you hint the roadmap for a second? Yeah, that's perfectly fine. Okay. So you may have noticed some conversations in the Jenkins community that we actually worked on the public roadmap for the project. So this roadmap is, it's just a draft. It lacks a lot of items and we will be putting more content specific for Kubernetes today. But one topic about the documentation that documentation is also on our roadmap. And right now, the process of discussing items we would like to put there. And no cheating. If you open the mailing list associated with this thread, you can see that there is a special roadmap item related to Jenkins and Kubernetes documentation, which would include a lot of solution pages, case studies, etc. And we really intend to document that and we invite any one of those interested to contribute in this area and to improve Jenkins documentation for Kubernetes. Because yes, there is a lot of things to improve and it will be great to do it together. Well, and already on your screen highlights that Victor Farsic's DevOps toolkit 2.4 book, which is Kubernetes and Jenkins has been open sourced, right? So we've got good resources to begin. Yep. And probably this is the only book in the world which has been afforded by Helen Charles. I'm the same as to something. Yeah. Okay. So it's the next question. Okay, let's go on. Why did we choose the deployment instead of stateful set? That is an awesome question. And I would, super awesome question. So why did I choose a deployment? This was mini cube and a deployment really works well with mini cube. Why would you use a stateful set? If you're doing a production Jenkins deployment, you would want a staple set and not a deployment. So why did I choose a deployment? Because this was just mini cube based. If when we do future demos and I actually show spinning up a cloud environment with Kubernetes for Jenkins, I will not use a deployment. I will use a staple set. I have the code available already for that particular demo where it will be a stateful set and ingress controller, ingress object and things of that nature. Very good question. But again, for development, I just chose to use a very simple for mini cube deployment, Ammo. Thank you. Another question is about what will happen on port response for a demo. Can you repeat that? What will happen on the port response for your demo and specifically what will happen is agents and how to manage agents in such case. So you'll notice when I did the configuration, I had the pod retention and I set that to terminate. So what that means is that the pod will be launched, the job will be run on that, and then it will terminate that. It will not respond. You do have the ability to say in the pod retention configuration that I'd like this pod to stay and not be terminated. And then you can run multiple jobs on there. But I felt that for the demo purpose, having the pod go away once I was done served a more better purpose than leaving it there. But you do have that ability in the configuration. Thank you. And the next question is Jenkins inbound agent tag and multiple question marks after that. Can you repeat that again? Jenkins? Jenkins inbound agent. Just a second. So this one. So what does it actually mean? So what's highlighted on the screen here is the actual location of the Jenkins inbound agent. So what I'm using in my configuration is the location for that. I chose to name everything for the name of the pod template as well as the label inbound agent to sort of keep everything unified. However, you can choose whatever you'd like. The only thing that you can't, you should not delineate from is the image that you use for the pod template should come from the repo that is on the screen right here. And why is that? That has the JNLP remoting capability in there, which allows the agent to talk back to Jenkins. And you need that communication. So that's already built into this. That's why we use this image. Yeah. So if you're confused about the naming, actually it's a change which was delivered just in the beginning of this week. We're renaming official images in order to clean up terminology. So yeah, this is the same official image as before, but under the different name. Awesome. Okay. So the next question. So when starting agent, it says that there are zero of two pods being created. So what does it mean and why there are two pods? So I create two pods just so I always have an auxiliary backup pod. If the first one should fail for whatever reason, even though logically it shouldn't, because they're the same, you can configure that to just say one pod. Okay. Good. And thank you. Next question is, since we're giving authority to Jenkins to launch my mesh, delete pods in the cluster, what steps are needed to keep it secure? Yeah. Very, very good question. So the first thing that you want to do is create a service account. As you'll notice, I created that service account. And then on the second part, I created resources where I allowed what that service account can do. So for my demo, as I explained, I did everything at the cluster level role. If you want to be more secure, you should never do anything at the cluster level role. You should always do it at the namespace. That way it doesn't have the ability to affect any other namespace in that cluster. That's one piece of it. The second thing is having authentication set. When you talk about RBAC, I created a service account and that's internal to the cluster. It's best to, if you're going to use internal authentication, that you separate everybody, every service account should be individual. And the communication within that RBAC for your cluster or your cluster role should be specific that you know what it's doing. Finally, I'll say in terms of cluster management from a security standpoint, having auditing turned on on the CUBE API, so you're able to see not only, so you have your RBAC, you know what you're allowing people to do, but to also be able to log and monitor what people are doing. And to do that, you want to have the auditing turned on. And in future demos, I can show how to turn that on. It's really, really easy to do. And then you can use a whole set of tools to be able to log the CUBE APIs to see what auditing, like to see what people are actually doing. Are they trying to create a namespace even though they don't have the ability? There's also other open source tools out there that will alert on such things like that. Example is Bowco is a tool that will alert when certain CUBE auditing is been hit. Hopefully that answers your question. Yeah, thank you. There are also other questions about security. So if something wasn't answered, please follow up in the chat. Okay. And the last question we had in the list before we go to the chat, if my pipeline triggers other jobs and the way it's finished, that means that I will need two different ports. Is it possible to configure builds to use the same port? Yes. So in that late, in the configuration of the pod template, you can specify that I'm going to have sort of serialized, I'm going to daisy change jobs together. And in my example, one of the questions was asked why I used two pods as opposed to one. There's another good example of why you would use two pods because you may want two jobs to run on two separate pods. So you do have the ability to do that. Okay. Thank you. So let's go to the chat questions. Yeah. So again, restarting ports. What is your recommendation for management for managing the persistent data on the master? Can the master use replicas and use the data? For this particular example, I used an empty directory, which it's basically putting it in, I think I specified slash 10. So you have the ability to do that. If you're using a cloud, let's say you're setting your Kubernetes cluster in a cloud infrastructure and you have Jenkins running in there, you're going to want to set that up, say for AWS, you're going to want that on some persistent volume and that persistent volume ties back to an EBS volume. And then their data is getting saved on the EBS volume. So for example, if you're Jenkins, I don't call it a master, I call it a control plane. So if your Jenkins control plane goes down, and then if you're using a stateful set, it spawns another instance, the configuration is already there, and then it'll automatically create that persistent volume claim, which is tied to for AWS. The same is true in Google. It will automatically bind that persistent volume claim back with the pod from the stateful set, and then will automatically connect it to your back end volume, which in AWS, it's being EBS. Thank you. And if you need to share your screen to show how it works, please don't hesitate to do that. Okay. Is it possible to specify a max number of pods? Or is it unbounded? You can set the max number that you'd like. You can be as many as one, you can do as minimum as one, or you can set, I only want 50 pods. I've worked in some organizations where we have jobs that will spawn many, many more jobs. And with that, I think at one particular time we had one that would spawn something like 400 different jobs. And we have that ability using the Kubernetes plugin. Crazy, but it's available. One of the things that you have to be, you have to be mindful of when you're spawning agents in Kubernetes, you have to make sure that you're not taxing the cluster. What do I mean by that? If you only have so much memory and CPU on that cluster, but you're spawning 15 parallel jobs that are running some machine learning pipeline, you could bring down the cluster. So it really benefits you to understand what you're trying to do in terms of the overall cluster. So with that, I think it's always good to have request and limits. And I would also strongly suggest that anybody that's doing this and thinking about doing it in production, I would think about not only a namespace-level request and limits, but setting a cluster-wide request and limits. That way you know that somebody's using a namespace and they're spawning a massive amount, they're not going to bring the whole cluster down. And a lot of these are best practices. And if you've not done it in production, it may seem sort of weird. If you ever have questions, I could talk about this all day long. So please, I'm a very collaborative person. Don't feel that you can't ping me. Yeah, the right chat's where you can reach out to Marky right after the talk. So Jenkins Gitter or Kubernetes Slack for Jenkins. Both channels, I believe Marky is everywhere in any channel if I've been Jenkins. Okay, next question. If you would like to offer to developers of the freedom to choose the image to be used to run their pipelines, let's say Python 3, what is, so then there is no other way than to build your own image on the top of the Jenkins agent. Is it right or are there other ways? So I want to be careful when I answer this question because I'm one of those people that likes to tinker with things. I will say the ability is there to do it. Yes, you can. Should you do that for production? No, don't do that for production. If you want to try starting off with a dev environment and making sure that you've ironed out every problem, I would start there. Why am I so hesitant to say, yeah, go ahead, just do it because the inbound agent Docker image has been tested with Jenkins connectivity in mind. So that's all it worries about. When you start to try to do something different in there, the ability to go, I call it going south or go astray is high. And you may not get the same level of support if you were just using the base image from the open source community. But it's definitely, I don't want to discourage anybody from trying it because I will say I do it all the time. But if you're not, I also know a lot of the underlying code so I can trace a bug faster. Yeah, one thing which was mentioned, if you don't want to extend the Jenkins official images on your own, there is also repository Jenkins slash JLP agents. And there is a number of images inside and one of the image since we were talking about Python. So here's the image for Python 3 and Python 2. And you can see that this image is, well, we will need to clean up the name and but it's actually based on the official inbound agent and it just installs additional tools. And that work is in flight. That work is in flight. Yeah, we will clean it up. But yeah, some of the images are already available. And if there are common use cases, we could provide official images in the community for that. And if not, you can easily extend them because it's quite easier to do in Docker. Am I still skin shank? Yes. Okay, because, okay, Zom was saying that I'm not and I was surprised. Okay. So yeah, then the next question, we want to migrate our visual machine Jenkins data to continue various Jenkins control point in Kubernetes, but we are facing some issues with when mounting the volume with the data on the Jenkins home. Any advice how to do that? Carefully. So when you do the migration, what I would strongly suggest doing is what I've done in other companies is I write a script to do the migration. I don't manually do it, especially when it comes to I've dealt with a lot of doing EBS volume migrations. And that can be super super complex. It's best to control it with a script. What, how would you do that script? I have found personally that if I write it in Python, especially using AWS, I can use the photo three library, it makes it more seamless, your flexibility is there. It is doable. I would strongly suggest writing a script to do it. Okay, thank you. So we have something like four or five questions left. I suggest we go through them. Yeah, nothing. So can we have performance problem or to have the master in a doctor? You can. Yes, depending on how the the control plane is configured. One of the things that I would strongly suggest doing is if you're going to move Jenkins into Kubernetes, that a you know exactly what your request and limits are going to be on Jenkins. On Kubernetes. So understanding what the jobs are that you're going to run. How you you really have to start doing a tree of decisions on how you set your request and limits up. But I have seen it become a problem. The problem has been rectified by setting good request and limits. And the next question, how do manage charging is backup some looks. That's a great question. And it's one that a lot of people struggle with. So I will tell you the different ways that I've done it production wise. I've I used three different ways. The first way would be having a general sort of script. And the script will just go in a, you know, copy up a full directory, tire it up and put it somewhere. That's one way to do it. The second way to do it is there's a lot of plugins that you can use. One of the most widely used plugins is a plugin called thin backup. And then so then backup gives you the ability to set a chron like schedule and control what you want to back up. You may just want to back up your jobs directory. You may just want to back up your jobs directory in your credentials file. You may want to back up the whole ball of the whole thing. And that you have that flexibility to use the thin backup. I have used the thin backup, especially in conjunctions with S3. So you can write a script that will, you know, the job runs on thin backup, you'll have a script that will go grab that what so how to send back up. Let me take a step back. Then backup will go based off of your configuration and your schedule and will grab everything and zip it up. Once it's done with the zipping, it will actually then after a certain amount of time will tire it up and make it an archive. You can then clean that up. So what I have done in the past is I had a script that will go through. It knew the schedule to run. So if I said I'm going to take a backup every day at 1 a.m., it knew I'm going to run every day at 1.15 a.m. I'll take that zipped file. I'll tar it and I'll put it in S3. And then I'll send out an email saying your backup was successful. If the job fails for some reason, I had a way that it would notify me, hey, your backup failed. And then, you know, we could then have the ops team go look and say, hey, why did this fail? There is a third way. The third way is using there is a, especially if you're in Kubernetes. Again, these are all open source tools that you can use. But if you wanted to back up your Kubernetes, there is a tool out there called Valero. And Valero will allow you to back up full namespace items. That includes EBS volumes. Let's say if you're using AWS. So that's a cool tool. Yep, there you go. Valero back up. This was a tool. I forget who put this tool out originally. I want to say it was Heptio, but I always messed that up. But I use this. I can tell you that one of my good friends, Seth, he uses this. He's the one that turned me onto it. Super awesome tool. And this, you know, when you start thinking about Kubernetes, now you have that ability. Thank you. Okay, so there is a lot of positive feedback coming in. And we still have several minutes to close the questions. The next one is about running in EKS and using volumes there. So one volume is, okay. So one storage claim is created towards my Azure vendor. How do I track the file system and its usage? So let's see if I understand. I missed one of the volume, but I raised the animals in my EKS cluster. It means it created a storage claim towards that. Okay, so one of the things that I would suggest not doing is using this demo for a cloud, because this is more meant for local development. It's not meant to use what you're referring to, Mira, I believe. It's called a volume claim template. I have code and I was going to save it for another presentation. But essentially what happens is, one of the earlier questions is why did you use a deployment over a stateful set? In a stateful set, when I would build something in a cloud, the stateful set gives me the ability to use something called a volume claim template. And what the volume claim template will now do is what you're saying, Mira, it will now track your file system. And then when that pod goes away, let's say your control plane for Jenkins goes away and responds, now it knows that when that stateful set is tied to a volume claim template, it will know to automatically go find the last volume claim template it knows it had and reconnect it. Now if you delete the volume claim template, well then you've got a lot more bigger problems. But to answer your question, Mira, what you want to use is for that type of a deployment is a stateful set that has a volume claim template set up with that. I don't want to put the code out there yet because I feel giving the code without explanation will create more questions than I think should, you know, it would be fair to the person wanting to use it. I will say that in a very soon presentation coming up, I will be doing this very same demo, but on a cloud infrastructure that will use a stateful set with a volume claim template. And, Mira, if you want to ping me offline, I can help walk you through setting that up. I'd be more than happy to do that to get you sort of moving in the right direction. Thank you. And I guess the last question is about the agents again. Isn't it better to just add another container to the port instead of customizing about the agent image? I think what you're referring to, Boris, is sort of like a sidecar. You can do that. Personally, I don't think there's anything wrong with it. One of the things that I'll say that may create complexity is now you have to track how that sidecar is going to communicate all the permissions through that. It also becomes an attack vector. So, if you have that sidecar running and it has certain level privileges, that could be a problem. You've now allowed that. In a production level environment, I would say that sidecars are extremely beneficial if they are built in such a way that security was in the forefront of your design. And oftentimes, that's not the case. Hopefully, that answered your question, Boris. Thanks for this answer. So, yeah, there is a lot of other comments. Do you want to take one more question? Yeah, we could take a few more. Tommy, hold on. I saw you turn your camera on and I want to answer your question, but I just want to get to one person's before yours and I'm just trying to find it. Bala asked, is there a way to recover a deleted job in Jenkins? So, the beauty of recovering a deleted job is that config.xml. If you still have the volume available, you then have the ability to recover that deleted job. It would mean that you would need to do sort of a restore. I've seen various ways to do this, but one of the most easiest ways to do it would say, my EBS volume, it has my jobs directory, which has the config.xml, even the next build number and all of that things, all you would need to do is there's a kubectl command that will allow you to copy over. You can do it that way. You can also, if you have a backup, you can restore it from a backup. So, hopefully that answered it. It is possible, Bala. Let me tell me your next. I want to just read your question. Your question was, will this be published? Is that right? Yeah. Okay. Yes, this will be published. Obviously, we have to go through and do some clever editing because we got Zoom bombed, but yes, this will be published to the Jenkins YouTube. I think we have a jam playlist, but if you go to YouTube, you look up Jenkins, you'll see a actual subscription, an actual link to the Jenkins that I have a playlist that you can subscribe to, and this will be there. It'll probably, I would say, give me at least 24 hours between Oleg and I. We're going to have to do some clever editing, but it will be published definitely by the end of the weekend. Yeah. So, yeah, you can see my terrible video choices right now, but what I wanted to show you, there is official channel, and there is a lot of playlists the way you publish actually everything. So, not only online meetups, we also have special interest Zoom meetings and other topics. So, if you want to participate in wider community and see what happens, you see some demos or work in progress stuff, you can go here, and here's Jenkins online meetup playlist, and be sure we will get it posted soon. I think we've gotten everyone's questions. Before you close it out, Oleg, I want to thank everybody that joined this. It's, like I said in the beginning of my introduction, I am a super collaborative person. I love to help people because I feel that I don't know everything, and there may be a different way to do things, and when two people or a group of people are collaborating on something, and everybody's learning, and there's no, you don't feel like, oh, I'm not that smart, or I don't believe in that. When people learn together, it's an awesome feeling like I'm getting goosebumps right now, but it's an awesome thing to be able to learn together, and in a way that we're empathetic to each other, and we're just being friendly and human to each other. In this day and age, we need that more than ever. So, please don't hesitate to ping me, to say hi to me. When conferences start again, if you ever see me come up, I'm a very approachable person. I will say I'm super identifiable because I'm the only guy that's tattooed all over the place, so don't think I'm scary or I'm, please talk to me, message me. I love to hear from people. Thank you for allowing me into your homes, living rooms, offices, whatever. That's just a beautiful thing, and I can thank you guys enough, and girls, and everybody. I don't mean to. Thank you. Thank you, Tomarke. Cheers, everybody. Be safe out there. Be happy. Love your neighbors. Yeah, thank you. So, just to whom I was interested to see more content about Jenkins and Kubernetes, the next step is actually next Tuesday, 4 p.m. UTC, and we will have Nikolai, who will talk about running Jenkins fully configured test code, use jobless, Jcast, and Helm3, and you're more than welcome to join this session. So, looking forward to meet you there. Yeah. Thanks all. I will stop the recording with this button.