 I am so thrilled to be here. Thank you very much for having me. So this talk is called configuration as code of Jenkins for Kubernetes. My name is Nikolaj Grasholt. I am a DevOps consultant. I work for Efficode, previously Pragma. I help our customers with Jenkins and I do Docker and Kubernetes trainings and I'm a certified Kubernetes administrator and I absolutely love a good documentation, examples and proof of concepts and creating those and when I say love that's not an overstatement because most of or all of what I'm going to show you here today is actually stuff that I have created in my spare time and I have opened my DMs on Twitter so please be nice but if you want to talk about Docker or Kubernetes or Jenkins and configuration as code then please just send me a message. Please don't make it help desky questions but all right. About this talk don't panic. Some of it is going to look a bit complex but so the complex part was creating the examples and figuring out how to do it and luckily for you I have already made all the examples and you're going to get all the examples so you can also just sit back and relax so you don't have to take notes off of the slides. There is a repository that you will get afterwards you'll get the slides afterwards so yeah just enjoy the show. Great so story time so my story starts with Kubernetes because Kubernetes is the most interesting thing in the world and of course I wanted to get involved with it so I wanted I had a love for Docker so Kubernetes came naturally to me afterwards so Docker for containers and Kubernetes for container orchestration and I wanted to become a certified Kubernetes administrator because I wanted to work with this and so the customers would hire me to do their Kubernetes so I wanted to practice Kubernetes and become better at it so I came up with a project because I'm a very hands-on person and I thought I love Counterstrike so why not figure out how to manage Counterstrike service on Kubernetes. So I did my initial commit almost two years ago well actually a bit over two years ago the the third of April and a month later I passed the certified Kubernetes administrator exam and that was nice but I want to continue working on this project. I also had a job I have a job so I needed some help with this project to get this going so I hired the nice Butler Jenkins running in Docker. I started creating my jobs for my workflows by clicking around in the UI but that quickly became tedious to have backups of so I created my jobs as code I converted them and afterwards so I was still backing up my Jenkins home folder so I configured my Jenkins with configuration as code and lastly I installed it with Helm into Kubernetes so I had everything running in Kubernetes and could manage it in the same place I'm going to assume you have some knowledge with Docker I'm not going to dive into those parts all of the snippets and all of the code I'm running is in the companion repository that you get access to and this is the part we'll focus on so going from Jenkins in Docker with jobs configured as click to jobs configured as code having our Jenkins configured as code and then installing it with Helm in Kubernetes and seeing that in the end when we have our Jenkins configured as code and our jobs configured as code it's actually super easy to get running in Kubernetes so the agenda in three pieces configuring jobs as code configuring Jenkins as code and running our awesome Jenkins in Kubernetes in the end so step one configuring our jobs as code so why do we want to do this well we want the easy reconfiguration right we don't want to have to if we lose our Jenkins instance then we have to click around in the UI and recreate our jobs it would be nice if we just had the specifications and could just load them into Jenkins our backups become lighter as well because we don't take copies of our job folders and with all of our content in it we just take a copy of the configuration of our job and lastly I have my configuration under version control so I can see who changed what when and from which value so I have the traceability of who's making changes to my workflows so how do I do this well there's two ways I can use job DSL to try it and true we've no time to do it for years a lot of Jenkins plugins that extend the jobs have native DSL so it's it's a great way of configuring your jobs as code there's also the declarative scripted pipelines those are the future oh there's a poll interesting yes I do so declarative or scripted pipelines the future they can resume on master restart they the declarative pipeline has job has a DSL which makes it easy to use a lot of different plugins and the scripted pipelines are in groovy so if you want to do something that isn't natively supported in the plugins or you want to do something highly customized then you can use the scripted pipelines instead you can also use a script block for the declarative pipelines but consider if you're using the right tool for the right problem I'm going to use job DSL for this because it turns out I can do an instant conversion of my job configured as click into a job configured as code it's the little thing I came up with it's super cool so a job in job DSL I want my actual pipeline in a job DSL spec I want a seed job that knows about my job DSL and I want to add the seed job to Jenkins rather than add my job to Jenkins and it gets smart I promise so the way I create this job I create a new file call it some job DSL that groovy I write the job DSL specification I'm just going to assume for now that writing job DSL is trivial and then I publish it to my internal git somewhere so Jenkins can get it afterwards a job DSL could look like this so I have a job specification it has a single step in my step it echoes hello world after creating my job I want to create my seed job so that's how I make Jenkins aware of my job in job DSL so I create a new freestyle job from the Jenkins UI I add a job DSL build step for the job DSL I just created I save the seed job and then I run it and then Jenkins will create the job that I have in my job DSL specification so just to recap I have my pipeline in a job DSL I create a seed job for my job DSL and then I tell Jenkins about my seed job um did I skip something no I didn't excellent sorry about that so the reason this gets smart is because so I have my single job as job DSL and I have my seed job but I can actually create a number of seed jobs for a number of jobs and then have a single seed job that a super seed job that reads all of the seed jobs so I get my entire platform into Jenkins but I'm only telling Jenkins about a single of these jobs so I'm only configuring one job through the UI this is very nice so the way I get into job DSL uh so job DSL has this very cool feature called the configure block and in my opinion the coolest feature except for being groovy code as well um and from the documentation so the configure block it says it's used inside the job DSL so give direct access to the underlying XML of the Jenkins config XML all right so the configure block looks like this so the intention of it is for to use job DSL for plugins that doesn't yet have a native DSL support so either you for the plugin you will have these nice commands so git it has a remote so they have implemented that in the git plugin otherwise if I want to do more advanced stuff I might have to use a configure block for writing directly to the XML um this is cool for two reasons so first of all it says something about a Jenkins config XML so it sounds like my job is actually already there configured as in in as code somewhere and secondly the configure block will give me access to interact with this XML all right so did you know that a Jenkins job is available as XML through the slash config XML endpoint so this what it looked like for this is what it looks like for a job that has a single task that says hello world I can actually go to the UI and go to my job and on the config XML endpoint get a specification of my job so this is pretty cool um so I'm going to show you now that a Jenkins job is already XML I'm going to show you the job DSL to wrap this XML and I'm going to show you the bootstrapping so we can just see that everything works all right so I told you about this example repository that you're going to get it has five folders uh the first one is the basic Jenkins in docker so that'll just give you get you running with the first example and you'll be seeing me just copying and pasting a lot of stuff from this example repository that's because that's how easy for you it is to uh use it as well so I have a preconfigured Jenkins it just has an admin user and it has the job DSL plugin because this exercise is about job DSL so it has the user admin admin of course I want to create a new job call it Nikolai and that's a freestyle project and I want to add a single build step I want to execute shell and I want to echo hello world and just run it so we see that it works excellent it ran and it printed echo hello world or printed hello world so in the URL I can write the config XML and I actually get the specification for this job awesome okay so I wanted to convert this to job DSL so in the job DSL folder I have a read me tells me how to do the demo says that I should get the config XML for a job and that I should follow the steps in the XML to job DSL template groovy file so I'm going to jump in there the steps says make a copy of the template file I'm going to do that and replace the XML job here part with the contents of your config XML you get that from the config XML endpoint add youths use the view source I'm going to use the view source to get the XML copying and paste it into this job configuration excellent and then it says remove the header of the XML document I'm going to delete that and replace backslashes and dollar signs with backslashes or escape them so this is a simple job so it doesn't have any backslashes or dollar signs but this is something I ran into when I did it on my own time lastly give the job another name that replaced me job DSL so I'm just going to call that Nikolai in job DSL and congratulations you have now converted your job to job DSL all right now I need to add this to Jenkins so I create a new job I call that seed job it's a freestyle project I add a build step it's I wanted to process job DSLs I want to use the provided DSL script and I'm going to paste the job DSL that I just wrote I save the job I run it and then after a couple of seconds it'll run it says it generated an item Nikolai in job DSL can run that while it runs we can look at the configuration if I can hit it oh no sorry about that I have a screen sharing bar in the top of my screen can you see my screen again yes we can excellent apologies for that so looking at the configuration I have my single shell step it says echo hello world and when I have run the job it says echo hello world great so this is how easy it is to convert your point and click job to job DSL and you can do this for any job because I didn't make any assumptions about what my job could do I just got the config xml pasted it into the job DSL wrapper and then the configure block did the entire translation and I had my job as job DSL awesome so is this cheating well no I'm I changed my workflow so if I have to change my job now I'm still doing it through the UI but I'm persisting changes with the generated config xml through git so I have my job configuration under version control and I get my easy reconfiguration with the supersede job so this is not cheating this is awesome all right my next steps converting this to a declarative or scripted pipeline for the complete as code flow so right now I am making changes by clicking through the UI and that gets tedious still and my wrapped job DSL is not that easy to read but this is a great first step for just getting my point and click job as code so the next step now I have all of my jobs as code so I want my Jenkins as code as well so my dream the example platform I'm building now is that I have a number of jobs as code I have my basic pipeline I have my config file pipeline and I have my Google cloud SDK pipeline I have those into Jenkins I want to configure Jenkins with an SSH key because I'm pulling my pipelines from some git repository somewhere where they are in version control I also want to add some configurations files for Jenkins like an auth JSON file for the Google cloud SDK and lastly I want my things to run in Kubernetes so getting our Jenkins to configurations code so why would I want this as configurations code well like before I wanted to be trivial to recreate my last configuration I want to be easy to create test setups and just being able to spin up a Jenkins with a certain configuration makes it great for containers because I can just spin one up at will it also makes my backup slider so I don't have to take backups of my Jenkins home folder anymore I can just back up my configuration and I have my configuration under version control so I can see who makes which changes to the configuration of Jenkins so how do I do this well that's a little harder than simply converting your jobs or wrapping your jobs and job diesel so I have to do one plug-in sort of or one setting at a time but I can't do it it's an iterative approach so I can I can just do it one plug-in or one setting at a time and then more and more have my Jenkins configured as code and then in the end when I'm done I can switch the backup strategy so I'm just saving my Jenkins Jamel the configuration of my Jenkins configuration as code instead of my entire Jenkins home folder there's a ton of great examples on the official configuration as code plug-in website sorry a GitHub repository they have a folder with demos that shows you how to configure a lot of different plugins that you might have installed on your Jenkins this is a very great resource for helping you doing that so so what's my first step for getting my Jenkins configuration as code in Docker well I need the Jenkins configuration as code plugin and do I want to install this manually through the UI no of course not I want to use pre-installing so I create three files one file called plugins txt it has the configuration as code id of the configuration as code plugin and has the version that I'm going to use I then create a Jenkins Jamel which is the config file for the configuration as code plugin under the Jenkins I add a system message I tell the system message to be configured as code so this is a nice sanity check for me to see when has my configuration actually been read thirdly I create a docker file for building my custom a Jenkins docker image with my plugins and these are just basic commands from the official documentation so first of all I copy in my plugins txt file I run the install plugins script with my plugins file so that's going to tell the script to install the configuration as code plugin secondly I copy in the configuration of Jenkins and I tell Jenkins through an environment variable where to find the configuration and this is just from the official documentation there's no magic going on and it is that simple so I'm going to show you starting the simplest Jenkins configuration as code in Docker so I have this here I have my docker file that I showed you before the only thing I have added here is that I'm skipping the setup wizard it's in the examples so you can check that on your own time I have the plugin configuration as code and I have my basic Jenkins configuration as code configuration file I have built the docker image and now I'm going to run it so stop the other one and go into the right folder and after a couple of seconds Jenkins will be running again it tells me to unlock Jenkins all right so I get the initial password from the log files of the docker container I don't want to install any plugins and I just want to continue as admin not now start using Jenkins and it's configured as code awesome so that was that simple just to see that my Jenkins works I can create a new job to see it only has the freestyle project because I don't have any plugins here it's also a very limited number of build steps I can do my echo hello world and run it so I actually have a working running Jenkins that is configured as code so Kavi is for doing this well when you are just installing your plugins pre-installing your plugins and you are mounting in a Jenkins home folder backup you will be overriding the configuration as code plugin so if you're not starting entirely from scratch you will need to add the configuration as code plugin through the UI through your managed plugins secondly I am mounting in the Jenkins to the configuration into the Jenkins home folder so if I were to mount on a backup on the location then it wouldn't know where to look for the file so you would have to mount those elsewhere so step one starting my docker with a Jenkins configuration as code so what's step two well I still have this clicking part of adding my supersede job to Jenkins so I might want to script that I can add a job to Jenkins by just adding it to the Jenkins YAML file so under jobs I can add scripts and under scripts I can add the job DSL so this is similar to the job DSL you saw before it's just been indented to fit with the YAML style this means that I can actually add a supersede groovy job to my Jenkins YAML and then the job will be there when Jenkins starts up so here I have my supersede job it checks out some code from git repository that will be the job DSL companion repository for the example project it uses credentials to check this out and it runs a single step it's a DSL step and the DSL step reads the external job DSL of the repository and uses wildcards so it just reads all of the job DSL and creates all of the jobs it will also delete jobs if I make changes to the git repository and run it again missing pieces for this well I am starting to check out code from a git repository using SSH so I'm going to need an SSH key I configure my Jenkins configuration as code YAML as well to add this key on startup because I don't want to click through the UI to configure Jenkins and this means I also need to add the credentials binding plugin I don't want to hardcode secrets into my configuration either so I am wrapping the an environment variable in brackets and dollar sign this means when I'm running my Jenkins when I'm starting my Jenkins and it reads the configuration it'll read this value from the environment so if I put this environment variable into my container that I'm running Jenkins will read the environment variable and use that to configure its credentials so that's super cool um and I'm actually also missing a couple more because I am writing job DSL so I need the job DSL plugin I am using a git repository so I'll need the git plugin I'm using SSH so I'll need the SSH plugin and I'm going to be reading pipelines scripted pipeline jobs so I need a pipeline plugin or the workflow aggregator which is kind of a meta wrapper around a lot of pipeline plugins so um this just means that I get the latest version of a lot of pipeline plugins in production you should figure out which of the plugins you need and of course pin the versions of those so you don't simply rely on latest great so demo time I'm going to start a Jenkins with the supersede job bootstrapped so in the jcasc folder I have an advanced folder there is a demo folder and there is a couple of configurations and a couple of shell scripts that run Jenkins with these configurations the difference between the scripts out here and the scripts in the demo folder is that in the configuration I'm checking out with SSH I can do this because my credentials point to a service account or bot user on my github that I have added to the repositories um when you are running this you will be doing it checking out with HTTP but then that made actually supplying the credentials a bit redundant so now you have both examples so in the HTTP examples you can do it without credentials in the SSH example you will have to supply your own credentials for your own repository all right so for the Jenkins bootstrap you can see from the slides I have the credentials configured I have my job configured for the demo I'm using script security false so I don't want to go through the commands I'm running in my seed job and allow those one at a time in production you should use script security and I have my run script and my run script simply provides the environment variable with my SSH key for the container when it's running so I can see the into the advanced example and into the demo example and use the run script and Jenkins is going to start up and I have my Jenkins configured as code and I have my supersede job added to my Jenkins automatically that's awesome and I can run my supersede job and after it's run it will have read the job DSL from the job DSL companion repository so folder with job DSL it has the supersede job and it has a number of other seed jobs as well it had it has added the auth job the basic job and the gcloud job and I know that after running the basic example I am able to run the basic pipeline and this is going to check out the the project from the configuration as code Jenkins k8s pipeline companion repository it's going to run my build shell script which will print that it's building and success great so my Jenkins is working and I didn't do a lot of work to add all of these jobs I just bootstrapped my supersede job and then it loaded all of my jobs in my project this is awesome um great so uh caveats for pre-installing plugins it's recommended that you create a base image with the plugins that you need um and that you don't simply keep the plugins txt because at a later time the plugins might not be available in the version that you need so I told you that I have been working on this project for almost like yeah well over exactly two years and when I put it down for half a year and picked it up half a year later and tried to build my Jenkins again I thought well great I have it as code so that's just going to work and then some of the old plugins that I was using had been removed from the from the from the public repositories so I actually couldn't build with the oldest plugins again the good part is that it forced me to update all of my plugins to latest but um it's not a good thing if you expect the plugins to be around and actually be able to use them in those old versions secondly so I told you about the the version of the workflow aggregator plugin that's just going to key to it is just going to load the latest of a number of pipeline plugins so you should figure out which of those you need and then pin the version of those thirdly there is the custom wow packageer project that you can also see for different ways of packaging Jenkins with libraries and extending it with plugins to to make it easier to reproduce the actual Jenkins that that you're going to be that you you're going to be building so more missing pieces well you saw that I actually had to run the supersede job once and I had to do that manually and I don't like manual work so can we automate that well we can give it a trigger we can just tell it to run every two minutes that's great because then at most two minutes after my Jenkins is put to live um it will run the supersede job and the supersede job is going to load all of the jobs on my platform awesome um it also means however that my supersede job will be running every two minutes and that feels like a bit of overhead but um then so I have my I have my supersede job I have added it to Jenkins with clicking now I have added it to Jenkins with the configuration as code and thirdly I have told it to run every two minutes so you saw that I had the supersede job in the in the in the job DSL repository as well and that's because a seed job can actually bootstrap itself so I have one version of my supersede job that I have added to my Jenkins configuration it has the trigger timer so that tells it to run every two minutes but then I have the supersede job in my repository with seed jobs without the trigger so I know that it's going to run at most two minutes after then it's going to overwrite itself and then it's not going to run anymore then I can add it could add another trigger to it in the in the in the repository so that it would run whenever I make train changes to the seed jobs but this makes it super cool because then I know that it will only run once and at most two minutes after Jenkins has been uh has been started so the third part we have all of our jobs as code we have our Jenkins as code and now we want it to run in Kubernetes and we're going to be using Helm to make that super easy so why do I want it running in Kubernetes in the first place apart from Kubernetes being awesome well a unified way of managing my applications so I told you that I wanted to have my Counterstrike servers running in Kubernetes so why not also just have my tooling around it running in Kubernetes secondly it allows me to scale with Kubernetes build agents so instead of adding these build agents to Jenkins and just have them idling around I could actually use them inside my Kubernetes cluster add Jenkins to my Kubernetes cluster then whenever I need to build something it'll just be using the compute power that I already have running and I can also use custom images for this so if my build agents need a specific compiler or version of a tool I added to the container which serves as the template for the Kubernetes build agents and then my jobs will be running inside of this container and I needed to do stuff with the Google cloud so I added the gcloud SDK which I'm going to show you caveats with this so I was just able to add environment files to my container when I ran it with the Docker run command I'm not able to do it in the same way when I run it in Kubernetes so what do we do in Kubernetes when we want to configure our application well we use config maps and we use secrets I would still not like to to have my to have my secrets have my passwords have my ssh keys as code so I would really like to still be able to provide them at runtime from the command line so I'm going to show you installing our awesome Jenkins with helm 3 I'm going to show you supplying the secrets for a helm 3 chart and I'm going to show you the custom agent images that I'm using so prerequisites for this a cd I want to go up I want to go into the Kubernetes folder with the Kubernetes example the Jenkins extra left as an exercise for the reader you can run that on your own time and see how the you can configure the config file provider plug in to supply configuration files for a job in a pipeline so I have a read me that tells me how to do the demo it tells me prerequisites I need to have a helm installed so I have a helm in version 3 3.1.2 I need to add the stable repository for helm this is also part of the quick start guide and afterwards I can search the repository and I can see oh no I can see that I'm finding a stable Jenkins help chart and I should have access to a Kubernetes cluster so I can do Kubernetes get oh sorry Qt control get notes and I have access that is awesome as well I have my helm sh so that's a helper script for doing the helm installation I have my install script I have my own install script the install script creates my two secrets they created from files so rather than cutting the contents of a file into an environment variable that I give to my container I just create a secret from the command line from a file so this is my SSH key this is my config file and then I'm running the helm helper script and this is so cool because I am supplying by using set the reference to the secrets um into a list of environment variables for the master container which means that I I could have gone into my value jammel and I could have supplied these as a list but then if I wanted to do it later from some ci I would have to use scd or some other tool to edit my value jammel file and in this way I'm actually able to do it just from the command line I think that's kind of cool the values file I'm using for this deployment looks like this I have my master I configure it to install a number of plugins a number of these you've seen before a number of these are just a quality of live plugins so job config history what changed uh the being able to rebuild a job time stamping output wonderful plugins I specify a specific tag for the Jenkins master that I want to run and then I supply my Jenkins configuration as code um configuration so this is not the config file this is not the Jenkins jammel file this is the configuration from the Jenkins jammel file but for the helm template so I enable jcask I provide a single config script and then I provide my Jenkins jammel and you'll see this is what I told you in the beginning that um when you have your Jenkins configured as code it's super easy to get running in uh Kubernetes with helm because you just paste your entire configuration you indented it then you're ready to go so I told you about the trigger I wanted to be run at most two minutes after this trigger will be overwritten it's just here to auto trigger one build awesome lastly I'm configuring the agent image so these are the agents that I want Jenkins to spawn and for you to use for building um and I'll get back to those so I will run the install script gonna create two secrets it's gonna do the helm install and with the output from the helm install so I'm just going to copy and paste these things this is the getting the secret from the for the admin account this exporting the pod name and then I'm port forwarding to the Jenkins instance running in the cloud so that I can access it from my computer um what does it say here oh didn't it work sometimes the copy pasting an entire line with a windowed terminal is a bit funny okay that worked and I'm just copy pasting stuff from the uh from the documentation I'm not making any of this up so my local host is running it has an admin account it has the I'm using the password that I was provided by the output there is the supersede job and so it should trigger in a second but I just quickly want to show you the configuration of it while it hasn't overwritten itself so you uh great so you can see here I have the trigger that it will build um every two minutes so now I have a build queued it's going to take a bit longer now that I have it running in Kubernetes with the Kubernetes plugin because it needs to spawn these pods that will run my build agents in containers for uh for executing my build inside it so while it's running and starting pods I want to look at the agent image I'm actually using I have a docker file I'm using the Jenkins inbound agent so this is the new name for the Jenkins JNLP slave images uh this is a nicer name and it will allow the Jenkins agent to communicate back and forth with the with the Jenkins master then I am installing the Google cloud SDK again that's nothing magical it's taken from the official documentation translated into a docker file I have put the the command line things after a run command and I've removed the sudoze because I don't need that when I'm running as root and in the end I'm switching to the Jenkins user again I have a build script um so you will want to push it to your own repository um I give it a name I give it a tag this just pushed publicly that made it a lot easier when I'm going to download it because it'll just pull from the from docker hop I specify the name of the image and I specify the version that I want to run and see here in the background that no executors are running so my supersede job has run and I'm now able to start all of my pipelines and they should be done in a second or well a couple of seconds depending on how how fast it takes for um for the pods to start so the ones we didn't look at the auth pipeline it's what yeah so what you'll see here is actually it not really interesting because um this will just be referencing a Jenkins file it'll tell me it needs to check out the pipeline script from a source code it needs to check it out from git and then it tells me where it is in the repository so this is the auth and job diesel there's a git repository and then there's a script path the interesting part is actually in the script on the pipeline repository which is the mono repo project example so I could have done this as three different repositories I chose to do this one because then I only had to create one repository the auth has a Jenkins file and it will check out this repository from source code it's going to run a hello world and then it's going to echo inside a config file provider clause that this is where I would authenticate my session and it just cats the auth json file so now that it's run we can see the output so there's a sanity check hello world and it prints my session file it says secret I'm a password file and looking at the resources auth json this is the password file password file so this is what I put into the um into the secret when I created it with cube control and which Jenkins then read when it started up and added as the auth json the auth json config file from the Jenkins auth json environment variable which I could then reference with the auth json json id this is very practical if you need to authenticate a google cloud for example so an example of using the google cloud a tiny example nonetheless I have my sanity check I have my country file provider as well and then I just do a gcloud version just to check that the gcloud SDK is actually installed into my into my Jenkins agent and when I'm looking at the output I have my hello world I have my secret file and I have the version of the different components of the gcloud SDK that I'm installed I'm doing something on the google cloud create machines etc and in the end I clean up the VMs another cool trick put it in a try catch have a finally clause that cleans up because then if you make if you make an error it still deletes your resources on the platform you get all these examples so to wrap up I wanted to do a lot with Kubernetes so I had an interesting journey that took two years and then last weekend I had a colleague write me hey man are you still hosting csgo servers in Kubernetes and I wasn't entirely done with the platform that I wanted to build but he told me that the free server broke and I thought well all right I have it running so I can start it up and 15 minutes later I was running csgo servers on Kubernetes and as far as I'm told they had a great match for a couple of hours so thank you and questions thank you very much for your great presentation we've got a number of questions we answered some of them offline but we still have eight questions to discuss so let's try to go through them and maybe rate a bit over on the meeting so the first question is where can I get the job this self-conversation that is a great question so you can get it on this repository so figure out configuration is code Jenkins k8s and you can I can paste the link in zoom so this is the repository that you'll get access to and also inside job DSL there is a read me and it has a detailed guide which takes you to a GitLab repository I created which has a lot more detail about what actually happens in the script and there's also some things about verification and yes go nuts and please tell me if it works or does not I made it 10 months ago but apparently yeah some of it still runs thank you so the next question rather than reading secrets from environment variables is it possible to configure the secret from a vault or other secret story in the configuration plugin yes so that is an excellent question if I was to get so I haven't tried this out but you will still need to tell Jenkins how to talk to vault so you need to give Jenkins a token in the first place which means that these credentials it's actually still great for giving your Jenkins your vault token and then in your job you will be using your vault token to communicate with vault and get the SSH key which you will then be using for Git or the the the auth file that you would be using for for Google yeah I also shared a link to official documentation on the jcask page because we have a special documentation page which shows all the various modes of managing credentials with jcask so if you want you can take a look and let's press it to the next question do you update your base image with installed plugins when are the plugins updates available or do you update them over a user interface both so I would want to I would want to test that the plugins work and if I have a Jenkins running I can easily update them through the UI and see that my Jenkins is still running and my jobs are still running and then in order to persist my change I would add it to the the plugins txt and I would build a new base image which I would then put in my my internal docker registry somewhere thank you so the next question is about secrets again do you actually put environment variables in for secrets or is a better way I probably don't want to have all my secrets as environment variables on the master easy I guess sure so well first of all if you have your Jenkins in the credentials then you have them in the credential store on your master so if you have access to the master you can jump through different hoops to still get the credentials but I would like to store my credentials in vault and then just have a vault token on the master and then on runtime manage my secrets in the different jobs so sure I can I can see that it looks like it blows up if you add all of your secrets as environment variables but when you have them as credentials you technically still have them on the master available you just have to know where to find them thank you let's go next is it possible to keep the build history and other data like artifacts or logs use Jenkins configuration as code maybe you want to take that Oleg you can get you can take the logs from the from the container it's running if you have artifacts from your builds that you want to keep you should put them in an artifact manager so Jenkins is not a database that's one of the ways another ways is to actually pass volume to your container use Jenkins master so that you can store Jenkins home on a volume for future we have a lot of pluggable storage storage it's already possible to do it for logs we created a log storage for pipeline and you can use for example fluent d and aws cloud watch or elastic search and yeah for artifact manager as Nikolai said we have an integration with s3 and in general we intend to keep working on that there is a public Jenkins log map which is being developed in the moment and any feedback will be useful but in principle just pass volume with Jenkins home and it will work and it will persist the data between restarts obviously make sure you also make backups using well or other services and marketing some presented how to do that at the last meetup yes so just to add in the example repository that you'll get under the basic Jenkins and Docker I was using the fresh Jenkins and I was mounting a volume onto the Jenkins home so this is the name of the volume this is the path inside the container and that meant that I could prepare it up front so I could install the job DSL plugin and it would also persist my my jobs through restarts okay thank you and yeah we will try to process all funny and publish the responses after the call so yeah I'm posting links in parallel and hopefully you will publish all this information soon yes okay now so we have well actually a number of questions increases so if everybody has 10 extra minutes we can keep answering questions yeah sure okay let's do that yeah so the next question is do you have any experience with hush corp vault as a source of truth for secrets no I have a tiny bit of experience with with running it but I haven't had it running in production and and work with the tooling around that yeah but but I would love to talk about vault offline but probably not at a Jenkins yeah right and also if you're interested you can join Jenkins configuration as code guitar chat because hush corp vault plug-in is currently maintained by jcast maintainers so if you have any questions just join the chat and other contributors will be able to answer these questions okay and next question how does it work with Jenkins Kubernetes operator I have no idea Jenkins operator yeah there is a separate project which has been started something like one year and a half ago yeah I would expect it works really well with it so it says it has support for the configuration as code and my I didn't really do anything special with my jobs going from a normal Jenkins to a Jenkins configurations code so I would expect you to just like dump your config Jenkins yaml in there and that it would run out of the box something quick and for the reason I'm not using a an operator is that I don't need the lifecycle management of my Jenkins so doing the but I manage the lifecycle myself I do the upgrades so the operator can help you with that and then I expect Kubernetes to do the rest so if my Jenkins crashes then Kubernetes will restart it and then hopefully if I've configured it correctly it's going to bootstrap itself thank you and hopefully we'll have a special online makeup for Jenkins Kubernetes operator it's yet to be announced but yeah we're working on it yes and also we talked about maybe doing a more extensive job DSL meetup at some point a plus one and if anyone has any other topics please use the feedback form I shared it in the chat so if you would like to request presentations or maybe if you would like to share something on your own just let us know we will be happy to do that okay now let's finish yeah so next question is why is the presenter including google cloud SDK in the agent docker image is because Kubernetes is running on jpe no not at all that was just because I am running my Kubernetes project on the I am running my counter strike servers on the Google Kubernetes engine so I needed to interact with Google cloud so for for testing like spinning up a new GKE cluster or configuring firewall settings and I wanted Jenkins to be able to do that for me so that's why I am adding the SDK so that I can from the command line but from a Jenkins job interact with the Google cloud there's nothing to do with where Jenkins is running it's this is actually running on on the new beta Kubernetes from Siebel the cube 100 beta program so the only thing Google here is is just yeah it could be any tool that I installed I just had this lying around and I'm actually using it so yeah great thank you okay so next questions what do you recommend to mask passwords in the logs as mask passwords plugin has a security warning now for some time I don't know I would expect you to if you use the credentials plugin then by default if you try to print the credentials it will mask them maybe Oleg yeah as a former maintainer of mask passwords plugin credentials binding in indeed tries to mask passwords by default mask passwords at some point it had integration with Jenkins pipeline can be used in some cases so the security warning for it is related to low priority but yeah if somebody wants to contribute please do so unfortunately I had to step down as a plugin maintainer in 2018 so there is some technical depth and yeah the plugin is looking for new maintainers credentials binding is your answer yeah credentials binding yeah okay next question how do you manage access control for users with jcosk do you define this in your Jenkins YAML as well yes I do so I am a one person developer army in my project so I just have the default admin user but you could add users in your Jenkins configuration as code YAML as well but what you'd really want to do is just hook it up with your LDAP and manage your users from there but then manage the LDAP integration from the from the configuration as code YAML so that when your Jenkins starts up fresh you can just log in with your users and groups right and if you want to manage authorization all major Jenkins plugins for authorization also support the groups so for example LDAP active directory and there are authorization management engines like role strategy or matrix authorization or for example github authorization which again can be configured in order to pass specific access configuration to the Jenkins instance and there are demos in jcosk plugin we show how to do that okay so just three questions left yes use the kubernetes agents how does one handle drone pipeline steps in docker container again does this need some extra config or just work so I am sponsoring them in kubernetes um I'm actually not sure uh yeah so you I think that when you create your um so when I have my um this was the job diesel when I have my scripted pipeline I'm telling it where to run so because I didn't specify any agents it's just going to run on the default agent but I would be able to in different steps uh to specify which nodes it should actually be running on so if you create so in my um so the helm chart is a bit basic you have an agent um but if you add several types of pod templates to your kubernetes configuration uh in Jenkins then you can specify in your pipeline which parts of your pipeline should run on which agents all right and again you can refer to the previous presentation by marco jackson he presented how to use uh kubernetes plugin which can private agents on demand in kubernetes and there are other plugins which can private site containers or if you run Jenkins in a pod and uh many other options so yeah you can provision them depending on your needs okay um yeah so let's try uh so in one of last iterations uh you try to skip Jenkins setup wizard but Jenkins still prompted for admin credentials and plugin installation so yes what happened I think uh what went wrong is that I was running the um I was running um this uh so uh Oleg told me about this cool feature that you can skip um that you can skip the setup wizard uh so I added this to the docker file and then I rebuilt my docker image and I called it jcask job um because I wouldn't want to break things that I had working in case this didn't work and then it totally worked and then I didn't go back to my readme and update the name of the newly built image so I'm actually running the very first image I created when I did the demo for Oleg and he told me hey this is a cool thing you should try out so uh it was just uh just the demo gods giving me a hard time um yeah thank you so okay um so if we have 100 of jobs it will be hard to define everything to the one yaml file uh is it possible to define each job in a separate yaml file yes thank you for that question um so in the in the companion repository called job dsl I have a job dsl folder and I have my supersede job um and this loads all of the job dsl in this job dsl folder in each of these folders I have a job dsl script which loads a single Jenkins file and I could have several uh job dsl scripts in this folder and I could have several folders so you can put each of your um yeah so for your hundreds of jobs so for each job you will have a Jenkins file and each of those Jenkins files will just have a one pipeline job which reads it from the um from your source code management and uh and loads it with the script path and also like so this is called basic Jenkins file in the uh auth example it's called auth Jenkins file most of the time if you have it in different uh repositories it will just be called Jenkins file and be in the root of your repository but I did it as a mono repo because I thought I'm going to have three projects that are almost identical so um I'll just put it in one repository for for making it simpler okay yeah thanks I'll say if you use just jcask yaml it's possible to specify a folder as a yaml configuration pass and if you do so or jcask plugin will load all the yamls and it also has logic to merge with them this logic is probably not like a different much but still it does have a job for many common pieces okay and finally the last question would you accept downtime for master during update or would you try to replicate the master and update them one by one that's a good question uh Oleg maybe you want to start what do you feel like yeah so it really depends on your setup because if you have a cluster of multiple Jenkins masters you can implement various things like rolling cup grates a beta thing for them if you want it's totally doable I know there is no automation or framework for that right now well um in cloud this we offer some tools for cloud this core and for BCI products but yeah for open source it would require additional scripting so if you want to do that it's possible if you just rely on Jenkins by default then you would have to accept downtime right now Jenkins doesn't have multi-tenancy we had a research project which we presented with 2018 at the contributor summit in Brussels but right now you have to restart your instance in order to apply upgrades whether it's configuration upgrade or plugin upgrade but did you see how fast it was to start up a new Jenkins so it should be fairly easy for you to to test out the new version of plugins or test out the new version of Jenkins take your configuration move it to a different instance spin it up click around in it verify that it works to your needs maybe add a couple of jobs to it and even run them if you want to if you want to verify it by that way and then when you're confident that the new version of Jenkins is good or the new version of the plugins are good then you simply apply the configuration on your production Jenkins and you spin it down you start it up and then it's going to be ready yeah Jenkins top time mostly depends on the number of jobs on the instance and we have some discussions about improving job loading time so so for the new features I have a cool trick where you limit the number of jobs on a Jenkins when it starts up yeah it's also possible sit their jobs why not so great that's it so thanks everyone for questions again we will try to process them and thanks a lot for Nikolai for this presentation it was really useful a lot of different cases and looking forward to see what's next so stay tuned for the next Jenkins on Kubernetes meetups pleasure yeah thanks all I'll stop the recording