 Okay, and we like thanks everyone for joining Jenkins online meetup to these main nineties and we have Nikolai who will be presenting to the story of migration from manual jobs to Jenkins pipeline with job DSM Let me share my screen and do some quick introductions. Okay, so this my screen Yeah, cool if you have never participated in Jenkins online meetup This is an event where the nice on the Jenkins community Basically, we invite contributors to share different stories about Jenkins to present to them us to share any kind of experiences and we invite everyone to participate you can ask questions during the representation and We also invite you to participate in the future events thanks to Continues delivery foundation and for helping us to run the platform Yeah, let's move forward. So speaking of contributions to Jenkins So we have recently published a new page which summarizes all the contributions So if you're interested in various aspects like just sharing your stories meeting people Contributing to court documentation Please don't hesitate to do so invite all people to participate in the project And the next week you have a good opportunity to do that because we run a UI UX hardfest Basically, it's an event where everybody is welcome to contribute to different stories I believe that during this presentation. We'll hear some runs from Nikolai about particular aspects of user experience and you're welcome to contribute and to improve the situation and You can also get special t-shirts and other prizes So if you're interested, please join us and it's going to be a quite a big event Hopefully it will be fun for all the contributors We're also looking for speakers and if you're interested in presenting and Jenkins online meetup Again any story which is related to Jenkins Including using Jenkins in different environments into a great use tools using particular plugins. Everything is welcome Recently, we have hosted a new page for speakers So you can go to Jenkins online meetup and there are some guidelines and right now we have a public call for papers proper Process so you can just submit an email and we will process so Everyone is welcome Also just a minute of advertisement Right now later today and also tomorrow. They will be called this connect event this event There will be a number of presentations about Jenkins Specifically about troubleshooting Jenkins Processing logs integrating Jenkins with different tools. So if you're interested to learn more, it's a good event You could follow and it's not too late to register and participate if you're interested Okay, now let's go back to our presentation So we will Be doing the presentation in a web and not mode you're welcome to ask any questions in So if you open the zoom panel, there is a corner beta button and there you can ask any question and We will be answering some questions offline and for the questions So we will keep them until the end of the presentation so that Nikolai can answer them Yeah, we've reserved a lot of time today for that for finding Connie you can adjust the contact Nikolai on Twitter and also we have shared a feedback form So any feedback will be appreciated and thanks to everyone who submitted feedback during the previous meetups We Processed that and we will make sure to take it into account today. We do less polls and Yeah That's it from me. So speaking of polls. It's our way to get feedback and if you don't hesitate I'll actually run one just to see you what job management tools use second and Promise it will be the only poll with today. Okay, is it fine? Okay, so while we run the poll, I'll just stop screen sharing so that we switch To the presentation by Nikolai and Nikolai and I'm sure he's himself and there we go Excellent. Thank you very much. Oh, like I want you to let you know that for the most part my use experience with Jenkins is great Alright, so welcome to the talk from freestyle jobs to pipeline with job DSL My name is Nikolai Grashalt. I am a DevOps consultant. I work at Effie code And I help our the organizations we help with Jenkins darker containers Kubernetes, I'm a certified Kubernetes administrator as well. And I spent a lot of my spare time on Just playing around with Kubernetes playing around with Jenkins. I Absolutely love good documentation great examples and break proof of concepts. I like building them I like using the ones that other people have built and So, yeah, I spent my a lot of my time fortunately just hacking around with this and I've opened my DMs on Twitter Please be nice. Don't ask like hope desky questions But if you want to chat about Jenkins containers Kubernetes more than welcome to send me a message Alright So during this presentation, there's going to be a lot of demos and I'm going to show you a lot of examples But don't panic. Everything you see is what you get Um, so I've created this Example repository With demo resources and these are all the examples that I'm going to do today Um, they're here. You can even open them in the Google cloud shell So if you don't need to have anything but a browser on your computer and Don't worry about the link either because you also get the slides in handout format So just sit back relax and enjoy the show Alright, so from freestyle jobs through naive job DSL or native job DSL to pipeline but um So a quick disclaimer when I talk about the freestyle jobs, I talk about manual freestyle jobs So, um, I'm not like I'm not bashing freestyle jobs. I think they're great I think they've gone done a lot of great stuff for us But it's the manual configuration of it that tedious process that I want to Say bad things about in a second So without further ado Freestyle jobs and the five stages of grief Also referred to as the kubler rush model Um, the first stage is denial So we think that the manual freestyle jobs are a great way of working um Second stage is anger because we start to experience this configuration drift. I don't know who's been changing my jobs I don't know I I can reuse my jobs. I could split the individual build steps up to into a different Parameterized jobs and then they could call each other and I would get a tiny bit of reuse But it's incredibly hard to manage and keep track of Um, I don't have reconfiguration of my jobs I with if I lost my Jenkins or lost my jobs I would have to manually click all of my jobs into Jenkins again And I'm just losing. I'm losing my work. I'm losing so much time and I'm just losing my will to continue working this way um The the next stage of grief is bargaining. So I try to um Change the way we work. I talk to my colleagues like can we agree on how we make changes or can we agree on how we create our Jenkins and manage it, but uh, this leads to my depression because It's not working. My colleagues aren't following the instructions 100 percent I'm not following the instructions 100 percent and that's okay Because maybe the instructions are too complex and intricate to actually work with efficiently, right? So the last stage of grief the acceptance stage Maybe manual freestyle jobs aren't that great All right So I want I want something else. So I want Pipeline I want configuration as code And I want my pipeline jobs in Jenkins But I can't just Put them into Jenkins. So I need I need to put them into Jenkins somehow So how do I get them in there? Well, I could create a job for my pipeline and add my pipeline to it but Then I I keep clicking in Jenkins still But that's okay Because I have the specification of my job as code. So at least it's it's slightly better um It doesn't scale very nicely because then for each of my pipelines I still have to click them into Jenkins. Um, so I'm it's just a pipeline job But I'm still clicking just a pipeline job for every single job that I have so The acceptance stage This isn't working out for me So I want to try something else What if I wrap my hello world pipeline In job PSL I can do that Uh, and then I can add that manually to Jenkins Okay, so that would actually allow me to reference all of my pipeline jobs from the job DSL And I'm not clicking as much in Jenkins anymore, which I really like So I've solved my clicking problem Um, but I still have this Change by modification. So I'm still making changes to the hello Job DSL wrapper whenever I have a new pipeline um so This isn't actually making me like completely happy um So what about this construction with a with a pipeline Wrapped in a hello world job DSL wrapper Wrapping the job DSL wrapper in a seat job DSL job And then having the seat job DSL job added to Jenkins Well, this would allow me to dynamically reference all of the job DSL From my one seat job DSL job So I would put them all in a git repository My seat job DSL would just add All job DSL in the repository and whenever I have a new pipeline I just add more job DSL so the repository So I get this change by addition Which means it's very unlikely that I'm going to break something else that was previously working cool, so Uh this and a disclaimer about the stars thing So I'm kind of using them like a grading system like which way of working is better For the wrapping of pipelines job DSL is absolutely awesome So it might as well have three stars. It's just for like discerning between the two that they are different So this is my focused goal. This is what I want to build And uh, this is what I'm going to show you how to uh, where how to get there today, so Talking about job DSL Every slideshow needs a quote. So my quote comes from the job DSL plugin. Uh, it's a groovy DSL for Jenkins jobs sweet DSL is just a domain specific language So it's not an actual programming language But it makes sense in the context of job DSL in the context of the job DSL API So a crash course in job DSL. This is a hello world job First I have a job tag. It means that I'm creating a new job a new freestyle job I give the job a name so I name this one hello world My job has a bunch of build steps So it's in a steps block This job has a single build step. It's a shell step And the shell step is echo hello world And uh, just to show you that I'm not making any of this up You can't actually specify a job like this. We are going into the first demo of the day. Yes, hands on and I like that and Uh, so I've cloned the repository onto my computer I've opened it in VS code. So you can see here all the folders There's the read me will tell you about the repository. The first exercise was called hello world So I'll open the read me And get it into a kind of visual mode. So put it over there And it's gonna tell me that the first step is create a new freestyle job in Jenkins I'm gonna go to my Jenkins and I'm gonna take a deep breath because The Jenkins I'm actually using is a Jenkins that I'm running in a container And you get that as well. It's in the basic jcask folder. There's a docker file There's a read me for how to build it. It's configured with Jenkins configurations code with all the plugins you need to do the demos and Lastly, if you don't have the docker installed, you can open the entire thing in the google cloud shell and You can run it from there. It's specified in the In the companion repository you get all right so The first demo So the purpose to show that we configure a javas code So create a new freestyle job in Jenkins and call it demo seat. Hello So i'm going to create a new job and call that demo seat. Hello. It's a freestyle job And it has a single build step The single build step is a job dsl step So i'm going to process job dsl and i'm going to use the provided dsl script The dsl script is in the hello world groovy So i'm going to open this one Copy and paste this And save it Then i'm going to run the job And when i reload i will notice that my hello world has been created And there's a typo in my In my companion repository and i can build it while i look at the configuration So the configuration of the hello world job. It should echo hello world and When i look at the result of it i can see that it echoes hello world So with this code Paste it into a new dsl job in Jenkins I'm actually able to create a job without clicking around in Jenkins that runs a shell step that echoes hello world crude right so Now i have my hello world job dsl and i've created my demo seed hello job in Jenkins Um, it's not exactly where i wanted to be but i'm getting there We we now know that i that a job dsl job can create another job Um, I can actually also inline This job so i can create a seed job That creates another job So instead of having a shell step I have the dsl step And then i'm actually providing the job dsl Inside the job dsl all right i'm gonna show you how this looks So The demo is called seed hello world inline seed hello world inline And I open the read me And it's going to tell me that I should create a new job in Jenkins and call it demo seed inline So I find my Jenkins And I create a new job call it demo seed inline freestyle project I should copy and paste the code from seed hello world inline groovy and that's the file next to it in the companion repository Looks like this so the same as the slide reproduced here for completeness I copy and paste this into a build step Which is a job dsl step And i'm using the provided dsl script I save it And I run it And when I reload I can see that it's generated an item so this seed inline job I can run it and look at the configuration while it's running So this job actually also has a build step which is process job dsls Which is the inline job dsl that I added to the seed job The indentation is very off, but um, yeah, it's there And when I look at when it's run I can see that it's also generated an item So it's generated the hello world seeded And when I go into this job I can run it while I look at the configuration I can see that it has a single build step. It's a shell step that echoes hello world And when I look at the job after it's run I can see that it echoes hello world all right, so um Now I have actually created a job in Jenkins That was a job dsl job that had another job dsl job wrapped inside it So I can create job. I can create jobs that create other jobs That create other jobs Okay um It's still not entirely where I want to be but we're getting into the idea of jobs generating jobs Uh caveats about this so in my Jenkins that I have running I've turned a job dsl script security off um You should have that on in production and only clear the scripts that should be allowed to run on your Jenkins masters um But for demo purposes, it's nice to just turn off You can also fix the formatting with the strip indent of groovy So the indentation becomes nice when uh when you're looking at the configuration in Jenkins all right, so Uh, but I wanted my my job dsl in another file and not inlined In a job dsl file. So how do I how do I get there? Well Um, I can actually reference an external file. So rather than provided as String inside the job dsl I can reference an external file So the idea is here that I have a repository with a couple of files. I check them out from a kit repository And then the seed external job will generate the job in the hello external file um, but I told you that I was going to show you a lot of examples and of course this is not going to run because It's just like soto code I can't just copy and paste this into Jenkins Um, but there is a demo not going to show it show it to you. I'm going to show you where to find it So It's in the companion repository It's called hello a seed. Hello world external There is a read me that takes you through the demo and the reason I'm not going to show it to you is because I am of course checking it out from an external repository And when you start doing demos with a third party dependencies things break and things take time So it wouldn't be as interesting to look at But you can do this on your own time and uh, it works because I just did it hours ago So it should still work all right So the cool thing about referencing referencing an external file is that I can actually reference files using wildcards So I can dynamically reference a number of files So if I have all of these different groovy jobs In my repository I can just use this any seed groovy job So check them out from the repository and then just create all of them And when I want to add a new job to your cell job, I just put it in the repository I run my seed any job again, and then it's going to be added to Jenkins So I get my change by addition and not change by modification All right So this is what it looks like when I have my external job DSL Reference by a seed job. So this could be the any seed And then I click the any seed job into Jenkins And then I have my jobs in Jenkins Awesome. It's not it's still not entirely where I want to be But I'm getting there. So Talking about the hello world pipeline So I'm assuming that you already have a number of manually configured freestyle jobs That you would like to have as pipeline or maybe you already have some job DSL that you would like to have as pipeline In this talk, I'm going to show you how to get the freestyle jobs as naive job DSL And afterwards I'm going to show you how to get the freestyle jobs as pipeline all right, so um From the manually configured freestyle jobs, there is a number of Paths I can take to get to the pipeline But no matter which path I take I still have to get out of the manually configured Jenkins jobs in the first place um Adding some labels to this I know that I can actually get out of a manually configured Jenkins job instantly And get into some configuration as code By just wrapping the job This is interesting. All right Secondly, I know that I could convert my manual job to native job DSL But that would take me some time And I know that I can configure my freestyle jobs to pipeline But it might take longer time And this is the reason that I chose I have a project that I've been working on with some Counter strike servers running in kubernetes Uh, and I can't manage everything myself. So I'm using Jenkins for this I'm clicking all my jobs and when I got tired of that I did the naive conversion because I could do it instantly And I want to show you how that looks like so The third or fourth demo depending on if you're doing them Doing them all Uh job DSL. I'm going to show you that a Jenkins job is actually already as code So it's there in xml in Jenkins I'm going to show you the job DSL to wrap the xml And I'm going to add it to Jenkins just for sanity checking so we can see that it actually works And I did this last month in a meet-up Jenkins on my meet-up The configuration is code of Jenkins, but I'm going to do it again. Just to show you how fast it actually is so The demo is called job DSL. There is a read me and The purpose to show you that we can almost instantly convert a job from a manual freestyle job to a naive job DSL My prerequisites are a job in Jenkins And I should get a copy of the xml to job DSL template from this repository So the companion repository for the other talk because why Recreate stuff when you've already done it once. Well, I'm going to take a copy of this And I'm going to create a new file here New file and call it xml to job DSL template Then it tells me that I should replace the xml job DSL part here With the context of contents of my config xml file And I can get the config xml endpoint from a job in my browser Okay, so Going to Jenkins I I actually already have a hello world job. So I can reuse this I can go to config xml and I get the xml specification of it I can do the view page source to get the raw output. So I want to copy that and then I want to edit the The template so it also has some instructions I'm going to delete those because I've streamlined the instructions I'm replacing the xml job here Let's just remove the xml header I'm removing the xml header And lastly, I should give the job another name. So I'm going to call this Hello world in wrapped job DSL All right Congratulations. You've now converted your job Cool So this is job DSL now and I can Use this in Jenkins I can create a new job I can call it. Hello I will call this demo seed wrapped Job DSL software developers are great at naming things That is a lie And a single build step. It's a job DSL build step. I'm using the provided job DSL script because I just wrote a job DSL script I'm going to save it run it once reload And I can see that it's generated my job And I can run it while I look at the configuration So I here I can see that it's actually created a job with a single build step, which is a shell step that echoes hello world And when I look at the output from the run of the job, it says echo hello world cool so You can Look at why this works on your own time. It is extensively documented Uh, I really like writing documentation. It is uh yes all right so I just instantly converted a manual freestyle job to job DSL Using this wrapper. Is this cheating? No uh I Like my workflow is not like inherently through code I can make changes to the job through the UI. I could go into configure and change it and then I could Take the config xml that I generate in Jenkins and I can persist that and get again I could just wrap the new config xml that I get So I have my job configuration under version control and I can easily reconfigure my Jenkins by using the dynamic seed groovy If I just have the wrapped config xml in my job DSL repository cool, um But I'm also not entirely done because having to go to the UI and making changes to my job and Wrapping them in xml every time becomes a bit tedious or wrapping them in job DSL becomes a bit tedious. So I would like to have them as The native job DSL or the or a Jenkins pipeline So, um This is the story that I originally wanted to tell you Because now I have my naive job DSL and I can convert that to native job DSL or I can convert it to pipeline Um Converting it to native job DSL looks like this. I think it's super cool Because it's an xml that you're just reading with the configure block Uh, which is a cool thing of job DSL. It allows you to interact directly with the xml the config xml of a job So I can iteratively Uh, I can iteratively unfold this I can do the small iterations to change my naive job DSL into pseudo naive job DSL until at one point I'm at the native job DSL and Then I can take the large leap of configuring reconfiguring my native job DSL to pipeline Which is nicer than taking the huge leap of just having my naive job DSL that I'm configuring into pipeline instead Um, and on the topic of large versus huge. Well it depends if you have a hello world job then the Difference in size probably isn't that big or if you have like small jobs in Jenkins Then you can probably go directly to pipeline instead without doing the iterations But I wanted to try doing the iterations I had this idea in my mind that by the power of the configure block I could unfold the xml and I could iteratively convert it to job DSL with using that as my elbow grease and um I I tried it and it turns out it is just crazy tedious. It's very time consuming But it's a super safe way of doing the conversion because you are taking the small steps and software developers sometimes like doing small steps um to like have a safe way of refactoring but um At other times you might be able to comprehend taking bigger steps at a time and getting further at a time so um Without further ado, I'm going to show you uh the job DSL iterative demo and converting six lines of job Of xml to four lines of job DSL with only 20 slides and how to put an audience to sleep Um, no, I'm I'm not going to show you this But I have the chapter um so If you want to take a look at it, you should Totally do it. It's nice to know that you can actually do this And I created the demo Uh, it's here in the job DSL iterative folder. You can see the different iterations So this is a hello world job And I'm going through different iterations and I'm converting it in to job DSL to the native job DSL Uh, I have the animated slides so you can see I'm going here. I want to change my description um and During these slides. I I changed the xml specification into some nice job DSL It's nice to know that you can do it. You yeah Um, I tried it So why is this iterative approach nice? Like just to repeat it if If you you only have this representation of your job the the ui representation, right? The other representation is the config xml So if you have a very complex job that you want to turn into job DSL You either have to do it in one go from the ui Or you can wrap it in the Configure block and then you can unfold it Do it if you have to it's nice to know that it's there I'd like to show you the job DSL api Um, because I think it's super cool what they have done Um, so they've actually made this interactive page where you can click around through the api So I can find the job clause so When it's loaded it takes a second Uh and under job I can find so I know that a job has steps so I can find steps And I can even open here so I can see examples And I can find the shell step It's uh down here and I can click it so I can see an example job that uses a shell step This is very cool I think that is a thing that is very very cool Is that it actually tells me apart from the limited build an api job DSL supports many more Jenkins plugins at runtime The complete api is available in your Jenkins installation On this url. So if I go there on the sorry On the Jenkins that I am running When it's loaded after 30 seconds more depending on the number of plugins you have it'll actually show you The support it has for your plugins And this makes it very nice to work with job DSL so The the api browser like the the the the basic one is great The one that's on your Jenkins is absolutely great and very relevant for how you can for what you can do on your Jenkins So that is very cool So great work on the user experience in the Jenkins community um so I I had this I had this thought I had this dream of of doing this this way of working right um but and just So so I had the naive job DSL that I've wrapped in the the config XML that I've wrapped in job DSL I do the iterations with the configure blog and unfolding I get the native job DSL and then in the end I have two DSLs So well defined DSLs for how to do work in Jenkins and I should be able to relatively easy translate one into the other in one go so It turns out that This is actually not the way I would recommend working Rather, I would recommend That if you have a lot of jobs a lot of complex jobs that you can't directly translate to native job DSL or pipeline Do the naive conversion Wrap them in job DSL wrap the XML in job DSL. Just Get them as code And then afterwards go back to the UI and convert them to the native job DSL or convert them to pipeline and uh Native job DSL. It's well defined. It'll work for all plugins Pipeline is well defined and it'll probably work for your plugins I just want to elaborate on that. So it's it's going to work for all your plugins with native job DSL because of the configure blog Because the configure blog can interact directly with the XML If the authors of the plugins haven't created the DSL Then you can just write the xml directly and configure the job and you can get inspired by what xml you should write By looking in the config xml of a job you've clicked It's tedious It's hard But it's nice to know that you can do it. So you can get everything as code What I would recommend that you actually do is to use well supported plugins that are actively maintained that are updated and Probably have support in pipeline and that you get your things converted to pipeline but use job DSL If you can't use pipeline Don't stay at manually configured freestyle jobs All right Yes So the path I chose the path I did for my project Is that I first converted to the naive job DSL and secondly I converted it to pipeline So talking about pipeline. Why not? Why why am I talking so much about pipeline? Right, I've already told you the cool things job DSL can do you can wrap an entire Job in it. You can Configure any plugin that you have in Jenkins and job DSL. So why do you want to go away from it? Well Pipeline can do a number of cool things. So from the Jenkins documentation You get the things as code you get better durability you get the ability to pause jobs that are running They become more versatile, which means you can Do more stuff with them in the world of continuous delivery And they are extensible you have it as code. So you get all the benefits of software design patterns and templating and shared libraries The things that are interesting here is the durability and the possibility So because we already have it as code with job DSL. They are versatile. We can do anything with the groovy language They're extensible. It's a programming language. We can we have the templating but the durability A pipeline can be automatically restarted when a Jenkins master restarts And it can be resumed from a point So you're not actually you're you might not even be losing jobs if your Jenkins master is going down When it comes back up your job will continue or your job will restart You have the possibilities you can wait for user input I'm not sure that you can do that in job DSL Um, I think you'll have to trigger some downstream drop jobs manually and then maybe reuse artifacts You can hack it and job DSL, but it's nice that it's natively implemented in pipeline that you can take this user input halfway through a pipeline Um, the documentation is here. So why do you want to go to pipeline? And uh, they of course add the as code as well But it's for the durability and the possibility. I want pipeline rather than job DSL All right. So how do we do the pipeline? Well, here's a crash course a hello world pipeline First of all, I have the pipeline tag I'm specifying that I want a new pipeline I give it agent any because my pipeline doesn't have any specific requirements as to where it should run It has a number of stages The first stage is a hello world stage The hello world stage has a number of build steps This one has a single build step. It's a shell step and the shell step is echo. Hello world all right um so reiterating the topic large versus huge well here the difference isn't that big, right? um So there's two points to make here You might want to go directly to pipeline because we're just using the shell build step It's natively implemented in pipeline. It's there The second point I want to make is the difference between the two so they actually look quite similar Which means you haven't wasted your time if you are converting your jobs to job DSL in the first place Because you can't go to pipeline Because at a later point you should be able to make a relatively easy conversion between the two It's two well-defined DSLs All right, so I know that I can get my code as pipeline now Excellent, but how do I get my pipeline? into Jenkins um Well, I want to use my hello world job DSL wrapper, of course, but I don't want to wrap job DSL anymore. I want to wrap a pipeline so Instead of creating a job I created a pipeline job And then inside the pipeline job. I added definition. So that's the definition of my workflow I added from the cps. So that means a groovy continuation passing style. It's the way I'm writing my code in groovy And I'm supplying it with a script. I'm just going to inline it It looks like this with the hello world job from before And it totally works I'm going to show you the demo hello pipeline inline I'm going to show you that the job DSL from the slide works so that we can actually just wrap a pipeline in job DSL And I have the disclaimer that the script security is still off Because it makes it easier to demo things So the demo was called hello pipeline inline. All right So To show that we can configure a pipeline job with job DSL It's going to tell me to create a new freestyle job in Jenkins and call it demo seed. Hello pipeline inline So I'm going to find my Jenkins and Can I close that one? I probably did because I opened the I opened the job DSL API and then I closed it afterwards. All right, but here we have it so I want to create a new job and I want To be called demo seed. Hello pipeline inline I'm still creating a freestyle project because I want to wrap job DSL The job DSL that I want to wrap is in the seed. Hello pipeline inline groovy file And the last thing I did before We started today was to Move all of the code into separate files because I thought that would be easier But in retrospect, it makes it way harder to Have all of the files on the same page. So, yeah We learn While we do So I have added the pipeline The job DSL pipeline job It has the inline pipeline I've created the seed job and I can run it to generate my job If I reload it, I can see that it generated my job. I can run it while I look at the configuration and It has a pipeline. So this one doesn't have a built step This one has a pipeline block where it has a pipeline configuration And that's because it's a pipeline job And when I look at it after it's run, it has the first first stage. So the hello world stage It has I can click here And I can click on the locks. I can see that it echoed hello world awesome Oh back into the presentation. There we go So I have my hello pipeline inline job With which is my pipeline in a job DSL wrapper I've added it by the demo seed job DSL wrapper job um But what I actually wanted was to have it in a separate file So of course just like we did with job DSL. We can make this construction as well So this is what it looked like when I had it inline So I have my hello pipeline inline job with the pipeline code inlined in the job But I can also reference using the cps scm So I'm referencing that my Jenkins file it comes from some git repository And then when I check it out the Jenkins file should be in my workspace And I can create and I and I have the job specification from that So that's uh, how the job will be created all right and my hello pipeline I can reference that in my hello pipeline seed job either directly Or with wildcarts to get it dynamically but just for the construction and I'm done I have the I have the I have my Jenkins pipeline file. I have it referenced from my hello pipeline job DSL wrapper I have my job DSL referenced in my seed job DSL so I can reference a number of jobs I add this manually to Jenkins But that's okay because I'm just adding one job and it's going to add my entire platform of Jenkins jobs, right? so um, I would like to introduce you to Freestyle jobs and the five stages of configuration as code Or the configuration as code model. So it's a bit different than the model for grief or kubla rush model, but please um Stick along So the first step is confidence. I feel confident I know who changed what when and from which values Because it's in version control I'm using well defined workflows that we use for software development in my example. I'm using git So I know what changes have been made And what the current state is without having to go into Jenkins and figure it out I feel value I feel valued I can reuse Through shared libraries with pipeline. I can use software design patterns because it's groovy. It's programming language I don't feel like I'm wasting my time. I'm not doing tedious tasks Clicking around in a UI just to try to keep up with whatever configuration I want my pipeline to be in I feel relaxation I have the automatically reconfiguration of jobs So I don't care about my Jenkins master if I have it in pipeline I might not even care at all what my Jenkins master is doing because it's just gonna it's still gonna restart my jobs and run them It's still gonna have my things I feel collaboration Which is not an actual emotion, but it should be I'm working with my colleagues. We're not working against each other and I'm enjoying working with my colleagues because it's not It's not like we're on a battlefield. We're actually working with each other And I feel happiness because this is a great way of working so Yeah, we did it We have we are from from freestyle manual freestyle jobs Into pipeline and we are adding our pipelines with job DSL and we're managing our job DSL in the same way Nice so Does this get better It actually does even Because so I told you about this change by addition change by modification um I of course want to use the wild cards So I just want to have my job DSL seed jobs or the sorry the job DSL wrappers in a git repository And I want the super seed job to just reference this repository. Give me all the job DSL in this repository And it gets better Because I can put the super seed job into the repository. Sorry And it can reference itself So it's constantly keeping up with the current configuration of it And I have the configuration of my super seed job under version control And it gets better as you saw when I accidentally clicked on the slide Because I don't have to manually configure my first super seed job I can add that to Jenkins if I'm using Jenkins configuration as code So no clicking at all just a readily configured Jenkins with all of my jobs But that's a thing for earlier. So we actually did last month um An online meetup the configuration is code of Jenkins for Kubernetes Uh in parenthesis moral of the story being if you just get your Jenkins jobs As pipeline job DSL and if you just get your Jenkins configured as code Afterwards putting it into Kubernetes is is trivial So Thank you Any questions Yeah, thanks for your presentation Nikolai Um, yep again if you have asking if you have any questions, please ask using corny feature in zoom And we already have a number of questions. We answered some for some offline Wonderful, so one question from blood. It would be interesting to see here pros and cons of using job DSL just Jenkins pipeline right so The pros of using job DSL is that you have everything configured as code It's an old way of doing it. Which means it's tried and true It works and it's going to work for all plugins because you have the configure block The pros of doing it with pipeline Is that you get the Restart you get the resume you get the durability of your jobs if the Jenkins master crashes Your jobs will might even still be running and able to pick up where they left um So yeah That's the so the cons of using job DSL. You don't get the durability of pipeline the cons of using pipeline It might not be implemented for all the plugins that you use But you should of course use updated and will maintain plugins great Anything to add Oleg No, I think it's a great summary. All right Okay, but yeah personally I use pipeline almost everywhere But yeah, I do appreciate really of job DSL, especially In conjunction I use jcask Okay, the next question Just like concourse that can we do all of these great pipeline right pipeline et cetera through common line Poof, that's a great question. Jenkins has an API. I know and you can fire post requests and get requests to it so yes but What I'd like you to do instead of Using the command line because now you're just sort of like scripting your way out of your creating like long shell scripts that will create all of your pipelines I would like you to create the pipelines in job DSL And then reference the job DSL in the seed job and then reference the seed job in your jcask configuration so When you start up your Jenkins You're not firing off your Jenkins post install script and waiting for that to continue You're just starting up at Jenkins. That's configured With all of your jobs and all of your plugins Oh like you're muted If anyone is interested in Jenkins CLI Share the link in the chat So your Jenkins has extensive CLI actually now we have two implementations one is official in Java another one is uh unofficial in gold but you can take any of them and You can create job DSL jobs so that you can invoke them to Generate final jobs. You can also launch them. So in principle, yes, you can do anything Using CLI Okay Let's move on How can you change a job that is already migrated to job DSL? So I think that uh All like you had you wanted to do another meetup at some point referencing some tools That could help you Oh, yeah, I can actually a quick Show them Just a second. So in Jenkins, there are tools which allow Doing some immigration. Uh, for example, Nikolai has presented tools For job DSL, but also there are similar tools which allow you to migrate to pipeline directly So for example, I'm showing you declarative pipeline migration assistant So it was created by Liam Neumann and Olivier Lamé And this tool actually provides some API which Supports the implementation so you can generate a declarative pipeline code that is similar to a scripted pipeline if you wish Um, and then you can lure them with job DSL so that you can incrementally do changes if needed and So do all the changes as code so Yeah, I hope it answers the question. Thanks for that Uh, so let's move on. Um, the next question is have you tried the same for maven jobs? For maven jobs. So instead of having my manually configured job as a freestyle job, I would have it as a maven job I have not tried that Um, but I would expect that if the job is in If the if the job is available in Jenkins, you could find the config xml and you could convert it using the wrapper and I would expect job DSL the api to have The to have a maven job Just like it has a job Yep So in principle maven jobs and freestyle jobs are quite close. They use the same internal abstraction called abstract project It might be a bit more difficult if you want to make that for example Promotion jobs from promoting builds plugin though. There is still some integration with job DSL And you could use that but yeah all classic jobs should be Supporting job DSL. Well If he is inheritance plugin or something like that. Yes, it's a completely different story, but for maven jobs It shouldn't be a problem So If I just quickly share my screen then You can see here in the Jenkins job DSL api You have a maven job on the left And if you click on it, you can see examples of what a maven job looks like Yes All right Okay, thank you So the next question Or would it be possible to define the dynamic As a job with jcosc under a bar result manual steps Yes, it is and Yes, it certainly is and Uh I even know I know the person who's asked this question, but I didn't tell him to ask it So I actually did this in the last presentation and if we look at the so This is available on the Jenkins online meetup as well is as a previous presentation, but So this companion repository had a demo for For auto bootstrapping a Jenkins configuration as code with the job And uh looking in the advanced examples of the Jenkins configuration as code If we look at the Jenkins configuration as code yaml file You can see that you can actually supply a job So I've supplied one job, which I've called supersede This supersede job References a url. So this is the configuration as code Jenkins as job diesel companion repository It uses credentials because it clones over as a sage and then it runs all of the job diesel in the repository And if you want to see how you automatically trigger this as well Uh, it's explained in the in the other video Thank you for that question Do so we answered uh six questions. We start from six questions. Now you'll have nine to go Oh wonderful Okay, let's continue. We have some time. Uh, so the next question is from ele What testing framework can you recommend to test the Jenkins as code with job diesel and pipelines? Oh, that is a great question. We actually talked about that internally Um, Oleg, do you have an answer ready for that? Well, I have some answers because we do some testing. Uh, for example, I Yeah, I can probably Show some examples Just well, I cannot show good examples for what it was Uh, but yeah So, yeah, let me share my screen. Do you see it? Yes So, uh, there are some foundation bits for testing. For example, if you want to test pipeline, there is a framework called pipeline unit so Jenkins pipeline unit this framework basically allows doing some unit testing for Jenkins pipelines It doesn't allow doing job diesel testing For job diesel testing, for example What I would use and what I'm planning to document in the next future. There is a framework called Yeah Jenkins file runner Test harness So, please don't be afraid about the name. So Jenkins file runner is a completely different project. It's fast capability for Jenkins But good news is that if you follow full configuration as code with jcast, with job diesel, with this pipeline, you can Run your project not only as classic Jenkins, but also as Jenkins file runner. There are tools for that And you can use this framework for integration testing So, uh, it's what I'm currently working for Jenkins pipeline library But yeah, basically this framework allows you to package your Jenkins instance to Jenkins file runner And then run this instance as a common unit test So it's quite handy. It's under development But if anyone wants to try please feel free to reach out to me and I will be happy to share some examples And I'm planning to do a Jenkins file runner demo maybe Next month at the Jenkins online meetup So I will be happy to share that Okay, I hope it Answers the question a bit But yeah, work in progress, but we are well aware that this area is important and in the Jenkins project we want So for better tools for that Okay, so I'll share the links afterwards. And yeah, the next question is What are the best practices that our dependency held these plugins? Yeah If it relates to the talk, but I thought that this is important to briefly discuss this question Yeah, sure I don't have a good question Don't install a lot of plugins that you don't need There is I think there is a plugin that will tell you which jobs use which plugins And then you can export that and then you can try try to figure out. Do I have any plugins installed that I'm not actually using? um So I would recommend use as few as possible to solve your to solve your problems I'm used to the the phrase dependency hell coming from Conflicting dependencies and the simplest technique I've ever seen was use the plugin manager itself and let it handle Worrying about satisfying the dependencies when I tried to manage dependencies outside of the plugin manager in Jenkins I tended to have much more of a challenging problem that if I let the plugin manager do the job Yeah, plus one and right now we are working on a better plugin management tools So if you are following Jenkins online meetup, uh, last year we were presenting a plugin installation manager Plugin installation manager tool which already offers some tools for advanced diagnostics for example showing coverable updates um Also Showing security warnings and other things. So this tool is already available. You can use it It has its own limitations, but you can try it for better plugin management Meet plan to keep adopting features from that. For example in official docker images so that they have better analysis of Transitive dependencies, etc It's in our plan Same, for example, if you use tools like customer packages that are company some company Some features which verify compatibility of plugins It's an advanced packaging tool, but it can be also used for configuration as code And yeah, this what we have in the development for users But we also try to address the plugin management on the Developer side. So for example, yesterday there was a meetup with james nord. He presented A bill of materials and new enhancements in plugin forms for plugin developers So it's not something like we totally rely on users to properly verify dependencies We also invest in the Jenkins tooling so that plugin developers can ensure that there is no binary conflicts and then that we can ship Bit and most stable plugins to Jenkins users So if you're interested you can go to the online meetup page and you can find the recording here There are more tools for Jenkins plugin developers Available and hopefully you will have more online meetups about that soon Okay, so yeah, it's a long question, but There are different different tools we shall have to check that Okay, our next question is apart from Ruby. It does We support any other language and this job this and pipeline um Oleg Do you know that? I actually I I'm not sure I expect not I'm happy to answer this question So Yeah, it's not me who asked that but I would be happy to ask it on my own So for pipeline, you may have seen the recent news that you now have pipeline as yaml plugin, which is in in the incubated state So it's On the adoption and if anybody wants to try it out Please do so and the surprising way supports defining pipelines as yaml For job this is right now. There is no such capability But technically it would be possible to implement that And I guess you could raise feature requests and maybe contribute it on your own So practically it's possible Yeah, it has a lot of Potential difficulties if you want to do that for pipeline. Yeah, we are we would welcome any user feedback You can install it. It's available in main Jenkins update center. So you can just find it And if you're hitting issues, yeah, there is A bit of issues in this project. So Yeah, please try it out and let us know what could we improve them That looks very cool. Actually Yeah, it's a new feature coming from kubernetes, which is cool and in yaml Yeah So, yeah the result of ongoing changes in Jenkins and If they will keep shipping changes Okay, so we answered more questions and we still have 10 questions to answer So what now? Yeah, would you like to go to continue for another maybe 15 minutes? Uh, so that we have answers on the record Sure Okay, let's do that and thanks to everyone who is still with us and Maybe recording these questions if you have to do So the next question is it will be easy to maintain Jenkins file and not the dsl files Have a sure dsl but keep all the flow in Jenkins file um Yes and no So I would like to show you because I actually have an example of this From the last meetup that we did with how it looks like when you're wrapping these Jenkins files So I'm sharing my screen. I'm in the A companion repository for the other demo. So it had two other companion repositories So I actually had my My seed job repository with all the job dsl and then I had my project repository, which was a Mono repo just for the purpose of the demo. So each of the folders they have a Jenkins file It doesn't matter that they are in the same repository because they are referenced individually from this job dsl repository So the job dsl repository has the supersede job Which reads all the job dsl in this folder? And it has a folder for each of the pipelines. So the auth pipeline, for example Has an author groovy Which references an auth pipeline job? It has a script path. So it's in the auth folder of the pipeline repository And if I just move this to the side You will see that if I look at the basic groovy It still references the pipeline repository Um, and it uses a different script path So sure, this is this is groovy You could wrap this in a for loop and you could try to figure out What all the files are in the In the in the in the pipeline repository. Sorry that was over here And then create so you could generate the The job dsl with the job dsl But I found that this is easy enough So I just have my supersede job and then I have a seed job per pipeline I'm also not able to add Multiple pipelines in the same job dsl job because then it will just use the first pipeline that it finds You could use Uh, what are there are different job types in Jenkins. There is the multi-branch pipeline job Where you're actually adding an entire project. So if you're using bitbugger or something like that, then it it says that for all of the projects For all of the repositories in this project I will have a Jenkins file in the root So whenever I create in a repository just assume that it's going to be there And then Jenkins will automatically create it for you And then even when you start creating creating branches feature branches things like that It's going to pick it up automatically in Jenkins Yeah, but then I would have my multi-branch pipeline in job dsl But I would then have one for the project in bitbugger or or other Yeah, it makes total sense Okay, next question. Is there an easy way How to avoid just keeping characters You're from some advanced shell scripts not only about job dsl but also about pipeline Mark Triple single quotes Right, isn't the only solution triple single quotes or triple double quotes And I'm not sure but even that doesn't really solve it because it's still your language inside a language And then when you're generating job dsl that's generating job dsl that's generating job dsl you're I've had 16 backslashes in a row because I was nested that deep Yeah And unfortunately, there is no good way to do that. Yeah Jenkins itself. It has some Features which escape things properly Well, or sometimes properly for example shell step. We'll try to escape the string Etc, but still all of it has a lot of limitations But but for the most part the multi-line string will help you a lot Yes Okay, so the next question Get other process wise. Is it possible to arrange a workshop to practice such things together? So Yes, organizer of Jenkins online meetup I think you would be happy to host such workshop if somebody wants to conduct that And maybe Nikolai does if you could product more than I such workshops online So Yeah, so we don't have any things like that in the pipeline, but Also, it would probably be a paid event. So when I'm doing things for effort code. I'm actually on the job and I'm Sort of this is borderline what we're also helping customers with so I'm not entirely allowed to give away all of the business um, but uh, but yeah, if there is Interest in doing such a thing then I can certainly ask Someone if I am allowed how much I am allowed to to do Yep, so if you're interested, please reach out to Nikolai and Twitter. So It might be a public workshop. It might be a commercial workshop, but if you're interested Definitely It's a good thing to ask Okay, so the next question When using pipeline using git checkout, uh, we are reference to a single Jenkins file located In a root git source. How can we store multiple pipelines in a single report? um That's a good question You can create multiple pipeline jobs that reference the same repository and job diesel Because you're not entirely able to reference multiple Jenkins pipelines Sorry, yeah multiple Jenkins pipelines from the same job in Jenkins Technically you can but yes, it's set in the open source commands There are all options I'm not that free though Yeah, by the way, there are plugins which you simplify that for example a remote file provider plugin and other things which can help you to dismantle multiple locations of Jenkins files and pipeline repositories, but the multi branch is quite limited by default Well, I would say opinionated but Was quite limited. Yeah, actually I I I challenge the phrase limited and instead say opinionated because I agree it is opinionated. It's and it's intentionally so Okay, but uh, yeah, there are plugins which allow to alter the behavior so for example, uh I don't share a backend a maintainer of pipeline as YAML plugin is also a maintainer of Remote file provider plugin for multi branch pipeline and if you're interested you can try it out for this use case Okay, so The next question can I fetch a job diesel from a normal htpsm point instead of github? I guess yes um If you can get access to the file and you can get it into your workspace Then you can reference it because it'll be in your workspace. So what i'm doing with the With the git tag in the job diesel is just to clone out a repository And then I have the files in my workspace But if you did a thing like You could uh, you can of course like create no files You can cut to files. You can curl download files Um, which you should always do authenticated And then you will get them in your workspace And then you could reference them as external files from the seed job diesel I took from your earlier instruction that it's really helpful to have multiple files defining And so whatever technique I use to get those multiple files in Is great, but Multiple files rather than one great big monster file that defines every single job in the whole system Yes, so change by addition rather than change by modification Because if I have this one big job with all the customizations in it, then it's very likely very likely more likely That I break some other custom configuration when i'm adding a new custom configuration Yeah, that's right Yeah, probably a job diesel could provide More different engines so similar to how pipeline provides different sources, etc Job diesel could also support the additional extensibility modes But I'm not familiar to this to the plug-in code base So I cannot say for sure whether it's already supported or could be implemented But if it's not supported, this is definitely something You could contribute to the plugin Okay, but that's not necessarily a limitation of job diesel though because like once you have it in your workspace You have it externally So whichever thing you can do to get the to get your specifications into your workspace like curl it from Some location copy it from the disk anything That you could do in Jenkins before the build step That is a processed job diesel script That will Let you read any job diesel into a seed job diesel. Yeah, that's totally correct Okay, so we answered two questions at once So the next question is We have a launch number of freestyle jobs that already in job diesel and we would like to move to pipeline jobs What would be your strategy? um I would Look at similarities in the DSL and then I would Wrap the pipelines and add them and then I would verify that they did the same thing either by Looking in the configuration of the jobs Which become a bit tricky um, because you're still like comparing a pipeline to an actual freestyle job that you have in job diesel um Yeah Yeah, would you suggest using shared libraries during this migration process? uh, yes If you whenever you have so like the first time you make something you don't know what it's going to look like the second time you Make something you kind of notice a pattern and the third time you should be able to generalize So if you have an overall way of working in your company or wherever you are that we always check out a repository We always clean our workspace. We always end up pushing locks somewhere Then as much as the general stuff you can put it in a shared pipeline and template the tiny parts that change And and work with that There is shared pipelines documentation with a couple of examples that show how you can actually create quite intricate custom pipelines and just parameterize them and then Yeah Create a number of of different jobs very easy using these shared libraries and using the templated jobs Right and also if you create pipelines you can test them So we partially answered the testing question before But for example, if you want to test pipeline libraries specifically We have a repository in the Jenkins project Jenkins Sympa pipeline library. So basically it's a pipeline library Which we use to test Jenkins and plugins and other bits So you can see that there is a lot of common steps which we share between our pipelines And here you can find examples of Tests which have been written using Jenkins pipeline units specifically for Building blocks which we use to migrate our pipelines Well, we migrated long to go into southern 16 But still we extend to this pipeline library and add more and more tests and Tests and features for that. So you can check this repository if you just want See a real world example of pipeline library with test coverage And there is also a pull request for integration testing these Jenkins fellow runner Yeah, not finished, but it should be some way here Okay So We are getting close to 30 minutes for questions left. Would you like to finish that? Um, like we can continue are you? If we we can take the four questions Okay, so yeah, we'll just finish the existing queue and for the rest, uh, yeah, let's follow up Um, in our same channels if you're having yeah, please continue asking questions on There was the slack ask on twitter like We're love talking about this stuff Yeah, definitely And probably next time we will go meet up without presentation just a discussion questions panel of experts session Okay, we're growing into the presentation Yeah, right So, uh, the next question of what when you're at a job they sell to your gate Do you still need to manually run the seed job or can you run it whenever the job they sell report updates? Yes, when I add a job they sell to my git repository I need to manually run the seed job, but that's because I didn't configure any triggers So it doesn't trigger the job on changes in my source code management But if I configured a trigger I could rather configure it to look for changes every second minute Um, that's a bit expensive Otherwise, I could configure it with a webhook to my source code management And then whenever my source code management changed It would fire an event to Jenkins and then Jenkins would notice Oh, this job needs to run and then it would rerun my job DSLC job And you should totally do the webhook trigger thing Um, but if you like me Don't have your own github or your own big bucket running Then you don't necessarily know where your code is actually going to be so it gets hard to configure up front And I also don't have a Jenkins running right now. My Jenkins is like all my jobs are as code My Jenkins is as code. So when I need my Jenkins, I started and then I turn it off afterwards Um, yeah Thank you So yeah, the next one, uh, how high how uh, Jenkins can discover parameters in Jenkins files without running their job once So it's rather an architectural question Uh, so firstly it really depends on which pipelines you use scripted to declare it because they process the bit differently Uh, but yeah too long didn't read Jenkins does a best effort of discovering Parameters using the abstract syntax train. It may work or may not And depending on the result, uh, if you It will definitely cache parameters after the first one so Personally my recommendation is if you really have to have pipelines with parameters don't rely on the first one Yeah, so you You can do some things where you trigger the pipeline and then you build into the pipeline that it will early exit If the parameters are not provided Um, but yeah, there there is no good way for Jenkins to know upfront What the job is doing because it actually doesn't have the job before it Pulls the job and reads it So there are some edge cases. For example, you can put parameter property in a pipeline library Which you load dynamically using graph or whatever Uh, I'm not sure whether I would recommend this use case in the good conscience, but technically it's possible I love that part technically it's possible. Know that it's there Yeah Okay So the next one, uh How uh, yeah, how to prevent the job the cell from detecting my job Changed manually since it was generated by this job of the parameters Has been added after the first run of my job. So I guess it's A follow-up of these keys Never allow anyone to make changes to the configuration of a job that you have configured in job diesel If you need to make changes to a job you've configured in job diesel You're doing it for development purposes and then you shouldn't do it on your production instance You shouldn't hack around on it Then you are creating the job in another place where you have rights to edit stuff And then you edit stuff and then afterwards you Make the changes in version control And then you rerun the seed job and you have your changes in production Right and we're actually working on some my errors for that So you can see that uh, if you go to the change log Of Jenkins, you can see a lot of system read references there So basically we're working on read on the UI. Thanks to team jack up and other contributors who work on that So our idea is to have read on the system configuration job configuration and agent configuration UI There is a blog post which is staged right now We are getting there and if you're interested to participate again, we have online practice Next week and the actually Read on the UI is listed here So we invite contributors to work on this story so that we ensure that nobody simplifies configurations configured by configurations That is awesome. Yes Because yeah, the reason I want to go to configuration as code I like had it on the slide right the configuration drift I want to know when changes are made by who and why and have it under I don't want the random hacking around Yeah, so it's coming soon. And if you use weekly releases, you can already try it out Okay So happy to deliver the good news Even even better not just can try it out should try it out in weekly releases because jenkins 2.232 35 is next lts And so good time now to test it good time now to experiment with it So lts is coming with this capability. It's not here yet But it's coming. Awesome Well, there will be some incremental changes. So I think that it will be a fully finalized by september But yeah, a lot of changes are already available Okay, so the next question Oh, it's not a question. It's just a comment. But yeah, I think we're I think we're done Yeah So, yeah, thanks to everyone for asking questions. Yeah, again, we got a bunch of them So we will publish the recording. We will publish All the questions, etc. It will take some time to process that but the recording should be available in 24 hours And Yeah, I guess that's it. If you are interested to share any feedback again, we will appreciate that Because it helps us to improve our meetups. It also we will also share feedback with nikolai so that Uh, we can improve the presentations as well. And yeah, thanks a lot to nikolai for doing this presentation. It was really great a lot of Content to study and hopefully it was helpful To everyone who participated Thanks