 So, I think it's already the time. So my name is Peter, this is Shimon, and today we would like to present you and Jenkins migration story which happened at our company which is obviously OVH. So a few words about us, why are we here and why we are presenting this to you. So if I should describe myself, I would say I'm a passionate Python dev and a clean code enthusiast. So I mean programming is generally art and nobody wants to look at ugly art, right? In general, I would say I'm a 128% IT nerd. So what about you Shimon? Yes, and when it comes to me, I'm also IT geek. I'm a proud Arch Linux user, as you may be aware, you cannot use Arch Linux without telling anyone use Arch Linux. And I'm a big fan of open source software and open technologies at all. My ask why? Well, the answer is simple. The open technologies don't give you any unnecessary limits. And I believe that living in world without any artificial walls won't need, in living in such world won't require any additional doors, gates, or windows, so life is simpler. So as mentioned, we are working for OVH and we are part of the public cloud team. So what we are doing is mostly R&D stuff regarding OpenStack. So implementing new features, maintaining them. But at some point of time, we decided we should specialize. And we did it by specializing to Jenkins. So in terms of building new packages, automating the whole platform, and so on and so off. So I hope there are only a few people that don't know what Jenkins is. But to be precise, Jenkins is one of the most popular automation servers. So it's an open source project so everyone can contribute to it. And it's really powerful. Well, you may not believe me, but I've got some evidence. Look at all those top IT companies which use Jenkins, including Microsoft, Facebook, OVH, and Red Hat. And there are many, many more companies. So Jenkins is basically really extensible. There are plenty of plugins. Whatever you want to do, you can achieve it. Either you want an integration with Kubernetes, with Docker, or with OpenStack. Everything is possible. Okay. And it's pretty easy to start with Jenkins. I have a little evidence for you to that. First, don't be afraid if you don't see details on the images because they are here just to give you some perspective. And let's discuss what are the typical steps you do when you configure Jenkins. First, of course, you install it and you start it somewhere. Then what you do is open your browser and go to a specific URL, the specific address where there is the first time, first launch configuration manager. That will configure the basics, okay? Then you log in to your running Jenkins instance and add some jobs, adjust the rest of the things, and that's all, right? So what they are doing, you are mostly clicking to fields and checkboxes in the UI. It looks easy. But what's done in the background, in the backend, there is this XML file generated. You may already see what I'm referring here to, but let's be professional and let's discuss it. There is an issue with XML files that they are not very human-friendly, okay? It's often visible that XML files are badly formatted. Here at least we've got some indentation, so it's not that bad. In XML files, we have a lot of redundancy. So, for example, for each opening tag, you've got the closing tag. So you double the information, sort of. And in the generated XML file here, you've got a lot of not really necessary things, like the version of plugin that was used during the generation of configuration file. You might think, okay, this version might be important, however, to our experience, it's mostly there, the structure of XML file doesn't change, no matter what the version is. Therefore, imagine you would like to have a review of changes in Jenkins depending on such XML files. It would be inconvenient for people because they would be afraid of looking at such abominations. And versioning such files will also be difficult because, for example, when you will update your jobs through UI, and in the meantime, if there were any plugin updates, not only the thing you wanted to change will be updated, but also the version strings in the XML attributes. So this was the main issues we want to solve. So how it started? So it was one beautiful morning, and we were together with Shimon discussing some new changes to Jenkins, and then suddenly our boss showed up and said, okay, boys, it's everything all right. Those changes are really good, but I see a problem in here. So can we perform any kind of code review? And can the rest of the team participate in those changes? So, well, we were thinking not really. So we needed to change it. Yes, and this also pushed us to further thinking, and we thought, hmm, we've got a friend. He's called Maciej. Maciej is doing a lot of amazing things, and whenever he invents something, he comes to us and says, hey, guys, I did this and this. Could you please create for me now a job in Jenkins that will build package or something? And we thought, hmm, okay, we are doing it for him, but what we are doing are rather basic steps. So maybe he could be able to do that by himself, but he is a programmer, and we need to find some programmer-friendly way to do that. In order to fulfill our goals, we decided to use free plugins. So we decided to use configuration as code, which regards the main configuration of Jenkins, then job DSL, which contains the definition of jobs, and last but not least, job pipelines, especially here the Jenkins files, which contain all steps of jobs, so what they are doing. And now we will go through each of them and explain how it looks. Okay, starting with configuration as code plugin. Actually, the name was pretty recently changed to Jenkins configuration as a code, so there is this additional part in the beginning now. And what can I say to you? It's a relatively young project. The development of this extension was started about a year ago, and about two months ago, there was this first stable release 1.0. And despite its young, it's very powerful. That we've seen on Jenkins Dev mailing lists that the developers want to introduce it as default extension in future Jenkins release. What this extension does, as Peter said, it allows you to define basic global Jenkins configuration things. So for instance, the name of the Jenkins instance, the welcome message, number of executors, credentials, some authorization things just like that. And with this extension, you are able to set everything using nice and tidy YAML files. Okay, Shimon, that's enough theory. They want to see some examples, right? Here you go. So, a pretty easy example. So what you see here are the basic Jenkins configuration pages. So what you can configure in here. So something like the system message for the number of executors of Jenkins. And on the right, we see how it's mapped in the YAML files. So as you can see, it's pretty straightforward. You just say, okay, we want on Jenkins number of executors set to 32. Okay, much less information than in the actual inputs and labels. Much more condensed, right? Also, Jcast has some integrations with existing plugins, like here, for example, the Fronting plugin. So here we define two Trotting properties for two sample preprots. So we say, okay, we've got four and two concurrent jobs that can run on these preprots. And we just map them into these YAML files. Jcast also has an integration with the JobDSL plugin which we will discuss in a second. So you can say now, but wait, you said that Jcast is only about the general configuration of Jenkins. And yes, you're partially right, but in here we can define also the jobs. But we don't really recommend it because each change that happens here needs to be approved using the UI in Jenkins. So something we would like to avoid. But here we defined some special type of job called the seed job. Its responsibility is just to read and file here given by this path and then parse out all the definitions that are passed in the JobDSL plugin and to create them. Okay, and then the rest of the jobs are defined using the second plugin. There comes the JobDSL. First, I would like to explain what is the DSL. If someone don't know, it's an acronym for domain specific language. So it's sort of meta language that is defined on top of other language to make some lives easier, okay? So it adds some additional function, some syntax sugar to achieve your goals. In this example, to define Jenkins jobs. And I would like to point that it's already, it's not a new invention, it's already pretty mature. The development was started a few years ago. And with JobDSL, as the name suggests, you can define basic job properties. Things like name, description, schedulers, triggers, et cetera, et cetera, okay? What this plugin does, it mostly maps things you can click and set in UI to groovy function calls. However, occasionally it may happen that there will be something missing in JobDSL that you can set in UI. What then? You don't have to throw away JobDSL. For such rare occasions, you have a XML integration so you can directly using JobDSL manipulate XML files and write the specific lines directly into resulting configuration file. Okay, so time for another example. So what we are defining here is a pretty easy job which launches unit tests on each changes that happens on Garrett. So what we can see here, okay, we want this job to be located in the review folder. Its name should be unit tests and it should execute steps that are contained in a Jenkins file called unit tests. We'll come back to this in a moment. Also what we see here that this job should be triggered using Garrett on projects that match this regex and the type of events that it should trigger on are creation of new patch sets and if someone types in a comment that contains the recheck string. So as you can see, pretty straightforward, right? Okay, but hold on, dude. It happens that I know JobDSL and there are some parts that are not mentioned in documentation. Are you trying to cheat me? Yes, you're partially right. So as Shimon previously mentioned, JobDSL is built on top of Groovy. So this allowed us to wrap the whole API into some custom functions which allow us to remove some code redundancy. So a few functions that we defined. As you saw previously, this was this basic job function. It is basically a wrapper over the pipeline job that provides JobDSL and we are just passing some variables and some, like for example, the SCM, so in this case the Git repository, is hardcoded in here because we use the same for all those jobs. Also we've got functions to define periodically triggered jobs. What we do here, we use this first function and then just add it a trigger which uses Chrome spec. Also we've got functions to manipulate the underlying XML file using Groovy's Node API and some functions just to create some folders based on a list of folders. Okay, and Piotrk mentioned that we created a job. We specified that steps in this job should be, are defined somewhere and here we come to the third plugin. It's called JobPinePines or Jankin Pipelines. This plugin basically adds a new, completely new kind of job to Jenkins called Pipelines and with Pipelines, there is introduced a completely new way to define jobs. Basically you define steps of the jobs using text and this text can be stored in our files called Jenkins files and what's important here that these files can be located outside the Jenkins itself, okay? So we can store them, for example, in your source called RepositoryS for specific project. Something like it's done in GitLab with GitLab CI YAML files, okay? Another advantage of Pipelines is that they allow reusability of previously defined Pipelines. So for example, if you have few projects and you test them the same way, for example, OpenStack projects mostly come on, you just launch talks after installing some dependencies, right? So you have four projects, glance, nova, sender, neutron. You want to launch unit tests for them. You need to define one Jenkins file with all the steps and then in five Jenkins jobs, you specify just load this Jenkins file and execute steps from there. Then if there is a need to update, you just need to update one place instead of four, five or more, okay? And one important notice here is that when you will be looking at Pipeline's documentation, please be aware that there are two syntax variants, one called scripted, the more traditional one and more similar to groovy. And new one, declarative, it's more similar to YAML. It's claimed to be more flexible and powerful. And last important notice is that as Pipelines are completely new kind of job, the existing plugins, if you want to call them, call them, have to be Pipeline's compatible, okay? Thankfully, there are more and more jobs that are now compatible. So it's not that much. Okay, so time for another example. A pretty general one. So what we see here, okay, we're declaring here a Pipeline job. So a job needs a place to run on, right? So we say, okay, here it's called an agent. We use any agent that is available and then we're executing some stages. So some blocks of logically, of some instructions that need to be run together. Yeah, so here we define four stages. So the first one is to call a new Vertel machine where we will launch our tests on. So we're executing some shell script, which basically is a wrapper of us above Nova itself where we deploy a new VM, wait for it to become online and so on and so on. Next, we need to prepare this Vertel machine. So we're executing some SCP and some SSH command on that machine. Then we are launching the test itself and on the final end, we need to clean up everything. So we are saying, okay, execute Nova delete on this VM and everything's all right. Okay, I would like to add one point here. We moved everything, every bash calls to scripts that we are putting inside the VM we spawned and we are just executing the script. You may ask why? So I'm answering now. There is a tricky way that Groovy handles strings and mixing in Jenkins files, bash calls and Groovy calls is kinda weird, okay? So don't bother about it, just put it in a script, load them and your life is much, much easier. I spent too many days doing work around on this. Okay, now. So we introduce you the three plugins you can use to define Jenkins configuration easier. And when you will use these three plugins all together, you will eventually find out that you have out of the box pretty good workflow for automated updates of Jenkins configuration. Let's see how it will work. Let's imagine we've got a friend called Pavel this time. Pavel is our very active teammate and he wants to make work better every day. So let's imagine he wants to add new job, for example. What he has to do, he has to clone the Jenkins configuration repository. Now we have it and then just introduce a change there and propose it to our review system. We use Garrett for this, okay? What happens then is regular review of the change. So some teammates are agreeing with the change. Some people disagree and some people just don't know what to say. But eventually after some improvements maybe the change is finally merged to the Git source code repository, okay? So we have the new configuration in Git. What happens at the same time? There is this Garrett trigger plugin launched on our Jenkins and this plugin catches, there was a merge event on Garrett so it triggers a job. In our case, the job is named refresh jobs. What does this job do? It's fetching the changes from the Git repository, the newly merged changes and then it places it somewhere on Jenkins so it would be readable by the second job and then it triggers the original create jobs and the job which result in re-applying the old Jenkins configuration on our Jenkins instance. Easy as that. So we saw what plugins we use, how our work from looks. So now we want to see how are the changes on each of the steps. So we prepared some examples just to show how a change would look like to the general Jenkins configuration to creating a new job and to modify an existing job. So here we go. Okay, first simple example is global configuration change. Let's don't consider here even the change in UI. Let's focus how it will look in review system and your review system or if you will decide to make that change manually if you will use XML files, you will have to go through all these long XML file. There are many sections besides global Jenkins configuration like credentials, view, some things. And well, you need to find this value you want to change, change it, propose and people won't be really happy to review it. In the CASC example, you can split all the configuration across few YAML files. So for example, view configuration, credentials configuration you have in separate YAML files and in one small, simple YAML file, you just keep the necessary things you want to adjust. So changing it here, reviewing, seeing it's much, much easier. Okay, that was not that spectacular, right? So now something bigger. So imagine you want to add a new job. What you need to do is prepare the whole folder structure, create some XML files for the folders and eventually create a new definition for a job. So as you can see, it's kind of a mess. A really long XML file, not everything is really visible in here. Well, nobody would actually look through it during the cold review, right? So I have bad news for you because that was only half of the file that you need to declare. So also going through here, also a mess, you need to look through it, make sure nobody will break everything. So how it would look like using job DSL? Wow, only seven lines of code. So and we clearly see what we're trying to do here. We're trying to create a new unit test job for Nova using the master branch. Then we would like to execute things that are in the Jenkins file with the unit tests and trigger it according to that Chrome spec. Once a day. Yeah, once a day. Okay. So it is that little of code that will even fit a little cat in here. Okay, Peyton, now. It looks nice. However, I see you only need to launch the tests for master branch. And as you said, let's do something more spectacular. How it will look when I would like to check to launch unit tests for all recent OpenStack releases. Nothing easier than that. You remember it's based on Groovy, right? So what we need to do is just define a list of branches that we want to use and then iterate through it and call the same function just with one field based on a loop variable. So there's even that much space that we can fit now six kittens in here. Okay, amazing. And then the last example, modify existing job. So here we get this example, how it will look when you will have traditional freestyle jobs kept in XML file. And when you've got a Jenkins file. In the first example, in the first case, you first need to look for a proper section in the long XML file. When you find the section, you see this here shell comments wrapped with XML tags. Not that clear. While when you got Jenkins files introduced and you know you want to just modify the job steps, you know you in Jenkins file, you will only find the job steps. So it's easier to find them. And as you see the difference here, it's much more clear with Jenkins files. We saw which plugins we were using, how changes on each of those plugins will look like. So you could say it starts reassembling a typical workflow with reviewing of changes, right? So I don't think that's really true. What do you think, Simon? Yeah, I think you are missing something here still. At least the workflow looks like something you should do. However, are you sure that after managing changes, your deployment will be, you will succeed? Yes, but actually no. So we are missing one piece in here, which is basically some non-human related checks of those changes. What we did here, we created some Ansible playbooks that are deploying a new VM, installing under Jenkins, then applying all the basic configuration and this new change. Then we check if everything is working all right and if yes, we are proceeding with the reviewing. Okay, so we are coming to the end. Let me sum up and summarizing I would like you to remember there is thing called Jenkins. If you are not aware of it yet, please change that. And while Jenkins is easy to start, easy to configure and try, it might not be the smartest way or maybe not the best way to manage it in a big team or a big company. So for this, if you want to do it like a boss, check these three plugins we introduced to you. Configuration as code, JobDSL and Jenkins pipelines. Using them together, it will allow you to introduce easily automatic changes of Jenkins code of Jenkins configuration. And it will allow you to progress further with your changes. You will be able to easily introduce deployment tests of your Jenkins. So basically what you can do now easily because it will force you to keep configuration in code repository. So you can use Jenkins to deploy Jenkins, to test Jenkins. So Jenkins inception. So that would be everything. Thank you for your attendance. And if you're interested in the slides, you can find it on Shimon's webpage just under this address. And I think it's now time for questions. Yes, could you please go to the microphone? Hi, it was a great talk. And I'd like to ask you, because I also have a similar setup that you have, but I have the probability of declarative pipelines because I generate the seed jobs and then the teams are maintaining their own pipelines and then they change, but the pipeline has to run once so the settings are update. Settings like geared, challenge merit events and stuff. Do you also have this problem? Because the pipeline doesn't run automatically the first time, so the settings are not there. We had something like this. I remember Peter did some magic so we can talk after the talk, we can take a look at it. We forced the refresh. Yes, please. So I have a question with regard to the testing. You mentioned that you test the jobs while they are under review, right? When pull request is opened. And do you do any other kinds of tests? So for example, some integration testing of the pipelines or unit testing while the code is already merged in master because what I'm asking is because there might be some infrastructure problems or some access problems, for example, SSH keys change and so on. And also, I think that with maybe it's more complex than what was included in the slide but I imagine that there might be some gating problems. For example, there are two PRs which work fine separately but will clash when they are merged together. And both are separately tested and pass and then they are merged. So I imagine tests after merge could be useful and the question is, do you implement this or do you not see a need for this for the time being? Okay, so let me answer. There are two questions basically. First thing, we've noticed that and for now we don't have such tests. Unfortunately, we started our journey pretty recently. However, we have that in mind. So for now, we just have a basic Jenkins deployment test to make sure everything works and the Jenkins Dev instance can be spawned. Now we are aiming to have some, like you said, all semi-integration tests. We want to simulate Garrett events to check if all Garrett trigger jobs are spawned and launched properly. So this is our goal. We don't have it yet. And the second question about these two changes that can be merged one after another, we use in Garrett this merge strategy called rebase. So if you merge, you cannot merge both changes at the same time. One has to be merged first and for the second it would be rebased if you want to merge it and there will be tests for it launched. So the program is solved that way. Thank you. You're welcome. Any more questions? Okay, so for last word, I would like to tell you that please remember we are from OVH and we have a stand here in Marketplace. So if you will have any other questions that didn't came up yet or you were just too shy to ask, please go to Marketplace, find us there. We are here for the whole summit. So please ping us and let's talk. Thank you again. Thank you.