 Alright, I guess we can get started. It looks like the last few people are wandering in. Okay. Everybody in the back can hear me from here? Yeah? Okay. Alright, this is how Jenkins can work for the whole team. My name is Tommy Keswick from the local Los Angeles Drupal community. Woo! See some friendly L.A. faces here. So, yeah, before we get started, let me pull the crowd a little bit. How many of you have heard of Jenkins? Oh, wow. Okay. How many of you are using it? How many of you are administers of it and how many of you are... Okay, administers of it. Okay, good. How many of you are just kind of a user of it on your team? Okay, good. Alright, yeah. This is what I want to cover. So what we are going to go over today is what Jenkins is, why you might consider using it. We are going to look at some of the project configuration screens of how you can configure some of it. We are going to look at what it is like when you are running a build in Jenkins through some code commits and then what happens in the interface and whatnot. And then whether it is good or bad, failures and successes. And we are going to look at how easy it can be to extend Jenkins. And then what we are not going to cover are installing Jenkins, scripting builds and deployments, and writing tests. So if you are here for that, I do not mind if you get up and leave. Because this is more higher level stuff that you are not going to get into the real guts of it. This is more like how you can use it and what it is going to do for you. Alright, a little bit about me. I am a developer and trainer at the Cherry Hill Company. We are a local Los Angeles firm. My background is in libraries. I am a trained librarian, but I have become a web developer over the years. And I am also an organizer in the LA Drupal community. Also an individual member of the Drupal Association, which you all should be as well. The Cherry Hill Company, we have been using Drupal since 2005. And we really like to use open source tools to help libraries and other nonprofits and educational institutions. We are a supporting partner of the Drupal Association. Okay, so what is Jenkins? Jenkins is primarily known as a continuous integration server. And so what that means, continuous integration is really the practice of building and testing your code after every commit or change that you make to it. So it will build the site or the project with the new code immediately after you do every change to make sure it is always working. But what Jenkins also can do, it is also sort of a triggering mechanism. So it can run tasks at certain times or after certain other events happen. And so you can run scripts or builds. They call them builds a lot of time in Jenkins. So you can chain a bunch of things together. It is also a great interface for managing all these projects. So when you have your builds or your scripts that you need to manage and you have to set them up to run after each other and whatnot, it provides this interface that makes it really easy. And also I think of it as kind of a message aggregator too, because it can collect all the outputs of your scripts and it can react to them and kind of consolidates all the messaging behind this stuff for you. And you can get your notifications through many different means. You don't have to be watching the terminal or anything like that. Okay, and so why would you use Jenkins? I think the number one reason is because you can empower your team members. For the CIS admins out there, you might want to offload some of your work. It will mean fewer bottlenecks for the developers to get their work done on their own. They won't have to... Well, they can go and find the issues with their code if a build fails. They can potentially go find the issues in their code by themselves. They don't need to pester you about what might have happened on the server. They can look at it. So for an example of that would be something like if you have an automatic build of your dev server after every commit and the server, say it fails, you can report... The server will report a failure. The developer can go in to the Jenkins interface and see what happened. They can see all of the steps that went through the build and find where the failure was. And that might clue them in to what went wrong and they can fix it themselves without bothering anybody else. Another thing it will do is it will reduce manual processes and workload that you've got going on for what happens during your builds. So you can do things like deploy code to your dev stage and production servers automatically based on certain events or it can be as simple as just a button push in an interface. So for example, you can have something like your development server can build on every commit. It will just use the newest code, spin it up on your development server. You could also have something like your staging and production servers could automatically build after you have some tests passing on the dev build. So if they pass, you could set some other server to build automatically based on that. Or if the tests fail or something like that, you can have the job that would normally allow production to be built to be locked down and not able to build because you know that the build is not passing. And so because of a lot of this stuff, broken builds are less cause for an alarm. Because depending on how you're scripting these builds, you can integrate rollbacks into your build process. So for example, like if something is broken in the build, the way we have it set up, I'll show this how it kind of works in a little bit. But our build, it basically works through KappasRano scripting. And so it makes some sim links and it'll basically just point to the working sim link. If something is broken, it'll point back to the one that was working before and it'll use a copy of the database that was working before. And so you don't ever have any worry that the server is down and broken because your site rolled back because it knew it failed. It knew the build failed. And this one also is applicable to developers can find their own problems. And so the whole team doesn't need to care about a broken build necessarily. And the other thing that Jenkins can do, which is I think actually a great feature of it, is that things that are kind of not related to building a site and deploying a site can also be added and scripted in a system. So some things like, like I mentioned, tests can be run with Jenkins. They can be one of the builds that happen. So if you have a server that builds and you've got your site up, you can then have a test suite triggered because it knows the build is done and has passed. You can run a test script against that. You can also do things like have a single push button to maybe migrate your files and your database from your production server down to a staging server or dev server, just like a click of a button, no command line stuff needed. We have something in there using just some scripting with an API that updates some documentation based on what's in a commit message for some things that we have. So if we say we want to keep a record of every time we apply a security update, we're looking for a particular string in a commit message, and it'll put a line of that commit and about it into some documentation automatically, so we don't have to manually stay on top of that. Another script we wrote that is pretty handy is we can monitor a mailbox and convert messages that come to that mailbox into tickets in our ticketing system, and that just kind of is a task in Jenkins that runs periodically every half an hour, so checking for new support emails. All right. Yeah, let's look at some examples of some of the interface. Okay, here is an example of what the home screen in Jenkins can look like. You can download different themes for Jenkins too, so it can kind of look however you might want it to. People have developed different kind of themes for it. So some of the, does my mouse show up over here? Can you guys see this? I don't have a laser pointer, so okay, cool. My mouse, I don't know if I can do that. This is a screenshot, so I don't know if I can actually do that. I wasn't going to try to do live demos. Somebody can follow me around. There we go. Let's see. So yeah, this is basically a list of the way a home screen can look. You've got a list of folders here that contain tasks. You've got over here some configuration options. You can manage the users that are in your Jenkins system. You can manage Jenkins itself with a bunch of the configuration options that are under here. Over here in the sidebar, this is where the action happens when you're running builds. You can see it in real time. Yeah, and so this is what it would look like if we clicked into that tasks folder. So when you have your list of builds or your tasks for Jenkins, they're listed out like this and we've got some really informational columns here. All these circles here are blue, meaning the latest build has been passing. You've got a weather column, which is interesting because when there are clouds, that means sometime in the recent history there was something that went wrong. But if it's all sunny, that means it's been good for a long time. You've got your name, you've got your last success time and the build number with a link to the status of that particular build. You've got your last failure duration and this button here is a build now button if you need to run it right then. And then this is kind of hard to show, but when you hover over a job, you get a little arrow with a drop down that's a quick links menu here. I find it really handy for getting into the configure section of a job, but you can do all these other things on it too. So when we go on, this is the landing page of a job. It's also like the status page. So you can see here the menu now contains all the items that were in that quick links menu. This actually is down here, the build history is all the recent stuff or all the recent builds of this project and their status with links directly to them. We've got downstream projects and what a downstream project is. I've talked a little bit about how you can kind of chain tasks and builds to each other. These are the projects that are triggered to run after this build. So we've got one that runs some B-hat tests after this build. And these links here, these are the ones that are permanent links. I don't really have a way to show what the URLs are, but the way they work is they don't have a build ID in the URL. They have something like slash last and so it's always going to be a link to the last build. And this one's always going to be a link to the last stable build and last successful build, et cetera. So you can share those links and people will always be able to get to the latest stuff that they need to. Okay, so now we're actually in the configuration screen. I'm going to go through, kind of like if I was scrolling through it, to show off some of the stuff that you can do in this. Obviously there's a project name. You can put a description in. In this setup, we have it so that Jenkins won't keep records of any builds past five just because it's going to kind of clutter up what we're keeping in. And it doesn't really matter what the build status was 100 builds ago. So we'll discard the old ones. We've got pretty granular permissions too. So we've got all these users in here and everybody in this demo system actually has all the permissions. But say you had a member of your team that you certainly didn't want to give permission to delete a job to. You can just easily uncheck that. You could also not let them configure it, but maybe only let them run the build on the job. You can give them only that configuration. So it can get pretty granular. And it lets you share with your team exactly how you need to, because obviously some users are going to be more involved in the system than others. And it allows you to give permissions for users to do something that they normally wouldn't be able to do, especially through SSH or on the command line. But you can give them interface permissions to do this stuff. And it's pretty protected from what they can actually do. Yeah, next there are options for source code management. You can see CVS or Git or subversion even down here are available. We're using Git here. That requires a plugin, but I'll show towards the end how you can add plugins to Jenkins pretty easily. And the options here are to point to the repository. Under some of the advanced stuff we could put in credentials if we need to. We can specify the branch we want to use and there's other more complex options if you need to under some of these advanced buttons. And the build triggers are, they really tell Jenkins when to run the build. In our setup, our repository, we use Unfluttle for our project management and it also has our repositories in it. It actually, we can configure it to ping a URL that's a Jenkins URL specific to a build saying, we have a new commit here in this repository, so build now. So that's really how we're handling our triggering of a build is when there's a commit to the repository, it says to Jenkins, okay, build this now. We also have this poll SCM source code management option in there, kind of just as a fallback. So it's going to check the repository every hour to see if there are any updates to it since the last one that Jenkins knows about. And usually that's never going to do anything because of the other setup that we have, where it tells Jenkins immediately when there's a new commit. All right, and there is the build itself. So let's see, down here, normally when this is a blank project, it's not going to have anything here yet, and you have to add it with this button. So there are a lot of different options here. We're using right now the execute shell, but you can do any of these other things. There are plugins that would provide more options in this, in this dropdown. But with the execute shell option, really this is what the build is doing. We have it running a bash script that has a parameter. And so this is kind of the meat of what this job does. It's running this bash script. This deploy.sh script is what handles all of the, I guess, configuration of what needs to happen for the deployment of the software on the server. That's some of the stuff I'm not going to go into, but kind of what it's doing is taking the new code, pulling it up, making a new directory for it, grabbing a copy of the database and associating it with that, making sure that Drupal can bootstrap. And then if it's all good, it'll change the sim link for the live site to point to that directory. Everything else in Jenkins is kind of built around what's happening in this script. So you want your notifications to happen because of this. You want your, you're telling it where to get the resources to do this, et cetera. And so some of the things you can do after the build happens are like notifications. Actually, let me scroll down. Here are the options for post build actions. Some of these are provided by plugins and some of them are default. In this particular project, we have editable email notification being used and trigger parameterized builds on other projects. But you can see there's a lot of other things going on. So for example, we've got this editable email notification where we can choose who is going to get the email about what's ever happening, reply to address, subject, content. And you can see that they're all variables. The variables are actually set in the main Jenkins configuration for the whole system. And so you can say who the default recipients are going to be and all that sort of stuff and kind of just set it to be generalized. But per project, you can see you could, you could type text in here. You could type email addresses. You could type new subjects. Some plugins will provide you new variables that you can add to this stuff. So you can, you can add all sorts of stuff. There's an attached build log so you could actually give it the whole console output of this particular build and send it in the email to people. Just to make it easier, they wouldn't have to go to the interface, et cetera. And then let's see, the parameterized build on other projects is actually a really powerful option. This is how you can chain events to kind of happen in sequence after things happen. So, so what we've done here is we have another project in our Jenkins setup called demo test be hat in Firefox. And so we've chosen this. I believe it's even autocomplete like it'll drop down with the other projects that are available. But you can also do a condition. So this project is only going to run or the build is only going to run when the particular build we're configuring right now is stable. So this build fails and is not stable. The triggered build isn't going to, it's not going to be triggered because the condition isn't met. And you can also add parameters. That's what this drop down is showing here. We are passing the get commit to it because this project needs a get commit to know what to run on. And, but you could pass all kinds of stuff to it, which makes it pretty powerful. And I think, yeah, that's it. Okay. So now we're going to look at what it's like to actually do your work and run a build. And this is going to be one where everything goes just right. I've, instead of doing a live demo, which is risky, I've recorded a video that has no audio. So I'm going to talk through it and explain what's going on. So this is me working on the command line. What I'm going to be doing here is I know in this particular site, there's a module that's out of date. So I'm updating the module and what's going to happen after the modules updated is that I'll commit the code and I'll push it to the repository. And then we're going to see how everything kind of goes together in the Jenkins interface. So the code has been updated with Drush. I've added the code back to the repository. And all this stuff, I happen to be doing this on the command line. If your developers don't work like this, it's however they normally do their work. They could be using a GUI for Git software or whatever, and it's still going to do the same thing. This doesn't really matter. It's just more fun to look at right here like this. Okay, so we flipped over after we've pushed. And this is what's happening in Jenkins. You can see down here, the build has started. And this is kind of, it live updates itself and starts to show you the progress of what's happening. And so I kind of forget what I clicked on next. But there's a lot of things to click on. You can click on the build and you'll see the ID number there, number 24. We could click on that. I think, okay, I click on the build history, which actually shows all the builds in Jenkins and what their latest statuses have been. And so we can see this number 24 is in the process of building still. It shows the history of a few of the other projects as well. And actually, some of the stuff doesn't refresh always perfectly. But you can see that this, okay, I refreshed. You can see that the test build started, which means down here we can see that number 24 is stable. It says it over here. And it took 52 seconds for it to run. And so that automatically triggered this test build to run afterwards. And that one is on build number 14. So that means the test builds only run 14 times. And then it'll finish and we'll see what happens. Yeah, this is the successful test. So everything should just pass. I think I refresh again here so we don't have to wait too long. Oh, what am I doing? Okay, I'm clicking into the build page itself. So this is the details of a build that you can see. So you can see our, that's our get commit message up there. The changes, number one. There's our hash for the get commit. So it tells us all the details about what happened in the build. Different build, different types of builds have different information on the screen, but it's a way to kind of see what the status is going on. And the blue ball up here is, or it's blue instead of red. If it was red, that would be, I mean, something's wrong. Okay, so that's when everything goes, goes well. Now the fun part is to watch what happens when something goes wrong. So I've got a template.php file here. And I am going to remove a semicolon. That's going to cause stuff to break. I've also got a CSS file that I added some, some code to. That's going to hide a search box, which the be hat tests relies on. So when I commit that, the tests wouldn't pass either. So I'll go through, I'll add this broken code to the repository. And we're going to push it up and see what's going to happen. So we'll go through this. Some of the things to take a look at in this build is how the test build isn't going to run and how just, just the way things look when they're broken. So we wait for it to get started. When we, when we come back over, we see it's queued up and then it's going to jump down to the, it's being executed status. You can see the ID number incremented by one. If we click on it, we can go to that status page, even while it's still being built. And it'll have a progress meter up at the top and it'll have a blinking icon instead of a solid one. But we can also see that it already knows about the, the git commit. It knows there's a new commit in there and it knows the hash. I forget what I clicked next. But there's other things we can click on. We'll eventually look at the console output. If you go back to the tasks, we can go see sort of what else is going on. Okay, so in between that click, the build finished and it failed because it's red and cloudy. So yeah, I clicked back to the build history of everything and yeah, you can see down here that everything is broken since this build. So that kind of made everything stop. The tests are not going to run. It's just kind of halted in place. So we kind of have to go figure out what went wrong. So if we go back into the build itself and we go over to the console output, this is going to be all of the output from what would have happened on the command line on the server that's doing this. This takes a little bit of training to show your developers actually what's happening. They kind of have to understand what's happening in the builds to scroll through this and look for the errors. But once, once you teach them a little bit, they can figure it out pretty quickly. So what we've got here is it's really like the Capistrano scripts are doing this. It's got a lot of drush stuff going on too. And eventually you'll get down to the part where you see errors. And that's what you look for because that's going to clue you in to what to go fix. And so right here we can see, I'll highlight it, that the template.php file on line 185 has a PHP parse error. So that's going to clue a developer in and say, oh, let me go check out that file. I've got something wrong there. And they won't have to have talked to assist Edmund to be like, what happened on the server? The build's broken. I don't even know where to start to debug this. The other thing they can look at is down here, the build step execute shell is marked as a failure. If you remember, that's that meat part of the build that I talked about that's running the script. And so the whole build stopped right there, that part of the script. So we go back and we fix the semicolon issue because we figure out where it's happening and fix it. So then we have to commit the code again. And it'll go through the whole process. I only fixed the semicolon, I didn't fix the CSS. So this time the build is going to, the build itself is going to pass. And then the test is going to be triggered, but the test is going to fail. And so we'll see what happens after that. And yeah, so it's already queued up. I'm pretty sure I refresh a few times, you don't have to watch all this happening again. And so yeah, you can see the last status, it still shows red and cloudy, but eventually it'll pass after this one's done. I think I refresh pretty, pretty quickly. Yeah, this is a refresh. Yeah, so that build passed. You can see down there back to normal for the build. Now the test is running because that's chained to a successful or stable build. Refresh that and it'll be finished. And we can see that the test is now broken. So now we have to go kind of debug what happened in the test. On this test build status page, there's a little bit more info. Like down here, test result, homepage use search, it doesn't work right. So we have to kind of go check what that was. Now this information is a little weird. It's going to take a little more training to figure out what's going on, but you can see there's some CSS selectors there where you kind of have to figure out what's happening. But you can notice there's a quick tabs search kind of thing that we know is something's not working right. So that clues us into what's not showing up in the test. And if you're familiar with the test you have written for your site, you'll know what test those are referring to. So I delete the offending CSS and save it. We will be able to run everything and it should go smoothly. I'm kind of running out of things to say on these screens. Yeah, so this is just debugging and making everything work. Yeah, so the builds are going to happen again. I'm pretty sure that I refresh here relatively quickly. But basically the gist of it is that everything is going to pass. The test is running. It will eventually pass. And then... Yeah, here's the status page of the test build and you can see the red triangles that kind of like number 13 was good, number 14 had an error, number 15 was good, number 16 had an error, 17 is good again. So that's how we can tell what is going on with our builds. Oh, and actually there's a new section here on the test build. It's got upstream projects. So it tells you where this project can be triggered from. And I guess the production build is considered a downstream project from the test. So this has got different status information. I think that video ended. Okay, so yeah. That's the failure when you got to go and debug some stuff. Next, okay. I'm going to take a breather for a second. But now we're going to look at extending Jenkins. So I've mentioned already a number of plugins that you can add to Jenkins to make it do more stuff. There are actually hundreds of plugins available for Jenkins. There's probably over a thousand. I don't know the exact number. But they come in many, many categories. There's things like trigger plugins. You can do a fire plugins. Plugins to change the UI around. There's plugins that help you with some command line tools. And plugins for authentication, that kind of thing. I'm going to give you a few noteworthy ones at the end. And then installation for the plugins can be very easy. You can go into your managed Jenkins that's installed on your server and go to a list that can be just kind of selected and installed in the UI. The kind of library of plugins that's available on the web is kind of baked in to Jenkins that you install. Or it goes and grabs and it knows what's available and you can install them right there. So let's, yeah, I'm going to show you how easy it can be. So here's another little video. We're going to go into the managed Jenkins section in order to, what we're going to install is we're going to install a hip chat plugin which is going to allow notifications. Go to the managed plugins section. It's going to allow notifications to a hip chat chat room. Here we have all the plugins that need updates but we're going to click the available tab. And on the available tab you'll see all the plugins that are available for your version of Jenkins. It's only going to show you the ones that are compatible with your installed version which is really nice. And so I'm going to search for hip chat, check that one off. You can install plugins a lot of them without even restarting Jenkins itself which is pretty cool. So you just click the button and you wait a little bit. It's pending and then it'll be successful. You could restart it if you needed to but you don't need to on this particular one. And so it's successful. Now we're going to have a few more configuration options available in our managed Jenkins section. If we go to configure system at the bottom down here we're going to have some hip chat information that we can put in. So we've got to put a server in. You can see we've got to put an API token in and a room that we want our messages to go into. I'm not going to show you the API token but when I made the video I put it in so it's going to work. You save that. So then you've got your general configuration setup but then we also want to configure it on a per job basis because sometimes you want certain jobs to report messages in different rooms or something like that. On the job version itself you're going to also have a new option. It's going to show up in the parameterized build option I believe or post build action or something like that. Yeah, post build action. So we've got a new one down here, hip chat notifications. It's got another room for an authentication token. We don't need that because it's in the general settings. We can put a new room if we want to and we can tell what kind of notices we want. Failures and successes at least. We could make a specific message or you can leave it as the default and that'll be useful. So if we save that and then we run the build we'll be able to see what happens. So if we click build now it'll start and then we can go over to our hip chat chat room and see what's going to happen there. I think I pulled that up right now. So I tested the API. That's why there's a notification in there. But then the build successful. It'll put a message right in there. It gives you the number and a link to go right to the build details. So especially if there's a failure it's a quick link for somebody who's hanging out in the room to go jump in here and see what went wrong. So you don't have to rely on emails because especially if you're building all the time those emails can really pile up and they get lost in people's boxes. I actually filter out most of these emails to myself because they happen all the time. And having the notifications come to a chat room that's kind of on your workstation is a lot more convenient I think. So yeah, that's how easy it is to install a plugin. So let me tell you about a few notable ones. Alright, so I mentioned hip chat. We showed that one. There's also a Slack plugin and I know a lot of people really like Slack and Slack is super extendable so you can obviously dump these kind of messages into your Slack chat rooms. There's a Git plugin which we were using and that's how we got all those Git options in there. It provides the way to put your repository information in. It also provides variables that are available to some of these parameters so you can put variables in that come from your Git repository. There's a plugin called Delivery Pipeline. Delivery Pipeline is actually really cool and it's worthy of a screenshot. What it does is it gives you this visualization. So this is an example of a bunch of chained jobs that kind of go together. So you can run a build that's going to run this test which then when it's successful will run this release which then runs two different jobs after that deployed a test and generate documentation which then also can run a pre-prod job and then a real-prod job. So it's just a really nice way to see the flow of all of your chain builds and what the status of them at any point in time is. So this is what I mean by there's visualization plugins that help you really see what's going on. Another example of a good plugin is the LDAP plugin. There's also a few other ones for different kinds of authentication methods. There's an email. It's called Email Xt extension. So what it does is it allows you to configure even more aspects of email notifications. It gives you full control over what types of things can be notified on. The Nested View plugin is one... We had it on the videos that we were showing. It's Y or N on the screenshots. It's the folders. You can put all your jobs into folders and those folders can be within folders, etc. It's super helpful when you have hundreds and hundreds of jobs. You can kind of organize them and it makes the UI a lot nicer to work with. And then there's LogParsher. We didn't have that one turned on, but what it will do is help with that console output. It'll make it much easier to find the error messages. You can turn that on. It'll make it a lot easier. We probably should turn that one on ourselves. There's one called Straw Boss which it's pretty good for organizations that want to monitor external jobs with Jenkins because it'll do that monitoring and so you can have Jenkins manage some of the reporting for those jobs and then it also will manage notifications. An external job can run and Jenkins can handle the messaging about what happens for those jobs as well. And that's really all the room I had on the slide for notable plugins. So, the takeaways that I would like for you all to get from this presentation are the following. For those sysadmins out there, you should consider Jenkins so that you can empower your team members. Letting your team do some of the work of debugging their own code and controlling some of these build processes. It's going to lessen your work and it'll let them do their work more efficiently without interruptions and that's what it's good for developers too because we want to disrupt or we want to reduce the disruptions between team members. You don't know what the other people that you're usually having to bother to help you with stuff could be doing when you have to bother them. If everybody can concentrate on getting the stuff done without help, that's going to be beneficial for everybody. And for everyone, being able to automate and manage your repetitive tasks, it's why we use computers and so you want to automate this stuff as much as possible because it's not fun doing it so you might as well have it done and have it done systematically so when there are errors, that's the only time you need to pay attention. And with that I'll open up for questions. Yeah, is there a mic in here? There's a mic over here. It'll help for the recording. Thank you. There's an on button? Oh, no. Okay. I'll repeat the question. Go ahead. Okay, so that stuff gets handled. The question is how do we win over how code changes get tested in the build? We want to know how update hooks and schema updates happen in the build. In our build script we're bootstrapping Drupal and so if that stuff breaks, Drush will know and complain and so I'm pretty sure that's how it's going to work. Ashoka's nodding his head so yeah, because Drush is running in our scripts, that's going to let us know when that kind of stuff is broken. No, yeah, it'll make a copy and run it on that. Oh, sorry. The follow-up was so it's a new update on a new database and it's not messing with the old one. Yeah, so it's it either makes a copy of the new one and runs on the new one or it backs up the original, runs it on the new one and either rolls back to the backup or whatever. It's all, it's going to have a rollback. The original's available if something goes wrong. The question is how do you version control a Jenkins task and I'm not sure about that. Can you write a Jenkins task in code? Ashoka, do you know that? Okay, so the answer that we got from the audience is that you can write out your Jenkins tasks in code and share them on GitHub and so it's however you want really how you could version control your Jenkins tasks and then just install them on your server when you're installing and that's just kind of stuff I didn't get into. So I mean it is possible and I guess people do share their stuff out there so that you can set up Jenkins with a bunch of pre-installed tasks. Yeah, so you can do it. I just didn't go over it at all and the way we use it is mostly through the UI. Yeah, go ahead. Yeah, that's true. So the question is you want to build and test with different versions of Drush and PHP etc. and how to write that stuff and I think all of that is going to be reliant on what you're putting in your build script and so in your build script like that one that's running that bash script that we have you're going to want to like declare what you need in those scripts somehow. I'm pretty sure Ashok has something he can add. Okay, that's where he was going. So you can start up containers within your so your Jenkins tasks can start up containers for you to test that kind of stuff too. I talked to some of these guys about this. Definitely. Yeah, so Jenkins is running scripts and whatever you can do with a script you can tell it to do. Can you repeat the beginning of it again? How do you stop the developers from doing what? Yeah, let me go back if I can figure out how to go back here. I've got a screenshot that should show some of these up here. So these permissions right here, so you probably can't read it. That probably doesn't help. So configure is its own permission. You can take that permission away from anybody in the system and they won't be able to get to the screen. The only thing they're going to get is like the build button if you give that to them if it's running permission. It's pretty granular. What you can let them do and what you don't have to let them do. Alright, if there's no more questions. Thank you.