 I set up a CI server for our project in Jenkins about two months ago and it was great. The art of software development has taken huge leaps and bounds in the last ten years. I think in circa 2003, if you weren't using version control, you were really behind the times. 2007 unit tests, 2012 continuous integration is on the scene. If you don't have it or if you don't acknowledge the power that it can bring, well it's like explaining to somebody in 2003, why aren't you using version control? How useful it is, it's great. It is actually amazing. I've been without it, I've been with it, and I prefer with it. For this example, I'm going to do a live demo. I decided to take a project called PyStash, I don't know who wrote it, but it's great. There's a Mustache templating engine in JavaScript, some guy wrote a Python implementation. I made my own fork to set up some Jenkins magic. It's available there, if the URL was straight. It has two great qualities. First of all, the unit tests are quick to run, which is important in a demo. And also the unit tests pass on my machine, which is also inherently important. Let me start with the demo. So I have literally a stock install of Jenkins. You can download a jar off the website. I did a war, actually, which is a web jar, whatever. It's all Java, it's Java. But I did Java-jar Jenkins.war, and it says go to localhost 8080, and I did, and I get this, right? So I'm going to create a new job. Let me call it PyStash. Let me open this in on-ground. See if I can get a boot. It's going to be a freestyle software project, because that's the type of guy I am. No description, advanced product collection, source control management. I'm going to pick Git. So this is all point and click. Thankfully, I tried earlier, and I managed to reach GitHub from here. So this points to my GitHub repository I have here with the GitReadOnly address here. What else do I need? ArchSpecifier, which I forget the hard way is StarMaster, and let's build a build figure. Just for kicks. This you put in the cron-style schedule, so I'm just going to put every five minutes you check on it. The docs are all actually surprisingly good on the page, describe everything you want. It's ugly but functional, kind of like me. And let's just add a build step, right? So you get access to a straight-up Unix shell, put whatever shell commands you want. What I do for all this is I define little shell scripts that I put in my repo, which executes what I want. So this is just a shell script that will activate the right environment and run the tests. PyStash has this weird command test framework thingy, I just trusted to do the right thing. What's great about the script is I can run the scripts on my shell, make sure it works, oh, it works, and then I can put it into my CI, so just run tests.sh. That way, I don't have to wonder what crazy stuff do I have in my build. If I update my environment, right, add packages, what and have not, I can update the shell script and I know my CI pulls the latest version and all is fine. So I highly recommend just put in a shell script, just execute it here. You don't need to worry about Jenkins environment, you can just all set that in a shell script. All right, I have my little project. Let's build now. I'm not going to wait for the scheduler, it's going to take five minutes every time, it's going to be too long. It's building, you can click on it, it's actually passed. It was a success. Isn't this great? And I got the little console output. All right, well, let's break something, right? Tests, no, where am I? FIM PyStash tests, test, simple. That sounds good. Is there an assert here? Assert string, and let's move a thing, right? So now the tests fail, right? No, run tests. Come on. The tests fail. All right, what happens if I build now? Oh, I'm an idiot. All right, test fail. Let me commit my broken test, because that's what I do on my team. I don't know if you've worked on with me. All right, let's push it up. Man, we see this commit all the time. You guys keep on merging them. Come on, get, oh, it doesn't work over the network. Sorry, yeah, I don't have SSH2I. Oh, OK, sorry, I thought I did. Anywho, it would tell you the build failed and a big retin button. I'll pass through that. Let's add a bit of magic to this. Let's add some other stuff. Let's see what we can add. Do, do, do, do, do. Oh, right. JUnitXML, right. What I'm doing here is I'm trying to make Jenkins think my project's a Java project, other than the fact that the tests don't run in five minutes. It won't know it better. There's a standard called JUnitXML. I don't know it. All I know is there's a great Python package that'll take your test runner, that'll provide you a custom test runner, that'll spit out the right XML for JUnit. And what's great about Jenkins, because in the Java project, it'll parse your JUnitXML and list you all the errors you have in your tests. So instead of, like, right now, all Jenkins sees is the shell script fail or pass, with the XML, it'll actually be able to break down into which test failed. To use this XML test runner, it's really simple. If you have any familiarity with the unit test package, I won't go through it, because that's boring. You just use it as a different runner. And in this case, discover directory where your thing is. So let me show you how I use it. I have another script called RuntestJUnit, which uses this test.py, which is basically the code I show you there, which uses this fancy runner. And let me stick it in. All right. So configure our project again, point and click, because that's how Jenkins can roll. So instead of my Runtest, I'm going to use my Runtest of JUnit, and I'm going to add a post-build action. Aren't those fun? Publish JUnit test result report. I love reports. This is what CI is great. It'll do plenty of stuff for you perfectly in the right order, and publish all these reports for you. So you can keep tabs on the crap you commit. Test report XML. Let me see what it is. Right. Output is Python test XML. This is a crazy directory of a crazy series of XML files. Oh, I need to give a star. Sorry. The great thing is I don't need to care. It just takes care of everything for me. Oh, I'm going to retain long standard error output. I love little flags. It's great. Build now. All right. And test results. Oh, I get a list of all the tests in there broken down, how long they took, all that. And in this case, they passed, because I'm so great and I can't push my break. Let's go back to see what else we can add. So that took literally all of three minutes and 26 seconds. Linting, who here wants to find you in amazing ways to annoy your coworkers? Who here doesn't get into enough stupid arguments with their coworkers? Linting, coding standards, they're all awful. Everybody has problems. Everybody else's coding standards. I propose you pick one out of a hat and you stick to it. As long as everybody has the same coding standard, it's all as important. Use automated tools to set that coding standard if you have to pick one. Go has go fmt, which formats your code automatically. I think it's some crazy C ones. We've got really two good ones in Python, I find. There's PyFlakes. PyFlakes is not a linting tool. PyFlakes is a catch all your stupid mistakes tools. It'll check your code for symbols that haven't been defined, right? So, oh, you mistyped the variable. So your compiler, which doesn't exist, didn't catch this for you. PyFlakes will. It'll figure out the modules you've got to import. Syntax errors, it's great. If you're not running, there's ways to import it to include it in VIM or where it'll highlight pieces of code where you screwed up. If you're not running of that, just download it, check it, Google it. It is great. Pep 8, a bit more ridiculous. It'll try to apply the Pep 8 rules. So from Python, the code standard rules, to your code and give you a long list of everywhere where you fail. So, all I did was I put them in a shell script. This is what I do. Two little picadillos. I put a set E so that the tests fail. They'll fail the build. And I put an OR through to make sure that all my errors in Pep 8 don't choke the test. So if I run it with lint in my shell, just like this. I get the PyFlake's log over all my code, which will tell me, oh, you imported this, but it's unused, blah, blah, blah, blah. It's all good stuff. Pep 8, a bit more picky. Oh, white space around columns. Your line is too long. Stuff like that. All these errors are identified with a code. This is very important. If you get into my rule of thumb, is I going to argument over five minutes over anything. I'm just like, just ignore that error. And my coworker will be happy. And that's all I need. So you can ignore how long your lines can be. You can set your custom line length. Check the docs. It's worth it. Instead of getting into stupid arguments about, oh, this rule doesn't apply in 2012. So let's see what we can do with Jenkins. Now, again, this is back to trying to make Jenkins think your project is a Java project. There's a tool called cobalt violations. Isn't that great, violations? Writing ugly code is a violation now. Because of your automated robot. This is like Robocop. It supports plenty of tools, including Pep 8. So I'll just put in pep8.log, which is my name on my file. And for pyflakes, pyflakes by chance uses the same format as pylint. So you can just stick it in there. You can set like bars and barriers of when do you break a build, because you'd use bad white space. Figure it out yourself. I might have too much success, figure it out yourself. Let's see, am I running the right command? I am not. With lint. Is that the name of my command? It is. All right. Let me save. Let me build now. Let me refresh. It refreshes automatically on longer builds. This is just so short. And now you should get your violations graph, which should show you all the errors in your thing. And I think if you build it again, you'll get an actual graph with a track. In my workplace, it's really convenient. Because whenever the Pep 8 graph really goes high, I know my CTO just committed a piece of code. Oh, graph. It is. So we'll keep a graph of low, medium, and high errors you've done and give you this breakdown line by line of this is wrong. You forgot to put a space in front of the class modifier and all that. Me and my co-worker were on the same project. We both shared a really great quality, being anal-retentive. So I think we went down. When I started this, we went down from 2,000 Pep 8 errors to 200 in the space of a week of just, this is annoying. Let me fix this. The machine says I have to fix this. Please. All the errors are codified. If you want to ignore an error, all the errors are codified with a little class. And then Pep 8, specifically, Pep 8, you can pass in a ignore errors and the error class. And I just put it in my shell script and boom, I go. And commit it. We run the CI. It picks it up. And we never hear about that again. And my commits get a huge poof in the graph because we just ignore the whole class of errors. It'll do the same thing. Of all the PyFlakes error, like Pep 8 is really annoying. PyFlakes, it's all common sense. You import something, you don't use it. Sometimes there's a bit of metamagic kung fu you do in test modules and stuff like that, where PyFlakes complain about something which is kind of all right. But usually, PyFlakes errors aren't that bad. Again, the point isn't get zero. The point is to get low enough that you feel comfortable with the code. As opposed to unit tests, which you always want to pass. Or the unit test, which you want high coverage. Hmm, I wonder if we can do that with Jenkins. Let's go back to our configure and click around more. There's a thing called cobertura, which does coverage of Java code. What's really handy about cobertura is there's a package called coverage. Python coverage, which we all know and love, which also can spit out XML. Let me see. If you pass it, so coverage will create like a coverage file which lists all the coverage in your project. Boom. And if you pass it in the XML, it'll spit out this coverage XML, which is a cobertura standard, whatever. Again, XML standard, I wouldn't know, heads of tails. All I know is I plugged that into this and it just works. Let me show you. Let's just, UTF-A just in weird case, but I think it works. Unit tests with coverage. All right, build now. This one's a bit longer because of the coverage thing. So we still get the violations, the test results, and now we get a coverage report, which will tell you to break down for all your modules of the classes, the files and all that. And I think if you click again, you'll actually get the actual breakdown of where the test ran, right? Which piece of code ran, which did not. So in red, it did not run in a green, it did run, right? One thing I don't like, well, one thing I prefer is coverage itself has a really cool HTML format report thing, which I would highly recommend. And if you go back to project and you configure, let's see, add a post-build action. The only they had, grab a beer, publish HTML reports. Let's see, da-da-da-da, where is it? Up, up, yeah. Here you can publish any type of HTML report. In all fairness, coverage has this directory. I dump all my coverage HTML reports into a directory. I just added, no, I wanna delete that one. Don't forget to save as I always do. Build now. Oh, no, I forgot one thing. You go to, in the publish HTML reports, keep past HTML reports, very important. All right, let's try building again. Oh, sorry. Boom, and it's taking a bit of time, chugging through. But again, this is all an automated process. I didn't show how to set up email because I don't have email, like an email server set up on my laptop, but that's really easy. It'll send you angry emails when things break, angry emails when things are back to normal. It's really a good HTML report. And now you get the really nice coverage reports from the base Python coverage tool we all know and love. And that'll, it's a bit prettier, I find. And a bit more pyphonic, like the Java corporatura. It works great when you're writing Java code as I don't. So I prefer the coverage tests. But you still get the stats from the corporatura coverage of like, oh, oh, your commit has lowered the line, the conditional's coverage from 100 to 89. Oh, this is not warning, Bill Ormanson. And basically that is it. What did I want to talk about else? Coverage, yes, it's a built-in coverage tool. You can pip install it, all this, you can pip install. I want to talk about the Jenkins plugins used. To install plugins, it is really easy. You just click, you poke at it enough and it gets in managed Jenkins plugins available. And all these I installed for the demo today, so it's four. And finally, sorry, so Jenkins was really easy to set up. In Ubuntu, it's app get Jenkins and you get a Jenkins server kicking up in three seconds. That's what I highly recommend. Have one CI server somewhere centralized and run it all from there. If you really want to get fancy, you can get like slaves where through SSH it'll communicate to a third server. I have tests running on our dev machine, which is great because our package builds our environment. So it'll try to update, like if somebody installs a new package into dependencies, like a Ubuntu package, it'll try to up get and then break and I'll be like, oh, we've got to update the servers with that commit and I have to do that and catch, it's caught me so many times. I want to quickly talk about Jenkins versus Buildbot. Buildbot is a Python tool, it's written in Twisted. It also does CI. I have used both. Buildbot is a lot bigger. Buildbot, you do not point and click to set it up. You write this configuration file, which is Python code, where you call classes and stuff. However, Jenkins, I just went over the surface over 15 minutes and it's about as deep as I want to get into Jenkins. Either feeling, if you want to do something a lot more complex, you're going to wind up writing plugins in Java or something kind of scary. Buildbot, it's easier to hack. You can dig into it. I've read the code. It's not, it's hairy, but it's not that bad. It's harder to get started. However, if you have an insane amount of tests you want to build and like five projects in 10 different environments and you need to build like a Windows client and you need like that machine to run this automated job. Buildbot, you can manage programmatically. Jenkins, you're clicking around for like days on end, right? So, but if you have one project like this or we had what, you can have series of projects. I think we had one for the unit test, one to reload the dev server. Well, you can be set up in half a day. Buildbot, you're going to be taking three days, three, four days to set up, to completely understand and set up everything. Please. The question is in Buildbot, you can set up multiple slaves on multiple environments, free BSD, different flavors of Unix, different builds of Linux, et cetera. Jenkins, you have an SSH thingy where you just create a new node through SSH and it's, oh, this is great. Simon, plug your ears. He's our security guy on our team. What it does is it SSH is into your box. SCP is a jar it needs to serve as a client, runs that and listens on a port and the master will send commands to it that executes a shell, a shell command. So you have to have an SSH? Yes. Well, with Buildbot, you can have the custom Buildbot slave, but... Exactly. Here you can just use SSH and Java for the slave. All the Jenkins does have a mode where... With Jenkins, all you need to do is SSH. In Buildbot, you need to set up Python on the other box. It's not twisted, but it's not easy. You need to have your virtual line set up. Here's just Java, and as long as you have like 1.6 Java, it's a self-contained Jar of Wars, and it works, I found it works surprisingly well compared to Buildbot slaves. But again, Buildbot, you can handle programmatically. So if you have anything past like four builds, Buildbot might be a very good idea. Again, the setup, I have a couple of shell scripts. It's not black magic. I think I have test PY for the XML runner. It took me like half a day to set this up. It's really easy. All right, is that it? Thank you very much.