 Thank you, and hello, everybody. My talk is about, as Matt said, about continuous integration. A few words about me. I'm a Python web developer since more than 10 years, and most of my professional and free time I spend on a project called Plone. It's an open source enterprise content management system written in Python. We have around 340 core developers worldwide, and Plone powers websites like NASA, CIA, NSA, Oxfam, Brazilian government, and many more. And since around four years, I'm the leader of the Plone continuous integration and testing team. So we make sure that our continuous integration systems work and that our testing is in a good shape. So what's continuous integration? I guess everybody heard that term at some point, and it's in contrast to what many people think that is like a software that you can just install, and then you do continuous integration. It's actually a software development practice, like test-driven development, for instance. So this software development practice is about team members that integrate their work into a main branch of a version control system frequently, and each of this integration, or commits, or pushes, or whatever, is verified by an automated build and test process. And this automated build and test process makes sure that code violations, test failures, or bugs are detected as early as possible and also reported as early as possible to your developers. We all know that those statistics about bugs, right, that the later you detect bugs, the more they cause. So if we detect bugs early, they're easy to track down and easier to fix. And one of the other advantages, big advantages of a continuous integration system is that if you run your build and your tests automatically on a continuous basis, then you know that your development and also your deployment environment is in a working state. So as I said, there are three important parts in continuous integration. The first one is that you integrate frequently into your version control system, that you have an automated build and test system, and that you report. So keep those three items in mind, because I will come back to that. Our first approach in the Plum community to continuous integration was actually BuildBot. Who here knows what BuildBot is? Oh, quite a few people. So BuildBot is a continuous integration framework written in Python. We had it set up, but it's quite complex. And as I said, it's more framework than an out-of-the-box solution. So you can't just install BuildBot and it will do everything that you want. You have to really know what you're doing. Yeah, it's hard to set up, so we barely used it. I mean, a few hardcore developers used it, but it wasn't really run on a continuous basis. It wasn't really integrated into our version control setup. And nobody really, as a regular developer, you did not even notice or knew about it. And around four years ago in 2011, we introduced Hudson, what's now called Jenkins, into our process. And one of the developers who started to play around with Jenkins wrote that it's like BuildBot with a butler. So in comparison to BuildBot, Jenkins is really out-of-the-box solution. You just install it and you have to configure it a bit, but then it basically works. So that was really nice. Also, Jenkins comes with a nice user interface, so everybody can just go there and check the status and stuff like that. Downside is it's written in Java. And as a Python developer, you always prefer, of course, to use beautiful Python software. But Java is a decent language, and it's a very good software product in my opinion. It has a huge open-source community around it with many plug-ins. It's backed by a company who offers commercial services on top of that called CloudBees. And we're really, really happy with it. So during my talk, I will give you examples of what we do with Jenkins, but it's not very specific to Jenkins. So as I said, continuous integration is a software development practice, so it's about the practice and the rules that you have. It's not about the software that you actually choose. So when we moved from Buildbot to Hudson, things looked a bit better. But we used nightly builds. I guess a lot of people do that because your tests take quite a while, and you don't want to run them on every commit for whatever reason. And then you run them on a nightly basis. If everybody sleeps, then you can just run them for a couple of hours or whatever it takes. And next morning, you will get a report to your mailing list saying, this is the list of commits. And now the build is either broken or it's fine. The problem with that is that you don't run your build for each integration. If you recall the definition that I gave you upfront about continuous integration, the important part is that you run your build and test process for each commit. Because that's the only way to figure out which commit or which code change actually costs regression. If you have 20 commits from different people, and next morning, we will see, hey, the build is red, then somebody needs to clean that up. And usually the person who cleans that up is not the person who causes the violation, so it's costly to do that. And nobody does that. If you are in a company, you can force somebody, like a poor guy or girl, to fix the stuff for other people. But in open source communities, it's even harder. Because there are 20 commits, and people say, hey, it wasn't me. My commit was really clean and perfect. So if you run them on a nightly basis, your build is broken 99% of the time. That's at least my experience. So our software development and release process in the Plone community was like this. The build was broken 99% of the time. And then before release, our release manager said, hey, guys, I want to make a release. And then two or three of the 340 developers, the really hardcore guys, started to fix tests for everybody else. Sometimes we had 400 or 500 test failures. We have around 9,000 tests in Plone. So people really, we sat together for a day or two, and we really fixed a couple of hundred bucks before we could even make a release. And then we started to make our 300 releases. And then our release manager could make the actual release. So that's what it took when we had those nightly builds. So how could we solve that nightly build problem? You can solve that by following the rule that you have one build and tests per commit. So how do you do that? By default, Jenkins use polling to poll the version control system. Like, you can set it to every 30 seconds or something, and it polls it. And if a new commit is there, it creates a build. The problem with that is you won't fetch all the, you will not get one build per commit. Because it could be that some two people commit at the same time, then you have two commits. And believe me, those two people will say it was the other one, always. So you have one commit, and you make sure that you have one build for that commit. And with today's version control system, that's really easy. Because GitHub has post-commit hooks. If you host it on GitHub or Bitbucket or on your own Git repository, you can just create a Git post-commit hook that actually triggers your Jenkins instance or your CI instance. And then you can have one build per commit. So you can trace the person or the commit that was responsible. So it's really easy to figure out what goes wrong. Inplone is a bit more complex than that because we have those 300 packages. And one checkout doesn't mean we have the exact same checkout of all packages. But I will come to that later. And then what's important is that you preserve this commit information through your continuous integration pipeline. So you pass it through the builds. And also so that you can, at the end, notify people via email or anything else. So we have those three steps. First, commit build notify. And in order to be able to automatically build and test your software, you need an automated build. So we have tools for that in the Python community. Inplone, we use build out. It's not widely used outside the sub-community. The pyramid folks use it. But most people use pip or easy install, which are also fine. You probably need to wrap them into batch files or anything like that. But you can automate your build, right? If you do that, you can use talks, for instance, on the CI system to configure what's run on the CI system. And on the Jenkins machine, you can, for instance, use tools like Shining Panda. That's a Jenkins plugin that allows you to create virtual ends or build outs and install things via pip automatically. So it's just a convenience tool. We're not using it in the Plum community because a batch script is enough. But if you want to do stuff with Python and you want a nice wrapper, then Shining Panda is the right tool for the job. So if you do your build automatically, you of course want to run your tests, right? Because you want to make sure that your software actually works. If you use PyTest, you're lucky because you can just configure PyTest to output files that Jenkins can read out of the box. Jenkins is Java software. So it has, of course, an XML interface. But with PyTest, it's really easy. I'm not sure about other Python test frameworks. We have Collective XML Test Report, which is the clone wrapper about the Zulp test run. I will bother you with that. And then you can present those nice statistics about your failing or passing tests. And the same is true for a test coverage. So you can use the coverage package and the Jenkins cobertura plugin to actually show that to your users. So you have a nice interface that you also can show to your project manager so he or she can track your performance and see if the build is broken. In order to make sure that your software is not only in a working state, but also does what it's supposed to do, you usually need acceptance tests, right? And I'm a web developer. So what you usually do is you write Selenium tests. And we used that in the Plum Community for a long time. But around five years ago, we started with Robot Framework. And that really gave us a boost when it came to acceptance testing. Robot Framework is a generic test framework with multiple plugins. And one of those plugins is Jenkins. Sorry. It's Selenium, Selenium 2. So you can write tests in this nice BDD syntax. Human-readable, not only by programmers. And Robot Framework and Selenium will run those tests. And you have all the integration necessary in Jenkins as well. So you have a Robot Framework plugin in Jenkins, or a Selenium 2 plugin that shows you all the nice outputs of Robot Framework or Selenium. The cool thing about Robot Framework is that it gives you a full trace back. If your tests fail, it goes step by step through it. And it does an automatic screenshot of the last where the test actually failed. And you have all that in a nice output that you can access and see what failed. And we are also using Source Labs, which is a software service that you can use to actually run your Robot Framework or Selenium tests on different versions. So they offer you all the versions that you could imagine because you don't want to set up your own Windows machine. We tried it. Don't do it. Those services are cheap. Sorry for the advertising or use any other service. But user service, don't do that yourself. We tried it. Then one thing that is especially important for Python because it's a dynamically-typed language is static code analysis. So you're able to track possible bugs early. I guess you're all familiar with the tools, PAB8, PyFlex, Pylons. We created a wrapper in the Plone community around those tools called Plone Recipe Code Analysis to have our best practice testable. You can use that without Plone, but only within build out. And you can present all those. If you run those code analysis scripts, you can present them within the Jenkins violations plugin. And it gives you also nice statistics about all your violations, not only for Python, but also JSLint and all the modern stuff, CSSLint. It's all pluggable into the violations plugins. So you can really easily present all the information that you have to your developers or to your project managers or everybody involved. Then one of the things that is really important is notifications because people need to be informed as quickly as possible about regressions. And there are many different ways you can do that in Jenkins. The best way or the way that is most widely used is via email. And there's an extended email plugin for Jenkins that allows you to define rules which people you want to notify. So you can say, if the build breaks, then I want to notify this mailing list and that, or if the build is still failing, then I want to do this and that. So you can really define all the rules that you want. Usually, if you have a larger organization, you want to hook Jenkins up with LDAP. It also comes with a plugin for GitHub, for instance, or Bitbucket, so you can use the authentication with that. That's really nice. That's the cool thing about Jenkins that it has such a huge community that you have plugins for everything. And you also want to show the current status to your users. So you can use the Jenkins dashboard plugin to have a nice dashboard or you can even build your custom frontends. It's all there, you just have to choose. So in the Plone community, we set up everything that I just presented to you and we ended up with this still. So why is that? I mean, we put lots of effort with a lot of people into that and we built it all by the book and the build was still broken. Why is that, right? I mean, there are two reasons, actually. One of the reasons is that for Plone it's hard to have this one commit, one build thingy because we have those 300 packages and if you do a checkout, then it checks out those up to 300 packages and you can't be sure that this all happens in a timeframe before somebody else comes along, right? So that's pretty specific, so I won't go into that detail, but that's a problem. As soon as you have two people that could be responsible for something, they will point to each other and say it was the other one, right? That always the case. And then the continuous integration and testing team needs to clean up and figure out what went wrong and after that you can point at those persons and say, hey, it was you, but I had to clean up your stuff anyways. But the second thing that is not specific to Plone is that people break the build and they just don't care. I mean, it's not because they're evil, sometimes you just want to do a quick fix or anything or you do a commit and you think that can't possibly break anything, right? I just did that like two days ago and a good friend of mine then just, it took him like two or three hours to fix my stuff because it wasn't obvious because the commit really looks perfect and then he wrote in the GitHub commit message that he wants to kill me. It was like all my fault because I was tired and I just went too bad instead of like waiting for Jenkins to pass. So it's not bad people, but sometimes those things happens, right? You break the build, maybe you don't check your emails or anything. Our build takes still around 40 minutes. So people break the build. So how do you prevent that? As I said like a couple of times before, continuous integration is a development practice. So what's maybe even more essential than a good software that helps you with that is actually that you practice that, that you have agreement on a team. And I think we gained a lot of experience with that because we have like those 340 core developers that's actually from our last year's conference in Brazil. We have over half a million lines of code. We have over 300 core packages. So we have quite a complex software and like a huge team of developers. It's not like a company where you can tell somebody to do things, right? So we need some agreement on the team how to like keep a green build. Fortunately, some smart people already thought about that and came up with a few continuous integration rules or best practices that allow you to keep a green build. The most important one is do not check in on a broken build. It's the most important one is do not break the build because that will not happen. People will break the build and it's okay to break the build. It's just important that you don't check in on a broken build because if the build is broken and somebody else comes in and checks in, then things get complex. You get more test failures and you can't figure out which commit was responsible and then people will point at each other and say it was that guy in. It wasn't me, right? And then things will become complex. So what you should do if you break the build, the team should stop, the entire team should stop and start fixing the build because you have a real regression, right? Your software's not in a working state and nobody can commit if they take this first rule seriously. So the team should stop and work on that. Sometimes that's not working. Then it's also fine to just revert your commit. Sometimes it's obvious what you can do to fix it and you can just fix it but there should be a time frame within where you should fix the bug within that time frame, right? Because otherwise you will block the build. But if you do that, if you stick with those rules, you can actually get a green build most of the time, like not 100% of the time because people will still break the build. This is what CI is for, right? Our tests take quite a long time to run. If you run them all not in parallel, like we do on the CI system, but sequentially, then it takes more than one and a half hour to run our tests and you can expect everybody to run all those tests, right? So people should use the CI system to break it but not for long. So if we go with the continuous integration rules and have our setup, we have proved that our software is in a working state all the time. That is pretty cool for our developers because if developers do a checkout, they know that the software works, right? Before that, they checked it out, wanted to fix something and they had like a broken build so they had to fix something else. That's frustrating. We could make faster releases because our release manager did not have to ask the two or three hardcore developers to like fix all those bugs for a day. He can't just make releases because our build is green, right? So you can deploy at any time. Just a few remarks about additional things that you could do, scalability is important. You should definitely, if you have a larger project, consider using a server node setup for Jenkins which Jenkins allows you to do. So otherwise, if you have a lot of jobs running in your Jenkins machine, then your UI will freeze because the server is busy. So do that on the nodes. Use provisioning, there's nothing worse than a CI system that does not work reliably and behaves differently on the nodes and you can use the Jenkins port allocator plugin to run things in parallel because this is what you want to do. Then if you have your CI system in place, the next step would be continuous delivery, not continuous integration. With continuous integration, you automate your testing process and your integration process. With continuous deployment, you automate your deployment. The idea is that for deployment, you just have to push a button more or less and automatically you will deploy. A lot of companies do that these days and Jenkins grew from a CI system to actually a system that can do CD as well. And we also started to work on that. We're using Zest Releaser, for instance, to do Python ac releases. It's an awesome package. If you do ac releases by hand, stop and use Zest Releaser, it's perfect. It's a really great piece of software. You can use DevP, for instance, to make ac releases or wheels releases to test then actually your deployment. And on the Jenkins side, there's a new plugin since like half a year called Jenkins Workflow Plugin. It's really a game changer in CI, in my opinion. It allows you to create really sophisticated workflows within Jenkins to run certain steps in parallel or sequentially and notify people. And it's incredibly flexible. I already played around with it and we definitely plan to move to it. So if you start with Jenkins, I would definitely check it out. It's really awesome. So to summarize, if you have a CI system and you integrate frequently, you have an automated build and test system for each integration and you report, as soon as possible, you can get a green build like most of the time, which gives you a proof that you have a software and working state that you can deploy at any time. You can ship software faster and better. It's more fun for developers, not frustrating for them because they run into like failing tests. And Jenkins in the last four years has been great. It's like you have plugins for everything. It's a great piece of software, even though it's written in Java. So yeah, use it. If you want to know more about continuous integration, I highly recommend that book on the left side called Continuous Delivery from Jess Humble and David Fairley. They came up with those continuous integration rules. There's another book called Continuous Integration from the same publisher. I would recommend to buy this book because it has everything and the continuous integration chapter in that book is really great. I have to, I bought two, both of the books and buy this one. You don't need the other one. And on the right side, this is a blog post. There's also below where the URL is below, where I wrote a blog post about our CI setup with all the plugins that we used and all the approaches, so that's more in detail. If you have any questions, feel free to ask me on Twitter, on IRC, on my blog. The slides are there. And thank you. Thanks, Timo. We have time for two questions. If there's anyone who has a question for Timo, put your hand up. Thanks. Hi. First, I wanted to say that with NOS test, you can also output XML, which can be interpreted by Jenkins and displayed in the web UI. And my question is, what do you do with flaky tests? With flaky tests. Flaky tests? Tests that sometimes fail. You can't prevent. Yeah, I mean, that's hard to do. What we usually try to do is to make them work reliably and if they don't work reliably, we remove them. Because personally, I don't think it makes sense to have a test that fails randomly because that does not give you any information. If a test fails randomly, it's to new use because if it fails, it gives you no information. If it passes, it gives you no information. So we try really hard to make it work reliably. That's especially important for Selenium tests because the underlying technology is fragile. But you can make it work reliably and Jenkins helps you a lot with that because if you run things in parallel, then you will see all kinds of effects that you don't see on your local machine. You have to make sure when you run Selenium tests that you make sure that everything is there because tests can run slow and fast and it's not easy to do. But in my opinion, it's worth the effort to have reliable tests. Yeah, my question was that could you quickly comment on how often like developers step on each other's toes when you have so many repositories and developers like does it happen often? Do you regret having split them out instead of having them in one git or do you use git sub modules? Could you please comment on these things? That's the big question that we always ask. Does it make, I mean, we split our, we had a big monolithic software block and we split it onto multiple packages, multiple repositories and it's really great if you can, as a developer, pick things and improve certain packages without having to download everything, right? So that's a great thing and we don't want to lose that. On the other hand, we see the amount of work that is necessary to release and keep track of all those multiple repositories and we haven't really solved that problem that you have one commit and one build. We are close, but we don't have it. So it's a trade-off in the end. It's hard to say. I don't think that we will go back to a one repository approach, but I can see the advantages that you have. Yeah, yeah, that's possible. Yeah, that's possible, but then you still do a checkout and then you can't be sure. That's basically the same. We are using actually Mr. Developer, which is a tool that checks out all the packages for you and makes sure that you have the right branches. It's pretty sophisticated, pretty cool, but it's complex and we try to store known good sets of this. So we had for all our 300 packages, we stored the version numbers or the commit hashes and stuff like that and we tried to make that reproducible, but it was just too complex. We failed at that. That just did not work. Great, thank you very much, Timo. Great presentation. Thank you.