 I'm here to talk to you about continuous integration, and a bittersweet love story. First of all, a bit more about myself. I'm head of product at ThoughtWorks and our product division. Ken mentioned a little bit about ThoughtWorks earlier, and I have one more slide, but that will be it. Basically what I do there is help our teams build products for software development teams and dev ops teams. We have a boost downstairs. We're happy to talk to you more about that. And if you want to get in touch with me, I have a Twitter account. Today I want to tell you... Oh, I took out my ThoughtWorks slide. So there we are. So if you want to come and see us, we're downstairs. So today I want to tell you a story about continuous integration. I will be referring it to it as continuous integration or CI. And it's a story about how we fell in love with continuous integration, how that love grew and progressed. And then we take a sort of reality check at where we really are with continuous integration and ask ourselves, you know, did we really fully commit to the practice of continuous integration? So I'd like to learn a little bit more about you. So I have a couple of questions for you if you could raise your hands to answer them. That would be great. First of all, who has heard of continuous integration? Yay! OK. Who is practicing continuous integration? I'm going to call that maybe 40, maybe 50% of you. You can ask yourself that question again at the end. Maybe I will ask you again at the end, but think about it as I go through this. What you know about continuous integration and whether you think you're practicing it or not. So for those of you who didn't know, because it looked like we had about 50-50, well, we had the majority of people have heard of it, but maybe only 50 were practicing it. I just want to do a recap of what continuous integration is and also talk to you about what I think is important about continuous integration. So basically continuous integration is a process where software engineers or we might call them developers, check-in code to a central source repository. That's the source repository that often a team shares, sometimes at the whole organization shares. We then push that code to a CI server. It builds that code, it tests it, and it gives the development team a result. It looks something like this, so our code goes to a source repository, it's then going to go to a CI server, and then it's going to come back around and tell us, did my check-in break the build or did it give me a positive result? As you can see, it's a cycle, it's a simplified version here, but it's a cycle, and the whole idea is that it's giving you feedback about the check-ins that you're making, about the code that you're writing. But I think when we describe this process, and hopefully those of you who are practicing, this is something that you're familiar with, I think that we miss some key parts about continuous integration that you don't really think about every time that you're doing this process or when you learn about what continuous integration is. And I think two of those key things are represented in this quote here from Jez Humble on continuousdelivery.com, and he says, in CI, developers integrate all their work into trunk, it might also be known as mainline or master, depending on what kind of source repository you have, on a regular basis, and he says, at least daily. So the two key parts there are that you are integrating to your master branch. Call it mainline, call it trunk, I will probably mix those words up, but I mean the primary branch of your source code repository, and you must be doing that at least once daily. So in case you forget, you will see these words a lot, master, and daily, I think those are two key parts of continuous integration, and when you think about your practice, think about whether you are doing those things or not, and I'm going to talk more about them as we go through. But there are also some other things that are really important as well, so I just want to quickly sort of revise those for those of you who have heard of this before, or think about them anew for those of you who are new to the practice. So check everything in. This would seem potentially a pretty simple thing, but you'll be surprised how many times I've turned up at a developer site where we have people working on their machines and they're not checking everything into their source code repository. They kind of make this assumption that their developer machine is probably like the staging environment or it's like their production environment, and it is not. So a good way to test this is if you have a new team member, join your team, or if you get a new machine to work with, just from scratch, from your clean laptop, have everybody check out your source repository, and if you can run your software, then you're probably there, and if you can't, you probably are missing something. Build every commit. This is super important. So continuous integration. It has the word in there continuous, and it's about integrating. We don't skip commits, we don't jump commits, we don't just do it on a nightly schedule. This is every single time I write some code and I put it into my source repository, I want that cycle and that process to run. I want to get feedback on every single commit, every single person, every single time. I also obviously need to have self-testing code, so it's all great if it goes through my repo and it goes to my source control and it's always a green build because I'm not actually testing that. So we need to be working in a way that we're building self-testing code. So when I check in a piece of code, it's probably great to check in an associated test to go with that so that I can confirm that it's behaving as I wanted it to and it's behaving that same way with the rest of the code that's already in my repository or that that's being checked in with the rest of my team. Because all these things, we're doing it every time and we want to have self-testing code, you're most likely going to need to automate that. So think about making that not a manual process. I assume at this point, most of you are not doing that, but for those of you getting started, you could start with a manual process but I highly recommend you think about automating that and I'll talk a little more about automation at the end than some of the benefits of doing that. One of them is that it lets you do this quickly. So this is a feedback cycle and I want you to do it every time you check in. So you need that feedback to be fast. You really want to get results quickly so that you can change your behaviour, make a change, put in a fix, do the same thing again. And things will go wrong. In fact, if you are not getting red builds, I question whether you truly have self-testing code and you want to embrace that failure. I think we've heard it, some of the other speakers this morning. It's not a problem if something goes wrong and it's the same case with continuous integration. That's why we're doing this, to bring that feedback forward, bring the risk forwards, break our assumptions and see failure. And the most important thing is that you make that failure visible. So every single person on your team should know if you have a broken build, if you have a red build, if something went wrong, because you all want to be fixing that. It could be a siren, it could be a light. Some of the teams I've had have a Homer Simpson who goes like, doh, every time it breaks. I mean, whatever you want, whatever makes you happy, have it be visible and have everybody understand what failure means. And then you're going to fix your failures. So I don't want to hear the doh and then hear it again and then hear it again. I don't want to go to your build monitor and I don't want you to go to your build monitor and see red, red, red, red, red, red. Oh, it's because so and so did something but we're waiting for him or her to fix that. You need to be stopping the line, fixing your failures immediately because that was the point in the first place. We don't want to leave it forever and then wonder what happened, what was that original breakage. The sooner you fix it, the better you get back to a state that is known to everybody and the better that you can get feedback to the rest of your team. So those are some of the key principles of continuous integration that are not necessarily obvious in the practices that you're probably doing every day. But lest you have forgotten the two things that I think are most important, check into your master, your trunk or your mainline and do it at least daily. Okay, so why did we fall in love with continuous integration? I think I've started to talk about some of the benefits there in the key principles but there's three or four things that I want to talk to you about that are truly beneficial about doing continuous integration and why I think it's a practice that a lot of us really think is a fundamental practice. Primarily it's going to help us avoid merge conflicts. So I am assuming most people are not writing software for a significant amount of time, so I'm talking months or years before they have to integrate that. But even so when we look at software development teams even in the agile world we're going to see people having conflicts between the code that they're writing. Be that within the same team, across teams or maybe across departments in a large organisation and the purpose of continuous integration really is to avoid that that merge conflict hell at the end of a project and bring the hell every day. Every time that you check in your code you're going to feel that little bit of a rub like is it going to work with the rest of my code is it going to work with the rest of everybody else's code and that's perfect because the tiny bit of hell happens very quickly you'll know that it's broken and you'll fix your failure immediately and then it won't be this daunting thing that makes your project push the dates out and be like oh hang on we just have to do our merge stuff which is going to take us three weeks you should have bring that forward. It also gives you fast feedback so I talk about feedback a lot we talk about feedback a lot in all of the things that you've heard today it's really about understanding what is going on with that code base and I really truly feel even as a product manager the power of continuous integration is that you truly get feedback quickly so for someone like myself who I'm really interested in did I get the requirements of my customers correct did I understand my business very well continuous integration is a way for me to do that on every single commit now I know that what is there is working if it's a green build and I can look at that quickly so it's feedback for my dev team it's also feedback for people who might want to move that source to a different environment and it's also feedback to the business because they can take a look at what you're building there and find out whether it's what they wanted or not and that feedback helps us reduce risk so we all make assumptions I make an assumption that I understand my customers as well as I do you might make an assumption that the widget that you wrote was the thing that I was trying to describe to you you might make an assumption that you all agreed the architecture as a team correctly or that you chose the right thing to write this in all of those things are assumptions there we're making a guess that they are correct and by doing continuous integration we have an opportunity to test that assumption more quickly and reduce the risk of it being wrong the final thing I want to talk about with continuous integration which I think is not something that people often talk about why continuous integration is important but I really feel this with the teams that I've worked on is that it really brings this idea of collective ownership to a development team so if you own your own code and it's in a branch and you work on that code for a while and you're all into it and you don't check it in it's kind of like this precious baby and then you sort of give it to your development team it's like this thing that belongs to you it's kind of hard for people to be critical about that it's kind of hard for you to give that ownership a way to somebody else with continuous integration I really see teams who are much better at saying this is the thing that we do together this is the software that we create for our business so it's no longer my precious code you get to have an opinion about it because maybe you wrote some of it too it's also not just my failure when I talked about fixing the build immediately if you have a broken build I don't mean you it doesn't have to be you if you broke it often times you're the most likely to be able to do that but this is more about us creating this together and now it's our failure it's our broken build we're going to stop the line together and it's not just your fault so I think if you're thinking about building a DevOps culture this idea of collective ownership that this is about doing something for the group rather than a personal project continuous integration is a really good way to sort of start to build that culture within your teams and your organisations so hopefully those four reasons explain why I think continuous integration is a great practice, why I think we started to fall in love with it I want to take you from the beginnings of continuous integration and look at where we are now so this is our love story so far so the first writings or the first mentions of continuous integration we see in the early 90s and they can continue to grow during that time before in 1998 we see continuous integration being cited as a key process in extreme programming which is one of the earliest more developer focused agile practices and then it continues more and more people start to write and talk about continuous integration Martin Fowler it gets featured in the continuous integration book a continuous delivery book we start to see continuous integration service the first being cruise control in the 2000s right up until now where even in these kind of really managerial heavy practices software scaled agile framework for enterprises safe which some of you may know even they say continuous integration is really important so I think you know you've hit the prime time when something like that happens and of course myself included there are tons of CI tools out there everybody's got a new CI tool so we're good right it's a great practice most of you have heard of it a bunch of you are doing it all the stuff is happening I'm going to leave it's great but is it really great I actually think our relationship status with continuous integration is actually fairly complicated and maybe some of you are starting to think that when you think about what it is that you're doing for continuous integration and what it is that I described that I think is kind of ideal continuous integration I want to take a look at that a little bit further and peel back from what it would appear on the surface to what I actually think we're doing when we say continuous integration right now so first of all this is you those are you you put your hands up so on the surface most of you knew what continuous integration was so that's a really great thing I don't have to spend 30 minutes of this talk explaining to you even beginning to talk about what continuous integration is because it's a phrase that most of you have heard and a lot of you are also practicing it so that's a really good place to be so that is our starting point and I want to take a closer look at some of the data about the whole community so outside this room what is our community doing this data comes from Stack Overflow each year they do a developer survey this is from 2016 and they asked their community how often do you check in or commit code and you'll see that nearly 70% of people in that survey said that they checked in code once a day so you will remember what I really cared about once a day 70% of people I would say that this looks pretty good too we will look a bit further down if we look a little deeper into some of that information this is information from Gartner and this is from their customers and they specifically spoke to their enterprise customers who were or were planning to be agile enterprises so these are organizations who have either adopted agile processes or are intending to adopt agile processes so one would assume that these are practices that want to move quickly deliver value and sort of be on the cutting edge and practice some of the things that we've heard today in those organizations we see that 40% of them are practicing continuous integration which is not what I would like to see but maybe it's not too bad but I think the more shocking thing here is that 40% of those organizations have no plans to practice continuous integration and I think frankly they're not going to attain what they were hoping for and being an agile enterprise like maybe they'll call themselves one and maybe we won't care whether they are or not but if they truly want to achieve that I think it's very hard if they have no plans to even begin with continuous integration so I think that really brings us home to what the reality really is in a lot of organizations the next thing I want to share is more is deeper still than that people's personal experiences with continuous integration and how people like yourselves people in our industry actually think about continuous integration and this information comes from a research project that myself and my colleague did last year as part of building some of our continuous integration products we did a research project using contextual inquiry what that basically is is we come to people like yourselves and ask you questions about what you do and we also observe how you're working and then we talk about you behind your back and we make some findings we learn some stuff and I come and share it with you so that we can all learn from these things so some of the goals of what we were trying to achieve it was a survey contextual inquiry research around continuous integration and continuous delivery and what we specifically were interested in is identifying what people knew about continuous integration and continuous delivery and we also wanted to identify what they really did when they were saying they did continuous integration and continuous delivery and what their pain points were around these processes in particular I'd like to share with you the findings from how do people define continuous integration so I'm going to read directly from some quotes here so these are just some samples from some of the things that people said when we said how do you define continuous integration you commit some code something picks it up and builds and make sure it still builds I push the code to the master branch after it's been tested on the test branch that in return triggers something that gets deployed testing your code all the time executing the code that you write all the time making sure the code doesn't go stale so as you can see the vast majority and these are representative of the majority of people didn't give Jess's quote and some of them didn't hit on the master part and some of them didn't hit on the daily part but you know what honestly if I had not prepared for this talk and I did not have Jess's quote there would I really give a textbook perfect definition of continuous integration probably not I think that we can see that they're doing something that sort of looks like that process that we talked about so we realized like okay definitions aside what are people really genuinely doing when they practice continuous integration so again I want to share with you some quotes from what people actually said and these are in the context of either what they were doing or challenges that they found with doing continuous integration the most challenging thing is when you end up with a lot of features piled together in one commit any time a feature is finally complete then it gets merged into master we tend to do anywhere between 15 and 50 branches in a push so again these are representative of the majority of people in fact I'd go as far as to say that 90% of people who said they were practicing continuous integration described something similar to these things we also heard which maybe you will recognize in yourself we also heard when we said are you practicing CI a response that was something like sure we use Jenkins or yeah we have a CI server I think that often times we get well I know because I saw it we think we have a tool that does a thing because it says it does a thing and we assume that we're doing that practice and in the case of continuous integration the case with the people that we saw they had not remembered or they'd forgotten those key fundamentals of the practice so I think you can see that on the surface it all looked really nice we had hands raised and our Stack Overflow survey looked pretty good but the deeper you go into that you'll actually see that perhaps we're not truly doing continuous integration and it is fairly complicated my colleagues and I would like to call this this thing this thing that we're all doing CI Theatre it sort of looks like CI but it's not quite the same you're sort of pretending at doing CI instead of actually doing it some common examples that we saw in those quotes there is either you have a shared mainline you don't actually frequently commit to that mainline so your integration really isn't continuous or you run builds with poor test coverage that would also be CI Theatre as well you allow your build to stay red for long periods of time or which I think is very very common and potentially what a lot of us are doing is you're running CI but against your own personal feature branch so you have a mainline for everybody but the majority of the time you work on your own branch and you run CI there so you're getting continuous something there but you're not getting continuous integration sometimes we call that continuous isolation like you're continually isolating yourself in this little part and you're not coming out to the rest of your team so I really jokes aside I want you to really think about is that what your CI practice looks like and maybe it's working fine but question whether it is or not so you know who cares you might look like a software developer but not actually be one I mean maybe that doesn't matter maybe it doesn't really matter to you I'd like to take the opportunity in the last few minutes to explain to you why I really do think you should heed some of the things I said and take a closer look at what you're doing first of all I truly believe that you're not getting the full benefits from continuous integration if you're not committing to your master branch at least on a daily basis and you're not practicing some of those things I said you might be avoiding merge conflicts much better than you were before if you merge in on a twice weekly basis you're going to be doing some of that but you're definitely going to have situations where you're going to have much bigger sets of code we saw in some of those quotes and I'll tell you a story in a minute you're going to be dumping a lot more you're going to get some more conflicts you're not going to be getting fast feedback you're not going to be building collective ownership it's also going to be really really painful and more painful the quotes that we saw there were genuinely what people said when we asked them about doing continuous integration and there's one more story that I want to quickly tell you it was about a participant in our survey and he described I think he used the word horror of having worked on his own branch he had changed 100 files and 7000 different lines of code and he had all of that in a single commit to the main line and he entered a code review and he suddenly was overwhelmed by everything that was in his check-in and everything that he had done over a period of a few weeks and what he said is I wish there was a nicer way to decompose commits because it's hard to keep everything in your head and my colleague she has a much better poker face than I do she just went oh this is really interesting and I think I was going you could have separated all of those changes out and committed those changes sooner and you will remember this person had previously told us we have a circle CI so we're doing continuous integration he had thought that horrific situation was what he had to do that was his lot and after it finished we did take the opportunity to talk him through some of it but I think those are some of the things that are going to be truly painful if you're not practicing CI and you're doing CI theatre instead and then finally and I think this is the main reason why I want to come to DevOps events like this and talk about continuous integration is that I think continuous integration it's a baseline practice that you need to get right if you want to start doing continuous delivery and you want to be like some of the organizations that you've heard about today I truly believe that if you can't get continuous integration right it's really going to be hard for you to move to continuous delivery and continuous deployment there's no way you can be deploying multiple times a day to production if you're not checking in multiple times a day that's going to be pretty hard so think about where you want to go means that you need to think about these fundamental practices as well okay so let's assume that I have convinced you that a love story is not as hunky dory as everything seems but you'd like to make it better I have three top tips for your continuous integration love life first of all commit commit to continuous integration commit more often if you're not checking in on a daily basis maybe start on a weekly basis if you are on a daily basis I challenge you to do it two times a day in fact I challenge you to come and tell me how often you're doing it in a couple of weeks of time when you've thought about this practice because I think you can do it just a little bit more and if you're not doing continuous integration which a bunch of you were not think about it when you're starting to build that practice think about that being one of the key fundamentals is that you want to be checking in code regularly to the main source repository I like this saying it's frequency reduces difficulty it sort of sounds like it's the wrong way around like why would something doing something that's hard if I have to do it more often it's going to be that hell that I talked about more often you actually find that the more often that you do something the easier it becomes the less difficult it becomes and I think continuous integration is one of those things it seems really hard when you begin but if you think about doing it more frequently you will reduce that difficulty and then continuous delivery or continuous deployment will seem like a natural next step for you automate so again if you're not doing continuous integration it is going to seem really hard if you don't think about your tooling and your automation so think about that at the beginning but that is not the fundamental thing that you should be doing don't pick a CI tool for the sake of it but think about the power that it can give you in terms of being able to do that thing repeatedly over and over again and for those of you who are doing continuous integration I bet there are things that you do not automate that are painful in your process that you could think about doing so I challenge you every single time you do something more than once ask yourself can we automate that again if it doesn't help you do continuous integration more efficiently and better for you it's definitely going to help you do continuous delivery and continuous deployment and then finally some of you knew this was coming I don't really want to be a trunk based development zealot but I can definitely tell you that in the majority of cases where we talk to people in our survey who had pains with continuous integration it was those that were practicing feature branch based development and not those that were practicing trunk based development so again this is a challenge it's a tip, it's something you can take or leave I really encourage you to think about whether a single source repository and trunk based development is something that you can do in your organization because I'm pretty sure it will make your continuous integration a lot easier for you to achieve and if you can't do this make sure your branches are not longer lived than a day make sure you're merging them back to your mainline as often as possible ok just in case you forgot make sure you check into your master branch at least once daily and daily is kind of lame, make it more than daily that is our love story about continuous integration I think that the end of that love story is perhaps unknown maybe it's known now but I think it can be whatever you want it to be so go forth, do continuous integration and decide how it ends