 Guess it's quarter to now So you guys enjoying Drupal gone enjoyed Drupal gone Come on. Have you enjoyed Drupal gone? Excellent excellent All right, so Continuous integration will solve everything I Think hopefully hey My name is Martin Bell I am a Drupal developer doing it for about five years That's Here right there about me if anyone knows about it Twitter Drupal.org go I joined cap last year Where I did a lot of this work on continuous integration Got a lot of experience with large development teams working Together and why continuous integration? It's very important on large teams So a little history first quickly The process is to continue integration use have been around for yeah since the beginning of computer development programming I found some reports about the IBM when they were developing the OS 360 Built their system four times a day. I have no idea how terrible that would be on that kind of a system Maybe it worked but in later years extreme programming and agile have kind of brought continuous integration into a fully-blown process and defined end-to-end system and Specifically Martin Fowler and came back are two prominent names there Martin Fowler has written a great great article about this continuous integration what it is and How how it's used I'd recommend you read it it probably is the one one true article about it in short so Yes, that's from his article continuous integration It's a software development practice where members of a team integrate work frequently usually daily Leading to multiple integrations a day What he also later says is the continuous integration then Causes integration to become a non-event as in it happens. You don't care about it It it you get a report back saying your code is good your code is bad As in the world continues it's never ending so it's all time going on however within that cycle you still have start and stop points so Every day for instance, it might be doing kind of 10 times a day You might have a continuous integration run at the end of each run You'd see a report coming out and then you can add some new code have it rerun That's when you start the point So who used to see I us developers Yeah, actually how many are developers in here? Wow awesome site builders Hey managers project managers Anyone I left out Themers. Oh, sorry. Yes. Themers. Sorry Excellent cool So, yeah developers Use continuous integration. We use it locally on our machine. We run a build script that Builds our sites. That's con considered continuous integration When we check in code into an SEM that's part of continuous integration because then it goes into being code reviewed So I builders use it because they can easily build a new site Mock around do whatever they want and not worry about the fact that They can mock up the site completely destroy it rebuild it in a split second and just go on about their life It makes life a lot easier for site builders as well managers love it because they get an overview of What's going on in the project? How is the code? How is the health of the the feature you're doing a application you're making so All-in-all continuous integration is a very good thing for everyone really So How many I've seen this? You know if you're a single developer Cloning code maybe two three developers, right? Cloning code hack code to the patch someone reviews it and This is a good patch. Yes, you watch and repeat if it's a no Maybe that guy comes over and clubs you over the head with a hammer. Hopefully not the hammer So this works this this works in small team, right? This works when you're three four developers Works fine. You talk to each other your round one table. Maybe What happens when you scale it? Let's say you have this. I love my art Let's say you have this so you're a senior developer you have five developers offshore and They are working on your module. You're responsible for a feature an application on the site they work way and all the communication they have with you is instant messenger email a video conference called at 10 o'clock in the morning every day and That doesn't even then count for the maybe six seven hour difference between you, right? So when your communication line is this thin because you don't actually have that human interaction between you How how How can you get that same kind of communication that you do when you only five people? And that's why I could continue integration is important because it helps a lot So This is your group, right? This is one kitchen. You're responsible this kitchen. There's ten others and They all talk together true. I am How do you how do you handle that one? cool xkcd good code We Usually like to get in some way in between there. We'd love to get some way in between their ways between fast and right now right codes Code well Don't know You know the picture you've all how many have you have been here? yes poor good codes So why UCI? It improves the overall view Wow, that came out actually a lot worse than I thought it would It improves the overall view so you can see further you can see Further into the future of your project. You can get a better overview of how your code is the health Makes goals easier Like I said future, you know where you are now. You can easily say where you're going to be By knowing that your code is working or not quality assurance so Those are time and continuous integration They talk about continuous integration. They always talk about simple tests and Qa right quality assurance in a In Drupal is an interesting beast Need to pass some tests, right? Yeah status of those tests passing And you set the flag in the ground But how do you know everything on your site works? Are you going to sit down click every single link and just hope and just you know Usually you you throw it out there and you you really hope right you don't know every single link unless you have 100 people sitting around all day long clicking links. It might happen So where to start on this where where do you start with this? monolithic task at the same place they're building by the way Wow, that's got that guy's got you know Sitting there out on there No, thanks. I'm all right build You start with your build You start with scripts Here are some On to talk about when people say continuous integration. These are some of the ones come to mind at least when it comes to the php world My colleague Marcus Douglas was talking about Capistrano Webistrano a kind of It's between servers and build scripts because you can define things in Webistrano and but you can also have it run the same things thing Is an excellent ant clone So in some ways You can define a lovely xml layout and it will Run it to its hard content and you get actually decent Status thoughts. I'll show something xml soon after this You can write it all in php Gives you great flexibility But it's kind of like writing your own cms again, you know You were sitting at home thought yeah, I'll write my own cms. That's a great idea I think I came to my third one before I felt this is terrible Drupal yes So now you have your build script Josh make to sort of build script You have your build script Service you need something to orchestrate it You need something to go off every day Burned to sleep no problem at all just run it and tell people what's going on On top of the list I'm putting Jenkins Mainly because it's been around as Hudson for quite a long time Some when they got both Oracle Oracle decided wow. Well Hudson. That's our name We'll not let you have that anymore. So they forked off and created Jenkins, which is pretty much Hudson We'll even find Hudson in the words inside Jenkins some places Cruise control not a Java one Have not used at all. Sorry to say Heard some great things about it. The UI looks really nice build bots python based one and it's the UI is Okay, I wouldn't put it in front of a first-time user Go on Wikipedia. Just look up continuous integration. You'll find massive list of service that will do all the same tasks and a lot of them proprietary Martin Fowler wrote one for ThoughtWorks called go So Now that you've got your build script and your server ready What's next? Well, now you need to create those tests right those status reports The things that managers love to see and the things that us developers really need to know what we're doing wrong simple test I hope everyone knows this one Anyone not know it Yeah, cool Everyone's with it triple simple test Write them in every module you make write a simple test, please it makes The work for guys like me who have to you know run these tests all the time much easier if there is one there Selenium Selenium is a cool tool It's pretty much automated human eyes in some ways You can sit down install firefox install selenium plugin and click around And it will record all it. It's kind of like a glorious macro script Um, but what's really cool is when you get into gridded selenium. Sorry Um So that you can have A massive grid of machines all running different os's all running different versions of the browser And they all will then selenium will then automatically check on each of these browsers, right? So that you will get back a report saying this pixel is not where it's supposed to be or But this image is not where it's supposed to be You can get really really good status reports from selenium about what's wrong with your actual graphical outputs Simple test checks your html It's all right selenium actually tests what the browser shows you PHP mess detector. How many have you heard of this? All right, it's not completely unknown yet PHP mess detector is kind of a It checks for security holes checks for At least that's what they said the checks for Uh, it checks for kind of Stupid things you're doing PHP code. It will look at your code and go like you're stupid. You've done it wrong redo it It it's uh, it's uh, it's one of these you add to the arsenal after a while once you've got the two top ones done You start adding mess detector. It's kind of a developer store But it goes into continuous integration. It goes into the Jenkins server Jenkins runs it each time and says if you're stupid or Coder In drupal code module Guys using it. Everyone using it. Hopefully. Yep. Cool It uh, you all know what it does. I'm looking to go into xh prof So your code is running It's Running great. It works. It's got all the features you want Is it running performance? So your page takes 10 seconds to load. I've seen it seen 30 seconds even Why Why does it take 10 seconds to load? How do you track that one down? I mean Drupal's got some built-in tools. Yeah, they work But What if it's in the boot script or why why if it's all the places? So why if it's not even in the Drupal? Why if it's all the parts of Of the php when you're going off to database or all kinds of things xh prof was written by the guys at facebook Don't don't this is right Don't worry It's quite good actually. Um It's straight into the devil module now and You literally install it you point devil module to it and it goes off and is happy And you can easily on each page load you will get up a link click it It will show you each function how long time they took in the cpu And how much memory they used I'm sorry to say my Jenkins VM died So I was going to show you xh prof that my Jenkins VM died like literally last night. I've been Hacking my way at it and it's annoying the live living daylight out X debug you want to step through your code? How many of you have tried to step through php code? Cool Excellent you guys are actually using you know tools that are helping you. This is good. You're not just putting in breakpoints on exit and exit book It's actually got some performance tools as well and it's got code coverage tools strangely enough Um, it will allow you to step through code And net beans has a great, uh interface to it Uh, I've never managed to get eclipse working with it And there's some others out there as well PHP cpd have any of you heard it before? Wow too php copy paste detector So you have copy and pasted code It will tell you Or not you someone else it will tell you you can go to with a hammer now and hit them over the head It's great running it sometimes then you just go like why so that was all the tools Let's move on again So in continuous integration The idea is to do it often right There are a few Three short mantras so to speak of this It has to be repeatable doesn't matter What steps you think are complicated? It's always repeatable and it has to be repeatable And it has to be automated repeatable you can't have a human interface going into it and clicking some buttons It has to be able to run at any time If you have a database dump coming back every day and you're deleting all the old ones then keep some of the old ones Uh, you need to be able to run it at any point at any time Otherwise, it's pointless in really having it because you might have checked some code that was I don't know Five hours old you've checked in a bunch of the helpers offshore and a bunch of the helpers here have checked in loads of code What the hell do you do now? They have to be known steps Don't use black boxes. Don't use something you don't can't look inside and actually know what it's doing Um It can be scary If you if you're using something if you're using some magical tool That gives you loads of interesting results that you actually have no idea what it's doing inside So yeah, it has to be known steps Right configuration. I'm not going to talk much about it. I'm just going to tell you Something about it because there's been plenty of talks is weak about configuration There's plenty talks about deployment developers hands off You're not allowed to touch configuration when the site is built No one on your team except for site administrators allowed to touch a site when it's built Doesn't matter how complicated you can code it in hook update features SQL script But the point is this is why the repeatable If You have to go into the site to click some button somewhere or set some smtp host or something like that It's not repeatable anymore. Continuous integration is broken You're stupid so Very simple If if you ever think all right, I have to go stop configuration Come back to again going back Configuration has to be able to work across multiple environments. I mean if you have one environment You don't really need continuous integration Uh, if if you're running it from your local machine You can try I mean Why not run continuous integration happy building a local machine every day and It could work Don't really need to worry about configuration too much If you have one site good for you, you have got a bit further Um multiple sites in configuration gets interesting So you have a few sites here And you have different environments. This is actually quite few. I've seen like 10 I think was scary That's your local dev machine You have to be able to connect the site a configuration with the local dev machine So in thing for instance, you can do this easily. You have properties files. They can override each other You can set a drosh to automatically go for the properties files and pick up certain properties to Use as your site configuration Then same for the environment I know this is in Drupal 8 loads of talk about this. I haven't followed it too closely. I'm sorry Same thing for testing this works in Drupal 7 so same thing for testing And but now you have to be able you have to check in that code Both of these has to be checked in because the next developer has to be able to come along and just run the script And instantly have the same environment as you Same thing for testing. You have to check that code in the configuration has to be in code has to be able to go production Now gets interesting in production. So if you're moving into production from testing, hopefully in testing and production, they're the same But they might not be you have to be careful because You if you accidentally check in some configuration that you know enables everyone to edit everything So, yeah, just a few things So handling the traffic You guys think I'm going to talk about traffic shaping now something don't you? All right well sort of This I like it So what if your offshore team you have 50 offshore people 60 70 They're all checking in code all the time while you're asleep. They're checking in code you Yeah You're talking about unbelievable amounts of commits You come in in the morning run svn up or get pull You sit there for a little while and you don't you can go dig through the log I tried a few times it was terrible because the interfaces for these logs aren't exactly beautiful and you You you don't know 100% whatever what all the changes are right and you're the senior developer remember Gary how many have heard of it? How many have used it? No one Really? Well, you should It's a git repository So it's You know if your project is on git good. I'm sorry svn. I looked around I couldn't find one equivalent Sorry It's kind of like the alas in stack where you got fish eye but There's differences It's also code review tool. So like I said, it's like fish eye However, Gary is Different and I'll show you why now. I'm I'm actually gonna do live demo. It's scary So It's disappeared wow the nexus multi display handling is Lovely. So here you go. We got a few projects going on here So We could start with some build files Can you guys see that too small? Can you guys see the back can't see it at all cool. I'm gonna try to make it bigger How'd you do that in eclipse? Sorry Gonna make it easier Gonna use something actually know a lot better Uh, no, no Now like I said, it's terrible It's all right. I try that one control plus It's fantastic, isn't it? Almost there I think control post works on this Can you guys see it now? The back Guys hit the back cool So this looks like an anscript, right? Few differences. We got that one there We got a few things here So this is pretty much a thingscript You have your drosh commands, which is in the dribble community. It's a add-on to thing So you have to go and get that one if you use thing It works really really well, even though it's still in sandbox mode Can really do a lot anything you can create directories you can set permissions And it has Some cool concepts around it targets So You have to for instance first prepare for prepare Sorry My mouth is not connected to my brain anymore First have to prepare your directories, right? So you run drosh make And I wanted to be a development version. So I've got work and copy research And then you have to install the site and literally site install It's easy Thing makes your life easier. It really does So is it a bit like puppets? um Yes and no It can deploy files And it can configure that's not a problem But it's really not for system configuration. It's for drupal and php projects So you don't yeah, I'll I'll take questions at the end. I still got plenty of time. Don't worry. I'll take questions at the end Try to remember them write them down something. I'm almost done So let's move on quickly to garret So let's make a code change Don't want to check in I'm going to go ahead and make a change to my make files Although they're not really that great. There's nothing in there really. I want a eh What what what modules should I get suggestions? Oh, come on paper module someone quickly. Sorry. Go there. Okay. Cool. Go there. And I want to put in my software I can show a thing wrong. Oh Sorry, if you can't see this, I will blow it up. It's a make file. You guys have seen a make file before. I hope really save that go to my Again, come on Linux screen So Yeah, how many have used git and how many use svm? So how many use git? Cool, how many used svm? How many feel comfortable we get? Cool So, you know a bit about how the uh branching and how the check-in can work sometimes with a git repository nice So, uh, just I just changed my make file, right? Stop this. Yeah modified base make cool. And I do a I can never remember this one Git push Actually, no, I don't sorry my bad I do a git commit first, of course. I'm stupid Already I'm messing up. That's amazing Message there changed All right Now this is the interesting one about gary So you have your push you push it through a ssh. You push it to a custom port So gary wants its own ssh server And There are a few reasons for that mainly because it's all in java. So it's all encapsulated So when you run gary, you can have all your git users Only in gerrit and you don't have to create them linux accounts or use any other means And it's very very easy to register new accounts. It's literally a google account Or you can use your own authentication with it You create an account you put in your pub key. That's it. You can start checking in code. It's like github It will be like running your own github on your own machine Except it's not really github so I want to push into Make files I'll push my head into the references for master Go Of course I'm using a I thought it was smart All right Don't fail Wow You don't believe The number of times I've actually checked in code and it's worked fine. Oh, hey, there we go. Trust master computer being this slow So You know, we'll get checking Even gives you a link to the new changes So now we've checked in What happens next So you used gary before some of you guys have seen this So I updated some files earlier obviously Now what this is is all the change requests that have come in all the patches that have come in And they're not actually in your git repository. They're not in your main git repository yet They're still in the queue waiting to be checked in So remember what I said earlier? How do you handle the traffic coming in from all the developers? This is how You you you you gate it you you stop them from being able to check it in to the main repository that goes into Jenkins And it works really well because you can have A verifier and a code review. So if you're going to look at the one I just did now It needs to be verified needs code review. How are you guys at the back seeing this or not? How are you blind? I was cool good So, you know, you got your all your author committer blah blah so you can do a diff side by side You can see I added my Contrib you can double click you can add comments which go back to the developer You can tell him you're stupid there um It it's really really nice. It really will help if you use carrot And each time someone checks in it goes in here before it even touches the main git repository that that goes Even if it's an unstable git repository It will help you catch The really stupid stuff So I'll go back to all my open months Changed even says review and progress Hey So I hit now I have two accounts. I put up two I can I've set up a permission so I can review it myself and I can even prove it myself But I'm not going to because basically what happens is that you would have two people Or you'd have Jenkins. I'm a little person The first person would say looks good to me But I need someone else to prove it This can either be Jenkins Which could be scary, but if Jenkins says yep, the code's fine It can improve it and say it's okay However, if you just want two people, you know Publish comments done And if we go to my Wow That's my firefox It's gone. It's completely gone. Excellent. I'll just bring it up again That was nice. Thanks firefox So we'll go into my Good review again I'll sign in I used a yahoo account my second account. Hopefully it still said stayed signed in Right. I can't actually sign in can I? Hey No, I'm not using the druplicon wi-fi. I've had so terrible. How how you guys have to do the druplicon wi-fi. It's been good Yeah, it's been it's been that one, hasn't it? Yeah Yeah There we go. Coming to my home. Hey connecting waiting for Sorry, this is going much slower than it usually goes, you know Live demo crazy Excellent weather is good Sorry Cheers Try this guy quick Come on Mean aren't you? My computer's being mean to me. It knows I'm doing something In front of public and people of us. Yeah, so let's go just back to the normal one. So If you had an excellent setup you go review and you say it looks good to me approved Publish submit requires verified still interesting So we can go and play around with We can now start looking under the hood a bit Access They were verified I'll move Myself in the much easier Go Yeah Right That's how permissions in uh gates are done Very easy. You can set up user groups. You can register people. You can add them to user groups. They get permissions. It's easy I think I'm putting people to sleep Excellent My mission is complete Forget that in any case If you review it you verify it goes into your main gate repo Jenkins gets a notification So there's a gary plugin in Jenkins. It runs. It gets a notification. It does the build So I'm not going to talk anymore. I'm going to Say that You've listened enough to my voice now gonna say thanks to the Drupal con group For doing an excellent job and say thanks to Drupal association For you know maintaining Drupal and community and etc. And I'm going to say thanks to you guys the community Because without you I wouldn't be standing up here talking to you now. Would I and without you we wouldn't have an amazing amazing CMS system I did use flicker stuff xkcd Thanks Questions I'm open. I saw some earlier. Don't be shy. I know you want to go you can buy all means yes It's more of a statement. Sure Yeah, of course. Yeah, of course. I mean I do it for my own neglected terrible sites And more my point is that you see most benefit when you have big teams Because when you just two or three guys around the table it it's not hard to talk to each other and saying like Yeah, I did that thing to the code now. Can you You know pull Right, yeah Right, yeah, okay There you go. Yeah, it works. Yes, right So do you run Jenkins on every commit? depends on how you have set up Jenkins If you have set up Jenkins to have projects for every single little feature and every single little module you have Then yes, you do Because then you get an instant response for that one module But if you have Jenkins run the whole site And if your site contains, you know 400 modules If they do exist Then no You have it check every 15 minutes or every half an hour And then you they will run then if there has been anything and collects together a whole bunch of them Anyone else? Yeah So if you have a preview environment So as in you mean if So I'll just reiterate question for the recording if you have a preview environment would be heavy to run Jenkins every hour Really depends on your setup But if I'm understanding your question correctly, do you mean all the content that's in there? How do you migrate that across or Right interesting, okay No, because that's a kind of environment when it goes to the client That's kind of a user acceptance so Jenkins Would have a task in its job to Push to that server But that would almost have to be a manual if it takes that long to build your site You have to have a manual button in Jenkins and when when you're happy with the state of the code You say go and it builds that site But having Jenkins running every 45 minutes unless you have a very split up environment where you have multiple preview servers And you somehow have a load balancing front that can choose which server that's just been built just finished building, etc right So when it takes that long to run against a preview environment, you do it manually You're going and press the button manually the project manager doesn't make it Yes One of the talks earlier this week was on kind of that small differences with feature they used features And an install profile So what you do is you'd have two different install profiles And you'd have a enabled in one install profile and not in the other Uh, alternatively you can do it in the thingscripts You can have it enable module you can have two different thingscripts that run on two two different environments There's several ways of skinning this cat of enabling disabling features and modules Sorry. Yes beside