 Okay, I should start. So, today I'm talking about achieving continuous integration with Drupal. Mostly what this talk is going to be about is more what that means and how to approach doing it. This is not going to be a talk about how to install and configure Jenkins or, you know, how to automate your post commit hook. We can talk about, you know, some of that stuff in the questions at the end. Mainly this is going to be about evangelism and why you should develop this way and how you can approach doing it. So, that's me. Hi. So, I thought I'd start off with talking about the evolution of a Drupal developer. How all of us start. How many people in here build Drupal sites or used to? Okay, right, almost everybody. Of course. This was me. So, I started probably the way we all started, which is to say I logged into my web hosting provider and started writing code. I downloaded Drupal right into the doc root directory and, you know, added modules. There's a reason people start this way. It's really, really fast and easy to get started. And, you know, you're building a site for yourself. It works great. So, it's definitely the quickest and cheapest way. And you make a change. It's immediately visible. It's easy to see. You can, if there's a bug, if someone visits your site and they say, hey, something's broken. You go visit the same site that they're looking at and you see the same bug and you fix it and it's immediately fixed, right? There's no overhead. There's no delays. And when you fix a bug, you know, your visitors immediately see the change. So, that's all terrific. You don't have to do release engineering, right? There's no, like, process where you hand off to someone. It's like, this is fun. It's immediate. If you didn't have to, if they weren't paying you for your job, this is the way you'd do it, right? However, there are many downsides, right? The slightest mistake, one missed semicolon and your site goes down. It doesn't work very well. And what happens is when there is a mistake and you're sitting there trying to fix it and then you make another mistake and your site's still down and you're trying to fix it in a hurry and everything goes crazy. I mean, obviously, it's kind of a panic. Also, a little bit more subtle is how do you demo a change, right? So, let's say the site's up and you're building it for your client and now you want to have version two and you have nowhere to show a version two because if you make a change, it's changed live immediately. And, of course, if you have more than one person, it just doesn't work at all, right? You can't have lots of people hacking on the same actual dock route at the same time. They're going to save files on top of each other. Version control won't even help you in this case. It's a mess. So, this never works, right? We all start this way and then very quickly we move away from it. So, what do we move to? We move to developing somewhere else and then having a production environment. So, somewhere else might be your local notebook or your local machine if you install MAP or XAMP or Dove Desktop or whatever, or maybe you have two separate accounts that you're hosting provider. So, there's some advantages, a lot of advantages to this, right? First of all, you know, missing semicolons will take down your site, right? You can start to do normal software development. You can use version control. You can have multiple people committing. You know, you can have, you know, daily meetings with your team and talk about features you're going to implement and have a version that's, you know, not deployed yet because you have somewhere to run it. Everyone's working on their own local copy of the database. Oh, another big problem with just hacking live on your site is you're testing on your precious live data. Everyone can have a local copy that their site talks to so they can, you know, all sort of work in parallel. But there's still problems with this approach, which is when it comes time to release, let's say you've got one or maybe two or three or four people working on their own local environment and now you want to do a release. What do you have to do, right? Okay, first of all, you get your team and you say, all right, everyone, we're going for a release now, all right? You take everyone's changes. Maybe they've been working on their own thing, on a feature for, you know, a week or two or a month or who knows how long, in an independent environment and maybe they've committed, maybe they haven't. So everyone runs, you know, Git pull or SVN update and the conflicts start happening and you have to get the conflict markers in your code and it doesn't all work and even worse than that is, you know, someone made an API one way and someone else called it a different way or the modules defined the same function name or what you get is integration errors, right? So at this point, you merge everyone's changes and you start trying to make it work and what you thought was going to be released today becomes the beginning of, you know, integration week and all kinds of frustration results and you iterate and you try to fix it and finally, okay, great, it seems to be working, you copy the code to the production servers and then your site goes down anyway because, oh, sorry, before you copy the code to production servers and now, first of all, you know, maybe you have to turn on a new module or create a new view or add some taxonomy terms or whatever and now you're going and clicking, like, on the user interface and that's a disaster and then it doesn't work anyway, right? Because you discover, oh, you know, as I was developing this feature on my notebook, I realized I needed the extra FUBAR PHP extension so I installed it but I forgot to put that on the production environment and so now my site's down. Oops. So this doesn't work that well either. It can function and a lot of people work this way but it becomes painful, and it's really around release time. So the question is, what's next? Now, what's next is this thing called continuous integration, continuous testing, continuous deployment. There's nothing really new here. If people who have been developing software for a long time you will all recognize lots of things that you do all the time. This talking about continuous integration really is about an attitude and approach to a certain style of doing things that many of us do some of or all of some of the time and it's just sort of making more rigorous a process that you're probably doing already. So the first thing is use the source code repository. How many people in this room develop against some version control system? Get SVN, CVS, RCS, right? You can't, this is step zero, right? You can't get anywhere, certainly with a team, without doing that. This doesn't matter which one you use. In the Drupal community, everyone thinks it's really cool, fine, use git. It's not that important. What's important is that you version everything. So not just the source code to your site. Also version your tests. Also version your technical documentation. If you have a doc that says, oh, we have this module, and here's how it works, it uses this data and talks to this web service and whatever it does, and you document it, which is great practice in six months, even you won't remember what you did. Put that code in your version control system so that a patch that changes the module, it also has to be a patch that changes the documentation, and now when you release it, you can look at the different versions of your documentation over time to remember how it worked three versions ago. If you use libraries, if you use any external PHP libraries, don't just install them on your server. Put them in your version control system and now you know when you deploy your source code, you're deploying your dependencies, too. There's no, oh, my goodness, you know, the prod environment had version 1.2 and the testing environment had version 1.4 for a library that, you know, you're managing yourself anyway. So, you know, in the Ruby community, they call this vendor everything. You know, they even advise putting, you know, putting binary objects in your version control system and then linking directly against them right there. And for version control, keep it simple, right? You can talk about sub-modules and sub-tree merges and re-branching and it's like there are advanced version control systems you can use. There are advanced features of Git you can use that have uses, but if you haven't already hit the problems that those solve, don't worry about it. Don't get into a conversation about it. Keep it very simple. One branch, your main branch, and make version tags. Okay, then maybe you get to... You have one guy who's working on some new feature. You create a feature branch for a little while. He works there so he can do local commits, merge it back in, delete the branch. Keep it very, very simple. Software development is hard. Version control should be as easy as possible. Sometimes I think Git's motto is if writing the code is hard, managing the code should be hard too and I don't really understand why that would be such a goal. Okay, so, integrate constantly. This is something that gets right to the heart of continuous integration. So, let's say you're implementing some feature and you know it's going to take you a month. Sometimes it takes a long time. You've got a whole bunch of stuff to write. If you go a month and then try to do a commit at the end and there's someone else on your team who's working, what's going to happen? You're going to have code conflicts. They changed something similar. So, take your big feature and figure out how to break it up into small steps so that it becomes more like a little chess game between individual pawns and pieces to get over there where your actual feature is and commit them incrementally. Every time you change something in your code that's going to require a change in production, don't leave it till release time to do that by clicking on the view to change the fields that you're displaying. Automate it. There's a variety of ways you can do that. I'll talk about it a bit later. Implement tests as you go. Sorry. And this is really important. It's very tempting for software engineers to think, either it's someone else's job to write the tests or I'll write the tests later. There's a whole theory of test-driven development where you should write the tests first. That works really well in some situations. It works not as well in other situations. But if you write the tests as you go, each little bit of effort to write the tests it just doesn't seem as big and daunting if you think, oh my god, I want to start by testing everything about my site. It's kind of like starting an exercise program. It's really hard to get started. You just write a little bit and then you write some little test for each little patch. And each little patch is a fairly small piece of code, a fairly small step towards the ultimate goal. It helps you just keep things going. Commit those changes often. You just get into small steps. You automate the test and you automate the upgrade and you implement the test. Commit it frequently so that other people on your team can always be downloading your code from the repository and running it and knowing that their code with their little change and their little test is going to work in the presence of your change with your little test. And then, of course, in order for that to work you've got to be pulling down everyone else's changes regularly. So don't go a week or a month or six months before you start trying to integrate. And, you know, what this does it just reduces that pain on release day. So I was thinking about this last night as I was writing it and I put in this slide called Sidebar because the manifesto of continuous integration, you know, it was on that previous slide which was, you know, update from the main repo daily and commit daily. They sometimes even say commit every few hours. And I thought about that. I'm like, you know, I build AquiaCloud. I'm the lead engineer for AquiaCloud and we have a team of nine people working on it. And I don't really want them committing every few hours or every day. We have a code review process. We have an engineer take a user story. We size it out. Hopefully it's, you know, a day or two or at most a week and they start working on it and they produce a patch when they're done. And then we review it. Someone reviews it. And then we kind of go around in a cycle. And then maybe there's some changes that are required and then eventually we're done and we commit it. It works just like it does on D.O. And I don't, it occurred to me as I was writing this that I'm not sure I buy into the full orthodoxy of commit every few hours. You know, I want that review process. So certainly I wouldn't want an engineer on my team to go a month without submitting a code review. So I would want them to, you know, maybe go a day or two or at most a week and submit a code review. If we have a really big story, I want them to break it up into small pieces and maybe submit code reviews for small pieces. But I also don't want to see a code review every few hours because then everyone's just going to be thrashing all the time going and looking at these small little code review pieces. And, okay, well, so we happen to use Scrum as our engineering process and so we have a planning day where we take, here's what we're going to do in the next three weeks and we sort of break it up into tasks. So that's where we figure out in theory, you know, what the small tasks are, but, you know, even if we're all thinking about it together on the first day, we don't always get it right. So if I say, if I were to tell the engineers, okay, you know, it's break it up into small tasks and commit them yourself. They may not make the right small moves to get to the right goal because on planning day we may not have discussed it properly. So what I'm saying is, you know, you don't have to do this every four hours in order to call yourself a continuous integration shop or maybe that means, you know, I'm not really, we aren't really a continuous integration shop. I think there's some middle ground, but clearly between every four hours and a month there's a lot of room in there for doing things more quickly and we have had engineers who take a story and go for a month and the problem is then they turn in their patch at the end and it's often not right by a lot, right? And then they've put a lot of time into it and we have to go totally refactor it and so I feel like, oh, if I'd seen that, you know, after every few days, then at least we wouldn't have gotten so far off track. So, you know, somewhere between four hours and a month you figure out what's your ideal cycle time and I'm not going to tell you that it's got to be every single day. Okay, talked a little bit about automated testing. I'm now going to talk a little more. How many people in here automate tests? Write automated tests for their site. A good number, fewer than the people who use version control. Okay, well, you can't go anywhere without version control and we all know you can have a site work without writing automated tests, so that makes sense. A good way to think about this is, you know, ideally you want to test everything. With the Acquia Cloud code base, we have a saying which is that if it isn't tested, it doesn't work and that's almost universally true, we find and we try to test everything. So with a site, you want to make sure that you have all these different models, you can have unit tests for your individual modules. Do they, you know, read and write things from the database correctly? You want to have browser-based tests. Does your homepage look right? Does your view sort correctly? There's all these different levels that you can test. You can do performance testing. There are plenty of services out there that you can point them at a site and they'll, you know, spin up 25 users concurrently and go through these page loads and test them all and make sure that it works. You know, so you can see over time how the performance is going. You can do upgrade testing, so I said it's really important, especially for continuous integration where you want releases to be really easy. We'll talk more about releasing in a minute. That your upgrade process is automated and not I have to go check on, you know, checkboxes in the Drupal UI. So if you have automation to do the upgrade, you have to test that, which means you have to take a database from production, bring it into your test environment and run the upgrade process and then make sure you get the right thing at the end. That's probably the most important thing to test if you're going to test anything. And I sort of said this before, but just start by testing the new stuff. If you say that you need to wait to start testing until you can go back and actually test all the different parts of your sites and all the custom modules that you've written, you never get started. So just start small. And by the way, so, you know, you want to test to make sure you're not white-screening of death. You can use Cucumber and Capubara and Selenium and there's all these really great tools out there. You can also write a one-line shell script that calls curl and greps the output to make sure it has your site name or your slogan in it. And that's something, right? And you do that one little thing. And then, you know, you add some other feature and you put another curl command, you know, and it starts really small and really easy. Now, eventually you'll realize, oh, my God, my test script has like 37 calls to curl and they're all custom hacks and you decide to refactor it and you put a little effort and you go to a better technology, but at least you've got a base to go from. And again, the upgrade testing is really critical. Um, run the tests frequently. Sometimes every commit, ideally every commit, if you have tests and they break and you allow them to keep breaking and you keep committing code, you're going to destroy the morale of your team, you're going to decide not to trust your tests, then it's a short step to decide it's not worth writing the tests and then pretty soon, you don't have tests anymore. Um, so it's really important that they do this. Now, confession. We have automated tests for AquiaCloud. They almost never fully pass. It's an extremely difficult thing sometimes to get automated tests to pass, but the difficulty of getting them to pass depends on the nature of the application. For a website where you're pretty much write how you have code that's talking to a local database, there's not too many excuses. That really actually ought to work. We have tests that fail because we're launching servers in EC2 and updating DNS and running SSHs across the network to 12 different machines and making sure they all work out correctly and there's all kinds of, you know, cosmic rays that can interfere there. When you're primarily dealing with a Drupal site, your tests are really ought to pass and it's worth calling a halt to future development until they do, because otherwise, you know, it's like your tests are telling you something is broken. Don't proceed, because that's what they're for. We actually, we have another saying, which is that our system tests at AquiaCloud, they're so incredibly fragile, they break if we make the slightest mistake. We hate that, except, of course, they're doing exactly what we wrote them to do, which is to tell us we just made a mistake. Okay. Oh, and of course, whenever the tests fail, tell the whole team so that everyone knows it, right? And everyone knows, because sometimes, you know, it does happen, you're the developer, you write your tests, they work, you do the commit, you do an update from the repo, you run them again, they still work, but then someone else runs them and they break. That can happen, right? Sometimes there are bugs that don't manifest every single time, or they depend on the time of day, or, you know, God only knows what. So when anyone encounters a bug, even though it might have worked when it was committed, make sure everyone knows it becomes a priority to fix it right then. So if you have an automated system running your tests, of course, it can send email out to everyone, if not, you know, just have a manual process where someone raises a red flag and says, we have a problem. And I don't know exactly why this is on this slide. I'm not sure it should be. But deploy frequently, so you've got your site, you've got your changes, each new feature, you know, you've got a little test for and a little documentation for, and it's committed and it's working. So you have to deploy it to a QA area so that not just the automated tests run, but now, you know, your manager, your CEO, your client, someone, some stakeholder can go play with the new thing in development, right? Before it's ready to release, but all the features that have been added so far work, right, because you've tested them, and they're however far along they are towards their ultimate goal, that much they should work. So you can play with it, your designers can play with it and be like, oh, you know, you've got the placement wrong, the font isn't coming out right, the usability isn't quite the way we designed it. It's really helpful to have a live environment after you know all your tests work. So, test in a clone of production, and this is where the conversation gets really interesting. It doesn't do you a lot of good to run your tests somewhere that's different from your production environment, right? I've sort of referred to this earlier. Running your tests on a developer notebook is better than not running them at all. But it's pretty close to useless because there's a million variations in the way someone's development environment is set up, which version of PHP they have, which extensions they have, are they running it, you know, some people run their tests under the Apache user, and some run it as their own user, and like, who knows what can go wrong? Drupal is very, very dependent on the details of its environment. If the wrong extension is installed, your site's going down. You need to run all your tests in as close a clone as possible of your production environment. You also need to run them in the presence of a production database. You don't need to necessarily update your test server with a production database every day. But if it's not automated and you don't refresh it every day, pretty soon what you're going to find is that you're using your database that's a year old because, you know, it's not automated and it seems to be working and you're developing code, you're running your tests, everything's great, and then a year passes because it wasn't a priority to fix it and then something blows up because finally you wrote some code that didn't actually work in what was in production because your database is a year out of date. When you do work on a production database, of course, it's very important that there's probably some critical data in there you don't want, like, take out your customer email addresses so you don't accidentally spam them as you're developing your newsletter functionality to take out the credit card data that you shouldn't have credit card data in your database, but, you know, scrub stuff. It's pretty easy to write little SQL statements that truncate the tables that, you know, or zero out the columns that have data that shouldn't be there. And then this is the one that I actually think most people don't do, which is if you have a production environment, you have a testing environment, they're clones of each other, terrific, and then your system administrator gets woken up at 3 a.m. some night because the site went down because of, you know, a huge surge in traffic from the other side of the globe, and they realize that, oh, oh, goodness, you know, we didn't configure, you know, Apaches, whatever feature correctly, so they fix it. Performance problem solved, site is back up, testing server not changed. So now you're running your tests again in an environment that's not a clone. Any kind of manual tweak or you update the package on the, you know, on the production server because, you know, your sys admin got an alert from, you know, that there was an update and they deployed it and all of a sudden your production environment just became not synchronized with your test environment, even though, you know, three months ago when you set them up, they were perfect. The only way to actually have a clone of your production environment is to automate the configuration of your production environment and then use that same automation to build your test environment. And this gets to stop, to a level of of system administration automation that my bet is that most people don't do because it's a pain in the neck, right? Every time some, you know, there's a new security package, okay, you can install it, but then when the machine blows up and you have to reinstall from, you know, from initial media or you start from scratch, did you document exactly what you did? Do you know you documented everything that you did? Are you sure you're going to run that in your test environment again? It's kind of hard to keep up. So, we'll talk about that a little later as things you can do to solve that. Okay, and then automate deployment. So, you've got your code, you've got tests as you've been, you know, you're writing little bits of test as you write your code, you've got a test environment, it's a good clone and now it's time to deploy to production. There's a sequence of things you want to do every time you deploy to production, but you don't want to have to do them by hand every time or you're going to forget. So, push buttons should, deployment should be, you know, push button. There's some great blog posts out there, you know, Etsy wrote a great blog post on their Deployinator, how they've got it down to a web app with one button. Ideally, you push the deploy button and you're so confident because your tests have been passing that you push it and then you get up and go get a coffee and, you know, go to lunch or whatever and relax. So, the set of things you frequently want to do, I mean there's many, but at least you want to put a tag in your source code system pointing to the code that you deployed at that exact time, so that you know next week, that's the code I was running last week, or more likely, you know in half an hour when the site blows up anyway, despite all the things you did to prevent it, you know the version that you were running before the release so that it's very easy to go hit the button again and deploy the previous tag that you marked. You want to make a snapshot of your database just before the release so again, you can roll back. It's really painful when you have to do that because on a busy site, you know even minutes after you deploy a new release, right, there's still user content coming in, you really want to avoid having to roll back, but in disaster sometimes you have to and if you don't make that snapshot automatically then you're sort of screwed. Then of course you want to deploy the codes to your server the code to all your servers, maybe multiple maybe single, you'd like that to happen sort of as quickly and synchronously as possible, then there's if there's any updates which you've automated then you need to run them hopefully you have automated them and of course after you run the updates if you need to back the code out depending on how you wrote your code sometimes that's where you need to restore the old database, sometimes your new code won't work with your old database schema so in that case you have to roll back your database, unless you can fix the bug really quickly and roll forward and that last part upgrading changes because you can't always roll back is why if you haven't automated the upgrade path you're not done, if you haven't tested it against a production database copied into staging and run your upgrade and visited the site your feature isn't finished. Alright, so that's a lot of stuff right, I don't know why I just went back how do you actually get there I've just evangelized for half an hour how do you get there the most important thing to understand is that it's not actually about technology this is one of the big names in continuous integration said this thing it's an attitude, it's not a tool it's not about using Jenkins and Puppet and Chef and Capistrano and Deployinator and all these other things those tools are all great you'll probably end up using some of them but you don't have to you can actually there's plug posts where they talk about there's one that talks about using a rubber chicken as part of your continuous integration process go search for it, it's very entertaining you can do this yourself mostly Drush has commands that automate almost all of these steps obviously the Drupal developers have encountered all of these things before so you can deploy your code with Drush it has a handy dandy arsync tag so you make the tag use git tag or svn copy or whatever system you're using and then drush arsync and it copies your code for you great, if you have multiple web servers you need a slightly more complicated script than that because of course you've got to update two or three web servers at a time and you have to address the problem if I'm updating two or three web servers at a time am I updating them all simultaneously Einstein had a thing or two to say about whether it's possible to do anything simultaneously at a distance in this universe so no, you're not so you try to make it as simultaneous as possible and anyway your script at least is going to have more lines in it because you've got to copy to server one, server two, and server three but, you know it's not that hard of a shell script you need to copy your files again, drush to the rescue it's got this great arsync command that you can say copy the files directory and it will, great database again, drush has a great command for that these things work fine, we use them everyone uses them, they're good tools you know, the people who work on drush the drush deployment code is our top notch the sql sync that copies databases even has a sanitized command built in that strips all your customer email addresses for you almost like they thought of that if there's more sanitation you need to do maybe you want to scrap, scrape out your orders database or whatever then you can just have another little sql command that you write and pipe it into your database and drush has a command for that too drush sqlcli, one of my favorites um, running your tests guess what drush has a command for that too it runs the Drupal simple tests right from drush, and if you did write some other script where you ran curl and piped, you know, used grep to look for your site logo or slogan, then you can just ssh to your server and run that too you know, you put your test script right in your code base so none of this is hard, right, we can all everyone writes bash script and it's not that big a deal um, and this works it doesn't even have to be automated right, you can say, alright it's release day, and have a little piece of paper and go through these steps if you're for a small team it's like, it's fine it's not about needing a huge initial investment certainly you can start here and then start automating as you go however if you're doing this yourself, you have that problem of your testing environment that I talked about and now all of a sudden you're administering servers and there's a lot to do here so you're using some version control system you need a server for that, someone's got to install it maybe you use github, that's terrific you still have to integrate with them you end up writing a lot of little scripts if you start using Jenkins or some other continuous integration server each little script becomes a Jenkins job you want to deploy automatically on commit remember I said always have a version of your site in a QA instance where people can play with it well that means that every time there's a commit you'd like to update that and okay, easy enough, post commit hook and git run your deploy script that's a little script to write if you want to copy your database from production down to development not that hard to do, there's a drush command just another little script to write if you want to run your tests all these little things that you want to do they become a job that you write in Jenkins not that big a deal you have to manage your own operating system maybe you're at a hosting provider that does that for you, that could be great the server build as I described you have Apache and MySQL and memcache whatever that has to be configured that's the next column you have to do security updates when they become available and you have to do these on your testing environment too you can't think to yourself our testing environment is behind a firewall no one can log into it it's not crucial for security you might be right about that it may not be crucial for security but if the environment is different you don't know that your site is going to work the same way maybe you upgrade to the next version of PHP because there's a critical fix to the accident in your site and you didn't know it and now it breaks in production God forbid you have multiple sites or multiple V-hosts you have to manage all of them put your domain names in the right place if you're configuring SSL if you have custom settings in PHP, I and I and of course you will have multiple web V-hosts because you have a production environment and a testing environment and a development environment so someone's got to go set all those things up you need multiple databases you don't need a separate site talking to your production database or vice versa so you have to separate MySQL instances even if you're on the same server and somewhere you've got to manage your credentials so you have all your settings files some people put them in the repo some people don't but you've got to get them to the right place on the right server without making a mistake so that every page load talks to the right database for the environment it's in depending on the different levels you've got to manage that too unless you connect to a hosted service that does that which could be a great idea but now you've got to work with that hosted service all these configure Varnish make sure Tomcat is installed to run Solar are you using Memcache which version do you have oh if you're using database replication because you want read slaves or you want an HA fail over database you've got to keep that up and running this ends up being a full-time job then God forbid you become successful you get a lot of traffic you don't want your best day to become your worst day because your site goes down when you got linked from the front page of wherever and automating backups how many people have completely reliable automated backup system for their site that they know works because they've restored from it fewer people then are which was the second question I asked that automate tests which was fewer people so it's a lot of work you don't get around to it because you've got other things you're worrying about but it's really important to do because otherwise at the worst possible time it's going to bite you in the butt alright and then of course 3am Saturday night your servers might blow up and someone's got to wake up and fix it which means someone has to know that it went down which means you need a monitoring system so now you're configuring Nagios to go with it you can do all of this and there are plenty of sessions here at DrupalCon about doing all of this and teach you how and if that's how you want to spend your time and there are some people who do, great you have another option which is pay someone else to do it so obviously I build Acquia Cloud, Pantheon exists there are other services out there that take everything from that last slide all of the server administration and also the automated deployment and running from version control and just hand it to you I see a lot of you know we have a we have a cancellation process for people who try out our service and when they're canceling there's check boxes that say like oh why are you canceling you know I didn't work well I thought it was too expensive I want to do whatever the reasons are and a lot of people check it's too expensive and I'm thinking I need to change the form so that when they check it's too expensive I open up another little form that says how many hours per month do you estimate you're going to spend on all that stuff on the previous slide because the answer is quite a few or you're not going to do it in which case you're exposing yourself to a lot of risk and like an extra hundred bucks a month how many you know like what's your time worth your job is to build a killer website and not to become a system administrator all the time if your job is to become a system administrator actually I'd like to hire you please come talk to me after the session so you know these exist there are others we sort of compete but really you know I have a great relationship with the guys at Pantheon and we're very consistent on the message that really our competition are all the people who are who are struggling to to make their Drupal sites just work consistently and reliably and growing the base of people who are working efficiently so you know we compete for individual clients but really we both just want to increase the number of people who don't have these problems all the time and I put this slide in because someone always asks so I'm just going to address it right up front okay you have a dev environment a testing environment and a production environment and we've talked about moving your code and bringing your databases back for testing but then someone says how do I manage the Drupal configuration problem where I want to change something in development I'd like to automate pushing it to production you know where's the checkbox for that the answer is there really isn't one the Drupal UI that entire administrative interface that we've built it's awesome if you're just learning or you're prototyping or you're just experimenting and plugging modules together and seeing how they all work but behind every one of those checkboxes there's either if the module is well designed an API or there's a database table and so when you say you know okay I want to change my view to do whatever or I want to configure Drupal commerce to have a new product there's some way you can automate that you can either write an update function Drupal's normal module schema update where and in those update functions you can do anything you can create nodes you can install new views you can change your taxonomy and access control and roles and all that stuff you have to write the code the advantage of doing it is you know that that code is going to work assuming you test it obviously there's the features module development environment export the view committed to code again it's code the thing about code is it's easy to version it's easy to automate it's easy to test checkboxes not so much I suppose you could actually automate a browser based upgrade tool which went and checked the boxes for you but really you don't want to and then having to do this we have this great UI and here I'm telling you you can't use it and the Drupal project is very aware of this so in Drupal 8 they've got this configuration management initiative which is actually an attempt to automate I checked the boxes and Drupal hands me the code configuration changes and there was a demo with that at the keynote I assume you all saw it so this problem may be going away for people who use Drupal 8 in a year or whenever it's released in the meantime update functions features modules there's probably a couple other techniques out there that let you manifest the configuration in code and that's pretty much what you have to do that's for configuration then the next question is how do I stage my content well this talk actually really isn't about that but again someone always asks so every time I've gotten this question I've always said you know there's a variety of techniques in the Drupal ecosystem for how to migrate nodes content from staging to production those all work on Aquia Cloud Pantheon but we don't offer anything excuse me specific to help with that this time however all of you when you registered for Drupal Con you got a copy of the Drupal Watchdog magazine and in that magazine is an article actually by another Aquia employee named Michael Cooper it's called enterprise Drupal application scaling which I would never read that article if it were me but what he talks about is how within the Aquia network where we have Networked at Aquia.com and Insighted at Aquia.com and library and forums and then a central customer database and all of our subscription nodes and users are synchronized among all of them he talks about how we do that so when we go to one of our websites and someone goes to node edit and changes a field and hits save it propagates to all of our other Drupal sites which are all independent sites they're not sharing their node tables or anything like that and so you can read that and it talks about it the short form is UUIDs and web services he talks about how we describe that there's also a variety of modules that can help the migrate module, node import, export, deploy there's probably some I'm not forgetting they will all work in a continuous integration environment they're really sort of orthogonal to that just thought I'd head off the initial question okay I always forget to show this slide so I'm going to show it now so I don't forget I'm going to show it again later they want me to ask you to rate the session please do I'd love to know love to get your feedback and questions that was a lot of brain dumping yeah the question is running simple tests takes a really long time so I actually based this presentation on a presentation that a friend and recently former colleague of mine named Mike Booth wrote and he had a slide that said how long can your tests take if you measure it in seconds that's awesome if it's measured in minutes that's fine if it's hours you have a problem and if it's longer than that it's a disaster right and so I totally get that Drupal simple tests can be very slow it takes a lot of effort to replace them with something else for that reason and I think I'm trying to remember the name Moshe Weitzman has a project is it called PHP unit it's based on maybe PHP unit the reason is so at Drupal simple test every time you start a new test case it completely reinitializes the database and that setup process just takes a while so I think there is an effort to fix that I don't remember the name off hand of the project but yeah it is an issue I'm sorry say again quick simple test that's another project okay so it is a sandbox project it is a problem so I'll tell you that our automated tests for Aquia Cloud they used to take 6 hours to run which is brutal because you come in the morning you make a change you start the test you don't even have an answer before you go home from the day we recently changed that we're aiming for 30 minutes actually in a couple weeks we're actually hoping to make a big step towards that so one thing you can do is run them in parallel so instead of having one test environment have two or three or four and when you kick off your tests just run class A here and class B here and class C there and it's at least something it takes a little more automation not that much more it does matter the thing with when they take too long what happens is you're not willing to run the tests before you make a commit so you can run them overnight and that's still good but then what happens is you end up committing broken code maybe you run the one test that was the specific functionality you wrote and then overnight you discover it broke and if you have that then you just have to have a really disciplined team which every day reviews the failures and figures out what's wrong and goes and fixes it and we do that and yeah it's painful life sucks other questions is version everything what do you use for database versioning so what would you like to version in your database so anything you can't put in code what I think you're asking is how do I version my configuration so I can back it out right I don't have a great answer for you what I can tell you as an offhand comment I know that I've never written Ruby on Rails code but I know Ruby on Rails Drupal has update functions and Rails has migrations I'm pretty sure that a migration is more or less what we think of as an update function you know it's database statements that you run to get you from one version to another and I'm pretty sure that the Rails migrations feature has a undo you know a function you're supposed to write to do undoing honestly I've never asked a Rails developer this but I'd love to know how many of them actually write that and I'd be surprised if there are many I mean I wouldn't you know it would it wouldn't be shocking if someone giving a talk much like mine in a Rails environment said yes it's very important that you write your undo migration just like I just said it's very important that you write your tests and I couldn't disagree with them I don't have a good answer for how to version your database you know automate your upgrades and test them frequently like many times before you do the deploy and that's having that test environment that's set up that on every commit or every night always runs your update with a fresh scrubbed copy of your production database gives you a lot of confidence that the update's going to work and other than that Drupal actually makes it kind of hard you know knowing knowing what we know now if I went back to you know 2001 and talked to Dries in his dorm room I would tell him maybe we should have this administration interface I'm not 100% sure but we absolutely should never have conflated configuration and content like I was just thinking yesterday what if Drupal actually had two completely separate databases one for all the content and one for all the check boxes because then you know your content database would get really really big as people add comments and votes and all that stuff but your config database would be pretty small and maybe you could actually just my SQL dump it and save it to a file and commit it to your repo and that you can't do that because you know we've conflated we've conflated the data and it's even hard you know if you have a site and you've got users submitting content and you have an about me page or about page the about page arguably that's configuration right even though in the node table the next row is you know some user submitted content that's content there's no good way to solve it it's part of Drupal's design unfortunately that design was one of its enormous strengths that it made it really easy to configure dynamically and I mean with field API you can change your fricking schema from the UI which is awesome until you try to version it so test your upgrade processes religiously that's why I said it's probably the most important thing to test again the universe is an annoying place the question is how can I replicate my lamp stack with all the various server configs and all that so my return question is how did you set it up in the first place so right so you installed the packages if you install the packages by hand the only way to replicate it that I know of is to install them by hand I mean so there are alternatives if your if your production environment is a virtual machine let's suppose it's in EC2 it's an Amazon EC2 right you can make a snapshot of your machine you can make you know either a bundle or if it's boot from EBS you can just create an actual snapshot and you just boot that machine again and that's a clone and that will actually work the and I think a lot of people do it that way and now you know it's an exact replica the problem with that is that now you know let's say you do that it works great and then in a week you're like oh I want to install a new package you install it in production you make a snapshot now you have another a new clone and it's current but now what you've done is you've started iterating your machine environment without automating building it from scratch and when you lose that AMI when it blows up for some reason or you realize you've made a mistake you're not going to remember the steps you went through to get there so that leads to tools like puppet or chef for automating the configuration of your machine from scratch and then what happens then is now you're building a Drupal site and you're also building a machine configuration application which is your puppet code or your chef code and you can do that and now you're running continuous integration on your Drupal site and you're also running continuous integration on your machine configuration code so this is now this machine configuration layer that's a software product that you're writing and you need to version it and you need to test it and you need a clone of that environment to test on it becomes a pretty big job and that's why companies like Pantheon and Aquia have implemented a server infrastructure because we do all that work so you don't have to other than automating it with the tools we used I don't have an answer for you but you can do it and there was a session I think yesterday that talked about doing that so that's available you know the video for all these are available was that your question or not how can you make sure they're exactly the same sorry yeah so I mean what we do is we always so on Aquia Cloud we always start from a base machine image which is we happen to use Ubuntu so we start from the official canonical Ubuntu distribution so we know that's always the same we're booting the same AMI and then everything that we need to change we put in puppet and then we always we have a test environment we're constantly building new machines from scratch and testing them and these tests that I described that take six hours to run we deploy to production we take the latest version of our code we basically do continuous integration at the machine level and we upgrade all of our servers and we test that upgrade process pretty carefully so it's a lot of work we've got nine engineers plus an operations team of about ten people running this environment which is why I can't imagine why anyone building a website would want to do with themselves but some do right so the question is about having you have a dedicated QA team and how does that work with your development when do you do the QA when do you do the code reviews so is your QA team are they writing tests or are they like poking at the product following a script so your question is how do you like when do they begin so in my experience and this is now just personal software engineering opinion here I and a lot of other software engineers believe that the idea of having a separate QA team that writes the tests that writes the automated tests which is why I ask that question is generally a mistake you want your engineers to write the tests for a bunch of reasons one is they're the one who knows the code best they're and are best able to automate it what we found when we started knock we a cloud we had a guy who was like the system test implementer he was writing the test and he couldn't keep up we'd be writing all kinds of crazy stuff he wouldn't even know what we were doing he was always behind it was a disaster it was horrible for him and it didn't work very well for us eventually we realized writing these tests it's really the engineer's job now that being said you still have after all the automation you still have a person who goes and pokes at it the way we do it when we consider the software done we test it as best we can automated we do some poking at it we release it if our QA team or our usability team finds problems they're bugs and they get entered back into the backlog and they get fixed on the next cycle I don't know does that answer your question that's how we do it we just think of it as you know so they're they're not we don't think of them as the QA team that is that's poking at the product we think of them more as you know user experience testers and then they just advise us on things to fix we do use so to be fair the product I build is infrastructure we also you know Acquia also builds you know we run Drupal Gardens and we have the Acquia network we have much more you know we have website development going on too I don't happen to do it and for that we do have we don't have the QA in-house but we use a company we use or used to use a company called U-Test which is a company where you give them a script and they you know they outsourced having people poke at your site and I honestly don't know how the teams that work with that that kind of QA integrates it into their site development go by the Acquia booth and ask one guy to talk to is Michael Cooper he's the head of the network Acquia network team there's some other people there that can probably answer that okay anything else yes so what I heard was are you asking about so what's the best way to deploy from development to staging to production but then you went on are you you're saying are you asking about code are you asking about configuration changes I couldn't hear that part can you repeat that the last part so what's the best way to deploy your code so this there's a variety of ways most of them are pretty simple it's a bunch of files you have to you have to get it there so an easy way is you've got your development server you've got some repository server somewhere and on your test server you can do you know a git clone or a or an SVN checkout or whatever option number one you know do the checkout right into your dock route that actually turns out to be not such a great idea because one of the things that Drupal needs is the the files directory the the place where user uploaded images and PDFs and all that go and if you do a you know a git clone right into your dock route directory that files directory has to be there or Drupal is going to get really confused but then when but then you you've got a a checkout of your repo that has stuff in it that's not in the repo particularly with SVN if you then try to do an SVN update all kinds of weird things can happen I think gets better about it but it just generally feels pretty dangerous to have your actual files directory right there so what we actually do in AcquiaCloud is we do the clone over here on the side and then we are syncing it into place with a command that says ignore the files directory and delete everything else and then it turns out actually our files directory is a symbolic link off to the actual file storage area since we're running in a multi-web server environment we have a network file system and whatever that's not really germane the and if you're running one server then some variation of that do a clone do an rsync it works fine if you're running in multiple servers then you've got the problem of well you'd like the code to update at least approximately at the same time as close to the same time as possible so there's a couple strategies for that we run our updates in parallel so when a code commit happens we have a job server and if we've got three web nodes and a commit happens we fire three jobs all at the same time and so they all end up SSHing into each web server and doing the clone and doing the rsync you can take that even farther with a technique called the swinging symlink where you've got your old code repo directory and a symlink pointing to it and then you copy your new one into place and then just flip the symlink which is the idea is you'd like all your code to update at once and that the swinging symlink achieves that goal on one server so if you only have one server and you do the symlink technique your code will update atomically all at once the thing is if you've got multiple web servers you can't swing symlinks on multiple web servers at exactly the same time so there's going to be a delay this one's going to swing and that one's going to swing and then the third one's going to swing and so ultimately you can still have page requests hitting your site hitting your database that are running on different versions of the code so the only way to be really sure this may not be what you asked the only way to be really sure 100% sure that all of your code updates atomically that there's a point in time where page requests are hitting the old code and then there's a big red line and then all page requests are using the new code the only way to be sure is to take your site down put it in offline mode do the update and put it back on so people don't want to do that most of the time it's not necessary most of your code updates first of all many of them just work they don't change the schema sometimes if they do change the schema maybe one page request or two will be a little funky and if it is you can write your code to be paranoid and say oh if my record has the new property and it works a new way and if it doesn't then work the old way and you leave that code in just for the duration of your release then you can take it out later so I don't know if I've really answered your question you can come ask me later if I didn't how are we doing for time? we are out of time so thank you very much please rate the session you can just go to the session node on the code of the Munich website it's number 409 if that's easy to remember thanks a lot