 I'm Mark Sonnebaum, a technical consultant at Acquia, a Drush Co-Maintainer, and this is Moe Schweitzman, also at Acquia, and the Drush Maintainer. Today we're going to talk about a framework recently built called Drush Deploy that should help you to deploy Drupal apps. The first, not a lot of people, especially in the Drupal community, use deployment tools. Many that exist already, but basically it deploys your code. It also does some other stuff, but really the focus is it deploys your code. There's a lot of different things that go along with deployments, and I'll get to how you do those later, but the important thing is this gets your code on the server. Three popular frameworks, Capistrano, Capistrano is definitely probably the most popular, Fabric in the Python community, lots of Django, people use that. Pantheon actually uses that. Glad the deployer is another Ruby one. Is anybody here using any of these three? Okay, some, very few. All Capistrano, anybody not Capistrano? So more importantly, what doesn't a deployment framework do? It doesn't deploy content. If you were here a couple of sessions ago for the deploy module session, that deploys content, so that's where you'd want to look for that. It doesn't install packages or manage comps. There's a lot of overlap here with, say, configuration management tools, like Puppet, Chef, Deconfig. Some of them will allow you to also deploy applications. Some people do that. It's a little more overhead than a lot of people need for deploying applications. I personally like to keep those two things separate. It doesn't manage submodules. I say that because there's also a little bit of overlap here with Dog. There's an asterisk on that one, but it doesn't do anything automatically to say any of your submodules for you. So yeah, I say sort of because you can kind of do all of these things in the framework if you want, but it's not really the main focus. So why use a deployment tool? Using the techniques here, you can get near instant no downtime deploys. I put a big asterisk on that, and especially as a near instant because it's impossible to do instant, right? But the idea is you do a new deploy, and you switch over from the old one to the new one as the site's live. Some sites can do that. Not all can't. You might be in a situation where you have some tasks that need to run that are going to take a while, and you might need to go into maintenance mode. I do that as I need to, but if you don't need to do that, like if you know you're pushing a deploy out that doesn't have, you don't have to run update DB, or you don't have to revert features, anything, most of my deploys happen live with no maintenance mode. Now rollback is another really important feature, and I should say first, a lot of these concepts are in all the other deployment tools I talked about before, but having the ability to instantly rollback a deployment if anything fails is very valuable. And version control is not a deployment tool. So that may seem like a silly statement, but right now, if you say clone your Git repository into your doc root, and then your deployment is Git pull, that's not, I wouldn't suggest that as a deployment strategy. It's not made for deployment, so things can happen because it's a version control system. It can do its job and say show you merge conflicts, and then you can end up with a broken site. There are probably semastrics around that, but in general, I would just say it's not really a good idea, and you shouldn't do it. Now a little bit about Capistrano. This is actually what I use most of the time. It's awesome. I don't really have anything bad to say about it. There are some Ruby on Rails assumptions. It was made specifically for Rails by one of the guys at 37 Signals. Many years ago. And so with using it with a non-rails application, you just have to override some of their tasks so that it doesn't say, try to, one of the tasks is restart. So after you deploy your code, you restart the web server, but that isn't really relevant for us, right? So we have to then override those so they don't do anything, which is a little silly. And it's a hard system to get going for non-rails apps sometimes because those things aren't obvious. There is a gem called RailsListDeploy that does that stuff for you, but it still doesn't exactly make it that much better. It's a DrushDeploy. It's a deployment framework built on Drush as a Drush extension. When I say it's heavily influenced by Capistrano, I definitely mean it. Basically all of the functionality that I thought was relevant from Capistrano, I built into this using nearly identical methods because like I said, I don't really have a problem with Capistrano. It's really nice. And I think the default merge strategy that it uses is really smart and works for the vast majority of sites out there. You might ask if Capistrano is so great, why should we do this? The number of hands that went up when I asked about deployment tools were pretty low. And so although Capistrano has been out there forever and it's stable and it works, none of you are using it. So I think there's something there. And we have a lot of tools. Nowadays if you have to know Apache, Varnish, Memcache, Solar, the list sort of goes on and on and on. Adding one more like Capistrano is sort of the breaking point for a lot of people. So we already use Drush. And also a lot of the concepts within Capistrano we already have in Drush. So we get a lot of things for free. We just have to build the deployment part on top of it. Now so what a deployment framework does is actually really simple. And right now, I mean, it's not that hard to deploy a Drupal site, but everyone pretty much does whatever they do from scratch. They come up with their own batch scripts. And it's not something that you should have to stumble over and then find all the problems with yourself. There are definitely a couple different ways to deploy a site that work for many, many people. And so we should have a way to simply do that. Now a little on how it's built, it's PHP 5.3 only. Is anybody here still on 5.2 and unable to move? All right, that's a surprisingly small number if everyone's being honest. Yeah, it's 5.3 only. We aren't really part of Drupal, so we don't really have to adhere to the PHP version we're using. And we used a lot of kind of need 5.3 features that helped along the way. Because converting a lot of the what was in Coptrono and Ruby to PHP was hard enough. And some of the things in 5.3 made it a little bit easier. Right now, it's only Git. I did that in the beginning just to make things simpler, but in hindsight, that was probably not that smart. Because I'm sure many of you are still deploying from non-Git sources. So that's something that's not in there now, but the source control will be swappable soon. And I would love help. Once I make it swappable, I would just have to have some help with, say, subversion backends if anyone is still doing that. And so site alias groups. Who uses site aliases right now? Okay, great. How many of you know about site alias groups? Many or, yeah, fewer? So this is another one of those features in Drush that no one uses that's really cool. You can actually have multiple aliases in the same file. And then when you specify that alias, you actually are calling all of them at once. And that's sort of an essential feature here, because when we're deploying, we could do deploying across multiple servers. It also makes heavy use of Drush options. So if you haven't used it yet, there's this really, really handy file called DrushRC. If you haven't copied the example in, it's in home.drush, drushrc.php. Deploy uses its own. So you need to make a deploy.drushrc.php, and you put all of your deployment configuration in there. Or you can actually put some of it in your site aliases file in the command-specific part, if you've used that. Now, this is the basic idea of how this deployment strategy works. Here's the, you set where it actually deploys to on your server, and right here it's saying it deploys to deploy, right? It sets up releases and shared. Shared is where you put things like, you put your files directory there, you can put settings.php. That's where it's gonna, that's where you have resources that are gonna apply to every release that don't necessarily change. Releases, you can see, has timestamped directories of every release, right? So when you deploy, it creates a new copy of your site in a timestamped directory. And you can see current is pointed to the last one. Now, the idea is you point your Apache v-host or whatever v-host to current all the time. So then when we deploy, if everything goes okay, we change that sim link to point to the next one, so it flips over automatically and you don't have to do anything. And then it's still there if you need to actually revert back to the old one. So, in the interest of showing things actually happen, let's demo this. So just to give you an example, if you have not used Drush aliases before, this is all the information I need in my alias. So just pay attention to the first one. Basically, I'm just saying there's two here. This is where Drupal is deployed to. This is the user to SSHN as, and that's the host. And this is identical, I just changed the host. So I'm deploying the two EC2 instances right now. So first, I would do drush deploy setup. And then you tell it at, is the name of that aliases file? You can see right there. And that references both at once. If I wanted to do one, I would do a C app one, but I wanna do both. And so you can see the commands that ran, it just set up those directories that I showed you before. And there's also a drush check, or I mean deploy check. And that'll go through the, go through the current dependencies to make sure that everything is in place so that you can deploy. So it checked that those directories existed, it checked that they were writable, and it checked that git exists. The git check was done by the strategy, the deployment strategy, which is a class that is swappable. The one I'm using uses git, and so it added the dependency to check for that. So whatever strategy you use, you can add your own dependencies to check to make sure that everything will work. And let's do a deploy deploy. So right now it should deploy here, and there's obviously nothing there. And hopefully there will be a website here now. Yay. Okay. And so now, actually, I mean just stage back into that machine. Okay, so this is what I've shown you before. Current is now pointing to that, and there's releases, and there's one thing in releases. There's actually nothing in shared. But let's go ahead and do it one more time so we have another one. So say something went wrong with that one, and you want to actually go back to the other one. You can do deploy rollback, and now current is pointing to the previous deploy. So that you would use if something happened after, I actually explained the, you can have tasks that run before or after different events in the deploy. One of the events is the sim link. When you tell something to run after sim link, that means run it after the deploy is successful. So at that point, the flush deploy cannot rollback for you. It wouldn't detect an error because you've already actually deployed it out there. And so if you say a ran update DB or something after that that failed, that's when you probably just want to manually rollback. But if anything happened before sim link, and if any of that failed, it would actually automatically roll that back and never finish the deploy, which is kind of a nice feature. Oh, and also, after a while you can end up with a lot of these releases in here. I'll show the configuration file real quick. One of the configs here is keep releases. And so I'll change that to one. And then after a while you can do clean up. And that'll basically get rid of however many releases are in there that, yeah, no old releases to clean up because we already rolled that back. It'll get rid of however many and to where you only have whatever number you specified there at a current at a given time. It's unlikely you'll want to have all of them there forever. And if for the few Capistrano users in here, this probably looks very, very familiar. All of those commands are directly from Capistrano. So a little bit about how this works is a Drush extension. How many people here have written custom Drush commands? Not necessarily like contrib, but just for yourself. Okay, great. So part of doing this was, it was an idea I've had for a while and I wanted to see how we could do this in Drush. And but I also need, there was a couple of things that we needed for it that were gonna potentially require changes to Drush Core, that things that we could send upstream that would make writing extensions easier. So it was a bit of an experiment in how to write this kind of an extension and what we can do to Drush to make it better. So one example is the idea of an alias as an argument. So for those of you who have used aliases, the first way there is how you'd normally do it. It's Drush, alias, then a command. What that does is it SSH's into that machine and then calls Drush to see all. There's no way with how those aliases work to just send it a command. And so in my case, most of the time I'm making directories, I'm checking if they're writable, I'm doing git clones. I don't wanna have to use Drush to do those things on the other end, that doesn't really make sense. Really all I wanna do is get the information from the site alias so that I know how to SSH into that machine, then do whatever I need to do. So the second example is how that works. The only example of this in core is SQL sync, right? Yeah, and so if you've ever used SQL sync, it's like Drush SQL sync, alias, alias. That's an example of a alias as an argument. But right now when you have a command that takes an alias as an argument, you sort of have to do a lot yourself. You have to do the parsing of the aliases yourself and then you have to figure out what to do with them. And in deploy there's sort of a framework built inside to take all those aliases and then whatever commands are given to it, it just runs them on all of them unless it's specified to run on only one of them. Another issue is, as I explained before, you have group aliases and then you can execute something across a bunch of group aliases, but right now we do them one after another. So that's a bit of an issue in a deployment because if you have say 10 or 20 nodes, I shouldn't say nodes at a triple conference, application servers, Apache servers, whatever. When it gets to the last step to flip the sim link, you have to essentially SSH into all of those machines one after another and then you have this weird rolling deploy thing going on. So one of the things we had to do was add concurrent command execution. And if any of you guys were in the Drush session in Chicago, you, oh, some people, I demoed this there and we had a patch that was gonna go in. We did a lot in the back end part of Drush, especially with now we have the ability to see command output as it happens. So that patch doesn't really apply anymore. So I recently refactored into this. So it exists in deploy and then that's intended to go upstream, but it does do this now. It executes all these commands in parallel. Now here's what the configuration file looks like. So the application name, all that really does is specify a directory to deploy it to so that you can potentially have more than one deploy on the same box. Deploy repository, you just tell it where your repository is. Keep releases, I was showing branch. Branch master just means no matter what you are, every deploy is just gonna take whatever is in master. So for every group of aliases you'd have, if it's QA production, you just change that to whatever it is. You can also put a tag in there if you need to, although there will probably be an option seemed to just specify a tag on the command line since you're more likely to do that in production. Deploy via is which strategy you want to use. Right now, the only strategies are remote cache and checkout, which the only difference really, there's no functional difference, it's just one's faster because it caches the Git repository. I'll talk about those a little bit more in a second. Deploy to tells it where to deploy the application to. This after stuff, I'll show you how that actually works in a second, but what that's telling it to do is after deploy Simlink, it runs the task called Simlink shared. And then after deploy setup, set up files. This is the way that you can attach your own custom functionality to the existing deploys since what deploy actually does by default is very, very simple and everyone's gonna have different things they need to do, so this is the way you attach events to this. And here's an example of a task. Now this is a concept very familiar to people who have used rake or especially fabric. It's not in Drush right now and I developed this in a bubble and without the other Drush maintainers input, so this isn't exactly gonna go into Drush, but if we can convince Moosh, maybe it will. It's sort of in between a Drush command and say just a script that you'd use with Drush SCR. Any function you put in your RC file and you put task in the doc block above it, I parse those and then make those available as tasks. It may seem really dirty to put say meta information like that in the doc block, but we're using PHP. So if you haven't accepted the dirtiness yet, you should. This concept comes pretty directly from languages like Python that actually you can have that at whatever and it's a decorator. It's a really elegant way of doing this and in PHP we just have to do it in a getaway and I'm okay with that. You should be too. And so you can see each one of these gets an actual copy of the deploy object and then you just run whatever you need to on that. And so this is an example. We'll set up files as an exciting, but SimLink shared. So you can see after the deploy happens, after the SimLink happens, this now also SimLinks the settings.php file and then SimLinks the files. Since those are likely if you have multiple Apache servers, you're gonna want to have a shared files directory, so that's probably on NFS or something or some kind of mount and then that's where you SimLink it. And I have, on the one I just showed, I also have SimLink Dockroot in there. Yeah, because my current or actually, my repo actually has Dockroot. The source is in Dockroot not in the root of the repository. Basically, that doesn't matter but the point is you can do whatever you need to do in these. So if you do say wanna install some packages, I guess, you could do that. You can do whatever you could do normally over SSH. And now before I showed like the before, after stuff in Dresch's RC, you can also do it in the dock block there which makes it a little bit easier. And so aside from custom tasks, you can also do Dresch tasks, right? So it's really common to do say like update DB or features revert all after a deployment. And so deploy will come with a bunch of default tasks since you shouldn't have to then make a custom tasks for all of the typical Dresch commands you have. So this makes one for update DB. And you'll notice it also says run once. Any task or any part of the deployment you do is gonna run on all the servers. You don't wanna run update DB on all the servers, right? Or if you have features revert all or basically anything that touches the database you only want it to run on one. And so if you specify run once, it'll only run on the first of the set of servers. Capostrano, if you know does support actually specifying like roles of servers or like a specific host there. I may support that if there is need for it. I've never really had a need for that. So I briefly mentioned strategies before. So the checkout and remote cache for there. Remote cache just keeps a cached copy of the repository. So every time you do a deploy it does a get pull on that. And then it just copies that into that new directory which is way, way faster than just cloning every single time, especially if you have a big repository. That's what checkout does. Now those are all, they essentially do the exact same thing, right? But because that's completely swappable and without the strategy class deploy does very, very little, almost nothing, right? So we do have the potential of having like a make, a Drushmake strategy or agar, dog and then even DevCloud and Pantheon. Once they have APIs to say do what you normally do in their UI to do deployments you could also plug it into here so you could have a command line way to do the deployment on those platforms since you may want to script that anyways. And if you just wanted to have, if you were doing something with get or you just really didn't like what deploy was doing you could just make your own. They're relatively simple. So, any questions? Yes? Question is, does it only deploy full sites or does it also deploy multi sites? Right? So that's something that you would set up you would set up in your custom tasks. Like it depends what your repository looked like, right? So say one example would be you might have, if it's a typical multi site setup say you have, your repository is just that sites directory for that other site. Then you would just change the deploy to directory to deploy within the sites directory of your parent repo and then you'd have it make the sim link for that in a task. So you could actually have the sites site name be a sim link back to the deploy directory so that you could deploy one site at a time that way. I think that, yeah, cool. Anything else? Yes? Great, great question. So, yeah, how does this integrate with continuous integration servers like Hudson? So as I said, there's a lot of overlap here with like configuration management. There's also some overlap with tools like Hudson but a lot of people use tools like Hudson Jenkins to do things like this. I personally like, I mean, I use Jenkins for many, many, many things. I like to keep my Jenkins configurations as dumb, as simple as possible. And so what I do is I actually run Capistrano. I have Hudson run Capistrano. And so because Hudson just runs tasks, you can essentially do whatever you would do normally in there. And then you could even have, I mean, say the tasks that I showed is like attaching before and after in the deploy framework. You could actually just have those as separate Hudson tasks that run if the main deploy command exited zero. So you could totally bypass it with Hudson if you wanted. Cool. So no question, this just perfectly fits everybody's needs. Schedule it. That's definitely not there now. It's an interesting idea. I had to have necessarily had that need but yeah, that's worthwhile. Yeah. Yeah, something to look at. I'm happy to look at a feature request. Okay, yeah, good question. I think I just assumed that everybody uses code. Oh, sorry. How do you do database updates and configuration changes? I made the assumption that everybody updates their configuration through code because everybody should. So when I said after you do the deploy, you do say like update DB features revert all. Generally, anytime I need to make a change on a Drupal site and that say with a modules configuration that doesn't support features, I put that in an update hook. I have a, actually I make an install file in one of my main features and then I put update hooks in there so that when I run update DB, that gets executed. If you're doing that, then all you have to do is then just attach the update DB task to run after Simlink. And if you're not doing it in code, I mean, there's really nothing you can do. Just sync that up. Excuse me? Yeah, okay, yeah, so that introduces some complexity and now it's, that's sort of, what? Repeat the question. What do you do if you have to roll back in that situation? So that's the situation I was talking about earlier where there are cases where you do want to go into maintenance mode. So ideally, you've tested things. I generally, like before this happens, bring in the database and the QA and then run all these updates to make sure they'll all work. There definitely are situations though that you can run into where this will happen. What I would do, I would put the site in maintenance mode. I would make a task for that, which really there will be a task for that soon. I would then make another task that does a drush SQL dump and I would put that to run before deploy. So that's the first thing that happens. And then I would, in the rollback hook, I would say on rollback restore that. So, and that's the idea. It's like everyone's situation's gonna be different. Everyone's gonna have different needs there, so pretty much any customization you do, you do in tasks. Except for the crazy scheduling thing, you can't do that. Anything else? Yes. You could maybe add checks for that as a dependency that would run and that maybe, are you saying this would be after a deployment ran? So in that situation, I think I would deploy to, not the whole group, say like one, and then I would comment out like the after tasks I had to do, the updates. I would check those things and then if that was okay, I would do the whole thing. Does that not work or am I missing something? Yeah, I'm not sure. I mean, we should talk about that. Any other questions? We have a few more minutes. So if there's like more generic drush questions, we can probably take a few of those. It was hard to hear. Could you speak up a bit? Sorry, that was a long question. So this was a question about, so let's see if I got it right. You want to execute a set of drush commands against a set of sites in a multi-site. And right now, all that drush provides is you execute one command and you say against a target of sites and you want to execute a series of commands against each site in a row. The solution that drush provides for that right now is to make your own command that encompasses the three or four or five commands that you wanna run and then target that against a site alias list. One other thing though, in drush five, we introduced the concept of shell aliases. And so instead of having to go through the trouble of making your own command, you can actually just in your options file do a shell alias that has those three commands together. I often do that for like, updateDB, features, revert, CCL. Yeah, I'm having trouble hearing the question. Could you speak up? Yeah, yeah, please. Is it possible to publish multiple database servers? So can you publish it to multiple servers? Updating to multiple database servers. And then what's happening on the multiple database servers? I missed the second part. If you have a server for user data and servers for content to... Yeah, so yeah, update source code, not data unless you do the data in your tasks, which in that case you can do anything you need to do. So I guess I'll answer a generic version of what I think that question is and that say if you're using press flow and you have masters and slaves set up. Actually, yeah, so say you wanna do the SQL dump off the slave, you would just do that in your tasks like you normally would in Drush. Question? Let's wait for the mic. I can't really hear. You said it was very easy to write your own strategy like remote cache or checkout. Could you show the code and quickly outline how to do that? Sure. And if you'll notice Drush Deploy uses the PSR zero namespace auto loader, if you're familiar with that. The checkout is really simple. All it implements is command, which then, yeah, which then calls the git class, which would later be just the SCM class and the checkout method. So the method of actually checking out is subtracted into that. And then you just give it the config revision, which Deploy automatically generates for you and then the release path, which is gonna be the timestamped directory that Deploy gives you. So that's a very simple example. Remote cache is a bit more. Actually, all the strategy class has to do is implement the deploy method. The other one didn't have a deploy method because the class that extended had one. So the deploy method here, all it is, is update repository cache and then copy it. And then it adds a check. Like I said before, remote or strategies can add their own dependencies. So that checks that the git binary exists. That gives you the path of repository cache. So update, all it is, is basically seeing that directory and then doing a git pull that I lie about that. Oh yeah, so git sync. Sync is a generic term that means pull in this context. And then it does the, yeah, and then it checks out the tag or whatever, actually whatever the tip is, right? Whatever you're gonna deploy. And then it checks that out before the copy happens and it just copies it into the directory. I think that's relatively simple. Another quick question. I was thinking about writing a deploy module for FTP. I know it's not the most used hosting environment, but it happens to come up again and again and I would like to have one deploy solution for all. Is it possible now in Drush to run Drush commands via PHP instead of the command line? Like remote execution? Are you saying Drush, run Drush commands through the browser? Yes. No. Short answer is no. Longer answer is Drush needs to do things that your web server should not have access to do. So if that works in your web browser, then you have a misconfigured and insecure web server. So I would say the best answer is no, you don't wanna do that. The point is the files could be deployed via FTP, but SQL, but up to be, and for example features with Word could still run directly from a web server. You wouldn't need to shell access for that. I missed that something about features revert. I mean features revert isn't actually doing anything to the file system, it just updates the database. Yes, that's what I meant. I could run this Drush command through a web server without it having access it shouldn't have. Well, so, okay, take those examples. So feature revert you don't even need a Drush command for because the features module provides you that functionality within the UI and then update DB is update PHP. So any of those things there already is a UI component for anything else that Drush is doing like downloading or whatever else you can't really do through a web server. So was there another question up there? Yeah. I did deployments for a year with examiner and I wish we had this, this looks very good. The only thing that I think that I'm not sure that you can solve is frequently one of the post deployment tasks. We'd have to write a Drush script that might run for 12 hours. For a long time and so we'd start that in a screen session. And that's the root of Kevin's question, right? Yeah. Okay, so it's already been asked and you're thinking about it. No, I think, but I could use more explanation. You might have a long running post deploy tasks that you wanna see status updates on. So you might have a task that somehow through the UI you get responses, I'm 1% done, I'm 1.5% done, I'm 2% done and it's gonna run forever. And I'm not sure how you would handle that through this. Maybe somewhat related, but one of the things we did fix in core in five is as you run remote commands you actually see the output live. So while the deploy was running you would see the output from that task. I don't know if that would help. And then you could, the percentage is hard to do but if you implemented that percentage logic yourself you could just print that to standard out and then you'd see it. Maybe that's good enough. Otherwise, yeah, I mean, I'd love to look at that more and see if it can be accommodated. Any other questions? Right now it's actually not only for Dresch Five. The question was, is this only for Dresch Five? Yeah, so it's not only for Dresch Five, mainly because the things that I needed to fix in Dresch Core are not in Dresch Core yet, they're still in here. So if and when a lot of these things do go into core I'll still probably keep them here so that it'll work on for and then default back to core because I would like for it to last a little more. Although, yeah, this and like I should say this everything I showed you works. There was no magic there. But there's still gonna be a lot of changes and this is not nearly 1.0. And I apologize if you go to the project page right now there's zero information, no releases and no example of the configuration file. The code is there. In the next couple days I will fill that out and have something there so that you can test but I would not use this right now. I would probably wait until it matures a little bit and we get a release out there but if you like all this stuff and you need this right now you might look at Capistrano, it works. Anything else? Okay, thank you.