 All right, well we better get started because we got way too many slides, so buckle your seatbelts. This is continuous delivery. My name's Jody, I am actually filling in for Howard, so if you came here out of your love for Howard, talk to you later. He has newborns and twins and they're sick. This is Mr. Gregg. Yep, Gregg Kanadisen with card.com and we've been working with some delivery continuously, so I thought I would share some of the things that have worked well for us. So I'm the CTO and co-founder of ZivTech, which is in Philadelphia where I worked with Howard for about seven years. I've been running ZivTech for eight years. Card.com about four years ago got founded and we're a mobile alternative to traditional branch banks. We do lots of fun stuff and we are hiring, card.com slash careers. Is that your real credit card number? It is not. Okay, so just to, these sort of like buzzwords, continuous integration, continuous delivery, continuous deployment. They kind of chipped me up a little bit, so here's what Howard told me. Continuous integration basically just means that you're not, you don't have separate developers working in separate branches for months at a time, and then try it right before deployment to bring all the work back together. Instead, you're continuously integrating everyone's work. This isn't a real problem in Drupal. It's a, was bigger problem like in the Java world, I think, where it has to all come together in this big build. In Drupal, most people kind of do continuous integration. If you're just all working in Git in master, you're basically doing continuous integration. You might not have tests, but you're probably already there. Continuous delivery is when you are always ready to do a release. It doesn't mean that you're doing a release 10 times a day or 100 times a day or a million like Etsy or whatever they say. You might still do your releases every two weeks or once a month, but at any point in time, if someone needs you to do a release, you can say, yes, I have a branch that's release ready. Okay, so instead of you having to say, oh, I have like some things that are ready, I could probably like cherry pick them out, then we could release. It's always being ready, having already tested everything that's in a branch that's ready to be released. Whereas continuous deployment, is it really that once you have continuous delivery, it's not that hard to go into continuous deployment because you're already ready to deploy at any time. It's more of like whether your organization likes to deploy all of the time or if they would rather wait and do it at once. It depends on how fast you really need to innovate and get changes up. So to remind everyone of where we've all come from, you know, we all, if you're, you know, my age, you at some point were working in Dreamweaver and FTPing things up to the live server. There was no other server, right? There was just your computer and the live server. And that was, I think where most of us were in the Drupal world eight years ago and of course where our WordPress brethren still lie. Sorry, I don't really plan what I'm going to say and then I say root things because I get nervous. So then we got a little bit more advanced as a community and as a software in general and especially, you know, first we started, most people were working in subversion, then most of us moved to Git and then we kind of moved our subversion, a lot of us kind of moved our subversion workflows into the Git workflow and so we would just kind of commit and push to master and then we would pull master on dev and we might have like five or ten developers or more all pushing to master and then reviewing everything on dev and that's sort of the, probably the standard workflow I think for most Drupal developers now. And of course then there's this whole thing that Acquia and Pantheon helped us to make more mainstream of which way the code goes and which way the database goes and so that's all pretty well understood. But for me at ZivTech I am obsessed with code review and testing and quality so I had a big problem with the way that workflow of just sending everything into master straight up to dev, it didn't enforce any review period before things went into the main branch which leaves me in a state where I'm having to come in later and review things and now I'm reverting commits or I'm cherry picking things out of places or they want us to hurry up and do a deployment, we haven't reviewed everything that's in master yet. So I don't really like that part of that workflow. So a lot of us have been moving to this pull request workflow and that helps with that code review problem because in the pull request which really is created by GitHub that concept of pull requests you're working in a feature branch for each ticket that you're working on as a separate feature branch and then you create a pull request in GitHub or something similar and then you can assign that pull request to your lead developer whoever on your team is going to do code review. This way the code review happens before the merge and now nothing unreviewed is in master or whatever your deployment branch is. So I like that a lot better but the problem was, not that I was stoned so I just always look like that the problem was that this is what my work, my workflow ended up just being coming more complicated than before because I would assign the tickets and my stereotypical hipster developer would, now I really is a recording this. So he starts doing his work, that's my project manager, he's trying to see what's going on he really has nowhere to see what's going on. He says it's done but everyone has a different definition. So he wants to do a review, well he can't really so I have to do the review because I have to go to my virtual machine and check out the right branch and revert the features and run the database updates and whoops something's wrong and now I have to get a new database and 25 minutes later I checked it and reviewed it. So I'm basically doing my own QA because you have to be a git master to even do it. So I'm doing all this stuff now and of course it's never right the first time. So he's scared and he's, I don't know, constipated. So then it comes back to me again next day, guess what, another half an hour. Hopefully it works this time. But of course I'm bottlenecking the whole process because I have so many things that I need to do and here I'm doing my own QA, it's taking me a half an hour. So finally I merge it. Oh great, the client hasn't even seen it yet, right? So it's merged, great. But of course it wasn't what he wanted, right? We were just basing off of his weird email that we didn't understand. And so now he says, okay, I know it's already merged, he doesn't know it's already merged, but it's already merged and he says, well we need to like do a deployment but only deploy some of the things, not the others because this one needs changes, right? Now I'm cherry picking 200 commits. That's my whole morning. Of course Jason can't do it because he's a junior developer. He's going to screw up somewhere cherry picking these commits. So it's going to be my whole day. So that's where we realized that we needed to change up our workflow so that we would have continuous delivery so that what was merged was always ready to be deployed by moving where the review happened earlier in the process. So what we found is that what we really needed was QA sandboxes. We needed an easy environment where people could do QA without having to be like a great developer setting up their own stuff all day. Because without the QA sandboxes we had master, they had a mix of things in different states. It was hard to test the feature branches which meant I had to do it. Only the developers could see anything on the feature branch because there weren't environments for them. So again it has multiple environments and a lot of things are moving this direction. And then the test contents getting wiped a lot. So when we added QA sandboxes that let us have continuous delivery because now master or whatever branch is always approved by the client and already gone through user acceptance testing. Everybody can easily see each feature being worked on. Review is so easy that I don't even mess around with my VM anymore. Like I don't even have time for that. I just like review things in just like a minute instead of a half an hour. And the test content is always there. I can have a QA person put the test content in before I review it. So I'm not doing that myself. And best of all we're testing the deployment steps every time we make a new feature branch so we don't have to worry about deployment. It's already been tested over and over. So this is the new emotional state. Still stoned. He's the same. He's checking in. Oh but see here's the difference. He checks it himself this time right? Not me. I'm not even here other than the beginning. He's happy. Well that's lucky. That's like one of those before and after where they also put on makeup. Then he, the client now gets in the mix. I'm not even there. The client's now in the mix. I don't know why he has a leg. It's sort of like dehumanizing. And then now all I have to do is the code review which is much quicker than the testing. Merge and deploy it. And everyone is really happy. Okay. But now Greg's going to make it real. I was just going to say I think that's sort of the ideal workflow. The makeup is on in the after photo and it's the ideal workflow. And I feel like myself about three years ago I was looking at this and I was like yes that's what I want. I want to automate all the things. But this is sort of like the QA Nirvana. But I think a big thing is to just remember you don't have to start there. So there's no need to start there. And really I think that the key things to do at the beginning are all really simple. It's just about adding this into your workflow. And I have to be honest that maybe some people that I work with, maybe myself, pushed a syntax error onto our live website. You don't always test all the pages of your site. So PHP-L is a really easy automated test that everybody can get running in about 30 seconds. And so add that into your flow of rebuilding your site automatically and just doing something stupid simple like PHP-L that does add value and will help you with this idea of adding some level of automation. And then just add more and more incrementally, do it slowly, and you will eventually reach ultimately the QA Nirvana. Okay, and in QA Nirvana you must have repeatable deployments, right? So no weird little list of things you have to change after the deployments, which means of course our friend features module for Drupal 7 soon to not haunt our dreams. And it's greatest friend, the amazing StrongR module. And of course HookUpdateN who's been with us even before features module, you know, forgetting out our configuration changes in a repeatable way. And then of course Drupal 8 configuration management system is going to be much better. I've gotten to use that a little bit. It definitely is a lot less painful for having repeatable deployments for Drupal. There's other things you can use in your deployment stack. You can run automated coder or PHP CS to sort of do some of the code review in an automated way. And then of course there is automated testing. And I think, I mean this is, you know, I guess how many people are not doing automated testing as part of their everyday workflow right now, right? I think it's a lot of folks and that's why I really recommend the idea of starting as small as you can. So David Pintato also did not do it before joining our team. And now, you know, this is a comment of his from a retrospective nine months into the process where he's like, I don't know how I lived before automated testing. I feel like, you know, whenever we used to do deployments without having solid automated testing it was always this really stressful event. And that's a big part of getting to continuous delivery or continual deployment is that you have to be confident that you can push that deploy button and that things are going to be okay. And so having solid code coverage in your automated tests is I think one of the key things to get there. And, you know, that's one of his points. Another element of this is that, you know, now he's moved on from that perspective of just like, I don't know how I live without automated tests to like working with test-driven development or TDD, BDD, where you're writing your tests first that match the, you know, match the ticket that you're working on, match the feature that you're working on. And it helps you avoid those situations where you're like, oh, this would be cool to work on in addition to this ticket I'm working on. Oh, also this other thing would be fun to work on. And then five days later you come back and you're like, wait, I was supposed to pick one small bug? So the TDD really helps with focusing in on what it is that you are trying to build and making sure that you're delivering the value that, you know, the whole team has agreed is the highest priority. And a lot of Drupal people are using B-HAT for testing. And if you want to get started with that, I think the best place is to go to the B-HAT Drupal extension. They have like a docs page and they tell you how to set it up and everything. But I've had a lot of success just having junior developers or even QA people writing the tests alongside the developers because these B-HAT tests, you don't, 90% of the time you don't have to write any code. And then when you do, you can ask for help. So one other important step in this process is, you know, we've got this idea that our databases are moving from one environment to the next. And along the way, I think it's really important to not just do that, but also to sanitize the database along the way. So what I mean by that is, you know, you've got a lot of passwords, maybe some private content inside of the database. You know, emails of customers, maybe millions of emails of customers that you don't want to get leaked. And so as the database moves on to, for example, a room full of laptops that might get left in a TSA bin, you know, you don't want that customer information to get stolen. So there's some really nice tools that have been developed for doing sanitizing of Drupal databases. So like the cool way to do it three years ago was to just run a script that had some delete statements and some update statements in it. And that sort of, that concept got added into Drush with Drush SQL Sanitize, but it was still actively a list that you had to maintain all the time to say, you know, oh, I added a new table. I've got to remember to also add it to my sanitization script. And then about four or five months ago, in workwithcard.com, we created a tool that will work on a whitelist basis. So it'll say, these are the tables that can keep content in them and then everything else should just get truncated. So that way when you add a new table to your database for all the super secret info, the default is that it's going to get deleted and your data will be safe. It also helps, you know, Joe was talking about the half an hour process of rebuilding your QA environment. If your database is much smaller because you've deleted all the tables, then that half an hour process turns into maybe a 20 minute process. So a little less time for Reddit, a little bit more time for QA. Okay, so obviously automation is the cornerstone of getting a lot of this stuff working because we're, you know, we're talking not only about moving through your dev workflow up to deployment, but now back from production down into your dev workflow. And, you know, our favorite Butler Jenkins is a big part of most people's workflows as they build tools that handle these things. So if you're not using Jenkins yet, it's very easy to install on a server. And just like a job runner, you can use it for backups or deployments or moving the database between environments and sanitizing it or whatever you want. It's better than just running things in cron jobs because you can see the output in one place that's nice to be able to see everything that ran and what it outputs. So that's really a cornerstone of a lot of this type of automation. And then how many people have worked with Travis? Okay, a lot. Or Circle? Okay. So Travis is, in Circle, they are continuous integration tools that will run, basically run your automated tests for you. So you might have like a workflow where every time someone makes a pull request, it triggers Travis to run all of your tests against that latest pull request and let you know if that passed or failed. So this is just an example of it's the Travis in our face just running its tests. But what we really felt was that automated testing is never enough. Now it really depends on the types of projects or the individual project that you're working on because Greg is talking about credit cards. So obviously he is really concerned about automated testing and unit testing and making sure everything is not messing up people's financial data. But if you're making a website for, I don't know, just like, we do a lot of brochure sites or just something where the data isn't really that private and critical and it's not exactly like a moon landing here. Then what you're dealing with more is people who are like, oh, that's a pixel off or I need you to change that color. We need to put some tabs in there and change the navigation. And so you can't really have automated testing for whether or not the client likes your new navigation. So most of what I work with on client sites needs feedback. It needs human feedback. There's no automated test that's going to tell us whether or not the client is satisfied with the feature. So I never really thought that those people with the automated tests, we have to do everything automated and then we can deliver it at any time. It's sort of like, what about feedback? A lot of us got into programming maybe because we don't like talking to people. But the truth is that it really is all about talking to people and you can't engineer them out of the system. You have to engineer the system for them. So what we have been working on, especially, this is really Howard's baby, not his real baby, who's sick, is probo.ci. So the idea with probo is that after you, I hate to confuse this, this is like the family circle following the dots, so after you submit a pull request, then it creates a build and can run the test if you have tests just like Travis would, but instead of just like taking it down afterwards, it just leaves it up and gives you a URL where you can do human testing. So it has GitHub and Bitbucket integration. It's built with Node.js and it runs Docker containers, so every single new pull request you do or update to the pull request is a new fat Docker container so that you have root access and you can like install anything you want in that environment as part of your build. It's mostly open source. It's github.com slash probo.ci. So you can run it yourself on your own servers. The parts that are software as a service are the web UI and sort of the coordinator for handling all of the different containers that it's setting up for all different people. And we run it on packet.net servers, which are excellent. So it's much faster than running on my local environment or my other servers. This is the website. It's like freemium. Oh, we're going to do a demo. That's what I'm forgetting. So what is really great about Probo, which you will see, we'll maybe give you a live demo if the Wi-Fi works, is that you just don't do any work. That's what I like. You just make a pull request and then it makes a build. Here, let me start this over here at the beginning of this. This is Greg making his pull request. Yes, I noticed that there was sort of a word that didn't make sense on our frequently asked questions page. I just went into the GitHub view of the tpl.php file that was responsible for that content, edited it, made the change, used the GitHub UI to create a new pull request for it. Do I narrate faster than I write sometimes? Use a really good explanation for what I'm doing and follow all the rules for good commit message naming and pull request naming. I do a lot that I call it my GitHub-only workflow. If I just have to make a small change, not a huge coding thing, I just work in GitHub so that way I don't have to set everything up locally. Because it's fine because as soon as you make the pull request, you're going to get to preview it. So here he is. This is actually a new feature of GitHub that they launched is that you can have pull request templates, which is kind of nice. It's a standard pull request template that I'm going to go through and fill out and it's got nice little happy characters like the sparkles for all the things I'm happy about and magnifying glass for things that I'm asking other people on the team to give an extra eye on. Okay, so he created it. And I'll go over here. Yeah, as soon as the pull request gets created, basically GitHub web hooks are kicking off. If you're using Jenkins, then you can have Jenkins doing pulling. If you're using Probo, Travis, Circle, those kinds of tools are getting a web hook from GitHub to kick off their build environment. And we've specified, in this case in our Probo YAML, a set of steps that it needs to do. So we've got some dependencies, some composer items. Phantom.js we're using for some of our tests. We're using Behat. We have some specific versions of things that we need to get downloaded and set up as part of the build. So Probo is going out and doing those steps on each build and then just running through everything that we've asked it to do. So the main tests that we use are a mix of Behat and Drupal simple tests. And it takes for us about 24 minutes, which feels a little bit long. And it's kind of amazing. They were running more like 40 minutes, and then we moved over to Probo and they got kicked down about 20 minutes. We keep adding more tests, so it's up to 24 minutes, which is kind of a good problem to have. Yeah, so that's the flow. And then I'm waiting for the build to finish, checking out some imager, looking at some dank memes, and then I go back to Probo and I'm looking at... Right, that's me. So I can see that all the steps have passed and they're either successful or failures. Of course they're successful, always. Definitely never a build fails. So yeah, so things are good. I can look at the logs that are available forever. Those are just available. You can inspect what's worked or what didn't work in previous builds. And then as Judy's mentioned, there's the live site that's just going to be available. And so I can click out to that and take a look at the Frequently Asked Questions page there and see that my change of one word did actually work as expected. But it is nice to have that. Obviously for a more complicated change, you can have that live QA environment be able to hit, see what's happened with your changes. Cool. Well, we really flew through that. But... Okay, well, I also am enjoying it for open source projects. So this is my installation profile. It's called Bear. I just use that for all of our projects. So we have... We just do everything as pull requests here. I got a lot of... I have a lot of pull requests. But it's very easy to deal with. So say I... Even sometimes I just do it just to play with something. Like if I want to try out a new module, I can just go into my make file here and edit it. And then... What's a cool Drupal 8 module I could add? What is it? I think I already have that. Let's see. The hipster profile has got all the modules before they were cool. Come on, guys. Give me a cool module. All one word, you know, separation. Underscores. Emojis. This just sounds like a joke module. Entity Query API. Entity... Entity Query API, no term. Okay. So I'll get this version. I'm going to go with dev. Point X. I'm also a big fan of pair programming. This might be the biggest pair programming session I've ever seen. Wait, why did you say I missed a U? Who said I missed a U? Yeah, you're one U short of a query. Um... Okay, so... I don't know if this is... going to work. Okay. So I'm going to make a new pull request because that's the whole game here. I don't like use a template. I just press the button. To each their own. Okay, so here it is. It's building. Mine runs a little bit faster than his because I don't have as many tests. So... It's going through. These steps are actually configured by a YAML file. I should show that to you while it's running. Let's see if I go back to the code. This one is a little bit complicated, but usually it's pretty simple. You can just run shell commands. And just put each shell command you want in there. But there's also like a Drupal plugin. There's a WordPress plugin now that makes it simpler. We didn't use it for this because it was a weird way that we wanted to build the profile. But here's how we run the behat tests. We just go over to the test directory, do a composer install, and then we do bin behat. So let's see if it's... So you can actually like change these steps and let's see how that blows up your features. Okay, so it's still going. So if you click these links, that's what takes you over to the Probo app. And then you can see the output. There's a lot of sadness happening here. It says it's downloaded entity query API. That's lucky. Okay, so now it's up to running the... It finished the Drupal install. And now it's running behat tests, which that takes a minute because it does a composer install and it gets a bunch of stuff. But soon... Okay, here it's running some tests. I have some really easy tests here because I kind of train QA people to write the tests. So this one is that as a site admin, I'm able to log in and publish content on the site. And this one is that they are able to add a menu item. It's almost done. It doesn't auto update right now. That's why I'm refreshing like I'm on eBay in the 90s. This is definitely going to work. Okay, you can tell it's done because it says view site. So there's actually two links here. View site and view build permalink. They're a little bit different because this is a pull request based link, which means if I give you this link, it's not always going to be necessarily the same build because if I update the branch, then it'll be showing a newer build. So that's useful if you want to just send someone a link to like what's going on with that pull request so they always see the latest. But if you have test content or something in your sandbox, then you would want to get the permalink to make sure you give them the build where the test content is. Okay, so you can go back to GitHub here. You can do a rebuild if you need to. I was lucky everything came out green, which is the opposite of Murphy's law. So the weird thing is because this is an install profile, there's no database. If there was a database that you needed for your build, you can just go here and build assets and drag in files like databases. Or you can do it. There's like a command line way to do it. Let me go back to my... These are just all of the builds that I have. There's even older builds. There's a lot of builds. So here is my link to see this cool module in process. Now, because there's no database, I have to figure out what the password is. So if I go to the run install step, it says installation complete. Use the name admin, user password, blah, blah. So let me copy that. Okay, so now I guess I can see if I got that cool new module. Ooh, REST style entity query endpoints. Okay, so it worked. So I like that better than trying out a module on my local environment because now I can pass this off to someone else and I can say, hey, check it out and send them the link. So what do you guys think about it? And can you finish setting it up for me? So that's about it. Any questions? There is a mic in this aisle right here. If you could speak into that, it's being recorded. Otherwise, Jody and I have to repeat what you said, which we can also do. Any questions? To repeat the question. So the question was, why do our tests take so long? I think it's a mix of a couple of different factors. So there's this philosophy about tests that they should be in a triangle shape, basically, that the majority of your tests should be unit tests and unit tests run extremely fast, but they only test a small component and then you should have a band of tests, a smaller number that are integration tests that go across a function maybe, that go across systems, go across modules, but that are still at the code level and therefore designed to be fast and that you should have a relatively small amount that are UI tests. It turns out that the tool for writing UI tests that we're using, Behat, is very easy to work with. Very easy to write those tests and it takes a minimal amount of time to get good coverage and so we wrote a lot of tests in it. It turns out Behat is also really slow in comparison to unit tests. So we're sort of migrating to writing more and more unit tests now that the time has become an issue for us and I should have mentioned that earlier. We have a small number of unit tests. Unit tests in Drupal are kind of hard because you'll write one and then you'll realize that somewhere in some function that you called, something needs a database and so the unit test just blows up. So for the unit tests that we write, it's mostly our custom code for the unit tests with unit tests. Do you use Selenium on the Behat tests? We don't use Selenium. We use Mink and for the JavaScript based ones we use Phantom. Oh. Yeah. Oh, you are actually. Yeah, get in line at the mic. Yeah. We just spin up like 10 environments, 15 environments and grab like, you know, prep for feature and find those and send 10 this way, 10 that way, 10 that way, whatever, right? There are, I don't know, Frank Carrey's here, but he's worked on that problem a bit. Like, I don't know that that's built into any continuous integration tools, but. It is built in Travis. Oh, in Travis, okay. Thanks. And at the microphone. Other tools I've seen which solve similar problems have ways to get a command line on like an instance that's like blown off so you can go put around and see what happens so you don't have to go out and see what happens. Is that something that's in the stool or is thought of? Yeah. It's on the roadmap. Okay. It would be great to be able to SSH in there and see what's going on. So. Which should be possible. I mean, I have, you have to just, ideally Docker enter and I go into them, but it's just a matter of the security and, you know, see, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, it's just a matter of the security and, you know, see, I, I even use Provo to, um, work on a WordPress site, which I would never have done because I wouldn't like soil my computer with it. But I can do it all in the cloud. I think it's the servers. Yeah. Yeah. So we were previously testing, we were using Jenkins running tests on, just an AWS instance. And I think it's probably because the AWS instance that we were using was not as performant as the Provo, um, servers and containers. Yeah. So packet.net, um, it, there, they might even have some people here. Um, but yeah, that's the, the basis of the hardware for Provo. And it's fast. Yeah. I mean, I, what these guys in Advomatic or something? Some, yeah, I think some of the folks at packet.net used to work at Advomatic. Advomatic's definitely here. Any other questions? Thanks very much, everybody. Thank you. Oh, I was supposed to say this. I was supposed to say this, that you're supposed to fill in our feedback form. Please. If you have something nice to say.