 Okay, how's the audio here? Good, at the bottom? Can you hear me well? Right, so we're gonna start now. Thanks for coming. Just to get an idea of who's in the audience, can you raise your hand if you're a developer? Right, mostly, right. C-Satmin, right? DevOps, pretty much the same. Some people say they are DevOps and not C-Satmin. Who's none of those? And we like to say what's his or her role. Right, pretty much that's it. Okay, great. So, little introduction. My name is Juan Pablo, but I like to be called Huampi. That's how everyone calls me. That's also my nickname in most of the social networks. Sometimes I take 72 whenever Huampi's taken. I've been working with Drupal projects for around almost five years, and I maintain some modules in core and in country. I wrote a book on Drush. It was almost three years ago. I'm writing a new book now for packet publishing, and it should be ready, I guess, by the end of the year. And since a year and a half ago, I started using AngularJS in Drupal projects, and, well, I love it, the more I use it, the more I like it. And, well, my employer is a lot of what I worked as, a developer, mostly backend developer, and a little bit of JavaScript as well. So, here are two tools that we're gonna mention a lot during this talk. Since I already asked what kind of audience we have here, I guess, pretty much everyone has used GitHub. We're gonna talk about how to integrate systems with pull requests so we can spin up testing environments with it, and Jenkins, if you never used it, it's a continuous integration system that doesn't mean anything to me, but if we would wanted to define it, we could say that it's something to automate tasks, like you do something every day a few times, either in your local or in a server, well, Jenkins can run it for you, either on demand or periodically. So, we're gonna start with a practical example. Let's imagine this workflow where we have a ticketing system, JIRA, for example, or even GitHub, if you like, for each ticket, whenever we start working on a ticket, we create a branch, and then we push that branch and create a pull request on GitHub, and we will have a testing environment for each of these pull requests where we can actually test our branch in action, and we can share it with other folks from the team or with the client. Once this gets merged, it goes to the development environment, it's kind of a main vision of what's currently in master branch, and then there is a later process to deploy a tag, like a release into staging first and then into production. So, given this scenario, if I start working on a ticket and I create a pull request, once I'm happy with the work I've done for this particular ticket, I go to GitHub and assume that obviously that we have our code in GitHub, and I create this pull request with the ticket number and, well, there's a little description there, and after a few minutes, I create this pull request. There is a user in GitHub, it's called TacBot, who has posted automatically this information. First is a URL containing the pull request ID that contains this branch, and you can navigate to that. And the second one is instructions on how to treat it down. Either you want it to be rebuilt or you're simply done with your peer reviewing. And this is just a Jenkins job that destroys a Drupal installation while this is the result of another Jenkins job that clones, let's say, a master Drupal site. So in this particular project I was working, we were using the IRC plugin for Jenkins. It adds a bot into a channel, and you can trigger commands to build a job and provide parameters to it. There are many other ways, there have been other projects where we would just post a link that you click on it and it tears out the environment, that the end is just a way of deciding how you want the job in Jenkins to be triggered. So then it starts the process that we call the peer review process, which is just a discussion. You and other folk are looking at the code and making sure that it works, testing it on the testing environment and maybe getting screenshots of it and then you get to a point where the peer reviewer says, okay, this pull request is ready, meets the requirements of the ticket. So I'm gonna merge it, so the pull request gets merged and this person, Labu, he runs the, he goes to IRC and says JDL, this pull request ID, and the tag bot user, which is actually a user that is managed by Jenkins, just mentions that, hey, it's been destroyed, that's fine. That will delete all the file structure, drop the database for the testing environment and that's it. So how does it help? Well, it helps everyone in the team to make the development process way more flexible. Everyone can work in an isolated environment for each ticket and you can group a bunch of tickets in a testing environment. So let's say if you're working towards a project that will last three weeks, you can create all your pull requests on top of a branch and have a testing environment for that branch and then you can have the client to be closer with you having a look at that testing environment before you merge that onto master. So I'm gonna go over some of the different stakeholders on a project and how this workflow with the pull request builder helps them. So first of all, clients and the QA team, they got a URL that they can test what you're working on. This screenshot is from the MSNBC project where we are using it and they run a live event where there was a panel, people were debating some topics and you could vote and they would read the stats live and discuss on top of them. So they wanted this to be ready in, I think it was just this tool to be in two or three weeks and obviously since it was such a short time for that, we needed to be like pretty much every day just being very close with us, working with it. But at the same time, we also had to keep on doing normal releases to the project. So we couldn't do everything in development. So that's why we created a base branch for this out of master. This was the main testing environment and we created, we had obviously a bunch of tickets to accomplish this project. And we would create sub branches on mini testing environments, finally, sorry. Getting them into the testing environment and every two days we'll ask the client, hey, this is the current progress, how is it going? So it gave a lot of trust to the client and the QA team that they could see things and not having to wait for us to prepare a release and deploy it to staging, for example, and maybe we had to roll back. So another stakeholder that benefits is, and we realized it's really helpful for the development team, our external teams, whenever we had to integrate JavaScript API from a third party team, we encountered this scenario where for example, this is a listing from the MSNBC project and we integrated the, all the social stuff was coming from Newsvine, which is MSNBC's social network. So they provided a list of widgets and API calls so we could bring up all the social media stuff in the website and in this view, this is a view powered by, it's an Ajax, it has Ajax Paging and we realized that when we were implementing these social counters, it was just a widget that their API would parse and render the counters. So on first page load of an article, this was working fine. But then when you would click on next, the next set of articles would load, but not the counters. So since we had a testing environment for this, we could just send an email to the team and say like, hey, this is something missing, how should we do it? And they helped us, so we fixed it, we rebuilt the testing environment, we confirm it with them and that's when we considered the ticket to be done and that's when we merged it on the main branch. If we wouldn't have had this, we would have to, for example, merge our branch on top of master, let it go to development, ask them and then fix. The problem of that workflow is that you are polluting master with work that is not ready and imagine if you merge something, goes to development, well, goes to master branch and someone else has to push something to production so that's a hot fix plus your code that is broken and this is JavaScript code so it's not as damaging that if it's PHP code and it has a bug, then well, you just got a bug into production so this way we keep it isolated. And for the peer review process, well, it helps with what I like to call the upgrade path, you know, every time you download a database from development or production into your local or every time you deploy code, you need to go through this set of steps, register, rebuild, run database updates, revert all features and make sure that everything works. This process has to be reviewed whenever, for example, you add database updates or you manage fields and export them in features, sometimes you need to have some extra code for that or sometimes database updates need to run in a particular order and the good thing about this process is that the Jenkins job will check out the branch using a fresh copy of production or development depending on how often you download a database and run the steps and if the steps fail, Jenkins will realize that some of the steps failed and it will post a guest or if you got access to access Jenkins, you can just see it there and post it on the pull request. So saying like, hey, there is something wrong in the set of steps to upgrade the database so have a look. So this is really good for both developers and peer reviewers in a way that they can prepare a database upgrade and test it without even having to download in the database every time they want to test it. Whenever the database starts going bigger than two gigabytes or three gigabytes, this saves a lot of time. Also for peer reviewers, you don't have to have a local. Like sometimes I've been with a different laptop and doing peer reviews because if I want to test someone as a ticket in my local, I'll have to check out the branch, do the whole upgrade process, make sure that I have a working environment. Maybe I have to also touch a note and put some certain conditions that meet the ticket requirements. I don't have to do that if I have independent testing environments. I actually, the developer can just prepare the environment for me and say, hey, just open this note. I prepared it for you. You can see the work I've done and you don't have to actually set it up in your local. So again, it's a time saver for both of them. This is a very simple architecture of how this works. Let's say on the top right corner, we can see the production environment. You can set this up with Drush or with Krone or with Jenkins just once a day, depending on how important it is for you to have fresh databases on your testing environments. We normally do it once a day. You download files on the database onto the testing environment. This testing environment uses a wildcard for creating websites for each testing environment. 72 would be a pull request ID from GitHub, same as 543. I don't know how this set up in Nginx, but in Apache web server, it's called a dynamic reader host. You basically set a star here and that number then maps with a physical path, like this would mean bar dot dot dot 72.pr, my site dot com, for example. That's how you can manage this. Jenkins on the bottom left is kind of orchestrating, like it's watching your GitHub repository for new pull requests or updated pull requests. It's actually a plugin that does it automatically. Whenever there is a change in your repository, it will do whatever you want. In this case, it will call the build job, yeah? Exactly, exactly. So basically, Jenkins just listens to these webhooks in a way that if there is a change, it will run whatever you have in your job in Jenkins. So either, yeah, there is a new pull request or either there is a new commit in the pull request. So when that happens, it will trigger the job that basically rebuilds the testing environment. So the actual build job in detail would be take out the particular branch or well pull request from GitHub. There is in the testing environment, what we call a master site, which is the template that downloads the production or development database. So it will clone it, like just adding a suffix, let's say the database is my site, it will clone it to my site underscore that pull request ID. Oops, sorry. In the new file structure, it would adjust settings, just append the database array, maybe overwrite some settings that you may need, it depends, we realize that every project is different, so we have to make normally a lot of adjustments on this system, so it really represents a production environment. Run all the steps to upgrade database and finally, post a comment on GitHub, oops. And as for the teardown, whenever you trigger it, it just drops the database, deletes all the project files and post a comment, it's very simple. Okay, so over time, we started using this system on the Tyson project because they needed this kind of functionality, and then we started using it on our corporate website at Lullabah.com. Over then from that, we went over to Martha Stewart and we used it there as well, and then it was MSNBC. During all of these projects, obviously the code base evolved and we, in some cases, we added extra jobs to test extra stuff, like for example, in the MSNBC project, some people would forget to teardown environments after a pull request was done. Sometimes we didn't want this to be automatic, like you merge the branch and automatically that triggers the job to destroy the testing environment because sometimes you merge it and maybe the client wants to still play a little bit with that testing environment, so we created a job that checks, like, hey, is this pull request closed? Is there a testing environment? Yes, it's older than a week, okay, then trigger the job and teardown. Also we have Casper JS tests on the MSNBC project, so after you get the message saying, this is the testing environment and here is how you can teardown, there is another message being posted in, like, happy face, hey, old test pass, or the guy of Doom dying, saying, I'm sorry, you know, test fail and you get the full log of all the assertions. In the Lullabot com website, we use in resemble.js to make sure that we don't have visual regressions, we realize, for example, that we deploy a new release and there was like a 20 pixels padding, like, at the header and it's really hard to spot that whenever you are quickly reviewing a release before deploying it, so there is a job there that takes a few screenshots and shows you, like, visual changes on it, so it's really helpful to make sure that you don't introduce any visual regressions. Kind of what Drush does, J, Drush ULI, well, it's a job that posts the login link, sometimes a client doesn't want to open the testing environment and enter credentials, so you just post a link there, they click and they are logged in as in the administrator in the testing environment and the create spare database, it was for the MSNBC project because the database was so big that creating a testing environment was like half an hour process, something like that and we optimized this project by, like, kind of in advance doing a database clone with a suffix, so if you trigger a testing environment it would use that spare database and obviously it will be just two minutes. If during the next 30 minutes someone else would trigger a testing environment, it will have to wait until the next spare database is ready but anyway, for the amount of people we were in the team, it was good enough. What's next, I am maintaining the system I'm explaining now, it's an open source project, it's, I think it's on the session description at the end of the slides but some of the folks at Lollabot are working on an actual product on top of this and they are rewriting it, it's not open source yet but it will, it's called a tag boat, there is a URL here and there is a newsletter that you can sign up if you want to get news on it and here is like a ballpark of what it's been going on there so far, it's using Node.js instead of a mix of bash scripts and drush which is what the current system uses. It doesn't really need to use GitHub, like there are some folks that for example don't use GitHub but they use Unfaddle in their projects for example and well the system will post comments on Unfaddle instead of GitHub, it would be just a different plugin let's say. It uses Vagrant to spin a new environment which is great, I haven't looked into that piece of code yet but it obviously makes things way simpler and faster because obviously the team can also use those Vagrant environments and everything is more standardized whenever you are testing stuff and yeah as I said for the moment it's closed source but the idea is to open it, I cannot say when but as soon as possible obviously. So what to do now, that's the open project that you can have a look at and test it. I am the one maintaining this. The problem with this repository is that it assumes that you are going to use Jenkins straight away so the scripts expect some Jenkins variables to be available so if you want to set up your local with it you will have to install Jenkins, prepare a sample Drupal project and set it up. In the last days I've been just simplifying this so it just uses Drush and just does the build job that's it, integrating with GitHub would be something external like an extra plugin but I haven't released a piece of code yet. And as for the other one, as I said, just subscribe to this newsletter if you want to be up to date in news. Announcement for Friday, if you've never been to spring I really encourage you to do this a lot of fun and you can be next with really great developers and learn, that's going to happen on Friday. Oops, another announcement, Drupal Camp Spain, that's going to be next year in May. We had a really nice Drupal Camp last year, I think we were like around almost 300 people. There was a full track in English, there were quite a few non-Spanish speaking folks who came so we are really looking forward to bring more people from outside Spain and make it kind of a more, little bit more international event and it's happening in Carreth in May, right, yep. So we're gonna go on for questions now. If you have a question, please go to the mic there so it's recorded, sorry? By hand you mean, right, so the question is, do you create new content for a particular testing environment? I would say if you create a new feature, do you mean a new feature as working feature for the client, right? Yeah, we didn't have the scenario where we had to create a huge amount of content. Normally, we would use the master data, I mean the production database and if we had to add some nodes, we would just add them ourselves. Like, we never had to have a bunch of content created for a particular testing environment. If we had to, we would just use maybe devil generate and run the command there. At the end, it's a server that is under our control so anybody from the team can just jump into the server and run any commands they like. So at the end it's code that you maintain so is it as if it was your local environment? Right, any more questions? Yeah, we started, we downloaded from production onto the testing server, the files and we use a stage file proxy module. That saves us from having to copy all of the files onto other testing environments. For anybody who has never used this module, the stage file proxy, what it does is like you define kind of a master server where Drupal is gonna try to look for image styles. So whenever you open a testing environment and it has an image within the files directory, it will see if it's available in the local server and if it's not, it will just use the master server that you say. So that saves you from having to copy the images all over, so that's how we do it. There are also some modules that make them handy to install at the end of the build process. For example, if you just don't want any sorts of email to be sent, you would install maybe email readout. If emails are important for you, you may want to lock them into a file and inspect them later, it depends on your scenario. Obviously sometimes in the testing environments we also enable UI admin modules like views UI, field UI, so you can have them available. Yeah, that's pretty much it. Any more questions? Yeah? Yeah. Right, so the question is, how do we test the tutorial workflow changes? And what we do, especially in MSNBC where everything needs content, we have a batch of Casper JS tests. We're using a country module called Casper JS and that it has a bit of integration with Drupal so you can manage sessions. You can do stuff like, okay, so as an editor, I want to do this and then as an anonymous user I want to verify that this happens. So that's the example I mentioned before that first the environment is built and you get the link to the environment and then another message is posted once this extra job has finished performing Casper JS tests and it will tell you like, hey, this works or not. And we have, like we discuss with the tutorial team like what's your workflow normally and we just wrote tests for that. So it will tell us if we broke something on the tutorial environment in the tutorial process because obviously for this project it was very important to keep it stable. It's another job that runs, the Casper JS module has, it's at rush command and it says, well, run all the tests that are located in this folder of the project against that testing environment and it will just browse the environment and perform all the assertions. Any more questions? Right? This slide, oh, do you mean this one? Yeah, this is a sprint, like this is like everybody in Drupal gonna ask. Oh, to this. Now I'm gonna, I think I'm gonna be around and I'm gonna be working on that rush version I mentioned. Actually if anybody wants to show up to me to show them, I'll just show it. It's like, it's a command that clones and then there is the virtual virtual house where you can see it. So, yeah, just look for me. I think I'll be, I think I'm gonna be doing the morning in the sprint. Any more questions? No, so far the biggest three weeks, the biggest project where we used it was, is the MSNBC project. We realized that we had to scale up the server whenever we were 15, 18 developers and all of, most of them creating testing environments. Actually, what we did before, what we did first thing to improve that was that, well, maybe not every pull request needs a testing environment. Sometimes it was just a CSS change, for example, or a typo in a statement, in a PHP statement. So we made it, we made it manual. Same as we were doing an IRC JDEL to delete the testing environment, we will have JTest to spin up a testing environment. So that obviously reduced the amounts of testing environments we had. But we didn't have many issues. Like, yeah, I mean, it's just a matter of waiting because we just set the Jenkins to just run this build process once. So if five people trigger a testing environment, it will just queue up and you will wait. At the end is testing, so normally you as a developer will just, you're done with your work when you're building a testing environment. So you would just trigger a testing environment, move on to something else. Once the testing environment is ready, you will get a notification through GitHub. And if you need to perform any actions for the peer reviewer to have a look, then you will do it. So it's asynchronous, right? So the question is how you keep the master database fresh with production data. We run drash SQL sync and drash rsync files from production onto the testing environment once a day. It's a two lines job in Jenkins. Yeah, obviously what we also take care in there is to sanitize. Like we run normal sanitization for user names and passwords. We also raise some data that is not needed because we don't have to have such a big database. So there is a little process there that does some cleanup of the database so it's more usable. Yeah, do you think it would be, well, but the thing is that we would be stressed in production because we would be asking my SQL to create a database dump every, oh, you mean asking it to the slave to do it instead of, that would definitely be faster. But so far we've been fine doing it this way. I mean, as I said, the major issues we had with performance was that a lot of people were creating a lot of testing environments and with that two gigabytes database that scales up pretty, I mean, you're in trouble very quickly. But apart from that, I mean, nobody, everybody is fine about just leaving the build you're running and moving on to something else. It's fine. Anything else? Okay, thank you very much.