 The title of this talk is Developer Workflow, Automation, and You. My name is Bradley Jacobs, and I'm here to talk about Developer Workflow and Automation. Work for a small company called Boston Globe Media, I am a lead developer and I specifically oversee all WordPress development at the company. I've been working with WordPress since before themes were introduced in Strayhorn, which was version 1.5. That translates to about 13 years of experience working with WordPress in some form or another. We have a lot of sites at the Boston Globe and nearly all of them run on WordPress. In fact, the only site that isn't powered by WordPress is BostonGlobe.com. The largest of the remaining sites in regards to both traffic and database size is Boston.com. As a result, we have a lot of automation set up around these sites to maintain consistency across them and across teams. Now, let's get back to the topic in hand. This image is a perfect representation about how I feel whenever I create a new repo and compare it to pretty much any other repo that isn't mine. Honestly, I think a lot of people can relate to me on that. The title of this talk is Developer Workflow, Automation, and You, but what it really should be is Empathy in Team Development. It's extremely important that we have an understanding of how we work on our code basis. It's not enough to just understand branching strategy and get or subversion. We need to have an understanding of the entire process. That means from the time a task is assigned, through the moment those changes are released to production. We need to have empathy when building our automation. We have to ask, what are tasks that are tedious and repetitive? How can we reduce the number of common mistakes? How do we communicate clearly when a step is complete or if an error has occurred? We need to consider all of these things, but add no additional steps in the workflow or we defeat the purpose of what we're setting out to do. Now, I'm not here to talk about theory or tools. I really enjoy those talks, but what I find really valuable is real-life examples of how this stuff is actually used. I'm presenting here today is the actual workflow for all development changes on Boston.com and by association all of our other WordPress sites. Before I get into the workflow, let me talk a little bit about the technology stack we're using. We use JIRA for all our projects and task management. Use Jenkins for all of our automation, CRON tasks, Fabricator for code reviews, and we are still using subversion for version control. For those unfamiliar with it, I want to take a moment and talk a little bit more about Jenkins. It is essentially software that can run custom tasks based on time intervals, event triggers, or just manually kicking a job off. Time intervals are useful for things like importing content on a regular rotation or clearing expired transits from a database and creating nightly database backups. Event triggers are something we make use of a lot. As you'll see, we have the ability to run a Jenkins job when code is checked into subversion or GitHub, when the status of a JIRA ticket changes, or when a code review is approved or rejected. We even have the ability to trigger a Jenkins job when a WordPress post is published. We use that to do a cache purge for that post, the home page, and certain section fronts. Of course, you can mainly kick off a Jenkins job task at any time. This is useful for infrequent tasks that need to be very repeatable. We're creating a Jenkins job right now that will copy a subversion repo and copy it to GitHub, maintaining all of its history. This isn't a frequent task, but it's something with a lot of tedious commands that will want to run multiple times before a final cut over, and we'll need to run that for many different code repositories. There are other alternatives to Jenkins that don't require you to host your own Jenkins server. Travis CI and Circles CI will not exactly the same offer some similar functionality that you could use for these types of tasks. But let's get back to developer workflow. Now, at the globe, we thought about our workflow in the context of our technology staff. What do all of the pieces of the process flow through? The answer for us was the ever-present JIRA ticket. All information regarding a task should be included within the JIRA ticket for that task. As one of my coworkers once said, it's not official until it's JIRA official. This makes it the perfect place to put clear communication about our automation. But more on that in a bit. JIRA tickets also have explicit statuses that indicate the progress of our task. These statuses make for great event indicators as to when our automation will need to run. We use the Agile plugin for JIRA, and I believe that's what gives us this nice hand-dan-like dashboard. You can see each column represents one of our statuses to do in progress, code review, in QA, ready for deploy, and done, or in production. I'm going to walk you through some of the status changes and how our workflow and automation work together. Move a ticket to do, to do, to, when we move a ticket from to do to in progress, we opted to have no automation at this step. There's a strong temptation to have Jenkins just automatically create a new branch in subversion. But in the end, we decided against this. Automation should be applied universally. And in this instance, there are plenty of tickets that get created that require no code changes. It might be an investigation ticket or a ticket to write up some documentation on a feature. It's vital that our automation adds no additional steps to the workflow or it'll defeat its own purpose. As a result, adding any step to mark this as a ticket that would need a new code branch violates that rule. So, branch creation is done manually as needed. When we make our first commit to the branch, Jenkins sees that. It takes that branch and creates a new web environment that we refer to as the test site. Jenkins then updates the JIRA ticket with the repository name, the branch, and the URL to that test site. When a developer feels their task is feature complete and tested in their local environment, they can manually move that ticket into the code review status. Without automation, this would be when a developer would go to the command line and generate a code differential per review. This is a tedious task requiring very specific syntax and is very error prone. So, we have Jenkins set to see the ticket status move to code review. This event triggers a code differential to be generated automatically. It'll use all the commits with the ticket ID and the commit message. This is what the code diff looks like. Old code on the left, new code on the right. We are adding a new file here for this ticket, so there's no old code to display it on the left. A reviewer can add inline comments to give feedback, then approve or request changes at the very bottom of this page. Once Jenkins creates the code diff, it'll update our JIRA ticket with a link to the code diff in question. This keeps our theme, this keeps with our theme of putting all information for a task within the JIRA ticket. When a code review has changes requested, it gets sent back to the to-do status for the developer to pick back up and address the feedback. When a code review is accepted, Jenkins automatically updates the JIRA ticket to move into the NQA status. Once again, we'll look at the JIRA ticket. We see here that the link to the code review now has a strike through. This concisely indicates the code review was completed. Below, we see the link was added from our first commit. That link goes to the test site we mentioned earlier. This gets used heavily during our QA process. You've all seen what a WordPress site looks like, so I'll refrain from actually following through on that link. But we are fortunate enough to have a few dedicated QA analysts that do manual QA. We don't have much automated testing set up at the moment. But if we did, it would have run before we got to this step. Before we let the ticket enter the QA phase. When a ticket passes QA, the status is manually updated to ready for deploy. Jenkins looks for this status change, and it does a couple of interesting things. First, it attempts to merge our code branch into trunk. If that merge is successful, it deletes the branch. Our trunk branch is a representation of what's in prod. So what happens after that? How do we track releasing the code once it's in trunk? Dashboards, dashboards, dashboards. The base for this dashboard is built during one of our regular internal hack days. It has seen a few iterations since then, but at the medical ants, we see all of our sites within one screen. We know exactly which ones have code waiting to deploy to the next environment. We drill into one of those sites, let's say Boston.com. We see all the information pertaining to releases for this site. There's a lot of information on this dashboard. Each environment not only gives us a status of how many tickets or builds are pending, but it also tells us what version of WordPress the environment is running. Now, I'm sure I'll get plenty of questions about this after the talk, but this is important. The latest version of WordPress gets pulled into every deploy to staging. So when a new version of core is released, we need to be able to track it as it gets released from staging to production. We don't want a surprise upgrade due to a deployment. This does at times force us to freeze deployments until the new version can be vetted and tested. Why would we do that? For a site our size, we feel strongly that being on the latest version with all of the latest security updates should outweigh any justification for theme or plug-in changes to override that. Has this caused frustration from time to time? Absolutely. Do we revisit this decision on occasion? Every WordPress release. Do we have any regrets? No. We have stood by this decision since we launched on WordPress and no one has made a compelling enough argument for us to change this process. We'll see how Gutenberg goes in 5.0. Next to our WordPress core versions, we have some handy links to the relevant Jenkins jobs for each environment. You'll note that the Jenkins link for our broad environment is actually a link to create a new Jira. Releases for production are actually handled by our operations team. So we need to create a Jira ticket for them. Even this link is something we tweaked and enhanced by using the Jira API to pre-populate the ticket with all the necessary information for the release. Here's a look at that pre-populated Jira ticket. The tier, summary, and release name are all filled in along with some other fields further below. The nice thing here is that it includes the build ID, which is that long number in the release name field. Before this, we had a few instances where copy and paste didn't work out so hot, and the wrong build got released. As with all parts of this process, we saw a small area of improvement and iterated, and that's important to know. Because while this seems like a large comprehensive workflow, we did this piece by piece. We saw an opportunity for change, something to improve, and we implemented that one at a time. Getting back to our dashboard here, below our environment section, we see all the builds waiting for deployment. A build is just the code changes associated with the Jira ticket. If a code change doesn't show up on the dashboard automatically after moving a ticket to ready for deploy, then we know something went wrong. And that usually means a merge conflict while emerging to Trump. This is actually an area of improvement for us. We need to make failures more visible. A build just not showing up in the dashboard isn't a very good notification that something went wrong. We should have a better, excuse me, notice that gives some information as to what went wrong. So, what comes next for us? We have an overall departmental goal of streamlining our tech staff. We use too many technologies for everyone to easily maintain and work on. We also aren't on industry standard technology, like Git. We're still on Subversion, not to knock Subversion, but most developers now really only know Git and primarily use GitHub. So, a goal for us is to migrate to GitHub. Not only will that put us in line with what new developers know, but it also integrates code reviews, which allows us to get rid of fabricator and use a service that we're already paying for. Linting and testing is another area of improvement, is running linters on their local machines. But we don't have anything checking when code is committed to the repo. We also don't have much in the way of a unit test suite. We do have a pretty good regression test suite. But once Gutenberg comes out, we're gonna have to rewrite that. Because it's keyed off of the admin. Finally, dashboards. We have some great dashboards right now that feed us a lot of information. But there's always room for improvement. When something fails, we should see a little notice or something along those lines without having to dig into the console output of a Jenkins job to understand exactly what happened. If you wanna try your hand at setting up some automation, these are some great technologies that you can consider checking out. For continuous integration, we're using Jenkins, but that's a self-hosted solution that can do a lot more than just continuous integration. CircleCI, Travis, offer hosted solutions as a service. They also have plenty of documentation. There are lots of opportunities for static analysis as well. In addition to any linting you may already be using, you can also run checks against coding standards. Be it the WordPress coding standards or the PSR standards. And version compatibility, that repo right there allows you to check to see if there are any red flags in your code for specific PHP versions. Which is extremely helpful if you plan on upgrading your PHP from say 5.6 to 7.1, 7.2. For code reviews, you shouldn't need to worry about that step if you're using GitHub. But fabricator is a great option. It's an open source solution that has honestly worked well for us for the past several years. And of course, dashboards, dashboards, dashboards. The more you know about the status of your tasks in the workflow, the quicker you can act to get them to done. So what are some common themes among all of the things we've seen here? Automation should be universal. It should be something that happens every time the workflow is followed. It shouldn't add any additional steps for anybody. The whole point of this is to reduce the number of things we need to worry about. If something goes wrong, we should know about that faster. And finally, we need to remind ourselves that this is just a snapshot in time. We should be adding automation a bit at a time. We want to make sure it works the way we want and helps in the way we need. We will constantly be looking to improve the process in the workflow. It's never okay to say, well, that's just how the automation is set up. We have to bear in mind that the automation is there to serve us. We are not there to serve the automation. So before I end things, I just want to give a shout out to the organizers for putting this conference together. And the volunteers who took the time out of their weekend to help make this weekend what it is. Shout out to the stenographers who not only keep up with the speed of our speech, but also the weird technical jargon we're just throwing out without even realizing it. And with that, I ask everyone to go take what you've learned this weekend and code something newsworthy. With that said, I have a ton of time for questions, I'm pretty sure. So I'm happy to take those if there are any. Sounds great. Thank you so much. I hope you've enjoyed your work. Oh, question. I'm sorry, I can't hear what you're saying. I haven't posted these just yet. I'll tweet it. Twitter is here. I will try and tweet that out today or today. No problem. Thanks.