 Nice everyone feel it feels probably better than they would at 9 a.m. So we'll take that Welcome we're here to talk about using the Robo Task Runner to improve developer workflows I'm Mark Dorson. I'm the CTO at chromatic Chromatic has been working in the Drupal community for over 15 years We work with all kinds of clients that have lots of content do lots of complicated migrations and Work on sites that have often really Complicated tooling needs and that's part of where the motivation for some of this comes from But hopefully you'll see Some inspiration for you know when you see some of the examples we go through on ways that you can improve tooling on your sites even in Simple ways start up for the warning There's going to be a number of code slides, but the good news is that they're purely to show concepts You don't need to copy anything down All but a few of the lines that you'll see are available in a public repo that will mention And if there are slides where it's clearly a lot of code just hold your breath for a second and I'm going to zoom in and You know to specific areas of that so don't panic. Oh And also We'll definitely have time for Q&A at the end. So if you have questions as we go through I'm happy to I'm excited to take questions at the end So what's the problem? If you're like me you like to automate things You've worked on a lot of projects that get festooned with bash scripts They start showing up everywhere and Bash is great. Lots people love bash, but just not for me. It's not my favorite I use it in situations where I don't have another runtime available And it's a great fallback, but in my experience I I'd rather work In the primary language of the project. So we're here at Drupal con So if we're talking about a Drupal project written in PHP, I want to write My tasks and scripts in PHP If it's JavaScript project, I want to write it in JavaScript I want whoever the developers are in that project the language that they're working with I want them to feel comfortable Also working with the tooling and automation pieces as well so We need a task runner in this case. We need a PHP task runner and we're gonna We're gonna use Robo. So what is Robo? Robo describes itself as a modern and simple PHP task runner inspired by gulp and rake Aim to automate common tasks So you might already be using Robo Without even realizing it if you use this tool, maybe you've heard of it Drush Drush is built on top of Robo Robo was first included in drush in 2016. It first shipped in drush 9.0 9.0 feels like it should have been recent, but it was seven years ago and Drush 12. I haven't here drush 12 is being prepped for release right now. I'm pretty sure They released it in their session yesterday. Yeah, so drush 12 is now out. We're on Can't believe that we're on drush 12. It's amazing a lot has changed in drush since 9.0 But Robo has been there for all of that. So the use cases we're gonna, you know explore and Think about look at automation Testing builds and deployments back at search stores the lists could go on and What is using Robo going to give us? It's gonna give us the HP package access It's gonna give us better readability in our code better error handling and A better developer and user experience. So, you know when people are running these tools from the command line It's easier to give them better output more informative output So there's a bunch of tools tasks that Robo provides out of the box. So these are things that You know, I've been sort of wrapped and abstracted There's file tasks easily work with files You can catnap replace File system tasks Copy directories things like that move directories Working with composer you can do Work with composer operations abstracted into Robo Things like composer update composer validate. We're gonna see some of that in a minute And you can work with version control systems like get so you can perform get operations Via Robo and last but not well not quite least or not quite last And then there's an exact wrapper which is sort of like this is your escape hatch anything that you can run from the command line You can run through exec. So if you didn't have let's say you didn't have a Get task and you wanted to work with get you could just exec running the exact tasks So anything that you're doing from the command line You can do it here and then now last but not least It abstracts like parallelization. So if you have specific tasks that are taking a long time you're working on servers that have Many processors available. This is fantastic in an easy way to paralyze tasks and Get great output It's a lot in my experience much simpler than using like the parallel command on the command line It abstracts away a lot of complexity So how would you set this up? Robo generally Start with composer require and a robo file one file In that robo file This is an example of Maybe the simplest command that you could create So the docs are great On this as far as all the tasks are concerned What we see here How does how does robo interpret this to determine this is a command? It's a public method any public method in your robo file or robo class is going to be interpreted as a command That you want to expose to the user you have helper methods Protected or private methods those will not be exposed It's going to parse the name of that method that method Into the command name. We'll see that on the next slide here. We have An argument For the year so that's going to be required since there's no default and Then we have the doc block above which is going to give the help text and then we've defined an alias so the main command name is going to be from the Hello Drupal con and we have an aliases of high now when we run robo just robo like, you know Without this command we want to see the list of commands that robo knows about This is what we're going to see So right down at the bottom we see that there's a hello namespace and hello colon Drupal con So that's parsed from the method name We see high in brackets, which is our alias and then we see our help text And then this is running the command robo. Hello Drupal con. We pass in our argument 2023 And we see our output. So this is the simplest implementation. You start putting commands in that robo file We just saw the simplest command that you could create but you could imagine That robo file you start with one command then you've got five commands Then you've got ten commands some are related. Maybe they're in the hello namespace Now you start having commands that are completely unrelated. You've got many namespaces so it started on the project I was working on it started to get a little unwieldly I was like This this file is a mess How do I find anything these things are really not related and we're working with PHP And this is not how we would really write a Drupal module We would be putting things into different classes in their areas of responsibility so we can do that auto loading to the rescue So now we're sort of at an intermediate step we started with the most simple implementation just a robo file but if we Tell composer where we want to store our robo commands In robo slash source We can break them out into separate classes So here's an example on a project where we've got 8-7 different classes that each contain Commands that are related to each other There's nothing special about you know, this is just how I organize things But you can organize them the way you want to but this gives you the ability to do that so This was our intermediate step and then we got to the point where we built a lot of great commands and Chromatix and agency we have a lot of clients, but we wanted to use some of these commands Across projects. We wanted to be able to roll out improvements And have those picked up, you know in short order without copying and pasting code You know across these projects and this was sort of the you know original Part of the original dream of like well, why you know move on from bash for these things We wanted to be able to You know keep things in sync when we improved one of our deployment Commands, we didn't want to have to then keep up with that across all the projects manually. So we wanted to build a package and We call it our package usher So you can use usher usher's open source Or you can build your own package But as you saw since we're using auto loading where you know, we're working in PHP We can use composer to pull in a package so So that's what we did and on our projects instead of running require robo we're gonna require usher so Three approaches we just looked at you could start with a robo file You could use your commands within the project break them up into classes Or you could share commands across projects by putting them into their own Own repository and sharing them Be a package us or a private package repository So now we're gonna look at a few examples of Commands that we've created That will give us hopefully some inspiration for what we could do So the first one is Validating Drupal configuration So we build every pull requests in a preview environment using a tool called tugboat shout out to tugboat if you don't know the tugboat folks they're great and And Chugboat is also available on Drupal.org. So if you are a maintainer of a contrib project or a heavy Contributor to a contributor project you can enable tugboat for free on your Drupal.org project All you have to do is add a tugboat config file And it's fantastic any merge request that's created for the contrib project It will get a preview environment built which is a live Site with a database everything you can click into it and actually test your changes So this is fantastic for our clients Especially our clients that are less technical We ask them to review a pull request and they look at the code and they're like sure not a good thumbs up But what they really want to see is how that change shows up on the website and tugboat lets us do that So when we build this environment We get to see our full deployment process happen, but we also want to check A number of things but one of them is that the Drupal configuration that's in the database Matches what's in disk so if there's some kind of change that hasn't been exported Or some change that fails to import correctly. We want to know about that and not ship it We want to find out about it before We merge that code So the way we're able to do this Josh has a command called config status so We want to run this after we do the deployment and we want to mark the pull request as failed So this is what one of our pull requests might look like and the important bit is Right there in this case The config Validation has passed, but if it hasn't we want to mark that as red. It's a required check So you'd be unable to merge the pull request If that has if that's occurred, so what do we need to be able to do this? We're building a preview and tugboat We need to run that drush command that operation with the right The right arguments the right options and then we need to talk to the github API to mark Mark this specific check as pass or fail So this is what the bash script looked like before I started and this is actually Like an intermediate Step because if you look down at the bottom there actually this actually is running a robo command And don't worry you don't really need to read all this but But what you do need to know is that all this could all this was just the piece to talk to the github API We're scaping strings. We're calling out to curl. It's gross And the the moment that I opened this file and approached it was the moment when this was working for One check like this, but I wanted to implement it for a second check and to do that Easily at least my level of bash expertise. I was going to you know I was looking at I'm like, I'm gonna need to copy and paste this into another bash command for that separate And that was a huge red flag to me. That was the moment when I was like, okay, I've put this off long enough This is the moment where we're gonna put this into php into into robo into usher in fact So this is where it started so we have our class the validate config Commands there's a trait the github status trait more in that in a minute that has all the logic for Setting the github status and this is part of what we get by working in php we can use We can use tools like traits to abstract that logic and then use it across a number of different classes Told you I wanted to reuse it so we got that and we've got constants so we can do things Nicely like that and another benefit to working in php for this stuff is now you get all the benefits of static analysis So if you're running php stand on your code You know, it's gonna catch things. Hey, you said you were returning a string from this method And you could potentially be returning something else. It'll it'll fail that if you're running So things like that that you're not getting With your bash scripts catch a lot of issues before we ship them with tooling like that So now we have our command that Robo is going to expose now. You can see that the dock block is a little more Full feature than the one we looked at before But the important bits to to point out or you see the That we have a public function and the name validate Drupal config we've got a site direct siteters Argument that has a default and then we have an options array which is our flags for the command And we've documented all that above so that that's up top in the dock block is what Robo interprets along with our alias But we also get to use things in php like I mentioned like we have a return type on the method And we go from there So now now we're actually looking at the trait now. This is where we're going to look at the code That actually replaced That big piece of bash So we've abstracted out the setting of github status You can see that we're giving some user feedback using the yell And the say commands that are built into Robo But we're able to break these into separate methods as well again great for static analysis So If we're suddenly introducing a typo, you know, that's gonna show up if we're calling a method that doesn't exist That's gonna show up and then here we have Our method that actually talks to The API and in this slide we see that we're setting everything up we're using environment variables to get the appropriate data We're setting the URL that we're gonna call to and then setting the body of the request And then we get to use guzzle to make the request instead of curl so this is an example of how we're able to rely on other php packages and Require them with composer pull them in and make use of those Instead of making some assumptions about what's installed on the system or what might not be in the case of bash So we're making that request it's a post Passing in the body and the headers and then we're handling the exception of what occurs That's how we set that value on the PR So this is a bit of a different example This one is an example. I think that seems kind of potentially Too easy at first but built up over time And I found it to be something that we ran into the real world so For us we've ended up with a lot of yaml configuration for different environments on our site. So we use Lando Often for local development environments, maybe use Lando or D dev or doxel So you in there along with a lot of other configuration You're often gonna have a string with your php version same thing on tugboat, which we talked about before Somewhere in there sometimes multiple places. You're going to have your php version Github workflows where we run all of our github actions workflows where we run a lot of our tests that don't need a database things that need a bit database we've run in tugboat, but other stuff we run in github actions and Same for production hosting if you use Platform as a service like pantheon or platform.sh You've got a yaml configuration with a php version string in there so We found that over time when we came time to upgrade php Someone would go through and create a PR with those changes. They change 7.4 to 8.0 and Open up a PR and Maybe they'd ask for me to review the PR and I'd look at it and say Hey, you've changed four files All the strings went from 7.4 to 8.0. That looks good to me and all the tests pass give it a thumbs up but Then there'd be like oh wait There was one we missed I can't review something that's not in the PR and I mean it would be if I was If I never made any mistakes, maybe I would remember a wait There should be six files that are changed, but there's only five But that's not me so I missed it and it goes out and then maybe weeks later we find out We've been running We're on 8.0 in all these places, but and we think we're on 8.0 everywhere But really we're still on 7.4 in this one environment. We didn't know that so I Created a command That's going to handle all that for us. Don't read this. This is the entire command a fair bit of this code is user feedback to the To the command line so let me pull that out So this is the meat of the command without the user feedback which the user feedback is very valuable Don't get me wrong, but just for this discussion today. We're going to look at this part so These are the pieces up at the top We see the function signature Config update PHP version is our command name We have a required argument Version which is a string and then we have one flag For skip composer update. I should add one thing One of the ways the important concepts about how we're able to share these commands across projects effectively is configuration yaml so There's a robo.yaml file that you can create and put any kind of configuration you wanted So what we do is write our commands With the idea in mind that this is going to be shared across a bunch of sites and any of the pieces That could be different That we would otherwise may be hard code We set those as values in that configuration yaml. So Here We're going to that configuration yaml and we're getting the current PHP version So we've told it the version that we want when we call our command So in this example, I think we're going to upgrade to eight point one And so we're looking for the current PHP version. Maybe it's seven point four Next up we're going to load the files that we want to check We're not going to do a find and replace over the entire Code base there are probably places in the code base where the string seven point four Exists in context that are not about PHP. I checked believe it or not it shows up in a lot of Like sass files or and things like that. So we don't want to do that. So we want to Limit our problem space. So we have on a per project basis developers in the project can say What are the files that this type of configuration is stored in so if you have one project that? Uses platform.sh You can list those type of files. If you have another project that use pantheon you can list those files At in that configuration array Same thing for if you use Lando or D dev those are going to be in different places. So We're going to load the configuration that we can that we want to check Same thing with a read me. Maybe you want to maybe it's mentioned in your read me These are the places that I forget to check when we make those changes so That's what that looks like in the configuration yaml. So we've got our PHP current version eight point one So we're starting there and Then our pads that we want to Check composer Lando tugboat platform GitHub workflow and the read me Then we're going to replace our version strings robo makes this very simple This is one of those tasks that I mentioned at the beginning replace in file. You give it the from the two and you run it if Composer.json is in that array of files We also want to make sure that our lock file is valid So if you didn't tell it to skip this step, which we provided as an option to the user They can choose to skip this Again something that working in PHP makes it very straightforward. I don't know how I would write this in bash Somebody here probably does but I don't We can skip that easily But because we didn't skip it We're going to run a composer update with the lock option, which is going to make sure that we don't update any of our Dependencies We're telling it which directory That we're that we want this command to run in So composer update dash dash lock in in a certain directory That's what's going to happen with this and then the last line We run composer validate to make sure that it comes back as green and valid and this is what it looks like When you run it So the current pre-HB version is 8.1. We've specified 8.2 is the version. We want to move to It checks composer.json in this case I did tell it to skip composer validate just to limit the output for the slide and then we can see all the other places that that Where we replace this So now in our documentation We can tell the developers on that project when you when it comes time to update PHP version All you have to do is run this one command. You don't have to remember all the different places You don't have to rely on a list that may or may not have been updated now It does rely on the config gamble. So, you know, if we start adding Places where that string is we'd need to update that but this Gives the opportunity for more developers on the project to feel confident in doing these types of operations And this is what it looks like If it doesn't go well, so if we were able to easily check again easily in PHP Hey, is what we're at with the current version the same as the version you've specified We're gonna give some some nice output to make it really clear why we're why the command is failing. So third example One of our projects is a Drupal 9 soon to be Drupal 10 Code base that runs many different sites in I think it's it's north of 20 now and growing and Essentially in the code base level. It's a multi site doesn't run that way in production every all the production sites are completely Siloed on their own infrastructure, but when we're creating things locally, that's how we you know tell Drupal what's going on So when we add a new site, there's a lot of files and directories that need to be created and updated some of those were Files that we mentioned earlier like those YAML files, but also Drupal settings files and We started with documentation and our read me is quite large on this on this project So again, this is a place where The read me is great and valuable and our dot we want our documentation to be great But the documentation can get stale too and it's kind of hard to test documentation if we created a command that would do these steps then we could actually run this command In our CI environments and make sure that it executes successfully So this command we're gonna update our Lando config update tugboat config get hub workflows update Drupal sites configuration with the latest versions of this command we even You know do things like create keys for two-factor authentication So again, that's stuff that started as in the read me a bunch of commands for a developer to run locally You know run this command take the output put it in a file move the file to this path But Now we can do that With a command and allow more developers to feel confident in this type of process We've got error-checking We've got our steps that I just highlighted, you know calling out to helper methods Lando config sites files tugboat config get hub workflows and tests and Settings files, so it's running all of this is kind of an uber command that runs a bunch of Other operations So the one that I'm going to call it and show is That tugboat config file so we talked about tugboat before So here's our helper method This is a protected method so it is not being exposed as a robo command on its own But as you saw two slides ago, we're calling it from our command Here we get to work with Drupal's built-in YAML class, so we don't need to deal just with that file as a as a string what we can do is load it and Decode it and that's going to put it into an array for us that we can work with and PHP and then here we're Essentially editing that yaml, but right now it's an array. We're going to create new items with the site name that we passed in We're going to add commands At the right place in the Maria DB container and the init method We had a create database command And then we're going to write it back to disk and give some user output so this is what Again, there's a I took a few things out to make it fit slightly better on this slide but just to give you a sense of The number of things that this command is doing and I would emphasize too This isn't where we started, you know We started with it doing one thing or two things and then it grew over time because it was effective and People were getting value out of it. So here we run a command at the top site create Drupal association Production URI we pass that in and then it runs the scaffold command Updates the config files another great thing with robo easy to ask for user input So you can say hey, do you want to do x or do you yes or no? This case we said no so that changed what was being written into some of our config files Make making creating directories moving files around Down at the bottom we move our example settings dot PHP file into the right Location that now exists and we do some string replacement with the site name so it has the right values and Then we even create some two-factor keys as I mentioned and put those in the right place And then give more user output for like what the next steps are hey You can now run a create theme command after this And it'll do similar type operations to create the sub theme for this So we've used robo to create to replace bash scripts We I showed how we can We started with a robo file and then move the robo file into classes So, you know, we can benefit from some of the organizational benefits of working in PHP We talked about how you could share these commands across projects if you have a lot of projects or even just more than one project where you'd like to share some of this functionality and Then we looked at examples for config validation PHP version changes and site creation This is just a little bit of Promotion there's a lot of stuff going on with Drupal 7 end of life If you know anyone if you or you know anyone that still is responsible for Drupal 7 sites We've been talking about all the implications of that on the Drupal 7 end of life podcast It's been we've had some really interesting interviews with Tim Lennon Matt glam in who just released a tool This week that will allow you to run Drupal 7 modules in Drupal 10 And the idea behind that being to ease the transition We know it's a big burden to move from Drupal 7 to Drupal to modern Drupal 910 And this will help that transition with the idea of being like if we can cut that Lift by 40% that's going to make a big difference for organizations that have a lot of complex code So that he released yesterday of another Drupal con release. So that's pretty cool I have stickers here for this if anyone's interested but check it out And that's it. Thank you all very much If I if I don't remember to repeat the question, please remind me. Yes Yeah, sure That's that's easy. I can do that Yeah No that file Thank you, I know thank you very much. So the gentleman asked for The usher package contains a number of commands that are you know able to be pulled into downstream projects you asked The configuration file that lists things out like the PHP version or the array of files that we want to operate on He asked if that file also lives in the usher package The simple version the simple answer your question is no that file lives on the individual projects So, you know, let's say you have ten Sites ten projects that want to use usher each one of those has a robo YAML configuration file so that you can configure those commands to run differently on each one So one site might be on PHP 7.4 Currently and so in you know and has a certain set of YAML files that you want to work with Another site might already be on 8.1 or 8.2. And so those files will differ site to site the slightly more complicated answer your question is all of what I just said and Robo has its own one because or usher has its own Configuration file because sometimes some of these commands will run we want them to run on that package too So we use usher to work on usher in addition to working on on these other projects. Does that make sense? Yeah, depends on the command so some commands I Don't have an example Up the top of my head, but some commands Maybe you don't need Anything in the configuration file and it'll you know you and you require usher and then You you run robo it'll list out the commands. You see the commands and you just run it Other commands that need some configuration like the PHP version one's a simple example If you require usher and run that command without setting up the configuration YAML the command will fail It'll say I don't know what your current PHP version is. I looked in this Config file and it wasn't there. So then you say, oh, I forgot to do that stuff. Let me create the config file And in usher I'm pretty confident. It's up to date. We have like an example Robo dot YAML configuration that has sample values for those that you would then copy into your project Great question. Thank you What else yeah Well one recent one was that I Started writing a command to work with something with our integration tests Thank you The question was Have we hit any pain points? In working in this way an example of a pain point would be I started writing a command to work with our integration tests About halfway through I was I was showing it to a colleague and he reminded me that our integration tests running in a container That does not have PHP In it. It's a it runs in job. It has a JavaScript runtime. No JS. So I was like, oh Well, it's got the whole repo there, you know, the whole repositories checked out in that container But I actually don't have PHP installed so At that point I alluded to this earlier like well, we could use bash that would you know That's available many more places and in that case would probably be there Or in that case it would be like well, we could use a JavaScript Task runner and there was already some precedent for that on the project. So that's what I ended up doing So that is the one thing you'd have to be especially in projects where which is not uncommon on Drupal projects We have a whole front-end ecosystem that there's a lot of JavaScript If you're breaking that up into places, you know with containers that where PHP is not always present, you know That's something you should watch out for You're welcome What else one more anybody last call All right. Thank you all very much