 Howdy. I'm John, and I'm going to present the 2021 Workflow Update Talk with the long-winded title there, and link for it is there. I think a lot about why Galaxy. And this interaction on Twitter a few years ago really stuck out to me. Devin elegantly describes why technically savvy people still use Galaxy. It isn't always about themselves, often it is, but it's also to enable others. It allows them to provide tools, depends on scientists to allow them to do their own analysis. As he says, Galaxy scales, my time does not. And I really love that concept. I think part of what makes Galaxy great is it allows people with all different sorts of skills to support research in different ways and empower others, bringing specialists together to solve problems. Admins can deploy infrastructure. Developers can write tools, bound for magicians, can become plugin developers, and write tool wrappers, trainings, workflows, et cetera, to enable better scientists. In this sort of model of Galaxy, I'm going to focus in on this particular interaction, the interaction between the method developer and the pipeline executor, and on the workflows that connect them. So Galaxy is good for method developers because it basically enables whole new classes of people to write workflows and become method developers. They can do experimentation, optimization, documentation, all without needing to be comfortable with command lines. For the pipeline executor, likewise understanding command lines and Q sub and cloud infrastructure might be daunting, and you don't need to with Galaxy. And then the outputs of workflows are enhanced by embedded visualizations, et cetera. But for these people, Galaxy is limited in many ways. For the method developer, many of these people are very technically savvy and prefer command line-driven tools like Snakemake. For the pipeline executor, maybe Galaxy exposes too many options, histories, upload options, et cetera. And workflows aren't really front and center in the UI and traditionally exposed a lot of extra options that aren't really needed. And so it could be a little daunting. So over the last two years, we've made bold strides to address these shortcomings. For the methods developer, we've sort of incentivized making best practice workflows and made them more powerful. The workflows themselves are more expressive and scalable, and for the pipeline executor, we've really simplified finding and executing these workflows. In the 2019 conference, Marius and I presented simple workflow parameters to help parameterize workflows in a structured way that is geared towards reuse and rich annotation. These have come a long way in the last two years and they're now a wide variety of types. They can be optional, have default values, text parameters can be restricted by connected values or values supplied by the workflow author. Regular data inputs can be extended, have been extended to likewise allow them to be optional or specify data types explicitly. This all makes workflows a lot more expressive and robust. Another way we've incentivized best practices is by allowing annotation of important metadata on tools and workflows. Galaxy now exposes license and author information to encourage giving credit to the plugin authors. This is exposed in the Galaxy DOM for tools and workflows as micro data and schema.org semantic data objects, metadata objects. Author information can link to many different kinds of resources, but special support for ORCID is provided, a sort of long-run question feature. License metadata is derived from SPDX and workflow authors can edit all of this right from inside the workflow editor. In the last two years, the expressiveness, documentation and tooling and debugging and reporting output for workflow tests have all dramatically improved. Check out the Plenimo docs for more information. Switching hats quickly, we have a Galaxy Works customer that needed workflow testing last year, set up on a bunch of different workflows that were all sort of existing and diverse. These were older style workflows and they didn't leverage input parameters, output labels, et cetera, things that are required for workflow testing. Also, they wanted to keep the workflows in Galaxy rather than leveraging GitHub and sort of typical Plenimo paradigms. So to address these use cases, we made the workflow best practice panel to rapidly transform workflows into best practice workflows. We'll show a little video here from Helena about this, but we've added this best practice panel. It's an opportunity to encourage the annotation of author and license declarations here, where sort of the, you know, encouraging, you know, it was missing an annotation, it was missing creator details, they're specified here. The best practice panel also encourages the use of typed and annotated inputs, as we've been talking about, and it allows migration from older style parameters to these modern ones. Here we're seeing that the creator can be people or organizations have all sorts of metadata associated with it. The best practice panel also encourages the use of output labels. Yeah, and these properly annotated inputs and outputs are needed for workflow tests. They vastly improve Plenimo tooling around running workflows, linting, et cetera. Part of the workflow testing was to enable a cluster migration for this customer also. So we developed tooling for testing and validating all the tool tests in a workflow. So to give the green light on running the individual tool units before the workflow integration. It's freely available on GitHub and PyPy. Traditionally, the workflow editor is just replacing the entire state of the workflow with each update. No information on what changed was tracked. We tracked the versions, but not how versions changed is another way to think about it. To power this best practices panel, we implemented a whole new API backed by a fully typed and structured pedantic descriptions of atomic changes to workflows that are persistent with each save. This infrastructure sets the stage for many cool things beyond just the best practice panel. For instance, the new refactoring API powers the ability to upgrade sub workflow steps with the click of the button from the parent workflow, as is shown here. This along with the ability to navigate into sub workflows from the parent makes sub workflows a lot more usable than they were even just a year ago. The refactoring API also powers really cool features being developed in Galaxy and Palima by Simon Bray for auto upgrading workflows to the latest tool and sub workflow versions. This mirrors work being done to keep tool wrappers up to date and bioconda recipes updated as the underlying tools are updated. So these are all things we're doing to sort of incentivize the methods developers. Now we'll jump up to the workflows where we've improved reporting and scaling. When we debuted data set collections, the point was to scale workflows and collections could allow processing a few dozen elements at a time. By the 2016 GCC, we were seeing sort of presentations talking about hundreds of items being processed at a time. And by the 2017 GCC, with a bunch of optimizations dealing with 3000 items as possible. By the 2019 GCC, we presented that another order of magnitude was possible. And in June of 2021, the current known record in production is a workflow mapping over 190,000 data sets. In my previous position, one of the last things I did was help set up a clinical Galaxy application where we really had these different stakeholders and we sort of allowed the method developer to create a workflow. The pipeline executor would be in the lab, would execute it and then hand off the results to the clinician all from within the Galaxy UI. We had to develop a lot of specialized training and custom documentation, et cetera, to sort of give the clinician some sort of context because Galaxy wasn't providing a lot of support here. I think part of the problem with that was that histories suffer from this lack of context. I think I've used this slide before. We do all these things to provide context for the big list of things in the history. Toolform enhancements, tags, collections, et cetera. But ultimately, even after this huge history, right, Galaxy's histories are still just a big list of, a big flat list of things that grows very quickly. Now imagine this isn't even a history that you created. You didn't even run the black box. The output of a workflow is just a bunch of data sets and then there's not a lot of context there. To address this, we've added the concept of an invocation report. It ties together the outputs of a workflow. We presented this briefly at the 2019 GCC as like a beta thing that we were adding. So we probably have seen these slides before. We had a nice markdown editor. We had syntax for inventing Galaxy objects in the markdown. But over the last two years, we've taken this beta thing and we've really invested a lot in it. And it's a lot better than it was before. One of the first things we did is we took those markdown-based reports and we allowed them to be used within Galaxy as another new backend for pages that are a lot more robust, sort of a lot more intuitive. And then we extended that to allow you to publish an invocation report as a standalone page, sort of decoupling it from the workflow. We allowed PDF export of reports so we could sort of make that handoff. We've been describing there very explicit. Sam went in and took these very rough concepts, these very rough UIs and polished them to make a really very nice final product. The reports look better than ever. He started with the components of the report itself but then also polished the editor. You used to have to understand the syntax and now you've got all these options on the side that each have their own UI for selecting things. So these complicated directives for embedding Galaxy objects into the markdown are easier than ever. Ulig has contributed a bunch of nice components for dealing with composite data types, digging into them, pulling images out, et cetera. Sam took all the visualizations in Galaxy and made them embeddable into the report. We'll see some of these new components here. Again, thanks to Helena for these videos. Here she is marking up some, specifying some output labels so that we can work with them in the reports. And we're sort of editing the report markdown here for a workflow. You can see we can specify the outputs. We can see the outputs. We can, where we're showing previews, the history data set peaks of those outputs here. We're gonna see job metrics and job parameters the same way. So this, yeah, all of these enhancements have come over the last couple of years. It's really exciting to see this component grow. All right. And now we're gonna show visualizations off. So on the side panel where we have all the different components, we now have all the different visualizations as well. You can select the output that needs to be visualized and you've got that same visualization, charts form available right there. So then when the workflow runs, you can see, you can visualize the outputs interactively. So that's things that we've done for the workflow. Finally, we'll sort of wrap up talking about the pipeline executor and how we've made finding and executing workflows easier. Talked about this lack of context problem. One thing that we've done to help improve it is invocation tracking. Before you couldn't see a running workflow and now you've got a representation for it in the UI. The admins and individual users can see all their running invocations. You can dig into it. We started by just adding the form for seeing the workflow itself and having the output of the workflow but also extended it to also have a drill down of it. So here we've got the workflow form. When we're running it, we can see the description of the workflow and now we can dig into the inputs, the outputs, the parameters and the steps. The steps include additional details like step inputs and outputs, job details. Data sets can be open and visualized right from the invocation. All of this is utilizing components from Mason's history, right? As you can see now, we've got a lot more context for the running workflow than we ever had before and that makes the workflows a lot more exciting. In the end of 2019, we got together a group of us and talked about how we could simplify the whole user interface around running the workflow as part of an effort to take some of the great features of Glaxio that's powered by the Galaxy API but move them into Galaxy Core. And we came up with a nice document that outlined things we wanted to see in the Galaxy interface. We wanted to simplify that workflow run form, hide unnecessary details, hide workflow steps, allow upload right from the workflow form and then figure out a way to take workflows and put them right in front of the user when they land on the Galaxy's homepage. One of these things that we did accomplish is that simplified workflow form. As you can see on the left, this is the older workflow form that has every step listed and the left looks a lot more and then the right is the new simplified form. If you've got a best practice workflow, it will automatically just show you the simplified form. It's a lot cleaner and it will load and run a lot faster and it uses the API in a better way because it's only sending the inputs in. So it's more structured, yeah. As part of this, we migrated that run form to Vue.js. I mentioned the, without having to load all the steps, the whole thing performs better, faster to render, faster to submit. We implemented uploading right from the form. So we took the upload component, we rewrote it in Vue and then we allowed it to be launched right from that tool form. And yeah, in theory, we'll let you hide the concept of data, hide the concepts of tools in the history from people who just wanna run the run workflows. A nice thing about uploading right from the form itself is that you can sort of tailor that upload component to the data types that are expected for that workflow input. So we can restrict the dropdown of possible data types. We don't need to display the composite data type tab if it's not applicable for that input. We don't need to display all the collection creation stuff if it's not valid for that component, for that particular input to the workflow. So we've enabled running workflows in a much more simplified form. And now we need to work on the landing, that sort of initial introduction to Galaxy Experience for those executors. And this is a work in progress. But we've got some work to do and we've got it tracked down and hopefully we'll get that done soon. In terms of finding workflows, we've sort of added integration with GA4, GH tool registry servers, so Doc, Store, and Workflow Hub. Maurice is gonna talk a lot about that in his IWC talk. Check that out for more information. And I just wanna say thanks to the whole Galaxy community for building awesome workflows and building awesome workflow integrations. Thanks so much.