 Thank you very much. Yeah, so also welcome from my side to the bite-sized talk and after Harshal introduced us last week to modules and DSL2 in general. I want to try and give a brief overview about how you can now make use of the modules in a pipeline. Sorry. Okay, so to get started, I just want to briefly recap what a module is. So it's an atomic process that cannot be reduced any further, and it usually contains like a single software tool like Fast2C, for example, and it can be used within a pipeline and also shared between different pipelines. And to make use of the sharing, there's the NFCore modules repository on GitHub, where you can find many of these modules already. So to make use of this modules repository, there's a new or already released, but there's an existing sub-command in the NFCore tools. Just a brief recap, you can install it with Kipp and Condor. And if you then run NFCore modules, you get a list of sub-commands that you can use to interact with this repository. And if you then run one of the sub-commands, with minus, minus help, you get some instructions on how to use them. So I just want to, over the next couple of slides, briefly introduce you to some of the sub-commands that could be helpful for using modules in the pipeline. So I think one of the first thing you may want to do is try to find a module that you could use, and you can use the NFCore modules list for that, and it will just print out all the modules currently available. And they are subdivided in tool and subtool because many tools like SAM tools or BCF tools have these sub-commands that are then their own module, basically. And then other tools like FastQC that don't have any sub-commands, they are just a tool. Okay, so if you look through the list and you find a module that you like to install, you can run NFCore modules install, and then your pipeline directory and the tool name, and that will install this entire module for you without you having to do anything else. And it then looks like this. So on the left hand side, there is a completely fresh new pipeline that I created with the template, and then around NFCore modules install, and then on the green, you can see where this module ends up. So the modules then basically creates, there's a sub-folder called NFCore software, and there you can find the FastQC module and with three files, the functions and F, the main and F, and the meta.yaml. In the meta.yaml, you can find documentation like what type of input and output does this module take, who wrote it, what does it do, these sort of things. In the main NF, that's where the extra magic happens, where FastQC is run, and then in the functions and F, there are some helper functions that are needed. All right, so one of the, I guess the opposing step to that is how to remove a module that you don't want to use anymore or that you erroneously installed, and for that you run NFCore modules remove, and that removes this entire sub-folder that had FastQC with this function's main and meta file in it, and it's gone. And then you can, you have nothing, you don't have to do anything else there. All right, then sometimes the module will get updated, maybe the software was updated and this was propagated already to the module, and you would like to use this newest update. So how to make sure, first of all, that you have the newest version, and then how to update it. So to check this module, you can run the NFCore modules lint on the directory to check our modules, or on a specific module like FastQC, and it will check, among others, whether or not you have any, missed any changes. And then to update it, you currently have to remove it and then reinstall it, but in the future, Kevin is already working on creating an update sub-command that you can then use. Okay, and then last but not least, what do you do if the software you're looking for is not an NFCore modules, and you basically have two options. One is to add it to NFCore modules, and the other one is to create a local module. So to add it to NFCore modules, this is usually really helpful if some other people or some other pipelines will use the software as well. And if you're unsure about it, you can always check the issues. Maybe somebody has already started working on it or requested it, or you can ask in the module select channel, and Kevin will talk about how to actually add NFCore modules in his talk in a couple of minutes. And the other option is to create a local module. So this is useful for software that is really specific to your pipeline. And you can run NFCore modules create, which will create a local module for you. If you look in this box, you can see the NFCore modules folder that we already saw and we've seen before, and the local sub-folder in which your tools will then live on your process will live. And a lot of the things that Harshal and Kevin covered in recovery in both of their talks is relevant here, even if Kevin's talk, for example, is more tailored towards NFCore modules, but a lot of the functionality will be similar there. Okay, so this NFCore modules create gives you this file here with many to-do statements and little help messages. And then you can start filling it out and try to get your tool to run here and hopefully the to-do statements help you figure out what exactly you will need to do in each and every step. All right, so now that we have modules either local or NFCore modules, we want to start writing actual workflows and pipelines. And they are two different types of sub-workflows. So these are chains of multiple modules with some sort of higher-level functionality, like all the QC tools that will be run on FastQ files, and then actual workflows, which are end-to-end pipelines, and then DSL1. We've known them as like these large monolithic scripts. But in DSL2, this is the combinations of modules and sub-workflows. And this is really taking one input and then producing a final output. On the right-hand side, I visualized the file structure a little bit. This is the current file structure of the Viral Recon pipeline. And for modules we have local and NFCore ones. I showed you those before. And then for sub-workflows, it will be a similar structure. Just some sub-workflows will be relevant to many pipelines, such as QC. And then we have the workflows. And in this case, I think Harsha separated by the input types of data here. So for Illumina and Anupor data, there are different types of workflows, which will then be called from the main NF. And these workflows are consistent of sub-workflows than modules. Okay, so to use a module, you have to install the module or create a local one. And then the next step is to adapt the modules.config. And this is really where the sharing of tools on software makes it easy, because here we can actually share the same software, but we can specify for our specific pipeline which parameter should be set. So for mark duplicates here, it says exactly which parameter should be used. And there are a couple of different ones. I highlighted here the arcs one to give you an example. And then some modules you will see have an arcs two line. And this is because, for example, for mapping immediately, SAM tools is run afterwards within the same module. So with one with arcs, you can specify it for BWM, for example, and then with arcs two for SAM tools. Okay, then the next step, if this is only has to be done once, you have to include this modules.config in the next for config, and then at last include the module into a sub-workflow or into a workflow. This looks like this. In pink, you can see the include statement you include, this module BWMM, from the path where it lives, so modules and of course software and so on. And then you add these options, these params.BWMM options that were previously specified in the config, in the modules.config, so this is how you can pass them down. And then the sub-workflow that we have here, we have three scopes. So take main and emit, and then take, it specifies the input data as channels. So here we have reads and indexes or indices for my little test workflow. In the main, this is where the modules come to work. So we have the BWMM module that was included up top and the SAMtoolSort module that was included. We can run BWMM and then take even the BWMM output and directly run it into a SAMtool, put it into a SAMtoolSort. And last but not least, from the sub-workflows, you can emit named outputs to then do other things and other sub-workflows or modules. So here we can now name our output from SAMtoolSort out BAM and name it, I don't know, sorted BAM. And then up in the workflow, we can access this sorted BAM directly to do something with it. Okay, so last but not least, we have to combine the sub-workflows to workflows or sub-workflows or modules to workflows. So here we, in the header, we have to include statements for the workflows or the sub-workflows in the FASTQC. So from once from the modules and of course software and one from sub-workflows local. And once again, we have to add the parameters to pass them down. And I highlighted here again in yellow for you to be able to track this a little bit. And for the FASTQC module, we just need to specify the parameter for FASTQC. So modules and then in the brackets FASTQC. And but for the sub-workflow, you have to specify all options for all the tools in there that you would want to specify. So here we have BWM and SAMtoolSort. And this is exactly now this field that was originally specified in the modules.config. Okay, and then in the workflow and F here, I've just run the module FASTQC on my input reads that I got and then the sub-workflow on also the reads in the index. I get the sorted BAMs output. And then I can do some more steps with that. Okay, so and then I want to show you how it looks like to add the workflows to the main NF. And here I took the ViraRecon one that I mentioned earlier. So here you can see now that we have three different workflows that are actually possible. And they are apparently dependent on the input data. And the whole main NF now becomes really lean though. So there's a bit of header still here, but overall the entire main NF is less than 100 lines currently. And it makes it really easy to now track for your input data, which workflow is run. And then you only need to look at the SRA download workflow to really see what's happening. Okay, and then two more things I want to mention that are important. So you always will have to adapt the multi QC module somehow to customize it for your tools. So you will have this is similar to the DSI1 version, but you can collect all the metrics in the workflow script, and then pass it down to your multi QC module on the right hand side. You can see all the different input data. So this is one where you have to create a local module for and you can then collect this data from the, I don't know, fast QC module, for example, here. And last but not least, you will have to collect all the software versions in your workflow. So each module must emit a software version. Just because we need to track all the two versions, and you can collect them all by creating an empty channel and then mixing these versions in from the sub workflows, you can propagate them up by these named emit versions. So for BWM version, you can get the version from the module and then access it again in your workflow script as test sub workflow out BWM version, then run your local module in the workflow. All right. And with that, I'm already in the end. Don't forget to join us for the next hackathon. And if you have questions, we will have a Q&A session in the end, or ask them in bite size later on modules.