 Hello, everyone, and welcome to today's Bitesize talk. I'm very happy to have here today Judith from the Max Planck Institute for Evolutionary Anthropology, and she is going to talk about how to use NFCOR conflicts in custom pipelines. Off to you. Thank you. Well, again, hi, everybody. My name is Rui Bayestero. I'm a PhD student at the Max Planck Institute for Evolutionary Anthropology. Here in Leipzig, Germany. And I would like to thank the space. I will be presenting today how to use these NFCOR conflicts in your own non-NF pipeline. And I have this necessity of running previous pipelines that I have already, or to automatize little tasks in the cluster that I currently am working. So what I'm about to present is what it worked at the end, and I hope it can be as useful as it has been for me. So over the next few minutes, I will guide you through this topic with the following agenda. First, what are the NFCOR conflicts, why to use them, and a step-by-step example in a non-NFCOR pipeline. Finally, a recap from all. So let's start. What are the NFCOR conflicts? Well, there are simple text files that describes the properties of your computational environment, which is basically resource allocation or managing hardware and software resources to run processes in a computing environment. And by hardware resources, I mean CPUs, memory and disk storage, and with software resources, the labor data models or pipelines need to exist. And additionally, these conflicts include scheduler configurations, which dictate the order in which tasks are executed and how resources are assigned. They can also limit the resources that a user consumes to prevent resource monopolization. So these files might appear basic, but you can see here, they are quite powerful for the pipeline to work. And here in this slide, these are all the documented profiles or institutional conflicts. So if you are lucky enough, there is one already for the institute in which you are working. In my case, for example, Eva is the one that belongs to the Max Bank, and you can access the link here to have an updated overview of them. But don't worry if you don't have or you didn't see your institution in the last slide, you can also make yours and contribute to the NFCOR community. I simply follow in this tutorial in grid inversion, following these QR codes or in grid inversion. And why to use these NFCOR conflicts or what are the benefits of using them? Well, there are several reasons of considering integrating this. First, they save time because they are already with their resources and settings predefined, saving you manually specifying job parameters or consulting notes on how to submit jobs to the cluster. But still you can customize your parameters to align with your specific needs. You can automate repetitive tasks by creating mini pipelines for specific tasks or steps, saving valuable time. And also with these configs, you can adapt your workflows and scale them as needed. Whether you're working with small data sets for testing or large analysis, they are flexible to allocate resource efficiently. And how does this work? Well, to include them, we need at least three steps. The first is to set the basic resources and you can set them either for the entire pipeline or for an individual process. Then load the institutional profile repository and check the, add a check max function to ensure that your task resource requests to the maximum limits defined in the institutional profile. And finally, run it. So let's go through an example to make this clear. And for illustrate this, I took the first script example from the next flow documentation to show that these are the needs to start from a complex pipeline to work with. And in this example, I organized the script into modular directories for each process with a main NF file containing the workflow, which is something that I would make in reality. And we would run it with additional parameters on our own computer with this extraction. Next load run main NF and the parameter for this example, which is a string. And open execution, you'll see that it indicates that it is running locally. So now let's apply these three steps to integrate NF core complex. And first, we may have already assigned basic resources for all the processes of our pipeline. And if you not have them assigned yet, this is the time you can set them for all the pipeline or for each process. And here, for example, I assigned CPUs and memory and time in a configuration file that I named base.config within a com directory. And then we need to incorporate that config file in the general nextload.config. But the real magic happens when we load the institutional profiling repository with these five lines of code. And within we're basically adding all the existing instructions who work properly in each of the environments or clusters, institutions, and also specify adding custom parameters from the profile. And you can see here what a chunk of the file looks like. So this link loads all the profile and each profile loads each copy. And even better, if your institutional profile is not there or maybe you misspell it, you will have an informative error, but it is not there. And finally, you need to add a checkmax function to the nextload.config and all these lines of code that you are seeing or indeed other resources that we ask for or task do not exceed the maximum limits that are defined in the institutional profile. And I think that this is good for new people because we maybe do not remember which are those limits and this can prevent that it crashes. And if you exceed these limits, you will again get an informative message about which type of resource you're exceeding. It can be memory, CPUs or time. And it will still run by using the default values. And now our structure will be slightly different for this example. We optionally added a base.config and the profiles in the nextload.config. And when we run the pipeline, you need to choose the corresponding institution and as a result, now the executor and switch from local to your cluster environment. In my case, for example, I'm using the EVA officer which uses SG to send the jobs. So in summary, you can use institutional profiles from NFCore with a non-NFCore pipeline by defining the basic resources, load the resources to the nextload.config, and load the NFCore institutional profile repository.config with custom parameters and adding this checkmax function. And remember that if your institution is enlisted among the existing profiles, you can consider creating one and contributing to the NFCore community. So this default not only benefits you but also your colleagues. So, thank you. Thank you for the space to present this brief topic here. Especially thanks to James for helping me to grow this tutorial and apply it to my own work. And you can find the good and the steps in the link of this QR code and the little example in the description. Now I'm happy to answer any questions you may have. Thank you very much, Judith. So, do we have any questions from the audience? There seems to be no questions. I was wondering how long did you need in order to get your way around all these config files for Nextflow? Was it a problem or was it very straightforward and clear? It was not that clear. That's why we make some drives and at the end we say, okay, maybe this is useful not only for me but for other people. And do you think there is something that we could improve in terms of documentation for those config files or is it more that it's just very complicated in itself? I think you're doing great in documentation and also with these tutorials and mini talks. Okay, thank you. Are there any more questions? If not, then I would like to thank you very much and it was a good talk, very good talk. It will help a lot of people I'm sure. And I of course also thank the audience for listening in and the John Zuckerberg for funding these bite-sized talks. Thank you all very much. Thank you.