 Hi, everyone. Welcome to today's NFCore bite-size talk. My name is Phil Huells, and today I will be talking to you about the NFCore modules patch functionality. This is a sort of very simple functionality, so I'm thinking today's bite-size talk will be very short. Stop sharing what I'm talking. But it's also, many people don't know whether it exists, and I think it could be quite useful, especially for people using NFCore tooling and the NFCore templates or pipelines, which are kind of private or custom pipelines, which are not going to be part of the main NFCore organization. This as well, this tooling really, really kind of shines. So yeah, if you want to use the NFCore template for stuff you're doing in-house, this talk is for you. I don't have any slides or anything. It's just going to be kind of a live demo. I'm going to walk through how I use it and try and describe what it's doing in the back end, and hopefully that will make sense to you all. We'll see how it goes. Those of you who have seen me talk before will know that I love a good live demo. Things usually go wrong, but that's kind of the fun of it. So let's join me on this, on this one, I'll ghost it. So just before I kind of kick off a little bit of background information, what are we talking about here? So for those of you familiar with this, apologies, but just to get everyone up to the same level. With NFCore, we have a pipeline template for the whole pipeline. And then in the last kind of year or two, with DSL2, we've been working with modules, and these are wrappers around specific tools. So this isn't on pipeline level. On pipeline level, there's one workflow all the way through, start to end analysis, and a module is just a single tool. And we have shared modules, which people can collaborate on, which you can install into a pipeline. Then when you make changes to a module, which is a centralized module, those changes can be easily integrated into every pipeline, but it's using that module. So the classic example and the one I'm going to be working with today would be FastQC, QC tool for DNA sequencing data used by, I don't know how many pipelines within NFCore, but very many. We've been chatting on Slack yesterday and today about some updates. There's a new version of multi-QC that's come out, and it's got some new options like minus, minus memory, minus, minus SVG and stuff. And we've been talking about those updates, and we can just do that in one pull request, one discussion on one module, and then all the pipelines can just pull in those changes across the board and get that new functionality, which is fantastic. So pipeline's modules. In order to make all of this system work, it's really important that the code within the pipelines, the modules within the pipeline is the same as the code in the central module's repository. That makes sense. If you want to synchronize or two, you need to kind of keep them, keep them tightly together. What that means, though, is you can't change the code in your pipeline. If you do that, the NFCore code linter will start complaining and tell you that you've not allowed to do that. So what people have done before then is you take the centralised NFCore module and you just copy it as a local module, and then you can do whatever you want with it. You can change it and the linter won't complain. The downside of doing that is you're no longer in sync. So if there are updates that come into a centralised module, you won't see them, you won't be able to pull them in, and you're sort of effectively losing that collaborative aspect, which is so powerful. So this is where NFCore module's patch comes in as a sort of a stopgap, if you like, a way for you to make changes to central modules in your pipeline and your pipeline alone whilst keeping that link, while keeping the linter happy, while keeping all the functionality about updating modules and so on. Hopefully that makes sense. If you want to ask me any questions at this point, shout, otherwise we can take questions at the end. Otherwise, I will dive into a screen share and let's show you how this thing works. So hopefully you can now all see my setup. I'll make the zoom tool as small as possible. So my patch pipeline is the NFCore methyl seek pipeline. It's one of the first ones I ever wrote, and it's one I'm still fairly involved with the maintenance for. So hopefully everyone is familiar with the idea of NFCore lint command, which runs over code tests on every single module and all parts of the pipeline. Today, for live demo purposes, I'm going to do modules lint, which just only lints the modules and not the entire pipeline. And I'm actually going to make it just the fast QC module. So things run nice and quickly. So if I run linting, make it a bit bigger, you can see that basically everything's fine. My pipeline's up to date with the central dev version of methyl seek. And I've got a couple of warnings about this module, but there's a new version of software available. And there's a new version of the central module available. But they're both warnings, they're not failures. So that's my kind of starting point. Right, we were talking about new fast QC functionality. So this is VS code. I'm looking at the methyl seek pipeline source code now. This is not the central method modules repo. This is my pipeline. So if I go into modules, I've got the local ones, and I've got NF core, scroll down, you can see I've got the fast QC one. And this is the shared fast QC module. Now, I could make changes and drop into local, but I'm not going to do that today. So let's say that things are moving too slowly. I want to do something here myself. What I can do is drop something custom in here. So let's say, well, SVG, I'm going to have a new input channel to handle these SVGs. And I want to do it just to my pipeline. And I'm going to hit save. So, you know, assume that I'm doing some valid change here. And I've tested locally in the next flow side of things that's working fine. And it's doing what I want. Now, if I rerun this linting test, it's going to be unhappy with me. Because this lint test checks the version on the web on the NF core modules repository, looks at the code there and checks the code that I have locally. And in this case, it says these, this code does not match the version of the module that you say you have. And so that's the hard failure. Obviously, I over continuous integration tests on GitHub will start giving a red cross and failing. And this is, this is not a good situation. So this is normally where you freak out and copy it to locals. But now I'm going to do some magic. So now I do NF core modules patch. I run this command to ask me which module I want to do it for the fast QC. That's where I've made my changes. And it just very quickly spits out some content. First things first, this is a diff. So this is where it's looked at the remote file and the local file for any changes. And you can see it's picked up here that there's some code in my local chain copy, which has changed. So this looks right. This is what I just added. So it says there's an extra line here. Now these diff files are really cool because with diff, you can, you can generate these kind of diff files or patch files, and you can apply them on top of other files. So we can reapply this change at any time. And that's what we do. We save this diff. And if I go to get status, you can see I've got changes to my modules.json file, which is used by NF core to track the synchronization between my local pipeline and the shared modules repository. I've got the changes in the fast QC file, which is the thing I just edited and saved. And I've got a new file here called fast QC.diff. I go back into the VS code, we can see that this diff file is just what was printed to the console here. And it's just saying alongside the fast QC module, hey, I've got some local changes here. And then if I go into modules.schema, you can see if I find fast QC that we've got a new line that's been added in the JSON file here. And it's just telling NF core that there's a patch file that exists and in this location. Okay, so great. What does that do? I can add all of this. Let's make a new branch. So now if I go back to the lint commands, NF core modules lint, which was failing, we're back to our starting position. Everything's fine. Everything's happy. Now, just to explain what's going on here in the backgrounds, I've still got those local changes. But in the background, when I do NF core lint, NF core code is fetching the remote version of NF core modules repository. But then it knows I have a patch file that diff file. And so it stores that local copy it's got from web, it applies the patch file on top, and then it compares. And that's why there are no changes. If I make some more changes in here again. So val some food, then that's not going to be in the patch file. And it's going to fail. Or in fact, it did a hard failure where it just couldn't even figure out what was going on. If I do it in a different place here, path, food, then it will just fail again and say that say that something has changed. And then I could run NF core modules patched again, it will update that diff file. So but now there's new changes are covered by the by the by the diff, and everything will work again. Hopefully that makes sense. And what's cool is it's not just linting. This applies to the same process works when I update pipelines, modules. So we've got this little warning that there's a new version of this module available. So I can also do NF core modules update. Let's just do first QC. And hopefully, yes, there we go. It has updated the module for me. So it's gone server NF core remote with a shared one. And it's updated my local copy. And then it's managed to still reapply the patch file on top of the new updated version and save that there. So NF core updates still works, which is like kind of magic, I think. And we can see these are all the new changes that come in. So there's a new get shot for this module of latest version when you can see the changes that happened when I updated the module. Right, hopefully that's all we need to know everything works beautifully. But I thought for completeness, I would also show you one small complication of when things could go wrong. We got a hint of it a second ago, actually. Something that could happen is if we find the fact if I do reset one, so just go back to when we first made the patch before we updated. Now I can add a different change here. So now I'm going to add minus, minus SVG on to the fast QC command itself. And if you can see it's the same NF core modules. Lins fast QC, you can see it's the same. This is all the same. And of course, modules. Patch fast QC. Yes, regenerate the patch. Okay, so now our patch file has got two changes here. That's good. And NF core modules lins fast QC. Okay, so it's a second change, but everything is exactly the same so far. Now the tricky bit comes now by go to NF core modules updates fast QC, like just like I did before, it will fail. And now what's happened here, I've got a couple of warning messages saying it's failed to apply the patch. You will have to apply the patch manually. So basically, this is a little bit like when you are working on code and you put in a pull request and you get a merge conflict. There have been changes that have happened on the central NF core modules repository. And there have been my local changes, which I've done with those patch files. And the tool can't automatically figure out how to reconcile those two changes. So basically what it's done is it's just clobbered my local changes. If I go in here, you can see it's made the updates, but I've lost all my custom changes, and it's just overwritten it with with what was on the remote copy. So all I have to do is have to go back in and I have to just recreate this patch. So I can go back in and go one minus minus SVG and then run SVG and then rerun NF core modules patch. That's fine. So just spare that in mind. Sometimes when you do updates, and you have local patches, you might need to do a little bit of fiddling, just be careful with always running and get commit before you do stuff, because then you don't lose anything and you can easily see which which changes are happening. Right. That's my live demo. Hopefully, everybody followed along with that and it made sense. Yes, sorry. Van says in the comments that when I said earlier about shouting, you can shout it literally if you want to, but I also meant just you can you can you can ask. So happy to take any questions and hopefully this will be be useful for some of you. Yeah, so you can unmute yourself now. If you want to, to ask questions. Lots of happy people. Very few questions. It's quite a nice neat little small topic to discuss. So I had plenty of time for sure. But what what happens if you realize that you actually have more changes than you actually want to include? Like, if there's some things that make the whole pipeline fail or whatever, can you undo the patch? When when you do the update, you mean? No, when you you have written something you get errors and it's like, ah, but this is because I did something manual that I want to be different. So you do a patch and then you realize oops, actually, that was something completely different and messed everything up. Then it's no difference if you made changes in any other way. You just you just go back and you you look at your get changes and you revert and get whatever. This is kind of this part of the workflow is specific to just a enough core tooling, just delincing the updating. So it's kind of coming in at the end once you've already fiddled around and made next flow in your pipeline work properly in the way you want. Cool. Then thank you very much. And also thank you everyone for listening. And as usual, I would like to thank the Jen Zuckerberg initiative for funding our bite size talks. And hello also to Maxime.