 Hello, everyone. Welcome to today's Bite Size. I'm Francesca Brunat, and I'm the host for today. With us is Edmund Miller, and he is going to give us his impressions of working with NF-Test. Okay. Good morning, everybody. I did not, I just wanted to, before we start, I did not write NF-Test. I've just been using it. It was actually written by Lucas Foreer and Sebastian Schoenner. I'm going to butcher that from IMED. Okay. So what is NF-Test? It is basically the ability to write unit tests for NextFlow workflows, which I looked at PhilStock is the second most requested feature in this year's NextFlow and NFCore community survey. So you can think of it as PyTest, but for NextFlow, if you're familiar with Python and unit tests, basically just a way to write either full pipeline tests or sub-workflow function tests and module tests. And we'll get into some of those. So what that kind of looks like. So if you haven't memorized the Hello NextFlow script, here's a quick reminder. We start saying all of these different greetings in different languages, and we pipe them through, and then we view it lastly. And so what that looks like on the NF-Test side is then we have a name of the test, this spoiler plate up here, and then we have a script, and we're actually calling, I believe, the remote script here. The name of the test on this line. And then we have what we expect, such as the success of it, how many tasks we're going to have, which I think is also important as you're changing things. And then we have several certs of like what's happening in the standard out of these greetings, for example. And that's kind of the power of what you can do with NF-Test here. And you'll let your imagination kind of run wild with that of all the things you can do with it from there. There's a little bit of background as to why we want this and like, if you're not sold yet on that. So, so far, our testing practices until now have been testing our pipelines with GitHub Actions, and then testing our modules with PyTest workflow that's running on GitHub Actions for those. With the pipelines and GitHub Actions, I've realized I was talking to you, Satish, as we were starting to play around with NF-Test, that CI is meant to run your test framework, not be your test framework. And I think that's kind of run true of all that it's doing really is it's just checking the workflow ran without failing. And that's a huge step up from not checking anything. But then that starts to create several problems as well, when we're doing that, such as a number of the test. Whenever that starts to go up, it starts to become more complicated to maintain. You start coming up with all these fancy matrices, things start breaking. It's hard to keep track of what's going on in the jobs per se. And I think Sarac as a example of that and all of the various tests that they had and and all of these and trying to maintain their huge pipeline. And so, running the test locally then and repeating those as well. In a fashion, you had to really translate from that workflow YAML and then convert it to a next flow command and then run those. And then if you want to run multiple tests, you have to do that a couple of times. It has also had a heavier reliance on CI runners as well for us to produce those rather than us using the CI test or the CI runners as a check and a settlement of an argument of whose computer it runs on correctly. So in our modules, we've been using PyTest workflow if you're familiar with that. I'm sure if you've contributed you've fought with PyTest workflow at some point. So running your tests should be as easy as running the pipeline. PyTest workflow however was not. And so we created an F core modules test. And that's a great utility and it's been a great way for people to test things locally and rerun them. It's a lot to maintain on our side and change and update and kind of keeps us from wanting to progress forward on those and we want to keep the infrastructure the same so we can focus on actually writing the next flow modules. There's a couple of problems though with PyTest workflow. As great as it's been, I've loved it and it's been a huge step up from nothing and being able to actually specify tests and lay them out. The issue is that we have no native support for next flow profiles which is kind of a real power of next flow. So we can't multiplex Docker singularity conda and yes, we've come up with several hacks to get around that in CI and locally but it's really difficult to reproduce locally and explain to new users of like, okay, well, next flow lets you do dash profile but then here you have to specify it before the workflow and like it's a lot of caveats for a new user and a new contributor. Whereas enough test allows you to just do dash profile just like next flow would. And so that kind of lets you, it's a lot easier for new users to say, okay, I want to test with Docker and you jump right off to that. There's not as many gotchas. And then the other issue that I really had with PyTest workflow was the pipeline output was tough to find locally. You had to go dig through temp, the directory changed every time. You can really go figure out where your output was for your next flow workflow. And so we wanted to convert PyTest workflow or convert all the CI to PyTest workflow and why were we putting that off across the board? We had done it in a few places and CERIC had done it in all of their tests as well and done lots of extensive work but it was a lot of manual work to get all the MD5 sums for the files that we wanted and the testing kind of stopped there of you can test for all the files and you can test for MD5 sums and you can do contains and other stuff as well with PyTest workflow but it was a lot of manual work to get all those there or we can then build out new infrastructure like we did for the next flow or the NFCOR modules and start to build out all that and create this box ourselves back in their corner as well for this rather than just kind of relying on a different framework to do that. And then the other big complaint that I had as well for PyTest workflow for pipelines was I could never get local modules working with the bin properly for the mocks that I wrote whereas with NFTest it just works out of the box. So after all of that, now a quick intro to NFTest so all that you need to do to get started is a simple curl which if you've installed NextFlow at this point you're pretty familiar with. It's also Java based and Groovy based on those and kind of runs with NextFlow and we're hoping to have the NFTest folks back on for a more in-depth technical talk on this. We just wanted to get started on it. So the next command that you'll run is NFTestInit and this just sets up the testing config for you and creates a testing directory and a default test config and you can play around with those. We haven't come up with the perfect best practice yet for what we want for the NFCore standard to be on that. And then the next thing that I think is really, really powerful is they have a generate command. So you can generate all of these tests for your pipeline workflows, processes and also really importantly that I haven't really gotten full like we haven't tested to the full ability is functions. Lastly, I think that'll really start to change things especially as we're starting to change to check checking our sample sheet with Groovy. For example, that might help us kind of specify those and walk some of that down. And then lastly, you just have to run NFTest after writing all of those tests. So that'll just generate all of the boilerplate that you need for basic smoke tests of all the processes, workflows, functions, et cetera. And so a quick little walkthrough of some code. So this is just a quick test that I wrote in Demultiplex and I don't think the highlighting is working yet. I don't know, there we go. And so what's really cool here is we can start to specify params. So if you've gone through and done this in your workflows and thrown it into GitHub actions and thrown this into Matrix, you know, you kind of start to feel limited after you get to like one or two or you start to really specify out like an entire test and it's just not very familiar. I believe you can parameterize these as well. So you can change your inputs and change various params for your workflow. And then this is just some boilerplate. So it boxes each workflow into its own separate execution directory, I believe. Phil and I have had some interesting findings with that. Again, you can check the size. Now this is the really powerful part here is this snapshot feature. And so what this does is it basically pulls in the MD5 sums for you of all of these files. So we have things like metrics that run manifest here. We have our fast queues that are zipped up and you can specify all these different paths. This actually isn't a perfect example even because you can just list a directory is what I found really powerful. So you can just kind of point it there and then it'll just go through all the files. And what that then looks like over here is then you get this snapshot file. And what this does is it just creates a quick little snapshot with the MD5 sums of them of all those files that you specified and you can do standard out, standard error, et cetera as well on those based on the workflow. But we found that to be a little hairy for right now on those, but as you can see, this is automated. And then whenever you rerun your test it checks for these MD5 sums and we'll tell you but the other thing is it's really easy when you're like, okay, well, I know I changed some stuff. I know these are gonna change. You just run it with the update snapshots flag and it updates all the snapshots for you automatically. And then you just commit this file and it will check again against them. And if we can go back this, and then for files that you can't MD5 sum you can also assert these files and check that they exist as well on them. And so that's one thing. And then there's lots of other powerful features as well. You can do regexes and then you can also check for, you can also start to build plugins for it such as like checking for FASTA files. I think there's plans for like a VCF checker as well and that kind of stuff. So I think that'll be really cool to see what comes out of that. Okay, so let's talk about some good testing practices as well as we start to adopt this and start to adopt more testing. I know we already have great testing practices in the NFCore tools. So there's things like test driven development and that's where you write your test first and then you write your code after. There's also acceptance driven test driven development and then so that's where you start to write your acceptance criteria first and then you write your tests and then you do your development. And there's behavior driven development and that's where you start to kind of talk to the team first and then you kind of go through those and start to write your tests and then you start to do your development after that. And then there's also pair programming which is kind of where you write the code. Someone else writes the tests. You come together and work on it together. And then lastly, just kidding, we're not gonna talk about any of those today. Those are all great and lots of great things that you should go read about. But I think as a community, we have more important things to focus on as well. So what are some realistic testing priorities? So we should probably first convert our CI tests to pipeline tests. I think those are great to reproduce locally and quickly. Next, we can then add more pipeline tests for various params and pathways that you might want to make sure don't break. Of like right now, I think most people are just testing that all their liners work and then hoping that everything downstream works. Whereas you might wanna check all of your skips, for example, and make sure that you get the right amount of processes and that certain things aren't getting ran. And then another great practice that I hope that we can adopt is to add tests when fixing bugs in PRs that prove that this bug is fixed. And we're not going to come back to this bug and checking for it in the future so that we don't end up recreating it. And then lastly, I think testing your local modules can go a long way. I think we have a lot of custom scripts that we could do some tests on and make sure that we're getting the expected output out of those to verify our scientific results. So the TLDR is let's avoid regressions and move forward with our tests. So a couple of us have been kind of like toying around with a lot of rollout playing could look like. This is just a quick update on this and is no way indicative of exactly what's gonna happen. So first we want to start with pipelines as a proving ground because I think that's where we have the most ground to gain on this as compared to being stuck with just GitHub actions. So it's aria and methyl seek and demultiplex if you're looking for some quick examples and you want to like implement this today and switch those over just that step one. And then we're gonna need to update the template. I've got a PR starter for that but we kind of need to settle on some like best practices or how we want to do things so that as you switch between NFCORP pipelines and contributing, things are familiar. And then we need to do some module infrastructure prep and figure out how we can convert those and run all these tests and what that's gonna look like. We're kind of waiting for them to come out tags for NF test and then we'll update the modules as they get changes hopefully on those. And that will kind of be a rollout as you make a change to a module you'll update the tests from Pytos workflow to NF test and then a final push to convert all the modules that didn't gain love at a hackathon possibly. And with that, I'll take any questions. Thank you very much. I am now allowing people to unmute themselves also to start their videos, really nice talk. Are there any questions in the audience? Don't see anything. Is there any logo for NF test? I don't think so. You should make an issue for it. No logo, that can't stand. Yes, no, definitely logo stickers. Yes, Marcel, you're right. I want stickers as well. Okay, but if there are no other questions then thank you again, Edmund. And I would also like to thank the Chandlerberg initiative as usual for funding these talks. And if you have any questions, you can always go to Slack, ask your questions there. I guess for this one, it would be in the general help channel if you have questions about NF test or about any kind of testing question. So thank you again.