 Thank you everybody for joining. This is a bit of a kind of last minute bite size that we've thrown into an empty slot that we had in the schedule and we thought we'd take advantage of that by telling you a little bit about what's happened in the past week or so with the NFCOR community and some kind of updates which affect us all. This has been a little bit of a last minute talk even by my standards. Apologies for it not being a very flashy slide deck but hopefully we can talk through the different points and please feel free to drop something into the chat if you have any questions or maybe Fran can relay any questions to me as I go along. This can be a bit of a kind of discussion bite size talk rather than just me presenting at everybody. Especially if there's anyone listening from the core team, please do stop me if I get anything wrong or miss anything important. So the reason we have some updates for September is something a bit out of the ordinary happened last week which is for the very first time the core team of NFCOR got together in person because of COVID and the lack of in-person events we've had recently. Many people in the core team had never met one another in real life although we spend a lot of time online chatting to each other. And also over the last couple of years, especially as the NFCOR community has grown, when we do have in-person events, hackathons and things, we're always so busy running the event that we don't really have any time to actually do any kind of core team work ourselves. So I kind of kicked this off this year for the first time that we'd have a little retreat just for the core team where we'd spend a few days getting to know one another and also trying to get through some work for the core team and make some decisions and really get into in-depth discussions about kind of community scale things which are very difficult to take over Slack and Zoom. I'm glad to say it went really, really well. We had a fantastic time. Everyone came to kind of hang out in over in Sweden. We got to explore our surroundings and we went on a nice walk in the forest and everything which is where this photo comes from and kind of played games in the garden and stuff like that and did also do a little bit of work. We managed to drop into SciLifeLab and use a couple of offices there to spend a couple of days really getting some work done. So we had our own kind of mini hackathon. So I looked over all the notes that we took over the days and kind of broke out these points which I'm going to talk over today. The first one's going to take up most of the time and I'll just kind of mention the others in passing. So the first thing is teams. So we started NF Core end of 2017, started 2018 and from very close to the start of that we had a core team which is again, it's not very good naming for NF Core core team but all of you will know who we are. We are quite visible and we have been kind of running the NF Core community and it's worked really, really well but the NF Core community is getting bigger and bigger and bigger and the core team has got a little bit bigger recently but it's only so far that we can kind of stretch ourselves and so they've been feeling for a little while now but it would be nice to formalize the community structure a little bit more, create some more teams and formalize responsibilities a little bit within those teams. So we spent a good chunk of a day talking through all of this and which teams we'd like and how they'll be organized and what they should do and by and large we try to structure around what is already happening within the community. So this means that it's pretty easy. We don't have to change very much. It should be no real surprises because this is basically how the community is already functioning but we're just kind of hoping to formalize things, make things a bit more transparent and a bit clearer and all of this will go up on the website pretty soon. We just haven't got quite yet. So this is an overview. You can see the core team is still kind of at the middle but we have a handful of new teams here which are on this plot. The one which is completely new is a new steering group and this is kind of the top level of the community if you like and the steering group won't do very much on a day-to-day basis but we'll be there kind of to look after things like finance. In the early days of NFCOR we didn't have any finance so we didn't really need to worry about that but with funding from channels like a Berg Initiative and other kind of actual personnel working on the project now full-time we do have some kind of higher level decisions to be made. So the steering group would kind of take charge of all the finance and all those large initiatives and think about some big picture planning. The core team is basically remaining unchanged, same people and but again we're going to just write down what we actually do. We're doing a day-to-day running for community making decisions and generally we make decisions by kind of committee. If anything's not clear we take a vote and what we've said now that we have a steering committee is that if ever the core team is split then anything, if there's no clear majority on the topic then we'll push that to the steering committee. I don't expect that to happen anytime soon. It's never happened in in the what four or five years that we've been running so but now we have a policy in case it does and the core team at the end of the day has administrative access to everything and another important task is they make final decisions on new pipeline requests and basically you know who should be within NFCOR and how the NFCOR pipeline community runs. So no big changes there. We wanted to clarify membership a little bit. Again this is basically how we've been doing it until now but we wanted to try and make that as transparent to the community as possible. So it's kind of a meritocracy. We basically invite people who have visibly been heavily involved in the community and if someone's interested in being part of the core team then they can join and we want to try and make it explicit that we will try and have as best representation as possible but to mirror the representation of our community. There's a new team here called Infrastructure. This one is kicking off with Julia and Matias basically because these two are now employed to work with NFCOR as of this year which is fantastic. So Julia is employed with the Translacoburg initiative money to work on NFCOR infrastructure code and Matias started recently at the SciLife Lab data centre to work on the same thing and because I've been historically involved in this a lot I'm kind of the leader of this team. So and this team will work purely on kind of the tools, code base on the websites and any kind of framework work around NFCOR and again we've been working on this stuff for a long time already so there's no big change here but now we have an official team. Outreach team has existed for a while but has been not super well organised and so we want to kind of take turn over a new leaf with outreach a little bit. We've set some new leads so Chris, Marcel and Fran and we will kind of go over the membership of the outreach team soon and check whether people want to be involved and who wants to do what and we're also setting explicit responsibilities here. We're hoping to pull back a lot at the moment the core team does a lot of the outreach work so we're trying to sort of separate those two teams a little bit more now. The safety team again has kind of existed for a while but has been not super obvious to find this information it's listed in the code of conduct and it's been sort of done basically on a completely voluntary basis so now it's still the same but now we have an official team which will be listed on the website. So Sabah, Michael and Chris have been doing a fantastic job with us kind of year or two and now we're kind of making it explicit about they are responsible for the code of conduct and they are kind of a go-to people if there's any ever any problems basically either events or on slack. A key difference with the safety team is that they skip the core team and they will go straight to the steering committee with any recommendations if action is needed. Another new team maintainers this is still kind of in flux at the moment but we're thinking because basically until now the NFCOR community is just everyone has been maintainers everyone has full access to everything everyone is expected to help out but it's not really the case in reality because many people are just not developing pipelines just using pipelines people involved to different degrees have different levels of experience within NFCOR so we're trying to add an extra kind of tier in here where there's going to be an explicit maintainers group which will be people not necessarily in the core team but who are heavily involved in maintenance work it'll be quite a big team and we want to try and do this so that we can scale with reviewing pull requests for example at the moment it's quite difficult to get a first release out of a new pipeline because we say that core team has to review that first release but there's not that many of us and there's lots of new pipelines so we're going to share this out a bit more if we can and basically share out some of this kind of maintenance task this would be really key for the NFCOR community and more information will go out soon we're going to come up with a list pretty soon of people we'd like to invite in the first round and we'll start rotating that list every six months and sort of see how we go and we'll see also exactly how we do this in the future but at the moment everybody has right access to every repository within NFCOR and we may change that and we may make it so that most people have read access to most repos and then the maintenance team basically adds people to the pipelines where access is required just to try and streamline everything a little bit more and make sure that accidents don't happen and for eagle-eyed of you may have spotted the word ambassadors noted a couple of times and the ambassadors team will sort of be a bigger extension of the outreach team and we have some kind of pretty cool stuff planned but there's a bit too much to write here and it's not very well settled yet so just stay tuned if that sounds interesting feel free to ping us any questions but otherwise to keep on the lookout and we'll be pushing some more information out about ambassadors soon right that's all the team stuff any questions before I move on or is everyone crystal clear sorry it's a bit of a dry talk I appreciate that but I gave everything the right to unmute themselves so if you have questions just shout out cool right well um hopefully everyone's happy with kind of a decision to have come to you like I say hopefully there shouldn't be any big surprises here because this is pretty much how we're already operating yeah Enrique says the teams will get slack handles to be easy to contact so we'll set up some infrastructure around these teams um some other things then we talked through the guidelines a little bit but NFCOR guidelines have been mostly untouched since the very start of NFCOR um 2018 um we've added a few bits over time and they were getting a bit kind of unwieldy the structure of the page like some stuff with bullet points some stuff with sections and things like this and some stuff was outdated and not really valid and some bits we thought were missing so um we've updated the the guidelines page a little bit underneath docs and then contributing and then guidelines and so now our overview page kind of lists all of them in one big list recommendations and requirements and each each requirement then has a dedicated page with a bit more kind of information uh just so that you can give space to dig into a bit more details about what we're talking about with the requirements more than just a bullet point and but also gives you an overview at the same time so hopefully this is this is helpful this is useful we point to this page a lot so and we're going to continue kind of linking to specific things when it comes up in discussion and if you're developing within NFCOR you should try and be aware of all the guidelines especially if you're developing a new pipeline so uh how do the guidelines influence existing pipelines the the new recommendations basically nothing's really changed um so there shouldn't be and if you if you've got an existing NFCOR pipeline um there shouldn't really be anything you need to worry about um we've made a couple of things a bit more explicit that we have previously been saying on Slack anyway um and mostly you've just fleshed out more detail about the reasoning like why does your pipeline name have to be lowercase without punctuation and why does uh does it work like this way all that way so it's mostly organization of the page and more detail um and yeah but any questions of course shout once you've had time to read through these and you know we're happy to take any questions and then update and modify as as appropriate um actually someone requested this I think this morning obviously uh was there anything written about uh guidelines for reviewing this is ties in with the new maintainers team as well that as we're asking more people to do reviewing um we need to kind of better standardize how reviews should be done we have some documentation already for the modules I believe about how pull requests should be reviewed and we're going to try and write up some more guidelines and some more help for this um and we'll probably do a bite-sized talk about it as well at some point um so this the aim of this is to standardize our reviewing process and also make it easier for new people to get into reviewing which is a big big part of NFCOR and can be very uh rewarding in itself and is really critical to the functioning of the community um test data we talked about quite a lot um so a couple of things are going to change one of the things we're bringing in is a new requirement that we want some more information about what test data is and we want that in a structured way um so we're going to have a new requirement pretty soon for each test data directory to have a yaml file um and that yaml file will have to have certain keys um so really just to make sure that we know what the data is where it came from how it was prepared and everything um because at the moment we're meant to have readme files but it's a bit kind of um hit and miss um having this will also allow us to add some continuous integration tests to be sure that those files are there and populated properly before a new test data is added which will again will make reviewing a bit easier um and then James has also written some new guidelines specifically for test data so again under docs contributing test data guidelines um so this is an entirely new page again spelling out what we had already been saying on slack in a informal way but now everything's listed here about if you're if you're generating new test data this is how you should do it um one of the things that came up several times as you might imagine is about maintenance um and how do we do this and when should we do it and i was very keen to try and make all of our plans kind of scalable sustainable not just like yeah we should do that and then forget about it because that's very easy to do um so one of the things we're going to try and start doing is do a bit of spring cleaning every year um James and Chris from the core team are going to take the lead on organizing this and we'll send out reminders and stuff but the hope is that basically everyone in the community can just kind of have a little think about this once a year and do a bit of tidying up um things like looking at pipelines to see if pipelines are being actively developed we have a handful of quite old pipelines which were maybe started at hackathon years ago and then sort of abandoned and and other pipelines which might know might no longer be maintained and maybe kind of getting out of date so things like looking for those and archiving them were appropriate and again we're going to write up some policies about how that process of archiving old pipelines should happen um and also just going through all the different pipelines and sort of checking for old branches which have been merged and can be deleted old pull requests where which have been superseded um and kind of are never going to emerge because it's already been done in a different way uh and kind of curating the issue list um checking for things which have already been implemented or duplicates and just just cleaning up all this stuff on github um this is a constant thing that we should be doing all the time uh but it's just a good time to go once a year and say have a real focus on it and try and have a community drive on it we tend to have hackathons in March so we're thinking it's kind of nice to do a little bit before then so that as we go into a hackathon we've got nicely curated lists of issues for people to work on and everything's fresh and ready to go um there's a lot of papers and published papers about NF call pipelines um so if you go to publications you can kind of see we've got a list of publications here that we collect for with different pipelines where appropriate um if you didn't know about this if you have a publication about a a pipeline um whether it's dedicated specifically for that pipeline or it just kind of describes the pipeline please do add it to this page because it's really helpful to have it here um and it's a bit of a difficult thing because with NF call we're a huge community project and especially when it comes to things like DSL2 modules um you know I could go and create a new pipeline today import all these modules that other people have written stitch them together and very quickly have a pipeline up and running which would be great um but it'd be nice to be able to cite the people who've actually put the work into those modules who actually wrote the code within the pipeline so we are going to try and kind of hit this head on and try and write up some some recommendations for how manuscripts and papers should properly acknowledge the community um this is we've already picked us up with again this week so it's obviously a hot topic um and basically it'll be about kind of thinking about where you should look for contributors um how you could potentially cite them and we might even come up with a little helper tool to automatically get a list of GitHub usernames and things like this um this will only ever be a helper tool for you if you're writing a paper it's never going to be like a hard requirement or anything like this um we realize that publications are uh highly political and very important for the people working on pipelines so at the end of the day it will always be your decision about who you acknowledge and who you cite within your paper um but this is just a bit of a helping hand hopefully um want to give us shout out for a couple of Slack channels there was one we created last week uh we're talking about how to try and make the community as inclusive to new members as possible and one of the things that came up is that um it's easy to sometimes get kind of imposter syndrome when you come and join a new Slack uh you know you join Slack as always people chatting about things you might not understand if you're new to NextFlow or new to the community and you feel like the question you've got kind of is a bit of a stupid question and maybe you should be able to figure it out but of course that's not true there's there is no stupid questions and there is no um every if you're thinking a question then chances are very high that other people are as well and that's what the Slack is for so we want to be as welcoming as possible and to lower that that threshold um and we created a Slack channel specifically for this this feeling called no stupid questions um which was based on another Slack organization um I forget who it was who suggested it and what the other Slack organization was now but the idea is that this is like a really friendly place to come and ask questions so you can find that um on Slack now and I think anyone who newly signs up to Slack will join it but I haven't yet added the whole community so please go and dig that out and have a look um and another one which I think is is a fairly fresh freshly minted Slack channel which deserves everyone's full attention is a NF core means so go and have a look for there and your copy break if you fancy it and remember that there's the channel browser in Slack you have a look through there there's a lot of channels on the NF core Slack partly because we have one for every pipeline um but there might be some interesting ones kind of hidden in there so do do have a poke around and see if there's anything in there which might have been added you might not have seen uh final points uh just to talk about the NF core website a little bit um Matthias especially has been working on this lately and will continue to do quite a lot of work on this and we've been doing a lot of planning work um first up you may have seen duplicate tweets appearing about releases or wrong times and things like that I think we're pretty sure that's fixed now so hooray um next on the list is if ever you try and go to a stats page you'll notice it takes a very very very long time to load and if you're if you're lucky it will load if you're unlucky or crash and if you're really unlucky it might crash the website for everyone who tries to look at the website for a little while so this is not scaling well and we're we're as a top priority now trying to fix that um so that the website's a bit more nimble before all of the events we're having in a week or two um so hopefully we're going to get get that sorted very soon a few kind of visual fixes and stuff coming up and things like uh in the on uh what we used to do for for pipelines is um in the documentation for pipeline we used to have any number of markdown files structured however you want it and we decided a little while ago to try and standardize that to two files uh there we go that's the error I'm talking about uh two files usage and output and that's good for standardization it means it's easier to do template syncs it's easier to build the website in a consistent manner um so that when you're used to looking one pipeline or another way will kind of look the same but we have found that there's a bit of kind of pushback against that because it limits a little bit um if you have a lot of documentation to write for example if you want to write tutorials or if you want to write kind of pipeline specific stuff in more detail then it's useful to have more kind of sub pages so we're reverting that decision and we're going to soon open up the website so that you can have any number of pages in the docs but you'll still have to have a usage and an output file as a minimum but you'll be able to add others um and then the big picture plan with the website is we're going to do a big refresh hopefully um at some point we're going to rewrite the whole back end so it's much much faster um scalable and actually built in a modern way and we're also going to restructure the website in a big way we've got some plans where we're going to try and make all the documentation much easier to navigate come up with a search bar which actually works and finds documentation within any pipeline or any part of the website and a whole load of big improvements coming so please stay tuned right that's all i've got uh oh two things i've forgot to make slides about one of them is quite a big one is um uh some of you may have noticed in the the DSL2 modules repository there's a folder called um sub-work flows which is being quietly lurking here for a little bit of time uh we worked quite a lot on sub-work flows last week and made a lot of progress um so this is coming and hopefully pretty soon i was hoping i would talk about it a bit today but i think there's some decisions which are still kind of in flux so maybe at the um at the summit we might have a bit more to say about it but certainly coming soon we will have um sub-work flows up and running we hope that includes both structure and kind of everything within the NFCOR modules repository and also we got a proof of concept running for NFCOR tools um to work with sub-work flows so that's really really exciting i know a lot of people have been looking forward to that and working on that for a long time and i think we're getting close um and one other thing i've already spammed everyone about this a great deal so hopefully you've not been able to miss it but the next flow summit is coming up and we also have a bunch of other events around that coming up so we have free online training which is next week which has got a ridiculous number of people signed up to it um it's completely taken us by surprise and i think we've got over 700 people now signed up to join the training which is just fantastic um but there's always space for more so if you're interested in getting some training please do hop onto that page get the details and um and register it's the first time we've run it like this where we're running in three different time zones at the same time so running for three days for two and a half hours but you can choose which time of day you want to run it in um so that's really exciting then of course we've got the hackathon coming up in Barcelona and also online and then we have the the main next flow summit coming up as well um we've been updating the the program a lot recently with different speakers and things and we've got keynote talk by Rob Patreau who's just put up his talk title last night um so lots of really exciting stuff happening there and hope to see many of you there right with that please ask me any questions or point out anything i've forgotten or got along happy to put more detail into stuff i i'm slightly on yeah Moritz i think you can um yeah go on um i was curious regarding test data if you have discussed in the core group and made a decision against support or whatever um what about having test data being built by pipelines themselves because it can be quite tricky to uh for example if you want to just add on to existing test data and so processes for them create another output it can be quite tricky to reproduce all those steps um that led to that old test data yep absolutely uh james mentioned this in the new documentation he wrote which is basically saying like do exactly what you're suggesting uh if the test data for your module can be quickly generated is don't add that to a test data repo instead in your test workflow that you've got in the modules repo as those other upstream modules to generate it each time um bear in mind that that means every single time that module was tested it will be running those previous steps so um kind of think about the polar bears a little bit think about the resources needed for the ci tests uh if it's heavy lifting to generate those files and you should do it once and upload the files that are just downloaded and used but if they're fairly quick to generate then absolutely don't feel like you need to add that to a test data repo instead put it into the ci workflow i actually meant um more what about starting putting pipelines into the test data repo that generate the test data um so pipelines are already in a test data sets pipeline repo we've got a branch for every pipeline that's how the ci tests run for pipelines but i feel like that's not what you're asking um generating test data sets somehow exactly yeah so so having uh some kind of more reproducible definition of how the test data i mean right now it's basically put put the commands in in a readme in whatever test data repo uh or branch but basically how about making that itself more reproducible by having problem pipelines and i'm with you now uh yes this is what this is about basically um but in a moment we uh we are not super happy with the the readme files uh which is just massive massive readme file describing what all the files are and where it came from and how it was generated so we want to move away from this kind of readme file approach where people can add as much of as little information as they want to a structured documentation format we're mostly thinking about origin but yeah absolutely we we haven't defined exactly what fields there are yet so we could also define you know a command that was used to generate that test data if it if it originates from some other kind of file um we've got a test data slack channel so if you if you have something specific in mind hop in there and spell out exactly what you mean and we can discuss it in more detail and put something together great question something else core team you're happy that i haven't forgotten anything maybe you should have included more more photos great okay in that case let's wrap it up and thank you everybody for for joining and listening um it's it's all kind of pleasure it's been a really really fun week um and i feel like we made tons of progress last week so i'm really pleased um so if you have any ideas or feedback on decisions we've made then please let us know and otherwise we'll continue kind of writing everything up that we took notes on adding it to the website and um i'm making it easily easily accessible for you