 Alright, I'm going to go ahead and introduce everyone and get the ball rolling. Welcome everyone. Good morning. If it's morning for you, if it's afternoon or evening, welcome. Very happy today to get the ball rolling and to discuss this special issue and the work that's ongoing in order to help improve the specificity and guidance around registration especially in evidence-based toxicology. Joining us today at the Center for Open Science is Paul Whaley. Paul, say hello. Editor-in-Chief at EBT Evidence-Based Toxicology. Yeah, good morning, good afternoon, good evening. Everybody, thank you for coming. Looking forward to kind of presenting this work. I hope you find it interesting. Also on the call from the COS, my colleagues. We've got Mark Call who will be demonstrating some of the key features on the Open Science Framework Registry and Katie Corker, who will be going over a couple of common issues and details with pre-registration. Say hello. Hi, nice to see everyone, glad you could join us. So, I'm going to get started with a general overview, a little bit of a very brief history of registration and some of the templates available on OSF and then I'll be passing it off to Mark to show some of the more detailed features. Paul will then be describing the special issue and what we hope to gain from this call of requesting standards and developing best practices for the field and then we'll finish things off with Katie. So, let me go ahead and get started with a little bit of definition and history. So, when we talk about pre-registration, what we're specifying is that it's a read-only version of a research plan posted to a public registry so that others can see that the work is ongoing. Now, there are a couple of qualifiers to that definition. We do know, we do allow and most registries do allow for transparent changes as the process evolves. Those changes to the pre-specified plan are allowed and some registries also allow an embargo period but generally speaking the process of registering a study involves posting it to a publicly available and viewable site. If you're coming from a clinical background, the term pre-registration might be a little bit different. Of course, it emphasizes the fact that it's taking place before the study occurs. Perspective study registration might be more familiar to other disciplines and some of the social sciences terms like pre-analysis plans are a little bit more prevalent than those are all... Some of them are slightly different but for the most part they mean essentially the same thing. So, pre-registration has been around for a long time. It's been called for in many disciplines for quite a while but it's been fairly ubiquitous especially in clinical medicine and since the early 2000s when there was a specific problem with the number of research trials ongoing funded by the federal government and constituents weren't able to access or determine where those cutting edge treatments were being administered. So that led to the creation of clinicaltrials.gov and around the same time registries in across the world in many countries were forwarding to solve similar problems. In addition to help open the file drawer, it helped increase discovery of ongoing research. And in basic research in pre-clinical work can increase transparency and ultimately what we're looking to see is a clear distinction between planned research from unplanned research. These two modes of work also known as exploration and confirmation of pre-specified analyses are equally valuable but distinguishing between the two can be quite nuanced. In exploration that's really a data driven process. We're looking for unexpected, unanticipated discoveries. We're looking for pushing knowledge into a new area and find unexpected trends or relationships. And this can be really used to inform theory. This is in contrast with confirmation of a pre-specified predicted relationship. It uses traditional tools of statistical inference, most frequently hypothesis testing, even more specifically null hypothesis significance testing. The context of confirmation is held to a higher standard of recovery. You want to be extremely sure of a relationship before suggesting that one exists. And it can have a more direct decision on impact. And we want to minimize false positives in the context of confirmation, whereas in exploration you want to minimize false negatives. If there's a chance of a relationship you want to be able to discover it but before saying that there's a strong causal relationship, for example, you want to hold that to the highest standards of rigor. So both of these modes of research are equally valuable for pushing knowledge forward and pre-registration helps make the distinction between the two more clear. So how can pre-registration help the overall research planning process? It can help improve overall planning. It can help bring additional feedback to the study creation and study design process. Importantly that feedback occurs at a very important time in the study design process in which the study, the data collection or perhaps the data analysis hasn't taken place yet. And so those changes or those bits of feedback are most useful at that early stage of research. Especially when using confirmatory tests, it helps to improve rigor because it really specifies that these were those pre-specified analyses that were not informed by the data exploration process. And it can help to limit what's known in some fields as questionable research practices, those QRPs. It can help specify that the plan is being followed precisely. Once work is registered and conducted, there are a couple of rules of thumb to follow in reporting the results of pre-registered work. Importantly, one must report the results of all pre-specified analyses. Any newly added tests are encouraged, especially unexpected trends or covariates or anything else that was done are enabled and encouraged through this process, but they should be transparently disclosed as unregistered or as unplanned analyses. And if there's any changes to the data collection or to the data analysis process that happened after the plan was registered, simply state that as such. Now, there are a couple of different ways to pre-register, and this is what we're going to be getting to in the webinar. Essentially, it's structured versus unstructured. When you use a template, it reminds it's a checklist, essentially, asking the author to specify all the important details of the research plan, the hypotheses, the data collection processes, what analyses will be used to address each research question. An unstructured registration is simply a narrative document describing what the process is. We have about a dozen templates on the OSF. Some of them are quite specific, like the recommended OSF pre-reg template. We have some that are specific to different modes of research, such as systematic reviews, secondary data analysis, qualitative methods, and we have some that are specific to publication types. Next report format is one of which the proposed study is peer-reviewed by a journal prior to it being conducted. We encourage the use of registration templates because we know that they help individual authors include all the important information needed to articulate prior to the study being conducted. In this evaluation of structured versus unstructured research plans, Bakar and her colleagues compared 53 registrations that were structured to those that were unstructured and found quite a bit of differences between the two. Essentially finding that the unstructured ones didn't include necessary details in order to fully pre-specify the analyses and authors also recommended improvements in ways to make sure that those structured ones included as many details as possible. Now I'm going to hand it off to Mark to talk a little bit about the workflows on the open science framework on OSF. Mark, would you like me to stop sharing my screen? Yes, please. All right. Thank you, David. David discussed about the concept of pre-registration, what it is and why we need it. If I can share, I'm going to show what a pre-registration looks like specifically on the open science framework platform. I am not going to go through all the small little details, but more of saying kind of like what does it look like and how do you get that process started. This is an example of a pre-registration template. David mentioned earlier that we have several different ones. The ones we're looking at is the OSF pre-registration template. It is one of our most rigorous ones and the more structured ones. And the intent is to really challenge researchers to think through the different nuances of their study to help you think ahead about what some of the challenges you may encounter. So kind of going through this, you will see here on the structure side or the left-hand side, we have an overview. That's where you all have all those different types of questions. How those questions are entered, it could be text fields, radio buttons, check boxes and all that is right through here. And some of them will also give you the option to upload files as well. So if there's something that has way too much context to put into a simple text box, you do have that option of some of those questions. And everything is structured pretty much in a section kind of format and subsections. So that way you can easily navigate through it. So you can scan through it and you'll see it through formatting. Or you can click the hamburger box up here, click something such as design plan and it'll take you right there. So everything is structured so it's easy to navigate. This also has a lot of metadata. So when the metadata is going to be here on the right, and what that really does is help others discover your research and it helps you be able to do other things such as if it's public, you will share your DOI. So every public registration has a DOI. So it's always linkable. It will also tell you when it was registered, so that way you have that kind of permanence in that timestamp saying, here is my study design. Here's one I submitted it, et cetera. It gives that kind of sense of kind of confidence, kind of reduces some of that concern that some researchers have and other things such as contributors and description of the registration. Just kind of like an abstract, except you kind of don't know your results yet. Along with that, there are other different pieces such as you can, it also be able to arch up all the files that are stored on a project or an connected add-on such as OneDrive or GitLab, et cetera. And then you can also add different types of resources. So we know that with different resources, depending on what your discipline is or depending on what your study design is, there might be other repositories that's best to house that research. For example, if you're doing EEG data, you have terabytes of information. And there might be some different types of requirements based on funders or journals or your discipline to put that data in a specific type of repository. Well, OSF, what you're trying to do is connect all those different types of objects. So you don't have to have them on OSF, you just need a DOI. So you can click on one of these types of resources. It will take you on this page where you can add that through DOI. You can add a description and all that information is right there. So pretty much since the pre-registration is kind of the first stepping stone of what you're going to do, then that's it will help be able to build kind of that structure, kind of that house that creates that foundation. And then you can add these pieces brick by brick until you create the entire house. Now, how do you actually start a registration? It's really quite simple. If you go to the main OSF page, you'll start at OSF home. You can click done here, like registries. Click add new registration. I want to assume that you don't have a project because we just created an account. Select which registration that you want to go with. Let's go with open-ended. It is our most flexible, create draft and you're off to the races. If you need any help or support, we do have help guides. Those are populated at help.osf.io, which I'll put in a chat in a second. And if you have any more specific questions that the help guides don't really answer, you can email me directly or you can email us at support at osf.io. And I will put that in the chat here in a minute as well. So with that, I will stop sharing screen and I'll pass on to my next colleague. How did you want to share your screen? Do you want me to? If you want to share, I think I've put my slides in the wrong place in the slide deck, so they are from slide number 20. Sorry about that. So if you go ahead and share screen, then I'll just say next when we want to go to the next slide. I'm just trying to get that slide up. Good stuff. Thank you very much. So thank you, David, Mark. So I guess I'm here talking about pre-registration templates for toxicology and environmental health. Next slide, please. So for those of you who can't remember who I am or don't know who I am or weren't here at the top of the hour. So my name is Paul Whaley. I'm editor-in-chief of evidence-based toxicology, which is an open science journal for environmental health. And I'm co-chair of the evidence-based toxicology collaborations, open science working group. So you've heard of our pre-registration templates in general from David and Mark, but you're probably thinking, why is toxicology environmental health journal getting involved? And I'm going to explain now. Next slide, please. So the reason really is that in toxicology, environmental health, we have the same challenges as many other fields with the reproducibility, accessibility, and so forth of research. But in toxicology, we probably, to some extent, don't really have that same rich tradition of discussion of whether the publishing practices that we're engaged in will have incentives and center structures that are working under a sufficiently supportive of our scientific goals, right? So we're kind of famously a bit slow on implementing systematic review and systematic evidence mapping methods for making sense of evidence. They were very kind of very much mainstream by the mid-1990s in medicine, more or less unheard of in our field until around about sort of 2014, 2015, really only kind of became a big deal probably around 2019, 2020. Protocols, so the plans for doing a study, making those available before doing the research, those are still very rare. Like they're rare for systematic reviews, there probably is still only maybe three dozen of those have been published in Environmental Health Journals. And pre-registrations for animal studies and in vitro studies are even rarer still. So it's just not really part of the research culture. And to some extent, there's probably some ambivalence, I think, about what the replication crisis means for our fields. No one's really done a lot of research into that in our domain yet. And then obviously then how to respond. Like we don't know what the problem is, we don't know how to respond to it. That's not really discussed. Next slide. But at the same time, we're toxicologists and environmental researchers, like many other scientists, very good at complaining about other people's studies, right? So we ask ourselves how much of the research that we're doing is really being done well enough. Particularly in research that's been done and is sort of intended for making decisions, a kind of regulatory toxicology, this kind of thing. There are lots of concerns about whether or not we really have enough information about the study methods to judge the validity and the utility of data for the decision-making context in which we're working in. And that's kind of seen as a big challenge for making research as usable and useful as it could be. There are questions about journals contributing to publication bias. We know that there's probably an over-prevalence of positive findings in the literature, even in our space. There are some interesting incentive structures which may mean that negative findings may in some areas be also over-prevalent. Generally quite tricky to get on top of. We know that authors are probably selective in the findings that they report from their studies. If they've looked at lots and lots and lots of outcomes or lots of intermediate outcomes, they might only be reporting the significant findings amongst that. We don't know how much research is being done that is basically redundant because we already have enough information about that thing. Yeah, next slide. So evidence-based toxicology is a journal set up to address this, we think, we may be wrong, but we think with the first open science journal for toxicology and environmental health were set up by the evidence-based toxicology collaboration as their official journal in collaboration with Taylor and Francis. And our mission is to publish manuscripts that utilize or advance evidence-based methods and toxicology, implement innovative open science practices as we handle submissions and hopefully demonstrate the value of open science practices to the environmental health community. All very grand. Next slide, please. To support this, we have a set of key editorial policies that are intended to improve publishing by putting science for impact and increasing transparency and accountability of the decisions we make as a journal. We've got some stuff that we're pretty pleased with. We're pleased about ourselves with. We require preprints because we think that basically solves the problem of publication bias. We do public manuscript evaluation and peer review. So when we make a judgment on a submission, we publish the judgment and its reasons as a linked record back to that preprint so everybody can see why we thought what we did and made the decision that we did. We try to base our decisions on rigor rather than the results of a study. So we try to focus on the methods, not how cool the findings are. We are aiming a very, very high open data standards. We're hoping to end up in the top 1% of top journals worldwide. And we give people the option to publish our methods before collecting their data. Next slide. Of course, improving publishing overall is very complicated and strong editorial policies help. But so does experimenting with different publishing formats. And we want to give people the maximum opportunity to innovate in the types of manuscripts that they submit so that if we are not aware of the submission type, that could be, if it became normal, beneficial to the research community, we want to give people the opportunity to make those submissions. We happen to be involved in the world's first randomised control trial of editorial policy interventions. That's going to be a slightly different submission type and you're going to be hearing more about that through Centre for Open Science because they're the people leading this and we happen to be involved. We do consider protocols for any study design. Then we're open to methods papers of any type and then the opportunity to try pre-registration templates came along and we were like, huh, this kind of strikes the philosophical hold with us, obviously. Next slide. And there's just a good fit for us because we encourage publication methods before collecting data. I think I said that three times already. This is obviously difficult to do without examples and templates showing how. So if we can publish a type of manuscript that helps people publish their methods before they collect their data, then that's only going to be a good thing, right? These templates, because you allow us to view the method as a product of a research process, that gives people the opportunity to publish and get credit for work that might otherwise have gone unrewarded in journals, maybe that only publish the results after the methods are done. And then we thought, well, how do you kick things off? Well, if you're a new journal, you kick it off with a special issue, right? So that's what we did. Next slide. So give you some ideas for templates. Next slide. The reminder of how it works. Basically, the gist of all this is that anyone with a method that someone else might use should create a pre-registration template. I mean, it really is a submission format for everybody who is interested in helping other researchers use their methods, right? So you've got a researcher, they develop a method, they conduct their study. They probably submit that study to a journal and get it published good for them. But then what they're thinking also is, well, I know that I've got colleagues who want to use this method. They're probably people in different labs have not met yet who might want to use the method. So what I'm going to do is I'm going to convert the method in my paper into a pre-registration template. You know, publish that template in evidence-based toxicology or with Center for Open Science or with both. And then a second researcher comes along, they're like, you know, the Googling on the internet or whatever, and then bing, they find this template that relates to the study design that they're interested in doing. They're like, well, this would make it much easier for me to pre-register or it gives me the recipe for the study if you like. They go ahead and they pre-register in confidence. They're covering their bases and they get all the benefits of doing that pre-registration that David was talking about and the previous researcher gets cited and everybody wins, right? Next slide. So lots of options for this, like talking in the abstract, maybe more difficult to understand what those options are. But we could think about minimum information for registering the existence of a planned epidemiological study. So that's quite easy, relatively easy pre-registration to do because you're only giving people the information they need to understand what study that you are doing and when. So they don't replicate your work by accident that they know what you're doing so they can plan around that. People will be aware of your study before you finished it, how it was publicizing it. You can plan the analysis of data. If you're doing this like you're doing data heavy stuff with cell culture things, this could be really useful just to have like an analysis pipeline template that everybody who uses the more or less the same method that you're using can use to pre-register their analysis plan. That's good for preventing kind of expectation bias and things influencing results in a way that sort of biases them towards like positive publication. And then if you wanted to aim really high you could do like a comprehensive study methods and data analysis plan for like say big kind of observational studies. And if you're seeking to comply with good epidemiology practice because you want the regulators to respond to your research then that could be kind of like a big deal thing that you're aiming at. So there's lots of levels to aim at. Lots of different ways of thinking about this as a journal. We're not going to be fussy if you go narrow because you don't have loads of capacity for doing like a really big template. If you want to go big and go wide, we'll support that. Like we aren't interested in prestige we just want good methods. So we'll be very open to pretty much anything that people can come with provided it's rigorous. Next slide please. See if there's checklist or guidance there's a template is basically what we're saying. So give them and regulate your compliance kind of issues. So you're doing a study you want to regulate your uptake you could interpret the guidance the regulator gives into the specific studies that you've got and that can help people really kind of crystallize what can be quite broad guidance into very specific standards compliant studies. You can do that for epidemiology. It could be really useful for animal studies where you're trying to show that your study is worth doing that you've minimized the use of animals that people will necessarily need to do an unnecessary follow-up study that would waste animals. Systematic view and evidence mapping that's very kind of structured and procedural and that lends itself well to templates. If these are necessary, desired, end up being helpful they could be very high impact publications. So I really kind of recommend that people consider doing these things. Next slide. I just say reward about eight slides ago if anyone's ears pricked up to that we will be giving the best templates or templates that we receive at the journal before the end of 2024 a cash prize at Society of Toxicology 2525. We haven't quite figured out the details like how much cash, because it depends a little bit on how many prizes. But we always want you to know that we are really keen and we definitely want to reward people who do this work well. Next slide. So thank you very much for listening. Thank you for Center for Opus Science for this initiative. I recommend you read the editorial. You submit your template to the link. You can follow the journal on LinkedIn for exciting updates about new publications when we publish evaluation reports of a manuscript. If you bit knows, you can follow that. You can, we'll place links to those. And if you really want to, you can join an evidence based toxicology working group. And I certainly recommend the Opus Arts working group as the coolest working group that we have. So I'll hand it back to, I think Katie now. Yeah, thanks very much, Paul. That's super interesting. It strikes me just in general that this is a process we would love to see duplicated at other journals. So if there are other editors out there that are interested in doing this sort of thing, please reach out and see if we can set up a similar process at your journal. Even just for people in other fields more generally, I think this gives a nice concrete example of the kinds of templates that we'd like to feature on OSF more broadly. And so we do have another process that runs alongside the process for the special issue where people can submit their templates to be considered for inclusion in the OSF. And all of the templates that are submitted to the special issue will also be considered for inclusion in the OSF. So it's sort of two parallel processes for evaluating those templates for publication and for inclusion in OSF. But it's just an opportunity for there to be even more impact for those templates to be of broader use to the community. We do wanna take your questions and have time for everyone's to get everyone's concerns heard and the questions answered. I have a couple of questions to sort of kick us off, but please be thinking of your questions while I'm asking these. And you can go ahead and put them either into the Q&A or into the chat and Zoom. So as a sort of seed to get us started, one question we might think about is why we need these pre-registration templates at all, right? So why is it the case that the pre-registration templates are providing something above and beyond what's in the method section of a paper or what might be in a protocol study by itself, protocol paper? Protocol paper. And this could be for any of the three of you or all of you. Thanks for sharing the slides as well, Paul. We'll definitely share the slides back with all of the registrants as well after the webinar is done. Yeah, that's a good question. So there's a couple of different ways to sort of articulate the benefits. So one is specific to toxicology. Every discipline is going to have slightly different norms, different terminology, different methods that are frequently used. And so the benefits of field-specific templates are that the language is most interpretable by the intended audience. We have several on the OSF right now that are very generic, but they were intended to be quite generic in terms of discipline relevance. But frankly, they were created by folks from within specific disciplines. So some of the terminology, some of the norms might not mean precisely the same thing when reviewed by a different audience. What's the difference between a registration template and a protocol? The registration template is really designed to mark somebody's claim that this study is going to be taking place by these people at this time with this population of data or of people. And it might reference, it might use a specific protocol, but it has more data, more metadata around it, articulating why this study needs to happen at this time and then using a specific protocol that might be included in the registration or some of the templates that are submitted to this might rely on a couple of externally previously published protocols and ask people to identify which one will be used. So the variation there is what we expect to see from this type of open call for templates. Wonderful, thanks. Sort of general question for Paul, and then I do see we have an audience submitted question, so I'll ask that one as well. Question for Paul, would evidence-based toxicology consider templates that are partial, so ones that are focusing on a particular issue that's relevant for evidence-based toxicology rather than say a full-fledged template? I mean, I know you addressed it a little bit at the end of your talk, but could you just say a little bit more about kind of the scale of the templates that you expect you might see? Yeah, so, I mean, we're very open to working with authors on what they think is a worthwhile amount of work and what they think is a unit of contribution, right? And we don't necessarily expect all units of contribution if you like to be equally massive, so whilst obviously big papers and massive contributions are very welcome, also small contributions are very welcome and we don't mind how big the contribution is and that isn't something that we take into account in our decision-making when we review a manuscript, like a small rigorous manuscript, as far as we're concerned, is as publication-worthy as a large rigorous manuscript, so what we care about is rigor. So if you have like a pre-registration template that's quite niche, that addresses just some aspect of, or some value in pre-registration, so like you say, like, we're just gonna concentrate on the minimum information that you need for pre-registering, so registering intent to conduct a systematic review, right, and that's it, then you'd obviously wanna show that you weren't just repeating what's on Prospero in some way, but you had like some unique value, that might be challenging given the Prospero already exists, but you might have something that makes it an additional unit of contribution and then that would be totally fine for us. You might be like, actually, we've got this complex, like all known chip toxicology study where we've got a whole method and analysis pipeline that includes some machine learning and some big data and all kinds of things, right, and biological pathways and then all sorts of stuff going on, and then we'd be like, okay, well, no, that's also equally valid, it's gonna be a big deal, probably, but again, we don't mind, we'll just take pre-registration template and we'll evaluate it the same way for everybody, which would be based on rigor, not on Skype. Wonderful, thank you. Patricia, would you like to ask your question live or shall I read it for you? You can read it for me, Katie, thank you. Patricia is teaching about research integrity and open science next week for first and second year PhD students, and she's wondering what would be a way to convince them that pre-registering their studies is good for them because of course we know it's good for science, but what is the argument considering that they still consider it something that's difficult to do? Yeah, this is a great question. So there are a couple of different ways to go about that. One would start with a sort of improved planning process, being an early PhD student, it can be quite intimidating to sort of figure out what goes into sort of a complete study and just use of the templates themselves can remind one about important design elements to include or the necessary details to add to the proposal as one gets closer to conducting their first sort of independent type of work. Using that template could be a good way to solicit feedback from advisors and committee members that helps both the writer of the plan make sure it's as fully fleshed out as possible, but then helps the advisor and the other committee members or other colleagues that they're reaching out to sort of reminds them about the different elements that should be included and can provide some better feedback there. If they want a strong reward of a promise to publish the results of the work, regardless of how it turns out, they can reach out to a journal that applies the registered report format. And after working with the advisor to sort of flesh that out fully, submit it as a proposed study to one of the over 300 journals that offer that as a publication option. That's a very strong incentive and reward to go through. So that's submitting the proposed research plan before it's even finally registered to the journal for feedback. And we haven't talked too much about that process here, but that's sort of a modification of the traditional publishing format. Another couple of benefits is that it shows progress on a research track in a way that's better than just something's in prep. So NIH, for example, encourages the use of registrations and allows that as an interim research output. So if you can identify that in progress reports and send a link to that, that's much more compelling than just saying something's under development or something's under work. You can actually sort of demonstrate that this proposed research has been fully fleshed out and is ready to go. So there's a couple of different of those types of benefits I would start with. Can I build a little bit on that? Yes, I see that very clearly. The four points you put out. The barrier we've been seeing here is how these students can convince their supervisors who are older, more traditional, not familiar. They don't know what cost is, what OSF is. Really, sorry, but they don't know. And then besides having to learn how to register something in OSF, they have to teach back their supervisors. Yeah, that's one of the big... You see that. One of the biggest kind of culture change impediments almost is just the fact that one might be being trained by somebody who's never heard of this, doesn't see it as sort of the necessary part of the process. It's not something they did. So why should they tell their trainees or even allow their trainees to do it? There's no magic bullet here. BI, sort of teaching about best practice and benefits that they have. You're encouraging the students to learn up on it so that they become their own strongest advocates. Kind of worst case scenario, kind of using the template and not even registering it, but just using it as that planning document is a way to get experience with what it is, even if you don't use it to its fullest potential and that can perhaps provide some more sort of concrete examples of those first two points. It really helped me make up a better planned document and I can share that with you in a little bit more concrete of a way than I would have otherwise. And that could be sort of the step in the right direction we want to see. So yeah, there's obviously nothing, there's no so-called magic bullet, but these are all kind of steps that we encourage folks to take one after the other. Can I also just chip in on this very quickly? So I said I work for the evidence-based toxicology collaboration. So what we do is as a research collaboration is trying to provide a supportive environment for early career researchers who want to use these pre-registration type processes, right? So if an early career researcher is working on an EBTC project, then it'll be the something that we've set up because we think it's worth doing, or something that a member has proposed doing through EBTC and through one of the working groups so they can collaborate with other people in the working group on a project. Then EBTC, we require empirical research and it's going to be like an official product of collaboration to be done on a registered reports format. So you have to do your protocol and pre-register it. So we provide a mechanism whereby if a researcher is not being supported by a supervisor in pre-registering their study, we'll provide that support instead. And if necessary, we'll mediate between the PhD student and supervisor if they want. Usually the extra free supervision that we effectively provide is pretty welcome and provided it's not costing too much or anything, then we just generally have to get on with it, right? So I think that's another way of enabling early career researchers is to steer them towards research collaborations who will support them in this type of work if they're not getting that support in their own internal labs. And there's other benefits that you get your network and everything else, you get the extra publications, you get to show you, you show some initiative. So yes, I think I'm basically saying goes to the evidence based on security collaboration. Hit that join button. Yeah. Great, thank you, Paul. Yes, thank you. I would just echo that in general, even for researchers in other areas, finding a community with individuals who are also interested in doing this kind of work is also very helpful for finding support and exploring strategies for navigating these kinds of tough issues. Yeah, just a second comment. We work here with a big group that does clinical trials. So for them, it's their nature to register, right? It's easy, but there are a lot of people working now on laboratory research and big data. And these guys don't know where to register, how to register. Even the systematic reviews don't, which are not, which don't have a clinical question. What do we do? Where do we register? I'm always sending them to you guys, so. Well, we appreciate it, thank you. Thank you very much. So we have another question in the chat about exploratory research. So if I could add on or elaborate to the question a little bit, I'm sort of wondering about the value of the pre-registration process for exploratory research. And then the question that the question asker was wondering was how are templates updated if in the process of exploratory research? I'll put in a link to the update help doc in just a second. Yeah, to answer the first question directly and then I'll expand on that. There's a feature to show updates or changes or revisions to the plan in the interface and that allows transparency to the reader to say, okay, this part of the plan what was updated at this time and provides a spot up to explain the rationale for that decision. You know, perhaps data, the best way plans once you get into the battlefield, of course, go up and smoke. And so as the study is conducted and reality sinks in, if changes happen, that can be transparently disclosed. The clear aim is to still make that distinction between the planned work and the unplanned work. And so there are nearly infinite number of reasons why those changes could have to take place. It could be that the data came in, didn't meet the assumptions of the statistical test provided. It could be that the data collection process was just unfeasible for one reason or another. It could be that a moderator, the temperature or age or anything else like that seems to account for a lot of the differentiation that you see in the results. The problem with just doing that without describing why or how it happened is that there are of course so many of those reasons that noise itself will creep in and provide results that look more surprising that come at the expense of the credibility. And so the transparency that that interface provides helps give context to when and why those changes were made. Especially if they occurred before the study was being conducted, there's extremely little justification needed. The later the changes happen, so the more detailed the justification can and should be. And the baseline recommendation remains generally the same of reporting the results of anything that was specified ahead of time. And then if the sort of post-talk explanations seem to be promising, report those as unplanned tests that show a lot of promise, but that deserve follow-up confirmation or follow-up replication. And so that provides that good avenue for showing what's promising and that unplanned discovery, but providing it with the correct amount of context saying that this can't be the main finding of the study. It's something that needs to be replicated and confirmed through additional research, which is something that's very commonly said in almost any paper that we read, of course, but it doesn't have that transparency of saying why the additional follow-up is really needed. Can I just jump in there and break a desire? So I think this is Adam Betman's question, if I understand. Yes, so I think Adam's also asking a little bit just about versioning of templates, right? So like if you've got a template and then you're like, actually, oh, we've improved our method, right? We want to update our template. Yes, you have to do that. So that depends a little bit on the technology platform that you're using and how you handle versioning. So I'd say I can tell you how we would handle it at Evans Space Toxicology, Center for Open Science will have a different method or approach, protocols.io will have another approach, cell-press protocols will have another approach, right? So what we would do is we would say, okay, because we're pre-print required, your protocol is that your template is, it's first of all posted as a pre-print, somewhere either OSF or Zenado or whatever, and then you would publish the template with us as a manuscript, and then you'd do some updates to the template and you'd revise, you create new versions of that pre-print registration on the repository each time you updated the template. And then probably at some point, you'd come back to the say, look, we've done five updates now, it's been a year, we reckon our template is a lot better, and then we would re-review your template, see if it has improved. Is there any questions we need to ask or anything like that? And then probably what we would do is we say, okay, well, we're probably going to publish a whole new template here, but what you probably should do is submit as a research letter, which is one of the lightest submission types that we have, which would be like three pages long, explaining how you've got new versions, and that's a way of announcing you've kind of got version two or version 1.5 or whatever of your template, and you can broadcast that through the journal and get an additional publication out of it maybe, is one way we would probably go about that. Okay, yeah, that's helpful. Thank you both for good answers. Yeah, I'm speaking specifically, we're working on a lot of in vitro inhalation, toxicology testing, and a lot of it's very new, and we're looking at a lot of different ways to kind of get to the same endpoint to evaluate cell viability, things like that. So yeah, my question came from a point of, if we do come up with our first method, and then we put it in a preregistration template, if that does end up changing, and we find out that there might be a more efficient way, or maybe the cells perform better using a different cell culture media or anything like that, then it may need to be updated with different versions. So, but thank you. Yeah, I think just talk to the journal, all right, and I was like, to the risk of pulling back the curtain and revealing a brinkly little bald old man, like when we're off, we don't know what we're doing either, right? Because this is me. So if you have questions, it might be that we just don't know what the answer is, and then we can work together to figure out what the most appropriate format and versioning process is, and we would appreciate that probably even more so than you would, and then we can implement that for other authors as well. And partly, one of the reasons for doing this special issue is to encourage some of this dialogue between us as a journal and the researchers who are publishing with us, and just try to recalibrate this relationship, but not a journal that is like a gatekeeper of publications. We are supposed to be collaborators in research. We have an editing role, which is to make research work better. So we're providing a service to you as a submitting author of making things better for you. So if there's stuff that we can do differently, we want to hear about it, and we want to change what we do as a result, not just sit here and just like in our ivory tower, go, yes, no, Bing, whatever, right? So yeah. All right, thank you. I can just say a little bit too about the process from the COS side. It is possible for us to version templates as well. So like our existing templates conceivably could get, you know, an update and have a new version. Our current procedure is that new templates that are submitted for consideration for inclusion in OSF go through a review process that's reviewed by the committee of peers, the working group that we have that considers these templates. And then our plan is to put revisions to templates through that same review process, or if there's a major update that needs to be made, that can also happen. But one of the factors that we'll consider when deciding whether or not to implement a template on the OSF side will be about the sort of maturity of the template or sort of how ready it is to be rolled out to the whole world. So I could imagine a situation where something that's a little bit more experimental or at an earlier stage of development might get published by the journal and not yet be ready for implementation at OSF, but eventually perhaps after it's been road tested a little bit more or seen a bit more use, then it would be ready for that kind of deployment. I mean, Casey, so that's because there's a resource implication for the Center for Open Science to maintain a template. Whereas as a journal, we don't have a resource implication because we just post the manuscript which has kind of like a document template rather than like a form with interactive elements and things. So yeah, they're quite different I think. So you kind of have like a two step thing you can publish with us. It may or may not end up being a supported template depending on Center for Open Sciences capacity and the availability of resources for supporting it essentially. Like that second filter is the supported template, but you can still get a template published with us. No problem at all. Provided you hear our regular requirements. Yeah, absolutely. David, did you want to contribute something? No, you covered it well. Thanks. Yeah, Patricia was asking about the peer review process for the templates at Center for Open Science. This is a relatively new process for us as well. We've had a working group in place for a little less than a year to consider new templates as they are being submitted. So the working group has developed some procedures and criteria for considering those templates. It is modeled after kind of what you would expect for a typical peer review process of the templates are submitted, the reviewers compare the templates to criteria, there's discussion, that's some sort of a hybrid between kind of like a grant review or a normal journal peer review. The reviewers, the people on the committee are other academics, experts in the process of preregistration. So people who have either created these templates themselves or just have some expertise in the process of registration more generally. And yes, Paul is one of the committee members. Yeah, I think what I was saying, it is basically, it's more like it's kind of like a committee peer review. So you would go through places where two people from the committee are a volunteer to review one of the preregistration templates as a form that we work through to try to improve and encourage the consistency and thoroughness of evaluation. We're still trying to work out what that form looks like. You're gonna have to do something similar at the journal, work out how we're gonna do the peer review process ourselves. This will help accelerate our understanding of how to review these things. And again, speaking to myself and not for anybody else, we don't know yet fully what we're doing. So we're kind of working it out by doing it to a certain extent. So I try to say way more about what they're doing than I do, so I'm just speaking for myself. But that's kind of the situation we're in. Yeah, no, I think exploring and breaking new ground is sort of part of making progress in this area and contributing. So it's exciting for me to see it moving forward. Well, I think we are about out of time. So I will go ahead and thank our speakers today and thank everyone for the engagement and the questions. Like I said, we'll send the slides around and we hope to see your submissions soon to the special issue.