 All right, let's go ahead and get started. Thank you, everyone, for joining. This is, I think, the third or fourth Registry Report Q&A webinar. All questions, all answers about the Registry Report format, of course, if you're joining, you should be fairly familiar with the format. Two-stage peer review, peer review before results are known with an in-principle acceptance if it passes that stage. The second round of review just to ensure that the accepted protocol and the registered protocol was conducted as promised. On the line is Professor Chris Chambers, leader and chair of the Registry Report Committee and all-around great guy. My name's David Mellor from the Center for Open Science. I work on our policy and program initiatives. And we're here to answer all of your questions. You should be able to use the Q&A button on the bottom of your screen. You can go ahead and start submitting those. As the questions come to your head, if you have them, please submit them. We will go through and make sure to answer all those live. We will be recording this to make sure that that's available afterwards. And we will attempt to use the transcription feature just to make sure that those are widely available. Chris, before the first questions come in, I wonder if you'd be willing to give a little overview of the rapid response Registry Reports that have been coming in on your end. Yeah, OK, thank you, David. Welcome, everybody. So I was just thinking, welcome to the Apocalypse edition of this Q&A. I think, as David says, we're on our fourth week. This is our fourth one. And third or fourth. And like I always say, this is a completely unstructured informal hour of just Q&A. So it's registered reports for nerds. Anyone who's got any kind of specific or general question about the format, I don't have any slides. I haven't prepared any kind of presentation or anything like that. It's simply here so that you can have an open discussion. And we are in a very strange situation right now, all sitting in our lofts and attics and bedrooms doing all this. But because of that, I think there's been a lot of talk lately about people doing Registry reports during this time, because we can't get to our labs. We can't necessarily collect data or run experiments or studies. So now it can be potentially a very good time to be thinking about completing a Registered Report. And I've certainly noticed an uptick in the numbers of submissions that we're receiving at the journals I edited out. Now, David asked me to just mention one thing briefly before we start taking questions. And that is the Coronavirus Registered Reports initiative that we started back in March. And I started this because I felt that with the surge of preprints and research across the entire spectrum of STEM or the end social sciences that's happening on Coronavirus, we really needed to try and adapt the Registered Reports model into this to test whether it's able to produce the kind of rapid high quality review that I hope it can. But also to try and introduce and inject some rigor into the flow of research that's going through at the moment on COVID-19. Because I think there have been legitimate and reasonable questions asked about the quality and the standard of some of that science is going out. So with that in mind, we launched this initiative at Royal Society of Open Science in March. And it was subsequently taken up by a number of other journals, including Nature-Human Behaviour, Plost Biology, PHA, and a couple of others. And the aim of this initiative is to provide a home for rigorous pre-registered Coronavirus-related research in any area. And the commitment from the journal is twofold, number one, to review these submissions as quickly as possible, ideally within seven days. And number two, to waive any applicable article processing charges so that we create as much possible incentive to use this and remove as many possible barriers and gateways from the process of actually doing a registered report. And as part of this, of course, we needed to have a review network ready that can respond quickly to submissions and is motivated to do that. And so with that in mind, we also created our rapid response registered reports review network, which is quite an acronym. And I'm happy to say we have over 800 reviewers across the entire spectrum of scientists now have joined that, which is a really fantastic response. And I'm tremendously grateful to all of those reviewers, maybe some of them listening and watching now. So thank you, if you are. It's been a great response. And because of this, we've been able to break all records on the list of registered reports. And we're taking them in fields we've never seen before. And you can read into the chat window, which all of you should be able to see, I've posted a URL to a blog post of mine where we introduced the initiative. And I provide updates as they come in on there. And you can also find them on my Twitter feed. I've got an ongoing thread. And if you go to my pinned tweet, you'll find the beginning of that thread. That's the short of it for that initiative. And of course, I'm happy to take any questions about this specific initiative on this webinar. If you've got any of you, maybe considering a submission, but you're not sure about some part of the process or eligibility and so on and so on. You can always put those questions to me here and to David as well. And of course, we're here to take all your other questions, as you may have. So fire away. I see there's already a general question. First question came in, Audra and Rosalind. He's asking, is anyone actually checking the registered reports? Or is it just a means of transparency if the results are considered questionable? Chris? When you say checking, I assume you mean is anyone reviewing them, like the stage ones? Yes. So let me be absolutely clear about this. These registered reports are reviewed the same way as any other registered report, which is to say, the stage one manuscript is submitted. It is said to multiple expert reviewers who assess the protocol according to the same criteria as we always have at RSOS or the other journals. It has to meet the exact same standards. It's just that two things are different. One is we are prioritizing reviewers who are part of our rapid response network who have committed to turning around reviews in 48 hours or less. And we're waiving the article processing charges. So those two things are the key difference. Otherwise, everything is the same as the standard registered report. I suspect there this might be a question also perhaps about the difference between registered reports and an author creating a pre-registration on their own using that for their research and then submitting a traditional manuscript. Right, which is not a registered report. That is just a registered article, which is something different. Registered reports differ from self-registration because there is a peer review process that's introduced before the research begins and before the registration is made. And it's then following that review process that that protocol is finalized and if accepted is then registered. And as part of that, the journal also commits to publishing regardless of outcome. So these are the crucial ingredients of the registered report format. Pre-study review in principle acceptance regardless of outcome. You won't get either of those or together, you won't get those for a standard preregistered article. Yeah. And the sort of two ways that standard authored initiated pre-registrations can sometimes be used. There's simple disclosure linked to the pre-registration and then that's available for reviewers or later readers to compare what was pre-specified to what was reported. And some journals will also take, that's kind of a level one approach that we refer to have a top guidelines, recommends starting these types of author-led pre-registrations. The second level is to really require that the reviewers or others on the editorial team compare what was pre-registered to what was reported and just make sure that any changes or differences are transparently described by the authors and to make sure that the authors report all the results of all the tests that they specified ahead of time. And that's again on author-directed pre-registration as distinct from the registered report. Anonymous writes, we were meant to be collecting pilot data now and then planning to submit a registered report, but of course in the current situation data collection is on hold. We're wondering if it's possible to submit them as one with a clear decision tree of how we will deal with various outcomes from the pilot work. Yes, the short answer to that is yes. We get these submissions quite a lot where rather than using pilot data to confirm one particular procedural path, which is good simplifies things if you've got it, but it's not always possible. It's not always necessarily the best idea because life is unpredictable. So sometimes researchers instead pre-register or submit contingencies, that's fine. Yeah, that's fine. Just make sure that it's really clear the nature upon which you're going to make the decisions. I recommend including a figure or chart which shows your decision tree in a very easy to digest format because reviewers are not always particularly used to assessing that kind of thing. So make it as easy as possible for a viewer to navigate and provided your rationale is strong and your methods are robust, that's absolutely fine and that would go well through review. Yeah, I would add make sure that those, sort of the criteria for deciding on path A or path B, make sure that's really crystal clear with sort of unambiguous cutoffs between the two. Yeah. All right. How does the Richard report tie in with the grant writing process? Given that a protocol needs to be in a relatively final shape prior to submitting a grant, how would the reviewer comments from a registered report affect a grant protocol? So it's not often that a registered report affects a grant protocol directly, it's more likely the other way around. So normally what happens is somebody gets their grant first and then with their grant awarded, they would then submit a registered report or one or more registered reports to journals. Now, this can be problematic if reviewers want you to change something about your design that you had approved and your grant to create the conflict. And most of the time this isn't a problem because grants in most areas, at least the areas in which I kind of serve as editor are usually quite generally written. They're a general roadmap of questions and methods and approaches. And there's quite a lot of space within that to navigate different paths depending on detail. So most of the time we haven't seen problems where stage one reviewers make recommendations which are incompatible with the requirements of the funding agency. There are or there will be of course situations where that arises, particularly in clinical areas where protocols can be more detailed and they can be much more specific at the grant level. In those situations, it is tricky. The best solution is to integrate the registered reports process with the grant review process. And this is called a registered report funding model or registered report research grant. There are a number of agencies which offer these now where you submit your proposal simultaneously to a journal and a funder. And if it achieves acceptance from both parties, then you get your money and you get your in principle acceptance at the same time. This is a really efficient model because it takes those two pre-study review processes and it combines them into one. Another way you can do this is to actually submit your registered report proposal to the journal before you have your funding. Now, some journals won't allow this. They will only consider stage one registered reports for projects that have the resources in place now to begin. But not all journals are the same. Not all editors are the same. And it's always worth having that discussion with the editor. As editor, I've accepted quite a few registered reports over the years that definite funding in place at the point of stage one consideration. And then those authors having got the stage one approval could then go to a funding agency and say we have an article guaranteed already, which could possibly increase their chances. That's always, it's always worth having that discussion with the editor before submitting to see if that's viable. And I believe there's at least four or so funding groups or agencies or foundations that are working on this model now. I know the Children's Tumor Foundation, the Flu Lab, Cancer Research UK has had a couple of grants given this way. And there are sort of more in the works there. Yeah, there's six in total. And it's growing. And I think it's particularly important. It's going to be particularly attractive in clinical areas where research grants can be very specific and precise. And we're in a lot of areas I work in, grant proposals are often very sort of general exploratory roadmaps. And there's a lot of, you know, there's no, the granularity level of granularity is completely different between a website report and a grant application. Question came through the chat window here from Francesca. I have a general question. What happens if the first step of the registered report procedure is successful? Does the manuscript is published? And then the authors never go ahead on the second step. Okay, so first thing to say there is that for a lot of journals that the manuscript is not published at the point of in principle stage form acceptance. It's usually registered, which can lead to being published depending on whether the authors choose to register it publicly or under an embargo. But for most journals, they don't publish the protocol separately. Now this varies from journal to journal. If you go to the registered reports page Centro for Open Science and you click on the participating journals tab and then you click on the policy features spreadsheet link, you will find in one of those columns at the end, I can't remember which one now, column 17, 18, 20, whatever, you'll find if that box is ticked, whether they publish the final protocol at stage one or not as a separate article. So most of the time, most of the journals don't, some of them do. The second part of your question, what happens if the authors never go ahead with the second step? Well, in that case, it depends again on the policy of the journal. The journals I edit for, that would be considered a withdrawn registration by default. So after a period of time elapses, if the authors become non-contactable, they miss their deadline by more than X number of months and don't respond to editorial inquiries, or they simply say we'd never went ahead with the research for whatever reason, then the journal publishes a withdrawn registration which consists of the abstract of the stage one submission as accepted, a link to the protocol and a reason for the withdrawal for whatever that might be. Yeah, as Chris said, several, I put a link to the policy spreadsheet that he was referring to. Some journals will, as he mentioned, publish those, many will not. We have a registry that encourages people to register. Sometimes editors will do this, oftentimes the authors will be instructed to submit their accepted stage one manuscripts in a registry dedicated to that. But any registry, clinicaltrials.gov or other registries would also be appropriate as long as that protocol is preserved. Okay, let's see, a couple more questions. In the second step, authors add further data or more results than previously stated. This is usually not allowed unless some, so authors aren't usually allowed to submit their pre-registered experiments or studies. And also we did another whole bunch of experiments that were unregistered, that's not usually permitted. There are a small number of journals that will allow this. In fact, this is one of the policy features that is captured again on that spreadsheet toward the end somewhere. Like I think there's a heading called allows the addition of unregistered experiments after stage two or something. Most journals don't allow this because it just opens up the opportunity for more bias. So the whole point of the registered reports process is to approve a plan of action and deliver on it. And there are however exceptions. So one of them would be if researchers collect more data than they expect for the approved study because of some issue with sampling. So sometimes for example, researchers might plan to collect 200 data sets but because of something that happened, they ended up with 210 because of the way the data was collected in chunks or something. There are always minor issues which are fine and they just have to be addressed as deviations from protocol and there are ways of dealing with that. But as a general rule, don't plan to collect more data than you get approved to. Stage one reviewers are the same as stage two reviewers? Yes, almost always. But the re-uptake rate of reviewers at stage two is very high. I think because the reviewers are often quite curious about what the results are and this is the way they can find out. It's not 100% reviewers sometimes back out for all kinds of reasons or for no reason or they're unavailable. And occasionally we will add a new reviewer very occasionally. That reviewer who's brought in at stage two if they were not part of the stage one process does so under very specific restrictions. They're doing so on the basis that the method and rationale and theory and everything else is not subject to re-litigation and they're really just focusing on the interpretation relative to the evidence, whether the protocol was followed closely and whether the exploratory analysis and so on are justified. So they're still following the same stage two criteria as a normal reviewer. It often takes a little bit of explaining in those circumstances because it's a strange way to review when you haven't been part of the process earlier. But in general, stage one reviewers come back. And shall we pay twice for publication fees? You'd have to ask the journal. There's 243 journals now offering this and I can't speak for all of them. I do know at the journals that I've edit for which is a tiny fraction of this number, and there's never two APCs charged. There's only ever one at the journals which charge APCs. There's only ever one charge. And depending on the journal, you either pay that at the very end of stage two or you might pay half after stage one acceptance and half after stage two or there might be other arrangements. Just look at the journal policy. There's no, what you've got to remember about registered reports is it's an article type with a certain standardized policy, but on top of that, there's a layer of journal specific policies which operate for all article types of which we have no particular knowledge or interest. So we just, we follow the core elements of registered reports but as to whether a journal, the way it charges its fees and whatnot is really for authors to find out by going to the journal itself. Yeah, we know of those 250 journals that meet those minimal standards for a registered report article type, peer review before results are known and the potential for a promise to publish the results regardless of outcome. Beyond those two specific criteria, there's a wide range of variations. All right, quick question. Sometimes the data features may require doing additional analyses or changing the analytical methods. How does this affect the second review of the study for authors, reviewers and academic editors? Well, it depends how essential the changes are, right? So normally in my experience, having edited about 220 or 30 of these things now, usually when authors want to change an analysis, it's not usually a critical change. It's usually while actually having seen the data, we think this approach is preferable and what they tend to do then is simply report both. So you report the pre-registered analyses and then you report a separate set of post hoc analyses perhaps using different approaches and which may be more sensitive that are also of course come with a higher risk of bias. Okay, so sensitivity and bias go together. In the, there are rare situations, very rare, where the pre-registered analysis is impossible for some reason. So something went wrong, something unexpected. There was some unexpected feature of the data or the data acquisition procedure or something and the original analyses weren't possible. Now, if those problems are identified during data collection, they can be addressed as part of a reopened stage one review process where you hit the pause button, goes back to the editors and sometimes we'll talk to a reviewer and say, what do you think about changing this and this? If everyone's in agreement that that's a good change to make, then that deviation is noted in the stage two manuscript eventually along with the point in time that it was made and the amount of data that was collected by that time and so on. So it's just all very transparently reported in that way. But that's rare. Usually the situation is that both are reported. Another question, what are the thoughts on including additional measures in the registered report experiment which are not planned to be analyzed in the final report? For example, collecting iMovement data as part of an experiment because it might be interesting to answer other questions which are not the focus of the registered report. That's fine. Yeah, we get a lot of submissions like that where people are focused so they're collecting a lot of measures but their hypothesis-driven investigation is focused only on a handful of them. But they collect other things for exploratory purposes and additional analysis and that's fine. And sometimes that might go in the registered report itself. So there'll be a very brief section in the protocol which says we collected additional data like eye-tracking or additional instruments, questionnaires, whatever. And we did this with the idea of looking at more potential moderating effects. These are not central to this research but we'll look at them. And then they might explore those at stage two in an additional results section. Sometimes they might not. So these might be something that you do for the purposes of a different paper altogether. There's no particular problem with that. The only thing I would avoid is making it too bloated. You don't want to have a protocol which has a billion extra parts on it that are not centrally related to the question you're trying to answer because that could end up being a little bit distracting for the reviewers and they might get all bogged down in all this extra stuff. So, you know, if you are in a situation where you're including a large number of additional measurements that are not going to be analyzed, I would just be very clear in the stage one manuscript why you're collecting them and what you're planning to do with them in a very general sense. Even if that's simply to say, these are not going to be reported in the current registered report. They are collected for the purposes of non-related research. One question came in as I've already answered. So, I put that in through text. Ronald asks, from what I answered registered report protocols are public at the moment they are accepted? Sometimes, sometimes not. Chris will answer that. But if so, there are two questions related to this. Can this be a threshold for people who are afraid of being scooped and two, what happens with registered report protocols that are never followed up by that part two of the process? And a similar question was answered that part two also. So, to answer the first question, at the point of stage one acceptance, the protocol isn't necessarily published. It's registered and there's a difference. So, you can register a protocol publicly or you can register it on a public repository under a private embargo. Now, the journals I edit for, authors are required to do one of these. They get an acceptance and they either register it publicly on the repository so it becomes public immediately or they register it at that point in time privately under an embargo that will expire at the point that they submit the stage two manuscript or they withdraw or something happens. So, there's not necessarily any kind of published record at the point of IPA of the protocol. So, this deals with the scooping fear. If you think you're doing something that's super important and other people are gonna jump on, then you can always register it privately under an embargo at that point and that protects you from scooping whilst also creating a time-stamped entry in a registry that you can release later on. I think the second part of the question was already answered, was it? Yeah. If they're never followed up on it, if the results never come in then the journal can publish sort of a withdrawn registration notice. I suspect a lot of the academic editors on the call are on the PLOS editorial board and their policy, I'm gonna put a link to their policy in the chat window. The default there is to publish that stage one accepted protocol and there are also some APC, the total APC isn't too much more. There's a reduction in price for the first one, I believe. But you can find some more details about that on that page. And so I guess to answer the first question, given that context, can this be a threshold for people who are afraid of being scooped? I mean, yes, it is a published, it's a sightable, it's very evident that this work is going on and the authors have claimed it provenance that that's an idea that they came up with at a very specific time. So I'm just gonna post a link to that policy in the chat window here. I mean, this can go both ways, right? So sometimes authors prefer to publish their stage one protocol because they feel that it gives them some kind of, it stamps a flag in the ground and so they can show that they had this idea at a particular point in time. Other times people fear that someone else might be able to do the research quicker and they prefer to keep it private. The only answer that matters here is that you have the choice because across the spectrum of journals, there is every possibility available. There are journals which publish the protocols. There are journals which... There are journals which require registration but you can choose whether it's public or whether it's private for a limited time. So it's really all options are available so there's no need to be worried about scooping. All right, Jordy asks, my lab submitted a register report prior to the pandemic. It is still in a review. The pandemic complicates data collection and interpretation as the studies about children's screen use, which I can see why that would be a complication. Should we wait for the reviews to address these emerging issues or write to the editor now to suggest edits to our stage one proposal? That's a good question. I, it would obviously depend on how substantial the edits were. Why would we attempt to raise it now? I think if you feel that you're going to have to change the design in a substantial way or even you want to put a pause on it or if you want to introduce some kind of additional elements that deal with the fact that the study is being conducted or will be conducted in an unusual circumstance, it might be an idea just to raise that issue now because there's a possibility that the editor could then flag this with the reviewers before they see it or they could even say, let's put a pause on this and you want to resubmit a revised version now. It really does depend on the context. I, if you carry on, you'll get reviews in at some point. You probably, they'll be a bit slower than normal because everything's slower than normal. And it might be the case that you have to make additional revisions in addition to the ones that the reviewers have asked you to make at that point, in which case you may have to make those changes anyway. So it might be an idea to head that off at the pass now and speak to the editor. Yeah, yeah, I think that's a good example of reaching out to the editor. Louisa asks, our stage one reviewers, the same as stage two, that's something we've talked about a bit. Oftentimes they are, but sometimes new reviewers can come in and publication fees will depend on the journal. Mercia asks, how is the register report different from simply conducting a full simulation study? In other words, simulate data and our performant analysis on this generated data, consider some inferences. I'm not quite sure I fully understand that question. So are you suggesting, maybe we can ask for a clarification on this. Are you suggesting that the register report would be a proposed simulation? Or are you suggesting the use of simulation at stage one in order to validate a protocol? We get that sometimes, we get submissions where authors are going to do a real experiment. And what they do is they in fact simulate data for power analysis or some other feature of the design. And in fact, they pre-write their result section around how they're going to report the data. I don't quite understand the context of the question. Yeah, I think there's, she might be asking about the relative benefits of doing data simulation versus going through a register report format. And they of course have sort of different but sometimes overlapping benefits there. We can do that. Yeah, so you can submit your simulated data at stage one as a way of validating your analysis pipeline. You can say, okay, we've simulated data with various assumptions. And when it comes to conducting the real experiment where we're going to conduct it in a way that's informed by these simulations. Use it in the way that a researcher might use pilot data. Yeah, I'm sure if you have a follow-up question please submit that and we'll get more into that. Are register reports suitable for projects that wholly use pre-existing data sets? For example, Biobank and don't actually generate any novel data? Yeah, we have quite a few submissions using the A-B project. All of these data sets are up for grabs. And the only thing you need to do is look at the, again, I come back to the policy features spreadsheet. I think it's column seven, perhaps. The one which says, we'll accept registered reports for existing data. And if that box is ticked, you can submit a registered report proposing an analysis of existing data. Now, the thing to just keep in mind is that you need to make very clear in your cover letter or your pre-submission inquiry if you do one, how much observation of the data you've already had, less the better. If you have already seen the data, then you need to make clear how much you've seen and how you're going to control risk of bias and overfitting. And different journals will set different thresholds in terms of how tolerant they are to risk of bias. Some journals, if you've seen any of the data already, they will say no, we're not interested. Others will be more happy to accept the registered report provided that there is a transparent declaration of exactly what data's been seen and a rigorous method to address risk of bias. Javier asks, should the registered report be registered in the trials registration websites such as clinicaltrials.gov? And the answer is yes. The protocol, the approved protocol should either be registered on a registry such as clinicaltrials.gov, the OSF registry, the American Economics Registry, there's half a dozen or so that would be appropriate. Or the journal, again, will occasionally publish the approved protocol. So the standard there is to just make sure that that approved protocol is preserved, sometimes under embargo, and different journals will do it in different ways. Eric asks, what kind of impact does the registered report format have on the timeline for manuscript publication? Is there considerable variance in registered report publication times across journals? So I'm gonna ask a follow-up question there. What do you mean by publication time? Do you mean publication lag from the point of stage two acceptance to appearing in an issue? Or do you mean handling time? I mean, there's lots of different ways of measuring this. So I'd like to some more detail on that question because otherwise it's a very, very long answer. Okay, Eric, give a follow-up to that, precisely what you mean about the time difference and we'll answer the specific question there. Anna asks, I'm an academic editor for PLOS ONE, a journal which is presently promoting registered reports. Yay. How will something being a registered report affect what I do with the paper as an editor? Well, it will affect the way you assess the paper because PLOS ONE, like all the other adopters, has stage one criteria. So it's important, of course, to be familiar with those. Otherwise, it's really not that different from handling a regular paper except that there's no results and there's this two-step process. So sometimes one of the things to perhaps anticipate from reviewers is less familiarity with the way of approaching these manuscripts. Sometimes you can expect the occasional situation, for example, where a reviewer will accept the review request thinking it's a regular paper. Even though your invitation email might have said it's a registered report and that's what this is and so on, who reads email, so they just click accept and then they find there's no results and they say, well, this is an interesting idea but where are the results? So, as long as you're ready for those situations, and that often can require just a little bit of back and forth with the reviewer to explain again, this is the process that you're in and there are no results yet, but they will come later. That will happen maybe one out of every 20, 30, 40 review, something like that, but that's reducing over time. I think more and more reviewers are now getting familiar with the format. So plus one is coming in at a good time, I think, where, you know, because years ago, plus one declined to offer registered reports because it was too different and the volume of submissions are too great to manage it properly. But now I think there's been an appreciation that the registered reports format has matured to the point where a large-scale journal like plus one can take it without much risk. So the first thing to go back to the first answer I gave really is check the guidelines in terms of how you should be assessing the manuscripts and make sure that reviewers are following those guidelines and you're following those guidelines. And this is especially important at stage two, that you don't allow reviewers to shift the goal posts on authors and reject or recommend rejection of papers for reasons that go beyond those specific criteria. Eric provided a follow-up question for the timeline. I'm interested in how the timeline for registered reports from stage one submission to publication might differ from the timeline for self-registered manuscripts from initial submission. Right, okay, so the timeline is different because unlike the regular article type, there is a research study done in between, right? So with a stage one registered report, you're submitting before you've done the research, you get your IPA and this can typically take anything from two to four months usually to go through that process overall. There's different chunks of that time taken up in different ways. And then of course the researchers go and do the research which can take as long as it takes. Some journals set a time limit, other ones don't. And then of course it comes back at stage two and it goes through review again in a different way now with the results. So that process overall will take longer because the review process now encompasses the entire research process rather than just the end of the research process. So you can expect much longer sort of time in a very general sense. I don't know that it's starting formative though to look at it that way because it's so different to a regular article. I, we don't, this is an interesting question for meta research as to whether registered reports, if you subtract away the time researchers spend doing the research, whether they spend longer or shorter time with a journal under review than a regular article does, taking into account the fact that regular articles are often submitted to multiple journals before they find a home because they may contain methodological problems or maybe they contain results that the viewers don't like or whatever. We haven't yet got the evidence that we would need to make a strong claim as to whether registered reports take longer or shorter compared to regular articles but I'm sure that evidence will emerge in time. Are there any implications of registered reports for blind peer review? For example, a stage one protocol is published, author's identities may be revealed at stage two. So you can, I think, and correct me if I'm wrong, David but you can register a protocol on the OSF at IPA without authors on it, is that right? Do you have to have authors? Can you, can you, I know you can have anonymized links to OSF projects. So can you read, can you enter the OSF registry without authors names being printed on the protocol? You can share an embargoed registration or even an unembarked registration with an anonymous link. And you can also, if the editor registers the protocol, they can keep the authors off of it for the time being and then add them on at a later time. So that's, those are two features that are possible on the OSF, for example. So what you could do then is if you were, so I guess what you want to first do is if you're submitting to a journal that does implement double blind review, like, for example, Eneuro, I think, offers registered reports using that format. The first thing to do is be simply to talk to the editor and find out how do you actually handle this yourself. So, you know, Eneuro does require registration at the point of IPA, stage one in principle acceptance. So what then, how then do they deal with that? And I suspect it's as David describes that you've probably registered privately. So that it's fixed in time, but it's not public. And then you can share an anonymized URL to that in the stage two manuscript, which preserves the blinding as you move forward. And then only at the end of the process when you get your stage two acceptance, do you then unlock the box and all the names are revealed. But it speaks to the editor. And putting in a link right now with instructions for on the OSF registry. If you wanna share that with the editor or if their editor's on the line that are wondering how to use this with double blind peer review, that link there can show how to point a stage two reviewer to a registration and keep it anonymous. Jan asks, as an editor, how do you recruit reviewers and communicate efficiently about the different expectations for a registered report at stage one? In other words, avoid reviewers recommending reject because there aren't results. So as I said earlier, that's kind of rare that reviewers do that. I think most, this happened more in the early days, but we've kind of pushed through that window a little bit, I think. And now registered reports are much more mainstream. So reviewers are expecting them, you know, they're not so unusual to them now. These still will account to this occasionally, particularly in new fields. And if you're a post one editor, perhaps be given that you're a broad spectrum journal, you can perhaps expect this to be a higher rate than if you're in your science journal, for example. But, you know, the best thing you can do is to really flag this very clearly in your review invitation email, make it very clear this is not a regular article, there are no results yet, or if there are results, they're not the results, they're just pilot results. And that, if you do that right, my experience of the review process has been very positive. It's not that alien for people to assess science before it happens. This is something the scientific community has been doing for a long time. This is just a particular mode of doing it, which is unusual in the context of peer review. As long as you make it clear to the scientist who's doing the review, that that's what they're doing, they will shift into a mode where it works. And I'm very pleased, very happy with the standard of reviews that we get and all the journals I edit for. There's a great deal of expertise that I think can be lost in a way with the regular review process where reviewers are confronted with the results before they've really had a chance to properly analyze the methods. And I think that there's something about the registered reports review process that captures the ability for reviewers to focus specifically on issues of methodology and theory. I'm trying to determine how a registered report would be written differently. What information is necessary in a registered report and how much detail should I provide to argue my case? Lots of detail, lots particularly about methods. Dave, perhaps you could chuck a link to the OSF registry, the registered reports registry. There are now hundreds and hundreds of examples of accepted registered reports on the OSF registry and you can find one in a field that's yours and you can look at it and you can see what's expected. In general, you want to have very clear theoretical rationale. You want to have a very clear, ideally enumerated list of hypotheses, a very clear link between hypotheses and analyses and a level of detail in your methods and your analysis plan that would enable another expert in your area to replicate pretty closely or exactly what you're doing or will do without having to pick up the phone and ask you what you actually did. So this is different from a regular paper which often provides just a very coarse summary of what was done and it's all about the story. With a registered report, the story is the method and the theory and the introduction really. It's that that's the bit that really counts, that's the meat. So they will be longer, but the good news is that most journals don't set hard word limits on that part of the manuscript. So you can expect more detail, but as I say, go into the registry and I see that David's put a link there now in the chat window and have a look at some of the registered reports that have been publicly registered there for inspiration. Here's a good one about qualitative methods. As an editor, I often discuss the possibility of registered reports in our journal with the editor in chief. We have about 20% of qualitative methods research and other editors feel that RRs are not really possible for qualitative research. Would you agree? Maybe share experience or case study? I would disagree. I think they are very applicable to qualitative research. And in fact, there are a number of journals which offer them for qualitative research. And if you go again to the policy features spreadsheet, it's one of four columns toward the end. You'll find whether or not the journal offers qualitative registered reports or not. If for those journals that do, obviously there is a lot that doesn't apply if you're doing a qualitative submission. There is no quantitative analysis plan. There may not be any hypotheses to speak of, but there is still a statement of the research question, a still a statement of the methodology that will be used to collect data or information in some form and process that information and generate some kind of narrative out of it. Whether that might be if you're doing, I don't know, if you're doing grounded theory analysis, that might be what's my methodology for producing my meaning units and how am I going to extract that from my raw data and so on. So there is still a process that can be described in advance. And therefore, it's all up for grabs. I would love to see more journals writing dedicated policies for qualitative registered reports because publication bias and selective reporting is just as much of a concern in an area that does qualitative research as it is in one that does quantitative research. In the end, selective reporting and bias is a problem for all of us that are reporting on evidence. So there's no reason why it shouldn't be as applicable there as anywhere else, but it does require experts in qualitative research to lead the way and I'm not one of them. So I'm always relying on qualitative researchers in those areas to say, here is a model that works in quantitative research. How can we adapt this to qualitative research so that it's valuable rather than it can't be done end of story. It really needs someone to take that and work with it. But you will find examples of journals which are already starting to tackle this. And in fact, David can probably point to, there's a resource on the OSF about this, isn't there? Yeah, in fact, Tamron Haven is a scholar in residence who has been working with us at the Center for Open Science. She's back in the Netherlands now. Of course, Hedemica Macdorff-Wrench in a lot of things. But she's been conducting a Delphi study on soliciting feedback from experts in qualitative methodology on what would be necessary to pre-specify for most types of qualitative research. And we're working to incorporate that into a sort of a dedicated pre-registration template on the OSF registry. So that will be coming forward in the next couple of months. And there are drafts of that on the OSF. I'm looking for a couple of examples right now, but I'll put that in the chat window. So there are emerging practices in qualitative research using this. And I know that the journal Exceptional Children has given IPA to at least one study that is purely qualitative methods. So it is possible and experience with it is relatively low, but growing. And I think it's, you know, I would add to that. It's important not to see the elements that are part of the quantitative registered reports model, you know, analysis plans, power analysis, all of these things, as the defining characteristics of registered report, they're not. They are simply the features of doing a registered report in that way. But really the key elements of a registered report are simply pre-study peer review and in principle acceptance, regardless of outcome. And within that broad constructs, you can create any model. You could create a model that is completely tailored to a qualitative research field in the same way that someone would assess a qualitative research grant, how would they assess a qualitative registered report? If it can happen for grants, it can happen for registered reports. It simply requires leaders in the field to take the model and adapt it and apply it rather than assuming that just because it exists in quantitative research, it's therefore not possible to apply it in qualitative research. Okay, Ronald asks, what happens if you, we've had sort of a variation in this, but getting a little bit more pointed. What happens if you need to recruit new reviewers for stage two because the original one wasn't available? And reviewer two in the second stage recommends rejection based on criticism about the protocol from that stage one, but differently should criticism about the registered protocol be allowed at that second stage? Well, reviewers can say whatever they like, but it's not eligible. So reviewers will sometimes attempt to shift the goalposts at stage two for whatever reason. They just thought they'd say something. Oh, actually, I think there's a problem here or whatever. Usually they're in those concerns aren't major, but sometimes that reviewers might think they are and they might mention them. Whatever, it doesn't really matter. They're not eligible. So if it's an open review journal, those reviews will be published with the article. If it's not, then sometimes the editor might offer that reviewer the opportunity to write some kind of reply or commentary if they feel that there is an issue here that was overlooked. As long as that commentary is transparent about the nature of the process that went on, then that's okay, that can happen. But if the journal is following the registered reports policy as it should, then criticisms of methodology that at stage two are not permitted to influence editorial decisions. And any journal which rejects on the basis of relitigated methodology at stage two should, you should tell us about it. Because one of my roles and David is with me on this is the Center for Open Science Registry Reports Committee where we try and monitor the way the policies are being implemented across the spectrum of journals. And we want to try and capture these problems when they arise, but we rely on the community telling us about it. And in some cases, I've been able to go back to editors and say, with the consent of the authors and everyone involved, we would like to raise this issue with you that you've rejected a paper for the wrong reasons and on occasion they have actually reconsidered those decisions. So editors make mistakes, they're human, particularly editors that are unfamiliar with the format. Just, you know, if you're an author, just be aware of what's allowed and disallowed in the policy, the registered reports policies are usually very specific about the criteria on which a manuscript can be rejected at both stages. So be clear with the editors about why they're rejecting your paper if they are rejecting it. Can you explain what is meant by a positive control? So positive control can be thought of any kind of measurement which confirms that the hypotheses are in fact testable in the preregistered study. My favorite example of a positive control and the one that I use in my teaching is a paper that was published in Nature in 1993, I think, with Carl Sagan as one of the authors. And it was the Galileo space probe and it was launched to look for evidence of signs of life on other worlds by using various kind of remote sensing equipment like spectroscopy and other remote methods. And as they were leaving orbit, they thought there's an opportunity here for a positive control to be run, kind of risky one when you think about it because it's already up in space, but they decided let's turn this thing around and point it at the earth because if we can't detect signs of life using our instrumentation on earth where we know it exists then what chance have we got of detecting it on another planet? It's kind of the best positive control they've ever done, really. And so they turned this thing around, they pointed at earth and of course they found evidence of life on earth because we know it's there already. Not intelligent life. Not maybe not, we found that out about 20 years later. The upshot is positive control successful and every experiment where possible should try to build in a measurement like this, maybe not as grand as that but certainly something on that scale. So for example, if you were doing another one a more closer to home example would be if you were doing a study where you were interested in the effect of a particular drug on activity in the brain and you were giving people this drug or a placebo and then sticking them in a brain scanner, what would a positive control look like in that context? And that might be some kind of validation that in fact the drug reached serum levels sufficient to produce a change in brain activity if it was there. So that might be a blood test, confirming that the drug reached some kind of threshold level. They come in all different shapes and sizes. There's quite a big literature on them in social psychology they're called manipulation checks. In experimental lab sciences they're more typically referred to as validation experiments or positive controls but you'll find examples of this in pretty much every area that you can draw on for a registered report. All right, I think this will be the last two. We're getting close to time and there have been a lot of great questions. Thank you everyone. Danielle asks from an ethical point of view what prevents authors from having their protocols peer reviewed with one journal during the first stage and then submitting the full manuscript elsewhere if protocols are self-registered in a public or private archive? Ethically, nothing prevents that from happening. That's not a question of research ethics or publication ethics. I mean, authors aren't under any kind of legal contract to stick with a particular journal or any kind of ethical contract of any kind. They can withdraw their manuscript at any time for any reason or no reason. And the only thing that will happen is if the journal has a withdrawal registrations policy that abstracts and a link to the protocol on the registry would be published and then the authors can do whatever they like if they wanna publish their article in another journal. Just be aware that the other journal is unlikely to publish it as a registered report. They'll publish it as a regular article which means that it would go through complete peer review again. So new reviewers would look at everything, methods, every all of it and you could end up getting rejected because of your results or because of anything that reviewers didn't like. For that reason, this basically never happens. So in seven years of doing this, I've never seen editor authors withdraw a manuscript for that reason or even heard of it happening. It's very unusual because why would you do it? It has to be a very good reason for authors to pull up a guaranteed publication out of a journal and submit it somewhere else but it is your right to do that. And there's nothing stopping you doing that as an author. And finally, the last question. When authors identities are revealed in a registered report, are there concerns that reviewers will use information about the researchers? For example, perceived institutional quality, quality in quotes to argue that the researcher is, that the research is impractical or unfeasible when really this is bias against the researchers. This seems like a problem at grant review. It seems like a potential issue for registered reports. Registered reports will inherit all of those problems that exist in grant review. And to an extent, although also in regular peer review, where there are doubts over a study, the reputations of the authors are known to have a biasing influence on outcomes. So any journal which does not blind everyone to the identities of the authors is going to potentially run into this problem. I don't see it as a specific problem for registered reports compared to various other forms of review that exist, but it has to be acknowledged that it's there and it's in the landscape. And that's why some journals offer registered reports using a double-blind approach to peer review, which I think has certain advantages, of course, in preventing or reducing the risk of that kind of bias, keeping in mind that it's not going to be perfect. There are still ways of inferring often from a study who's behind it. And I would add that if, as an editor, you receive reviews about whether or not the proposed research is practical or infeasible, you know, it's up to the authors to, or it's possible to prove that wrong. And so the journal doesn't have to publish their final results if those positive controls, for example, that Chris mentioned earlier, don't pan out. And so there's very little risk for the journal to give that imperensible acceptance, even if concerns about practicality are raised earlier on because there's built-in checks to make sure that the work is conducted to high quality, regardless of where it's conducted and those same types of checks can be applied to, should be applied to every institutional submission. Okay, that's all the time we have for today. We've answered, I think about 25 questions. A couple came in through a couple of different venues. I think that's a record. So that's great to see everyone. And we'll go ahead and say goodbye and everyone stay safe out there, follow the guidelines around you about whatever the best practices are for wherever you are in the world. And just to say, if you've got further questions that we weren't able to address, you can use this hashtag on Twitter and I'll keep an eye on it over the next few days and see if I can answer any additional questions that came up. And of course, we'll do this again, hopefully soon. We're gonna be stuck in our houses for a while yet. However many weeks or months, thank you everyone.