 how long it's been in existence and its current iteration. Chris Chambers will be giving an introduction to what that is. Action describes the process of, in this case, registered reports. And we've got two presentations then on the evaluation of the concept of registered reports. We've got an all-star task today. As I mentioned, Chris Chambers, he's a professor of cognitive neuroscience at Cardiff University. He's the chair of the Registry Reports Committee for the past roughly about eight years. And he has helped support and implement registered reports at dozens of different journals, helping to bring it along to, in its current format, over 300 journals, now offered as a submission option. We're very excited to have Chris Dornis today, Stephanie LaCosta from Nature, Human Behavior. She's the chief editor and the founding editor of Nature, Human Behavior. And throughout her editorial career, Skarvula has been a strong advocate for rigorous research practices and has a research background in linguistics and experimental biology. And Shiel in Courtney Soderbergh will be talking about evaluations of registered reports that have occurred in the past few years. And Shiel is a doctoral candidate at the Polytech Institute in Adenhoven, Netherlands. She studies meta science and increasing the reliability and efficiency of psychological science. And my friend and former colleague, Courtney Soderbergh, quantitative UX researcher at Facebook, formerly at the Center for Open Science, will be talking about evaluation of registered reports that have been recently published. So without further ado, I am going to pass the screen control over to Chris. I will stop sharing my screen and allow Chris to talk about the implementation of registered reports. Thank you, David. Right, welcome everybody. Let me just share my screen here. You don't wanna see my inbox, let's go with that one. All right. Shout if that's not working. That looks good. Good, all right. So I've been tasked with talking about the concept of registered reports and really bringing any newcomers up to speed on what the format is and why we offer it, why it's now a publishing option in science. And I'm gonna stop at that point and shift on to other speakers who'll cover implementation and as David said, evaluation. So I'm just gonna introduce you to the concept of registered reports. And I think when we think about conceptual issues, it's important to go back to the very basics. And from my point of view, the motivation for registered reports stems from a rather fundamental paradox in science and academia, which I think explains part of the reasons why we are suffering from low reproducibility in many fields at the moment. And the paradox goes like this. If I ask an audience of scientists, which part of a research study that they believe should be beyond their control as scientists, after some thinking, the answer always comes back as the results. Because the results are the one part of your research that if you're a good objective, dispassionate investigator, you should try to keep at arm's length and we shouldn't try to push results around or get certain outcomes, we should just let the universe give us the answer. But then I ask the same people, which part of a study do they believe is the most important for advancing their careers, particularly if you're early career researcher and publishing in the high impact journals, achieving success in grant applications and so on, getting promoted, getting hired. Unfortunately, the answer is also the results. So we're told on the one hand, if you're a good scientist, don't try and push your results around, don't touch this bit here. But at the same time, make sure your results are amazing, make sure they're positive, clean, convincing, compelling, they tell a good story and so on. And there's no shortage of advice in the literature and in formal circles on the mastery of the message and the story and the spin that's very important for publishing in prestigious outlets. This paradox is, I think, at the root cause of many of the problems we face in science at the moment, particularly the social and life sciences. And when we put researchers on the pressure to produce great results, they will do it. If you push the individual to do something that is in their interests, they will do it in order to have a career, but we may not like how they go about doing it. And what we've learned over the last 10 years is that the scientific method simplified here in terms of a deductive cycle is prone to being broken at all different points from the pressure to get great results. And this manifests as a lack of replication, low power, so a lot of salami slicing of doing a lot of small studies rather than a smaller number of more definitive larger ones, various forms of selective reporting, otherwise termed p-hacking, changing a hypothesis to reinvent history and retrofit it onto outcomes that in fact are unexpected. This is the so-called practice of harking or hypothesizing after results are known. And then up here in the top left, the specter of publication bias, which hangs over everything, and lack of data sharing. And there are various reasons why these practices are very common. These questionable practices are common. And one of them, in my opinion, is because we put researchers under this unrelenting pressure to produce great results at all times for their careers. Now, it's this problem, which has motivated the core essence of registered reports that if we're existing in an outcome-oriented ecosystem at the moment in academia and in science, then the solution to that is to just make results a dead currency, take the value of the results out of the equation when it comes to evaluating quality. And the premise behind this is that, at least when we're talking about hypothesis testing and perhaps even more broadly in certain cases, when we think about hypothesis testing, what gives a piece of research its value, its scientific value, what gives it its merit? Well, we can look at the question. So how important and insightful is the question that is being asked? How important is it for us to answer that question? How rigorous, how reliable, how innovative is the methodology that is being proposed to address that question, to answer that question? But drawing a line at that point is saying, never the result, never the result that is produced will influence the quality. The quality itself is separate from the actual outcome of the process. And this bends us all the way back to one of Richard Feynman's famous quotes that we must be very careful not to fool ourselves. We're the easiest people to fool in science when we have particular disposition to wanting to get certain results. We are all human. We all want to see certain findings more than others in our research, but we have to build in protective measures against that. And this is what registered reports comes from. It comes from this philosophy that if results don't contain any information about study quality, then it makes no sense for editorial decisions at journals to be based upon results. Because if you do that, all you're going to do is fall prey to bias. So we should therefore actively blind the decision process to results. And this is not a new idea. This is not a new idea by any stretch that's been around for over 50 years. Robert Rosenthal was one of the first psychologists to propose the idea in the 1960s, where he thought that to get around a lot of the biasing effects of knowing results as an editor or even trying to get certain results as an author, it's very important for journals to assess research regardless of the results based upon the procedures, based upon the theory. At this time, there was nothing implemented. It was really just a proposal buried away in a chapter, but it emerged consistently throughout the next five, 10, 20, 30, 40 years in different forms in the social sciences, in medicine, as a precursor to what eventually became clinical trial registration. We saw the discussion continue in various forms that in fact, results blind evaluation or pre-study review is very important for addressing this kind of bias. And for a long time, of course, nothing much happened in many fields. Otherwise, registered reports would have been around for a long time, but it wasn't until neuro-skeptic blogger really revived the discussion around the early part of 2010, 11, 12, in light of various fraud cases and the classic study on false positive psychology. But we started thinking about this pre-registration in a very practical way in the life sciences and in the social sciences. And this discussion got revived. And then from that discussion emerged the registered reports model as we know it today. Now, I'm not going to go into a great deal of detail about the implementation because I'm sure Anne and Sirula and maybe Courtney will even cover aspects of that as well. But just to highlight the four central pillars of the model that we originally launched at Cortex in 2013. The first is that researchers decide their question, their hypotheses, procedures, their analyses before they commence data collection. And more recently, in fact, for data that already exists, this can be done after data collection but before analysis commences. And that's offered at many journals now. What happens then is that part of that review process takes place before the research is conducted. So pre-study peer review of the proposed protocol. If you pass that stage of review following revision and adjustments and improvements and all the usual kind of elements that go into methodological and theoretical review of research, then the work is granted in principle acceptance. So the journal says we virtually guarantee to publish this at stage two provided you meet certain criteria. And this is how this is a very simplified description of how this process works. For those who are uninitiated, there's this stage one peer review process that happens at the design stage before the research is conducted. Reviewers assess the theory, the theoretical framework, the rationale for the hypotheses, if there are hypotheses, the methodological rigor and robustness. As I say, this will, if all goes well, after review and revision, get in principle acceptance or IPA. And then at stage two, reviewers come back and after the research is done and the results are in to assess the compliance with study protocol and importantly whether the conclusions are based upon the evidence. And this is a very different kind of review process to normal, right? So they're not assessing things like the, how clean the results are or the subjective impact of the research or those types of elements that they are out of the equation, okay? All we're interested at this point is judging whether the research did what it was supposed to do, whether quality checks were passed and whether the conclusions are based upon the evidence. And a lot of the heavy lifting in the review process happens at this earlier stage at stage one. So we don't really litigate theory, rationale and method at stage two after results are in. So the idea of this split review process is to take away outcome bias from the evaluation so that we make results literally a dead currency when it comes to deciding which articles get accepted and eventually published. And as I say, none of these things matter. So when it comes to assessing a stage two registered report it doesn't matter whether a hypothesis is supported it doesn't matter whether results are statistically significant whether they're novel, whether they have impact all of these indicators that those of us have been around in the academic system for a little while are very familiar with they're completely meaningless in the context of a registered report. So it's a completely different way of thinking about and evaluating scientific research. I'm gonna zip through some of the advantages of the format as I see it. First of all, for the scientific community this really have to be understood on a number of levels. So we can talk about three main categories of benefit of the format, at least theoretically. One is reproducibility. So what we find with registered reports because of the amount of detail that has to go in to the methods in order for them to be assessed properly at stage one and for reviewers to be convinced that the work is worth doing and will be done properly. We see method sections that are often longer and more detailed and by that very nature more repeatable than method sections in regular articles. And attached to that is another requirement. A lot of journals set minimum requirements on sample size, power, statistical sensitivity however defined. So you often end up with larger, more definitive studies than you would get on the general track. So registered report, it's not uncommon for example at Cortex for a registered report to have a sample size it's maybe two, three or even four times larger than normal. The second main category of benefit is transparency. So there's two ways this manifests. One is the fact that for most registered report formats open data and open materials, open digital materials are typically required as well at many journals or at least strongly encouraged. So we see a lot more data sharing and a lot more sharing of the ingredients, crucial ingredients of the research. And there's also an interpretive transparency that becomes clear in a registered report at stage two because in reporting the outcomes, authors are free to conduct exploratory analysis that were unregistered and were not planned. That's absolutely fine. And there's no restriction on the kinds of creativity and exploratory thinking that can be done. But it's the main requirement is that those analyses and those outcomes are clearly distinguished from the outcomes that were based upon the preregistered confirmatory tests that went through stage one review which have to be reported in their entirety. So there's this additional layer of inferential transparency between confirmatory and exploratory outcomes which is beneficial for the reader because you can then apply the level of confidence that you're happy with in each of those separate sets of results. You don't typically see this in a regular article. In a regular article, you won't know whether or not the hypotheses were planned in advance or invented after looking at data. You won't know whether an analysis that looks like it was a hypothesis test was in fact an a priori hypothesis test or a post-talk exploratory analysis. There's no way of distinguishing them. And finally, the final category of benefit for the community is credibility. So we're taking away various forms of bias out of the equation, provided the format works as it should. So we're taking away publication bias which is logically impossible because the decision to publish is made before the results are known. So it's impossible for the results to influence the publication outcome. We're also doing our very best to limit hindsight bias. So this problem of harking or reinventing history to present unexpected results as though they were expected because it helps telling a good story to do so. And we also do our very best to prevent selective reporting in the confirmatory analysis because of course, authors have to specify their analysis plans. What about the individual? It's all well and good saying these are the benefits for science. Why do we do this at all? What's the benefit for you as an author in using this approach? It's no good. And over the years, I spent a lot of time thinking about this. It's no good coming up with solutions for improving scientific practice that only benefit the community but carry either no benefits or even severe costs for the individual. They just don't work because they rely on altruism. And we can't expect anyone to sacrifice their career for the good of the cause. So we have to make sure that we align the benefits for the individuals and for the community. And this is where our registered reports are gaining really win, I think. So first real benefit is that as an author, you get your review of feedback when it's most useful before you've done the work. There's not much point getting a great, really insightful review from a referee about something you could have done to fix a problem in your design after you've done the study and got the results in. And so it makes the review process more efficient and helpful to get that feedback at the point in time when it can actually be applied. And therefore the review process itself has a much more constructive tone. You're much more working in concert with reviewers to try and build something that's strong, robust, insightful and hopefully impactful rather than find reasons to gatekeep and shut stuff down and to reject papers. And the second main benefit, the one that really is the main big one, this is the big draw, is that as a researcher, you can get your paper accepted before you even start your research. And regardless of how the results turn out in the end. So no more playing the p-value lottery, gambling on certain results going a certain way. Otherwise you won't have your PhD or you won't get your next fellowship or your next grant. It takes all of that pointless and I think quite foolish gambling out of the equation completely. My final slide before we move on is just to give you a very brief overview of where the format is now, how the concept has evolved. So as David said at the outset, we have over 300 journals across a wide range of fields now offering the registered reports format. And we've seen a steady rise in adoption over the last seven years. Some would say this is a rapid rise. Others would say this is a slow rise. It's a very small percentage of the total number of scientific journals. But at the same time, it's a new initiative and it makes people change the way they think about their workflows. So it's not unexpected, I think that it's going to accumulate fairly gradually. On the other side, we have, there's about 600 or so fully completed registered reports that have been published. They're stage two registered reports with results. So there's a growing corpus which will continue to expand probably exponentially over the coming years. And the format most recently has expanded beyond journals into what's known as the peer community in registered reports which is an initiative that we established in April which performs stage one and stage two review of registered report preprints before they even get submitted to a journal. And we have a number of journals on board which endorse our decisions without further review or show interest in our decisions even if they will review in addition to our reviews. And this is another track that authors can take in using the format. If you wanna read more about that, there's a link here to the peer community website. I think that's all I'll say for now. I wanna make sure I don't cut into anyone else's time but my aim here is really just being to introduce you to the concept and before we move next into discussion of action evaluation implementation. Thank you. Chris, thank you. You're perfectly on time. I appreciate that. One bit of housekeeping I didn't mention is that most questions will hold until the end for discussion. If there are any clarifying questions, please feel free to raise your hand and we will address it as quickly as possible. I don't see any raised hands right now. So I'm gonna pass the screen to Stavrula to talk about the implementation of registered reports at Nature Human Behavior and perhaps beyond, thank you. Thank you, David. And thank you very much for the invitation to participate on this panel. So I'll try, I'll do my best not to repeat things that Chris has just said, even though a little bit of repetition will be unavoidable. I'm the chief editor of Nature Human Behavior and I'll be talking about our experience with registered reports. When I took on the launch of Nature Human Behavior back in 2016, I wanted to create a journal that aligned as closely as possible with principles of rega and credibility. And I also wanted to redefine what constitutes a significant advance, placing discovery research on an equal footing with confirmatory or disconfirmatory research. And also moving away the evaluation of research from the results, which as Chris said, are beyond researchers control to the research questions, the methods and how substantive the work is. So based on this description, it was an entirely a no-brainer to adopt registered reports which formalize these aims. So we adopted the format from the start in our first issue. We published an editorial calling the community to submit their registered reports to us. And in our first year in 2017, we received 28 registered reports by 2020. That number had more than traveled. And although registered reports still constitute a very small proportion of the content we receive, approximately two to 4% per year. To me, and as Chris noted, it's no surprise. Adopting, this is a radical departure from the traditional way of doing research. And any radical departure requires culture change and culture change requires time. With respect to the disciplines, we've seen come to us, submitted to us. Perhaps it's no surprise, but just over 50% of submissions over the previous four years were in psychology and 18% in neuroscience. This makes sense both in terms of where the format originated, the disciplines from which it originated and also our scope. Although we're multidisciplinary, psychology especially forms a very core component of the journal. But we have seen a development in that respect. Back in 2017, nearly all of the registered reports we received were in neuroscience and psychology. In 2020, the picture had changed, although psychology and neuroscience are still the majority at 60% approximately. 40% comes from other disciplines. Such as political science, economics, but also genetics. And we've accepted in principle, one registered report in genetics, psychiatry, communication studies, as well as a host of other disciplines, including sociology, education, public health, and many more. Now the outputs, what have we produced over the past four years? We have accepted in principle over the previous four years 24 stage one protocols. At the point where a stage one protocol is accepted in principle, we require that authors deposit it either under embargo until stage two publication or publicly at that time in a stable repository. And if the authors agree, we also offer to upload the protocol on their behalf to own fig share in a specific space dedicated to the journal. And I've included the link here for anybody who wants to peruse the AIP protocols at NHB. Out of these 24, we've published 10 stage two registered reports and we have quite a few coming up as well, which is great. We've collected those as well on our webpage and I've included the link if you'd like to go through the papers we've accepted and published, now published at stage two. This is a question that comes up frequently and of course, this is a very small number, 10 papers, but everybody asks about outcomes, the results. And we found out of these 10, three reported positive results that supported the hypothesis, five reported null or negative findings and two were inconclusive. Now, when it comes to the peer review decisions for registered reports and acceptance in principle decisions for registered reports versus regular articles. Over the previous four years for regular articles, we have sent out for peer review, 14% of the papers we received. The remaining 86% were rejected editorially. For registered reports, that proportion is 21% and registered reports are 50% more likely to be sent out for review than regular articles. This is also true, these are the aggregate data for the four years, but this is also true if you look at year on year, the data for each individual year, every year, regardless of fluctuations in overall out to review rates, registered reports are approximately 50% more likely to be sent out to review. We have sent out to review for four years, approximately 50 stage one registered reports, so our corpus is small, but looking at that corpus, 43% received acceptance in principle as compared to 37% for regular articles. So what have we learned? What have we learned as editors? Back when we were launching Nature Human Behavior, there was no other highly selected journal that had adopted the format and in fact, there were quite a few voices saying, oh, this is going to be just for non-selective journals, journals that don't select on impact. That is not true. You can select for impact, but not on the wrong principles, not on the results, but on the basis of the methods, the importance of the question, your audience and the criteria that you have for defining importance so long as you leave the results out of the question. As for authors, we also needed to learn. This is a different way of approaching editing papers and also because of the low volume, they are a very small proportion of the papers we see day to day. So especially at that time when there were far fewer journals around, the learning process for us was perhaps a bit steeper, but we had a lot of help. Chris, who is an advisor on the journal, was extremely generous with his time and with his engagement and also provided us with materials, template materials that we could modify to fit our goals. In the meantime, registered reports have grown substantially and the community of editors, those of us who offer the format and our journals has also grown substantially. So currently there is even more help and I'm sure I'm not speaking just on my behalf but there are a lot of editors in the community who will be very happy to help anybody in the audience who's interested, who's an editor interested in adopting the format. Registered reports also involve some additional work. This is partly by default in the sense that registered reports require a minimum of two rounds of peer review. One before data collection for making a decision on acceptance in principle and one after. For regular articles, the minimum number of peer review rounds is one. We see that translated in practice as well, the median rounds of peer review like nature, human behavior for registered reports after stage two is three versus two for regular articles. But we have much greater engagement in the work. So it's very different when you receive something that is all done and something where you can actually help work with the authors and the reviewers to make it as good as it is as possible for the authors then to move on to complete their project. And that's really satisfying for an editor from an editor's point of view. At the same time, this comes with greater moral responsibility. And what I mean by that is when you get an article where everything's been done, your role as an editor is to ensure that you vet it appropriately, but you have no contribution to its quality or how the author's effort and time and money were invested. When it comes to registered reports, you do have that responsibility. That responsibility is greater because the authors haven't spent the money, the time and the efforts to go off and do the research. So there is a very higher element of responsibility as an editor when you handle registered reports. But to me, that greater responsibility is actually a very positive thing. That's with registered reports, I feel we're actually making a much more important contribution to science and the scientific community. So I wouldn't count it as a job, quite the contrary. Regardless of how you see the process, what I feel that makes the greatest difference and I'm a huge supporter after five years of working on registered reports of the adoption of the format is the reward when something is ultimately published at stage two because you see something that as an editor, you have near 100% certainty that it's rigorous and credible. To me, that makes my day. So I think that's for me the most important driver. What have we learned from the author's perspective that at nature, human behavior at least and I wouldn't be surprised if that holds at other journals as well. The chance of having your stage one registered report peer reviewed and hence because more go out to peer review or some more our AIP is greater than for regular articles. From our authors who have undergone the registered report process, we've been receiving very positive feedback. People are very happy to work with a format. Chris summarized a lot of other points that are really important for science and for authors. One underappreciated aspect, I believe, of registered reports is that they are, I think one of the best training in rigorous science tools available out there. You learn how to draw a firm distinction between exploratory and confirmatory research, which in my experience as an editor is very hazy at best in the regular published literature. You learn to be very, very specific and clear about what your hypotheses are and to state them in a way that they're testable and to derive clear predictions from them. You learn how to build direct links between your hypotheses and predictions and your analysis plan. You learn robust sampling methods and, of course, there are no incentives for questionable research practices. There is no reason, the decision is not made on the results. There's no reason to engage in harking, pee hacking and you can't actually do it. So to help authors build robust protocols, we've created a template. It's very detailed. It's based, of course, on our criteria, but it's free for anybody to adopt who's interested in it. There are two aspects that I don't believe are yet fully appreciated. The value of pilot data or simulations with dummy data before as part of the stage one protocol. And I need to be entirely clear. The reason why I would encourage all registered reports, authors, except those who do replications to consider piloting their study or doing simulations with dummy data is not because we want to have a preview of what the potential results are. That's beyond the point. What is the point is registered reports, perhaps more than other research articles, require a greater investment of time and effort because of the requirements all journals have for power. Although we differs, we ask for studies to be powered at 95% or base factors equal 10 or 110, others set the limit at 90% power, others at 80%. But still, no matter what the power requirements are, that is a much higher standard than your average article. So you will need to invest more time and more money to be able to do your registered report. Before you throw all that time and effort and money, it's really good to make sure that you do everything you can in collaboration with the reviewers and the editor to make sure that you've optimized your design to study the question that you're interested in. And another aspect that's perhaps not sufficiently appreciated and perhaps that's a personal take is the value of adopting for Bayesian sampling methods rather than null hypothesis significance testing and power analyses for your sampling. With power analyses, you need to meet a certain power level depending on the journal, and you have to commit to a specific number of participants or samples that meet that requirement and there's no movement. It's not negotiable. That's it. You stick to what you have to do to collect those data. You can't peak and you can't stop. With Bayesian sampling methods, especially when you are using base factors and the journal sets a level of evidentiary level required, you can set a maximum number of participants, but if you meet the level of evidentiary support required, you can stop testing. This means that you can save resources. It doesn't work out. It won't work out every time like this. Nonetheless, it's probably a more efficient way of using your time, your money, your resources, but also more consonant with treating participants when they participate in research, they want to do it because it helps science. If there is a level of participation that may be superfluous, that's good to take on board. Sorry, I'm so sorry. I'm moving back and forth. So my take home message would be if you're an editor in the audience and you are thinking whether it would be a good idea to adopt registered reports, by all means do. I believe that there are benefits for every party involved, and especially because you get to actually play a more active role in your capacity as an editor in making sure that the work you do makes a difference. If you're a researcher, Chris summarized some of the benefits. I'd say you get the best training available, one-shot training available with doing registered reports, and at the end of the day, you can publish your work much, you are guaranteed to publish your work if you received AIP and you play by the rules at stage two. For reviewers, again, there is much more active engagement. You make a difference in a much more positive way. It's very different. It's quite disappointing for a lot of reviewers. They accept to review because they read a really interesting abstract. They're excited about the work. They start reading all good until you get to the methods, and you see, gosh, this is definitely not working. They're not answering the question they posed or the design is flawed. And all you can do is say, sorry, go back to the drawing board. That's really not satisfying. And that's not what happens with registered reports. For funders, registered reports are an investment. It's a good investment of funding money. And it's important for funders to support the initiative as it develops as well. And finally, as for every initiative, we need to evaluate it and to evaluate it continuously and also to optimize it as it develops. And that's where the role of meta-scientists is really important. So that's it from me, and I'll hand it back to David. Thank you so much, Deborah. There's a great few of questions coming through, and we're getting through those sort of in typing. Two questions came up that spoke to a point you made right at the end. So I just wanted to make sure to give the opportunity for you and Chris to also emphasize any points or feedback you've heard from the reviewers. Temerinde and another asked what's your feedback been from reviewers who have been involved in this format so far? Do either one of you want to elaborate on that point at all? Sure. Perhaps, Chris, would you like to talk a little bit about that? I can also chip in. I actually missed that because I was reading one of the questions in the Q&A, because you repeat, David. I'll read that, yeah. Two questions, but I'll just read Temerinde's here. What has it been your feedback from reviewers that have reviewed registered reports? And how do you acknowledge their contributions or what feedback have you heard from them? No, it's a good question. So, well, I'll just jump in quickly. So pretty much all positive. You don't get a huge amount of feedback from reviewers because reviewers are so busy. So you don't hear from reviewers every time. But when you do hear from them, it's almost always to say how interesting that process was or different or they felt it was more constructive. I always remember one of the first comments we had from a reviewer back in the early days of Cortex. He wrote in after the whole Stage 1 process had finished and he was like, I've never done anything like this because I feel like I can actually help them avoid problems. It's not even like grant review because you can't help people in a grant review. You're just gatekeeping. The interactive dialogue of the review process he felt was really useful. Occasionally, you get negative comments. I've had a couple of reviewers not understand the process. Like they get annoyed by the lack of results or maybe they don't read the invitation letter. Usually, if I explain a little bit more about what it involves, they get it. But occasionally, there can be misunderstandings. But broadly, I think reviewers have responded very positively. I guess on the question of rewards and incentives, there have been proposals to give reviewers more formal recognition when it comes to reviewing registered reports at Stage 1 because they are potentially influencing matters of design and theory and issues for which you might think authorship would be on the table. I have had cases where reviewers have become authors. They've had to, of course, stop being reviewers. You can't be both. There are problems when you start formalizing, I think, rewards based upon reviews of registered reports if those rewards depend upon the registered reports getting accepted because you could then incentivize kind of soft reviewing. That's not as insightful because it's somebody get something out of it. So I think you do have to be very, you have to be handled, it has to be handled delicately and carefully. But there is potential, I think, to build on it. Thank you so much for that, Chris. Now, I'm going to pass it off to Anne Shield. If you're just coming in, Anne Shield is a doctoral candidate at the Polytech Institute at, I didn't have it in the Netherlands, studying meta science and reputable research practices there. And you're going to talk about some evaluations that have occurred that you've led in directly on the registered report format. So without further ado, I'll pass it off to Anne. All right, thank you very much. Okay, I'm just trying to serve the screen share. One second. Yes. All right, does that work? Yes. Okay, perfect. All right, yeah, thanks. Thanks a lot, David. So, yeah, as you've heard my part here is to tell you a little something about evaluating registered reports and some research that I've done myself on this in particular on the questions, do registered reports reduce publication bias? And as you've heard in the, if you've been here for the whole session, you've heard one answer to this in the beginning when Chris said that they do this by default and basically, logically, I will get back to that point. But first, yeah, first start with publication bias as a working definition of this that I use is publication bias is a selective publication of positive results. And this can happen in at least two different ways. One I call reviewer bias. So that would be if authors submit both positive and negative results to journals, but reviewers and or editors prefer positive results or more critical of negative results. So negative results end up getting rejected more often than positive results. This is one way in which bias essentially would come in and we would select for positive results. And the other mechanism is file drawing. So that's from the author side who would decide to not even submit negative results to publication or not even bother writing them up. And I think it's quite, yeah, quite intuitive to think that these two processes might be related. So authors might be disincentivized to submit negative results because they anticipate reviewer bias and I think it's not worth the effort, but they could also be different aspects to it. We have some evidence for publication bias in psychology and the social sciences is where I come from. And one is indirect evidence in particular in psychology that is an implausibly high success rate. So if we look at the published literature, we typically find that an extremely large amount of studies has positive results. Mostly most studies that have looked into this at all report rates above 90%. So 90% of published articles have positive results. And this is strange because we also have relatively good data on statistical power in psychology, which is does not look as good. And as you probably know, statistical power is essentially the probability of detecting an effect if it exists, which is as several studies have suggested on average studies in psychology tend to have less than 50% power for medium sized effects. So this would mean that it's basically impossible to reach that 90% of positive results unless all hypotheses psychologists study are true and have large effects that they can detect with their small samples. So this sounds not very plausible. The other part is we also have some direct evidence of publication bias in recent years, most notably to studies by Franco Miltra and Simone Witz who looked at a grant program and have some longitudinal data of studies, basically a full population of conducted studies and followed them over time to see which of the studies end up getting written up and published. And here they also find that no results are mostly not published and a lot of them not even written up for publication, whereas studies with strong results were overwhelmingly written up and also published. So this is exactly this publication bias effect that we would expect. All right. So now, as we can talk, as Chris has explained in the first session, registered reports essentially designed to reduce publication bias. So the whole process is a structure such that the decision whether to accept or reject an article is made before the results are known. So taking the results completely out of the equation. So at stage one, before stage one, an article or proposal is reviewed and gets in principle acceptance before the data are in or at least before the data analyzed. And in the second stage at stage two, when the second run of peer review articles can only be rejected if the authors substantially deviated from the protocol or if important quality checks were failed, but not for the main results of the paper. All right. So one question is, does that work as intended essentially? And in particular, so that would mean we would expect that registered reports don't show this implausibly high success rate that we know in the general literature. And my colleagues and I ran one study on this, which is now published in Advances and Methods and Practices and Psychological Science. And our goal here was basically just to compare the positive result rate. So that is the proportion of articles that have positive results in registered reports to normal papers, which you hear called standard reports. And we did this in the discipline of psychology because that's where we have the most expertise to evaluate those papers. The way we did this was we had two samples. So our registered report sample is actually basically the full population of registered reports at the time. I should say registered reports limited to the discipline psychology. And we also excluded registered reports that did explicitly not test the hypothesis. In the end that was at the time we started the analysis that was late 2018. There was only 71 papers that we ended up with. I have to admit it was a little bit disappointing at the time because we started with a database of 150. And that ended up not being as many. I think this is definitely an analysis that I would like to repeat sometime in the future because you've already heard that there are many more published registered reports now than there are currently. We compared this to a sample of standard reports, which we randomly sampled from the literature, from a time to in a timeframe matched to the time in which registered reports have existed. And we again only included articles that tested hypothesis. We selected them through the search phase test the hypothesis. Which was like if they had this in the abstract and the abstract mostly they would be included and then we drew a random sample of 150 of a sample a little bit. So ended up with 152. And what we did then we basically measured positive results by extracting the first hypothesis mentioned in a paper. Either mentioned in the abstract or if it was not clear enough in the abstract went to the full text to see more clearly and then coded whether the authors reported that hypothesis to be supported or not supported. And what's important here is that partial support was coded as positive results. So full or partial support comes as a positive result not supported no support is counted as negative result. And we also coded whether oh yeah and I should say we double coded this coding was not blind which is certainly a limitation. We decided that it would have been very difficult to remove all mentions of registered report in our sample. But that is definitely a potential source of bias. We also coded whether studies were original studies or replication studies. And this is important because registered reports have or we know that registered reports are very popular for replication studies psychology at least for the early adopters I would say. And so we expected a lot of the registered reports to be replication studies and very few normal papers to replication studies. And we have some reason to think that replication studies are more often negative they might be conducted because of the skeptical of the original study for example. And so that might influence the reals. And we also this I won't go into I don't have the time to go into this but I'm happy to answer questions about this later. In registered reports we also looked at how the hypothesis were described or introduced. And that is because as I just mentioned we selected the comparison papers and normal papers based on this phrase test the hypothesis. And the registered reports were not selected this way. Since we don't know how representative this phrase is for hypothesis testing papers in general. We looked at how what language was used in registered reports for this as an exploratory analysis. All right, I should say that our way of sampling standard reports and the way of coding positive results was a direct replication of a study done by Daniel Fanelli in 2010. It was quite influential. He sampled 150 papers from several disciplines and compared the positive result rate across them. And he had this result that psychology had the highest positive result rate of all swear the black sheep. So we took his exact method and applied it in this new context. All right, so now the results might already be familiar to some. We see here on the left the standard reports have very high positive result rate as we would expect as we've seen other studies before. So this is 96% positive results in the normal literature and psychology compared to registered reports only 43-44% positive results. So a huge drop. What I find interesting here, yeah, that the rate we find for the general literature is really so consistent with what we've seen before above 90%. And that registered reports are really so much lower, it even surprised me, I have to admit. We also analyzed, as I mentioned before, original studies were original or replication studies. And as I mentioned, that is because we suspected that replication studies might have more negative results. And so here you see it's split, only original studies, this is only replication studies on the right. And you see that the general pattern holds for original studies only. But you also see that a lot of registered reports indeed were replication studies. So more than half of our sample in registered reports, sorry, were replication studies. And if we only focus on original studies and we see that the positive result rate in registered reports was slightly higher, was 50%. But still way lower in the extremely, again, extremely high positive result rate in the general literature. And for replication studies, yeah, the positive result rate of registered reports was a bit lower, 39%, compared to the four standard reports we found that were replication studies. Four small studies, they were all positive, which I also find a little ironic, actually, that in the standard literature we find basically the opposite effect, that the replication studies were more positive. But yeah, this is of course a tiny sample, so who knows if this would hold up. Okay, I should also say there was another study similar to ours. It came out earlier by Chris Allen and David Mellor in 2019. They audited registered reports that were published at the time in psychology and biomedicine and also looked at how many hypotheses were positive. Their method is a little different from ours, but they generally came arrived at the same very similar conclusion. Yeah, it's a little difficult to directly compare their coding to ours, because we coded the four hypotheses that we evaluated in their data set we don't know exactly which they were, so we can't directly compare them. But still I think it's good to see that we're not very far apart, so they have around 40% positive results whereas we have around 3-44%. All right, conclusion. So what we see registered reports have dramatically fewer positive results than regular studies. But we cannot establish causality in the study. This was only an observational study in particular things that are confounds that are possible that authors who choose to submit registered reports, especially in this very early cohort they might be systematically different from authors who don't do this. They might be less biased or more prone to submitting negative results anyway. And one thing that I find quite interesting is that registered reports might be used strategically for risky hypotheses that authors expect to get to write negative results for in which they anticipate could be difficult to publish as regular articles. And if that is true, we basically have a different base rate of true hypotheses in the registered report literature and the standard literature. And this would influence the results that we should expect. But we can still say that the standard literature has way too many positive results. It is really very, very difficult to explain these numbers without assuming bias. And so, yeah, in my own research, I'm one of the things I'm interested in is especially looking into this idea that what happens if registered reports are used for more risky hypotheses or what that might do to the things that we actually end up with the questions that researchers study. All right, so then to sum up, do registered reports reduce publication bias? Again, as I said, as Chris has already answered it, essentially registered reports are designed to reduce publication bias. So essentially, if the procedure is followed as intended, there is no way that articles can be selected based on the results. So I think part of this question is not empirical. But to the extent that it is, I think what's also important here is that we should verify the implementation of registered reports to see how well is the procedure followed, what are the cracks through which bias could creep in, do the editors and reviews and also the authors actually do as they should. And the other question that as I just mentioned that I'm also very interested in is to monitor the consequences of the change incentives. So as Chris mentioned in the beginning, registered reports are set up to change the incentive so that bias isn't even, the publication bias isn't desirable anymore. But I think the way in which the incentives are changed might also have consequences that we don't completely foresee. I think it's very interesting to look into that and see if this really gets us where we want to go in general and that there are no unintended side effects. All right, that's it. Thank you. Thank you very much. And thank you so much for that. And I look forward to you zooming in on those, you know, looking out some of the alternative explanations or especially kind of, I think there's a lot of interest and a lot of discussion that will happen, monitoring how the change incentives, what the consequences, perhaps sometimes unintended consequences of those might be. So I eagerly anticipate the future there. Thank you. And now, last but not least, Courtney Soderberg will be presenting on some additional evaluation of how reviewers were considering registered reports compared to comparable studies. Courtney, hand it off to you. All right. Hello, everybody. So today I'm going to be talking about a research study that I and some of my former colleagues of the Medicines team worked on while I was still at COS which was looking to evaluate how registered reports compare to the traditional publication model. So, you know, Chris and others have talked a lot about the proposed benefits of registered reports. I'm not going to elaborate on that. I'm not going to elaborate on that. I'm not going to elaborate on the labor of this fact. But, you know, in theory, if they're working as intended, they should be minimizing publication bias, improving rigor, improving the importance of the research questions that are asked. But as Anne mentioned, whether registered reports are implemented in a way that actually leads to that is that some registered report skeptics came out with also some potential costs of registered reports which were they may lead to less creative research, less innovative research, or less interesting papers. And some of the logic behind this goes that, you know, registered reports, as we've heard, require kind of a lot of upfront knowledge and upfront planning because you have to provide, you know, exactly what analyses you're going to do. You have to have a really good idea of effect sizes so you can properly power in things like that, which may be harder to get if you're kind of doing research that is really moving into a new field or new direction where there may be less information. Additionally, you know, it might be easier to do registered reports if you're doing like really small incremental work and so maybe people are just not going to transmit really interesting novel papers for registered reports because they don't feel like it's possible. You know, these are not what is supposed to be happening with registered reports but people have, you know, offer these up as potential costs and it's important that with any intervention we actually check is it having the benefits it's supposed to have without having unintended negative consequences. So we set out to test do registered reports increase the quality of research without stifling innovation? So do they have the benefits without incurring costs? There are different ways of measuring quality and innovation, right? There's no kind of canonical scale or measure for this. So we actually took two slightly different approaches. We did some factual coding of information, which I'm not really going to get into you today. And we also did subjective assessments. So if you think about what researchers are asked to do during a review process, whether it be an article or a grant, a lot of times they are specifically asked to review a paper in terms of its quality, its rigor, its innovativeness, how impactful you think this is going to be. So this is something that researchers are asked to do a lot and so we actually leveraged that sort of reviewer mindset to have people provide subjective assessments. So the way we designed this study, it was a quasi-experimental observational study. So we had participants read an actual pair of published articles in their subject area of expertise. And subject area of expertise was assigned based on the self-reported area of the researcher and based on their choice of some keywords that best fit their research. And so the actual participants were 353 psychology and neuroscience researchers. These were primarily U.S. faculty members that we did have some grad students and some international researchers. And they were reading actual published articles. So these were either registered reports or non-registered reports that had been published in journals between 2014 and 2018. And they rated these different articles on about 19 different outcome measures. So some example questions, they were always asked to compare to the average study in their area. How novel is the research question of the study? How creative is the methodology of the study? How rigorous is the analysis strategy? How justified are the conclusions based on the paper's results? So I mentioned that everybody did rated two pairs of articles, a registered report and a non-registered report. Because these were studies that actually occurred in the literature, we get quite a bit of ecological validity, right? These actually look like the types of studies that get published, registered report and non-registered report. But that does mean that there was a self-selection process that happened, right? Articles, well, researchers select articles into going down the registered report pipeline and because of the self-selection that happens, that could bias our estimation of the treatment effect in this case, whether something is a registered report or not. And so to try and really get at what the effect of being a registered report was on these perceptions of quality and innovativeness of a paper, what we decided to do was try to create matched pairs of articles. So a registered report would be matched with a non-registered report where we tried to make the articles as similar as possible on everything that isn't wrapped around that registered reportness of the paper. And so we did this using two different matching frameworks. One was a journal first matching framework. And so that emphasizes kind of what sort of publication selection criteria a journal has, right? Maybe some journals publish research than others or some journals require longer papers than others. So a journal match is going to try and kind of smooth out all of those differences between the registered report and the non-registered report. And then there was an author match version. And this really emphasizes writing and research styles, right? Maybe some authors just write really boring papers regardless of whether they're registered or not and other authors are just better at making their research sound novel. Or maybe some researchers just tend to power their studies more highly than other researchers. So that author match is going to try and smooth out those differences. I should also mention that only novel research non-replication articles were used for this study. And that was to allow that matching to happen. At the point we did this, there were very few non-registered report replication studies that were published. I think for example, it was what for? And so if, for example, we saw that registered reports were seen as a lot less novel, but most of the registered reports were replications and most of the non-registered reports weren't replications, then a very easy explanation as well. You're comparing registered reports and non-registered reports, but you're also comparing replications and non-replications. So to make a better apples to apples comparison, we only included novel research for both registered reports and their matched non-registered report articles. So within 2014 to 2018 articles that were published novel research, we ended up with 29 registered reports which was nearly all of them that fit that criteria at that point. There was like three in polysci that we kicked out just because that would have added a complexity to the study. So we ended up with 29 article pairs. So 29 registered reports and then each of those registered reports had a journal and an author matched article and these were all in psychology and neuroscience just because that's kind of where registered reports really got it to start. So before I actually show you the results, I want to kind of give you a little bit of an orientation to what I'm about to show you. So because there was that matching framework in order to estimate the treatment effect, what you're going to be seeing is basically the average of different scores. So for each pair of articles, we calculated the difference between that article and its matched article. And then we averaged those differences for the article pairs to take that amount of time to match into account. And then our analysis strategy was a Bayesian mixed effects model. So I'm going to be showing you posterior distributions with credible intervals. And then we have the 19 outcomes and they're broken up by when somebody saw them. So when we presented these articles to folks, we broke them up a little bit. So they answered a number of questions after reading the intro and the final section of questions. They answered another slew of questions after they read the results and then they read a final section of questions once they finished the whole paper and read the abstract. I should also say that we took out the word registered report from the registered report. So we did like a slight bit of redacting to the articles. We didn't go crazy because if you if we tried to redact them too much, they started looking like Frankenstein but we did take out any obvious like this is a registered report type of language. All right. So what did we find? So we saw that reviewers proceed registered registered reports to be more rigorous provide more learnings and be of higher quality than matched non-registered reports and you can tell that because you see this quite large shift to the rate in the posteriors more positive numbers mean the registered report was favored over the non-registered report period article. So we are seeing good evidence that registered reports at least under people's perceptions have the positive results they should be having. And then there's this question of what are the perceptions of those potential cons? So we do not see evidence or registered reports decreasing perceived creativity novelty or importance of research and this is pretty consistent whether people are kind of being asked before or after they see the results now those effects do tend to be smaller they do tend more towards the zero point which would be registered reports and non-registered reports are the same as compared to some of the other outcome measures where we see this stronger pro-registered report bias or pro-registered report leading so they don't really we don't see evidence for a registered report advantage for those but we also don't see evidence for a non-registered report advantage so a registered report decrement which is really what we were trying to show we want to make sure that registered reports are hurting these areas and we don't see strong evidence for that so in terms of areas of future research I definitely like to call this study initial evidence just because it it's an observational study it's very neuroscience focused it's very much what registered reports looked like when they were first being implemented right it was in that 2014 to 2018 range and it is perceptual so I think going forward I would love to see studies that actually look at do those perceptions of greater quality and rigor actually map on to less perceptual and more tangible measures of quality and rigor so for example one thing that people have kind of posited but hasn't been tested yet is that if registered reports are working as they should they should be more likely to reproduce or replicate and I think that's an open question that I would love to see somebody look at you know the big pie in the sky is to do a randomized trial so you can actually get that causal impact of registered reports I think it's probably hard to pull off but I would love to see somebody do it and then I think actually a really important point for all of this which Anne mentioned as well is these projects we're looking at papers kind of closer to the heart of when this implementation happened both in time but also in discipline and as Chris has shown in his first slide there's kind of been an exponential rise of the number of registered reports happening in the journals offering them in what sort of disciplines they're taking place in and so interventions there can be some intervention creep as interventions get further distant in time or in space from where they're originally implemented and so I do think it's important to see if these effects tend to hold for studies that maybe happened later but also in other disciplines that are maybe further removed from the registered report starting point you might think that the implementation might be less valid to the original and so you might see these effects decreased but at the same time now that there's this bigger community of supporter and registered reports for editors and for journals you might say well it's actually easier to do kind of really good faith registered report implementations now so I would love to see more work in that area as registered reports move forward. Alright and that is all. First join me in thanking all of our panelists for presenting today I'm going to take the moderator's prerogative so to speak and some questions Roland we have a few more minutes for questions Courtney thank you so much for providing that overview of that study there's a real neat way to evaluate a lot of those questions that have originally arisen about registered reports these are just going to be so boring that nobody wants to read them or anything like that so taking a look at some of those questions with reviewer's feedback has been really interesting to see a question for the panel at large Courtney you sort of mentioned with what you think some of your future directions might be so you and Anne what do you think future areas of research hold and you mentioned that you're sort of interested in how registered reports might be changing incentives and or looking into that I'm wondering if there's areas there that you're particularly curious about or that deserve additional insight yeah sure first by the way can you change my start my video again because I can't switch on my camera anymore yes okay great thanks perfect so yeah as I hinted at the end of my presentation so I'm quite curious about whether registered reports encourage authors to to study different research questions and this could be in the sense of studying different hypotheses with a different prior probability of being true it's one way but it could also be different research questions in another way and that might be a good thing that might be something that we actually want and I mean part of this is almost implied and also what Chris and and several that talked about that it's you have to do more work up front you have to be more clear and you have to find your research question differently than in other than in in in normal articles and so kind of by definition that changes the research question but I wonder how what we can expect to happen and how that might impact the science at large and so I'm currently working on this a little bit and with formal models and with simulation models basically and and trying to think what are the incentives and so how would researchers who respond to incentives and science kind of change the behavior in response anything to add to that Courtney or I think one thing I've always wondered I saw somebody have this question and I know a group at Bath is working on this but I don't remember where they are at this point you know registered reports the peer review happens before either data is collected or data is analyzed so like the peer reviewer can have a much larger effect on the study design or the analysis strategy or something like that and somebody asked you know couldn't you just do this with a normal study and just blind people to the results and I think a group at Bath is actually looking at an implementation in a journal where you do that but I think one of the one of the questions that is that going to work hinges on is what exactly is it about registered reports that is having the largest impact on various pieces you know one thing the registered reports has over that sort of model is that the reviewers can actually help you change the research design whereas in the research story happen you're just blind to results that can happen and so I think as people kind of come up with different iterations on not quite a registered report but not quite the traditional model that are kind of middle grounds understanding kind of what pieces and parts affect what part of that quality and rigor advantage will be important to know to figure out kind of like what pieces are needed for which parts um yeah the uh this question I suspect always comes to you and I'm not going to force you to get it it did come through the chat I think I always ask you what other journals in nature are sort of considering that and I won't force you to you know divulge anything that's in confidence but I'm wondering how these conversations go with colleagues with other editors um what types of recommendations you provide when you folks approach you about um expanding the format into other disciplines other types of journals etc uh I bester everybody yeah um nature communications also launched the formats adopted the formats last year scientific reports as well um it's been filed in a third journal but hasn't been announced so I started trialing um um handling uh the first few submissions before going fully live and uh it's either in the works so towards adoption in some journals or being considered in some journals I'd say that uh not in the physical sciences so as you know half of the nature journals are in the physical sciences and that there is less need for those fields to adopt the format so it's not it's not something that would provide the benefit that it does uh for the life sciences and social sciences journals but watch this space there's more coming yeah and um follow up on that I think came from Veronica in the chat um in the q&a um pointing to an NIH report recently that is looking at the um perceptions of uh registry reports in particular but pre-registration more largely I think um which researchers use um animal models and there's a lot of confusion a lot of misunderstanding not doing uh you know what these processes are necessarily um I wonder if anybody on the panel has kind of recommendations for how in this case I think the specific recommendation would be for what funders and other publishers can do to raise awareness um what what fits of confusion you might have encountered uh in your various uh in your various lives that could be opportunities for education or or or clarification I think that's something that comes up uh disciplines differ very much um to in the extent to which uh a research project uh is composed of a steps that one builds on top of the other versus you can plan everything in advance and you go forth and do it so psychology and neuroscience are like later as is clinical um clinical research uh so clinical trials are fully planned in advance very rigorous protocols for any interim analysis or any changes so this this would work well uh and very straightforwardly in other disciplines in some of the life sciences every step you take experimental step um may depend on the previous steps you took that makes the process more complicated we have peer reviewed registered reports where there were a series of five studies that each had to build on top of each other and what was done in those cases was to specify the dependencies the second way of addressing that problem is with incremental registration clearly we would need to optimize the process for incremental registration having of his wage for a very long time before they take their next step clearly we want this to be an efficient process so that's something we need to focus on and how to make sure that incremental registration is something that can happen really easily and quickly and reviewers are on board from the outset and they know what the expectations are and quick turnarounds I believe that there are solutions but um I have heard several times how it works for psychology or it might work for clinical trial but it doesn't work in cell biology and it doesn't work in cancer research so I think that there are way we just need to think harder and work with the communities that do the research to find the best way that the model could be applied for those projects that are confirmatory I believe that exploratory research if it's bound in that way um it's it's not it doesn't give any benefit if it's confirmatory if you have any hypothesis and you want to test it then register reports can be for you for your discipline we just need to figure out how to do that best yeah and I think that alludes to some of the other incentives that Anne was just mentioning a little bit before um sometimes there's this tension between our research reports just uh reconfirming a love affair that we have with hypothesis tests that might be unfounded um and I don't think that's necessarily true I think the um if one thinks the hypothesis tests are the right way to go about your particular research program this can be an appropriate way to move forward with it but that premise is not always of course true and yeah I think a big unanswered question is what's the appropriate proportion what should a very healthy um research discipline look like how would anyone go about answering that but to sort of that appropriate proportion of exploratory um to confirmatory research um it it's likely going to be something that changes over time changed by discipline changes for a large number of factors so the question that we have on the panel has um sort of opinions or insights or or think of what that what that healthy balance could or should look like from mine go ahead now so you now from my perspective science consists of two equally important parts discovery and if you don't explore you don't discover full stop and confirmation you need to make sure that you explore and you generate hypotheses to then go off and test in a confirmatory manner and that's how you discover new knowledge but you also need to ascertain whether the knowledge you think you have is actual knowledge or not and whether it's credible or not those have to go hand in hand otherwise if either of them takes precedence you either stifle discovery or you'll create science that's not credible um so for me it's a balance, it should be a balance so maybe if I can add to that this is this is an area that I find extremely important and I've also been working on a little bit of that um lately that I think um the reform movement especially in psychology where registered reports have been taken up most so far and a lot of um there's been a lot of debate on reforms and how to improve things there's been this very strong focus on hypothesis testing and I think the assumption was this is most of the things we do and if you do a pre-registered hypothesis test or one in a registered report you can still explore after the fact but I think as we've done this we've recognized, we started to recognize how what an intricate thing a really strict hypothesis test actually is so this and this leads into um several of what you mentioned about that often pilot data will be really important and that it's such an investment registered report we really want to make sure that everything is in line is lined up to have this one really severe test and there's so much that goes into that there's so much knowledge and information we need to be ready for that and a lot of that is not just falling out of a previous confirmatory study I think and so I think we need to embrace that more that there are more different things that we need to do and find out and some of those might seem boring like a feasibility study it's not super exciting but it's actually important there needs to be space for that but also not just we need outlets for that but there's also risk of bias in those studies I mean it's different yeah like you know it's not quite the same as with hypothesis test but still there are risks and I think it would be great if we could shift a bit our mindset on to think well how can we like what are the risks for different types of research and how what are the options that we have in different types of research and minimizing risk of bias without like suffocating that research you know so not like a registered performer just is not going to work for all sorts of inquiry but maybe we can have like some elements that work for some things and it would be really nice if we could explore that a bit more I think future in our last moment before I thank everyone and dismiss class so to speak I've been asked by conference organizers to share some summary points about what the what the session is included so I'm wondering if each panelist would you know provide one take home message that you think is worth worth making sure it's documented to the wider metascience community by the end of this I'll start with you I think my take home message might be that it's a win-win-win-win-win situation all around for authors, editors, funders, reviewers to actually get on board and start looking at the format as something that we need to scale up because then we will stop or we will worry less about how authors select which project to do as a registered report versus if you're doing confirmatory research the community standard at this time is the registered report where you can then go off and explore at the end of the paper as much as you want so long as you stick to what you said you wanted to do in the main part so I'm looking forward to the scaling up of the format Courtney can I put you on the spot for the take home message do you want to make sure it gets documented sure I would say that the take home message for me is that you know at least we're not seeing any we're not seeing evidence that registered reports are having these potential negative consequences that I think was feared by some skeptics so you know as long as the implementations are good faith implementations don't seem to be registered reports doesn't seem to be having these knock on unintended negative consequences and yes I think I will also echo that that registered reports are an incredibly great cool idea and they seem to be working so far also I think one thing to highlight would be that in the last years we haven't seen any great big scandal of something going terribly wrong I think that's also very interesting to note and but at the same time I also think we should be a bit cautious in how we interpret like what county gets published as registered reports because it's very confounded right like it's the with time with the people who do it with the research questions and we just should keep that in the back of our minds as we go along and see how things work out. I want to thank you on behalf of all the attendees also thank you very much for your contributions thank you for sharing your experiences and the ongoing research and we look forward to more of that in the future thank you very much and with that I'll close the session I think we'll put up the closing slide right now and encourage folks to hang out in the remote session and be looking forward for future events thank you everyone goodbye