 Okay, welcome everyone to the Center for Open Science webinar series. I am Brian Nosek, I'm Executive Director of the Center for Open Science, and also in the presentation team today is Tim Errington who is director of research at Center for Open Science. This session is to provide an overview of some new evidence that was recently published in Nature Human Behavior about registered reports and their association with rigor and greater research quality. And then ideally, we would like this to evolve, devolve into some discussion about what is it that we can as a research community do to further investigate registered reports and their potential value for advancing knowledge and accelerating progress in science. We will encourage you to raise your hand at any time. If you have questions that you would like to address, just use the raise hand feature in zoom, and we can enable your microphone to be able to speak in this context. And we will have some break points during the slides and then discussion time at the end to really try to flesh out where we stand with registered reports and what comes next. So Tim, you can begin the slide deck, and then I will give a couple of minutes of stage setting and then Tim will present the key new results. So the work that we will present was supported by the James McDonald Foundation primarily and our adventures provided some additional support. What we want to do first is provide some context for registered reports for those that are less familiar or with what current evidence there is. So next slide. So this is the cartoon version of how research gets done. You develop an idea, you design a study, you collect and analyze data, you write the report, and then you publish the report. But of course, in the standard model, there is a big barrier after writing the report before it's published, and that is peer review. And in this context, all of the incentives for researchers to get through peer review to get their goal of getting a publication are to have outcomes that are innovative, novel, exciting, and create a neat and tidy story, so that the paper is understood to be advancing knowledge and being innovative and creating something new that hasn't been seen before. And of course, as is the discussion across many meta research and other kinds of commentaries and investigations, that kind of incentive structure can create troubles for the credibility of the research that is produced at the end, and particularly in publication bias, favoring positive results over reporting of negative results. Another innovation of registered reports is to add an additional peer review process earlier in the process after the design phase before one has looked at or even collected the data. And by adding peer review here, what happens in registered reports is that one, the authors get feedback on their research designs before conducting them so they actually might be able to revise them in the context of how they will publish their research questions. But more importantly, the journal makes a pre commitment to publishing the results of the research, as long as the authors follow through with what they say they were going to do, regardless of what the outcomes are. And so the presumption of registered reports is that we will reduce or entirely eliminate publication bias based on the outcomes because the outcomes aren't known at this stage. So when peer review occurs after writing the report, that stage of peer review is not about, are the outcomes what we expected or are they exciting enough, rather, the evaluation is about, did the authors do what they said they were going to do, and did they interpret their results responsibly. So the existing evidence that has been accumulated with registered reports in practice finds that it is effective at this primary criterion, that primary criterion being, does it allow enable encourage negative results to be published, and eliminate that bias against publishing them. And here are two different investigations Alan and Miller and she'll at all that looked at this in slightly different ways, comparing registered reports with articles published in same journal or similar journals around the same time. What you see just referring to the example of the data on the right is that in standard reports, almost all of the hypothesized findings were obtained. Papers are almost perfectly successful at finding evidence in favor of their hypothesis, whereas with registered reports when no one knows what the outcome is going to be. More than half of the studies found lack of support for their initial hypothesis. Similar type of approach on the right, showing negative results are much more prevalent in registered reports, whether they are replication studies, or novel studies compared to the traditional standard model of evaluating research. So this is key evidence about the primary criterion for registered reports in how it addresses potential credibility issues in the published literature. And so the line of work tried to look at what's the whether there's implications in terms of how researchers respond to register reports, for example, by, are they less likely to cite registered reports because they are more boring, more likely to report negative evidence. The investigations that have been done to date suggest that there isn't a difference or even there might be a slight favorability to register reports in the attention that they get through citation, compared to other articles published in the same journals around the same time. So this is just one display, showing that there's very little difference between the likelihood of citation on average for register report compared to similar articles. And there's other investigations that's some of the key findings to date about registered reports. The context of the present study that we would like to summarize is wondering if this review prior to knowing the outcomes changes the actual rigor and quality of the research itself. There are a couple of possible mechanisms one could imagine for that. If I have as an author, know that I'm being peer reviewed based on the importance of my research question the quality of my methodology to test that question. And I don't have results yet. In my attention as an author will be more on the importance of the question and the quality of methodology. So I make work harder in making those more rigorous more robust in order to get through peer review. And the second possible factor is that as a reviewer, when I'm evaluating a registered report, then I might be able to see flaws and provide feedback that instead of the authors just being frustrated I wish we had thought of that before we did the research in the standard model. In a register report model that feedback can potentially be incorporated into the design and improve the overall rigor and quality of the research. There are various line reasons to expect that rigor and quality may actually be improved in registered reports. And so the current study was designed to test that. So I'll hand it over now to Tim to take us through the research. Thank you, Brian. So let me give a bit of a background into what we were doing in our research on this, this recent paper. So one thing that we did which is what some of the evidence that Brian was just summarizing beforehand is we use the existing literature publish already published records to kind of investigate this question. At the time when we began the study preparing the study materials. There were 160 16 identified registered reports of which we only use 29 that's because we were excluding replication studies. We wanted to look at novel research, largely just because that that's what's more prevalent in the traditional literature at least how it's traditionally published. And also, by looking at the scope of the disciplines that it was covering we went to the majority which was psychology and neuroscience that left us with 29 registered reports at the time. One of the very first things that we wanted to do it just identify what those comparison articles were at least from the peer review standpoint. And so, as we outlined in the paper there's two main areas that we decided to focus on. One was the author, and the other one was the journal that it was published in. And in both of those what we wanted to do is to match not just on the journal but also trying to focus in on the topic. And the year or this time span it was published right so all of these are relatively close to each other within a year of the publication of the registered report and an attempt to try to find similar research published at a similar time and focusing again on these two key aspects. Then what we wanted to do in terms of investigating this was to be able to have it essentially reviewed again and that was a lot of really fun discussion about well what's the best way to do this what we landed on was was peer reviews right is using peer review just like we review journal articles just like we review funding calls. We thought that that would be the best way and we actually mirrored that in terms of the different criteria the different questions that we were going to have the reviewers. Investigate. So let me let me walk you through a little bit more for those who didn't read all the supplemental information of the paper. Kind of how we did this especially in terms of how we match those reviewers to articles because we didn't actually do that we actually had them self select into the process. So we started with first having them identify so these are reviewers that we identified that were active researchers largely in the United States was at the majority of our sample, but we also did do some researchers that were active in the EU. But we first, when we invited them into taking our survey we had them self select into what was the research and academic field that I associate with based on my experience. So right is it cognitive psychology versus social psychology or subset. And it might be that there's none of those in that case, they would they would self exclude themselves out of the survey because we wanted to have this kind of assignment to your own expertise. So after they identified which area they were in, they would then see keywords that were matched to the registry report and the comparison articles. So essentially, once again they get presented with a screen to let them identify with their expertise which one most closely matched it and if none of them matched it, if they came into this saying I'm a cognitive psychologist but none of those keywords fit my area, they could select none and be removed from from taking the survey. I wanted to do that because of the association of being able to have an individual self identify, not by the article but by features of the discipline and the keywords that are associated with the research. What's the best expertise that they can have. Once they went through that, as I said, the keywords are brought each table each row of keywords comes from the registry report and those comparison articles I was talking about. At the end of the survey they're randomly assigned to a registered report, the one associated with the keywords and one of the two comparison articles that we're just describing, either the journal or the authors that was a random event. They were also then randomly assigned to which one they'd evaluate first. Was it the registry report, or would it be the comparison articles that's another random randomization that we built into the design. And then when the authors were going to review their very first article right either one of those two. We also treated it similar to how grant articles on the registry reports model is which is asking questions before those results are now. So we'd first break out the introduction and the methods and the results of everything except for that last study that's in the article the one that was being registered in the registry report. There were eight questions there. Then we'd show them the results in the discussion of that last study so there'd be seven more questions and how they see those results. And then we'd show them the abstract at the very end, let them ask questions about the entire paper. Globally, and there were four additional questions there. And then they like I said they'd repeat it for the other article. All right, so let me go through. And I'll go over kind of a main findings and I'll take a break and pause and see if there's any questions. So this is the main figure that we presented in our article and what you see here this is just of the introduction and the methods. And these are the posterior probability distributions for the primary estimates and this is looking at the within subject design so taking into account both both studies that were going on. And what we see here is the black horizontal line. You guys can see my cursor there's a dot that's the mean, and then the dark black and each one is the 80% credible intervals and then the light line that goes past it is the 95% credible And we ordered them right as you can see here from which one would have the highest performance advantage for registry reports so what's on the right versus the ones that would have the least performance advantage registered reports. So again if it's on the left side of zero that means that the comparative article has a higher performance advantage. So we see here and these very first ones right this is before learning about those findings of the last study, compared to the controlled articles that reviewers evaluated the registry reports last study is having a more rigorous methodology, higher quality methodology, higher estimates of what would be learned, higher quality research question. So those research questions aligned with the methodology, the quality, sorry the importance of the research question regardless of whether the results would be observed the creativity of the methods, and the novelty of the research question. And again anytime that we're spanning about zero indicates that it's not statistically significant, or at least not not statistically different from the confidence intervals from the comparison article. So after we had the participants read then the results and discussion, right so at this point now they've seen the intro the methods and now they get to see the results in the discussion. They also compare to controlled articles we saw registered reports, having a more rigorous analysis strategy, more justified conclusions based on the results, higher quality of the results are quality of the discussion, higher estimates of what would be the results, and more innovative results and more important findings, again compared to those non registered report articles. And I'll just go through the last one, the last section there. Which is when after they finish the paper and they're presented with the abstract, compared to those control articles they evaluate those registry reports is having higher overall quality producing more important discoveries, better alignment between abstract and findings and more likely to inspire a new research. And so we get between all of these what we saw is that registry reports did outperform these comparison articles and all 19 criteria and reviewers basically they were rating these registry reports either similar, right, or more positively than the non registry report comparison articles, which of course we're really interested to see. No, I'm not showing you here this was the within comparison as I mentioned because we randomized we could also do a between subject comparisons are just looking at the very first article someone rated. And we saw similar results as well but it allowed us to investigate this a little bit further and the robustness of the findings. We'll go one more slide and it will take a pause. We also are colleagues coded the objective features of the articles. And this is now just at the article level. And here's a summary of some of these open behaviors that we found. So again of the 29 registry reports and the map journal and author based comparisons we found that they were more open materials and data and pre registration compared to registry reports. The pre registrations actually an interesting one to call out only in the sense that we would expect it to be 100% right these are registered reports so this shows us that there's still improvement. And we actually some have published on the need for that improvement of just making those pre registrations discoverable there's a couple great articles that that highlight that and this reinforces that finding. The other thing that's worth calling out is that the sample size of these studies were also larger on average, then compared to the comparison articles. And showing you I think a lot of the kind of objective features that we'd expect in and more open and rigorous reproducible articles so this was actually a really nice finding to have even if the sample size is relatively small. Okay, I'm going to pause here actually I'll start here just in case there's any questions I'm going to hand it back over to Brian but I want to see if there's if there's any questions about what we've shared today so far. If there's any questions you can put them in the chat or you can just raise your hand and we can let you ask your question about the findings as they are. Okay, well you should feel free anytime to raise your hands I will continue with a couple of final slides and implications and then we'll transition to hopefully some discussion about next steps in this. Oh, we do have a question. So let me take that where the reviewers blinded to whether an article was a registered report or not. Well excellent question. There were partially blinded in the sense that we removed mention of registered reports from the articles if it said something like registered report. But it is actually very difficult to blind in total in this context, because there's a lot of factors that are associated with register reports in behaviors that are also factors in evaluating the methodology. And so what we opted to do in this context was just remove explicit mentions of register reports, but otherwise leave the papers relatively untouched. So this creates an interesting situation of it's possible conceivable and in fact, half of the time authors could detect when we asked afterwards, was this a registered report or not, they were able to say, accurately, whether it was a register report, whether it wasn't registered report. So the, so there is information there that the reviewers are drawing on to help identify whether the papers are registered reports. So then an obvious question is, well, what's the implications of that, right, if, if authors know that does it affect their assessment in a way that is different than what their assessment should be. And so what you're seeing on this figure is actually an evaluation of that question, or at least one part of the evaluation. What this is is it's the 19 different criteria outcome criteria that were evaluated. And in the rows are the, the reviewers ratings of whether they thought registered reports would improve the quality of research compared to standard articles. Substantially more means those that rated said yes I think registered reports would improve quality substantially more down to neutral no I don't think see I think there's any difference. And then the bottom there is negative no actually I think registered reports will make things worse in quality compared to standard articles. There's only one category for negative, because very few people articulated that belief. And so what this is is five bins, or six bins, pardon me, aggregating participants to be able to estimate whether there's a trend a moderation by beliefs about registered reports to their evaluations of these particular articles on these criteria. And what you can see in the aggregate, if you're just sort of looking at the figure holistically is that there is a trend. So the other way right the trend from larger effects for those that believed registered reports in get more quality to smaller effects for those that are more neutral, or even negative about registered reports compared to others. So there is some moderation observed based on the beliefs of reviewers about the quality of register reports. A second thing to notice is that none of the estimates move substantially in the other direction, even among those authors on those reviewers that believed that registered reports actually might reduce quality of research. For them, many of those credible intervals overlapped with zero, meaning that the true estimate could be negative, but the preponderance of evidence lean still positive to credible intervals that were relatively to the right still favoring registered reports, even among that skeptical group. So there's both pieces of evidence here that beliefs about registered reports do matter for how reviewers evaluated these articles, and the register reports still show favorability, even among the most skeptical just much less so across the various criteria. So this then requires some thought and unpacking about how we interpret this. So the next slide please. So if particular interest areas of interest to think about what does it mean that we see this moderation. Is it a demand effect, meaning that there's something undesirable about this moderation, or are we seeing something about real evaluations. So a few thoughts about this and then this might be a good area for discussion. The first is that peer review itself and assessments of quality is subjective. So there is not in this context, an objective sense of is this higher quality or not. It is peer reviewers read a paper. They decide based on the qualities of the paper the things the features of the methodology, whether they rate that as higher quality or not. So in that context peer reviewers look at a lot of things. For example, if peer reviewers see that there is random assignment in one paper and don't see random assignment in the other paper and rate the paper with random assignment as higher quality on methodology. Then we would say yeah they're you they have a belief about random assignment that leads them to evaluate that research as being higher quality. So blinding random assignment in that context to get a fair comparison doesn't make sense because it is the subjective assessment of the feet that feature that is informing the judgment. So in that sense, it's a natural evaluation, if you believe that features of register reports lead to higher quality, then you would expect perception of register reports to be associated with higher quality. However, there is a big concern in psychological research about demand effects that go beyond that natural evaluation. And that and just to take the register the random assignment as an example. If I have a belief that random assignment is a higher quality thing to do, all else being equal and research. Then what if I see random assignment, and then say well, I have that belief, and I, you know, there are two potential negative effects. The experimenters may also have that belief so they may want me to say right random assignments really good so I may artificially adjust my evaluations because of what the experimenter ones. But I also may artificially adjust all of my evaluations because of what I want. Well I'm a supporter of register reports. I see that this is a register reports. So I'm going to say that it looks more rigorous or the question is just as interesting or novel because of my belief in register reports, rather than because evaluating the question leads me to think that it is a higher quality. Those are very hard things to separate right because because ultimately what we need is some kind of insight into the reasoning process for that subjective assessment. How much is how I'm naturally evaluating based on my beliefs about that feature, and how much is it an artificial adjustment away from that in order to ideologically, for example bolster my belief in that. We can't eliminate that issue entirely and in fact there's a rich psychological literature finding that it is not possible to effectively eliminate that entirely. But we can find ways to address it to some degree. So for example Tim mentioned that the findings are consistent whether you do a within subjects or between subjects analysis. I must reduce the concern about demand effects to some degree in the between subjects analysis for the following reason for it to be an artificial adjustment where I recognize that this is a registered report, and then I come up with a mental decision matrix of, oh I like registered reports. And so I want to adjust my evaluation of this register report. And so I want to rate it more highly. There's a lot of noticing that has to come occur, and then deliberate adjustment in my process as a reviewer. In the between subjects, I haven't yet seen both conditions. So when I'm in the condition where the standard article is first I'm just reading an article like I usually do. It's not likely I'm thinking about registered reports, in particular, unless I've been queued to that for my participation in the study. And so there isn't a lot of effect, likely on me adjusting unusually my ratings of these different factors. So when I'm in the register report condition first, one I may not really notice the register reports until I see a comparison, right that may be more likely in the within subject. But even if I did, I don't have a benchmark to adjust against. I'm just like well I'm, oh yeah this register report and I'm making my ratings. And so I don't necessarily have a comparative element of adjusting one up and the other down by comparison. So here's a direct comparative judgments. Here's a registered report, and here is the standard report, which one is better on that dimension. That's where you maximize the likelihood of people invoking their demand or ideological adjustment of people like registered reports. I'm going to push that one up. But by separating in time, and just comparing the things that people got first, we can reduce that to some degree. But again, it doesn't eliminate it. There's always the possibility that people are over adjusting based on what they would naturally evaluate subjectively. So the ideal would be if we could have objective measures of quality, right? No, let's get rid of all this subjective evaluation. Let's objectively decide how much quality random assignment is, how much quality data sharing is, how much more quality is this feature of rigor. Obviously that's hard because they're not, they don't translate objectively very well. So the closest that we probably can get in a program of research trying to understand rigor and quality of register reports or other kinds of methods is to evaluate things like robustness, reproducibility, replicability, right? Those outcomes that one can observe in a quantitative basis of here are the findings reported through register reports, here are the findings reported through standard reports. Are the effect sizes overestimated on average when you try to repeat them? Are the effects more fragile in the context of a standard report versus one that's a registered report, et cetera, et cetera. So those will be the areas to continue to poke at and pursue on trying to understand this phenomenon. Before I go into some of the more extended things, why don't I pause there and see if there's other questions that people want to raise? There are. Yeah, there's a couple more questions there. I think we can take a pause and answer them. There's one in particular if we go back before we get, I think, into where you were going, Brian, because there's a question on that. There are some questions about the pre-registration not being prevalent within registered reports. I think that's a good one. And so somebody was, I was asking if there was a pre-registration that did not mean the Stage One Registry Report wasn't available somewhere. There's, I'm going to put in the chat, there's two articles that are worth reading. Remember, these are older papers. This is actually that issue has been described very well. Tom Hardwick did a great paper mapping the University of Registry Reports. It's a nature human behavior that basically outlined this exact issue that not all registered reports at the time were making that Stage One acceptance available. Right. That essentially being the pre-registration was not available. So that meant is it was very idiosyncratic to like, oh, was a journal or an author doing it or not. There's also I attached another article, which is a response by David Miller at Center for Open Science and Chris Chambers basically saying, thank you. Yes, let's do that. It's vital. Like, let's improve the process and make sure that it's available. And so now we have a better system to make sure that those Stage Ones are available going forward. And I think this is exactly why we want this type of research just to keep improving, even something that's as simple as the ability of making sure it follows process in terms of the materials. So I think that's a great question. That's, I defer to those articles as a great example of where to go. And I think that also hopefully explains a little bit of a question about like pre-registration versus registered reports. Like the Stage One is in essence the pre-registration, but you're right. If it's not available, then you're kind of left with, well, how do I even know? So hopefully that answered that question. I thought that was a really good one. There's another one I thought maybe we jumped to before we get into it, which was Brian, maybe I'll let you take this just because I think you asked me about it. So I'm guessing you already had a question similar to this, which is, why wasn't this article published as a registered report or an exploratory report, but the registered report is one that's a great question because we tried. Yeah, very good question. The, we did submit this particular study as a registered report. And it ended up getting rejected, not entirely, they did say if you wanted to revise and resubmit you could, but it was rejected on the basis of being too exploratory. We reviewed it as a set of questions and a variety of outcomes and we wanted to report all outcomes because we didn't have a strong theory to test other than the question of we presume based on some general evidence that registered reports may show some performance advantages. And then pursue a revision because of timeline. So one of the practical challenges of registered reports is where the time is spent on getting peer review and the commitment to publication. In this particular context, we were put this is a grant funded work that had a particular timeline where we needed to do the data collection. And by the time we got those reviews in, we thought it was too much risk based on our grant commitments to then go back through the peer review process again before we started data collection. So we went ahead with data collection, and then published it as a as a standard article. So it'll be interesting to see if in the future we can vary that. There's some additional comments that want to tackle now Priya raises a great comment. She says what about the reverse experiment, everything else about the study is the same but one is explicitly a registered report, and one isn't. It's a great way to look at demand effects in trying to equate everything except the particular feature of it being a registered report to see if people adjust based on their just general belief oh it is a register report therefore it must be better. And that would be a great study to do. The key would be how do you, which parts get built into the being the same and which so that the just saying it's a registered report or not is the difference of interest. Would the control article also be pre registered with the control article also have received comments from external reviewers, just not through a formal publication process as a registered report or not, because if authors, or if the reviewers are saying I believe registered reports for higher quality, because I believe that they've been evaluated by independent reviewers prior to doing the research, then eliminating that feature would mean that the basis of their subjective assessment why they believe that's of higher rigor is getting But then you can start to unpack and say well hang on a second if that's the basis of the belief and it isn't really anything about the content of the report itself it's not about their sampling design, or any other feature of the rigor. It's their belief about the process that generated this, and then we start to just get into the really complicated part of subjective assessment right which parts of these really are where rigor comes. And the previous example would be a great additional study to start to zero in on evidence. Tim do you have something. Yeah, and I see if I can find it somebody's done that study, or at least they attempted to do it. Yeah, I'll define it again they pre registered there's and I believe now it's been published but the reason that they can. They don't talk about it is because they, the manipulation check that they had in there about making sure that somebody could could understand if it was a registry report because that's actually part of it right you're flipping the questions you have to be able to pick up the fact that it was indeed a registry report. Didn't happen. So, so they did that as a registered report. They really can't interpret their results very well because basically they weren't able to create that manipulation that they were intending to do which is what you described Priya. Yet they still published it so I'll find it if I can real quick to see if I can put in the chat if not I'll add it to the notes, because it's a really interesting study and I would think that also that's a good benefit of registered reports which is even though it still ended up being a hard to interpret finding being from a study, it's discoverable. So I'm going to make it my job to see if I can quickly quickly find it because I do remember this one because I've looked it up myself I was excited to see the results and I was, I felt bad for them when it turned out that the large study that they didn't pan out. Great thanks for that. Patricio asks a question also says it seems that pre registration is not necessary for register reports or am I missing something depends on what you mean. It is built into the register report process in the sense that the researcher, the authors have to make a commitment to their design and their analysis plan and what they will report prior to observing the outcomes and so that stage one commitment is a pre registration in that form. And at stage two, the reviewers are evaluating the extent to which they adhered to the state what their stage one pre commitments. But pre registration is as a term can also include me that pushing that content into an independent registry. And that part is not required right it could all just be managed privately at the journal and not have it be posted to an independent registry. In addition to the definition of the Center for open science we think oh no we want that to be public that part shouldn't be private in the register report workflow at the journal. Any reader should be able to look at the register report final report and compare it with what the commitment was, rather than just rely on the reviewers to do it. I would consider when Tim pointed out that that wasn't 100% and we think it should be. That's really sort of an organizational commitment. No, if we're going to adopt these. We really want to try to promote 100% adherence to making those stage one reports public to really maximize the value of register reports, but but the point is good and that it's not incumbent on the design of this that that occur. So let's see the. Oh and the related questions stage one protocol availability rate has improved but still not perfect. 50% of journals require transparency in 2018 87% in 2020. Yeah, this is still moving in the direction of getting these protocols to be more transparently accessible over time so thank you for that comment as well. And others that you want to answer now before we go on to the next part. Nope, I think that's great. I did find that article real quick, and I answered it to Priya's question because really wanted to answer it so for those I think this is great. This will get us to kind of what are the next experiments and where's the direction research should go. But there's a paper there that was published last year in Royal society that that was attempting to do what what Priya's question is so sharing that for everyone in the audience. Great. Thank you, Tim. All right, you can go to the next slide we'll have a couple of additional comments about sort of next steps of how we can move forward through some of these issues, and then would really love the engagement and ideas that you generate as a consequence of this as well. And that's what we were just discussing, whether there is objective assessment of quality improvement that can be done and certainly using external standards, someone in the questions and answers mentioned the equator guidelines. There are a number of these where they have, if not entirely objective there at least consensus based factors that are seen as being associated with higher quality, and using some standards like equator and other reporting standards might be a mechanism of doing that, at least they're disclosing these things, in addition to doing things like study the reproducibility or robustness of the actual findings as an indicator of quality. But this will always be an area of uncertainty and opportunity to sort of really keep pushing the boundaries on evaluating research quality more generally. A second obvious next step for this area of research is to start to look more for the boundary conditions of the benefits of registered reports. We were frankly surprised at the extent to which the estimates were all to the favoring side of registered reports across these various criteria. These selected ones that have been offered as to me very plausible ones of that would favor standard reports over registered reports such as creativity or novelty or importance of the findings themselves. With the expectation that part of really understanding unpacking these data would be to start to look at trade offs. Where is it that you would choose to do register report where is it that you would choose not to. And what's the right admixture of those in the research literature to really promote the most robust research possible. The fact that we observed slight to relatively strong favoritism for register reports across all of those criteria, make some of that discussion more complicated. Are we saying now register reports should be how all research is done. That would be premature. But what is the I think these this evidence demands is that we start poking at that. Where is it that we will see the register reports continue to provide value similar to or greater than in terms of quality standard models. And where is it that the reverse will occur, because it must be the case that in some scenarios that the benefits of register reports would not be realized. And that could be by disciplinary fields, we only sample the small number of disciplines here that had. In fact, the very first register reports that came through those disciplines. And it started in a relatively narrow set of methodologies that are used across the sciences. And so it could be that as register reports becomes more common across disciplinary and methodological approaches that its benefits become more diffuse or even counterproductive in some circumstances. Because as register reports becomes more popular. It is going to be less about those early adopters that may be unique in how it is they approach register reports because it was a new thing because they wanted to try this new innovation, and now more standard practice. As we've seen in lots of areas of intervention or implementation science, the initial implementation can often show much larger effects than the implementation effectiveness as it gets diffused across the population and into the mainstream for a variety of different reasons quality of implementation, how it is that people approach it. These are some of the unusual use cases that occur in how the model gets used. So that's going to be a really important part of continuing to evaluate registered reports. Next slide please. Another thing that is an obvious one to try to do is do it as a randomized trial. This was an observational study. We did our best to try to do matching in order to make it so that we can make an inference about registered reports being uniquely associated. So the matching by author and matching by journal is intended to do a lot take care of a lot of the obvious confounding influences right different authors. Well mostly took care of that. Oh different topics. Oh mostly kept it the same. Oh different evaluation criteria by journal well matched by journal to. But it doesn't resolve all possible confounding influences, and that's a, you know, limitation of any observational research. So it would be fantastic to get a randomized trial that is a realistic kind of randomized trial for registered reports. And it's hard to think about how to do this in a realistic way that isolates the right variables of interest, right you could imagine doing a randomized trial where you say, okay labs, you're going to volunteer and if you're selected into the treatment arm then you're going to do all of your research as registered reports for the next year. And if you're in the control arm your labs going to do all research the normal way. It's not a very plausible design to pursue. My thought is the most likely plausible next step for evaluating registered reports as a randomized trial is to take advantage of a feature that occurs in the peer review process itself, relatively naturalistically. And that is in a variety of subfields of different research areas or different disciplines. It is not uncommon to ask for an additional experiment in a revised and resubmit. So the reviews come in the editor says, there's a lot to like here but there's some evidence that really needs either reinforcement or that's missing or you need to test this particular condition because we're not quite convinced by the evidence. So if we do a new experiment, then resubmit the paper then we'll consider it again for publication. That's a perfect time for randomizing registered reports. Right so in the, in the model this this approach, what would be what would occur is if the editor is going to ask for another experiment the action letter. The editors are randomized to be invited or are invited into the study if they agree to be in the study are randomized to standard treatment or register report standard treatment would just be there, get their action letter like they normally do. Register report what would be, we want to do another experiment, what we want you to do is submit the experimental design back to the journal beforehand, and then we'll make the pre commitment right follows register report model, and then everything follows the same. This would mimic a lot of features of the natural process and allow a randomized trial that could be very useful. So, there may be other innovations to pursue as well but that's one that we hope can get funded at some point and there's a number of different journals that are willing to participate in that model to really help push the boundaries of understanding for register reports. On the other side. One other important innovation that registered reports enables and that deserves some additional scrutiny and evaluation is whether the register report model can really help to facilitate alignment of incentives and and culture change approaches between different stakeholder groups, and particularly between funders and journals. The register report model is a lot like applying for a grant a joint approach between journals and funders provides an amazing opportunity to create a simplified review process and align incentives between journals and funders. And the idea of a partnership here is that authors submit their registered report proposal stage one proposal and get it peer reviewed with a budget attached. And if it makes it through that peer review process, the journal commits to publishing it, the funder commits to funding it. And everybody wins in this process, authors say what a submit one time and I get the money and the publication sounds great. Funders say, Oh, we get high quality research coming to our journal. Yeah, we're a game funders say, Oh, everything that I fund will get published, ultimately if they can follow through with the work. That's a much better return on investment than I normally get which is so much of the fun things that we fund, never did report it at all. So it is easy to make the case for those various stakeholders about the potential value for this. Another thing that it may do is actually facilitate culture change by making it much more visible that there are different stakeholder groups that are all aligned on trying to promote greater rigor transparency, pre registration all the things that come along with these reports. And so there are a number of funders there have already done this model as a pilot, and who are planning to do some in the next couple of years. And in fact, we got a grant through a National Science Foundation that's linked at the bottom of this slide to do an evaluation of the recommendations with these funder partners that are noted for whether their intervention in partnering with journals to offer this actually helps to facilitate culture change within their research communities that they're trying to support. So there's a lot of interesting potentials both on register reports for its own sake, and getting stakeholders promoting register reports for culture change initiatives more broadly. Okay, so that's it for highlighting some of the next things that are in the in the mix for register reports. There are many others that are occurring and maybe some of those will come up for discussion. We also just want to make sure that you have access to information that we just reviewed in this as we break now into open discussion. I will also point out there at the bottom that these slides that is a link to the slides themselves so that you can get all of these links as well so take a snapshot of the screen. And then you can get those slides for following up on any of this material. But for the rest of the time that we have, we can do open discussion and address some of the additional questions that have come in. Awesome. You have particular questions that have arrived that you want to tackle. Sure, yeah, there's one I just answered it's a great question about, you know, do we think there's artificial intelligence algorithms that could be used to obtain objective assessment of papers mythological quality. It's a great question because the yes we're trying. I think it's a great question to preprint of the score program that you should check out that's very related to it and there's a couple organizations as well that are trying to do that as well. In terms of, you know, basically developing algorithms that can try to do objective assessment, it'll still and this is I think a tangential area to go but a good one to think about which is it gets back to what Brian was saying which is, you know, what is it gets back to look at that gets at the kind of assessment of quality right so this is really getting closer to like not just openness but like trying to hunt into rigor. And then obviously if you want to do like reproducibility or replication you go farther and farther and more and more resources are there but I do think you know development of tools is a great idea. So I'm glad someone raised that because that connection is is a good one. Tim, anybody that would like to ask a question live feel free to raise your hand, and we will enable your talk, or if you want to type into the Q amp a or the chat. You can do that as well. Martin asks, could the register reports model work with observational studies. Yes, and in fact, there are many registered reports that have been done with observational research, including research with existing data. So I'll give one example is a project called the AI ID project made available a small sample of a very large data set looking at attitudes and identities and individual differences that's psychological research. That small sample of data was made available publicly for anybody to do exploratory analysis of different kinds of questions that they would like to investigate. And it was explicitly linked to a partnership of a dozen or so journals that were that explicitly said they'd be interested in reviewing register report proposals based on this data set. What they do is develop their stage one register report proposal on this existing observation of data, submit it to one of those journals, and when the journal accepted it for the ones that got accepted. The managers of the data set would make the confirmatory data set available to the authors, and then they would write up there, they would just rerun their scripts that they used to generate their exploratory findings, and then put the confirmatory evidence into the paper and submit the paper back to the journal. So there's a lot of interesting models, especially for data that is sequentially released for secondary data analysis, or for one's own generated observational data to be able to leverage the register reports to try to improve research rigor. So thanks for that question. I've added the link there for anyone who wants to dive dive in a little bit deeper. There wasn't the open call the ID ID project that Brian described as it's obviously a Google doc so you can kind of look at that a little more detail. Brian I'll let you answer this one there's a question there about what changes have you noticed I think Alandro asked this in editorial policies related to pre registration protocols and the publication of negative results. Another question. There have been some changes over time, and they in the broader sense of adoption of the top guidelines. So if you go to co s.io slash top. The policy framework for journals or funders or institutions to promote greater transparency and rigor in the research that they publish or fund or support in their institution. The adoption of pre registration as interest by a journal or requirements, likewise publishing negative results is increasing across journals, but increasing more slowly than some of the other areas of open scholarship. For example, engagement with open data or open code or open materials protocols, etc. Has grown much more rapidly across journal disciplines, then has pre registration particularly and commitments to reporting negative results. For journals that are sort of leading lights on that you know plus one was founded on the principle of we don't care about the importance of the results we're going to report. We're going to publish papers based on their rigor and the questions that they ask, not those whether the outcomes are positive or negative. And there are a few others that have explicitly promoted that approach. But that is not catching up as quickly as commitments to open data and otherwise the register report mechanism gets there by default, because suddenly you are publishing negative results because a lot of them are negative it turns out. It isn't an explicit policy hit statement about wanting to publish negative results in the standard form. The fact that there are now almost 300 journals that offer register reports is a positive signal and it's starting to diversify across disciplines that this will become more acceptable practice over time. Thanks for that question. I don't say anyone else yeah if anyone wants to raise their hand and say something feel free. Just add one just comment just to, because I think it's some of these have been really good to kind of push also like the boundaries of where we're going on studies the registered report RCT proposal that Brian was describing and shared on the links there so you can look at it and one please give me a little back I'd love to hear that just to kind of improve that design. Something that we did to help understand that, and especially within this disciplinary boundary that we were just talking about is also to do a survey to ask people about that hypothetical revised resubmit registered report study, kind of trying to get an understanding of like well, you know, there's tons of journals that are doing this with the authors of those journals actually be open to that. I think that that we, at least in terms of what people are willing to say of this hypothetical study that they do that and I think that's actually really helpful because what we're trying to get ahead of, and in terms of designing these studies I think like everyone else is trying to understand when those biases are going to creep in. I think, again, unless you have ultimate control over everything like the hypothetical Brian that you described of we will pick labs fund them and they must do their research a certain way. And that's just the way it has to be will always have these kind of, you know, moderating effects that influencing what we observe. So it is really interesting to think about it again I was something that went really heavily into the design of the matching of the articles in this registered reports project was thinking very heavily about that. So just want to highlight that because I think that was a nice second mini study to get into a proposed project, but I'd love to hear other people's thoughts on that when they read that RCT maybe they can send those back to us or add them to the notes. Yeah, great. And then I did see one more question the q&a that we didn't answer that's an excellent question about the study. And it's when investigating how registered reports influence publication bias does it include all reports, or only those that end up being published. For example do authors tend to follow through and publish all the results, or might there be some risk that no findings still get lost between the registration final publication. That's an excellent question because it must be the case that that's going to occur to some degree, but we don't yet have documented evidence of it occurring, or how often it occurs. So there are two ways that it will play out. One should be addressed in the peer review process, and that is, I as an author said I was going to do this, and then when I submit my stage to report something else. So that second stage is to hold me accountable to report what I said I was going to do at the outset, and if I'm not willing to do that then the paper gets rejected and it doesn't get to be published, because I'm not willing to do it, or whatever reason, unable to do it. The second is that the registration occurs but I just never submit the paper as follow up right I'm super committed to that this particular finding. I put in a register report because I want the stamp of credibility, but then I don't get the results that I want and I say, oh, we never finished the study, and I just just never submitted. There would be still a problem for publication bias in the system. There isn't yet evidence or at least a lot of evidence and may have happened on a singular anecdote I actually don't know of any examples where the findings had been suppressed, but it's going to happen. The question will be how often does it happen that findings don't get followed through once they are conducted. So this will be an important part of the ongoing research about register reports is to document and identify ways to address these kinds of challenges. Yeah, that's a really good question. Before you keep answering I have to go to another call at on the hour so I'm going to jump off and say thank you to everybody, and Tim will take us home for the remainder of the questions. So thanks again, great questions and discussion. Brian, I was just going to add one more thing and then we'll see if there's any other questions. I do see a couple more questions. One thing I was going to add on to what Brian was mentioning is, is this a really good question that was raised about, you know, follow through, essentially, that's a really interesting study in the sense that one of the other projects that I'm doing the recently project in cancer biology. We actually have that where we've had both interestingly two things happen one is a rejection of a registered report. Because of deviations that we had to do in order to execute it and the editors reviewers not believing it was it was done. So that's one one way of doing it right which is that second peer review. So why are they rejecting if it's not done. How much of it is on the author very much deviating. How much is it still on interpretation, potential gatekeeping. Another related part of that is not just doing the study like Brian was saying and then not submitting it that's obviously probably occurring would be interesting to study it, but also maybe the study was never able to be done for some other reason. Like in the cancer biology project that's happened to us either resource constraints or because they're replication projects we couldn't, we couldn't get the models to work so we actually just put the brakes on the whole thing and thus there's nothing left to submit because we actually never finished the studies we can actually get the experimentation to work. So I'd be it's a great question it'd be really good to tease apart and I'm sure there's more than just what Brian and I had said. In terms of the possible landscape or universe of those, those reasons the nice thing is, is if we can make sure that those initial stage ones are public someone can actually start to do that. All right, let's see that it looks like there's maybe a couple more questions I see Jackie asked a question how do registry reports fit in the culture change funding proposal mentioned I think. Hopefully Brian made that a little clear, the culture change funding proposals those those kind of RTIs, they're largely fit as these maps between where a funder would have a call. In terms of trying to encourage say replication research or publishing of null results, or whatever that other kind of kind of open rigorous reproducible behavior that they're trying to do. And to encourage it, you know, not just you can either make it a policy, everything we do must do that. You can have it where it's just a reward okay if you publish. Say your null findings in a journal and you send it to us will give you some monetary reward or something like that. A lot of the times what we see is it paired with with journal or multiple journals. It could be like a registered report opportunity so replication works really well for that if a funder wants to call for having a lot of replication studies, mapping that and having it be a registered report with a funder with a journal that funder journal pairing is really powerful because it one kind of reduces the barrier of okay this can get published right replication research is hard to get published currently outside of registered reports. So it's a great advantage because it also means that it's going to find its way out there right they're not funding research just to have it be kept in the file drawer they're funding it to make sure that it's out there and maximize this impact. So I think that's a great question. Hopefully I answered that Jackie. Oh, and then there's another question I'll answer. And again if somebody else was raised their hand, I'll stay on for a little bit longer. On the scale, can we identify registered reports in the literature, is there a metadata tag or similar that says this paper is a registered report. Yeah, it's another very good question so this is again I think where the technology is still catching up as registered reports become more common hopefully this starts to get improved the right way. Part of the way that we did it and I know a lot of other people who studied it is there's a Zotero library that that we have the Center for open science maintain in our policy team. We can identify which ones are registered reports of doing this with within the editor community to try to say, can you tell us which ones are registered reports, because most of them tag it but it's not in the metadata yet. So ideally that finds its way over into something like cross ref and I know there's been a lot of discussion to start making that more formal that gets you into the publisher realm. That's a great question and I think once that becomes a format that you can search on such like review articles. That'll make the research on it much easier but in the meantime I think right now we're still leaning on it being a bit of a grassroots manual effort to make sure that we're curating, which articles are registered reports, and maybe even ones which ones are not because there's been in the past I know a couple publishers that accidentally identified their articles are registered report and then had to quickly correct and remove kind of that local paper level metadata that they had on their site. So it's again a great question by by someone. Cool. I think I think I'm up to date with all the questions. I want to thank everyone again for for staying on for the whole hour. I'll wait just another minute and then if there's no other questions and I'm going to say thank you and then I'll be on the lookout for when this gets published the webinar will get published for you to share with others who missed the miss today's discussion. I think I think we've hit that point. So I want to thank everyone again for joining me and Brian and be on the lookout for for more exciting stuff from registered reports in the future. Thank you.