 Thank you for participating. This is a one-hour webinar on Pre-Results Review in Economics, Lessons Learned from Setting Up a Register of Ports in a Couple of Journals. With us today, we're very happy to have Andrew Foster, Professor in the Department of Economics and Health Services Policy and Practice at Brown University. He is the editor at Journal of Development Economics, has implemented this process and will be sharing lessons learned from that initiative. Erin A.S. Wolfe has a postdoc at the Department of Economics in the University of Constance. He has been the guest editor at the Journal of Experimental Economics, also sharing experiences with setting up Pre-Results Review in that journal. And then finally, last but not least, Alexander Ognoski, Senior Program Associate at the Berkeley Initiative for the Transparency and Social Sciences will be giving tips and tricks and lessons learned from setting up a good author workflow again with Pre-Results Review equivalent to literature reports. My name is David Miller. I'm from the Center for Open Science, a non-profit organization located in Charlottesville, Virginia. Our mission is to increase transparency and reproducibility in scientific research through meta science, through advocacy, and through building infrastructure to enable the types of actions you want to see happen. I'm going to very quickly give a little bit of definition and some disambiguations about some terms that are commonly thrown out. These practices occur in clinical research, in psychology, economics, other social sciences, and there are some very similar terms that are sometimes used in slightly different ways. So just to make sure we're all on the same page on some disambiguation, when a researcher posts a pre-analysis plan ideally to a registry that will become publicly available, that's defined as pre-registration. In clinical sciences, it's simply registering a trial or more precisely prospective trial registration. And those can and often should, I would say, include what's known of course as a pre-analysis plan, the purpose of which is to distinguish what was planned from what was reported in light of the data. Windows proposed study plans are sent to a journal and have the prospect of being guaranteed publication regardless of the main outcome of the study. That's known as register reports or in this format equivalently known as pre-results review at a journal. Registration is mainly about submitting a study design to a registry designed to distinguish confirmatory versus exploratory research processes. Registration reports is that, again, two-stage peer review process where these proposed study plans can be provisionally accepted by a journal before the results are known. And they have a similar but in some cases different benefits to each other. Pre-registration and register reports can both distinguish between confirmatory hypothesis testing and exploratory discoveries. Both of them can increase transparency into the process of the work and they can help to open the file drawer. When the journal is involved in both steps of the process that can help reduce publication bias and it can help improve study design by getting peer review at a point in the research workflow where design considerations can be addressed before the study is conducted. So with that, I will stop sharing my screen and pass it off to our first panelist, Andrew Foster. All right, so I just repaired some notes. I understand I'm supposed to talk for about 50 minutes. Is that correct? That would be great, yes. Okay, and Alex should feel free to correct everything I say that's wrong since he knows almost as much about all of this or probably twice as much. But let me just sort of start with this sort of one of the questions always comes up. Alex is like, why do we do this? And I think that the best answer is that really there are a lot of different things and I think different those of us who are involved with the JD's pilot on registered reports, we had different motivations and the funder's had motivations. I think the I'll see if you have the journal had motivations. For me I saw it as a chance to get some good papers into the journal on sort of a much more specific. You know, I thought that there would be favorable selection. It was good advertising for the journal. It would sort of speak to my colleagues in development that the JD was paying attention to what people were interested in. And this was clearly what a constituency was interested in. And so I thought it was a good opportunity to do that. So that's my sort of Machiavellian perspective. But I think, you know, I also understood the arguments. I think they're very good that, you know, that open science, sort of the focus here was really important that pre, you know, pre-results review had at least in principle the possibility of reducing the extent to which people sort of undertook work that wasn't published. Certainly, I think the evidence on that in some some places I think is quite strong. I had always felt as an editor that I tried to take that perspective, regardless of whether it was pre-results review, that I should evaluate a paper based on the nature of the question and the nature of the design and pay less attention to the results. But I also understood that wasn't the fact that I did that wasn't so apparent to authors and referees. And there may have been a fair bit of censoring that would happen even before we got ahold of it or the referee comments might be more focused on the results that I might be comfortable with. I think another motivation was we really wanted to spur, especially young faculty to take on work that was more creative, that perhaps with the less risk of coming up with an unpublishable result of one side of one type or another, they might be willing to take on something and it could be rewarded fairly early in the pipeline in the form of a prospective publication and therefore keep them from sort of, you know, waiting to do their best work until they have tenure. So those were kind of the sort of the set of things that came together that sort of made this happen. In terms of sort of, you know, what happened, we received our first publication, our first submission, so we, you know, it turned out to be fortuitous that right at the same time we were interested in doing this, Elsevier was rolling out a new platform that had registered reports built in and so I saw this as kind of an opportunity to kind of, so as soon as that platform came out in 218, we started accepting registered reports as part of this pilot. You know, I can, I now look at the records, we've moved on to another platform, but it's still retained that. We've received 172 publications through that since February 18, okay. Now it turns out about half of those were mistargeted, okay. So there's a special field that you're supposed to fill in, you know, pre-results review and then there's a second field you're supposed to do if it's a phase two submission, but in fact some people pick those things with papers that have nothing to do with registered reports or they start submitting and they don't. So I would say about half of the 172 are just mistakes, okay. So, you know, then you're down to sort of 70 or so and of those about, or 50 of those went out to referees, okay. Now we normally reject sort of about 25% of the papers that are submitted go out to referees, so this is a much higher rate than we would normally get. Now some of that is because I think the favorable selection, the people who were sort of attuned to what we were doing or sort of a little bit more experienced and so forth I think, but also we were a little bit more flexible. I think we felt like that we should, particularly as a pilot, we should be a little bit more generous if somebody was going to try this out because we wouldn't know what a good one was and what a bad one was, so let's start and be fairly flexible. And Dean Carlin who is my co-conspirator here was very much willing to take on more than his share as a result of that. The rate of submission has been really quite steady. I looked at a histogram of that. It's been about two to four months, two to four per month, you know, since February 18. There seems to be a peak mid-summer. We've seen it sort of been higher in mid-summer, lower in other periods. We presume that has to do with the academic calendar. I'm not yet seeing a peak this time round. I'm guessing it's because, you know, field trial workers in such trouble in general and people are not sure what to plan. So anyway, but it's been fairly steady and, you know, I was worried that maybe it would sort of peter out after the original excitement, but it really hasn't been the case. I think if they're just kind of sort of flowing in, you know, I think Alex and Ted have been pretty good about pushing out things like this to sort of keep it on the front of the radar of people and I think that's probably helpful. But I think it's nice to see that it's just sort of continued on. It hasn't sort of accelerated to something unmanageable, but it hasn't dropped off to zero either. In terms of our current status that of the phase one, we have sort of in the current status, we have two accepts, 28 rejects, and 12 under revision in phase one. And then in the phase two, sort of on top of that, we have two accepts and two under review. So, you know, by and large, you know, over what, you know, now, since February 18, we sort of produced essentially, you know, four, four papers. Again, probably it will be like, you know, four papers a year, I would say, going forward and steady state, which is exactly where we'd like it. We don't want it to get much bigger, because not everything is suitable for that. But we think that's, I think it's a good pace. Mostly there are RCTs that we're getting. We sort of talked about prospective observational studies, but that's not been sort of the kind of submissions we've received. Most of them I'd say are RCTs. There have been a couple of cases where people submitted pre-results, review papers, registered reports for observational data that had already been collected. And originally we weren't taking that kind of data. But sort of along the way we decided that was just too complicated because it was sort of, you know, the problem is someone would submit something and if the data was already collected, they would sort of start analyzing it. And it was difficult for them to commit for the three to four months of not looking at the data before we actually got back to them with comments, right? Whereas, if somebody has a planned data collection in four months, then we kind of have a deadline and everything seems to work better. So it's just cleaner that way. But it would work with prospective observational studies. It's just to say we haven't had those. Topic-wise, it hasn't been restricted. It's been all over the map, human health, agriculture, credit, all the different kinds of things that development economists look at. And as I said earlier, we definitely are getting, at least in the initial stage, more seasoned authors, not necessarily sort of the most senior people, but sort of more the sort of mid-race, maybe they are, you know, at the cusp of 10 or 15 to 25 ranked departments are sort of the mode. So they're sort of, you know, upper tier, but not the sort of top tier of the people submit to the journal in general, okay? In terms of my own thoughts, you know, I think one of the best or my thoughts and experiences, the referees have been very willing to help out. I think that probably reflects the spirit of development and people kind of get what we're doing. One of the things we've had gone round and round, I'm glad that David, you mentioned this, is that we have to go back and forth with the authors about the difference between a pre-analysis plan and pre-results review. You know, people say, I'm going to submit my pre-analysis plan. No, we've already submitted one. Is that enough? And no, that's not right either. So that's been a major sort of education. I think people are beginning to get it. I think that one of the surprise lessons for us is that, at least for me, as I said, I had tried always to try to focus on the design of the paper and not on the results. But I think the referees found, and I think even as an editor, we found that it was harder to judge than we expected. And I think one of the key issues is power, okay? That when you have results, regardless of whether it's significant or not significant, at least you get some sense of how tight the standard errors are. Now, people in pre-results review do these power calculations, but you know, my feeling is that the machinery we have for power calculations is really, really poor when it comes to the kinds of study that development economists do. There's sort of covariance that matter. There's the variance within clusters and across clusters. All of these things are guesses, right? And so in the end, you can't, based on a pre-analysis plan, get any idea how precise your estimate is going to be ex-post. And so when you have, and that's one of the issues that we face, and we have to just, you know, judge based on imperfect calculations. So that's been something we've thought about a bit. The complexity of data collection time for pre-review is really important, right? I mean, people were in a hurry when they submit the pre-review. They're maybe at sort of midline stage of an RCT. They want the result, you know, they want to learn what the story is before it comes in. But our referees, you know, like to take their whatever, the three months that they have. And so, you know, the, and the co-editors were busy people and sometimes their delays there. So we've tried to push them along a little faster and we have been successful in that regard. But review process takes time. And that's a barrier. That's a problem sometimes. I, you know, I want to say that Elsevier has been incredibly supportive through this process. They've really allowed us to do this. And, you know, one of the big issues was, could we allow people to submit papers and accept it for pre-review in phase one to another journal? We wanted that to be the case because we didn't want to have people, you know, avoiding us because they thought the paper might be too good. And they've been, they've accepted that. And, you know, we've had a test case. It seems to be fine. Other generals seem to be okay with that. So I think that's something to watch. But so far it hasn't been a problem. And, you know, the only thing I can say is that Elsevier did have this registered reports module, which we thought was really cool. But it didn't actually do anything. Right? It wasn't as well thought through as, you know, it would have been if I had done it. You know, if we had really sort of thought of what we needed. I mean, one of the things we sort of learned is phase one and phase two aren't actually really linked together. So you can't accept a phase one. You have to send a revision for a phase one, which isn't a very natural thing to do. And that confuses authors. It confused, you know, Dean at 1.2. And we had quite a big mess that took us about a year to unwrap. Okay? And then the final thing is that we do see there's some problem of a moral hazard and revision when people are doing the phase two. You know, maybe their incentives at least initially aren't quite what they would be if they if they didn't already have a in principle acceptance in their in their pockets. I don't know that that's a big problem and people, the authors have been responsive. We said, look, you just didn't do enough here. But it could be a problem down the road. There's a future plans. You know, I think the big thing for us is a lot of fieldwork is either suspended or has a rapid turnaround right now because it's COVID related. And that, you know, the rapid turnaround we can't handle in pre results review. The suspension is probably going to cut into the number of people doing the kind of work that fits here. You know, I think we want to encourage submissions of the right kind, the publicity and talking to people and keeping it on the, you know, on the front burner. I think it's important. And I think there's been some discussion about partners, you know, possible partnerships with project funders. There was some discussion about NSF and so forth. Is there any way we could kind of integrate the journal review with the proposal review? I think that's kind of too complicated for me, but it's certainly been something on the table. So let me stop there. Thank you so much for those comments and the sort of big picture overview of what you, like what you've seen. The, I forgot a couple of housekeeping notes before I introduce you. Everybody feel free to submit questions. There's a little Q&A button. Most of those will be held towards the end and we'll make sure to have plenty of time for Q&A. And I also want to remind folks that this webinar is being recorded and we'll make it available for sharing afterwards. With that, I will pass the baton to Ranius. Would you please share your screen and provide your insight. Thank you. Yes, thank you. Yeah, should be on now, right? Yep. Are you seeing it? Yeah, perfect. So some of this is going to be repeating what Andrew already said. So this is, this is our experiences at the, this experimental economics symposium, which was planned as a pilot that should not feel a full special issue that therefore we called it a symposium. It was initially targeted at having five to seven articles. We won't quite get there, but I'll talk about that. What I'm going to say is, is mainly focused on the, on the questions that the, the COS put in front of me. And so there's going to be some slides with a lot of contents and one slide without contents because I don't have much to say on that. That said, let's start. So the first question is, what was the motivation for implementing? And there's, of course, as Andrew said, there's different levels here. It's the question of what the motivation was for us, and then what the motivation was for the journal. So for us, we had, of course, publication bias in, in the center together with, with the file drawer problem that, that was brought up already and the incentive for writing up papers that, that don't produce exciting super strong results. That was the main, the main focus, but of course, there's a lot of side effects, positive side effects. One would be the better targeted use of resources. So if you're the 20th person to try to demonstrate an in existent relationship, that's bad. And you're only going to do that because there's 19 others that you don't know about. Sometimes you try and follow up on a paper that was a false positive. And then we all know if you, if you, if you can reproduce the original results, you have a much harder time publishing that. So it usually takes probably five to six papers to correct one false positive. Yeah, bad design choices can be spotted more often. And earlier, that was mentioned already as well, not just by referees, but also by the authors who have to think more carefully about, about their own studies, because they have to really put everything down in detail. Acquiring funds might be easier for young researchers if they have a pre acceptance already, they can go to the funding agencies and say, look, I have this, this is going to be published, no matter what comes out, wouldn't you give me the funds? At the first site, it might seem like it requires much more effort. But maybe that's not true because it does require a lot of more, so much more effort in terms of before running the pay or other the experiment, of course. But after that, after you have the pre acceptance, you no longer have the need to rewrite your paper time and again to fit a different journal. So kind of in general, in, in, in total, you might have the same workload or even less potentially. And then there's a theory paper by, by Martin Duffenberg, who first proposed this, such a mechanism already in 2007, together with, with Peter Martinson that came out in 2019. And they focus on the incentives to cheat in terms of producing your own data or P hack or, or any of that sort. Another positive side effect that I see and that I think hasn't been implemented so far in economics is that in principle, we can go back to double blind review again. Because I mean, we abolished that in economics because you can always Google the working papers that are out and you already know that the authors of the study. But now nobody's going to publish their proposals. So in principle, it would be feasible and potentially sensible to go back to double blind review. When we proposed this to, to experiment with economics, they were not as enthusiastic as we were. They did see the, the positive effects of this and also said because of that we want to give it a try. But of course, there were a number of reservations that so there were a bit more hesitant than, than we were in the beginning. And I don't know what the state is there. Maybe we find out a little bit later in the conversation. Here's some buts that came up or that come up. And this is not all from the experimental board. This is that buts that come up time and again, if you talk about pre results review. First of all, some people say there will be no submissions. There are no incentives for submitting. First of all, low quality ideas, you have to, you have to put in a lot of effort in order to get your pre acceptance. So you won't spend that much effort on a low quality idea. However, for a high quality idea, unless you have the terms that the JDE has in terms of that allowing you to go and first try the AR and then come back if the AR rejects the paper. But if you have the terms that we have at the experiment, that at Experimental Economics, then you might not get the high quality submissions because people might want to save them for, for having their chance at the AR, right? So another issue was that people might use the, the pre registration, sorry, the pre results review as a cheap improvement device. So you have a nice idea, but you don't want to spend too much time on thinking about the design. So you send it to the journal and let the referees improve your design. I mean, in the end, it's going to be an empirical question, like most of these questions. But the question is how cheap is it really? So if you already spend time to write it all up and you have to put in a lot of effort to, to match the requirements for, for a proposal, right? As Andrew was mentioning, there, so, so we didn't have this experience, the experience that we had to reject papers because the pre results, the proposals were not proposals in the sense that we wanted them. So they were mostly off the format that we wanted them and that we, we had expected in that sense, kind of experimenters in, in experimental economics seem to be closer to the idea in some sense. But given they have to, to put in so much effort, the question is whether a, a submission just for, for improvement is really so cheap. There has been the question of whether there's really need for registered reports and lab research because you can always go back to the lab and, and rerun your study if you don't find a result or whatever. Well, that of course does not solve the file drawer problem, right? So that's, in that sense, I don't see how, how there's no need. And if you look at psychology where it's at least as easy to rerun a study, they have hundreds of journals implementing registered reports. So the question is, why would that be an argument against registered reports in, in experimental economics? Question, another question that has been asked is, won't people try to cheat? Well, if we talk about faking data or p-hacking data, we know that incentives are reduced. So there's this paper by, by Duffenberg and Martinson. And it's, it's pretty obvious that that's the case because you no longer have to produce stunning results, because it's, it's all about the question. How about submitting after obtaining a null result? That of course by the, by the rules laid out in our call is fraud, right? If you, if you try and submit your, your study after having gotten the data, you don't talk about this. You don't talk about your prior experiments and then try to sell it as the final thing. Of course that runs the risk because probably referees are going to have some, some improvement suggestions for, for the, for the experiment itself. So you will have to change it. And in that sense, I don't see that this is a large danger either. Another question, will there be a seniority bias problem? Maybe, especially if you do this whole thing without double blind review. Of course it's still an empirical question, one that we can not answer so far because we haven't had the numbers. There may or there may not be judging by what we got. Will there be low quality work at stage two? Andrew was always already saying they, there might be a slight tendency for that. On the other hand, of course, there's, there's also the review at stage two. So you can't, you can't just, just not do what you promised. And then finally there was the question of, of we'll, we'll be publishing too many uninteresting results. And I mean, if, if we look at the numbers that Andrew just presented with four papers a year, that doesn't seem like there's, there, there would be more than, than the share of uninteresting results than, than other journals would, would have. So that says that's to the bots. Our submissions, what did we receive? So this, oh, I should be saying this was, yeah, I think I said this, but the setting is a bit different than at, at the JDE because we had, this is about symposium. So this has to come out at a set date. So we also had a set deadline for, for submission. Right. And we had, I think, some eight to 10 months prior to, to the deadline when we announced that. So the number of submissions is about two, two plus. We had 21 submissions for this. There was a second, 20 second in the making. They asked us whether they could still submit. We said, no, that's past the deadline and it's too much past the deadline anyway. So, so we, we refused that. Topics was, topic wise, it was very broad, perhaps even more than what you expect for standard submissions. The authors, there were 27 authors for these 21 submissions and there were three to six, depending on how you see it, with what I'd call keynote status. So, so that's people that we would potentially invite for, for a keynote speech at our local workshop. So however you might judge that. Referrer recommendations so far and decisions. So we, we didn't have, so we had final decisions. We had 90% rejects, both in terms of, of recommendations by authors and in terms of, of decisions. And that matches exactly the, the experimental economics standard, so to speak. We have three revised and resubmit decisions. And two of them have keynote authors on board. So in that sense, there might be a seniority bias. There may not be, we don't know. So the numbers are very low and hard to judge from that. What was not received that you'd like to see more of? I don't have to say much on this. So we would have liked to have more paper and more publishable papers, of course, but that's, that's what we got. Knowing what I know now, what advice would I give to others considering this format? Well, I would still say do it. Try double blind review. I think that's, that's sensible and that should get rid of the seniority bias problem. You should might want to consider the, the JDE terms in terms of allowing people to, to go to the AER first. We had the idea that you, you do this with kind of when you submit that you agree to, to cite the call for papers that was published in Experimental Economics. So Experimental Economics would at least have some benefit from being kind of a step holder for, for the AER. But, but then again, that's like a quality insurance stamp that you put on. And that's, that's a sensible reason not to do it this way. So that, that was just our idea back then that you could, or you could not consider. And then, of course, use the wealth of materials that are available. And they're just, yeah, they were very impressive. And we benefited so much from, from what was there already, that the work was, was really reduced. And, and perhaps also that made it easier for us to, to get the kind of submissions that we were expecting. One thing that I forgot to say now, and that's not on my slides is in terms of refereeing, we've had about 50% of people accepting to review. I don't know what the, what the standard at Experimental Economics is. We've had hoped that that would be, that would be higher. But yeah, that's, that's how it was. And I think that's everything I had to, no, no, no, no, future plans, of course, push for perpetuating the, the, our registered report submission option at the, at Experimental Economics, also in light of, of kind of the fact that it seems like judging by the JDE experience, it's not overwhelming what you get in terms of numbers. But it's also something that seems to be, that, that some people really, really would like to see. And, and we've had a lot of comments in that vein. And then, of course, ask additional journals to introduce it in the field. Okay. So that's all I wanted to say. And I hand over to Alex. Thank you so much. I just want to share, we have two questions. Keep those coming in and we'll make sure to address those. At the end, Alex, would you be happy to take the baton now? Yes, just a second. I'm trying to, okay. Thank you, David. Thanks to you and to Claire and for your colleagues at the Center for Open Science for hosting this webinar and for inviting us to share our experience with Registry Reports. My name is, my name is Alexander Bogdanovsky. I'm a Senior Program Associate at the Berkeley Initiative for Transparency in the Social Sciences. And in my presentation, I want to reflect a bit more on the operational perspective of the process on how to make the results for review or Registry Reports work. First, a few words about BITS, the organization that I'm representing. BITS is an initiative of the Center for Effective Global Action, which is a UC Berkeley-based hub on research development, on research hub on international development. We work to promote ethical, transparent, and reproducible research practices as means of improving the integrity of science. We do this by conducting and supporting meta-research. We do this by providing access to open science education. And let me here advertise our ongoing call for applications for our Research Transparency and Reproducibility Training, or RT2, which is taking place at September 21st to 25th. You can find details on our website. And finally, we work with various actors throughout the scientific ecosystem to help them adopt policies and research protocols to support open science. So we started working with the JD in March 2018, given that this was a novelty in the discipline. We decided to first introduce Registry Reports as a pilot. And the pilot ran between March and 2018 and November 2019. Andy spoke that the pilot was an overwhelming success. We received plenty of submissions and acceptances. But I want to take you a little bit behind the scenes and tell you how we made it happen. So in preparation of the pilot, we surveyed the editors of the journals that accepted Registry Reports. Back at the end of 2017, there were roughly 70 of those. I think there are over 250 journals accepting Registry Reports these days, which speaks of the exponential growth of the format across the disciplines. And based on the feedback from journal editors, we learned that among the biggest challenges that they faced were the low number of submissions. The other panelists spoke about that. And then the very few submissions that they did receive varied highly in terms of their structure and their quality. So it was sort of apparent that it wasn't particularly clear to authors what a Stage 1 Registry Reports submission should look like. From the author's perspective, obviously, going into a new discipline, we faced the challenge of relatively low familiarity with pre-results reviews. So we had some educating to do. And then we recognized that there are higher upfront costs for preparing manuscripts in this track compared to conventional peer review. So to address these things, we developed a series of author and editorial resources. One of these are the author guidelines for authors. We used the Center for Open Science Generic Template Guidelines to get us started on this. Over we adopted it in order to fit the disciplinary focus of the journal, but also to introduce a little bit more flexibility at Stage 2. I'll talk a little more about that later. In addition to the guidelines, to further clarify and provide examples, bring them closer to authors and reviewers, we developed sets of frequently asked questions. These were largely informed by questions that we would get asked every time we spoke about this. And then probably the most interesting item in the package of materials that we developed was the Stage 1 Submission Template. So this includes series of questions and pointers that authors can answer in order to come up with a complete Stage 1 Registry Report. Just to give you a general idea of what this looks like, this is not the actual template. However, this is a submission checklist that includes all the items found in the template. So you can see that there are some of the formatting requirements mandated by the journal, but then we also have pointers for how to report the importance of the research question, the research design, data collection strategy, empirical analysis, etc. Beyond putting out these resources, as I said, since this was breaking new ground in a new discipline, we understood that it was very important to spread the word about this. So we did targeted outreach and promotion. We used quarterly blog posts targeting the most prominent media outlets in development economics. We sent emails to authors who had pre-registered prospective experiments at the AAA RCT Registry, which is the go-to registry in economics. And as I said, we gave presentations such as this one to promote and help spread the word out. And finally, in addition to all these things, we also helped us. So this was essentially me trying to, the best of my ability to answer two questions from authors, received roughly 90 questions from different authors during the pilot phase, which kind of tells us that authors didn't need quite a bit of support in the initial stages of the process. So if you're an editor and are interested in bringing your, bringing registry reports to your journal, you can do so in six steps that we identify these based on our experience, and they may be relevant to other journals in particular in economics. First of all, decide the scope of adoption. So is this going to be a pilot like we did initially with JDE? Or is this going to be just a part of a special issue? For example, the way experimental economics is doing this will largely influence the amount of effort that adopting registry reports will mandate from the editorial team. Then appoint an editor who's going to be mostly focused on handling submissions in this track, and then also identify a roster of reviewers. Ideally, these will be folks who are already familiar with registration and pre-analysis plans and can provide constructive feedback to prospective work. Then Andy talked about this, and it's very helpful to have an editorial platform that supports a two-step peer review process that will carry over the correspondence and peer review feedback from stage one into stage two, but even if this is not in place in your journal, there are ways to work around it just the way we did. And the most importantly, prepare author guidelines, and you don't have to reinvent the wheel here. Obviously, the Center for Open Science Generic Guidelines and the materials that we put out are a great place to start. I want to talk about a few interesting editorial policy editorial points that might be consequential in this regard. First of all, think in terms of the data and research design eligibility. Are you only going to accept experiments if you're, for example, accepting secondary analysis of existing data? How are you going to ask authors to verify that they haven't looked at the data before head? So this is all important. Articulate evaluation criteria for stages one and two. As I said, we understood based on experience in the discipline that authors probably needed a bit more flexibility at stage two, so we had to modify the evaluation criteria a little bit to allow for that. Then think of in terms of the, think about the timeline to submit relative to data collection. So at what point, how early before starting to collect data must authors submit? Based on the experience of the JD, we found that roughly four months before collecting any follow-up data is recommended. However, having some sort of data, for example, baseline data for with field experiments is useful to understand the study context and conduct some power calculations. Consider how you're going to ask authors to report and evaluate report deviation at stage two. So we noticed that a lot of the journals in other disciplines explicitly ask that authors ask for permission from the editor before moving on with, before deviating from the stage one submission that was accepted, that was granted in principle acceptance. The JD, as I said, since we're dealing with field experiments and a lot of things may go out of control, we opted with a more flexible solution where authors can deviate by the need to transparently report everything and justify it in their stage two submission. Moving down the list, again, submitting to other journals after being granted in principle acceptance is another interesting issue that both Andy and Iranius talked about. I'm just going to skip that. Designating papers published in PR in Prove Results Review might give an additional incentive for authors to submit their work in this track. So for example, a system of badges or some sort of a special designation to signal that this paper is in this track underwent floral peer review. After you all do all that, you should appoint a contact person to answer questions from authors. As I said, in our case, authors did require a significant amount of support, at least in doing the pilot phase. And then finally, once you have all this in place, the chair on the top set a launch date and develop an outreach strategy. So it's very important to spread the word out. Think of what the most prominent outlets in your disciplines are. So think the blogs and the newsletters that people read and try to get on those. And then finally, consider the familiarity with pre-registration and penalized disciplines of authors in your discipline. This will mandate how much support authors will require. So if you're interested in, if you're a journal editor and are interested particularly in economics and are interested in bringing the pre-results of registered reports to your journal, we have made all the materials available at JDE slash pre-resultsreview.org. They are free to use with attribution to bits in the JDE. And then feel free to get in touch with me if you're an author or an editor with any questions or suggestions. Yeah. Thank you. Alex, thank you so much for providing that insight and that overview. We do have four questions and if more come in, we'll make sure I get to them. But I wanted to go through those and give you all the opportunity to address them. I'd be happy to chime in if there are opportunities for that opinion. But let's get me down the list. Paul asks for you, Andrew Foster. Authors, you said that they had retained the right after first stage acceptance to send the paper to another journal. If yes, doesn't that make it more likely that Journal of Development Economics publications through this channel would be predominantly null or small effects, which are traditionally harder to get into top journals? Yeah. In principle, that's possible. I'd say two things. First, we feel that our role is to serve the field. When an author asked us recently, wanted to make sure that this was really our policy and we actually wished him good luck in his submission to a higher rank journal. Because if that's published, that's good for the field. But I should also say that we do part of the sort of registered contract for us is that something gets published in the electronic version of the journal, which says that this paper was withdrawn either because it was never completed or because it was submitted to another journal. So there is actually an archival record within the JDE that this paper is published elsewhere. So we're not obviously competing with the AR in a case like that, but there is a record. And so that's what we think is important. And in some sense, whether we get credit or not, really doesn't matter. But if someone looks in the JDE, they'll see that this phase one acceptance does have a result. And we aim to keep it that way. Yeah. I think that's a great point. It's scientifically, it doesn't matter as long as the results are disseminated. And I think it remains to be seen how often authors will go down that route. I think there'll be some, you know, a little bit of obligation to the journal, especially for having gone through the pre-results review process. But, you know, if there's an extraordinarily surprising result that could be disseminated elsewhere, that's not necessarily bad. And it's, you know, of course the journal helped make that happen in the first place. Some journals have published these after stage one acceptance published the protocol. And that's something that could be cited. A couple of journals do do that. I don't think any in economics do that at the moment. And of course, a lot of authors are, you know, concerned about that as well, because it would be an earlier publication. So sometimes those can be embargoed. So there are a couple of different ways to look into that. I'm not sure if anybody else has any opinions or experience with that, or if anybody's actually seen one of these be published elsewhere yet. I know it's been a question for a couple of years, but I haven't seen that occur to my knowledge. All right, we'll go into the second question here. From Victor, if only a small percentage of studies can use registration, I think that was Andrew from your example about our focus on RCTs. How much does this do to improve science, or is it exactly the problematic part of the studies? So, you know, I don't, I think science gets improved one step at a time. And so I think, you know, if there's a set of a class of studies that are better as a result of this, science is improved. And, you know, I'm not trying to change the world here. I have a sense that within my field that there's a recognition of the value of this. And if us doing that, us doing that, you know, causes the field to be a little bit better, that seems enough. Now, obviously, you know, if this webinar has, I think, what its desired effect, what we did in the JD will actually have bigger legs and will actually have a larger impact. But that's not why I did it. And we do know, and I don't have sort of a summary at my fingertips amongst what's been published amongst many disciplines so far. But certainly the format can be applied beyond RCTs or beyond field experiments. I think it's particularly well suited to those for a variety of reasons. A lot of this was sort of designed with a randomized controlled trial in mind, but it's applicable to certainly to any hypothesis testing research, particularly null hypothesis significance testing, if that's important to the inference of the study. A journal would be well put to consider that as a submission. But I think a lot of journals taking this up for the first time are, I think, for decent reasons, focusing on a certain set of studies just to experiment, gain experience, develop best practices, and depending on how the fields evolve, I think the future will be different. We'll just see how many different design types would be appropriate for the format. Anonymous writes, and we kind of got to this in a couple of different answers, but are the PRRs normally published so that authors and reviewers can see concrete examples of how such a report looks like? The short answer, yeah. I think the answer is they wouldn't normally be published, but some are published, where actually they're not published in the journal. We actually are maintaining a website that is sort of separate from the JD because we're just dealing with that would have been too complicated. The Dean Carlin's Outfoot put together, it's quite an elegant thing and it tracks the phase one submissions. It has examples of successful ones. It has the templates that Alex made and it has a user interface that Dean and I can access and so we can update things. One interesting sort of anecdote is that in one case, the author wanted to be sure that we wouldn't publish details of the PR because he was concerned that his subjects might actually read it and it would change their behavior. I don't know how legitimate that concern was. Thank you, David, for putting it up there. I think the idea that we would like to get those things available through the web, through that website, but it's really up to authors what they want to release, when they want to release it. We just wanted to make sure we had a track record so we could sort of answer questions like how many were submitted and what was the status. Yeah, I think it's a really valuable resource too to see what's been accepted and it really gives a good insight, a good registry basically. We generally ask the authors, can we make this available to other reviewer or other people? So we're not doing that at Experimental Economics, but in some sense, I don't see much of a need because as you said, the templates are there and what we received looked pretty much like what we expected so that was good. I'll share a link in a moment to sort of some similar registries with other journals, but let's move on. So you're just watching me look through my bookmarks there. Viola asks, Andrew, again at JED, mentioned that it's often difficult to judge the power of a study. I think that's an understatement. Could you maybe give examples of how authors can do really well in describing their power calculations? Well, I mean, the stuff you teach in statistics class and I think that's okay. I mean, my inclination, what I always recommend my students is to actually construct a data generating process, simulate out the data and run the regressions you're going to run. And so I think at least when you do that, you're explicit about what your assumptions are. You could even provide this data code of the simulation, for example, so that people could see it and play around with it if they wanted. So I've never seen that done. It just came to me just now. But I do think this power issue is something that we want to think about long-term on this. But I think a lot of it is unanswerable questions because it relies on stuff that we just don't know in a particular context. Yeah. And after results are known, it might be tempting at that point to, of course, use that for what turns out to be a post hoc power analysis, which does at least give you some it feels like it's better because you've got something to seek your teeth into and it seems a little bit more precise. I should say it is problematic. Sometimes we often get, in fact, the majority of the cases have been people that where they've already conducted a baseline. So at least when you conducted a baseline, you have some sense of what the correlations are in the at least the covariates or other variables. So you get a little bit better sense of what sort of the residual is going to look like, even though you haven't treated it. Now, obviously, if it's a treatment heterogeneity issue or treatment interacts, that's not going to help you. But at least that's a basis on which to make a power calculation or power simulation, as I suggested. Yeah. And I would like to put out a common morning, but just as a public service announcement, don't be tempted to use the results of a very small pilot study for the basis of a power calculation because that's a very small representative sample. Yeah. We often would advocate for split sample designs. But this does get a little bit more complicated with the pre results review. But sometimes a method would be to collect a large sample size, randomly divide half of it and do lots of exploratory or design work power calculations on half of the data and hold off analyzing or even opening that second half of the data set for later confirmatory work. Of course, now it's a little bit more complicated with journals if you want to get that pre results review in there. So it's not, check with the particular journal you're working with if you're tempted by that feature. But that's something that is well known in machine learning and AI research, for example. Emma reminds folks, thinking beyond RCTs, there are 84 journals that offer register reports for meta analyses. And the format fits them well. I'm going to put in a link to the COS website with register reports that has information about journals that offer it across several different disciplines. And there's a table there also that mentioned some of these issues about journals that do offer it for existing data sets or meta analyses or other. So if you're interested in learning more there, there's some background. Okay, Jeffrey asked. And I should mention we have three more minutes, so we'll get through as many questions as possible. But to respect everybody's time, we will be ending in about three minutes. So just, we've already talked a little bit about sort of non-experimental work. So I'm just going to skip that. I apologize. Magdalena asked, what would you say was the experience from reviewers? Did they understand the format? Did you feel that the additional training reviewers required over and beyond specific instructions? And I would just add to her, did they seem to like it? I would add to that question. Ian, did you want to say anything? I would say we used better quality reviewers, experienced reviewers than we would in general. And our experience, I think, was very positive. They were curious about the process. They had been experienced at RCTs. They kind of knew why we were doing it. So they had sometimes asked us questions, but our yield was higher than average, not surprisingly, I think. And I think they were surprised at how hard it was, right? So I'm not sure, you know, enjoyed is quite the right word, but they felt it was a learning experience. And so they were willing to participate. And I don't think anybody said, I'll never do this again. I agree with most of that, except for kind of it seemed to be a little bit harder than we expected to get people to review. But apart from that, reviewers did a good job. They did exactly what they should have done. They focused on the right questions that were also in the materials that we gave out. So yeah, that was worked perfectly. And to corroborate what Andy and Irenaeus said, we did speak with some of the authors after they submitted about their experience at the JDE. And probably the most positive line of feedback was that the feedback that they received from reviewers was in much more constructive tone because, you know, you and I really have any results to shoot down at stage one. So I guess that's why it made it much easier to reviewers to provide actually useful actionable feedback. All right, I would just like to say thank you very much for your participation. Panelists, thank you very much for sharing your experiences and recommendations and your opinions on how the format's going so far and your recommendations for the future. Everybody attending, thank you very much. Thank you for logging on through submitting your questions. There are a couple of questions that we'll try to answer in written format at a later date. So we'll make sure that these materials are available at the recording and any slides that are available will share with the group. Thank you very much and have a great morning, evening, wherever you are. Take care. Bye-bye. Awesome. Thanks, everyone.