 All right, thanks a lot for the invitation and for being here just after lunch So I'm a little brother and I'm going to be talking to you about the instance for application This is joint work with a lot of people One of the co-director is also in the room Fernando. Not sure where he is, but Anyway, really excited to talk to you about this I'm not going to spend a lot of time trying to justify why I think we need more replications I think this is a crowd that's really well aware of some of the problems that we're facing This presentation is going to be mostly about the economics, political science, finance, social sciences And if you just have a look at the number of replications or comments that are being published in some of these disciplines Well, they're just there's just not that many in economics per year There's something like 20 comments of replications being published. So it's a very small number So when we started the sense into the idea was how can we actually? massively increase the number of replications being produced being disseminated and hopefully being published and The idea is really to try to move out of this bad equilibrium in which we don't have a lot of replications and when there's a replications usually it's quite adversarial and negative and We're pretty sure that a lot of people are actually doing a lot of replications and those are positive So the idea is how can we generate how can we generate a lot of new replications? And hopefully many of them are going to be positive I'm not going to spend a lot of time about the definitions But like in the previous presentation with the score project here The two key things that I want you to understand is if I say reproducibility or reproduction I really mean running the codes and when I see replication I mean something much more than that It can be using new data. It can be doing robustness checks. It can be a myriad of different things So when I see replication, I really mean something much more involved than actually just running the codes I mean running the codes reproductions are very important But here what we're trying to do is to generate actually a replication to try to get to the robustness and repugability of claims and results in published research most of the Repugations that we've generated so far are for non-experimental work But there's also some experimental work that we're doing right now Basically lab experiments and try to generate new data in a new lab So the first question is which studies we're trying to replicate We're focusing on economics political science for the moment We're going to be moving to sociology management Demography, terminology and other disciplines starting next year But for a moment you just want to keep in mind that we're interested in political science and economics And we're looking at papers that were just published in leading journals So for political science will be top three published in 2023 2022 Economics will be top five adding some additional journals So we're looking at papers that just got published and we're trying to replicate as many as possible And I'll get in a second to how we manage to generate new replications So we basically have a huge database of all the studies that are being published For these journals and what we do is we collaborate with data editors with Researchers and we try to obtain information about whether the study has actually publicly available data Or maybe the data needs to be accessed in a specific cloud. Maybe the data is Only available. Maybe only the final data set. So this is something that we document for each study On our website Then I have this huge database for which I know that some of these studies are actually replicable We do have the data or the data is asexual in a specific Lab or maybe research center with this list what I do is I give it to a board of editors We're not our journals But we have a board of leading economists and leading political scientists and basically what I asked them is that they give me Suggestions of names of potential researchers who could be replicators who could actually run the codes looking for coding errors doing robustness checks Etc. So that's one of the way that we generate Replications the second way is we develop replication games So these are events that we do each month all across the world So for instance last week we were in Vienna and in Ottawa next month We're going to Melbourne then we're going to San Diego Montreal Tokyo, etc And these replication games are basically a one-day event in which people register and I match them to other people in their field So for instance if you work in health economists I'm going to assign you with other health economists. I'm going to propose you five ten studies and you choose one So this is one month before the actual games and then when the game starts So let's say you're doing health economics or you're doing American Politics then you have your study you think about like the robustness checks that you want to do or the type of recording You want to do for a full month then that the the game starts This is the day of the game and then you have a full day to actually do everything that you've planned to do So this can be a lot of different things. So for instance, you're a team of let's say for Political scientists, maybe one of you is going to take the code from our bring it to Python another one is going to take new data adding new years another one is going to be looking at the All the years are a cluster looking at odd liars another person could do something else and then at the end of the games after eight hours of coding We finish the games and then what we do is what we go for drinks We're having fun and then in the following weeks and months. You need to send me a report and The same with with the previous stream So when a replication is done I receive a report and this report pretty much puts everything that you've done and you send it to me and What I do is I send it to the original authors So the Institute is the intermediary between the replicators and the original authors So I deal with all the email exchange and it's very important because when I send an email to original authors They reply right away Whereas when the replicators send an email usually the authors don't always answer So the fact that we have this institute in between is really useful And the fact that I deal with all the emails is also useful because then we want the response from the original authors And we do not disseminate publicly the replication report until we get that response It can take weeks can take months But then we release both at the same time and what's interesting is sometimes the original authors are not really pleased with the replication They're going to ask for maybe changing some of the tone languages and so on and then I go back to replicators asking them for this They might ask also the original authors to change a response and so on and usually this is one or two iterations of this And then we publicly release both of them on our website or discussion papers and also on Twitter Okay, last thing I want to mention is so far one place with 750 participants for replication games for 2023 And we usually start the games with a song and it's a lot of fun. So if you want to participate, let me know now The other thing that we're doing is the replication games and the first thing is really useful for when the data is publicly available But if you're an economist or if you work with data that is administrative data You will know that it's actually really hard to actually have access to the data to the data So starting this summer what we're going to do is we're going to pay $5,000 US dollars to replicators. We need to replicate a study that is done using admin data So it's going to be a special key for Studies using economic data. We're also going to start doing lab experiments in other lamps. We're using new data so basically so far just to give you an idea of What we've done is we've started a year ago or a year and a half ago We have 135 replications that are being ongoing or complicated and now that we've finally received funding We're going to be able to actually pay replicators and scale up our activities by a lot So what are the replicators? replicators or PhD students faculty or researchers at the World Bank IMF, etc but for the most part of PhD students and faculty They can remain anonymous. So Once they're done with their replication report usually remain anonymous I send it to the original authors and once they see the answer from the original authors They decide whether they want to put their name or not Okay, if they don't want to put their name, that's totally fine We release on OSF if they want to put their name then we release as a discussion a discussion paper But we disseminate pretty much everything that we do Okay, we don't disseminate only if it's positive or negative we disseminate everything the other thing I forgot to mention is all the replicators invited to co-author a paper where we're gonna be combining all the replications that we're doing And we're gonna be writing a paper like this every year So this summer we're gonna be disseminating our first paper And if you participate in replication games or through the other streams in terms of replication papers Then you get co-authorship to that meta paper even though maybe you remain anonymous for the single replication report There's gonna be so many authors that there's no way someone can actually tell which papers you replicated Okay, but what's key here is that the replicators are kind of referees, but they're super referees The paper is published just came out of the journal just got published But now as a reviewer with your teammates you actually have access to the data So you able to uncover coding errors and we've uncovered many major coding errors many minor coding errors But also you're able to see stuff that the referees haven't seen I mean maybe in the paper It's not written that you're actually waiting your observations But as a replicator just looking through the code that takes you one minute to figure out that actually there's a waiting scheme And maybe the results are not robust to this and so on so as a replicator You have a lot of advantage in comparison to what refer reports or referees are doing We obviously have a conflict of interest policy. The last thing on this slide is very important We do not tell the replicators how to replicate this study They're the expert This is their field they could have been a referee for that paper And so they should do whatever they feel that is a sensible robustness checks or any type of recording or Redownaling the data and recalling everything from scratch They decide what they do and what is appropriate for for instance someone working with like time series is very different And what's appropriate for someone working in political science doing an experiment So it's really important that we allow this flexibility in terms of the type of replications that people can do All right once a replication is done They send me a report as I said then I send it to the original authors and as I've mentioned then we release those as discussion papers Okay, so as I said worry we've completed 140 something like this Completed or ongoing replications These are starting to be disseminated on our website so if you're on a website and you're interested in a specific study You can look whether the data is public available or not where you can download the data whether It's been reproduced and whether it's been replicated and you have access to this and the idea is I won't point down the line when we're going to be Having thousands of replications before you start a new project and you want to sign a specific study You're going to look at whether the data is available whether the paper has been successfully replicated or not Okay, so that's kind of the idea Also I managed to put together three special issues dedicated to replication So the goal is to try to publish Some of the replication reports for those that don't want to end up That don't want to remain anonymous. So with Kevin Easton in who's sitting right there We put together a special issue at research in politics But another one a king of general economics at the economy can quarry the editor-in-chief told us that we can publish as Many replications as we want as long as they're of high quality. So the goal is going to try to Generate a lot of demand also for replications. So here we're going to generate a lot But we also want some of them to get published. We also work with journals trying to put together Replication section etc etc. So I know so all this that are presented so far is achieved with zero funding Now the cool thing is a couple of weeks ago. We actually got lots of funding So now what we want to do is to massively scale up our activities so starting right now or Goal is to replicate something between 100 and 250 replications per year for economics Over or let's say 100 replication for political science. Then we want to move to management sociology etc etc So just to give you an idea for economics This would be approximately 25% of all studies using empirical data published in the leading Let's say eight or nine journals So we want to get to a point where if you publish in these journals, you know someone's going to look at your code and Maybe there's quite a likelihood that actually someone's going to replicate it so You know look at your coach carefully and Maybe ask someone else also to actually look at your coach to make sure you didn't make you know Some stupid coding mistakes, which we find a lot and most of these mistakes are just by accident But if someone would have looked at it clearly you figure it out really fast So we want to get to a new equilibrium in which actually there's not a tread But just someone is going to actually going to look at your codes. I won't point down the line So that's kind of the idea The last thing I want to mention before I move on to the To the Q&A is we're really hoping to get at least 1,000 parts spent per year for the replication games and If you're interested in organizing replication games and for any of these disciplines at your institution Institution just to let me know we just need coffee and a bunch of rooms to be honest So it's really low-cost and it's a lot of fun I'll stop here This is your institute. We're doing this for you guys So if you need something or you want something or you need help you've done a replication We want you want us to reach out to the original authors We're happy to do that if you're an editor and you want to put together special issues or replication sections Please just contact us. We're happy to help That's what we're for. Thanks a lot Yeah, please go ahead. Hi Dolmarosh. I'm working with shirk in Canada Lovely presentation great to hear that you're putting so much work towards this important effort as wondering I know this is this is recent and so There might not have been as much times as you would have liked to see or check the impact of the reports that you're publishing But we know that when studies are published They're often covered in the media and they can have a really big impact and they get cited even papers They get retracted keep on getting cited for years and years afterwards as wondering whether In your case you've seen that some of the reports that you've published maybe highlighting Deficiencies with some studies or a lack of replicability. Has that had any impact you think on how these studies have been used by researchers downstream? Yeah, so that's a great question is something we deeply care about I'd say that One big difference between what we're doing and what others are doing is we replicate like right off the bat So the study is published and we replicate like right away So that's a big difference between repeating something done ten years ago and you change a clustering So I think we're going to be able maybe to change the path in terms of citations that a study receives because we Provide this report really fast in terms of whether the results are robust or not The other thing is some of our repetition reports have already been published as comments So for instance, there was a case in top general economics were 75% of the data were duplicates We released this on Twitter and we got 200,000 views in 24 hours The comment was published within a month at the journal I think can have an impact to be honest. Thanks. Thanks. Thanks for that