 to interact with this group here. There really are a lot of links between what BIDS is doing and what BIDS, this other initiative I'm involved with, is doing, so it's great to connect and share. I'm gonna talk about issues that I think might interest you in economics related to promoting research transparency. I'm really gonna think a lot about and discuss pre-specification of research hypotheses. So start with a little bit of background where I'm coming from and then talk about some potential applications of pre-specification of research hypotheses as a tool for achieving greater transparency in economics research. The starting point for all this is growing recognition in the social sciences, not just in econ, but in psychology, political science and other fields that there are some pretty pervasive problems with the research that's being produced. There's more and more evidence that whole literature is maybe full of false positives or there's a fear that lots of results that we think of as solid results are really just false positives. There's a lot of evidence that data mining goes on. There's also a really great new study that's the Franco et al 2014 study in science. It was a really unusual study because they were able to keep track of all the different social scientists who had gone access to a particular data set. It was really a data set that the researchers themselves helped design. They got to add questions to online surveys. And Franco et al followed up with all the researchers that had been involved in this multi-year project and got information on their research results, whether they had statistically significant results or not, whether their findings were expected or not, et cetera. And they found this really striking fact which was null results. Results that were well-powered but not statistically significant very rarely got published, very rarely appeared in the published literature. Significant results almost all got published. So if you extrapolate from the findings of Franco et al to the rest of our social science research literatures, and perhaps this holds outside the social sciences, we should really be concerned about the bodies of literature we're producing. Those are the problems. We're figuring out new ways of diagnosing these problems, but I'm gonna discuss today one possible way of beginning to solve some of these problems and discuss how widely it can be applied. So to sort of lay out a set of possible solutions, I was involved in a paper that came out in science a couple years ago. They talked about three different sets of approaches that might improve transparency and reproducibility in social science research. So the three are listed up here. One is what's called disclosure, really increasing the reporting requirements for researchers, really setting up or trying to set up a norm of full disclosure of research designs, full disclosure of data. So that's one. The second one, which is related, is shifting norms in favor of open data and materials. Again, in many research fields, people's data, researchers' data remains their own and is rarely shared with the research community. That's changing now in a lot of fields. I think the world is changing. It's definitely changing in economics, but for instance, in psychology, it's still pretty rare to share your data sets. In medical research, it's still quite rare, partially for privacy reasons. So open data and materials is valuable to allow people to replicate results and sort of build on existing research. The third type of approach or potential solution we talk about in this article from a couple years ago is preregistration of research hypotheses. That's what I'm gonna focus on mainly. Today, the idea here is, when we think of science, when we think of research, we really think of preregistration type approaches to research hypotheses. We think that the researchers out there with a clear idea of what they're gonna test, they go and they get their data, they run their experiment, whatever they're gonna do, and then they compare their results to what they planned. In reality, research doesn't proceed this way in many cases. Sometimes it does, but it often doesn't. People get their hands on data, they mine the data, they look for correlations, they look for different patterns. Maybe they've run 20 different hypotheses on this data. And maybe at the end of the day, they only report that one that's statistically significant. That's the one that gets published in a paper. That's a big problem. That's part of the reason why we think there may be a lot of false positives out there in research literature. So the idea here is, if only we could get folks to publicly register their research hypotheses beforehand. A, we're gonna know all the tests that were run. So we're gonna get a better sense of what results are missing in the literature, unpublished. And when people run their statistical tests, we're gonna have a better sense of what they mean. If I've run hundreds of tests, and I only report one of them, the one with the smallest p-value, that p-value isn't that meaningful. There's a sort of multiple inference problem in the background that's just not being dealt with in any sort of serious way. Pre-registration of hypotheses could help us deal with that. We could lay out what we wanna test. We could lay out how we're gonna deal with the multiple testing problem. We can try to solve it. Okay, so what does this mean in practice? And I'll tell you what it's starting to mean in economics. So in economics, we actually are beginning to have an approach for pre-registering research hypotheses. I'm gonna tell you what people are doing so far, and then think about whether we can extend this into new realms of research. That's really the goal of the talk today. So what would pre-registration look like? Well, researchers, myself, I have a research question. I could post them in a publicly available registry that could be queried. So if anybody else is interested in this research topic, they could kind of figure out I was interested in this. I've been collecting data on this issue. You could imagine that that sort of plan or that research hypothesis document could be very detailed. It could be very sparse. And one of the things I'll come back to in my discussion today is there's still a lot of debate within economics and other social sciences about how much one should pre-register. How detailed one should be. Should it be a one pager with the broad outline of what you plan to do or a 50 page document step by step, mechanism by mechanism all the way through. I'll come back to that later on. In some sense, it's second order. That debate about how detailed we should be because almost no one's doing it at all right now. So even just getting that two pager out with like I'm kind of interested in this question, that would be a huge step forward. So let me tell you about an effort which is the new American Economic Association Registry. It's called socialscienceregistry.org. So I guess they had a kind of, maybe there's an imperialistic ambition there to make it social science wide, not just economics. It was only founded a few years ago. So less than three years ago. It's called the RCT Registry, Randomized Control Trial Registry. And part of the impetus for establishing this registry were all the concerns I was talking about before. The fear that there was a lot of research that was disappearing, et cetera. An economist got together and voted through our association to set up a registry so we could have a kind of central registry to store this information. It's a nice website, very user friendly. And the thing that's been very impressive for us in economics is just how quickly it's taken off. I was involved in some early discussions about the registry. I've been through bits, trying to promote the preregistration of research hypotheses. It was really unclear when the registry was launched three years ago if people would participate, but they are. There's already over 600 studies registered in the last few years. Numbers are rising rapidly. Some of these are older studies. So the registry has two goals. One is to allow people to preregister hypotheses. The other is folks who have existing studies, existing randomized control trials in economics are encouraged to register on the RCT Registry so we have the complete body of literature basically there, registered all in one place. So it's a combination of both, but the numbers are rising. So every month there's more registrations here from the start in 2013 until earlier this year. And the numbers per month are rising too. So the trend is, this is really accelerating in terms of registrations. And I will say that in my own field, I'm an economist, I'm a development economist. I work on international development issues. Development economics has been the subfield within economics that has done the most field experiments and randomized control trials of any subfield of economics. We got into it early on, we teach our students how to do them and now it's spreading sort of throughout economics. Within development economics, in the last few years it's really become the norm to register your work, put up hypothesis documents, et cetera, before studies are analyzed. So the norms are starting to shift in my field for experiments very quickly. So this is a real success. In a few years we went from no registry in economics, almost no pre-registration of hypotheses to just hundreds of these studies coming up a year. Big change in a small amount of time. The data here, going back to the issue I raised before about the level of detail that's expected, you can put up a pretty sparse registration right now. The A.A. registry was set up and was meant to be pretty light touch, pretty low demand, to impose low demands on research or time. So you can basically list your main outcomes, the broad design you're gonna use, the country you're working in, a little bit on the sample. But to register your study you don't need to lay out every single statistical test you plan to carry out. There's an option to do so, there's an option to attach documents with very detailed analysis plans if you want, but it's still light touch. And I think part of the reason they did this was to make sure people weren't scared off. They wanted to get everybody in the system, in the registry, even if it's at a minimal level, to change the norm and then maybe over time norms will shift towards more demanding sort of standards. But it's easy, you click on it, start with the name of the trial in 15 minutes you can register a study. So this is a success story. What are the benefits that I think we might see from this move towards pre-specification in economics? I'm gonna survey this quickly for like five minutes and then tell you about this whole other realm of research and economics that isn't being pre-registered where I think these norms could be usefully applied. Let me just give you the, this is like the win, the success story. RCTs are being registered online. Why is this good? A, I mentioned the Franco et al study, all this research that just disappears. We know it's conducted, it just never gets published, it's never circulated. If everybody's registering their research hypotheses ex ante, that research isn't gonna disappear. It's gonna be registered. I'll be able to search on a particular labor economics topic or particular health economics topic and get the 20 studies that were registered on that topic, even if they were never published. That's really good. It rounds out the body of evidence. A, B, it's gonna reduce the risk of biased reporting of results. So if I lay out in advance, I'm planning to look at a particular health outcome. I'm interested in the effect of a particular treatment on high blood pressure. And I registered that. But when I got the data, I didn't find anything on high blood pressure. Well, maybe without registration, I would sort of like not report high blood pressure. Maybe I found that it affected weight loss. I looked at 50 different health outcomes, 49 just had no effect. There was this one relationship between this treatment and weight loss. Sure, that should be reported, but if I write my paper as if all along I was planning to look at weight loss, it's just a really misleading paper. It's a really misleading piece of research. And maybe the high blood pressure result never gets reported. If I'm registered publicly and I go on the record saying, I think the hypothesis I'm interested in, my primary outcome is high blood pressure, someone's gonna ask like, what was the high blood pressure result? You said you were gonna run. You could say, oh, that was a zero, but there's at least some accountability. Without the registry, it's hard to even know what people are doing. So that's a big advantage of the registry. Closely related to point number two, P values have a lot more meaning when I know how many tests have been run. If I know you were gonna run these five tests, I can adjust for multiple inference across those five tests. But if I don't know what you plan to do, maybe you ran 500 tests, and I have no idea what statistical significance means if I don't know how many tests you ran. We have corrections for this stuff, but if I don't know what the number of tests are, I just have nowhere to start. So that's another advantage. It makes open data more effective in the sense that if stuff is registered, it may point me towards the data set that's out there. If I post my data somewhere, but no one knew what I planned to use it for, or no one even knew the study existed, they're never gonna find that data set. If there's a centralized registry where everybody's posting their research hypotheses, I'm gonna find out about more data. I'm gonna be able to build on existing research better. Another big advantage of a registry. Wow, registry is just sounding awesome. This is sounding so good. A side benefit, and one that people don't typically talk about much, which I think as someone who's been writing pre-analysis plans and pre-registering hypotheses now for the last six, seven years, a big benefit is it forces you to really think through your research in advance of doing it. So you're planning your data collection or you're planning your analysis and we have endless debates now about what is the right econometric specification? What is the right statistical test in a given setting? And we do our due diligence to figure that out before registering it in a plan, because that's pretty serious. The world's gonna see that was our plan. You can't wing it as much, which a lot of researchers do. They get data, they're not sure exactly how they're gonna use it. Sometimes when you think things through, you get the right data. When you think things through, you sort of structure your tests in a better way, et cetera. So I was presenting related talk this couple of years ago down at Stanford and there were folks down there who do lab experiments in the social sciences and some of these folks were saying, well, I like to be able to wake up in the morning and just like with a new idea and just run the lab that day. Just like, oh, I wanna run this lab. I'm gonna get these subjects in today and run a lab. That's pretty exhilarating, I think, as a researcher to wake up in the morning, have your cup of coffee and just have a new idea. But I might suggest you take a few hours and think about it before you use all these resources to run a test. So again, there's something a little more deliberative, a little more careful about writing a plan. There may be others. Again, I'm not sure how interactive we wanna be. Maybe in other setting, everybody be shouting things out. Let's do it. I'd love to. I want questions, I just wanted to know about the mics. Please don't wait, go for it. Okay. Don't know if it's recording. It's on? So going back to, you used the word pre-registration rather than publishing with at least peer review and maybe more darkly publisher's lock. That's sort of a situation. So you're interested in that distinction between publishing it versus pre-registring it? Yeah, you could imagine applying a publishing point of view to or use the word at least to this process of hypotheses with peer review at least. Yeah, yeah. Or have the agenda of working it the other way and getting to a more open publishing regime. I'm gonna come back to related points but I'll just say one thing which in some field, so in clinical trials, medical clinical trials, people often do publish their pre-analysis plans, their hypotheses documents. So that is something that is done in some field. I will talk about exactly this notion though of getting your hypotheses peer reviewed later on. I'll talk about what are called registered reports which is a new publishing form that's taking off in now in psychology. There's a political science journal that's doing it which allows folks to get their research hypotheses peer reviewed and accepted for publication before they even touch the data. So I'll come back to that because that's a kind of next step in some of these ideas. Exactly, so you make exactly. So in the grant process, we're already writing proposals laying out our hypothesis. In some sense, we already do have these plans but those are very rarely shared with the research community. They're sort of proprietary, the grant making organizations don't publish them typically unless they're funded. So there are already things we're doing that are along the lines of what I'm saying but it's not formalized, they're not put out there. They're not, I guess what I'm, the big difference is they're not searchable by the research community. We're making them but we're not putting them in a place where everybody can find them and a registry makes them findable. Okay, so how about some concerns? In this push towards pre-registration economics, there's been a backlash actually within economics and other fields. There's a number of scholars who have been very vociferously arguing against pre-registration. There was a very senior scholar in my field in development economics and I got an email from a friend, literally just yesterday, this distinguished scholar in my field was giving a seminar at his department and he said apparently I hate pre-analysis plans. My model is my pre-analysis plan meaning he has a theoretical perspective so economists write down formal models. Those models have testable predictions. He's saying look, here's my model, my model has predictions. If I run tests, trying to figure out what a parameter estimate is in my model, my model's already constraining my behavior. I'm not data mining, I'm just basing everything off my model. The junior colleague that contacted me about this said to the senior scholar, well why didn't you pre-register your model then? If the model's everything, pre-register your model. Get it up there, make it public that that's what you wanna test. If you don't do that, I have no idea that was the model you started with. You might have gone through three or four different conceptual frameworks before you hit on one that gave you results that you liked. So I'll come back to that as well but there's this back and forth in economics where a lot of us who are doing field experiments, especially younger scholars, are registering but there's a backlash, especially among more established scholars I think. One of the critiques that's made is you know if we have to pre-register, it's just gonna stifle our creativity. So many famous scientific findings were just done by chance. Someone stumbled across a pattern in the data. You know the discovery of penicillin, we could go on and on about these accidental discoveries that no one planned, there was no pre-analysis plan for them. Are we gonna lose that? Are we gonna lose all that scientific progress if we start pre-registering? My own view on this is it's not either or. When we can pre-register and we have data coming online or an experiment coming online that allows us to pre-register, let's pre-register, there's always gonna be tons of data, non-experimental data, other data floating around, historical data, retrospective data that people are gonna be analyzing and looking at and maybe there, these kinds of fortuitous discoveries are gonna be more common. But it doesn't have to be either or. And I just wanna, I put this slide kind of early on in the talk, because very often when we talk about pre-registration, people say, oh god, if we start pre-registering, only pre-registered work will get published. That's not how it works. About 15, 20 years ago in economics, people started doing field experiments. I was lucky enough to be in getting my PhD and my advisor was one of the earliest economists doing field experiments, Michael Kramer. And I started doing field experiments. So I was part of this wave of economists doing field experiments. And when we started doing them, we started hearing a lot of the same stuff. Oh my god, now that they're field experiments, those of us not doing experiments are just never gonna be able to publish again. It's over. Experiments are just gonna crowd out everything else. That hasn't happened. It's been 15 or 20 years now. And last we checked, there's some recent data. About 10% of papers in economics journals, leading economics journals are experimental. Most papers are still observational. So it went from 1% to 10%. That's like a massive increase. The trend is going in that direction. But experimental work is still a subset of all work. Same thing in medical research where clinical trials are very well established. The vast majority of papers in epidemiology, public health, et cetera, remain observational studies. So we don't have to think that, just because registration's available and usable in some settings, nothing else is ever gonna get published again. So again, I just wanna kind of move beyond that. The advantage of pre-specification is then we know sort of what was, what is being tested that was planned on ex ante versus what's more exploratory. And that's a really valuable distinction to know scientifically. Those are different sorts of findings. The interpretation of p-values are different, et cetera. Okay, all that was background. Long-winded 15 minute background. But what I wanna really get to today's, the starting point was, we've kind of gotten pre-registration to work for experiments in economics. How widely can we apply this approach? Can we go beyond experimental studies? In my view and in discussions with others working in economics interest in these issues, this is like the big open intellectual question in our field, methodologically, around transparency. Is pre-registration, are pre-analysis plans just gonna be confined to experiments or could they be applied elsewhere? Well, one area beyond field experiments where they could very easily be applied is laboratory experiments. So people called experimental economists typically do lab work similar to psychology labs, very few of them pre-register their experience. But they really could, just like we do with field experiments. It's the same idea. They're planning a lab, they're gonna call in subjects, they know what the treatments are, they presumably have a research hypothesis there that they're working with. They could register their work. There's been a lot of backlash from experimental economists. So a very distinguished experimental economist, Muriel Netterly at Stanford, says, you know what, pre-registering work in the lab is really a waste of time because labs are cheap and easy to replicate. It only costs a couple thousand dollars, maybe to run a big lab. If I find something interesting, I can go back and do it again. Someone else can just very easily redo my lab. So when replication is really easy and cheap, maybe pre-registration is just superfluous. That's her argument. And there's something to it, compared to some of the field experiments we do in development economics. We've been involved in multi-year field experiments in, I mainly work in Sub-Saharan Africa, where the cost of the intervention is millions of dollars. You can't easily redo those experiments. That's pretty hard and expensive. A lab that costs $1,500 or $3,000, if it's an sufficiently interesting result, people will replicate it. So she has a point. That said, more and more lab data and lab work is being linked to field experiments these days in economics. More and more lab work is also being linked to the real world context. So for instance, some very interesting experimental economics work done at Berkeley here, looked at how people played labs before and after the outbreak of the Great Recession. And behavior changed. People became selfish. People started acting in these really selfish and uncooperative ways after, I think, October 2008 when the market collapsed. You can't redo that real world experiment. That's a case where you might actually want a pre-analysis plan. So I think there are cases where pre-analysis plans will be useful in labs. There may be other cases where replication can solve a lot of problems. So that's another one area. The area I'm going to focus on though for the next 20 minutes, really the bulk of the talk, is going to be what I call perspective observational studies that are non-experimental. So what do I mean by perspective observational studies? Experiments are pretty much by definition perspective. Like I'm planning the experiment so I can write down a plan. Presumably I have a plan of what I'm going to do. I could register that. Observational studies with existing data though it's harder to know when you've seen the data, if you've been mining the data before you register a plan. But perspective observational work is work that's not based necessarily on an experiment but where I don't have access to the data yet. That's what I'm calling perspective work. So for instance, in economics, very often we know there's going to be a policy change in the future. Right now in California, Jerry Brown just signed a bill into law, I think a week ago, two weeks ago, raising the minimum wage. So for the next five years, we know the minimum wage is going to go up to $15, maybe six years. It's going to go up to $15. That's a policy change that we know is going to happen. We can take advantage of that policy change to study the effect of minimum wage changes on the labor market. I'm going to come back to exactly this example in a few slides. I could write a pre-analysis plan now laying out how I'm going to exploit that policy change. It hasn't happened, I can't mind the data. I don't know what the data is going to look like in 2020, you know. That is perspective. This is a, I'm going to argue today, like the bottom line of my talk today is this is a realm where we could be doing a lot of pre-registration in economics and in the social sciences, but like no one is now or almost nobody is right now. Another case, not just policy changes, in political science, future elections. We know when the election's going to take place. We may have hypotheses about how particular things are going to affect voting outcomes, but the voting outcomes haven't been realized yet. There's actually a case of some pre-specification of political science along these lines. Why isn't every political scientist who's studying election outcomes just pre-registering their hypotheses so we know they're not cherry picking results? Seems very natural to do. Almost nobody is doing it right now. So there's this whole realm of activity in social science that's unregistered that could be just as easily registered as RCTs. And I'll talk about an early case, and basically I've almost forgotten paper, this Newmark paper I'm going to talk about that does this and talk about what we can learn from it. Okay, so beyond these big policy experiments, any time a new round of data is going to be released, we could be writing pre-analysis plans before it's released, before a census is released. PSID, sorry, I'm using economics terminology. This is the panel study of income dynamics, a multi-decadal scale longitudinal survey of American households. And every however many years they come out with another round. Well, if there's been something interesting to study, why don't I just register my hypotheses before they release the round? I know exactly what variables are going to be in there, et cetera. So that's what I'm calling for. Now, the current AEA registry is meant to be for RCTs, randomized control trials, but there are actually other platforms one could use to register a pre-analysis plan. So one is the open science framework, which some of you may be familiar with, put together by the Center for Open Science in Virginia. They have a very flexible, you can basically register and make public whatever you want, point people to it, create a permanent URL, whatever, to make it pretty easy to find. And they're also trying to increase interoperability with the AEA registry. So my own view is the AEA registry, and that's part of the kind of point of this talk, is also just to make the case, the AEA registry should be open to more than RCTs. It should be open to any perspective study where it's credible you could register your, and timestamp your hypotheses. Let me give you a few other examples. There's other applications that one could use pre-registration for. Let me give you a few again. These are a bit econ-specific. Try not to use too much jargon if you have questions. Go ahead and ask. In a lot of macroeconomic research, researchers run calibrations. They have a model of the economy, the macroeconomy, and they subject this economy to different impulses, interventions, and they work through and see the effect of these sort of simulated experiments within the model on, say, economic growth. But there's a lot of choices and flexibility in a particular model in which parameters you're gonna choose. And those parameters are gonna govern the outcomes. And there's a fear among those of us not in macro, and even among those in macro, that researchers sometimes have a leeway to get the results they want by manipulating four or five parameters enough. That there's basically a lot of degrees of freedom is one way to put it in these macro models. A macroeconomist who hasn't, say, gotten the latest data yet that they're gonna use for one of these calibrations could go on the record and pre-register. These are exactly what I think of as the right parameter values for this model. I haven't seen the data that I'm gonna use to generate predictions, but I'm on the record with these parameters. Here are the results. It'll reduce some of the fear that results are being rigged by researchers, just to be frank. Very similarly, in some economic models, which are called structural models, again, there's very specific mathematical, functional forms imposed on behavior. They give you very specific predictions about economic behavior. These models tend to be quite complex and quite sensitive to parameter values. And again, there's a fear that you can get almost anything you want if you tinker enough with what the demand curve looks like, what cost curves look like, et cetera. Prespecifying, going on the record with what you think of as the right parameters would really give people a lot more confidence in your results. For people who are doing Bayesian statistics, there is a sort of degree of freedom there. You can make different assumptions about the prior distribution. Again, before I get the data I'm gonna use an analysis, I could go on the record and pre-register what I think of as the right prior. No one can accuse me then of sort of data mining based on choosing a prior to get the result that I wanted based on the data if I haven't seen the data yet. So again, before you see the data, you can go on the record with a lot of your research decisions and it could give the research more credibility. So it doesn't just have to be experiment. Anytime you don't have the data yet, you can do this. Okay, so let me discuss the Newmark paper for about 10 minutes or so, 15 minutes. I see this paper, I've been writing a review with Garrett Christensen, your guy's colleague here at BIDS. We've been writing a review on transparency and reproducibility issues in economics and trying to make sense of where this literature has come from and really for decades there have been papers that have appeared one or two here or there and then kind of been forgotten or trying to make the case for some of the practices we're talking about. Now there's more, I think critical mass, more momentum. But an early milestone that's almost forgotten is this Newmark study from I guess 17 years ago, it was ultimately published 15 years ago. So my knowledge was the first pre-analysis plan in economics. This is someone who got on the record about his research hypotheses. And he was studying minimum wage, the topic I was talking about before. Do you have a comment in the back? Do you wanna ask a question? The mic is here, why don't you use the mic? I was wondering whether we can keep update our assumptions like when you're doing BASian prior has to be kept updated by existing literature or available data. So is it possible for researchers to update and change their assumptions? When new data comes in or? Before data comes in. Before the data comes in? Yeah, so like when you're doing- You're saying if you register one document could you then update it before the data comes in? Right. Oh, definitely. I would think that would be the right thing to do. It shouldn't be that the pre-analysis plan is set in stone. If I register a set of ideas, I still don't have the data. I'm like, oh my God, I made a terrible mistake. One of the first pre-analysis plans after Newmark, Newmark is the first, but there were a few pre-analysis plans written in 2009. I was involved in one of those projects. I think there were three of us in different groups that around that time started doing this in economics. And I think overall we did a good job. I mean, it was like a 60 page document, our pre-analysis plan. We had everything really detailed to the exact covariates, exact outcomes, exact groupings into mean effects. I mean, we really spent forever putting it together. There was a hypothesis we forgot. We were studying a program in Sierra Leone that was providing assistance. You know about it. You know this paper. So provides assistance to communities, but we forgot the most basic hypothesis which was like, was the assistance from this program actually given to the villages? Like we actually had data on like, did the money arrive? Did they actually implement the program? Because we were just, all of our hypotheses were about the outcomes, right? And this just basic implementation hypothesis we forgot. So in the published paper we're like, okay, this isn't in our pre-analysis plan. It's so obvious that you'd want to actually check that the money arrived before anything else. So we're adding it, mea culpa. So our view is we can't be dogmatic and say, oh, something that wasn't pre-specified could never be looked at. You just have to be clear about what was pre-specified and what you added. And we say in the paper, hey, if you don't want to consider this, don't consider it. That's that you probably want to know if the money arrived before studying anything else. So our view is we need to iterate, improve over time. Okay, let me talk about this new mark piece. So just like today, the minimum wage is a contentious, politically contentious issue. I think it was even more contentious probably 20 years ago than it is today. And there was a literature, David Card and Alan Kruger were involved. David Card is here at my colleague at Berkeley. They had written a bunch of papers showing that, you know, you raise the minimum wage there's basically no employment effects. That was their finding. They had a bunch of studies. Newmark and some other colleagues had written other papers with different data and different statistical approaches showing that when you raise the minimum wage you get these negative employment effects, which is like the neoclassical economic explanation. You make labor expensive, people hire less labor. Everybody kind of agrees that's probably the right direction of the effect. But if it's minuscule and close to zero, who cares? If it's really big and negative, it matters a lot. So Newmark's findings were it's really big and negative. Card and Kruger's were at zero. It was very contentious. And that's where this study kind of emerges. So this, not to the labor, but you know, these were just the existing estimates in the early 90s. There were some zero point estimates. There were some positive point estimates. Hey, you raise the minimum wage employment goes up. There were a bunch of big negative point estimates. There were sort of estimates all over the place. And Card and Kruger in a study of theirs really argue very strongly that they thought there was a lot of publication bias. That, you know, you basically saw lots of studies with really big estimates in both directions with T statistics of exactly two. Like just what you needed to get published and sort of nothing in between. So it was like nothing null ever got published, only like big positive and negative effects on both sides. And it just seems so consistent with a really screwed up journal screening model. So they said, okay, you know, we've got to do better than this. So what ended up happening? And this is really a Berkeley story. So I'm glad to be telling it here. There was a journal, Industrial Relations, which is a labor economics journal. The editor was here at Berkeley, is still here at Berkeley. David Levine in Haas School of Business. He was the editor in conversation with Alan Krueger about this exact topic. They said, you know, there's got to be a way to get around these publication bias effects and these author effects. And they came up with this idea of saying, why don't we tell people to write down what they're gonna do, exactly what specification they're gonna do, exactly the analysis, exactly the data before the next few waves of government labor market data are released. Let's just like be totally hands above the table about it. So they kind of came up with this idea on their own and said, this is the scientifically appropriate way to work in this field. So David Levine and Alan Krueger did it. They brought in David Newmark. And the idea was that Krueger, Card, Newmark and co-authors would sort of agree on a statistical approach and the data they would use and then would kind of like generate the estimates together. That was the idea before getting any of the data. So this is what's sort of written up by Newmark where he kind of describes what he did. You know, we're gonna prespecify the design. We're gonna subject it to peer review beforehand to make sure that like expert researchers in the field agree this is the right way to test this. And then we're gonna carry it out when the data becomes available. And this should allow us to eliminate what they call author effects. Author effects means kind of like researcher bias in the estimates. So very, this is 20, you know, this is 1997. So this was 20 years ago. No one was talking about this, but somehow the scientific method like burst through the, you know, through the fog. It's very similar to the idea that Danny Kahneman, I guess some people are familiar with him, he's a Nobel Prize winner in economics, was a, he's a psychologist by training, but as an expert leader in the field of psychology and economics. He was here at Berkeley at the time too. So this is all kind of mixed together. And David Levine said he used to talk to Danny Kahneman all the time about this notion of adversarial collaboration. Getting people with different results who are on different sides of a literature together and saying we agree this is the right way to proceed scientifically and we're gonna co-author a paper. That's actually what Danny Kahneman has done apparently on a bunch of projects. He'll reach out to his most bitter adversary in the field and say, you know what, let's agree, let's sit down, do the experiment together and make scientific progress. This is great. And this was the goal of the industrial relations pre-specification. It's also very close to what is now called a registered report, which is what I was mentioning briefly before. So this is an article format that has become common just in the last few years in a number of psychology journals, some scientific journals, one political science journal, comparative political studies did a special issue where again, folks submit their plan, their research design, their data, it gets peer reviewed and then it gets what's called in principle acceptance, IPA. And in principle acceptance, it's sort of like getting a revise and resubmit for a journal from a regular standard paper in that they're like, you know what, this sounds great. If you can actually get this data and you do what you say you're gonna do, we'll publish it. Obviously, if the data collection falls apart or you don't do the analysis the way you say you would or you don't write it up, then they're not gonna publish it. But so that's the idea behind a registered report. Effectively what the industrial relations journal did was a registered report. Peer review, then get the data. We commit to publishing it as long as you do what you say you were gonna do. So this is just a kind of schematic of what publishing would look like both the standard model and this new kind of registered reports model. The usual model is we design a study, we go get the data, we analyze it, we write it up and then we decide whether or not to submit it. If it gets submitted it goes under peer review, maybe the editor asks us to modify it, maybe we edit it, eventually maybe it gets published. There's problems all along the way here, mainly at that writing up and submitting stage. So you know we talked about the Franco at all paper before, people collect data, they plan to get something published, but for some reason they don't like the results, they're no results, the results don't conform with the priors of others in their discipline, this evidence disappears into the ether. So you may never even get to that publishing stage with a traditional model. With the results blind review, you do all the design stuff, all the heavy lifting, all the conceptual work, all the literature review, all the planning up front and you submit it to peer review. And if it's a good enough idea they give you in principle acceptance, then you go out and do all the work. There's a bunch of advantages to this. A huge advantage is getting lots of detailed feedback at a point where you can still improve the science. So this has a bunch of advantages, even beyond getting around the whole publication bias issue, which is a massive issue. So this is a new and very exciting way forward, then you collect the data and you go through and hopefully you get the paper published. But the difference here is even if you get a null result, even if you get a surprising result, you're gonna get published. Cardin Kruger, I talked about their minimum wage work two minutes ago, collected a bunch of data in the 90s showing that minimum wage increases really didn't affect employment, as I mentioned. There was a lot of resistance in the economics profession to that finding. Neoclassical theory suggests when you raise wages, employment's gonna fall. People were upset with them. They said they lost friends. There are people in the profession that wouldn't talk to them for years because they felt like they were traders to their discipline, seriously. So if you have registered reports, you're kind of protected from that too. Like, hell, look, we can all agree on the design. There's just a zero here. Don't blame the messenger, right? People were afraid that those zeros were author effects, that they were biased, that they were left-wing, that they were whatever. So there's a lot of potential advantages to this sort of approach. How did it work out in the Newmark study? And this is a terrible eye test for all of you that are not right up here with me, I apologize. But basically that arrow there is when the April 1997 symposium for industrial relations took place. Newmark submits his pre-specification of analysis and it's only a few months later that the data gets released. I mean, the US government just hadn't released the data. He couldn't have mined the data. So he said, I'm gonna use the data that's gonna be released later this year to study these things. He registered it, eventually came out as a working paper and then it got published in the journal in 2001. So it's basically a registered report, what they did. Okay, so I'm running out of time. So let me skip over this point. This point about the cost of pre-specification has a little bit to do with what I talked about before about flexibility and exploration. So you're kind of tying your hands. When you pre-specify, and if you kind of get the specification wrong, that's a problem. But hopefully if you do, you'll just admit you did and run a better specification, explain why. I think if there's more and more pre-analysis plans and pre-specification, people will learn that one out of every three or four papers, there's an error in the plan. We have to be flexible as long as we explain why. Newmark's point, which I think is important, is in cases where there's already a large body of literature like the minimum wage study, this is a bit less of a concern. When people have been studying an issue for decades, they know all the data sets, they kind of know the econometric or statistical issues. You kind of know there's maybe a few choices you're gonna make on the statistics, but it's pretty well established what you're gonna do. So this highly politicized, widely studied issues are probably the best cases for registered reports. You get around any accusation or possibility of bias and you really know what you're doing, ex ante. So this is sort of, he basically makes this point. In cases where there's a lot of past experience, we're not so concerned about it. In other areas, and the paper here, the Casey et al paper that's cited here was that Sierra Leone study I was alluding to before, we were doing work on the impacts of a government, of sorry, yeah, a government assistance program to villages in Sierra Leone that attempted to build local institutions and social capital broadly defined. No one really knew what the right outcomes were for that study. In our study, we have hundreds of outcomes, gender empowerment outcomes, voting outcomes, meeting participation outcomes. There's just literally hundreds of outcomes that we designed in our surveys because we really didn't know where to look or what to do. And in fact, when we wrote our pre-analysis plan, we made sure to get a lot of input from our government partners because we wanted to make sure we were studying what they were interested in. So we had tons of outcomes. It was pretty exploratory. The way we deal with it there is we group these outcomes into families and we do multiple testing adjustments. We deal with the multiple inference problem formally. So it's so many tests in that case. But that's a case where pre-specifications useful for other purposes. It's a case where we really didn't know exactly what to study when we started the project. Again, when there's a lot of outcomes to look at, it opens up scope for lots of data mining. So pre-specification is still useful. Okay, so what did Newmark find? He did all this, he registered his plan. I will say that originally, Alan Krueger and David Card were meant to take part in the symposium, but they backed out. They didn't want to participate. I don't know all the reasons why. So it was just Newmark's study that went through the refereeing process. Newmark basically found zeros or small negative effects. He sort of found these effects that were neither the positive or zeros nor the big negative effects. He kind of found this kind of like weak negative effect kind of close to zero. It wasn't a very well-powered study as it turned out. So it wasn't like the definitive study. It was kind of in the middle of the range of existing estimates. So on one level, that's kind of reassuring that when you can't mess around with the data, you kind of get something close to what you expect. A bunch of studies since then have sort of honed in on estimates in that range that he found. So it was actually scientifically useful, but it was underpowered. So it wasn't the perfect design. One thing that Newmark does, which I think everybody who pre-specifies work should always do, is everything that wasn't pre-specified in his paper, he denotes very clearly. It's like, oh, you know, here I didn't include a lag or I should have include a lag defect. I'm gonna do that. I'll show it to you both ways, but I really think this is the right way. So there's a clear, any deviation, whenever there's gonna be pre-specification, it's very important any deviation from the plan is well-documented. Or else, what's the point? Okay, so let me just move on again because I only have a few more minutes. Again, the Newmark approach is very close to what we're calling registered reports. It's sort of forgotten episode. I've been at a bunch of conferences where folks from the Center for Open Science and elsewhere are trumpeting this new article form. But there are some precedents for this approach. And I think the Newmark case is a really interesting one. And my general view is there's just a ton of applications of this work. Again, new data releases, future policy changes, future elections. And we should all think if we're gonna do this kind of work, you know, ask ourselves, why aren't we pre-registering our hypotheses? You know, if marijuana's gonna become legal in California, we were talking about that before, folks are working on that. If we know on a certain date, there's gonna be a law change, pre-register your hypothesis about what's gonna happen to the price. I mean, it would make it so much more scientifically powerful. If everybody knows, there was no possibility of tendentious reporting. Yeah, in the back. Is the mic, do we still have the mic? It's been a while. Chris Chamberlain did have a several cortex. Sure, it's been like three years they've had registered reports. And it's still not happening, right? I mean, it's still absolutely not the mainstream. Very, very few papers have been registered. Why, what is the social or whatever argument that, I mean, what's the cause? What's the reason for that? We've had endless debates about the issue of why it's not happening. I think I'll kind of answer it in two parts. One is it's starting to happen in the sense that, let's say, for experiments in economics, in development economics, people register their papers now. They register a pre-analysis plan. It's almost ubiquitous. So in certain subfields, it's starting. And I think that's how it's gonna work. There're gonna be networks where the norm shifts. Shifting the norm in the broader network isn't gonna be one year's work. It's gonna be many year's work. And then I'm gonna get on the record here. We're being filmed. When it happens, it's gonna happen very quickly. Would be my guess. I think so. That would be my view. So I think with experiments in economics, it's starting to happen. A lot of people aren't even that aware of this stuff in the mainstream. I think there's something of a generational issue. When I've talked to grad students and younger scholars, the kind of immediate, median reaction is, that's cool. Like, I wanna do that. I wanna be part of that mission. I think for folks who, and I'm in the middle because I'm the first 10 years of my career, we weren't registering anything. And in fact, I'm just publishing a paper now where we have to put in and put in the footnote because it's becoming the norm in my field to have a pre-analysis plan. I just wrote the footnote this morning saying, we did not register a pre-analysis plan because we did the data collection before anybody was doing it. We're sorry. So I think there's gonna be resistance among those who say, wait a minute. If this is the only type of good research, what about my last 20 years of research? So I think it will take time. I think it'll be generational. I will tell you just in 30 seconds or a minute at the end of this talk, in a couple of minutes, what we're doing at BITS to try to speed it along. But I'm still hopeful. So let me come to that. Let me come to that. Okay. I think we're supposed to go till around two. I was gonna get into another paper. Maybe I won't go into it. I'm gonna make one point. There's a really nice paper. Del Ray et al. It's co-authored with John Ioannidis, so maybe many people know. It's Stanford, eminent scholar, medical researcher and scholar of transparency, where they make the case not just for registering perspective observational studies, which is at the point of my lecture today, but for registering all observational studies. And in other words, the question is, is it desirable to register non-experimental, non-perspective observational studies? They say yes, and they give a bunch of arguments why. I'm not gonna go into all of them because I know we wanna get to the question and answer. I will say one thing though. They raise a concern about this approach, which I feel very strongly, and which makes me reluctant, even though I'm heading up BITS and I'm pushing for transparency as much as I can, I'm pretty reluctant to support registering non-perspective studies. And I'll tell you why. Let me just jump ahead to this concern. When there's pre-existing data that's out there, like tons of data that I could have had access to already, it's basically impossible for me to credibly validate my claim that I'm registering these hypotheses before seeing the data. And it would be all too easy to mine the data, go in, register the hypotheses that I like, that work, that correlate properly, and publish something and say, oh, it's pre-registered. Look, my p-values are right. Ian Edes and Dal Ray's response in their piece, and it's a very nicely argued piece, is you know what, norms will change. Researchers fundamentally are ethical. They won't wanna do that. They're going to, a norm will emerge where before you touch any data, you're gonna register your hypotheses. They may be right, I could be wrong. I just think it's all too easy for somebody to peek. And so I'm gonna look just for 15 minutes, like I don't know if that variable is even clean enough to you, I don't know what the distribution, maybe it's bimodal. Maybe, you know, you could imagine a bunch of reasons why people, actually what is the correlation? Oh, it's no longer pre-registered at that point. So I think there's a concern with non-perspective work that even though it could be useful to register these studies and they raise a bunch of issues about why it would be very useful, and maybe we should encourage it just to start the process and see if they're right, that it'll become a norm. My fear is it'll undermine the credibility of all the other registrations that are out there. So with experiments, with prospective non-experimental studies, you can pretty credibly know someone is in data mining. You pretty much know what the p-values mean. We haven't even gotten there yet. If we try to go this far, people say, you know, this registration stuff's just all a scam because my friend was looking at the data before he registered his hypotheses. So I don't think we're ready for this yet. Maybe someday, but I'm a bit skeptical. Okay, let me wrap up and I won't get into the details on this paper. All I wanna do is just put up a slide on our initiative here, which in some ways is like a sister initiative with bids and Kevin and I were just talking before and we determined that bids and bits, in addition to having a brilliant, brilliant names, were founded in exactly the same month, you know, because I think he was kind of like, you know, what, who came first? Which name came first, you know? So I think it's been exactly three and a half years. So what do we do at bits? And this gets the question, the gentleman who asked before, bits has training courses and workshops. There's gonna be a summer institute here at Berkeley. We've had a few already. We've done a bunch internationally to teach people about these approaches, how do you pre-analysis plans? Do you replication, et cetera? Do you meta-analysis, other tools in this area? We have a great bunch of online pedagogical materials and how-to guys. I'd really recommend you look at the website. There's a lot of slide decks. There's a lot of lectures, videos on all these topics. It'd be great if you're teaching a class, you wanna bring in one lecture, a half a lecture on these topics. There's like off-the-shelf materials here to use. We have grant competitions to fund research on transparency, replicability, meta-research. We've also just started this past year to award prizes. We were very generously funded by the Templeton Foundation to award prizes to young leaders in the field. The young leaders who are really pushing the envelope on these topics, and we're also creating, and this may be really valuable for people in this room, we're creating what we call the BITS catalyst network. And I think we have like 30 catalysts so far. Catalysts are folks who make the case in like a one-pager why they're interested in transparency and reproducibility, why this is part of their workflow, why this is part of their intellectual agenda, and then they're eligible to apply for several thousand dollars to hold a workshop, to fly a speaker into their class. Basically it's just seed money to do work you wanna do in this area. And it could be to hire an RA. So just, there's a bunch of stuff going on in this area. And I think between BITS being here on campus and BITS, you know, we're coming from different disciplines mainly, although there's some overlap. I think we're indicative of the fact that the movement has momentum right now, and people are viewing data, and viewing reproducibility and transparency differently than they did 10 years ago, five years ago. Thanks. Sure, yeah, it's fine. There's an issue that I didn't hear you mention at all, which is if you're gonna publish exactly what you're gonna do, how are you gonna do it? Well, what's to prevent somebody else from doing the work? Or would that actually be a good thing? Cause then you don't have to do it, but you get a lot of the credit for having thought of the idea in the first place. Yeah, I skipped over that. I made a strategic decision. I think I have two bullet points in there. Let me mention how it works. Both the AEA registry and the open science framework allow you to timestamp a document, a pre-analysis plan, et cetera, but not make it public right away, but not make it public right away. So they have a window of privacy, and I forget if it's three years, something like that. So there's a pretty decent window, and again, I think the idea there is they don't want people not to post because they're afraid of getting scooped. And so that's actually, that could provide some protection. So you have at least a few years to get your research out. Now in terms of some of these, like what I was talking about the Dal Ray, the observational registries and whatnot, again, if you don't have that kind of timestamp and the data's available, if I'm running a $5 million experiment in Sierra Leone and I've got the grant and I'm working with the government, hell, I can post that online. No one's gonna redo my project. I mean, it's like a unique project. But if it's an existing data set that anybody can download in 30 seconds, I'm really concerned about scooping there. So any of these proposals for observational study registries would have to give people some kind of grace period after which it becomes public, right? So there's another fear there, which is almost like patent trolling, which is like people could post 500 things. And then anytime anybody works on any topic related to their 500 hypotheses, they would claim credit for it. And maybe they should, maybe they shouldn't. It's just really different than the way research credit is allocated right now. It'd be a really different model. So, and you hinted at that with your point. Like if I posted and someone else doesn't, maybe I get credit. It's another reason why some of the kind of perspective, the non-perspective observational study registration would face challenges, I think. We'd have to figure a lot of stuff out. Thanks a lot.