 Thanks for taking the time out of your day and your life to join us today. I know for many of you, it is pretty late in the evening slash approaching early in the morning. So I especially wanna share my thanks with all of you. And today we're gonna be doing a deep dive on replication in education research. And my goal is to talk about a little bit, give you a little bit of my background, give some explanation and justification for why I think replication is relevant in education research. Actually define what do we mean when we're talking about replication and education research. Then spend a little bit of time talking about what support and funding there is for replications in education. Make some initial connections between replication and equity in education. And then end with what I think are some, instead of FAQs, some FA comments, which I view as frequently asserted comments that I hear when these conversations come up pretty regularly. And then hopefully we'll get to hear what your questions and comments are as well. My hope is that by the end of the session that you'll go from being curious to at the very least being conversant about replications. And then hopefully we can also begin to set the stage where you can transition from just talking to talk about replication, but to also begin walking the walk with replication. So my goal, we're only gonna be together for an hour, so let's see what we can do. So my experience with replication wasn't quite from birth, but it was pretty early on in terms of my career as a researcher in that for my dissertation, I sought to replicate a finding that I believed to have been well-established in many different areas, but I wanted to see how it worked in a slightly different context. And so I say that because I want it to be clear is that I was trying to replicate some previous work because I believed it to be true, and I expected to be able to replicate previous work in a different domain. But then when I collected my data for my dissertation and sort of analyzing it, I failed to replicate every aspect of the previous findings that I expected to replicate. Essentially nothing, nothing worked. And my initial reaction was that this was my fault and that I didn't know what I was doing as a researcher and started to wonder whether or not I was a bad researcher myself. Now in the year since that time, I've seen that others have also failed to replicate some of that foundational parts of the work and maybe my research results didn't have anything to do with me or the quality of a researcher that I was. And that dissertation experience kind of started to be down the road to thinking about the importance of things like replication and not always assuming that published findings are gonna be universally true. And things like just because it worked in one setting does not mean that it's gonna work in a different setting or for different students or for different researchers. At the same time, the opposite I think is also true is that just because something doesn't work once does not mean that it will never work at all. And knowing whether a finding can be replicated and knowing if or when or how or for whom a research finding might generalize is really important when making decisions about education. And I think the bigger take home that I've gotten in the years since this dissertation, since my dissertation was that in education and in research more broadly, I think our actions do not always align with our ideals. And I realize that's a pretty big assertion and for many of you, you maybe should be nodding your head and say, yep, I've seen plenty of evidence already to agree with that and others may not be there. And so I wanna spend a little bit of time illustrating how I came to believe this assertion and hopefully get more of you to see where I'm coming from. And so I'm gonna spend a couple of slides talking about what are our actions now and why is any of this a concern? So in 2016, this paper was published in Nature asking many researchers, is there a reproducibility crisis? And as hopefully you can see on this slide, 52% of researchers said or were half said yes, there is a crisis and it is significant. Another 38% said yes, there is a crisis, but it's a slight crisis. 7% didn't really know if there was and only 3% of respondents said that no, there was not a crisis in a reproducibility crisis and research. So the vast majority said 90% said yes, there is a reproducibility crisis. Why might they have thought that? Well, in the same survey, oh, so what does this tell you about what scientists think and what does it make you think and made me think that, yeah, there's pretty good consensus amongst those people who are doing research that there's a crisis. Why did they think that? Well, the same survey asked them, have you ever failed to reproduce an experiment? And so they asked two different ways, have you ever failed to reproduce somebody else's research? And have you ever failed to reproduce your own research? And this is in a variety of different fields, none of which are education, but across the board, the majority of people responding said they had failed to produce the research of others. And in most of the fields, the majority said that they had failed to reproduce their own work. And so if this is happening in fields like chemistry, biology, physics, engineering, medicine, where a lot of the research is done in much more controlled settings in the laboratory than where education is done, makes me wonder, what does this say about replicating research and education? Is replicating research or reproducing findings easier or harder than in these fields? I don't know, I don't know that we ever tested that. But for me, my takeaway when I see things like this is it makes me a lot more skeptical about the research that I consume. And it actually makes me a lot more worried about my beliefs and my behaviors. What am I basing that on? And what is the strength of that foundation if so much research can't be reproduced even though it's published? That makes me a little worried and a little skeptical. And so why should we care about all of this? Well, I think it's because we make a lot of decisions assuming that they're informed decisions and that, oh, this works or this doesn't or I know that this is gonna work because somebody has found this before. And at least that's how I make a lot of my decisions. And so I want to know whether or not the research that I'm basing my beliefs and my behaviors on can be reproduced by another group. And if they've tried to reproduce it and can't, that makes me a little worried. Do I know whether or not they've tried and failed or do I only see the things that work? I want that information because I want to make informed decisions. And so I'd love to say that I've got this great analogy about the slide with the rope bridge over this big valley and how if we deviate it, what from happened and there's danger and consequences, but actually this is just the image that showed up as a suggestion. And I thought, boy, this looks great. So I'm gonna go with it. But I've talked a little bit so far about the examples. Those are not in the education world. What do we know about education research in this area and this topic? So here's three examples is that some researchers looked at the database of dissertations and what the outcomes are reported in the dissertations and then tracked them to journal publications. And they found that the non-significant outcomes in dissertations were 30% more likely to not ever show up in the published article. And my guess is the published article is what gets seen far more often than the actual dissertation. So those non-significant, oh, this didn't work or we looked at this and it wasn't statistically significant, that disappears from the conversation far more likely than the things that did work. That's what is more likely to show up in published research articles. And I think it's been pretty well established. The effect sizes in unpublished studies are typically quite a bit smaller than the effect sizes that are in published studies. So that means if we see a publication and we say, oh, this intervention works and this is the expected finding that we get and we're reading about it in a publication that might be quite a bit more inflated than when that intervention has been tested but that's not published. And that's gonna affect our expectations for what the consequences or outcomes of conducting an intervention in education should be. A third finding that I thought was really interesting but again, not particularly surprising to me was that a research commissioned by those who develop the intervention is about has results that are one and a half times the effect sizes when independent studies are doing it. So if I make an intervention and I publish the research on it, I'm gonna find generally bigger effect sizes than if an independent group tries to apply that intervention. So what does all this mean? Who's it for? I think this is all kind of showing that if our ideals and research are we're gonna inform the conversation and we are dispassionate and we're unbiased and we're giving an unvarnished, honest, clear truth of this is what's happening. When I see these types of examples add up and when I see the examples of researchers saying I've tried to reproduce somebody else's findings but I can't, I really start to believe that, wow, this isn't our actions and what we're consuming maybe isn't not as clear as what we hopefully aspire to. And it shows, I think, that we're not always living up to our aspirations. And why? Why not? What's causing us this gap? And I think there's a lot of behaviors and if you were in the opening session for this conference, you heard Brian knows I've talked about some of their work surveying researchers from various fields on whether or not they've performed various behaviors or not with some colleagues, Jonathan Plucker, Brian Cook and Jared Hodges. I also published a paper focusing just on education research asking researchers what types of behaviors they have done and this is a table just of their self-reported use and now the font is very small but what we basically asked was have you ever used this practice in your research? And so we asked them things like have you ever omitted an analysis that you conducted from a publication and two thirds of our respondents said, yeah, I've done that before. All the way up to have you ever filled in missing data without telling the readers that you're filling in missing data and 10% of our respondents said, yes, I have done that before. And so what we found was we asked about 10 different behaviors and we found that five of these what are called questionable research practices by many had at least 40% of our respondents saying that, yes, I've done this behavior before and 90% of our respondents said that they had done at least one of these questionable research practices. And so if a researcher is analyzing some data but not including all of their analyses that's where that omission can show up why are they omitting and which results are being omitted. In our paper we didn't just ask about the questionable research practices we also asked about open science practices and we asked about five in particular and today I'm gonna focus specifically on replication and in that, in our respondents we found that 43% of education researchers who responded to our survey said, yes, I have tried to replicate something before which I thought was great. And similar to what Brian talked about from his CLS data was that 98% of our respondents said that they believed that replication should be used at least occasionally in education research which I thought was, which surprised me was much higher than I expected but maybe what I learned this morning was that maybe I should have expected that a little bit more. And so why is it again going back to what's going on with these questionable research practices and open research practices I think it's important that I reiterate the point is I'm not here to say that researchers are bad people. I think sometimes there's some can be bad incentives and bad situations. For example, here's a tweet that's getting a little old now. I don't know Ben Jones at all but I know the feeling that he is communicating here when he got a decision letter from a journal saying that they won't publish because the data didn't support his predictions. And the editor suggested that they rewrite the beginning so that their paper, their predictions the beginning of the paper matched the predictions that they got. I know this is just an anecdote but I have one of my own early in my career. I submitted a manuscript that had two studies in it. The first paper had relatively clear the first study had a relatively clear straightforward results in the second study attempted to build on that a little bit but it had mixed results that weren't as clean and clear as the first study. And after the first round of review the editor suggested that we drop the second study because it didn't tell a clear story. I didn't know what to do. I was relatively young in my research career and then I thought, well, I can just publish the second study as its own paper. And so we published the first study and then I tried to publish the second study and it got rejected and then it got rejected and then after a while I kind of had to move on. So right now in the research literature the fuzziness showing that another similar study didn't get as clean and clear results is kind of in my file drawer and the world doesn't know about that. And I know these are both anecdotes and so I say them to show that it's not necessarily always about some analysis or some studies not being reported it's not necessarily always up to the individual researcher. There can be some incentive systems and structural issues that get in the way as well. So I think that's important to say and I think these individual stories can be really powerful when we think about how do we want this scientific and academic research system to actually look like? And there's some people, including people I know and love quite well who say that we shouldn't let the facts get in the way of a good story. So maybe when we get the suggestion of cleaning up the front end of the paper to match the data then yeah, it just tells a clear story and it's easier for people to remember it's easier for people to understand and then that's just a good story and it's a good narrative. I love my dad and I think he's a wonderful storyteller but he's not a scientist and he's not a researcher. As researchers, I think it's our duty that we absolutely let the facts get in the way of a good story. But not everything has a clear story and I think that's okay, especially if we want our actions to actually live up to our ideals and I'm hoping some of the evidence that I've been giving to all of you kind of shows at least my journey and my understanding of why I think our actions don't always align with our ideals and to that big assertion that I made. So where does that leave us? If what we're doing doesn't actually align with what I think maybe we're trying to do or maybe what the rest of the world believes we're actually doing, where does that leave us? I think that leads us to some pretty big problems. On the bright side, it does say that it leaves us with some opportunities for improvement. And some might say in the bright side, well that in education, schools and policymakers don't always base their decisions on our research anyway. So maybe we're not doing all that much harm by not living up to our aspirations. But that's pretty cynical perspective. I think, yes, we have room for improvement and hopefully we'll spend some time in this session and at this conference talking about ways that we can improve. And hopefully that a year from now or five years from now, this gift is does not necessarily represent the situation quite as well. So what can we do? If that's where we are now and this is where we are now, what can we do and what should we do? And I wanna go back in time to something that was a quotation that was actually stated before I was born, John Tukey. So that confirmation comes from repetition. Any attempts to avoid this statement leads to failure and probably to destruction. That's pretty gloomy perspective, but in this context, Tukey was talking about trust and stability in data analysis and that confirmation also comes from repeating data collection, data analysis. And I think this is where replication comes in but the key pieces we actually have to look if we want to assess whether or not something can be confirmed. So that's a whole lot of background context and assertions to get to you to actually the point of what we'd like to talk about today. And that's actually replication. And I've got pie on here, not just because for me, it's just it's lunchtime, but also because replication and what is it and why is it and why should we do it and what does it actually contribute to the research perspective? There isn't a universally accepted answer to any of those questions. And there's actually many different ways you could slice the pie to answer those questions. And what I'd like to do next is to not give you one definition and one conceptualization of replication but to give you a brief thin slice of several different definitions and conceptions of replication so that I'm not giving you a skewed or biased perspective and just sharing with you what I think, but also what others have thought and what others are doing in the replication world. And that's where I think we have to do some look. So this will be fast, but it's just a broad conceptual framing to give you a taste so that then you know maybe where you want to go next to do a deeper dive and to learn some more about it. So the first conceptualization and view of replication I'd like to talk about, this is from the US National Science Foundation and Institute for Educational Sciences where when they're talking about replication, they're talking about studies that involve collecting and analyzing data to determine if the new studies and the new data yield the same findings as a previous study. So that's really about collecting new data and comparing the new findings to a previous finding or a previous results. And I emphasize the new analyses and new findings because there's a lot of different terms that some people use interchangeably and other people use to refer to different things that I think it's important to emphasize here. So for this example, which I thought was a really nice, clean and clear design from the Turing way where when we're talking about replication, that's where they mean they're talking about it's the same analysis, but with different and new data. If you're trying to reproduce something that's the same analysis with the same data. So that if I collected data and analyzed it and got a result, but then gave that data set too, could you reproduce my findings if you conducted the analysis on my data? Replication, many people do it specifically only about new data. And then different analyses are about whether or not robust or it's generalizable. But I think this is a useful framework for when we're talking about replication. It's often about using the same analyses but with different data. Here they're saying if it's different analysis with different data, that's testing whether or not it is generalizable. Other folks may call different analysis with different data, is that also replication but a different type of replication as same analysis with different data. And that comes up in this type of definition we're looking at not just what is replication but what types of replication are there. And these are a common framework which I've used in my own work of looking at a direct replication versus a conceptual replication. And in direct replications, that's where researchers are trying to stay as close as possible to the original methods of the original research. So they're still collecting new data but they're trying to follow that same research recipe that the original researchers found and their goal there is to verify or corroborate the original finding. Now conceptual replication is different because in conceptual replication, this framework, the new data are being collected in a purposefully different way to test an underlying hypothesis or to assess generalizability. So they could say, okay, the original research worked with first graders. We want to know if we can replicate it with kindergarten students or the original research was in rural schools. We want to know if we get similar results in urban schools. They're intentionally altering one component of it. It gets tricky when a conceptual replication changes multiple components. So if you were to change the age of the population you're looking at, the setting or context of the United States versus someplace in South America versus Africa versus Asia, you're changing the context and the culture or which could also include changing the language or the measure that's being used. If you make many changes, it makes a little harder to make an accurate inference about the results because if the conceptual replication gets different results, well, what caused the different results when you changed potentially four or five different things? You don't know what caused that change to appear or to show up. So this is different types of replication research, breaking it down even further, Huffmeyer and colleagues actually reconceptualized for replication as a sequence of different studies, each of which have a different alteration and kind of a different purpose. So there's the original research but then they call the next step in the sequence an exact replication and that's often done by the same researchers, maybe in the same lab or the same classroom using the same materials, trying to keep everything exactly the same as close as possible. The next step in the sequence, they call the close replication and that's again, we're trying to keep all the processes and components the same, but it's now it's by different or independent researchers. Can independent team get the same results? The third step in the sequence would be constructive replication and that's where the first time in this sequence where new elements are purposely being added where now you're purposely changing a process or you're purposely changing the component. So now it's well, instead of looking at first graders, now let's look at kindergarten nurse or it worked with this particular standardized measure, what if we change to a different measure? Then that's where the contribution in what's new. The fourth sequence would be, what about a conceptual replication in the lab? And that's when you're starting to really make many operational changes in the steps of what you're doing and how you're doing it, but you're still testing the theoretical relevance, the underlying hypothesis that was being tested, but you're still in the lab because these were psychologists who developed this and then the fifth step in the sequence was a conceptual replication in the field. So it's leaving the lab and going to a real world setting. So my initial take on this is I love how sequenced this is, but I don't know if this is how necessarily replication is currently happening. And I don't think they were asserting that this is how things happened. They were trying to reconceptualize it as a sequence of different studies for how replications can be conducted and what the contribution can be for each of them. And I know this font is gonna be really small, but they mapped out kind of a sequence where they start on the left if an observed effect is and if it's successful going kind of all the way up, you can go through each of the sequence of the steps. So if your exact replication at first stage, which I have blocked off and read, if it was not successful, then they say that repeated failure at the exact replication stage, what does that mean? They said that well, the underlying theory and the hypothesis is being discredited. If the original research team that found a result can't get it again, that's after they try it over and over and over again, then that hypothesis is probably discredited and should be rejected. But if each sequence in replications are successful and have successful results, then we go all the way up here where it's an independent group in the lab, they've changed a bunch of different components and pieces of the study and they keep getting successful replications of results in the field, then and only then can we conclude that the theory applies and has a real world relevance. Every outcome other than this little red box that's blocked off here, every other outcome leads to either rejecting the underlying theory or you accept it with much more limited conditions and it's very, very narrow. And what I find this is a really useful tool in framework to think about when we're consuming research and we read a headline of one original study, I think often the way it's framed is to make this universal broadly applicable. Here's this theory that works and has real world relevance, whereas under this typology, we can only make those types of claims and assertions after a finding has gone through this full sequence of results. Another area that I'd like to talk about is this causal replication framework that Vivian Long and Peter Steiner have proposed. And this is specifically focusing on replications in randomized controlled trials and this causal replication framework provides a framework for designing replications to make a causal inference. So if you have an original study that you want to replicate, this is this causal replication framework helps you design that research and to be able to test the assumptions and the inferences that are made from the primary research. And just real quickly that I wanted to overview this is a really complicated and great set of papers that we don't have time to go into fully today. But it has a lot of assumptions and that these assumptions are all needed to actually design and test replication of whether or not you're actually replicating original research. And if your project and your trial doesn't meet all these assumptions, then a lot of the inferences that you want to make comparing your results to the original results break down. And so this is I think an important thing to keep in mind when we're interpreting research results and does the replicating research successfully replicate the original finding or not, we have to be very careful about understanding what the design is and did we design our replication particularly well. Now, the different models that I've talked about so far in replication really do all focus on these procedural and components and ingredients of what a study design sets. The next framework doesn't build on that. Last year, Brian Nozick who we heard from earlier today and his colleague, Tim Arrington who I believe is the director of research at the Center for Open Science, they said that replications not about repeating study procedures, really the fundamental aspect of replication is about assessing the prior inferences and advancing particular theories. And so that replication, the behavior is actually about the opportunity to test whether or not existing theories, existing hypotheses and existing models are they able to predict outcomes or predict results that haven't yet been observed? So can you take a theory that was maybe supported in one study and then design and conduct a research study that assesses that and tests that theory or test those hypotheses? And in their framework, they're saying a consistent outcome in a replication would increase our confidence in the prior claims whereas an inconsistent outcome would decrease our confidence in the prior claims. And this framework is very different because it's not about copying or altering purposefully one particular component of the research procedure. It's really about testing that underlying hypothesis and that underlying theory, which is a pretty radical change from how many others have been framing and discussing replication research. So as I promised, hopefully delivered, that's a whirlwind, it's a lot to consume and it's a lot to keep track of. Fortunately, there's some tools to help you keep track of all of this. One of my favorites is called Fort, which is the framework for open and reproducible research training. It's a great website, which includes a glossary of terms, which is fantastic. And I actually had a paper come out earlier this week in nature, human behavior that introduces and talks about what the glossary is and what its goals are. But this website is so much more than just a glossary of terms. It also contains a ton of great introductory and advanced materials that will help folks incorporate replication and other open practices into your own research, but also into your own teaching practices. So if you want to review what are these different perspectives and what are these different papers, I think this website could be a really great resource for folks to visit to learn more about it. And more about replication and different frameworks and perspectives. Now, I've talked a lot about is, could a replication be successful or not? There's actually been some research, some of which I've helped conduct on replication rates in education. So we've talked about all these different frameworks and replication, how often is it actually happening and what are the results and what are folks actually finding in this? So in a paper that I published with Jonathan Plucker back in 2014, we took the top 100 journals or the journals with the top 100 impact factors and we just looked to see, are they ever using term replication anywhere in there? And so we searched for replica asterisks which would take care of replication, replicating, replicated. We want to know are folks using the term? And so that's that black solid line of what's the rate at which the term replication or replicate is being used. But then we actually looked at those papers and said, are they talking about it or are they actually saying we conducted a replication to get an estimate for how often replications are being published in education research? And that's where that dotted line shows up. And what you see is that there actually was some but very, very little for the first half of the last century but then starting in the 1950s, we started to see that line go up a little bit. So that by in the 2010s, about 0.2% which is one out of every 500 education publications was trying to replicate a previous finding. Now, I don't know what the right amount is. We can have plenty of conversations about this but one out of 500 felt a little low to me but how does that fit with some other fields? Well, we actually looked at a couple other fields and education, which is the red line at the very bottom that's the same results but just a different why access compared to say psychology which is now the solid black line. And you can see psychology, the increase also started in the 1950s but the increase was much more rapid so that by the early 2010s the estimated replication rate in psychology was 2% of publications were explicitly saying that they were trying to replicate a previous finding. So 2% is far higher than one out of 500 whether or not that's the right number or not. Again, that's a great topic of conversation for folks to have or for the field and the community to have. I don't know what the right answer is but this is all kind of dated now because this is early 2010s. Very recently, hot off the press, Thomas Perry and his colleagues looked at just the last 10 years of education research similar looks a little different but it's a similar estimate to what I just showed you was they started kind of will be left off and then kept track up through 2020 and you see a slight increase just in the last 10 years. And so their results suggest that replication has gotten more common in education in the last 10 years whereas we ended up at one in 500 publications. They said now it's gone up to one in 400 publications which is actually a pretty big increase. I don't want to belittle it at all but one in 400 still isn't that big, all that big of a number. And so if we want to talk about well, how can we get this relevance or this prevalence increased while one way to do it is to look at what type of support mechanisms is there for replication and education research. And in the United States, as far as I know the Institute for Education Sciences which is part of the US Federal Department of Education currently has one specific funding stream that is dedicated to funding systematic replications because if we want the prevalence of replications to go up and we want to know what replicates to other contexts to go up, we need to support them and funding is an important part of that support. I do want to specify that this funding stream I believe is specifically dedicated to conducting randomized controlled trials replicating previously assessed randomized controlled trials. And so that's a limited scope of what type of replication that they're looking to support. And so that their goal here with that support is that systematic replication building on the prior evidence to better understand what interventions to improve educational outcomes work under what conditions and so for whom will they work for? So that's getting away from that generalized, oh, we have one study that works, therefore it works for everyone in all situations and all times. This funding stream really looks to say we shouldn't make that general inference from the get to go but then with replication we can start to learn more about where interventions work for whom and in what contexts and maybe the magnitude of the effect might change as well. So again, this particular funding stream they want you to under that conceptual replication framework where you're intentionally trying to replicate and duplicate the procedural recipe but they want researchers to purposely change one particular aspect of that. So that could be a conceptual replication whereas the original research was in an urban setting now will it work in a rural school? They're purposely trying to change something to assess how systematically things can be replicated. Now, if this is the sole funding stream dedicated to replications and educational research my first takeaway is that we're gonna need a bigger boat that if replications are informative and helpful for all of us having one dedicated stream that's only dedicated to conducting and replicating randomized controlled trials is not sufficient. Now, what can we do? Obviously we could change massive policy. I don't know how in control all of us are in doing that but we could do smaller things like building replication into our other projects. So if you're collecting data somewhere else or if you have another grant are you trying to at least replicate your findings with a different sample or in a different project to be able to see and build into replication to make it a normal part of your workflow. And when I think about needing a bigger boat I think of the line that the sea is so great and my boat is so small like what can I do? And that's where I think I can start to replicate when my other projects and not just only when I have a specific grant to conduct a replication but if there's only so much that we can do I think this is where prioritization comes in and this is a big conversation that I think the field has to have is what types of findings do we really think need to be replicated first? What are urgent prioritization or urgent replication that we need to conduct think about it rather than what would be easy or what could be funded but what do we really need to know? And that's where at least I start thinking about things like well, what do we base our decisions on? And if we're gonna use an idea or a finding to influence a decision or to influence instruction or school policy to me that rises up maybe a little bit more to the top from an applied setting, others who approach replication from a theory perspective they may say well, what theories are getting a lot of attention or rebasing our actions on let's test and interrogate those theoretical findings. Another area of prioritization that I think has gotten a lot more urgent attention and important attention which I think is very needed and I think is a very interesting growing conversation in the open science world is how does open science connect to equity conversations? And equity in education is a huge, huge priority and I'm currently working on a project where I'm trying to talk about how open science values and practices align and can support what I believe to be a very urgent priority of equity in education. And I think open scholarship generally and replication specifically can be really important components to the equity efforts. Ron Daniels who's the president of Johns Hopkins where I worked actually with a book last year titled what universities owe democracy and open scholarship was one of the four things that he said that universities owe to democracy. And I think it's very relevant to education because in education I think the ultimate consumers of our research are parents, are practitioners and our policymakers and in my experience when I'm talking to anyone from those three groups their primary interest is usually around how does that research finding apply to my kid? How does it apply to my classroom or to my schools? It's very practical and very applied. They want to make informed decisions and they want to know what they should expect if they take a particular course of action or if their kid goes into a program or has a particular experience. They wanna know, can I expect a similar outcome in my context is what you're talking about in that particular research. And those are big questions about generalizability that require I think replication and I think they're also certainly relevant to equity efforts. When I think back to my dissertation lessons just because it worked in one setting we doesn't mean it's gonna work for you in your particular setting. I think there's knowing whether or not a finding can be replicated and knowing for whom it can replicate and what context and what situations and for whom it generalizes is really important when we're making evidence-based decisions but that's also a driving force in many equity conversations and assessments that are looking at whether or not something works for the majority is that gonna work for others as well or if something works in one setting is it gonna generalize to other contexts as well and should we expect similar results or not? We don't know the answer to those questions without replication. And so in my mind these conversations really align quite a bit with the goals and values of Open Scholarship and Replication as well as with the goals of equity in education. Where equity is a very big concern for a lot of folks I'm wondering if folks who are interested in talking about growing open science and growing support for replication can we align ourselves and help support the equity initiatives that are happening in education to be able to say we can help answer some of these questions and where those questions we feel that we do have answers I think often it's because replications have been conducted. I'm sure this is something I'm working on this is actually the first time I've presented it I'd love for you to reach out to me if you're interested in adding to this and building on this because I think it's an important conversation of how we can build support for replication in Open Scholarship but also how we can apply it to help address and answer questions around equity that already have a lot of attention and already have a great sense of urgency around them. So that was a whirlwind I know I'm sure you have a lot of questions and I want to leave some time for that but while we do that I wanna maybe preempt some of your questions a little bit with some questions that I often get and often hear and I see there's some questions in the chat but I can't actually see the chat because I'm sharing my screen and so I wanted to share with you and maybe we'll see if this is a preregistration to see if my previous experience replicates to this particular conversation. Some of the frequently asserted comments because when questions come up it's there often not questions there are often comments. These are the types of things that I often hear with around replication is like, oh, but all classrooms are different so why should we expect it to results to replicator but I do qualitative research that has nothing to do with replication or my fundamental foundational beliefs of qualitative research say that we shouldn't expect things to replicate or but the person that I'm trying to replicate might get mad at me or what if someone tries to replicate my work or I use secondary data or replications won't get published. Hear these all the time. I think these are great very valid comments to make. I'm less concerned about all of them than others but that may be because I've spent a lot of time thinking about it. So I'm happy to talk about any of these topics if folks are interested. So let me check in the chat here briefly. Yeah, you go for it. Yeah, for it is great. Who said that in terms of what to do democracy what do universities owe democracy? That was a book that came out last year that was written by the Johns Hopkins University President Ron Daniels. That was the title of the book what do universities owe democracy and Open Scholarship was one of the four things that he said. Thank you for putting that in the chat. The answer there's the book I'm sure. Oh, and that link is also great because I believe if you follow that link it is published open access. So you can access the book without having to pay for it which I think is great. Those are the questions that I see in the chat if folks want to spend some time on any of these particular comments or others or other questions and issues I'm happy to spend some time talking about that now. My kind of big take home that I hope folks believe in support that have been convinced of is that I think replication is an important part of a healthy balanced research diet. How big replication should be a part of that diet? I think there's a question that we haven't answered quite yet but I think having it as part of the education research diet will help us have be more confident about knowing what works for which students and in which contexts in which situations. I see in the chat how do you prioritize replication since we can't replicate everything? I think there's a couple different things to do that. There we can look at what would get funded and so I like to get a paycheck and so I would wanna structure and prioritize what would a funder want to do but right now the only funding that I know about is for randomized controlled trials. I think replicating results or testing theories that are the basis of how we're educating students and how we're creating schools is probably the most urgent thing. And so in terms of replication, there's a big paper that came out a couple weeks ago from Vanderbilt's testing whether or not universal pre-K leads to long-term positive educational consequences and because a lot of schools and states have been spending millions if not billions of dollars on the assumption that pre-K is very beneficial and it leads to positive outcomes. If we're spending a lot of money on it, I think most stakeholders generally want to know what is the expected consequence of it? And so if we're gonna be spending billions of dollars on something I'd wanna know has it been independently replicated before state, local and federal governments start writing those checks? I think from maybe another perspective is what do we think really works? What do we think is really, really effective in helping students learn or grow or be more confident and participate in school or prevent negative consequences like dropout or being suspended from school? If we think something is really, really effective in helping something we want or preventing something we don't want, before I trust it and know whether or not it's actually effective, that's the type of educational outcome I would like to see replicated. Okay, also in the chat, what are your thoughts about infrastructures we need to do replications? Should we do open source infrastructure? For example, to store and share data, I am a big proponent of all aspects of open science and open scholarship. That's where this conference is talking a little bit more about open source infrastructure for open data. If the more data that are open, the more we can test whether or not that data and result can be reproduced because again, that's looking at the same dataset. And if we can't reproduce the original results with the original dataset, I would say maybe let's look at what is causing that problem before we spend the time and energy collecting new data to see if we can replicate it or not. But I think having shared data is a really, really great relevant stepping stone to testing and figuring out what needs to be replicated. Here it says one out of 500 publications that is a replication is low, but what would be a good number? One out of 50, do you think we should have replication bounties based on what citation prevalence is an incentive structure? So something that my colleague Jonathan Plucker and I recently proposed in a paper that came out in educational psychologists, which is a special issue, which had a special issue about open scholarship. We're trying to reframe the prevalence conversation, which we've helped been a part of going away from what's a good number into thinking more about context than what's ready for something. And so the analogy that we made is that I don't know that there's a good number that I would suggest about what percent of adults should be on blood pressure medication. It doesn't make sense to say what percent of adults should be on blood pressure medication. I think what makes more sense is to frame the issue around based on particular individual contexts and their symptoms and their health behaviors and their biological, I'm not a doctor, I'm not speaking very eloquently on it, but I think it's on that individual context that should be the basis of whether or not they get put on blood pressure medication. I think the same thing is for findings and theories. It's not necessarily about numbers of what results or what papers and how many should be, but it's based on the particular context. And that should be determined, I think that should help determine what needs to be replicated. And so if we're assuming a theory is true or we're applying the results in our schools or in our policies, I think that's what creates the demand for what needs to be replicated. And if something gets published and no one's talking about it, no one's assuming that it's true, I think there's probably less urgency around replicating that. In terms of replication bounties, bounties is a pretty scary word in my mind. I think of the context of finding a flaw or finding something that's wrong, which I think has a host of concerns and potential problems. I think having rewards for replicating and I would love to see funders and I would love to see even journals say, hey, here's a finding that is getting cited quite a bit. Can you replicate it? Do you want to collaborate to collect some data so that we can see if we can replicate it to kind of lower the bar and the barrier to entry to conducting a replication? I think that type of maybe more carrot than stick, I think having rewards for conducting replications, I think might have a little more positive spin because I think there's already some apprehension. As I said about, oh, what if someone tries to replicate my work? Does that mean that they think that I'm wrong or that I did bad research or they don't believe me? I think it's a real concern for a lot of folks and I'm a little worried about that. I see a clarification. That's what I mean. The more something gets cited that there's more incentive to replicate and to advance it. Yeah, I think citation is one good potential metric. If something is getting cited a thousand times or a hundred times, clearly the research community is talking about it and has something to say about it. That could be one metric for looking at what needs to be replicated. But I think also that because education is such an applied field, I think is the results being assumed to be true or assumed to be informative about educational policy or practice? I think that's another potential useful metric to use when we're trying to prioritize what should be replicated and what should not. Any other questions or comments or frequently asserted comments that folks would like to see covered? I can go back to that slide. If not, I wanna thank you all for joining me and hopefully this has helped with some of your thoughts about going from being curious to at least being conversant about replications. Oh, I see a couple other questions come in. What's my opinion about replication and qualitative studies? I think it can be very relevant. I recently published a commentary that asserts that replication is relevant for qualitative research. I don't quite have enough time to go into it all now, but I think the one sentence or really brief version is that I think replication research supports some pretty well-established values in qualitative research around transparency and intentionality. And I think replication can also be used to more formally assess the well-established tradition of transferability of findings and that right now I think qualitative research often puts a lot of inference about transferability on the reader. And I think replication can be one way of potentially more formally testing transferability of findings. And I think also replication can evaluate connections between reflexivity and position statements in qualitative research findings. Right now, I think we believe, I think with some good reason that reflexivity and position statements help communicate how the researcher in their perspective can potentially play a role in what the results look like. Replication of qualitative research for those with different position statements and different positionalities may be a way to formally assess some of those things that's new. I am not primarily a qualitative researcher, but many of my co-authors for this commentary were and we had some great conversations and we're hoping that our commentary sparks more conversation about the applicability of replication in qualitative research because I think replication's applicability in quantitative research is also not a well-established tradition and it's gonna require some time for the community to engage in dialogue and think about how is it relevant? And personally, I think replication in quantitative research, many of these conversations about transferability and generalizability is the quantitative research world catching up to many of the concerns of the qualitative research community has been discussing and trying to handle for decades. And it may just be that in some areas, this is the quantitative research world catching up and acknowledging and addressing some of the limitations and problems the qualitative community has been having for many, many years. One last question, do I think there should be seasons to replication incentivization, reasons to replication incentivization? I'm not sure I understand the question. Should there be seasons to replication incentivization? I'm not sure I'm following the seasons part. Boy, I hate to end on this note. I think replication incentivization there should absolutely be incentivization to conduct replications, but that there needs to happen from a variety of different places, I think from in terms of hiring practices, rewards and tenure promotion. Are we being rewarded? Are individual researchers being rewarded or are they at least not being punished for conducting replications? Are funders willing to publish, are funders willing to support replications? Are journals willing to publish replications? There's many different stakeholder groups that need to become aligned in their incentives and their actions in order to support a replication culture in education. So on that note, I wanna thank you all for joining me and hopefully this is just the beginning of our conversation and relationships around replication and I look forward to continuing the conversation and seeing about your thoughts and work and replication in the future. Thank you so much Matt. Yeah, happy to join up. I'm not quite sure I answered Jonathan's question at the end there, but Jonathan, if you're still there and wanna follow up with me, please feel free to reach out to me and hopefully we can chat.