 Hello everyone. If you're just joining us, welcome to MetaScience 2021 and to this symposium, which will be to replications make a difference. My name is Fallen and I will be moderating this session. We will have three speakers followed by I understand plenty of time for quite Q&A. So as we go, please do put your questions in the Q&A chat box and we will get to them at the end. And with that, I will hand over to our first speaker, Bob Reed. Over to you, Bob. Okay, am I, is my screen being shared here? I think so. Yes, you just need to go into presentation. Yes. So I'm Bob Reed from the University of Canterbury. I work with a research group called UC Meta and this paper I'm going to report on is written with my colleague Tom Coupe also at Canterbury. So the research question that we're going to be looking at in this study is do negative replications affect the citations of an original study, more so say than positive or neutral replications. So that's our research question. And to address this quantitatively, we're going to have to assemble four pieces. The first piece is we have to have replications. And then the second piece is that these replications have to be assessed. Was it a positive replication was a negative replication. So we need an objective assessment of the replications. The pieces we have to match these replications to their original studies. And then the last piece of the puzzle in some ways the most important piece is we have to match control studies to those original studies. So that's really the, the, the nuts and bolts or the guts of what we're doing is being able to have a good match. The way we're going to match is look at the citation histories of original studies to control studies. And if their citation histories match very closely, then we'll pair those studies up, and then look at how their certations change after the replication study has been published. Okay, so we're focusing on citations, we're going to match on the basis of citation histories. We're going to assemble a sample actually numerous samples, and they're going to consist of three types of studies. We're going to have original studies that had negative replications original studies that had positive or mixed replications, and then we're going to have the match controls. Okay, so, so that's the, the individual pieces of our statistical analysis that will become later on. We're going to draw our replications from economics and from two sources, replication wiki and the replication network both of which post replication studies publicly and focus in the area of economics. I'm not going to go through this but how do we assess the studies what basically we don't we let the authors of the replication studies, give us their assessment. And so, oftentimes in the abstract or the conclusion of a paper, the author of a replication study will say something like this. They might present robustness and placebo tests that cast doubts on the validity of the original study, or they might say this study suggests that the results are very robust regarding blah blah blah. And so, we let the replication study assess itself so to speak, and we take that as our, as our category of whether it's a positive or negative replication. Also the impression a reader would get when they read the paper most readers are going to basically take the assessment of the author. They might investigate it some more but that will be the, the impression they'll have of whether that replication was positive or negative. Now comes a tricky part of collecting control studies to match with our original studies. Well, the simple way is just go to the same journal or issue that the controls that the original study was published in, and just grab all the other articles that appeared in the same journal or issue. If it's in the same journal or issue it must have some commonality with the original study. So that's a natural place to look for control variables. But we're also going to cast a much wider net. We're going to look at all studies and scopus that were published in the same year as the original study that were also published in the same area the same very large general study economics or business or finance, and we're also published in those journals that published the original study so we have a collection of, you know, several hundred journals that published our original studies and we're going to choose our control variables from those journals. So we've got to two places we're looking for controls. And in this first stage, we're just casting a big net. In this first stage, we're going to end up with 422,918 potential controls and 225 potential originals. Now that's stage one. What we want to do in the next stage is narrow down those those matches. So we only choose ones that are very close. And that really is the core of our analysis. So two dimensions to finding good matches. First of all, studies differ in how much time has elapsed between when the original study was published and the replication study was published. It would be a lot easier if I'm matching on citation histories to find a good match when there's only two years or three years of citation history to match on much harder to find a really good match when I have seven or eight or 10 or 15. So as you can see from this from the histogram here, we have a wide variety wide range of years that have elapsed between when the original and the replication study was published. We do restrict our choice of controls to have to have at least three years between when the original was published and the replication was published. The first dimension we got to be concerned about the second dimension is the total number of citations right so can be pretty easy to match a control study if the original has, you know, five or 10 citations there's a lot of papers out there. But if our original has 500 or 1000, it's going to be pretty hard to find a really good match. So what we're going to do is basically have a sliding scale. That's going to be a little more relaxed when our original study has a lot of citations as measured at the time the replication study was published. Okay, so so nuts and bolts. Very quickly. Here's an example of original study that was replicated and the replication was published three years later. That gives us a citation history of two intervening years. And what we're going to do is compare our potential controls to the original for each of those years and look at the absolute value of the difference. The absolute value of the difference and equals minus one the year preceding when the replication was published and T equals minus two year before. And we're going to construct this variable called total absolute difference as the sum of the absolute values of the individual year differences between citations between the original and the control. So we're going to take all these controls and match them up and we're going to try and choose good ones. Here's here's the same example for a four year gap. All right, so now I have three years of a citation history to work with. Okay, we're going to go ahead and for every original every control. For every sample of three year intervening gap or four year intervening gap or so on. We're going to construct this measure of total absolute difference. And you should be able to figure out that when total absolute difference equals zero, boom, we got a perfect match. So our control study has exactly the same number of citations in every single year, as our original study does during the citation history. Okay, but that's a pretty high standard to meet. So we're going to relax that a little bit. We're going to relax it as a function of the total sites of the original study at the time, the replication study was published. So again I'll go back to that three year gap example. These published in T equals zero the original, the replication published T equals zero the original T equals minus three. So I'm going to count up all those intervening years count up citations during those intervening years before right before the replication was published. And I'm going to have a sliding scale, based on that number. Alright, and we're going to compare the total absolute difference in the citation history with the sliding scale I'm not going to go through the details of that. I want to point out this one PCT variable that's percent. So the sliding scale is going to be a function of the percent of the total original site so if the total original sites is a pretty small number, then we're going to have a pretty tight band, where we're going to allow the control studies to match with respect to differences in citations. If the original study had a huge number of citations, then the band gets bigger. Okay, and so rather than explain that formula, I'll give some examples. Okay, so when percent is zero, no matter how many sites the original study had at the time the replication was published, we're only going to include controls that bring the entire citation history only differed by one citation from the original study. That's a really tight criterion. When the percent is 10% that gets relaxed when the original study has a lot more citations, then we're going to allow controls and have differ by a larger amount in their number of citations during the citation history period. And likewise, when percent is 20%, the gap gets a little bit larger, flat gets a little bit larger. Okay. All right, so here we go. So what that means is, we have a large number of specialized samples, and we have different matching criteria. And you can see, say when there's a three-year gap, and I use the 0% matching criteria, I've got 34 treated, 34 original studies, 4,553 controls. Okay, so I do that when I got a three-year gap, a four-year gap, a five-year gap, up to eight-year gap, where you lump all the studies that have more than eight, we combine three to eight, and then I throw everything in there. Okay, so you can see, and I'm going to focus in the subsequent empirical analysis on this yellow highlighted row. I got 0% as my matching criteria, and I got 74 treated, 7,000 plus controls. When I have 10%, I got 103 treated, and 7,500 plus controls. And when I have 20%, I got 140 treated, and over 11,000 controls. Okay, so that, when you do that, that's our best way to try to control for any unobserved differences between our controls and the originals. Okay, how well do they match? Well, pretty good. You can see by the mean row that for the 0% and 10%, on average, the absolute total difference over the citation history is less than one citation. The median value, the median absolute total difference is one citation over citation history for all three matching criteria. So for the most part, we do a pretty good job of getting matches. Okay. All right, so now we get to our statistical analysis, right? The dependent variable is going to be the difference in citations between the original and the control. And we're going to regress that on a constant term, and on a dummy variable refute, which is going to take the value one, when we have a negative replication, and zero if it's a positive or mixed replication. Okay, that's the regression we're going to estimate. And we're going to estimate that for every single year in a window around the period when the replication study was published. So we'll take the immediately three years preceding the publication of the replication and the immediate three years after replication study was published. And we're going to do that for each of the individual samples. So samples that only have a three year difference between when the original was published and the replication was published four years, five years and so on. Okay. Although I'm going to focus on this combined three to eight year. Here's our results on this is this is about it at the end here. So first of all, let's look at the matching period. We would not expect that the outcome of a replication would affect the citations of original studies in the in the years preceding the publication of the replication. And what this number here in the red box is telling us that is difference between our control studies and our original studies for studies that had negative replications compared to ones that had positive replications. Okay. And so we would expect those numbers to all be very close to zero because the outcome of the replication was not known yet. So it should be an irrelevant extraneous and all those estimates are insignificant. The units here are number of citations. So that number minus 0.168 saying that studies, the original study had minus 0.168 citations less than the matched control study, if the replication eventually ended up being negative. Okay. And again, sometimes negative sometimes positive. So that's what we would expect if we did a decent job matching. Now let's look at the post replication period. Now the results of the replication study are known. And so let's look at those numbers. Right. So again, we've got three years following location replication. 0% M% 20% different matching criteria, different numbers of treated and controls and each of these samples. Positive numbers. Right. Most of those estimates are positive. What does that mean. It means that when a replication was negative. The original study actually received more citations than the control studies did, compared to studies that got positive or mixed replications. It's kind of the opposite of what you'd expect. However, before you make too much of that results. Notice, all these numbers are really small. Right. They're all less than to the units here are citations to get a sense for how much of a effect this is, you can go back and compare that to the number of citations he studies had when the application was study was published. If I add one or two extra citations to the number of citations the original studies had that is a negligible effect. So the estimates are positive, they're small and across the board, they are statistically insignificant, despite having a fairly large number of controls although that's a little bit misleading. The bottom line here is, we find virtually no impact on the citations of original studies that have negative replications, compared to those that have positive remix. In other words, no evidence that negative replications affect citations. Thank you very much. So I'll stop my share and pass it on back to Fallon. I do see we have questions but I believe we've decided to hold all questions to the end of the three talks. So we will move on to our next speaker, who is Tom coupé. Tom, if you can start sharing screen. Tom's talk is titled paving the road to replications results of an online experiment. I shared it. Do you see the screen. Yes, absolutely. All right, good. All right, indeed. So this is joint work with Bob, and also Christian Zimmerman who is at the Federal Reserve Bank of St. Louis. So as I assume already many people have said before me. There's a lot of evidence that lots of studies cannot be replicated we have evidence for that from psychology medicine economics management nutrition science. So there's lots of evidence that many studies cannot be replicated. Many observers have argued that while this suggests there's a big role for replications that in fact, many people should be doing replications and we should see many published replications. However, in practice we see that very few few studies get replicated and very few researchers are doing replications. And so one one can wonder why is this the case. And one possible argument there is that in fact there are few replications, because replications receive less attention than original papers. So, from our study kind of here we have a kind of we collected information on about 320 pairs of replications and their original papers. So if we compare over a six month period, how many times the web pages of a replication paper gets visited and compare that to how many times over a six months period original papers get visited. We see that original papers get about four times more visitors than replication papers. Okay, so this is kind of visits in a big archive of economics papers. So clearly there is much more interest much more attention for original papers than for replications. So, if you're a journal editor and you have to choose between an original paper or replication paper, you probably say okay I'm going to publish your original paper because that's going to be leading to much more interests in our journal is going to lead to many more citations. Similarly, if you're a researcher you might say okay if I have to think about doing a project that is a replication or I invest in original research maybe it's a better idea to invest in original research, and that is going to get more attention. And that is going to lead to more citations for me. So, what can we do about it so if if indeed the reason is that we see so few replication because people are not kind of paying attention to replication as much as they pay attention to originals, what how can we change this. A number of proposals have been made on how to improve that and how to get more citations for replications. For example in 2015 Kofman and Niederle suggests that, well, journals they should require that whenever somebody sites an original paper. They should also cite the replication of that original paper. If the journals require this, we will see a strong increase in citations for replications. This might give more incentive for authors to do replications. Kind of what we have been trying to do in our paper here is to do something different. We have tried to see whether providing information about the existence of a replication can lead to more attention for that replication. So the idea here is being that, okay, many people know about the original paper, they don't know about the replication, and that's why they don't cite the replication. So that if that is true, then we would see that the original papers are much more cited than the replication papers, just because nobody knows about the replication papers. So what we did is we did an intervention experiment. So we teamed up with RAPEC, which is a big archive of economics, working papers and economics journal articles, which is run by our quarter Christian Zimmerman. I think it's kind of probably the biggest archive has about three and a half million research items from over 5000 working paper series and over 3600 journals. So when you go to that website when you try to find economics research from RAPEC, you will get things like this. So this is the typical webpage for a random paper. You search for replications, or no, you search for articles that are about replication, you search in RAPEC, you find an article like this one on the need for replication, for a replication journal. You get information about the abstract, the authors, whether it is downloadable or not, and so on. You might see here some blue and red tabs. RAPEC is using this kind of course to draw attention to specific things you can do on that website. So now what we did is for papers that were replicated, we added two things. We first indicated under the title, the paper had been replicated. So we added a star behind the title and indicated that the star means in fact this paper is not just a random paper, it's a paper that has been replicated. Moreover, we added a yellow tab. Again, we made it yellow, different from the existing colors, again to draw attention to the fact that the paper has been replicated. Now, if people saw that paper had been replicated and they were interested in the replication, they would click on that yellow tab. This would bring them a link to the replication paper. So basically, people who didn't know there was a replication, by going to this website, they would find out there is a replication. And if they're interested in it, they could go to paper through this link. Now, what we did is we made this an experiment. As I said, we had about 320 different pairs of replications and originals. And we randomized which of those 326 pairs would receive this kind of information, and which would remain as before, like, without explicitly without a yellow tab or a star. Okay. So how did we do the randomization, we didn't do it entirely random we stratified it kind of we wanted to make sure that we had a balance we didn't. So what we did is we looked at six months of visits to these pages, and then try to find pairs of replications and originals that had in the six months, similar visits to the similar numbers of visits to the replication, and to the original. So what we want to do is we want to try to get controls and treatments that on average are have the same number of visits to the originals and the same number of visits to the replications. We divided in one group into groups group one and group two. So, you can see group one had about 10 visits in a six months period before we started the experiment. So group two didn't immediately get the game didn't get treated right away group one is the treatment group group two is the group that originally serves as a control, so they don't get the yellow tab. And you see, before the treatment started, we have similar numbers in terms of visits to the replications for the two groups and visits visits to the originals for the two groups, which basically suggest that indeed before the treatment we make sure there is about that we have a balance that we can compare at least on average the treatment group and the control group. So now let's see the results of our intervention. The first thing we can do is we can measure how many people clicked on the link to the replication paper. So we have somebody visiting an original. We have a count of all these people who visited this original. How many people then went to this replication and actually clicked on the clicked on the link to the replication. So we actually can see how many people use our link. What do we see here in this graph. Here we have share of y axis we have the share of visitors to the original study that use our link. Before we started our experiment and somewhere in August 2020. We don't know exactly when the link was created for somewhere in that month. So we only use data from earth after that. We see there were some people using links from the original to the replication before it could happen because Rebek also has citation lists. And that allows people to go from an original to replication where we can see this is very, very rare before we actually the yellow tab there. Then, in August we the yellow up there, create this extra visibility for a link to the replication paper. In the group that got the link treatment group see about 1% of the people who visit the original, then go to the replication. In the control group, most nobody we can see a clear effect of our intervention by highlighting the fact that there is a replication, we can see that some people use that to go to the replication. And then in March, we have extended our treatment to also the control group. So from March, all papers all replicating replication papers and all original papers that have replication in Rebek are, they are getting such a yellow button. And we see again, one to 2% of the people actually are using this time. Now, this is share of people that click on the original relative to how many people visit the original. Now, since many people more people visit the original than the replication, increasing the kind of a small share of visitors of the original that go to the replication can mean a bigger impact on the views for the replication. So here, that's exactly also what we see. If we look at the share of clicks relative to the number of visits to the replication webpage. It's about while it rise from one one to another month goes between four and 12% on average, it's about 6%. So from the moment we see, from the moment you have the treatment, we see it for the treatment group. If we then extend the treatment also the control group, you also see it in the control group. It's clearly our intervention has an effect, but that effect is fairly small. We can't increase a click through rate for if you look at original papers. This leads to an increase of 5% 6% of visits to the original to the replication. We can also look at the overall page visits for the replications and compare the control and the treatment group. What we see there is again, after we do the intervention, always have more visitors to the replication studies that were treated, compared to those that were not treated. But the difference is very very small. We also do some econometric analysis basically this difference is not significant. What does this mean? Well, our original idea was that some people might not cite replications simply because they're not aware of it. So we wanted to see how does increasing awareness about the existence of a replication affect the visitors of original studies and we see in fact the impact is very very small. The 2% of the visitors of original papers then go to the replication paper, leading to somewhat of an increase 3 to 7% of visits to replication papers. So the big question here is, why do we observe this? Why do we have such small effects? One could argue that while we didn't provide enough information, maybe we shouldn't have just the yellow tab, maybe we would have to have some pop up window. But kind of we thought that would be too annoying and RAPEC didn't really want to want to go to that level of annoyance of visitors of their website. So still we think this is a very visible way of indicating there's a replication. So we don't think that that's going to be a big that that can explain our small effect size. One could argue that in fact people already knew about it. But if people already knew about the replication, kind of why do we see such big difference in visitor statistics for replication and originals. So our kind of our feeling is that it suggests that kind of the research don't really care about replications, right. So even if they know that there is a replication, they still don't go to that application and find out more about the replication. So that's the question maybe it's consistent with what Bob was presenting in the first presentation, kind of after a replication the citation of the originals are not really effective. Well, one possible reason is that indeed people just ignore replications. That's all I have to say. So I will stop sharing now so we can do the next presentation. Thank you Tom. That was an excellent way to an open question to leave that we can come back to in Q&A hopefully. So we will move on to our third and final speaker in this session. And that is Rose O'Day, who will be presenting a talk titled feasibility of replication studies in behavioral ecology. Over to you Rose. Thanks Valin. Hello everyone. Thanks very much for sticking around to the end of this session, especially on a Saturday. To start, I need to acknowledge that what I'm presenting is very much a work in progress. We're actually less than halfway through the data collection. And I want to thank Megan Head for doing most of the work so far. If you would like more information about this project that we might not get to today, you can read the registered methods by following this link on the screen, or just send me an email. Okay, so those previous talks were asking do replications make a difference. But in behavioral ecology, we just can't answer that because we don't have enough close replications. And this is true of ecology and evolution more generally. The estimate from Clint Kelly is that fewer than 0.02% of papers in ecology and evolutionary biology describe themselves as replication studies. Now this metascience audience by now is very familiar with this general phenomenon, and whatever discipline you come from you've probably got your own literature of papers saying replications matter and we should have more of them and what does a replication even mean. So this is just some of those papers from ecology and evolution over the past 20 years. Basically there's been a lot of talk but not much action. So why aren't ecologists replicating more. It's not that they don't think replications are important. They do. We know they do. Hannah Fraser conducted this survey which found that ecologists think replications are important, and we should be doing more of them. And then if you say well why don't we are the first thing people point to is the incentives. So the difficulty is funding or publishing replications. People also talk about that just to cool constraints environmental variability and just the academic culture. But as we know we've just heard there are lots of ideas for how to overcome those barriers and encourage replications. So one day Megan and I were talking about our frustration that although we have all these ideas, not much seems to change. And we thought, well, maybe it will help if we had some database of studies that are worth replicating. And then behavioral ecologists who were replication sympathetic could search that database and find studies that they were capable of replicating. But how would you go about choosing studies to include in this database. So my thinking on this has been heavily influenced by this preprint led by Peter is a guy. I recommend checking out this preprint there's also a talk on YouTube, and it presents this model of how if you had a candidate list of studies that you might wish to replicate. How would you prioritize them which are the most important. And basically what I took away from this is that there are four factors to consider. The first is how valuable is this knowledge that we're seeking to generate does this thing matter. The second is how certain are we in that knowledge already. If we're already sure that something is true or false, then there's no point replicating it that would just be a waste of time. Relatedly, what is the ability of this particular study to affect a level of certainty in that in that knowledge. And I think this gets to a couple of things. The first is what is the quality of that study, if it's poorly designed, if it doesn't measure what it's trying to measure, then whether or not it replicates shouldn't actually change our level of certainty in that knowledge area. But I also think this gets to the culture of the research community, and this is what that previous talk was about so with the community pay attention to the outcomes of the replications, and would they update their beliefs based on those outcomes. Then fourth, is this replication feasible. If it's going to be super expensive to do a replication, then it better be worth it. Now out of this list of four things, I think that the feasibility question is the easiest one to think about and to imagine measuring. So this is where Megan and I have started sort of attacking the lowest hanging fruit first. So that's the subject of today's presentation, how feasible are close replications in behavioral ecology. Now I mean this as the discipline but also more specifically behavioral ecology, the journal. We're planning to submit this to the journal behavioral ecology so we're using its own papers as a case study to illustrate how we could possibly do these feasibility assessments. Now this is a well respected society journal that's been around since 1990. And we aim to sample across that 30 years span of the journal. And we have made the decision to focus on the more cited papers, assuming that those are the papers the community has paid more attention to, and therefore will be more likely to want to replicate or pay attention to the outcomes of those replications. So on this slide I'm showing you histograms of the number of papers in each of those publication years with the citations per year on the x axis, and we've sampled from the top 25% cited papers in each year which is shown by the red bars. Now from those papers we then screened them to only include the primary empirical studies so we were excluding reviews theoretical papers those sorts of things. And then, of the eligible papers we just took a random sample in each year to end up with our database. Now I should say from the year 2021. There were no citations yet so we just took a random sample of those most recent papers. Okay. So now we have this database of 100 papers, and we're going through them to pull out the basic methodological details and assessing their replication feasibility. So far we've done 35 of those papers, and this is just showing you the spread of those publication years for the sample that I'm showing you today. Now each of these papers presented the results from up to five studies that we can estimate replication feasibility for. So overall what I'm showing you represents 62 studies. Now to do these feasibility assessments, Megan pulled out basic information from each of these studies including short descriptions of what was done and why, what the main variables that were measured, and what were the limiting resources. So if you were going to do a replication, what about it seems like the hardest part, as well as what was the focal species, and where were those species sourced from. So this slide is showing you photos of the species that are represented by those studies so far. Now most of these are fairly common species that multiple behavioral ecologists could have access to. You might notice there are a lot of birds behavioral ecologists love birds. This is a well known taxonomic bias in the species that we tend to study. And there are also geographic biases so here that country shaded in dark other locations where those species have been sourced from in those studies. As well as recording how common the study system was Megan also recorded where the data were collected so was it in the lab or wild or semi wild setting, whether the study was experimental or observational. Whether you would be able to collect the data at any time or if it needed to be in a particular season, and how many people you would need to collect the data either full time or part time. Most of the studies had smell fairly small sample sizes, although there were some big exceptions from long term studies. And the amount of time it would take to collect all the data was also usually fairly short, so fewer than six weeks. If you already had access to the study system, then you could also start collecting data fairly quickly within six weeks. But if you need to establish the system such as building the animal facilities, or finding your field population or getting ethics permits that will take a few more months at least. Okay, so those are the basic details about each study that Megan recorded. And next we set out to estimate the feasibility of replications on these four different dimensions so system intensity duration and complexity. We use sort of a traffic light system, there's four so a four point scale for each of those facets of feasibility. And we did these estimates separately so Megan had read the method section of the papers as she was creating this database, whereas I was just going off of that database of information that Megan had coded. So as I show you the results from the next slide, this is the color scheme I'll use. So remember the green as color means that a replication would definitely be feasible, whereas the red as color means that would be really really hard. First I'm going to show you our independent estimate side by side for each of these four dimensions. And then at the end I'll put them all together with our consensus answers so you can have an overall picture of replication feasibility. Okay, first of all, how feasible would it be to replicate these study systems so measure a similar population in similar conditions over 60% of the time we gave the same answer and most systems seem fairly feasible to replicate. Second, how feasible is the intensity of the workload. So, could you do this, you know, on the side or would this require your full attention. And we're considering these things independently of each other so this is not counting the intensity of say setting up the system because I've already thought about that. This is just the intensity of once you've already got the system. And again this had over 60% level of agreement. Third, how feasible would be the duration of this replication study. So would this just be a short project could a student do it or would it take years of your life. And this was the easiest simplest thing to assess we had the highest level of agreement for this dimension over 80%. Fourth, how feasible would the complexity of the study be. So most studies had fairly simple designs with simple measurements we had over 80% agreement again. Now if you read the register methods for this project you'll find that we'd also plan to assess a fifth aspect of replication feasibility resources, how much extra funding would you require. But we found while doing these assessments because we were trying to consider each of these aspects of feasibility independently, the resources category just didn't make sense. Because once you've already thought about the system duration intensity complexity that takes away the things that consume resources, the setup costs people costs equipment costs. And so we found that there just wasn't anything left over in this resources column and it, it didn't make sense so we've decided to not not score this for the remainder of the papers. Okay, so as well as finding out that this resources column was weird. As we sat down together and compared our answers we, we decided on the consensus answers for each of those studies for the aspects that we disagreed on. And this is what it looks like when you put all those consensus estimates together. So each of these rows represents one of those 62 studies. You can see by the amount of green and this lighter shade of yellow that most studies seem like they would be feasible for at least some behavioral ecologists to replicate. And the hardest parts tend to be the intensity so studies that have intense workloads or the duration studies that require years and years of your of your life. This just answers one aspect of this replication worth. If we're going to prioritize studies for replication, then we do need answers to these remaining harder questions. If replication seems so feasible in behavioral ecology. Why aren't we doing them. Is it that it doesn't actually matter whether this knowledge is true or not. I find this question of value just really hard to think about for curiosity driven field like behavioral ecology. Well maybe it does matter if what we're studying is true, but it's too hard to work out what our level of certainty is. Or maybe we do know where the knowledge gaps are, but it's not obvious how any particular study affects our level of confidence in a particular knowledge area. And I think this is at least part of it. You see when we were planning this project which originally thought that we would also assess the feasibility of conceptual replications. So those are replications that assess the same concept or idea, but in a different way. We quickly gave up on that idea though because it was just too hard to pin down exactly what any specific study was testing. We couldn't define its conceptual boundaries. This is not a new observation and it's certainly not unique to behavioral ecology. I'm thinking of these general meta science topics of the crisis of theory and the crisis of generality. This also gets to this general problem that often what seems most important is the hardest to measure that not everything that can be counted counts and not everything that counts can be counted. And I don't know what to do about this. So in this presentation, I showed you a way that we can count the feasibility of replications in behavioral ecology, but I don't know how to count what those studies are worth. So after thinking well behavioral ecologists at one point decided that these studies were worth enough to publish in the first place. So if it's quick and cheap and easy to replicate them. Why aren't we. Thanks for listening. I'd be keen to hear your thoughts. Thank you very much. And so now I think we have about 10 minutes for Q&A. I believe that Shanichi had a question because he raised his hand quite early. Shanichi, I'm just going to allow you to talk to answer your question, so you can ask your question. Hello. Yes. So I actually raised hand accidentally I was listening on my phone but that. Okay, I last question to roles. So the, it's really interesting that like you know that we don't know the value of the, you know, how important it is. I was thinking like, maybe many of those. I was wondering whether you are think sort about assessing replicability because maybe easy. There's a you know psychology and all other different disciplines are doing like what is a, you know, predictability how we can predict the replicability and we are quite good at doing this. So maybe if it's easy to do, we first also wants to like, you know, the quantified replicability of these potential studies. Yeah, I guess that was part of the idea of the project is that at the end would end up with with the database of studies that that are feasible for people to replicate so that could be a catalyst for some sort of replication project although I think we'd want to think about whether that was that was the best use of time and resources which those other talks were sort of getting to if we do these replications will it change anything. So I think, I think we've got to think a bit more carefully before diving head first into a big replication project. Yeah. Okay, thank you. I just gonna, it's okay I'm going to ask the question to everybody. Why don't people care about the replication. Like, you know, the theme of all the, you guys talked about what why do you think that is because I do care. That's an excellent question to me and actually when I was going to pose to all three of you as my question, just your own personal reflection so who would like to take it away. I'll go ahead and jump in. Let's get it started. So, short answer. Don't know. I suspect most people really don't care that much, but that's a personal bias. I gave a lightning talk on this earlier on saying we should ask people. You know you get all these people who cite the original study and don't cite the replication we should ask him why didn't you cite the replication study. You see we're interested enough to cite the original. Maybe get some information that way but yeah so maybe it's possible she needs you that you and me and the rest of us are a little weird in that we think replications are really important and most people don't know that kind of when you cite the paper, you cite it because it was a kind of it was something new, something that stood out, maybe more than because you believe it's true. So in that sense maybe we don't, we don't release kind of when we say we don't really care whether we believe it or not. If you want to close would you like to add anything I just see is being it's just added something. I think I covered at the end of my talk and I don't want to say anything too definitive or because this is being a recorded but I can't. I worry about what it means if we don't care about replications I worry about what it means for the quality of the science we're producing and our values and and aims. I just said that she was reminded of Steven Goodman's talk at the last Meta Science Conference where he talked about how we had lost the language to talk about uncertainty, because of over reliance on dichotomous significance testing. And she's wondering if there's a parallel here, we've spent so long out of practice in replication that we don't really know what to do with it now. And that sort of follows on from the talk that she, the question that she put in the chat asking whether the softer question is do research is know what to do with replication evidence and I'm wondering if each of you would like to reflect on that. Maybe that's part of it kind of when I read these replications I also often don't really know what to do with it. I think that kind of often replications they don't replicate everything they replicate only a part, then the conclusion is not always too clear. So with many replications indeed it's hard to kind of how to know how you should update your beliefs after reading the replication. Yeah, I agree. One thing I would say to that I think it's really hard to know how to process a negative replication for the reasons that you guys mentioned earlier, but I think there's a lot of value in a positive replication. So if you go in with the prior that hey I don't really believe, you know, I don't know what to believe, and some guy did a study, and somebody independently comes around and confirms that study. Maybe two other people confirm that study. Well then my my priors are updated a lot. I have a lot more confidence in that result. So I think while it's hard to process conflicting replications. There's a lot of value in confirming replications and I'm, which is kind of a shame because I think journals, at least the anecdotal evidence is that journals prefer to have negative replications, because that's what people pay attention to and read. Well, it has has been a very interesting session and I'm wondering if there are any more questions if not, I'm conscious that most of our audience are on a Saturday in Australia New Zealand. So we might leave it there. Unless you have any final concluding remarks that either of you would like to make. I say one more thing. Rose, there's a journal in finance called critical finance review, and the editor puts out replication studies that he wants to see replicated. Or original studies he wants to see replicated. Maybe if you could get an editor like a behavioral ecology to put out a list there. It would communicate to people that, hey, this editor is interested in this and like to publish it. And that kind of solves the issue of how do we choose one because the editor says do this and I'm interested in publishing it. Oh, it's a it's a highly ranked journal. It's a really interesting journal I could talk about it for a while but no it's a highly ranked journal relatively new kind of catapulted to the rankings because the guy who is the editor is a very idiot eccentric and really smart guy, and has very influential in the field. Interesting thanks. I spoke to assume that is one more question. It's for rose from Eden. In addition to the lack of interest in replications when they are feasible. I'm interested to know about how to think about those studies where replication isn't feasible. For instance, what alternatives are or should be used for assessing credibility when replications aren't feasible. Thanks, Eden. I think for those those long term studies where they don't seem feasible because they would take years and years to set up. You can kind of consider them as internal replication so that if you can, you can see the replicability of effects. You know, five, 10, 15 years later, and I think those really valuable studies in a college in evolution to study all sorts of, you know, context dependent effects and, and, and there's. Yeah, interestingly, a lot of those populations long term studies first get established because they got lucky initially that they had some cool effects show up so that you know got the more grant funding but then those effects disappear down the line. And for those ones that are just a lot of work. It's not that they're not feasible to replicate, but they would require additional funding so I think that more gets into the weather, whether we care enough about replicating them enough that they could give you money for that to be your full time thing and not just a side project. But for the question of what, what else can we do I mean we have we have computational reproducibility and conceptual replications, if we can define our concepts, precisely enough or have strong enough theory that it makes sense to do those and interpret them. I think we are genuinely at time just a reminder to everybody that with this panel we are concluding what has been a fantastic and robust Friday night Saturday morning meta science 2021 session. So there is at the end of each session, a schedule 30 minute networking component on Rimo. I have popped those details and the link to Rimo in chat. You can go there to continue all of these conversations. If you are missing the link. It's also on the meta science 2021 website, and in all the emails that you've received so there's nothing left, except to say, thank you again to to all of our speakers. Thank you for joining us. Thanks.