 live now. Wait for just a second and always working on a tape delay. I have I have a lot more respect now for like radio DJs. Oh yeah. I'm sure it's this weird kind of out of body disconnected like. Alright, yes, everything should be good. Now I believe everybody's pulled across into the session. So without further ado, the last talk of the of the meeting, a PS stick around. We will do a little tiny I promise not too long. I'm not we're not going to bore you. We'll do it. We'll do a very quick closing remarks after this. There won't be a break for that. We'll just swap as quick as the machines will will let us pull over into a into one little little final zoom, one little final session for that. But the last proper talk of the meeting from a dear old friend of mine, Dan Hicks, who is going to talk to us yet again, looping back today's been a sort of articulated reticulated looping kind of kind of day to to the question of open science to talk about open to to argue for it. So the title is the thesis. That's so nice. Open science can't solve the replication crisis. So please take it away. Yeah, well, so thank you everyone for sticking around for the last talk. Thanks to Charles and Luca. Is that how it's pronounced? I'm going to guess it's Luca. Yes, for organizing this conference. I'm on the west coast of the US. So because of the the time difference and other responsibilities, I've only been able I've really only been able to watch the talks on YouTube. But super interesting really looking forward to seeing the other open science stuff this morning that I had to miss because I was teaching literally an hour ago. But yeah, okay, getting into my talk. So just as an overview, I'm going to start by giving some background on the replication crisis for anyone in the audience who might not already be familiar with that. Then I'm going to introduce a distinction between reproducibility and replicability and talk about some of the problems that have emerged in the discourse surrounding the replication crisis because of a failure to distinguish those two things. Then I'm going to talk about open science. I'm going to give you some background on open science, talk about how open science is supposedly going to address the replication crisis, and then give my argument that in fact, open science can't solve the replication crisis. So first, the replication crisis is an ongoing epistemic crisis unfolding primarily in social and behavioral sciences. As sort of a tagline, the crisis is that experiments, especially some high profile experiments such as priming studies fail to replicate much more frequently than one might have expected. The replication crisis really came to the attention of sort of the general educated science minded public around 2015, when outlets such as 538 published pieces with titles like science isn't broken. And the Atlantic published a piece with a title, How Reliable Are Psychology Studies. However, the the clouds of this epistemic crisis have been gathering for some time. Already in the 1960s, the psychologist Paul Mille has raised a lot of the concerns that we see today in the replication crisis. In the 1980s, Gerd Geiger-Rinzer, the psychologist was raising some concerns about the misinterpretation and misuse of p values and statistical hypothesis tests that also appear today. In 2005, Gerd Geiger-Rinzer published a paper, I was going to double check the title. It's something like, why most research is false, which attracted an enormous amount of attention. But really, the current crisis precipitated around 2011, especially in psychology with a couple of major events that happened in that field. So first, Daryl Bem, who at the time was a very highly respected psychologist, published a paper purporting to find evidence of precognition using methods that at the time were regarded as perfectly acceptable. That same year, another prominent at the time prominent psychologist Dieter Ekstapel was fired from both of his university positions for fabricating data. These two events really caused psychologists to take a close look at the methods they were using, the established standards in their fields. And within the year, the first large scale replication studies in psychology had launched. By 2015, one of these large scale studies called the Open Science Collaboration published the figure you can see up at the top of this slide, which there are a few different ways to interpret this slide. The most common interpretation was something like only about a third of psychology studies can successfully be replicated. I think that reading of the figure is a little bit strong, but certainly it attracted a lot of concern and attention. This slide is just a quick and dirty bibliometric analysis to give a sense of the disciplinary scope of the crisis. So you can see up at the top, the crisis really is has as its center or epicenter psychology, which has about twice as much discussion of a replication crisis as the next field that seems to be the most concerned about it, which is business and economics. But you can see there's a lot of spillover into other social and behavioral sciences, primarily quantitative fields. So you can see government law, which probably refers to political science, sociology, a bit further down geography, communication, education research. There is some partial overlap with some fields of biology. So you can see up towards the top biochemistry molecular biology with some discussion. And then down at the bottom, cell biology just barely making the cutoff for this slide of more than 25 hits in the web of science search. In molecular biology and cell biology, though, a lot of the concerns about irreplicability relate to things like being able to properly identify cell lines or reagents, which don't quite aren't really the issues in social and behavioral sciences. The crisis has prompted a enormous and very vigorous literature, which has proposed numerous diagnoses, or I'm later going to call them bad habits that are thought to contribute to the crisis. I'm not going to go through all of these. I will highlight P hacking down in the analysis and interpretation session, because that may be a technical term that some folks haven't encountered before. P hacking is a habit or an analysis practice in which researchers try variant statistical analyses repeatedly with the same set of data, until they're able to achieve an analysis that is conventionally statistically significant, so a p value below the conventional point oh five threshold, and they only report that final analysis that was statistically significant. And the problem here from the perspective of statistical hypothesis testing is that this approach completely validates invalidates what it is a p value is supposed to report. I'm not going to really make claims here about which of these factors might be actually or not contributing to the crisis, which factors might be more important than others. The replication crisis is a social phenomenon and like most social phenomena, probably it's over determined with a lot of these different factors, not only contributing to the problem, but also interacting with each other to make really complex causal relationships here. So with that background on the replication crisis. Now I want to draw out a distinction between reproducibility and replicability. And in this talk, the definition I'm going to use for those terms come from a consensus report published by the US National Academies of Science, Engineering and Medicine, or the NAS in 2019. Early on in this report, the authors note that this distinction is really fundamental for understanding what the nature of the problem might be and what might be done about it. They define reproducibility in the blocks here are quotations as obtaining consistent results using the same input data, computational steps, methods and code and conditions of analysis. So reproducibility is really a very computational concept. The idea here is that if I'm attempting to reproduce some of your research, I have the exact same data that you use the exact same code that you used to analyze it. And in addition, I'm expecting to get if the reproducibility attempt is successful, the exact same numerical outputs from running that script on your same data. So it's a computational concept. Whereas replicability on the other hand, as they put it, is obtaining consistent results across studies aimed at answering the same scientific question, each of which has obtained its own data. So as reproducibility is a purely computational concept, replicability is really about running another study and looking to see if we get similar results. A key difference here that the consensus report authors emphasize is that reproducibility involves the original code and data, whereas replicability involves new data collection. And this difference also corresponds to the criteria for success of these two notions. So again, reproducibility is looking for, as the authors of the consensus report put it, full bitwise reproduction of the original values with the exact same numerical values. Whereas replicability is about qualitatively similar findings from qualitatively similar studies. And O'Yana Feast in a few recent papers has looked very closely at replicability disputes and examined the ways in which this qualitatively similar notions open the space for possible reasonable disagreement about assessments of similarity and assessments of whether one study is even a replication attempt of another. I want to emphasize that reproducibility and replicability are logically independent desiderata for computational or digital science. On the one hand, we can have a study where errors in the data analysis inappropriately exclude a range of data resulting in biased estimates. So this would not be considered replicable. Another study that gathered similar but different data and used a similar but corrected analysis strategy would reach very different findings. This was a case in macroeconomics in the study that was very influential in sort of arguing for austerity politics after the Great Recession. This study, however, can still be perfectly reproducible. We can have the original data and the flawed data analysis and get exactly the same numbers from it, but it wouldn't be considered replicable. Similarly, or conversely, we might have a study that is replicable but not reproducible. For example, suppose the original data have been lost, there's no way to reproduce the exact same numbers that were produced from the analysis because we don't have the original data to plug into that analysis. But if similar studies that collect new data from the same population tend to produce similar findings, then that original study would still be considered replicable. I emphasize this distinction because there's been a lot of terminological chaos in discussion surrounding the replication crisis about these two notions. So for example, I have a quotation here from letter in Zoran 2018, which is a white paper published by two economists that's been very influential in arguing for really stringent open data requirements in science used to make regulations in the United States. And they say access to the data necessary to replicate scientific studies is essential because the results of so many peer reviewed scientific publications have been proven to be impossible to reproduce. And they cite a bunch of studies to try to support this statement. However, the studies they cite look at either replicability or reproducibility, but they don't cite any studies that look at both of these things together and establish the kind of relationship that they're asserting between replication and reproduction. And in addition, that opening few that sort of opening clause access to the data necessary to replicate scientific studies, that's incoherent because by definition data from the original study is never necessary for a replication attempt. A replication attempt is always by definition about going out and gathering new data. And finally, in this section, it's worth emphasizing that the replication crisis is really a replication crisis. It was really precipitated by concerns about replicability, not reproducibility. So in the precognition study by Daryl Bum in Dietrich Staples fraudulent work, the concern wasn't Oh, can we get their data and produce the same numbers that they reported in their paper? The concern was, if someone else ran the same study, would they get similar results? This is a replication concern. In addition, the the the sort of large scale studies that have gotten the most attention and seem to indicate that there really is a replication problem. These have been replication studies rather than reproducibility studies. This means that evidence of a replication problem isn't necessarily evidence of a reproducibility problem and vice versa. And it also means that things that promote reproducibility don't necessarily promote replicability. And I think open science clearly reproduce clearly promotes reproducibility. But it's not obvious that it promotes replicability just because it promotes reproducibility. Okay, so now let's talk about open science. Open science and specifically a movement for open science has been a very popular response to the replication crisis. Proponents of open science often adopt an attitude that kind of facetiously I call open science to the rescue. One very prominent proponent of open science has been the psychologist Brian Nozick. And I have some quotations here on the bottom of the slide with various papers where he's been a co author. In one relatively early piece, he called open science the ultimate solution to the replication crisis. Elsewhere, he's argued that reproducibility increases confidence in results, and that transparency is superior to trust. So here making assertions that I think about the connection between the reproducibility promoted by open science and replicability. The open science movement has been rather successful, especially in psychology in getting some of its recommendations adopted by journals, for example. So up at the top, this is a screen cap of an article that was recently published in I forget which but one of the major psychology journals. I've highlighted some badges sort of towards the lower right of the screen cap. These badges indicate that this study has implemented various open science policies. So authors are being recognized and thereby explicitly rewarded for adopting open science practices. There are a lot of different things that people mean when they talk about open science. I really recommend the survey study by 11 at all from 2016, which looked at I was researchers in the UK about the various, not just policies that they mean by open science, but also the values they associate with open science. I just want to note very quickly how I'm going to define open science for the purposes of my argument. Specifically, when I talk about open science, I'm going to refer to open data and open code. That is practices where researchers publish the data used in their study and the code they use to analyze that data, where researchers publish those things in publicly available repositories. This might be GitHub, this might be OSF, the open science framework, which was developed by Nozick and other advocates of open science. I want to be explicit that today I'm not talking about pre-registration or the specific form of pre-registration called registered reports or open access publishing. These things are often included when people talk about open science or open science practices. During the Q&A, I can explain why I want to exclude them from the argument I'm making today. Now we have the question of whether open science will help with replication problems. My claim is that open data and code will help with replication problems only in so far as three criteria are satisfied. First of all, those bad habits, those purported causes or factors contributing to the replication crisis back from part one. Those habits need to leave traces in the data or code so that by inspecting open data or code, we discover that these things might have played a role. Second, these bad habits shouldn't already leave traces in papers or other traditional academic publications because otherwise we can just look at the paper. We don't need to go to the data or code. And third, open data or code need to help prevent or mitigate the bad habits. It's somewhat helpful if they help us detect that there's a problem, but what we really want are new habits or practices that will help prevent the problems in the first place. So what I've done on this slide is I simply went through the list of factors, quote unquote, bad habits from section one. And I've evaluated them according to these three criteria. What you should notice immediately on the bottom half of the table is that for a lot of these things, I don't think open science will help at all. So take, for example, imprecise theory and under-specified phenomena. One concern in the replication crisis is that often social and behavioral scientists are kind of vague about the relationship between the theory they're trying to develop, the phenomena they're trying to characterize, and their data. This vagueness isn't going to leave any sort of trace in the data or the code. It's really a matter of how things like the data and the experimental design are interpreted. And so I don't see how open science can help with these kinds of issues in any way whatsoever. Jump up to the top for one issue where I think open science will certainly help. And that is detecting and dealing with software bugs, where that could be either code that the researchers have written themselves or packages or other dependencies that they're using in their analysis. If I publish my data and my code and you access it, attempt to reproduce it and you get an error message or you discover that I'm using a package that's considered to have flaws and shouldn't be used anymore, then, yes, open data and code have definitely helped you detect that problem and can help mitigate that by, for example, suggesting changes to the code through a system such as GitHub issues. I think open science has a much more situational value when it comes to issues like p-hacking or data mismanagement and just for the sake of time, focusing on p-hacking. So if I am a researcher working on my analysis and I try a dozen different variant analyses and include them all in my code and then publish my code and you go and look at my code, you can see I tried a dozen different variant analyses. I only reported the last one. You can see that I've engaged in p-hacking. So that's a case where open science could be helpful in directly detecting p-hacking. However, suppose rather than dutifully recording every single variant analysis I try, I try an analysis, I write a few lines of code. It doesn't work out in the sense of not giving me statistical significance. So I erase those lines of code and write a new variant analysis and I keep repeating this process until I have something that's statistically significant. Well, then what you will see when you go and look at my code that's finally published is just the one analysis that I reported in the study. There won't be any sign of p-hacking. So in this case, open science doesn't help detect p-hacking. In terms of prevention or mitigation, so suppose I'm already planning when I'm writing my code to publish the code that's used in the final paper. So I know I'm going to be publishing open code and I know other researchers are going to potentially see this. I don't want them to think that I'm p-hacking and so I'm not going to p-hack. With that train of thought, then open code has helped prevent p-hacking. On the other hand, my thought process might be, OK, I'm going to publish open code. I don't want people to think I'm p-hacking. So I'll p-hack anyways and then I'll very carefully erase all of those other analyses that quote-unquote didn't work. Well, now open data hasn't done anything to prevent p-hacking. OK, so the ability of open science to prevent or mitigate and detect things like p-hacking, I think is much more situational. It depends a lot on how researchers actually respond to. Open science, I think you can tell a similar story about data mismanagement, but I'm already a little over my target time. So I will come back to that in Q&A if people are interested. I will note about fraud. I think while open science can probably help detect fraud in the present day, a really committed fraudster who wants to fabricate data, I think, can adopt simulation techniques used by statisticians to evaluate methods as is to create simulated data that would completely pass current data forensic techniques would look totally legitimate, but in fact is completely fabricated. So I don't think open science will really help with fraud either. Now, I do think open science can be good and valuable, but not because it will solve the replication crisis. In my own empirical work, I use open science practices. I teach data science methods courses in my graduate program. Where I encourage my students to learn and use open science techniques. I think open science is good and valuable, but because of the way it's valuable for teaching and learning by understanding how a researcher conducted an interesting study. I think open science can be valuable because of the way it can be used to extend prior research with reanalysis or data aggregation. And under the right circumstances, I think open science can support participatory research and research in the global south. I think open science can be good and valuable for all of these reasons, but I don't think it will solve the replication crisis. Thank you. Fantastic. OK. Again, I'm going to keep doing my usual because this has been working so well. I'll take advantage of the of the tape to get my own first question in. This is like what you would never do if you were the chair in real life. But for some reason, it makes sense here. So this is related to the I assumed that the trace is not being being findable in the in the papers point. I want to I want to pick up the open access publication question. You might have expected that I that I might have picked that up because it's one of my things. So so, yeah, what do you do? How does that change the landscape? Do you think I mean, I mean, obviously, there's there's questions about there's separate ethical questions about financing and democratization of access and all that kind of good stuff. But as far as it being an arm of the open science movement, how do you how do you read that? OK, I just inhaled some tea as you started asking your question. So I want to make sure I understood the question correctly. So. Were you asking like why did I draw this line here? Why did I exclude open science here or open access publishing? Yes, in short, yes. Yeah, I'm good. There are there are a couple of reasons here. One is so on the one hand, I think open access publishing is included because it sort of exemplifies a lot of the values that proponents of open science point to in other contexts, sort of notions of transparency, giving sort of everyone access to knowledge and sort of this mertonian communism sort of sense. OK, so I think open access publishing gets bundled in because it exemplifies a lot of the same values. However, I haven't seen anyone sort of argue, for example, that open access publishing will somehow do something about the replication crisis. Specifically, this is a longer answer. That I will try to avoid giving because it can turn into a rant. I got specifically interested in this over some proposed policies in the US federal government. I kind of gestured before at a strong open science requirement that was in some sense implemented for about five minutes at the US EPA and the Department of Interior that really would have caused a lot of problems developing protective regulation. That was sort of how I got interested, a big part of why I got interested in developing the argument of this talk was responding to that initiative. That initiative was focused entirely on open data and open code with nothing in there about open access publishing. So there was sort of that arm of why I was less interested in open access publishing was because it wasn't sort of the problem I was trying to address in here. That makes that makes good sense. A question coming in from from Stefan Hesprügen, who writes, OK, yeah, good. So eating our own dog food, what's the value of reproducibility, not replicability in in digital science studies? And what's the value of open science and the kind of stuff that we do? Yeah, good. So I actually I had a slide that I said that that I decided to cut for the sake of time a pair of slides, actually, one talking about the value of replicability and the other talking about the value of reproducibility. What I say on the reproducibility slide is that I think reproducibility is good and valuable in much the same way that internal logical consistency is valuable in scientific theories. OK, if a piece of computational science isn't reproducible, specifically in the sense where, you know, I get your data and code and try to re-run it. And I get a bunch of errors or I get different values from what you reported in the study. Like we have reason to think something has gone pretty seriously wrong. It's hard to tell what just from that. But like that is at least a yellow flag that your numbers don't seem to match up with the numbers in the paper. I think that's a reason to be concerned. So much like logical consistency in a theory, like if a theory is logically inconsistent, we've got some reason to be concerned about the quality of the theory. That said, once that criterion is satisfied, reproducibility doesn't do much to establish the credibility of a piece of research. Right. So you might actually go back to this case right here. So this macroeconomics example, completely reproducible, actually the way so Herndon, Ash and Paulin were the folks who got the Excel spreadsheet from the original authors and figured out they were dropping a bunch of countries from their analysis. And when that was fixed, the analysis actually supported exactly the opposite conclusion from the original authors. OK. This was entirely reproducible, right? They could get exactly the same numbers because they had the Excel spreadsheet bundling together the data and the analysis code. But that didn't establish the credibility of the study that actually helped people figure out why the study was not credible at all. And so I think the same thing about other cases of reproducible research. It's like it's pretty important to have. But once you have it, it's not going to take you very far. So jumping back here now. So in terms of promoting open science in sort of digital HPSTS, what I would give are the arguments I give on this slide. I think it's incredibly valuable for teaching and learning to be able to look to see how another researcher conducted their analysis and kind of poke through it and work through it step by step. This is actually how I learned a lot of the data science I learned was reproducing published studies that had open data and code. And it's also something I use in my own data science courses. I have my students look at a study and try to reproduce it. And we learn a lot from doing that. Similarly, making the same points here about this can allow us to extend our research if we can build on each other's data sets. And this can also make our research more accessible to community partners if we're doing community based research or researchers in the global south. Fantastic. Next question. Let's see. Yes. Next question comes in from from Sarah Weed, who writes, says, thanks, Dan. I've wondered about the social forces in play in thinking of of open science as the solution to the replication crisis. So for example, is it less directly confrontational to say, please be transparent about what you're up to in your studies? That it is to say, please stop doing p hacking, designing studies badly, behavior, any kinds of bad behavior in science. Yeah, yeah. I was DM DMing with Sarah yesterday about this. So I know some of the broader context that I said in our DMs that I wasn't going to talk about the particular things you was asking about. And I'm going to stick by that. I'm not going to comment on a very recent open science bad thing that that's been unfolding over like the past 24 hours. So I won't come on comment on that in terms of. So I think part of what Sarah is asking there is sort of carrot versus stick. And so commenting on that. Yeah, I think in general. Carrots are preferable to sticks in this context. You know, what I'm giving on these arguments here are really carrots, right? They're here all of these additional good things that your research can do if you do it using open data and open code. I also think, although. I also think that, for example, software bugs using open science practices and being sort of really conscientious about trying to learn how to do it well can lead you to become, in effect, a much better software engineer, which is something that, you know, even in sort of the fields of science where open science and computational methods are widely used. So, for example, psychology and and cognitive science. The students generally aren't explicitly taught software engineering principles that will make their code like more reliable and more reproducible and potentially more replicable. In terms of sticks and the social dynamics of it. I think strong open access requirements are a mistake just because there are. I didn't I didn't have a slide here. I didn't have time to put it together because this week has been a mess. There there are downsides to open science. There are a lot of concerns about, for example, publishing open data with. Sort of anonymized patient medical information, because a lot of that can be D D D identified or re identified. There are concerns in the context of ecology about publishing locations of endangered species because that tells poachers where to go. There are a lot of concerns in archaeology and certain areas of genomics about publishing data on indigenous communities, because, for example, then, you know, outsiders can know where their sacred sites are and trample all over them, which is, you know, really not good. I think there are enough sort of provisos on open science that it's better to teach scientists how to be thoughtful and in other ways, virtuous about their use of open science rather than sort of more, let's say, deontological approach, that there are these rules that you must obey the you must follow these practices. And we're going to wrap your knuckles if you don't do that. Very nice. By the way, I just want to say because I'm violating the upvotes, but I want to note that that that question is so cool. I'm holding that for the last question. I actually want that to be the last question because that's awesome. Oh, no, no, no, no, good way. You're going to like it. You're going to like it. You're going to like the next next one then coming in from from Rose Travis, who says, you know, maybe I missed this, but don't you at least think that the publishing data makes fraud more difficult? Like now you can't just copy, paste and mix and match values in an Excel spreadsheet like the like actually a lot of the real real scientific fraudsters have been doing. Yeah, so I would agree that open data makes detecting fraud as it's currently done easier and thereby makes it harder to get away with fraud under current circumstances. But however, I think this is a very short term thing. The methods used to create what's sometimes called fake data or simulated data, these methods are widely used by statisticians and methodologists to basically look to see, you know, do your analytical methods do what you think they do? This is something that status that in undergraduate statistics courses are increasingly being taught because they're actually really, really useful to get students to understand like how actual data gets translated and like what things like statistical tests are actually doing. So these methods are already being widely adopted and they're very easy. It takes, I mean, depending on the complexity of your data, it can take just a few lines of code to write a simulation to produce data that I think would really pass any current forensic methods for detecting fabricated data. And so I think over the medium term, much less the long term, I don't think open data can really help with fraud. I think a committed fraudster can find, you know, can take a couple of days to learn how to use these methods and then simulate perfectly legitimate looking data. Sure, sure. Next question coming in from Max who writes, would you share some thoughts on the role of theory in replicability and expand on what you already briefly mentioned. It seems as though your point is that open science is sort of overtly focused on methodology. Could we say that in order to get better replicability, besides open science, we're missing good theory? Yes, yeah, okay. I don't want to just say like I really think sort of the study conceptualization level is probably the level where it's probably the most important level for a lot of these replication problems. I kind of think that, but I'm enough of an outsider to psychology and psychological practices that I don't want to like really state, like really assert that very, very strongly. I will note there is a lot of literature happening in psychology, other social and behavioral sciences, philosophy of psychology on these sorts of issues about the way psychologists use theory, about what they mean by theory, and also like sort of down the ladder at sort of the level of phenomena characterization, like our psychologists actually measuring the things that they think they're measuring when they measure them. Again, Olyana Feis, her recent work on this is really fantastic, really, really like fine-grained, careful case studies of how these things play out in practice, like how psychologists try to characterize phenomena, and how sort of a failure to recognize disagreement over phenomena characterization has led to replication controversies in some particular cases. So I really recommend, I mean, really it was her work that made me think like, oh, you know, maybe it's not sort of so much on sort of the statistics data analysis end of things. Maybe there's like something quite a bit deeper here. And I'm in, I'm not in a philosophy department because we don't have a philosophy department here at UC Merced. I'm in a cognitive and information sciences department. And so my colleagues are cognitive scientists, including a few psychologists, and this is something they talk about quite a bit as well. Cool to see that it's getting, it's getting uptaken with the practitioners. That's that's good stuff. Next question comes from Eugenio Petrovich who asks, so sometimes we have to work with proprietary data, full text of publications, records from Web of Science or Scopus that just can't be open because they're proprietary or protected by copyright. And there are entire fields like Scientometrics that are that are affected by this kind of issue. What do you think about this? What are people, what are people saying about this? Yeah, that is something I have run into repeatedly. I started doing sort of Bidliometric studies in 2015-ish and I've published several papers, excuse me, that use Scopus data. And what I have to say is, well, I can show you the code. Here it is on GitHub. I can't give you any of the data I use because Elsevier says that that belongs to Elsevier. I know there have been initiatives. I want to say it's the umbrella organization that does DOIs. Crossref? Crossref, yeah. I think it's Crossref. I think they have an initiative to try to include at least like the reference lists, making those as open as possible. But that requires working with publishers. And so whereas like PLOS is totally cool with it, like Elsevier and Springer have been really hesitant to make the references open. I think it's Elsevier and Springer. I might be misremembering exactly which companies. So I don't want to necessarily get people yet another reason to dislike Elsevier. Yeah. Oh, and I guess I should also note for a couple of years I had a postdoc that was funded by Elsevier. Just a full disclosure there. That's in my past now, but just so people know that. Yeah. So I think there have been initiatives to try to get at least like the reference lists or the citations openly available, but it requires working with publishers who have been really hesitant to do that. JSTOR has done a lot of really cool work, of course, on making the texts of the journals that they archive available for text mining and bibliometrics, which is really, really cool. I have a couple projects that for various reasons have been installed, text mining JSTOR data. And that's great if you're working with journals that are in JSTOR. Yeah. So I share the frustration there. And I don't unfortunately see any immediate solutions on the horizon. Yeah, that seems right to me. And I agree wholeheartedly. Two more questions here. One that I think, let me lean on you to answer a little briefly and then one that I think will take a little more time. One from Nicola Bertoli who says a great talk in the end. So would you say that the replicability crisis is more about technical problems like statistical power and significance or more about collections of social and epistemic norms? Okay. I'm going to do a very characteristic thing from me and say it's both. If you know my work, you know that I love sort of arguing that like these very technical problems are also social problems and matters of social norms. And I think just to pick out p-hacking and I think illustrates this really well, this is both in some sense a technical problem because it's technically violates the assumptions of statistical hypothesis testing and so your p-values become completely meaningless. On the other hand, it's also a sort of social problem of norms because like one of the big things people realized after Darryl Bem's publication is that as I understand it right, I haven't looked at it too closely but like the way he got his results was p-hacking. And so there was this realization that's like, oh, we have this practice that is perfectly normal in our field. Like we kind of teach our students to do this and it completely violates like all of the statistics that we teach them at the same time. So I think it's both a technical issue as well as a social issue, a matter of the norms in the field. Fair enough, yeah, yeah, I like that. And then lastly, so I love this as a final question. So for Marley Belovos who writes, so we can make our data open so that everyone can see it. We can situate ourselves in terms of indicating our personal biases. We're aware of power dynamics, we're respectful of indigenous and community ownership of data, we're as transparent as we can possibly be with our methodology, our funding, et cetera. What else do we do as researchers to build the kind of ethical data that we seem to be aiming for? A nice tiny, a nice small question for you to finish up. But I love this because this really brings together our whole day, right? This has been the sort of underlying theme, I think, of a lot of the talks today. So I wanted to get your thoughts. Okay, I'm going to bring in an answer from the major area of my work, which is in science and values. So if you know my work as a philosopher of science, you probably know me for the work I've done on science and values. Are you in for this kind of a risk-detilian approach to thinking about values in science? Something I've really emphasized in this work is building relationships. Because a lot of the science I focus on is environmental public health. It's very directly tied into public policy. That's the kind of science I'm most interested in. So what I emphasize in the context of science and values is strengthening relationships between scientists, policymakers, and the public. Where I give as sort of an exemplar of doing this well certain instances of the environmental justice movement. Where members of communities, often impoverished communities of color, situated right next to sources of pollution, suffering all kinds of health effects as a result of that, are doing their own science, partnering with scientists in order to sort of get the kind of institutionalized credibility that they need in order to get regulators to take them seriously. Where I want science to go is sort of in that direction. So if you ask me, you know, you step back from that question and ask, like, what do I mean by good science? I mean, science that is used to make people's lives better rather than worse. And I think the way this really needs to be done is sort of building connections to other communities, especially for example, environmental justice communities that like our most desperate have the biggest need for the support of scientists to pursue goals of justice and human well-being. So that's why, you know, we come back to this slide. That's why I think actually, like the last argument on here, the ways that open science can, under the right circumstances, support this kind of participatory research and also research in the global south. To me, this is actually the most compelling argument for open science because under the right circumstances, it really can move science in a really powerful way towards promoting justice and promoting human well-being. The citations I have here do a really nice job of sort of looking at how open science in practice doesn't necessarily do this super well and like things that might change to do this better. Fantastic. With that, I think we will leave it there. Again, as I mentioned, we'll be back in, in as quick as the platform will let us be back with a, with a closing, with a closing message. So, so, so thanks again, Dan, much appreciated. We will, we'll be back in just a few moments. Thank you.