 Hello and welcome to this event on critical perspectives on the MetaScience reform movement. And I would like to first welcome you all, and now we hear a little introduction first from Tim Arrington. Great, thank you. Hi, I'm Tim Arrington. I'm the Senior Director of Research at the Center for Open Science. I'm really excited to introduce this symposium, you know, critical perspectives on the MetaScience reform movement. We have a great lineup. I'm very excited to hear what everyone has to say. I'm sure we'll have a rich discussion. Before I pass it back to Seven to get us started to introduce, I just want to say a couple words about this event, why it's important, particularly why I think it's important. First is there's a lot of enthusiasm, just listening to the speakers before we got started. I can hear that from them as well to have this event included as part of the MetaScience meetings that have been ongoing. This has actually been occurring since the very first meeting in 2019. That critical perspective was more on replications. When we had the virtual event in 2021, there's also a critical perspective that was included. And I'm excited that we get to do that again today, include that perspective alongside the MetaScience meeting that's occurring. So hopefully this is something that continues. And the reason I want that to continue, and this is the last point I want to emphasize, is this is healthy, right? This is a very healthy thing to have for any discipline and any scholarly work. We want to embrace kind of the self-scepticism enterprise as much as possible. And in many ways, to function well, the most important and valuable insights come at that intersection of differing perspectives. So with that, welcome everyone to the symposium. I look forward to everyone engaging and hearing from our speakers. Yes, let me just introduce myself. I'm Sven Ulbs, I'm a PhD student at the Danish Center for Studies and Research and Research Policy at Aarhus University. And I'll be moderating this symposium. We have six interesting presentations today. And I will shortly introduce every presenter prior to each presentation. And after each presentation, there might be time for one or two short questions and answers to each presenter. And in the end, there will be a 40-minute long Q&A panel discussion with all presenters where you can ask questions addressed to a specific presenter or just post a question to the whole panel. And you can post questions throughout the event in the chat. And I will select questions and post them to the presenters. So I guess we can pretty much start with our first presentation. And that is from Bart Penders. Bart is a associate professor at Maastricht University. But currently he's doing a fellowship at the Keite Hamburger-Kolleg-Aachen, Cultures of Change. Bart, the stage is yours. So thank you very much, Sven, for that introduction. It's the Keite Hamburger-Kolleg, Cultures of Research, not necessarily Cultures of Change. Although change is something that we deal with a lot, of course. So thank you very much for inviting me, for allowing me to speak about some older work, as well as some new work. The title that I chosen, Shamed into Good Science, is, well, it's actually the use of the word shame, is mostly a signal word. I won't speak about shame too much. But actually, to me, shame is part of a constellation of elements, of etiquette, so shame, disgust, guilt, pride, joy, empowerment. We feel all those things, and they guide our actions, they guide our decisions, and in our daily lives, but also in our scientific activities. And this etiquette, I will talk about that in a few minutes. To me, it's very much part of a civilizing process. At least we can conceptualize meta-science, even up to a certain point, as part of a civilizing process. And if you take a look at the entire list of adjectives that the Center for Open Science included in the invitation, civilizing is one of them. And it's maybe the kindest one when you talk about critique. But it is very substantial thinking about what it means for how we do science and how we interpret what we do. So knowing that I would kick off, I hereby present, let's say, an idea of a starting point for what, for meta-science and scientific reform. The realization that the literature, scientific literature that we know, that it is not actually a collection of true claims, that science is not a prose-optimized to produce those claims, and that scientists are not actually incentivized, nor do they virtuously adhere to do the right thing. This realization that this is the case is, to some common knowledge, and to others, possibly new, or at least at the beginning of the meta-science movement or reform movement. And it has been described as a moral panic, or in the words of Peterson and Ponofsky, as a scandal, in a sense, that the reform movement managed to scandalize the social character of scientific practices. That is not to be overly critical of that, because if you manage to scandalize something, you also are able to mobilize a lot of attention and also a lot of resources in order to get something done, which means that you can try to fix it. And that is, of course, the objective of meta-science to create the knowledge that then the reform movements can use to reform to change scientific practices. And these reformed scientific practices, I will argue, are actively positioned on a gradient from regular, or perhaps primitive, incarnations of scientific processes to more advanced processes. And of course, everybody wants to be at the advanced end of that. And in order to get there, you have to show that you are engaging in this more advanced process of doing science. Reform science has to, in some way or another, display its reform in order to distinguish itself from bad science or more primitive science or regular science. And there's a strong moral appeal in that. So the claim, open science has just science done right, displays that what we should be feeling, thinking, or doing is that the right thing is the reward in itself. And that means that we have to think about science, not focused on its outcomes, but focused on its process, the how of science, not necessarily what it produces. And to that end, it requires certain characteristics. Certain characteristics need to be promoted. Transparency, obviously, in the context of open science, but also to a certain degree standardization. If you want to compare one experiment with another in the context of a replication, but also beyond that particular context, then you need some degree of standardization in addition to the transparency. You need a lot of documentation. And some of that documentation can take the shape of bureaucracy. And Tom will say a lot about bureaucracy, and especially how bureaucracy is also a type of work. But to me, bureaucracy is, above all else, a type of politics. I'll get to that. But by doing all of these things, by showing that you participate in a process of science that upholds very specific norms for transparency, commits to standardization, displays all these documents, and sort of lives up to bureaucratic requirements, is a way to display that you focus on the process much more than the outcome. And if we conceptualize science primarily as that process, then what we need, or what we have actually, up to a certain point right now, is a series of innovations at our disposal, a series of changed ways of doing science, changed incarnations, sort of of scientific practice, populated by different things. One of those different things is a set of bureaucratic innovations. I just mentioned bureaucracy, but quite important are preregistered reports, but also very detailed reporting guidelines, forms and templates and formats that are made available for all of those things so you don't have to reinvent the wheel or come up with what you yourself think might be important to include in those. Those forms and templates work greatly, of course, also towards standardization. And I've written about bureaucratic innovations before, but what I want to talk about today also is social innovations. And actually, whether or not innovation is a good word is up for discussion because what we see in reformed science is not just a sort of design of a process to live up to new criteria and to use these bureaucratic innovations, but also to self-organize in ways that, to some, are new to others, maybe not so much. But we've seen the emergence of various collectives and collaborative forms over time, over the last decade or so in order to uphold the very norms and values of open and reformed science. And these are the many, many labs approaches, big team science approaches, but also adversarial collaborations as ways to optimize, in a way, the penetration of these open science values and reformed ideas of what science is supposed to be. I'll talk today only about the many labs of collectives. But before I do, I want to do a really short excursion into a little bit of theoretical literature. This is a quote from the book, The Process Genre, by Skwirski. And take a look at just the read a little bit. Of course, feel free to read everything. But what she does in her book on the aesthetics of labor, and the book is called The Process Genre, is show how by our ways of displaying labor and displaying processes, we attach certain value systems to them. And we evaluate relative to one another different types of processes. And what we have here, or what we can talk about, are developmental trajectories from primitive to advanced. And so in this process genre, the advancement from primitive to advanced is, that's the last few words, by extension, a statement about the status and character of a people or even civilization. And because this is about distinction and because this is about civilization, it means that we have to talk about, at least briefly, about some sociological theory, French and German sociology, that we most of us, perhaps, even know. But you, Foucault, Elias, all talk about how different ways of self-presentation and displays of distinction allow us to set ourselves apart. To show that what we do is a different end, of course, better than what others do. And I've chosen to rely primarily on the work of Norbert Elias, the German sociological interpretation, and conceptualize metascience and scientific reform as a civilizing process, a process in which, through subsequent and continuous processes of displays of distinction, slowly manners, morals, norms shift and expectations of what scientists do change. And a lot of that is formal in those bureaucracies that I just mentioned in sort of governance structures of collectives. But a lot of it is also informal, just how people work together, how they relate to one another. And these bureaucratic tools fits this notion of civilization, but these collaborative processes, too. And for this, for instance, we can think of the possibly famous quote by Kropotkin, that competition is the law of the jungle, but cooperation is the law of civilization. So to work together is the more civilized, the more advanced process towards knowledge construction or the making of knowledge. So what are those many labs? Many of the people in the audience might actually be working in one of these many labs, but others may have never heard of them before. So there are many, many labs. So the original many labs were conducted under the auspices of the Center for Open Science. Many labs, one through five, a series of centrally governed replication studies in order to assess whether or not studies would replicate under certain conditions. And these are in the past, but they inspired a lot of other people to also organize their work in similar ways and also adopt as a consequence similar labels. So I'm showing you here not all, but many of the many labs out there. And they cover many topics and even many disciplines. And they are all different and all similar. What I did is I interviewed, and I'm actually still currently interviewing, people who are working in these many lab studies in various roles, leadership roles, data collection roles, in order to get to the core of what it means to work in the context of one of these many lab studies. What's it like to do science that way? And here too, the etiquette of process is obvious. And there's a lot of formalization going on and a lot of bureaucracy associated with that formalization. So what we see are formal agreements, not always, but often between partners to make things clear what is expected of you and why. All of manuals, some are huge. Standard operating procedures, videos. So videos of a pilot experiment, for instance, that is then shared with participants who then send video back of their experimental procedures in order to, well, on the one hand, convince each other that they are doing things the way they should. But also in the context of data collection, that video is also data. And most of this is very remote. So in some of these collectives, people meet each other, but often not. So there are quite a few of these many labs, collaborations where people don't meet or if they meet, it's actually accidental instead of part of their collaborative relationship and much, much more. So what does it mean to formalize collaborative relationships in such a way? What is the consequence of this type of formalization? So there are many consequences, but one of the things that I'd like to highlight is that it really does change relationships between people. So this is a quote by one of the people involved or actually working in one of the many labs, reflecting on the commitment of participants to many labs. Most of the people love working in those many labs and they're having a lot of fun. But a fraction of the people of the authorship, this respondent tells me, just respond only to emails critical to their own authorship. Like they don't provide any comments on the draft, they don't seem to care. And that makes sense. They've done what they've been asked. They fulfill the terms of their contract. That is what you get with contracts. This is not exemplary of how people work in many labs situations. Actually, most of the time people are heavily involved with one another and go way far beyond what is expected of them. But formalization allows people to stick to the letter of the contract instead of the spirit, which is interesting, of course, but luckily, mostly the exception. But what formalization does do is establish power structures, quite formal power structures through the centralization of, for instance, the design of research. So if you have a many labs collaboration with 300 people in them, they can't all equally, at least not easily, co-design the study, even though they can all participate. But that creates differences between four teams, key teams, leadership roles, and collaborators. And some of those collaborators are referred to as collaborators, but sometimes also as participants. So that creates a little bit of confusion because many of those participants then collect data from participants. But it creates low task uncertainty for some. Another quote by one of the many labs participants is, it's exceptional, really. You give me the stimuli, and I could just start testing. I felt like a research assistant. It's, in this particular case, absolutely clear what to do. But power structures always create insiders and outsiders in various gradients, of course. And if you have, if you use these bureaucratic tools like, for instance, pre-registration and registered reports, you have to decide, at one point, you submit your registered report. And one of the many labs researchers shared with me that this is also a bureaucratic tool of control. So in a way, if you don't have anything like a pre-registration or a registered report, a lot of people will try to change something in your design. And I don't think that's a good idea. And when we receive questions like that, we could count them relatively easily because we said it's approved like this. We can't actually change it. So we avoided having many changes along the way. It made it a lot easier because we now know we have to do it this way and we can't change anything. Our initial idea stays our idea. Here, pre-registration is used as a tool to exclude certain voices. But you can also flip that out, of course. At the same time, and these are not quotes for paraphrases because I don't have the approved transcripts yet, we see people who go above and beyond to get everybody's input, get consensus across all intended collaborators before we move to registered reports. And that can take sometimes years to do in order to do it well. But that's an investment that people are willing to do. Or, more differentiated, we collect views of the underrepresented and freeze those, preventing those with a lot of power to shape it their way. So the decision when to submit a pre-registration or registered report is a decision you have to make. You cannot not choose. But whether you use that to amplify your own voice, democratize the inclusion of voices or amplify a very specific subset of voices is a decision that is guided by whatever moral economy you have underpinning the study that you're trying to do. I'm going to skip this one because I'm running late already. If you want to know more about this civilizing process, I wrote that down. But the question is that if you continue this process of distinction, where does it end? So a focus on process and the eradication of all sorts of bias from that process means that we have to focus on the weak link. That weak link is us. We humans are imperfect and that will never change. And if it is us that threatens the integrity of science, then it's us who has to go. And that's something that we see throughout the history of science of course in the production of this ideal of mechanical objectivity that we also see in the context of meta-science. An ethos of self-annihilation as the price of knowledge. Science as something like death, removing yourself through dying. Or very much applied to notions of meta-science. The post-human theory of science requiring total transparency and machine intervention. Or, and this is from a paper of Sarah Anne, who you will hear in a few minutes, a push towards maximal and mechanical objectivity is reinforced through the infrastructures that we've set up. So this ideal of post-human epistemology, frame science not as labor, but as this disembodied process. But these big collectives, these many, many labs, they are, if anything, more social than before. And collaboration is contrary to Kropotkin's quote, not actually the opposite of competition. Big team science or big science in general provides structures within which you can compete with one another and structures that compete with one another. And from time to time, the term frenemies pops up as a description of how scientists operate relative to one another. Big science is something that we've been studying for decades. So the little book I'm showing here, Little Science, Big Science by the Solar Prize is from the 60s. And since then, we've seen a lot of work on big science. One of the things that's interesting also to highlight very briefly is the imagined cultural power attributed to this approach of science. And the cultural power is a term that Charlie Ibersol actually used on his blog. And what it does, it sort of produces expectations among the people inside the many labs collectors but also those outside on what it can do. And it looks like that it is acquiring a lot more authority than some people are actually comfortable with. So quite a few people responded similarly than the first respondent whose quote is shown here where they expect the many labs to provide the final and definite answer to a scientific question because of the organization of science facilitating that. So kind of answering this question once and for all, we want to have a 100% sure answer. And the authority of these big projects is also a problem if you disagree with them. So one of the people involved says even though this person is involved in a many labs, if you look at it from the outside, imagine that you disagree. What could you do to actually disagree? Last sentence there, it is just too big, too powerful to actually start to disagree with. It also means that if there is so much authority attributed to this way of organizing research, then it might become possible to use it as a tool to adjudicate between wrong and right or to use it as a policing tool. And one of the things that we've seen here is in the context of the assessment of the work of Gino, for instance, the label many has also been adopted in the context of this many co-authors. So wrapping this up, many labs are big science and big science is managerial, formal, hierarchical, centralized, political, but we also know it rather well. A lot of work on big science has been done. Interestingly, the many labs are not yet very much driven by central funding, which has been one of the main drivers of other older big science initiatives. But in the end, what we're all trying to do here is convince each other of the value of our claims through organizing our work, organizing our labor and the way we communicate. And in many ways, scientific reform and the many labs as an example of that are performances of an imagined civilized process. And that makes me end with what I should have called this presentation, namely, and it's also the paper I'm going to try to write about this, renovating the theater at the theater of persuasion. Thank you. Any questions? Thank you, but this was really interesting. I personally kind of liked how you started with saying that science is not maze for truth claims. And then one of your last slides was where someone said that we want to get truth. That was quite fun. And now I think that since we don't have any questions in the chat yet, but if anyone has any questions, please put them in the chat. Nicole has raised a hand and maybe has a question to ask. So I'll open the floor for Nicole. Sure, thank you, Bart. This was really interesting. And I see why you said over email that we should talk about some of the links between what we're working on. I'm wondering if you collected any data on people's motivations for joining these projects. And if you think if there's any way of accessing information about the so-called non-users here or people who do not want to join a sort of many labs style project or feel alienated from that style of research. And if we could access those types of people, what would it tell us about the kinds of cultures that are growing up inside of these large collaborative projects? So yes, I have collected data on motivations of people to join. I have asked every respondent why they joined and for some of them how they actually or why they set it up to begin with if they are in this funding role, for instance. But because I'm restricting respondents to people inside these many labs, I don't know why people would not want to join them except for reflections of insiders on outsiders. But that is not necessarily actually used for that. So you'd have to specifically take a look at people who do not participate and sometimes actively also critique this type of organization. You'd probably have to seek them out specifically just like you'd have to seek out people inside of these collectives equally specifically in order to find out. But this comparison, of course, makes perfect sense. But I do want to add that one of the things that I've learned and I also imagined to be the case, of course, is that the people who work in these many labs consortia do not spend all of their time on working in that many labs consortium. They spend a part of their time on doing this type of inquiry. And at the same time, also participate in other types of inquiry and actively reflect on the differences between them, sometimes at least not all the time. And in that sense, there is quite an interesting contrast to be detected. But in my case, at least, only in the subset of people who already are willing to at least devote a part of their time to this type of inquiry. Thank you. Time-wise, I think we have to go on, but I like that there are now a lot of questions coming up in the chat. We will come back to those questions probably in the Q&A at the end of the event. But now we will come to our next presenter who is Thomas Ostler. Thomas is a senior lecturer in the Department of Psychology at the Manchester Metropolitan University. Thomas, thanks for joining us. Yeah, thanks for inviting me to talk. So, yeah, my talk today is going to be called the Invisible Workload of Open Research, and it's largely based on a paper I published of the same title last summer. So the critical perspective that I'm kind of trying to bring to this webinar is that a lot of metascience is too focused on the abstract research process. And that might seem a bit of a strange thing to say because I guess almost by definition, metascience is research on the research process, but it is also an applied field. And most of the time we're doing this research because we want to change the research process, we want to improve it, we want to kind of have a new intervention in it. And so I think we need to think about the context in which these changes happen. And I think typically the context that people adopt is what you might call the research ecosystem context. So in this, you know, metascientists, we're interested in the kind of the main individual or the group that we're interested in is the researcher. And the context that they exist in is one of journals and the journal publishing policies of funders of third-party kind of tools and infrastructure, things like the open science framework or independent projects like Code Check. The researcher is embedded in a global scientific community of researchers and we might kind of even be interested in sort of comparing across national policies and how different countries are sort of promoting open science. And I think whilst this is very valuable, there is another context that we can look at which is the university context. And thinking about it like this, we acknowledge that actually most of the people who are interested in their researchers, they do research, but they also do other things. And typically they're not necessarily just employed as a research assistant, they're employed as an academic by a university. And so the context that the academic exists in is slightly different. We need to think about the services and resources that are provided by the particular university that employs them and that they have access to. We need to think about the managers and the leadership structure in the university that often sets expectations and sets policy for those academics. We need to think about all the work that academics do that isn't research. So teaching, grading papers, academic service. We need to think about the colleagues. So not just the academic, the researcher is being part of a global community, but the colleagues that they interact with on a day-to-day basis, be that fellow academics, but also non-academics and support staff. And finally, the local policies that exist for any one individual institution which can be quite varied. And so I think that this perspective has sort of a lot of potential to improve or to contribute to meta science. And there's a huge literature already about higher education which is not just about education, it's about the kind of academic context. But what I'm gonna focus on today is thinking about the sort of the non-research work that academics do and how open research impacts on the workload and time that academics have. So this is a very nice infographic that I found that kind of helps to visualize what academics spend their time doing. So yes, we as meta scientists are sort of interested in the research portion of this, gathering data, analyzing data, publishing, disseminating research, but that's only one portion of an academic's time. There's also everything that goes with teaching, assessment planning, delivering teachers, teaching sessions, training, supervision, and then everything that kind of might broadly fall under academic service. So serving on committees, attending career development workshops, organizing conferences, things like that. And this way of visualizing it is actually very similar to the way that academic work is often conceptualized by the university and the university management. And in fact, it's often formalized in this way through academic workload models. And I'm aware that this kind of differs from country to country, but in general, it's about the term projectifying time, so kind of allocating the time available to someone into specific different projects. So in the UK and Australia, I think they're called workload models. In the US, I think they're more commonly called like your teaching load. And this is my workload for this year. So this is what the university thinks I do and expects me to do and spend my time doing. And you can actually see from this that actually research is almost only a fifth of the time that I'm kind of employed to do that. The rest of it is teaching, leadership, seminar committees, training, things like that. Now, you can see from this that I have, you know, 358 hours per year allocated to do research, but it's not broken down further than that, right? It's not specified exactly how I should use these 10 hours. So if you kind of agree with the sort of, I think it's quite, it's not necessarily controversial to say that open research generally does take more time than closed research. Just something like sharing data. It takes more time to, you know, format your data set, put it on a repository than it does to just not do that. Then if the new demands to make my research open or the requirements or even just incentives means that it will take me longer to do it, am I going to get more time in my workload to do that? And I can tell you right now, the answer is no, because that would involve basically taking something else away, right? So I can't go to my head of department and say, oh, I'm going to be doing a lot more open science now and sharing my data. You know, can you take away some of the essays that I have to mark? It's just not going to happen. And this idea then that we change the expectations of what a researcher should achieve in a certain period of time, but without actually giving them sort of more time on a workload is known as workload creep in this higher education literature or workload intensification. And so it's basically expecting people to do more, but without giving them more time to do it by taking away other activities. Now, I think that's a sort of useful thing to bear in mind then when we move on to my next point, which is that you might say, okay, you know, yeah, okay, open research does take more time, but it's not that much is it? Like if you're pre-registering something, if you're sharing like a data set, quite often it's, you know, it's maybe a 20 minute, half an hour job or something. You know, it's basically just kind of filling out a form and providing some information. And I think that's true, like I agree with that, but I think in a way that can make it almost more of a problem than if it was a huge investment in a new activity. And a lot of open research practices or interventions through, from journals and funders or other actors to increase the uptake of open research, thinking about them as basically minor administrative tasks is also quite a useful perspective. And to be honest, I think that as much as we as meta scientists can kind of wax lyrical about pre-registration as this amazing tool to reduce bias and, you know, prevent p-hacking, there are going to be a lot of researchers out there who look at a pre-registration document like this and go, oh, great, like another form that I need to fill out before I can actually start doing my research. So this might apply to things like pre-registration, data availability statements, transparency statements, transparency checklists, also contribution statements, metadata descriptions. It is essentially admin. It's filling out a form and providing information that records what you are doing. So what is the issue with that? If it does only take 20 minutes, like why is that a big problem? And as I said a second ago, I think it's because it's a cumulative problem and that the existing administrative burden is quite high for a lot of researchers. So this is only my personal example, but if I want to do a piece of research at my university, then I'm looking at probably filling out a 16-page ethics form application, another 12-page research protocol document to upload as part of that, a 10-page data management plan. If I'm collecting sort of sensitive data, then probably another 14-page data protection impact assessment, maybe another six-page information security assessment if it's particularly sensitive data. If I want to do any research involving the UK National Health Service, the military, prisons, police, I'm looking at at least double that, because they're all going to have their own ethics processes to go through as well. And there's not a huge amount of research on this already, but a study from 2009 that looked at the time that investigators spent on funded grants found that they estimated about 42% of an investigator's time was actually spent on the administration of a project rather than doing the actual research as in collecting and analyzing the data. And this is worth bearing in mind when we kind of return to the idea that even that is only one part of an academic's job. And so the administration that is associated with research and the new administration that's associated with open research, maybe that does only take 10 minutes, but across all of these aspects of academic work, there is administration, and typically it's increasing as people introduce new initiatives and ways to improve it. So we're in a webinar right now about meth science, possibly somewhere else on the internet. There is a webinar of research safety officers debating a new health and safety procedure that is going to improve the safety of research, and that's going to involve filling out another better improved form that takes about 10 minutes. Maybe there's another conference going on about pedagogy and teaching that involves another new intervention there where maybe that takes about 10 minutes. So across this whole sort of diagram of academic work, there is administration everywhere, and it's always increasing. So although any one individual instance of an open research practice or administration, it can just be dismissed as trivial. Like 10 minutes is a trivial amount of time to complain about. You can't even really formulate an argument against it, but it's not just the one piece. It's the cumulative burden of all of these that has been described as it's death by 1,000 10-minute tasks. So thinking about open research as a type of administration is really useful because there are already a lot of literature and theories out there about administration and administrative burden and how we can decrease it. So I'm going to share with you three ideas from that that can be helpful in thinking about this. So the first is the distinction between administrative sort of justified burden and administration and red tape. And this distinction is that if it has a benefit, then it can be justified. But if it entails a burden, but it doesn't actually make a contribution to achieving a rule's functional objectives, then you can think about it as red tape. And one example of this is something like on an ethics form, the ethics application that goes across my whole university has questions like this on it that says, does the project involve ionizing radiation? Does the project involve lasers? Now, 99.9% of researchers in the university, the answer is going to be no because historians and people working in the English literature department are just not going to do a project that involves radiation and lasers. Even people like myself in psychology, like I don't do research that involves that. So asking every single person who's doing an ethics application whether their research involves lasers is not actually for most people fulfilling the objective of trying to protect participants because they would never even get to the stage of being able to use that. Another idea that I really like is about rule redundancy. And this is the overlap of administration from different organizations which creates this again unnecessary burden. So if you think of something like a data availability statement, this might be something that you need to provide in a grant application, on an ethics form application, to your university sort of research services if you're hosting the data with them, for a data repository and for a journal. You can almost guarantee that although they all require the same information, the exact form that it comes in is going to be different. So maybe the word count or the word limit for this is different. Or just the specific way they ask the question means that when you are copying and pasting it from all these different organizations, you're having to sort of adapt it slightly to fit the question that is being asked. So that is again kind of unnecessary time cost of having to duplicate it but change it every time. The third idea, which has kind of got quite a retro-sounding name of robotic bureaucracy, I think it was a theory sort of developed about 20 years ago or something, is that often we think, okay well we can automate a lot of this administration and that is going to reduce the burden. But this would say that in theory that works but it can actually inadvertently end up increasing it because the forms are difficult to understand, they're misinterpreted and then you actually need to kind of contact someone and go back and forth through it and sort of explain why it's not relevant to you. And this is a particular issue when we think about, well particularly relevant to a lot of research done in the humanities because a lot of existing research administration like ethics applications often by default kind of assume that your piece of research is like a biomedical clinical trial and they're asking you about what's your hypothesis? How are you going to collect informed consent? But a lot of research, those kind of concepts are relevant. So I think Nicole is later is going to be talking about auto ethnography and the idea of consent in ethnography is a lot very different to like if you're signing up for a drug trial. And so we have something like the level three top guidelines for journals which say you must post your code to a repository and it needs to be independently reproduced prior to publication. You need to pre-register your study. You know a lot of research people are going to be going like well that's not relevant to me. So I either have to sort of like explain that to the person doing it or write some statement that explains why it's not necessary. And again that is in terms of fulfilling a functional objective there's not really doing much. And the sort of final example here before we wrap up is that I've talked about examples where it's sort of new bureaucracy created by open research but the open research can indirectly sort of make existing administration more burdensome as well. So if you think that conducting research openly tends to facilitate big team science projects like Bart talked about before this is one that from the psychological science accelerator had something like 500 authors on and I can't remember where I heard this but they said that when they submitted one of these to a journal they got to the sort of journal submission page and they had to add the authors in and they sent an email to the editor saying like oh you know there's information in a spreadsheet can we just send you that and they were like no this is the only way that we accept submissions and so someone had to spend nearly two days manually entering every single author one by one into the manuscript submission portal which is obviously an unnecessary administrative burden. So I'm going to wrap up with three implications and solutions then so from thinking about administrative research sorry the time cost of open research and thinking about it as a type of administration and I think the first implication is that we do need to think more about the time costs associated with open research practices if all open research is associated with the time cost even if that is minor compared to not doing it and we need more discussion around whether these costs are justified are initiatives fulfilling their functional objectives even something like pre-registration where the objective is to reduce bias in the research it's not fulfilling getting everyone to pre-register research it's not going to fulfill that if actually no one is reading these pre-registrations or checking whether they've been followed. It's not easy to design administration that is not a burden but equally having 100 different forms for 100 different kinds of research is a burden as well but I think you know one thing that we're probably not doing enough is just getting feedback from users about their use of these practices and these forms to see whether they do meet their requirements for their areas of research so the administration of open data is going to look very different compared to the historian compared to an astrophysicist so should we be getting them to use the same form? The second is building on that idea of rule redundancy and how we can reduce that can we work together to collect and standardize information one idea that I really like that I've kind of not seen takeoff as much as I thought it would is the registered reports funding partnerships where you basically report but you apply for the money to do the project at the same time so it's like a grant application and a journal submission in one to reduce that overlap of the bureaucracies and even little things like reducing formatting requirements accepting format free information you know this is something that a lot of journals are doing now which is a godsend when you've been rejected from one journal and you want to send it to another you don't have to spend a lot of time reformatting it and although it is quite cliche to say like oh maybe AI could solve this you know that is actually something that it is quite good at giving it a load of text and saying can you rewrite this under the following headings so maybe there's potential for that to sort of reduce this as well and then finally then you know I think going back to the title of the talk about invisible workload I think we need to see that open research takes time we need to change research culture to account for the time that it takes we need to make sure it is seen in workload models and accounted for in the metrics and the expectations that we assess researchers on we need to make sure that job roles and responsibilities are formally recognised the work that goes with open research this is particularly for kind of non-academic support staff who may have previously kind of been a technician but now suddenly they're expected to help with making data open making sure that's recognised in their job roles and kind of formalising those careers a bit and setting expectations for students and ECRs about the demands of open research and overall final point is to not promote open research uncritically it does take more time and if you think about academic work more holistically you kind of quickly come to realise that time is an academic's most valuable resource so I've gone a little bit over so apologies for that but thank you very much yes thank you Thomas I think we might have time for one quick question so I think an interesting question also because you hinted towards this in the end of your presentation is how can we understand and reconcile misalignment of functional objectives of researchers, employers e.g. universities versus those of employers like customers like funding agencies yeah I mean good question I think I think that is already there is more connections between universities funders so my university at least we have a whole department of people who will help you write your research grants, help you choose a funder who fits your area of it and I believe that they are in quite close contact with funders and kind of feeding back towards each other so I think there is potential for those links to become stronger and that might involve things like building in ethics applications with grant applications so again if you can kind of do those at the same time your university approves your ethics for a project that you want to do at the same time as it approves your funding application to this agency because you are going to have to do it anyway right you are going to have to get ethical approval at some point if it gets funded then sort of bundling those together can help reduce that bureaucracy thank you it was very interesting now after the next presentation we will have a five minute break and before we will have one more presentation which is from Stefan Guttinger Stefan is lecturer for philosophy of data and data ethics in the department of sociology, philosophy and anthropology at the University of Essex great thank you Tom could you maybe stop sharing your screen so I can share my sorry thank you let me just share this so just once I'm sharing here thanks to Sven and everyone who was involved in setting this up and thanks for having me I really appreciate that I'm also, I realized when I looked at the abstracts that I'm quite the old one out because I don't really talk about the reform movement necessarily but I do hope it's interesting and I do think it should be relevant so as you can see from the title I want to talk about why I think replication is underrated and by that I mean that also the people who are really rating replication for instance people, certain people in metascience or in the reform movement are actually not rating it enough so that's kind of where I want to go and the title is for those who know the paper is a play on a paper by Uliana Feist who's actually in the room so hi Uliana I can't see you but I'm good to have you here and so her paper is why replication is overrated and one of the claims she makes is that replications are less useful and important than is widely assumed at least in the kind of psychological research I have focused on in that article and to confuse everyone I'm going to say that I actually fully agree with what Uliana is writing in that paper and I think actually can be expanded it's not just psychological research the kind of uncertainty that she highlights I think they also apply to molecular cell biology and other fields and so yeah I'm at that point where I'm saying basically replication is overrated and replication is underrated so the question is obviously how are you going to how do you pull this off and the point is obviously that how I split what I mean by replication and that's really where I want to go so the move I want to make here is that I want to say that replication understood as dedicated replication studies is overrated in the sense in which Uliana for instance describes it so we should still rate it but not overrated so it's not like a fool it's useless or anything but just we should not overrated and then the other point is that replication as what I will describe here as inbuilt replication samples is actually underrated and so the two things I need to do obviously here is I need to first explain what I mean by this distinction dedicated versus inbuilt and then from that I need to explain or develop what I mean by underrated in what sense and why so that's what I'm going to do so for the first step this distinction here the first part of it like the dedicated replication study that's kind of just what we usually experience or encounter when we talk about replication especially also in metascience or the reform movement this is an experiment that is a dedicated replication it's set up as a self-contained experiment or a larger project depending on how it's structured and it's not part of the everyday research process like Bart said that people don't just work on many labs they do other things as well they have their own research strength going so that's a methodological point these are like the studies that we know and many more obviously and they can also be smaller scale and are not published but it's kind of a dedicated self-contained thing on the side almost obviously related to your interest maybe it's a quantitative point it's an observation that always comes up we rarely performed replication studies dedicated replication studies and now this is obviously what it's being pushed for as part of this kind of replication crisis narrative and the activism that it generated and then kind of the goal of these types of replications that's something that the authors listed here have highlighted that this is kind of a diagnostic aim a point in there we want to say something about those studies that are being replicated so the previously published results just quickly on this slide you can disagree with that in different ways I know that like distinctions between direct and conceptual replications are not that popular anymore I think they work for certain purposes I'll keep to that more or less for now but if we think of roughly what many people describe as direct replications you know we see different things they can do they can at least help to detect fraud or kind of questionable practices and they can help us increase trust in the reliability of these previously established results so they just once we can replicate the result for instance we can reduce the likelihood of a chance result due to sampling error or there was a fault instrument experiment to mistake and so on and then people often highlight like all the things they cannot do so they cannot check for systematic error because they kind of the idea is to use the same setup so if there's an inherent flaw in there you will reproduce the artifact so you can just say you've got a robust artifact you cannot necessarily test generalizability even though you're always generalize because you do it again in a slightly different context and so on but it's kind of you're not using a different population for instance or you're not extending it to different materials because you try to do the experiment with the same materials and also this kind of idea of robustness like you cannot reestablish the robustness of your result because too many people that means you would have to show kind of the same result using different methods or different operationizations so it can do some things it can't do others so they cannot do everything but who can so they still do necessary work so they should be rated but not overrated maybe so yeah that's where we are with these dedicated replication studies the other side of that distinction that I want to make here are these what I call inbuilt replication samples and so what I mean here is that this is a replication that is part of the everyday research process it's really integrated into the experiment that you are performing as one sample among many others in that experiment so again that's the methodological point how it's done, where it fits in into your research practice process for the quantitative point I would argue that is an abundant form of replication Peterson Panofsky in their paper where they interviewed scientists they come to the same kind of they have this claim as well and here these types of replications can have for instance also an integrative aim and that's kind of you try to build on previous work you try to integrate work that is happening it's not necessarily just diagnostic it can have a diagnostic element to it but that's not necessarily the main aim or the motive for why this is being done as Peterson Panofsky put it just quickly that's actually we can trace this kind of idea of these intrinsic kind of replications back quite some time so here's Hunt from like 50 years ago says replicating is an intrinsic part of science and not something which is missing unless someone deliberately sets out to replicate which is what I would call kind of a set up a dedicated application study Schmidt in kind of well-known 2009 paper he writes at the end in a follow-up study parts of an earlier study are directly replicated within that new study but then there is either a second condition within this experiment or a second experiment that assesses a new hypothesis that was not tested before it's kind of building on this previous work and Stuart Firestein also says experiments get replicated because people from other labs use to publish results and the methods in their own experiments so again this is kind of built in replication that happens and so this is really kind of this integrative mode or the motive is integrative it's like you replicated because you want to build on on these existing findings again I wrote about this in a previous paper where I looked at the experimental life sciences and here I said that I'm basically almost all experiments in biology contain such an in-built replication which is often quite direct in the form of the humble positive control sample so it's one sample among many and back then I called these micro replications because they don't necessarily replicate the whole previous experiment but only the bits that are relevant to that particular experiment and I think this matters for the quantitative point because I do think dedicated replication studies are very rare but these built in replication samples are highly abundant and that's something that also Peter observed in their interviews so there's actually much more replication work happening than we usually kind of talk about or recognize right that addresses the quantitative point I just want to give a quick example of what that might look like so this is a random study I picked just on google scholar and just open and checked who has a positive control some papers don't use positive controls which I think is problematic and anyway we can discuss that later but here's a study just for Arabidopsis plant biology details don't really matter but it's more this is figure one so it's kind of the opening figure of the paper and here they characterize their model system right there their experimental setup and so here WT means their wild type and then this here is a mutation a mutated gene that they introduced and then over here if you can see hopefully my cursor is like the rescue phenotype and so this is repeated here you can just see if they mutate this protein or the gene CTL1 and it doesn't function the plants are smaller they don't grow as tall and over here you can see that the root structure is very short so why do they go through this well if you look at the text they say well we examined the phenotype of this mutation in detail blah blah blah we found they're smaller and so on and then it comes here and this is consistent with the previous observations and if you go to this paper labeled number 15 you'll find a study that has reported for the first time if you mutate this protein you find this phenotype you find this type of effect and so you will not find if you google this or google scholar this you will not find a replication study of the CTL1 phenotype mutation but it has been replicated just in the form of this kind of integrated positive control and somebody has actually directly replicated this particular experiment and so that doesn't get published in that sense so if you google titles and abstracts for replication studies you will find nil there's just zero but actually they all happen so it's also a measurement problem I guess for metascience where do you look to find replications so that's just one example I could bring up others sometimes the point here is I think introducing this distinction between these kind of dedicated replication studies and these replications that happen as part of everyday research practice does not change not only change our quantitative claims how much is there is out there and how we measure it but also claims about the function of replication practice and if we go back to the dedicated replication studies and this arguably flawed but hopeful informative slide you can see what I highlighted here that the main work that is being done is that trust in the reliability or stability or some about truth of previously established results that's what it works towards and that's a backward looking function you generate new data to diagnose the previously existing data right looks backwards and I think that also includes what Petersen and Pinovsky describe as the integrative mode because in the integrative mode or with that motive in mind when you replicate you're trying to establish does that bit of existing literature work for me can I build on this can I extend on this or should I ditch this and choose something else and so on so it's kind of this do I integrate this with my own work it's kind of in a ways backward looking still what I want to highlight here is that these inbuilt replication samples this kind of integrative mode of replication does not just simply corroborate existing results either for diagnostic or integrative purposes but it's central to increasing the trust in the new data that is being generated because as we know a positive control is really kind of about is this experimental setup that I have working as it should oh yes you know this is the proof of principle sample in a sense yes it works it shows what others have seen I can trust the way I have set it up and so the data that I generally that I generate with the new data we have a higher you know we're more likely to trust it as like okay yeah they have a working system and so I think that's also really important for how researchers read papers and data so they're not like sometimes and Peterson Pinovsky also say that like replication starts with trust or at least in the integrative mode well where does that trust come from I think it comes from reading these kind of these built-in samples of replication the way controls are integrated how they work within the experiment and the cross experiments and so this is like this whole web of trust establishing practices that are being performed and so that really gives replication this forward-looking role it's a really I think it's a really important epistemic role that is overlooked and that's why I think replication is underrated in the current debate because we're not taking into account this particular forward-looking role and so that's all I have to say basically I repeat my key claims replication as a dedicated study as a backward-looking function it assesses strength reliability truth or whatever of existing results a replication as an inbuilt replication sample or however you want to whatever terminology you want to use as a forward-looking function it can tell researchers about how much trust to put in newly generated data so it works both ways I think the exclusive focus there's very dominant in the debate about replication crisis about how to improve science the exclusive focus on these dedicated studies means that an important aspect of an important functional role of replication work is overlooked and so yes it's generally underrated in pre-debate that's all I want to say and I hope I'm on the time I think I am great thanks for your attention I'm going to stop sharing thank you Stefan that was really interesting and now we have 12 minutes time for questions so if you have questions just put them in the chat or the Q&A now I also found that one it's when you're the one person between the break and yeah yeah we have one question is what happens in forward-looking replication when a lab attempts to replicate the results in order to build on them is unable to and they move on to another question so they want to build on something they try to replicate it it doesn't work I think it depends a bit on how central that particular piece of information is to their work and plans what stage they are because if it's really central to let's say it's a description of a particular gene sequence and they want to work with mutations they want to study okay if we delete this particular area of the gene does it then not do what was reported but they can't even actually reproduce the wild type activity of that protein for instance which would have to be their positive control say okay this is the level of activity we see then we do this mutation activity goes down or up so if they can't even then reproduce that I mean they would that would trigger a very that would trigger them to investigate and just the things we do when stuff doesn't work and talk to the other those who published it and exchange materials and just figuring out who is going wrong and where the whole messiness of failed replications in other bits I guess if the kind of this you know if it fails well then you might just drop it and you don't work on that so I think if your positive controls fail you just go into a very deep mode of of not trusting your system at first or not trusting the new result it depends a bit on what you use your positive control as well I would say like a brand new result you're probably less kind of trusty if it's something that you've known has been used and kind of reported many many times over you will not trust your PhD student or whoever does the work so I hope that answers the question a bit but not sure let's go to the next question here's one that asks what about differences between different cultures of research and their forms of doing replication yeah very good point I don't really touch on that my focus is firmly I probably had like this claim or something my focus and my thinking in this part comes firmly out of kind of experimental life sciences biochemistry molecular biology you see these with the examples that I'm using like plant biology but kind of genetic work I'm not so I originally I trained in biochemistry so that's kind of the area I know I've never done a research in psychology I do assume they use kind of they have a culture of positive control as well but I wouldn't know so I would restrict my claim to these kind of experimental life sciences and leave it to future investigation how that actually plays out in other fields I would also not say that this form because I do think that replication work should always be thought of in a very localist manner that just certain ways where certain areas of practice of research practice where how you control doesn't really works differently I wouldn't want to make universalist claims what I do think is universal is like this repertoire of trust establishing practices and that's something I think we should focus on much more in our thinking about science is kind of how how do these work these have this can be communication between people it can be specific tools we implement in the experimental setup but again it's kind of where should we put the focus and personally I think the moves towards for instance automation that are happening they might disrupt these networks of trust establishing practices for better or worse but I do think it's that's kind of the level where I want to think in a more universalist manner with this particular bit I would say yes experiment life sciences other cultures let's do more research yes another interesting question I think is can you understand dedicated replication as forward looking as well it tells you how much trust you should have on the finding that is being replicated the distinction between forward and backward looking doesn't seem significant to me old claim okay yeah yeah I do think I'm struggling with the forward backward looking and it's I do and it is the limitations of words I haven't come up with a better way of trying to capture what I'm thinking of of course in a way research always moves forward and doing a dedicated replication studies you know looking at previously published results is an important element of looking forward and of can we trust this and then it's a yeah it's a good point I wouldn't say it's irrelevant it's just bad choice of words maybe or like just not optimized yeah there is just something about there's like there is an element of there's an element of replication of kind of let's examine and let's build trust in the things we already have whereas the moment the motive or the mode that I want to emphasize is replication as a tool to build trust in the newly generated data because I think in the Nozak and Errington 2020 paper there's like the generating new data to diagnose the existing results and I think it works the other way as well existing results help us generate trust in newly emerging data so that's kind of the dynamics that I want to emphasize but yeah I give you the point as forward maybe everyone yeah our next presentation is going to be from Saran Field Saran is assistant professor in the department of pedagogy at the University of Groningen and please go ahead thank you can everyone hear me my screen thanks for having me I'm most happy to to be here it's a pleasure to be able to share a platform with people like this and I always enjoy having a chat about metascience stuff that always makes me happy so I'm talking a little bit about the idea of a community of practice and the conceptualization of this within the science reform movement so I have been only ever been a metascientist I never had one of these original research areas and moved into metascience later on in life or anything like that I started in metascience in 2013 and so I've seen the movement if we can call it that community groups evolve and develop as time has gone by and my own perception of what's going on in my own approach has also become more and more nuanced and has developed over time too and part of what I'm talking about today kind of reflects that a little bit I'll just move these faces away so often in the literature you read about the science reform movement or the open science movement as if it's a monolith as if it's this homogeneous thing and certainly we can think about this as science reform as being a single community of practice now community of practice is kind of what it says on the tin it's a community of people that is drawn together based on practice a joint interest or a joint enterprise and so in that sense science reform has got a joint enterprise the joint enterprise is science reform or improving how we approach science practice there's a shared repertoire shared discourse obviously we talk about things like replication and registered reports as though the most obvious concepts in the world but if you step outside into industry for example a lot of people won't know what the heck you're talking about but we share these discourses these debates, discussions we share practices a lot of us share data if we're in fields that that's appropriate for example a lot of us use registration registered reports and we see that we reify our values so our values you know as a group as a community tend to revolve around doing better science so more transparent science more reproducible science for example more valid science and so we see reifications of these values in the fact that we generate practices that support these values in terms of research practice so I did an ethnography in my PhD because I was kind of just interested in studying the reform movement from an ethnographic perspective it took place over for you I'm sorry about how wordy some of these slides are it always looks worse when you see it in practice but it involved a virtual ethnography so I kind of started my PhD at the precipice of the pandemic and so it very quickly went from in-person field work to Twitter work, online work which was interesting and as a result you know the question for me was really revolving around you know who's the community of people I'm talking about is it a community where's the field in which I'm working as an ethnographer I ended up settling on sort of being interested in just the people who talk about science reform the people who practice science reform stuff the people who post information and science relating to science reform open science reproducible science that kind of thing and so I conducted a virtual ethnography on Twitter on the community of people who appeared at the time on Twitter and I used the theory of the community of practice as an analysis tool as a way of sort of framing and legitimizing some of my observations and I saw a couple of different things that I thought were interesting for one thing membership in the group is continually negotiated so what makes you an open scientist we see discussions about you know well if you don't share all your stuff then you're not an open scientist or people don't tend to say it bluntly but often there's that assumption that if you don't take everything from the buffet then you're not the real deal but you know what makes you a science reform what makes you an open scientist these were things that were discussed you know we see forms of reification of these values we see people really centering preregistration and registered reports and that kind of thing replication as this symposium for example shows these are forms of reification that explicitly and implicitly focus on certain traditions in research for example the positivist traditions tend to be spotlighted and focused on a little bit more than other traditions for example so these reflected values are interesting to kind of observe and we see central versus more peripheral agents obviously for example the center of open science Brian Nozak associated with that you know we see those as being quite central figures entities within the group but there are of course more peripheral people people who are just kind of you know interested but on a peripheral level they don't need to go to all the things like the open science stuff to be interested in the discourse and so that was some of my observations that I made during this ethnography that I think are interesting and to a large extent my research supported the idea of a single community of practice in the district in the time I don't want to run over the concept of a single community of practice was to some extent supported but at the same time it was a little bit less homogeneous than I'd expected as well there was variability and diversity and some interesting factions and cracks in the landscape of the research reform or science reform movement that I saw and so I thought okay I'm interested in the structure of this group so what might a network analysis suggest through the API which used to be open to researchers back in the day using keywords like open science, open research, reproducibility in their bios now obviously this is not a perfect proxy but it is to some degree a way that we can sort of get a sense of who self identifies with this movement I looked at these people and the connections between these people to create a graph of just over 2,000 nodes or 2,000 user accounts and this is what I found this is in technical terms a herible and it's a whole mess of people now the nodes in the middle that are a little bit bigger and a little bit paler they're especially central nodes meaning they have a lot of connections but to a large degree there's a lot of peripheral players and a lot of people who sort of are involved but not to a very central degree and I've listed some of these user names next to these nodes too which is just interesting so I noticed that the density for example it was not a very dense network I expected it to be a little bit more dense and yet only 2% of possible connections had actually been made within the corpus so that's not a very dense corpus despite this though about half the users in the graph are mutual they follow each other which is interesting this tends to indicate a somewhat flat and hierarchy within a community and so this high degree of reciprocity was kind of interesting and I wasn't sure how to sort of interpret that but it was an interesting idea does it make more sense to look at this as separate sub-communities because I thought what's this diversity in these cracks that I'm kind of seeing there are these different approaches and different perspectives on the science reform movement I thought how can we kind of capture that and doesn't that kind of suit what's going on a little bit more so Etienne Wenger who is a social learning theorist he wrote some configurations of communities are too far removed from the scope of engagement of participants too broad, too diverse or too diffuse to be usefully treated as single communities of practice he implies here that it might be better to think of separate smaller communities of practice which is kind of where I'm going now so I took a look at this big network of 2200 nodes and basically tested it for modularity to see how modular it is or to see how many kind of does it make sense to look at little subgroups within the bigger group I don't want to take up too much time what am I going for time I basically I will go a little bit quicker through this because I'm I don't know okay lots of different algorithms are used, two key ones kind of come into play here I'll focus on what I used however I ran the LOVA algorithm for modularity or network detection and I found that it consistently robustly detected the same communities in the same number it varied a little bit but I hung around four or five communities which was interesting because when you look for modularity in a network you want it to be interpretable you want to be able to look at that and go okay what does this actually mean can we kind of understand these little communities or is it just seemingly random groupings of people or is there some kind of thing that links these separate groups you know or the people in these separate groups to one another so this network is indeed somewhat modular Q tends to sit between I think 0.3 and 0.7 so this is a somewhat modular network it's not crazy modular but there is certainly evidence of some kind of subgrouping within this network I'll zoom through that a little bit but I found four main communities and it was really interesting so this community for example is populated with individuals and accounts we're very interested in platforms and the statistical side of the science reform movement this one second community was consisting almost entirely of journals that were associated with life sciences and that kind of thing but also open science and that was just a community that kept coming up it was very interesting to me this seemed to me to consist of people who've kind of been in the original crisis of confidence sort of side of things people were very active in the start of the open science movement if we could call it that and who had a lot to do with generating some of the early discourse surrounding the issues and finally community four was very full of people who are interested in open access libraries that kind of thing infrastructure basically supports open access and finally I broke down the main network into these separate little groupings showing you just where some of the main the biggest or the most prominent user accounts were sitting and a lot of them sit kind of within communities of their own but are also between communities as an interesting observation I don't have time to go into now but that's kind of how the communities were sort of settled within the broader the broader group so there's evidence of of a sub community structure it would seem and the question then is you know does that point to actually separate sub communities or communities of practice and in Wenger's terms a constellation of communities of practice obviously you know we're not detecting networks this is a an algorithm with which partitions the network based on a certain set of parameters so there are technical limitations there but the facts that these communities were very seem to be very easily interpretable was useful and interesting for me so working with a constellation concept this idea of you know of a constellation instead of sub communities there are a lot of reasons for why it makes sense to work with the idea of these sub communities rather than again the monolith of science reform but there are reasons that all of these sub communities kind of share historical roots in this greater cause of science reform there's a lot of overlap some people are groups of members of multiple groups and some kind of you know overlap and up brokers across groups there's often competition for a lot of the same resources despite the fact that there are different approaches and different priorities and needs in separate sub communities there's of course natural diversity there are different groups of communities of practice simply because the science reform movement is comprised of various kinds of different people and various kinds of approaches so is it beneficial recognizing the joint enterprise and the related interconnectivities as well as diversity does it make sense to do that is it useful for the movement itself or movements I think so I think it really can highlight different perspectives so it facilitates intersection of different perspectives which is part of what we're talking about with the whole reason for this symposium considering the constellation approach allows room for plurality and for different perspectives for different needs to be appreciated to be brought to the table and it motivates development of more and different practices so for example when we make space for multiple different epistemological approaches for example we can open the scope for more stuff so I like the idea of Bart's bureaucratic innovation we can broaden that scope you know what are these forms and this standardization that can occur we allow that to be inclusive for more kinds of participation in science reform and I would argue that it develops the effectiveness of the movement that it you know this civilizing process that Bart talked about that it brings that along a little bit we develop and become more nuanced as a group when we take into account the plurality that can be represented in the science reform movement I feel like oh terrible if we ignore a constellation structure we're going to have problems, we're going to inhibit diversity and perspectives obviously that's not a good thing we reduce collaboration opportunities when we have such a great smorgasbord of all these different approaches there are so many cool opportunities for collaboration and cross-fertilization or cross-pollinization which is great there's also less flexibility to adapt you know there are going to be different we're going to be pulling this movement in different ways in the future and allowing for a little bit of different perspectives here and there allows that to also adapt as we go and I think we see potential fragmentation if we try to get everyone to conform to the same ideals and the same priorities I mean then replication do not apply to all disciplines and so if we make that a thing then we're going to exclude people by default for fragmentation so there are loads of different ways to sort of exercise this constellation idea in practice but I'm going to leave it here because I am already out of time and I don't want to be that guy so thank you for your attention and thank you Sarah-Hann we have one question already which is did you pause at a taxonomy flat enumeration of edge types in your network graph? sorry I missed that question what was that question? I'm sorry yeah it was did you pause at a taxonomy flat enumeration of edge types in your network graph? to be honest I don't understand the question sorry are there any other questions for Sarah-Hann? there's a clarification for that question so are nodes just connected or not connected? sorry my audio is really strange can you repeat the question? so are nodes just connected or not connected? are nodes just connected or not connected? is that the question? no so all of the nodes in the network there were kind of basically three different kinds of connection one is no connection at all one was a single connection or followed or following and the third type is that there was mutual connection which was obviously the thickest kind of edge which means that there are two connections between two different nodes yeah okay maybe I can ask a question you hinted on this a little bit in one slide but I was curious whether you could elaborate a little bit on negative consequences of not actually recognising this diversity or people writing about the reform movement without actually specifying what exactly they mean with this term yeah I mean I think there are kind of different ways of looking at this you know I think ultimately if we consider the science reform movement as being a single thing I think we just missed some of the detail that goes into the science reform movement you know obviously we have this big goal with how we do science that's a really important goal but I think if we focus on that and fail to consider the different ways that we can approach that goal and the different strengths and contributions that we can bring to the table I think that we really really miss out on all of the benefits and the progress that can come from that you know when we have to work together and we we end up with really cool interdisciplinary for example collaborations you know where we're pushed into ways that we wouldn't otherwise have been likely to be pushed into you know pre-registration for qualitative research is only somewhat new and if we didn't have the room for that kind of development of that kind of approach then it would be very difficult for qualitative researchers to participate for example so I think the more interest we have in different contributors and different kinds of contribution the further we can go as a movement thank you I think another question could be or it's more like a comment but I think you could maybe say something to it is I think the idea that describing a community as a community must imply a monolith is one worth digging further into to be honest yeah no I don't disagree with that I think that's a great point I think what I mean by that is just that it's often treated in that way so it's not so much what we call it it's more that what that implies for how we discuss it I think if we talk about it as being a monolith we we're allowed then to just ignore the different kinds of contributions and it gives it this solid homogeneous quality that I think just it doesn't really bring out those beautiful colours and different textures that this landscape has so it's not so much that it implies that I think that just in practice what ends up happening is that's how we think about it yeah I think another comment that might be interesting to consider is it would be interesting to link the Twitter network to the published literature to look at similarities and differences between the structure and inequality of discourse on Twitter and in the scientific literature what do you think about that I think so I think that'd be fantastic I mean I think the only issue is that you know our Twitter network is now no longer so many people have moved to different platforms just because of what happened with Twitter so I think unfortunately you know this is an issue with any kind of social media study is that platforms become obsolete that's a massive limitation and so I think it would be really cool to be able to link some kind of network structure to say a structure within literature citations I think that that would yield some very cool results if we could do that another question is if you don't take everything so it's about your comment about the open science buffet and it's about do you have any thoughts on how we can expand the movement by making it easier to enter without having the whole buffet table shoved down your throat and people being turned off because it feels overwhelming yeah and I think that's what I love about the about the buffet idea you know we I think part of it is just expanding how we talk about it and how we think about it you know we're always focusing on replication, no offence to all the fantastic people here who have talked about replication today but that seems to be such a focus such a focus on registered reports and pre-registration I think framing the discussion to be about bigger things and values and just opening the discussion I think already allows room for the buffet idea to take root I think that's really part of it sorry my daughter's here and she's distracting me a little bit I'm so sorry okay I would just say thank you and I think we can now come to our next presenter which is Nicole Nelson and Nicole is associate professor in the department of medical history and bio-essics at the University of Medicine Wisconsin sorry Nicole the stage is yours okay thank you very much and thank you for these great talks so far I think that my slides are way off of where they should be let me just fix that you get a little preview of the talk to come so I think that my talk is going to be very complimentary to a lot of the talks that we have talked about or that we've had so far in that my interest as well is in this idea of epistemic heterogeneity and in thinking through various reforms that we've seen come out of the meta science and open science movements and the impact that they have on heterogeneity in terms of the methods used and the sort of techniques used within the literature now fortunately for me I think because Bart has already introduced you to some of these big team science forms like the many many projects as well as the psych science accelerator I won't spend too much time on this and instead save some of my time later on but what I do want to say about this is a lot of these movements have explicitly styled themselves after other kinds of big science projects in the physical sciences sometimes making direct reference to these projects is the inspiration for how it is that they think new forms of scientific work could look so in addition to the physical sciences as inspirations we see some other kinds of inspirations here for example the camarades group in their multi part study looked towards the clinical sciences as an idea for a new type of work form that they could adopt where they were trying to run pre-clinical animal studies in a form that was very similar to the way that clinical trials are run where they're distributed across a large network of hospitals who all participate using a shared protocol and then they have a data safety and monitoring board that then groups all of these things together so here the exemplar is not physical sciences it's clinical sciences but it has a sort of similar flavor to it although maybe a little bit more distributed another initiative that I think is worth putting on the table here as another example of big team science is the movement towards cloud labs or so-called automated or self-driving labs and these are movements not to necessarily group together the people but to get people using a common platform that shares a little bit more with some of the movements we've seen like for pre-registration and other tools that we're trying to get people to use in this case the idea would be that you have many people using a common suite of laboratory instruments and that makes their data more comparable and easier to sort of interpret across the many people who are using the same platform so in this talk today what I want to think about is whether or not these open science, meta science innovations that we see in terms of work practices might have the implication of reducing heterogeneity in scientific practices. I don't mean this as a sort of conscious intention on anyone's part, what I'm asking about is whether these types of practices make it more difficult to maintain heterogeneity. Now just a few clarifying things this is not an argument that I want to make about the need for exploratory research vis-a-vis confirmatory research I think there's lots of good work already that argues that the meta science movement should not be about eliminating exploratory research, that there is a need for that but that confirmatory research is really sort of the target of intervention on which a lot of meta science techniques or reforms are aiming to operate. So I am also thinking particularly here about confirmatory research and heterogeneity within that space rather than the heterogeneity of exploratory versus confirmatory versus other kinds of research. This is also not an argument about why replication or meta science reforms are not appropriate standards or tools for all fields we have several good arguments in the literature about that including a paper by Bart Penders and colleagues that are arguing that the kinds of reforms being proposed by the meta science movement are not necessarily reforms that are effective or reasonable for all different disciplines and I agree with that but that is not the kind of heterogeneity that I'm interested in today. My target is more in line with what has been called the generalizability crisis or I see that in the chat here somebody has linked to a very recent nature commentary piece talking about academic monocultures as being one potential risk of AI focused solutions and this is the thing that I am interested in talking about today is whether the kinds of innovations that we see taking place in meta science could lead towards a monoculturing of different methods or standards or people or disciplines. So this is what I am going to be focusing on. Now rather than giving you empirical research from the meta science movement itself as Bart has done today what I am going to do is actually use a historical case study to outline the premise for asking this question and it's my view wearing my historian hat because I work kind of half-hats between history and ethnography that a better metaphor for thinking about what collaboratives look like within the meta science movement is not big team science thinking about things like the Large Hadron Collider or something like this that are as Bart pointed out centrally funded have these large administrative structures but instead the example that I want to point to as being comparable and good to think with are the history of model organism communities. These communities as I will show you are a little bit more self-organized, they're more distributed they have flatter hierarchies and they didn't depend largely on central funding and in many ways I think that these communities actually look quite a bit like the types of communities that we see arising today associated with the open science and meta science reform movements. So let me give you a really quick run-through of two communities that I think are exemplary for thinking about what's going on in meta science today and that is the Drosophila community and the C. elegans community in the case of the Drosophila community we really have sort of an almost an accidental collaborative emerge in some ways and that Thomas Hunt Morgan in his original lab was not really intending to grow a giant community of people that were looking at heritability and mapping chromosomes and Drosophila but it happened almost by virtue of the material properties of what he was working with being flies. He had all of these mutants that he was making in the lab and he was trying to map each mutation that he saw, you know, white eyes back to a specific place on the chromosome and at some point had so many more mutations that he actually had time to research these mutations and so started trying to recruit people in to study some of these other mutants that he had come up with and so he began sharing openly the mutants that he had developed as well as requesting researchers to share back the fly mutants that they had developed which led to the creation of this informal network of different strains that you could order from a centralized service run by Morgan's lab. So in the image on the left you see an image of a researcher early 20th century holding the Drosophila information service newsletter which was a list of all of these different fly strains that you could order and then the little milk bottles of flies where you could write in to them and say please send me some eggs for this mutant. So I think in this case we see a lot of features that are quite similar to what we would see aspirationally speaking in open science communities open sharing of resources. We have these informal publications that also included things that looked like preprints where researchers would share work in progress before it was published and we also see this coordinating sort of ethos emerge not from a centralized body but from the network itself. That type of community structure is very similar to what develops in the C. elegans community post World War II all this time with our intention. The core member of this community Sidney Brenner was somebody who was very interested in developing a community around C. elegans as a research organism in order to use it as a model for understanding human nervous system. So it was not an organic process but it was an intentionally directed process where he was trying to recruit people and trying to build up a community that would be structured around principles of openness and sharing. Once again we see things that look like that in the C. elegans newsletter people are sharing works in progress. There's also sharing and disclosure of methods so many of the entries in the C. elegans newsletter are actually describing different techniques for how it is that people do things sharing and disclosing the knowledge and we also see movement of materials and strains between this community. Now these community structures are ones that in the history of science make them sort of famous for the productivity that they managed to achieve in terms of collective projects of trying to map different parts of organisms. So in the case of the fly community, what the fly community was most interested in doing was taking all of these different mutations that they were seeing and using them as a tool to create one of the first maps of a chromosome and understand which traits map to which locus on which chromosomes such that they could build out these collective maps. Because they were sharing materials that they could use materials in common, sharing their work in an early stage and communicating all of this within the group they were able to sort of self-organize so that they didn't have as many overlapping projects so that everybody could kind of take a piece of the chromosome or a couple of mutations and work out what that looked like and then add it back in together in order to create the map of a chromosome. Likewise in the case of the C. elegans community, one of the main projects for this community was to create a complete map of the nervous system of the worm and so here again we see a sort of distribution of resources and labor and focus such that people in this community were each taking a little piece and then collectively donating it back such that this group of people was able to create a complete map of all of the cells within the worm and their sort of destinies in terms of what nerves they formed or not. So I think that this shows us one of the benefits of collective labor and open systems of scientific working is that they can be very efficient when people all get interested in a similar topic and work together to share the resources and you see a sort of self-organizing emerge from these communities. But I think that what I would like to focus on more today is that this had consequences for how life science was done more broadly that they don't necessarily think those model organisms intended or model organism communities intended at all. So these communities were trying to make progress on a very specific question of interest to them. What are the sort of physical substrates of heredity, the chromosomes and can we figure out how they work? How is the nervous system organized and can we figure out how developmentally cells give rise to different parts of the nervous system? But the system that they created in order to do this of sharing information and practices meant that their research community style was incredibly productive and therefore started to out-compete other kinds of styles within some areas of the life sciences. The image that I have for you here on the screen is the study that is published in the journal Genetics that looks at publications in that journal from 1960 to 2010 and it does a very simple analysis of asking for each article published in that journal does that article use an organism which is considered to be a model organism just an official designation that the NIH gives later on in the 90s versus a non-model organism and they use the NIH definition to basically, you know, separate out presence to absence. Is this a model organism or a non-model organism? And what you can see is that from 1960 up to 2010 over the course of 50 years we see a huge change in the way that science is done where organisms like flies like C. elegans, like a raptidopsis become not just in the mix in the community but really the dominant form of actually doing science. So from a 50-50 split within the journal Genetics to something like an 85-15 split where the strong majority of publications in that journal today are publications that are using model organisms. This is something that life scientists have really wanted to sound the alarm on as being an area of concern because it's a type of monoculture that they see as coming with substantial risks. So I've given you two examples of some of that discourse. One is a little piece by life scientist Jessica Volcker who is talking about the difficulties of trying to use organisms that are not rats or mice or flies as a researcher who wants to study more in different organisms because it can be difficult to get funding for these entities because there's so much more work and time involved. And so the productivity outputs that you see can't really compare to people using these standard organisms who now have all of these tools at their disposal, like all of these maps of the nervous system of the worm. They don't have to build these out, they can just use them. And so in terms of trying to ensure heterogeneity within the community, it becomes quite difficult because the more successful the model organisms are, the more costly it is to try and study any other organism. The article that is on the right side of your screen by science journalist Daniel Engler summarizes a bunch of different discussions in the field that are about the particular problems with the models that have risen to prominence as models. Now, remembering that everybody selected these models for their own individual purposes, nobody was necessarily saying hey, mice, that's what absolutely everyone should use. And yet that is what has happened historically in model organisms research, is that a few models ended up becoming the de facto standard. And so we see these founder effects basically of the one thing that people chose at the time, now becoming the standard even though it has flaws. So one of the things that this article discusses, for example, is that Black Six, by far the most popular strain of mice used in biomedical research today, has a number of properties that really make it an outlier on a bunch of common physiological and behavioral tests. Like, for example, it likes to drink alcohol, which is very unusual for rodent species, probably because the alcohol doesn't taste so bad to Black Six mice as opposed to other mice. And so it is a weird model in a lot of ways because it's unlike its rodent counterparts in its proclivity for alcohol as well as a number of its other behavioral properties. Now, within the metascience literature, the contemporary metascience literature, I think that we have discussion of these kinds of problems. The model organism community story gives us a story about how it is that people might focus in on one particular organism, but we also have a bunch of literature that talks about focusing in on one particular method or relying too much on studies conducted in one particular environment. And so, for example, some of the work coming out of Hano-Verbal's group in collaboration with the camarades folks is to try and look at the problems of relying on even well-powered studies that are coming out of only one lab. So in this image that I've taken from one of their plus biology papers here, you can see that this is a visual argument essentially for why it's actually better to have three different labs conducting an experiment under slightly different conditions, because you get a more realistic sense of the distribution of a phenotype than if you had one large powered study but only conducted under one set of circumstances. And so, the question that I want to raise for the discussion is in these collective forms of work that we see emerging associated with open science and meta-science reforms, like psych science accelerator or the multi-part study or the many-baby study, are they going to end up looking more like giant labs that are all conducting things under the same set of protocols, circumstances, etc. Or are they going to look more like this other future where you have many studies being conducted under slightly different conditions which then allows you to do an assessment of variation and heterogeneity. So, I think that there are two fair objections which I agree partially with to thinking about open science or meta-science reforms as encouraging homogeneity. And one is that for many of these reforms, I think people would argue that we're not asking people to standardize at all. All we're asking them to do is document. And so, one example that probably many people know is this one case study coming to us from oncology wherein two labs were using ostensibly the same protocol but getting very different results and they did quite a detailed analysis to figure out what differed between their labs and the difference of those trivial as one group using a shaker to more vigorously agitate their cells versus one using a magnetic stir. And so, the conclusion of the article that they published here in Cell Reports was just to say make sure you report this. They didn't say use this one, not that one. They just said make sure that you report this. But, I think it's also a little bit disingenuous to say that reporting alone doesn't change practices because time and time again when I hear people talk about the impact of reporting checklist, they do intend them to be things that change practices as well as just things that increase transparency. For example the idea that if you're going to submit an article and you have to tick off whether or not you actually blinded or masked your studies you may feel embarrassed about enough about the fact that you have to check, no I didn't. The next time around you adopt that practice. Now in the case of blinding or masking that's not something that we might think of as especially problematic because it doesn't really create a monoculture that's just kind of a bias reducing measure as opposed to the focus on one model organism or one protocol. But I think there are other ways in which we see that meta science innovations do in fact create a default standard which could encourage homogenization. So this is kind of a blurry image here but it's taken from a screenshot of a cloud lab infrastructure software where the idea of the cloud lab is that even for something as simple as washing cells or stirring cells there are a whole number of parameters that when you do this through robotics you have to specify and that in fact it's a great idea to get people to use these platforms because then they are forced to specify all kinds of things that would otherwise go unrecorded in their procedures. The trick here that I want to point to is that this piece of software comes with default settings and so in order to actually change the settings you got to pop open the box and change the parameters manually which I suspect means that more people than not will end up just accepting the default parameters meaning that we get essentially one way of washing cells or at least much less variation in the way that people wash cells such that something like this might not have even come to light because people weren't using different methods of stirring if they were all adopting the same default setting suggested by a common platform. Now the second objection that I can foresee that I think has some merit to and then I have some sympathy for is it perfect standardization as impossible already that the natural world kicks up an absolute ton of variation and so we don't really have to be that worried about too much standardization because there's buckets and buckets of it so if a little bit of homogeneity sort of creeps in whatever fine because the natural world gives us tons and I think that is both true and not true and because I'm pretty much at time I don't have a great amount of time here to give you the detailed version of the story coming out of the mouse model literature on the background effect but suffice it to say that the moral of this story is that in this case what we had was one lab who was creating a mouse model that then was used in a number of tests by a whole host of other labs and because that lab was the only person creating this model it took quite some time to figure out that the specific way that they were making the model had a bunch of variation going on in the background that was actually significant they thought that they were making a model that was missing a gene had a gene knockout that it didn't really matter what was going on elsewhere in the genome because the impact of this knockout would be so profound that turned out not to be the case in fact the other genes were interacting with the knockout in significant ways but because there was only one construct available initially it took time for that to come to light so I will stop there now but I would like to open it up for things that I would love to hear your thoughts on and the first is this so what extent do you think this model organisms case that I have given you here is similar to or different from contemporary open science and metascience practices does this make a better metaphor to think with than things like you know giant physical sciences projects I'm also curious to hear what elements of your work practices in the area that you're practicing if any do you think are becoming more homogenous because they're easier to use or the tools are better understood so even if no one is telling you use this protocol this is the standard you may choose this protocol because it has so much literature attached with it because it's well described because the instruments that you need to use for it are already out there and freely available and in this way I think people do potentially get encouraged to focus in on a few sets of techniques so I'm curious to hear if this resonates with your experiences and then finally I'm curious to hear from the audience what initiatives you think I might better should follow in order to better understand these changing work practices that are part of meta science and open science reforms so there are many many projects as well as you know the psych science accelerator tons of different things so I'm curious about what you think might be interesting empirical sites to follow to look at this process of methods development in order to be able to understand whether or not some of these work practices sort of tip the scale in favor of homogeneity rather than heterogeneity okay I will leave it there thank you very much thank you Nicole we have one question in the who and a which is whether the risks of collective labor could be recast as the benefits of collective modeling shared models yeah I it is it is inarguable I think that this is a style of work that had a lot of benefits to it and I don't want to say that this is not something that was beneficial it very clearly was right you see these communities making a lot of progress that are pretty rudimentary tools for the time on these projects like you know mapping the fate of every cell within C. elegans so there are certainly benefits here the question is about the balance of risk and benefits like at what point do these tools become so good that other tools are no longer competitive and that is really the question that I am interested in is if we see uneven distribution of meta science open science reforms such that we have one suite of tools that becomes really easily accessible to people while other tools are lagging behind does that then create these founder effects where we see people making rapid progress with only one set of tools that may have some intrinsic flaw that might get discovered only 10-20 years down the line yeah another question is standardization has occurred a lot in molecular biology take sequencing plus mid-preparations etc there are kids for so much now that is standardized and enabled innovation on top of that this doesn't preclude someone going back and sequencing by hand for example but it has led to advances not possible before how do you think what you discussed relates to this type of technique homogenization yeah that's a great question and in fact I years ago tried to launch a project on exactly this looking at DNA sequencing technologies because one of the things that I was seeing in laboratories is that I was working with some laboratories who would for example describe themselves as an Illumina shop and that they were in no way going to adopt new sequencing technologies because they had invested all of this time and effort in being able to understand the particular quirks and foibles of the Illumina system and so what I'm not totally clear on is the extent to which if you become an Illumina shop you miss out on opportunities that other sequencing platforms might actually give you what is clear to me is that people are not using multiple sequencing platforms that is something that is too high in investment so I think the question that we need to be able to ask is what is lost basically when this homogenization actually happens it seems to happen and so do we lose something meaningful when people become an Illumina shop and don't experiment with a variety of other tools in order to understand how the failing of one tool might supplement the failing of another such that we get a more complete set of information yes again thank you Niko and now we are coming to our final presentation today which is held by Bernhard Wieser Bernhard is associate professor in the department of business at the University of Idaho please go ahead right well thank you everyone for being here and for sticking around onto the last presentation so today I want to talk to you a little bit about the dogmas and the evidential standards in metascience and why this is a good time for a makeover for metascience first I'll give you a prologue by Farah Albin to set the stage for my talk as you may know Farah Albin stood firmly against scientific indoctrination and cautioned us against many dogmatic traps of scientism and this is a called how to defend society against science he says about the cautious dogmatic approach to learning and generating knowledge says this will slow us down no doubt but are we supposed to charge ahead simply because some people tell us that they have found an explanation for all the misery and an excellent way out of it and concludes this essay with the following statement with a hat tip to Socrates the hardest task needs the lightest hand or else it's completion will not lead to freedom but to a tyranny much worse than the one it replaces so I will draw some parallels between Farah Albin's warning about charging the head with a heavy hand and the risk of making things worse than before while carrying the conversation forward to our current post-replication crisis context. Given the topic I will assume that most of you most of the symposium audience are very familiar with the background of replication crisis so I will save you a historical overview but I just want to emphasize that science reform and metascience emerged as a backdrop of poor scientific practices unreliable results and a massive resistance to change and as such the predominant focus of science reform so far has been on instigating motivating and activating social change so in this first scientific utopia paper in 2012 by Nozak Anbar Anant they put it as follows we argue that the barriers to these improvements are not technical or financial they are social. The barriers to change are a combination of inertia uncertainty about alternative publication models and the existence of groups invested in the inefficiencies of the present system. Our ultimate goal is to improve scientific research efficiency by bringing scientific communication practices closer to scientific values and perhaps out of this urge to tackle these social barriers first and foremost certain priorities emerged in the reform agenda and I used an earlier version of this slide that I will just show you three years ago and the previous critical metascience symposium by COS and I'll just show you a revised version of it with much of the content intact because it's still relevant. Science reform has so far prioritized certain issues such as larger sample sizes preventing and discouraging or drawing attention to at least p-hacking large scale adoption of pre-registration and maybe resistance report as well can be counted here. Replication studies and increasing research transparency openness encouraging open methods as well and things certain things got de-prioritized in turn and I do not mean that intentionally rather this was a de-facto de-prioritization just by way of other things dominating the core reform agenda. So among these things that remained out of focus are included sample quality model billing and selection issues and model mis-specification or specification approaches the systematic exploration theory development and inference quality and formalism in meta methods. So from my perspective this picture translates into a prioritization of procedures over the actual content of meta science. So such a prioritization eventually gave rise to some cultural artifacts and heuristics that's going to be the focus on my talk today. For example, meta science has become motivated by and need to activate the scientific community to change rather than to understand scientific phenomena better. So like the meta focus became much more important. Meta scientific practice has focused on research and maybe larger studies to validate or debunk results rather than slow careful medical scientific progress and understanding. And the scientific community has begun consuming meta science in a particular way too maybe in a way to confirm their belief that this change needs to happen rather than trying to understand what we actually aim to understand in terms of again the development of science. And some of these heuristics have become so normalized as to cause a threat of turning into dogmas preventing us from freely and rigorously thinking about science. So let's take a very quick look at some case examples to illustrate what I mean by science being relegated to a secondary role. So the first case example that I'm going to present is about replications and the heuristic that I refer to here is the more the merrier heuristic. The assumption is that every replication is a good replication and we need more of them and sometimes even indiscriminately so. Take the infamous now infamous BEM studies the ESD studies of 2011 that contributed a lot to the instigation of the replication crisis. There was an initial failed replication of these BEM studies in 2012. Then a while later another set of two studies with very large sample sizes followed this time in 2021 this time in German version of the priming task and then two large sample studies also reported failed replications in 2022 and finally a multi-stage mega study was published in 2023 with similar findings again of the failed replication. Over the same period scientists also critiqued BEM studies from multiple fronts so the theory was found to be insufficient to be properly tested in these experiments. The study designs were found to be noisy and subject to experimented degrees of freedom and the results were found to be not robust to different types of alternative analyses basically. So based on my work on the theoretical foundations of replication and how they work I always get questions whenever I see a new BEM replication or other similar replications so why do we keep running such flawed studies or re-running such flawed studies and when we will decide that it's enough and we'll stop running further replications and maybe most importantly to me is what is it that we have learned about BSD from all of these replications over time that we could not have said a priori or based on these other critical evaluations and another way of looking at this is if we were really interested in possibly understanding ESB and we were kind of entertaining the idea that it could be actually real how would we start studying it would we just perform another BEM replication or would we go elsewhere so whether a replication experiment is what will serve our epistemic goals in a given setting is something that we should consider more carefully before performing our next study so these considerations are exacerbated typically in mega studies like larger multi-site replications so my case, second case example is going to come from these growing number and popularity of multi-site replications as previous speakers also mentioned the one example would be the menelab studies that's the one that I'm going to use as well so here the heuristic that is at play is the bigger the better heuristic so one of the if we look at the example of menelab studies some of the things that we find here is that while trying to replicate a finding across multiple sites new sources of heterogeneity are introduced at every site due to seemingly trivial differences in conditions procedures and materials regardless of the extreme levels of standardization that is trying to be implemented the menelab study designs also got extremely complex due to multiple labs varying sources of error and restricted randomizations at multiple levels so we end up with things like split plot designs with several nested factors and due to this complexity meta-study analyses are typically performed under mis-specified models kind of ignoring some of these complications of the study design and sometimes such as in this particular menelab for study even a higher level inference is being thought such as testing a meta hypothesis by assigning different sites to different treatment conditions and as a result as we find out in this 2023 paper Arkambu's bus my colleague and I non-exactness of replications will exacerbate errors in estimates in larger models such as those used in testing meta hypothesis the reason is that larger models often require nuisance parameters also to be estimated in addition to parameters of interest even if we take the best approach to inference the meta-exactness of replication will be reflected on multiple estimates resulting in undesirable estimators the often times bigger means more complicated and that means worse inference while creating the opposite impression of higher credibility so once again I wonder what do we learn substantively about the phenomenon under study when we design such large studies without systematically controlling for the sources of heterogeneity or optimizing the experimental design for better inference moving away from replications for a bit my third case example is going to perhaps unsurprisingly to some of you is going to be pre-registration and the heuristic that I will refer to here is that if it exists it's good so it's becoming increasing the common to use to register study label as some sort of quality signal or signal of rigor I just basically did a very simple google search, google scholar search for the phrase pre-register study and after 2020 it turned it's returned over 3,000 hits just for the term itself so the phrase became very popular it's being used in a lot of studies nowadays to refer to you know like the very generically the pre-registration status but the more important thing is the heuristic so I'll use a specific example of this heuristic that is presented in the 2023 nature human behavior paper which is a systematic review paper and the authors here evaluate systematic evidence for happiness strategies and they exclusively look at pre-register studies so I'll quote the authors here and they say because pre-registration should substantially reduce the likelihood of false positives we treat the pre-registration as an important component of evidentiary value and further add we provide citations to small non-pre-registered experiments in supplementary information but we do not discuss these studies below due to their relatively low evidentiary value so basically the authors use the mere existence of pre-registration as a discriminating evidence for research quality and rigor without even needing to scrutinize the content of pre-registration so Barton and I have written a very short paper letter on this and there's more that we address there and of course this is just a single example but further evidence has been recently provided by Moinslet where he posted this preprint a few weeks back basically and he analyzed open peer review data and found evidence that editors and reviewers do not mention, access or compare pre-registrations against the article during the review process so that's the blue part of the bars that you see there showing the proportion of mentioning, accessing or comparing pre-registrations and maybe the lowest one here is less than 3.5% of reviewers and editors compared the article to the pre-registration but it says about this state of affairs if a study is pre-registered but nobody checks it during review doesn't mean anything what is the value of a pre-registered article when the pre-registration was not actually evaluated during the review process very little I would argue if reviewers do not examine the pre-registration plant will the casual reader current data suggest no again instead of helping move science forward such heuristics are likely to be detrimental to progress by biasing the scientific record in ways we cannot quite even comprehend or anticipate yet so my last case example will concern the whole of science reform and our overall evaluation of it regarding the evidence for effectiveness of science reform basically and I will refer to this heuristic as any evidence is better than no evidence so to which of course this kind of statement I would say evidence for what that's the relevant question so in this recent major human behavior article that came out at the end of 2023 it's called high replicability of newly discovered social behavioral findings is achievable the authors report high replicability of some new results in social and behavioral sciences and conclude that this is due to reforms such as excuse me registration large samples and transparency so this paper and its preprints have already been cited about at least in Google Scholar 48 times and Jova Coleman and I wrote a commentary on this paper which is so being evaluated by the journal and in that commentary we discussed how the meta study design does not actually support any causal conclusions before the reforms were not subjected to experimental treatment so there is no control condition within the study and that the reported replicable metric is very idiosyncratic and not necessarily robust as you compare other replicable metric some of them reported in the supplementary information of the paper moreover I think the reported replicable rate is not actually that different from some of the other reports other metrics reported in the literature I gave you three examples here but none of these papers were conditioned on reforms unlike the current Nature Human Behavior paper also the effects on the study in this paper were selected on statistical significance causing selection bias or if you think about it within the study there is internal publication bias but most importantly from my perspective is that all of the significant findings in these replications are referred to as discovery without really any discussion of why that should be a discovery or what has been learned in these domains of study over these set of studies why would that be the preferred approach to discovery along with most of the papers that cite the higher replicable to paper the Center for Open Science itself has disseminated these findings with the following phrasing the reforms are working evidence that the credibility of social behavioral sciences can be improved so some other citations refer to these results as conclusive evidence that reforms work without necessarily specifying what work means there with regards to scientific progress this kind of attitude to me about jumping at any kind of evidence in support of science reform is over eager although that premature and careless because we cannot do the evidence is there we just need to read it correctly and evaluate it accordingly and maybe it sometimes even comes across to me as although desperate like are we in such rush to confirm this conclusion that we can't even take the time to actually look at the quality of evidence or the relevance of evidence so based on these these case studies are just to illustrate my point there are many other examples that I could not fit in this part presentation but in conclusion in some ways I think we have started putting the meta before the science and there is a growing tendency to do meta science for meta science and to follow or endorse the reforms for reforms sake rather than taking the time to articulate how when or why they would contribute to scientific progress and there is also some evidence for use and misuse of procedural signals as evidence for quality without further scrutiny there is also evidence for lowering our evidential standards and then evaluating pro-reform results as people who are pro-reform and finally something that fire oven warned us again some evidence for staff congratulation and hubris as if we have figured it finds out and everything is better already and I know if we ask this question to anyone like do you think this way and people would deny it but sometimes actions actually speak louder and they may actually type our more honest opinions about this as well and that's kind of what I'm kind of trying to read now that the reform movement has gained momentum and there is growing adoption of many of these procedures I think we can afford to take a step back and slow down slow down to of the meta makeover that I suggest here is not just the critique of meta science it's about actually meta science in critically it's about it's an endorsement of more critical deliberate mindful practice and consumption of meta science so first I say let's keep our eye on the price we got in this in the name of science so let that be our guiding principle and let's evaluate everything based on how they contribute to our science in concrete terms so what have we learned how have we made progress in advancing knowledge and how do our meta studies how are our meta studies going to help us epistemically do they help us reach our goals we need to replace our short term focus on existence not existence type of results to scientific understanding by a long term genuine painstaking exploration and we need to be more intentional about what we study and why and maybe divert our focus to stuff we are actually curious about and we aim to continually explore for example as opposed to out of a desire to debunk things that we don't actually believe in and we need to design better studies first and then better meta studies which we can properly specify we must raise our own evidential standards both when testing hypotheses we want to be true and when evaluating them and collectively we need to keep questioning the quality and rigor heuristics that we have established we need to keep questioning and resist meta science dogmas and we need to openly invite criticism like this symposium but not only that genuinely engage with it and use it to improve the way we have been doing things and the way we are going to continue doing things lastly we must actively cultivate intellectual humility and accept that we could be wrong accept that we may have made wrong decisions along the way and resist hubris so I will conclude with a quote from Claude Shannon in 1956 he wrote a piece called The Bandwagon in response to the large scale quick and mindless adoption of information theory by scientists in meta disciplines so this was the concluding paragraph of this short paper and I'd like to invite you to mentally replace information theory that I italicized here with meta science and scientific reform and communication theory at the end with science so Shannon says we must keep our own house in first class order the subject of information theory has certainly been sold if not oversold we should not throw our attention to the business of research and development at the highest scientific plane we can maintain research rather than exposition is the key is the keynote and our critical threshold should be raised authors should submit only their best effort and these only after careful criticism by themselves and their colleagues a few first rate research papers are preferable to a large number that are poorly conceived or helped finished the latter are no credit to their writers and a waste of time to their readers only by maintaining a thoroughly scientific attitude can we achieve real progress in communication theory and consolidate our present position thank you I'll conclude with this and invite any questions thank you Bernhard this was really interesting I don't really have a question yet but what I do have is an interesting comment that I think a reaction would be quite interesting because it addresses the issue of communities actually engaging with each other so it's a longer comment but I will only read the end out it's the general point that insufficient attention is given to theorizing is well taken but I don't think this is in a position to miss a logical reform framing miss a logical reform as un-rigorous science is unhelpful and unnecessarily antagonizing I kind of would have expected that kind of comment well this is a critical meta science symposium and that's what I meant to bring to the table nowhere I think here I said that the whole reform is just this I am just trying to caution people in that kind of heuristic representation of reforms not thinking about their mindless implementation of them it doesn't mean that everything reform has done is this so this is just a warning that this is possible and it's happening somewhere and it's increasing in frequency as well in terms of you know like not reading pre-registration for example or automatically doing replication I am a meta scientist myself so yeah I know like there is a lot of good work I believe that my work is good yeah another comment is formulated as a question is while I understand the criticism here I wonder if this truly reflects how the majority have adopted open science practices yes there are many projects and approaches but most simply now to register they pre-print and maybe make data and R code open is this really a runway train a runway train interesting comment no I don't know I don't know whether this is the mainstream trend I don't know to what extent this represents how much or how much of meta science or scientific reform is being implemented I think this is a fair warning regardless regardless of how frequently it is being hated this way I think it applies it doesn't necessarily have to apply to everyone for it to be relevant that's my point thank you Beno and now we are going to have our panel discussions panel discussion where you can ask our panel everything that comes to mind related to reform movements or anything that you would like to ask about presentations but I think we could also quickly start by asking our presenters whether they want to ask another presenter anything because I saw during the presentation sometimes there was also hand up by our presenters so maybe we can give them a chance to ask each other questions and spark the discussion yeah I would like to open that I think by talking about something that came to mind during many of the presentations that I heard essentially all of them but was never made really explicit namely how do we think that these trends in meta science whether that is bureaucratization homogenization using heuristics instead of thinking well etc how all of that is related to notions of de-skilling like these molecular biology kits that Tim typed about in the comments and Nicole referred to that was already then they were introduced in the context of debates of de-skilling if you can't sort of make your reagents yourself and do all the work yourself really worthy of being a molecular biologist is this these tools that we are being handed these procedures that are being developed and optimized is that de-skilling science in some way or another do I need to answer it myself or do I need to point at somebody oh Thomas wants to go okay I'll have a go I think this is something that I've seen crop up a lot in the debates particularly around when it comes to workload there seems to be almost two schools of thought which are we need to train researchers to do open science and how to do pre-registrations how to share data how to share materials how to write reproducible R code and then they're kind of at the same time there's this perspective of saying we shouldn't expect researchers to be specialist software engineers at the same time as being experts in cancer or things like that and actually we should move to team-based research to be the team that has these skills I've kind of never really seen that sort of resolved really about you know should probably the answer is not to move one of the extremes entirely but it just seems to me to kind of be a bit of attention here about saying like well it seems a bit odd to kind of try and do both things at once like either we try and specialize people to have certain open science skills or we should try and develop like more rounded scientists who can do everything and I think probably the most interesting perspective I've read on this sort of comes from the higher academic the higher education literature where there's this really cool paper I'll put it in the chat by a guy called Bruce McFarlane who basically argues that even the term even what it means to do research in the kind of current climate of sitting down planning a research project that you do and then you write up is the skilling a broader perspective of what used to be like an academic where you know research isn't just something that like an individual project that you sit down and do it's a kind of intellectual engagement with a whole topic and involve sort of sitting around and discussing with other academic things and reading stuff and that in itself is all research rather than a sort of doing a specific project but yeah I'd be interested to hear some of the people's perspectives on that idea of like yeah should we be training specialists to do all these different things or should we be kind of upskilling academics in all these different areas and how do you kind of resolve that when you're trying to plan reforms Yes Stefan? Yeah just quickly I don't have an answer to the question you just posed I was just thinking with because there's a comment as well with AI and de-skilling and all that and there's this interesting narrative which actually I think is fed by a lot of the replication crisis narrative activism about you know if you look at any termoscientific, termo-fisher whatever all these companies they also increase your reproducibility and blah blah blah by bringing automation and often then there's this sentence like immediately you're worried that the human goes out and all this but then it's always like it's mere manual labor that will be taken over the researcher will still do the real science so it's kind of goes back to the point that Tom just made as well kind of where is the real science being done and what kind of skill or what does it entail but like I'm always curious about this notion of mere manual labor because a lot happens when I just do the pipiting and kind of I don't know it's just kind of there's these narratives of what we can lose and what not what skills are relevant and what skills are just like can be blended out and just it replied to Bart's original question I'm awesome maybe I'm not worried about de-skilling like when I did my PhD in biochemistry we started using more and more kits I didn't think it was getting easier or I lost skills or kind of but right now I'm more maybe worried about out-skilling because again with automation people are saying well we don't need to use normal pipettes anymore we can transfer liquids in different way and kind of it moves into an area where it's different kind of processes maybe being used that we cannot you know it's it goes into a pakeness and all that kind of where can we still tap into the process and kind of establish trust and reliability and all this I'm not sure but that's just one of the worries maybe Yes, Nicole? Yeah I might argue that we could reframe the de-skilling problem and maybe think of it in two different ways and I'm going to be historical case study guy today but one good analogy that we could draw I think the evidence-based medicine movement and there's some really nice work showing there that this tension between sort of like skilling and de-skilling is there as well and is maybe a fundamental tension or tension in reform movements so for example in EBM the reform of the curriculum for med school was all focused on people becoming critical consumers of the literature and figuring out how to use these sort of critical reasoning skills but in terms of the EBM reforms in practice it was a much more technocratic here evidence-based guidelines and you must follow them all and so that may be a skill that's just inherent to reform movements or a tension that's inherent to reform movements it may be also a thing that we could think of as a problem with technocratic approaches which maybe relates to what Berna is talking about that when you have a sort of technocratic prescriptive way of here's how you need to do science then that is maybe de-skilling is maybe a good word for it I think yeah my thoughts I think I have a question in case no one else has one it is kind of an aspect that hasn't been addressed yet so much but you read a little bit about it when you read critical perspectives on the reform movement and especially on meta science that people tend to have a certain background when they are in meta science or when they are in science of science and I think an interesting question would be how much do you think the background of the actual researchers who do meta science or are involved in these reform movements influence the appropriateness of the reforms for types of research or research cultures I'll have a go so one of the things that we hear most in the context of that type of discussion is the over-reliance on quantitative studies when it comes to meta science and if you take meta science as critical evaluations of research practices in the broadest sense if that's the definition then meta science has been around for centuries and most of it has not been quantitative so essentially the quantitative element of it is an innovation post world war 2 innovation for the most part in that sense and sure so if that is the epistemic culture you come from yourself with the moral programs and ideal articulations of science embedded in that program and as a result also in you then that means that you will bring a variety of ideals with you into your meta science project that is in itself actually not that much of a problem as long as alternative articulations also exist but they don't or they do actually but they exist in different spaces so we do have history of science, sociology of science, philosophy of science and we have meta science we speak to different audiences in different voices and these meetings are specifically orchestrated to get them all together and then for the moment we do and then we sort of articulate the plan that we should do this more often and make this sort of structural engagement but it doesn't stick at least not until now not so much I hope it does and I hope it changes maybe as the moment that it changes but probably not and I think that that is something that we need to well consider as one of the biases if we want to use that term anyway one of the biases that meta science essentially introduces in its project to reduce all sorts of other biases thank you what do the others think I certainly agree with Bard on that one it's hard to tell how much of that is our own biases as qualitative researchers towards the value of what qualitative research can bring but I think it is the case that a trap that sometimes meta scientists can fall into is assuming that they understand what the experience of doing this work is like for scientists who are not like them because meta scientists having been born out of a particular scientific field have experienced a spectrum of scientific training there's sometimes a problem wherein they use their own experiences to interpolate out to the rest of the community and what their experiences have been like without actually doing the qualitative research to see if that is indeed the lived experience of other people in the field I think that meta science is getting a lot better about that but I think there's a lot more room for systematically collecting information about what science is like for differently positioned people such that we can have a better understanding of how a particular reformer intervention would fall on people who are living and working in different scientific careers essentially Yeah, I agree with that as well and I think sort of a good example is that a lot of people, well myself included, my background in psychology I got into becoming aware of open research through the whole discourse of replication crisis and I think that a lot of people especially potentially well known people within the movement had similar backgrounds coming from psychology and quantitative replication crisis and a lot of the reforms like pre-registration were perfect for dealing with the issues that we encountered there, right, you know, PHAQ quantitative studies but I've definitely been in discussions where people have kind of almost taken perspective like oh, we need to, you know, spread the gospel of open research to other disciplines, you know, because it's been useful for us to like improve the way that we do science and psychology you know, so let's go and see how pre-registrations can help with like history and then it always kind of strikes me as like, it's almost like you have a solution and you're looking for a problem which doesn't always seem the kind of right way around to do it. Yeah, I think an interesting question maybe also for everyone is a question that was posed for Bart before is can you speak to the relationship between mechanical objectivity and I guess also this process of automation and what he called this post human epistemology with the aim of the very widely asked for fair principles and also with this machine action I think this is also probably related to the issue of that machine learning is getting more and more involved into the actual science processes maybe Bart you can start since the question was first posed to you Yes, yeah so this whole idea of mechanical objectivity of course sort of speaks to the idea that we are the problem and if you get rid of us then at least you solve that problem and maybe not all problems of the world but at least that one and well let me disclose first which is a sort of a positionality statement that I do not subscribe to mechanical objectivity I firmly believe that it is unattainable in the sense that you cannot remove humans or traces of humans from the process so if you remove the humans you are still left with the traces of the humans and whether that is in a racist AI or in some sort of imperfect machine so that means that in many situations it might not be problematic to strive for something resembling mechanical objectivity as long as we all recognize that it is essentially unattainable just like the actual truth is unattainable and you can strive for it the same goes for mechanical objectivity but the fact that we do try means that we also have sort of developed tools and instruments and concepts and things that should get us there and some are and yeah playing that part really well but none of them actually do it because they all essentially rely on human actors to work even AI does so the question is well can you speak more about the relationship well sure I would say that the relationship is fictional but very powerful and we all have to somehow relate to that fiction all the time even if we don't really want to yes Nicole yeah so I thought I would chime in a little on the question of mechanical objectivity since I've been thinking about this a lot as someone who is about to become the embedded ethnographer in a large-scale automation project to create a self-driving lab at Carnegie Mellon and I think that what we could say from the very extensive literature on ethnography of the bench sciences at least and this may not hold true for other areas of science is that automation tends to displace rather than replace tacit knowledge and so over and over again what we see is that when you automate a process rather than the tacit knowledge being the manual skill of how to pipette well then it becomes the tacit knowledge about for example which well plates you can use with the robot so that it doesn't go bonkers or some things like this so it's not that we don't reduce the human intervention or the human knowledge in the system we just sort of substituted out for other kinds of human knowledge and other kinds of errors now one of the things that I think is maybe really worth thinking about in terms of mechanical objectivity and trusted data is where that tacit knowledge is displaced to so for example in emerald cloud labs which is the parent private company of this project that's being implemented at CMU one of the things that I find quite interesting is that initially all of the staff that were actually working within the robotic lab setup carrying around the plates and test wells were basically biology undergrads kind of entral level job for somebody who had some training in biology but interestingly the company found them to be too interventionist trying to fix too many problems that they wanted people to behave more like the robots and ended up hiring former Amazon warehouse workers basically saying like I'll pay you double to you know instead of put product A in box B put like test tube A in slot B and so in this case we have a displacement of the tacit knowledge that's being gained into a workforce that is kind of fundamentally a different one in that they're not exactly part of the same sort of system of scientific norms and socialization and that to me is something interesting to think about in terms of how it's changing scientific work and trust because the movement of this tacit knowledge around through automation means that it's being displaced into places where we don't know as much about what the norms are governing these kinds of work communities and that I think is something really worth studying. I think we also have a question about math proficiency here. I just wanted to ask it. Oh okay then go ahead. Yeah it's so there's one comment about I do worry about the level of knowledge of statistics among scientists but also among meta-scientists so you think that the level of mass among biologists is a serious barrier to progress. I have some things to say about this but I don't want to be the talky-talky head so if anyone wants to go first go for it. I can go first here. I do more math oriented theoretical work myself so I don't know like to what extent it's a generalizable problem. It is a problem in some kinds of meta-science for sure. It is a problem when quantitative evidence or quantitative flames are presented to researchers to follow certain statistical procedures for example and when people who are presenting these procedures are not doing it either a good job or maybe dumb it down assuming that the audience will not get it so you leave a lot of nuance and a lot of auto-precision in terms of the claims themselves. So we just like in 2021 we published a paper on that the case for scientific reform in case for formal methodology in scientific reform and that was basically the focus because there's a lot of the I think reform initiatives addressed quantitative research and how data was analyzed you know P-hacking and everything and a lot of that is on point but some of the recommendations were not they were kind of being a little bit over eager to again instigate action some of the edges were kind of rounded to make some conclusions seem a little bit more appealing to the audiences but I think we need rigor in statistics as well and we need to understand it to be able to give people recommendations as to how to use it and that was kind of the point in my priorities of science reform versus what was the prioritized part was about that like some of the things that got prioritized was in the field of scientific practice you know what are our temples like what kind of models are we assuming are these models do we satisfy the assumptions of these models even in a lot of these multi-fights replications but the experimental design is never formalized so you try to figure out what the model is to put it in a mathematical form and it's sometimes impossible to gain that from the details of the experiment so yeah I think there's some some missing sophistication in that regard maybe it's on purpose that you know like assuming that the audience does not have the same sophistication so we need to make things simple maybe it's just lacking I don't know I don't have evidence in either way so I'll say my view on this is there is certainly a role for education and there are certainly people who are misusing statistics right like those things are real and true things about the world at the same time I worry that there's a bit of a deficit model assumption underlying this question that basically the only reason that people use statistics differently than you might use them is because they are lacking the knowledge and if we just fill them up with knowledge then they will behave in the same way that you would want them to behave and I think that that sort of undersells the problem both because we have lots of evidence that that kind of deficit model thing also because it doesn't acknowledge other things that might be contributing to people's usage of statistics so for example this little chat that Bart and Tim and I were having in you know sidebar in the chat here about the extent to which different disciplines think of things in terms of presence absence this is one of those I think underlying assumptions where it's the assumption and not the knowledge of statistics necessarily that's the problem so if you assume basically when you're studying a developmental system that if you have seen it's going to result in a presence absence phenotype it has it or it doesn't that fundamentally changes the way that you analyze the system then if you consider it to be something that's going to be drawn from a distribution that may be an assumption which is better in some cases than others and I think that I think happens a lot and this is something that some you know biologists have wanted to talk about to me almost in a guilty way is that they may have started their career studying phenotypes that really were better understood as binary presence but as they moved on towards studying more and more complex phenomenon it would have been more appropriate for them to switch over to thinking of something as you know drawn from a distribution of phenotypes but they never really made that transition and so in that case maybe the problem is partially knowledge you know they didn't have the statistical skills because they didn't need them initially but it's also partially about unseating that assumption that they have to approach their thinking about the data in a different way because they were using a frame of thinking that was appropriate for one question but now not appropriate for another Yes, Thomas your microphone I was going to ask a completely different question to Nicole but if anyone wants to pop in about that previous one please do so first okay but if not so my question for Nicole is about the platforms right that researchers are using and you talked a lot about how you know they become they homogenize the methods that certain disciplines use I was going to ask if any of your research is also on like how those platforms grow and survive and what the implications of that the sort of platform capitalism sort of area so obviously a lot of them are not capitalist products designed to make money but the bigger they get the more resources, the more people use them the more resources that are needed to sustain them and eventually that there was some chat before where people were sort of talking about ties into burners stuff about incentives to prove that metascience is working does it sort of get to a point where a platform becomes so big and so well used that it kind of needs to keep being used and promoted just to sustain so many people's jobs and is there kind of incentives there that are problematic Interesting questions and I would be interested to hear other people's experiences with kind of platforms that are relevant for their areas of study too but what I will say based on my sort of background studying this model organism approach of the stuff that I'm doing now on automated labs it is absolutely the case that we see instances in which technologies become too big to fail not in my mind as much for the economic reasons although they certainly are there but because of the sort of continuity of data lock in effect problems where essentially that if you switch to a platform which is too different then it becomes really difficult to draw a line between existing bodies of research and new bodies of research so when I was speaking before about some of the people who described themselves as Illumina shops basically one of their motivations for not wanting to change was not just the start-up cost of tacit knowledge but also the fact that they had samples that they'd analyzed over a period of decades that they wanted to be in as much sense as possible consistently analyzed so that they could go back and compare these sort of sequential analyses over time so I think it's not just economic self-interest which might generate these sort of effects where we lock into a particular platform it's also this general desire for intraoperability that I think maybe a lot of what we think about is kind of truth in science is actually really more so intraoperability like can other people work with this result can other people get this result almost like you know can I plug my USB-C thing in when I travel in Europe type thing and those intraoperability things if we think of science problems in that way we start to see them as these problems that are really analogous to a lot of other standard setters in the world and so that's how I would do it maybe if I just because I have to hand up if I'm just on the same point but on a much smaller scale I thought of that when before we came to this why is variation not necessarily factored in but just like absent presence in phenotypes and all that it's literally also not just to make like science work on a larger scale but literally just your own experiment like you need to generate when you work especially now I'm thinking of biology living systems all you do is stabilizing what is not stable you're trying to create some sort of stability that you have from time A where you start your experiment you still have the same thing at time B because otherwise you can't say anything you don't have to continue to you need to make reliable claims even within your own experiment so you're constantly you're constantly trying to create and maintain some I think that's maybe not it's it's real state you are distorting in order to enable certain forms of knowledge generation and that happens even at the smallest level even if there's no commercial interest it's just kind of inherent to what you need to eliminate variation and so on so it's just you can't get past some things but like with the platforms just to come back to that point as well I think and this is Nicole I have you know you will have a lot more direct insight into that but I'm just it's not even I'm not sure where the cloud lab for instance is to write sites no it's one interesting site and one relevant site but then there are a lot of it's maybe the cloud itself not the cloud lab like you know regular laboratories where a lot of work is now fed through the cloud and where certain automated systems are or fed through specific cloud systems that creates new platforms and monopolies that are hard to break and that are obviously integrated with commercial interest as well and kind of you know you see collaborations already with like Gilson pipettes to have a cloud based you know certain automation help but they actually have a deal with quiet and you know the kids so you automatically get those protocols uploaded onto your Gilson pipette pipette man whatever they call it like your apps that you can use and so on so there's like the way diversity of experimental practices is reshaped happens in the very micro space within each laboratory maybe and that's just something to look at as well I think it's interesting thoughts and I think you know I might reframe slightly the idea of it happening within micro spaces because your comments really nicely point out how capitalism capitalism not just platform capitalism as Thomas was pointing to really impacts the way that standards proliferate in that it's not just certainly open science and meta science and advocates who are doing things that encourage homogeneity right obviously corporations instrument manufacturers are also trying to encourage homogeneity where homogeneity is everyone uses their product and so I want to you know be clear that we should put those kinds of courses alongside other things like reforms that are you know probably not even as potent as some of these other forces like okay if you buy our kaijin kit you get a suite of tools that become easy for you to use such that you become a kaijin shop like that's a real phenomenon as well yeah we also have a question in the chat that is related to this it's kind of like the question of whether interoperability will degrade interdisciplinarity into a mapping of meta standards are the participants actually able to speak about a limitation of the platform I would love to hear from Alexander more about what he means on that so that we could add his voice into the discussion I don't know if that's a thing yeah that's a limitation of the platform it's only they can only comment does anyone have any thoughts on this or well maybe I'll kick it off that I would love to hear where people chime in in terms of what they think this means I think that you know if we were to draw an analogy from thinking about the tech world more obvious in the tech world is that nature is not going to enforce a standard for you that there are many potential ways to design a technology and there needs to be some human intervention on deciding what the standard actually is whereas maybe one of the ways that we could think about the reproducibility crisis is an assumption that the combination of the constraints of nature plus the constraints of you know statistical methodology we're supposed to keep us you know sort of bound enough that there wouldn't be so much variation and now what we see is that that's really not the case that both nature and stats are malleable enough that we can get a number of different answers and so it becomes more appropriate than to think about it in this way that it's like all right of the many human standards that we've developed for bringing a phenomenon into being which one do we choose as to whether or not this will turn into a standard or many standards I sincerely hope it won't turn into a standard I mean the point of my talk today was to try and argue against that world I think sometimes it can be hard to defend heterogeneity because heterogeneity feels like messiness it feels like everybody doing their own selfish weird little thing but there are real benefits actually to accidental heterogeneity that isn't to say that we couldn't manage heterogeneity by deliberately including it in projects but that will always be limited to our assumptions about what variables we should be changing and part of the value of unmanaged heterogeneity is that it allows for a bunch of circumstances to kick up variation that we might not otherwise see because we haven't thought to look there so I would really like to argue for some lack of standardization and not just deliberate heterogeneization because I think that the unmanaged heterogeneity plays an important role in science that really ties into sort of many discussions about researcher degrees of freedom which is sort of the designated label in meta science where other reflective disciplines use actually quite different labels such as professional maneuverable spaces or just essentially just judgment and I would say that when you close down every route towards or shut down every researcher degree of freedom you are deleting potentially valuable heterogeneity from the system, from emerging and in that sense limiting yes but removing no I think another interesting question we have that touches on at least three of the presentations we had is in my experience of 12 years the tasks required for open science are much more than 20 to 30 minutes per administrative intervention I estimate it adds days if not weeks to certain projects why do I accept that burden because it raises my own confidence in my findings and my persuasive powers, thoughts? I feel like time is question oh good, good no I just wanted to say I think that a lot of practices do add time and this goes back to a comment I made in the chat earlier I also think that we need to reframe the issue such that we're talking about the distribution of time spent the distribution of labour so for example with registered reports yes it feels like they take heaps more time but I think if you spend time on a really good pre-registration getting it methodologically reviewed then collecting your data and getting your principal acceptance and collecting your data you spend so much less time later on trying to get it accepted at a journal and have all these reviewers saying oh you should add this analysis collect this more data do this other different thing than you did I think you decrease the possibility of more labour later on with some of these practices so I think yes there is sometimes this extra labour feeling but I don't think that has to be the case and I think if we reframe the discussion about where the labour actually takes place and how you distribute that labour of the research life cycle I think there's a good argument for a lot of these things not necessarily taking more time but taking more time upfront Yes Nicole I think you also want to say something No I was just going to encourage Thomas to add in his two senses that seems so relevant to his presentation Oh okay then Thomas would you like to add something? Yeah I mean I definitely think you know as much as I've said we need to think about the costs of the time costs of open research I do prescribe to the benefits of it and having just done a registered report myself actually yeah the process of doing that did create a better outcome than if I had just done it a normal standard sort of piece of research I do think that the costs part of what it comes down to is that when I was saying about the kind of the workload and the expectations of researchers is if we are and this speaks to broader things like incentives is that if we're still assessed on the number of publications rather than the quality of them then anything that takes longer if we've only got a limited amount of time to do research then if it takes longer then we are going to get fewer publications out of that time that we have available so the shift to trying to evaluate research based on quality over quantity is a big thing one thing I'm sort of unsure about though is that whether quantity in the end when you're trying to assess researchers is always going to come out as the ultimate thing that is assessed on because at the moment you might say oh well one really quality and this is particularly like like Bernard was saying kind of use things like pre-registration is like a heuristic for quality because then you're looking at researchers outputs and you say oh well you know they've got two publications which are fully open pre-registered open data open materials and that's like better than the sort of researchers five completely closed non-open outputs but then there's another person who has five pre-registered open top quality papers like they've got more than that other person with two so you kind of I don't know how the quality quantity of quality still ends up trumping quality and I'm kind of unsure how we get out of that kind of rat race really yep thank you I think we have to come to a close already over time I just would like to tell everyone that this event will be put on YouTube so if you want to revisit it or share it with anyone it will be openly available and before we end I also would like to thank everyone for attending and participating I would like to thank the Center of Open Science for providing the platform for this really interesting event and I would also especially like to thank our six presenters who made the event possible in the first place