 Well, let's get started then. Ladies and gentlemen, attendees, I welcome you to this webinar of the major science 2023 conference. My name is Alex Bouter, and I'm your chair during this webinar. We have 90 minutes for the topic replication and replicability in the humanities. And most of you will know that the awareness of replication started in the biomedical and the social sciences. And it became quite a big thing once it became clear that replication was not always successful. And by now we know that, on average, it only works the replication with the same findings in 50% of the studies. That was labeled as a replication crisis. And the humanities still did their own thing. And then a few years ago, Rick Bales, one of our speakers today, and you're surely, we wrote a few comments suggesting that replicability and humanities might be interesting and important and maybe desirable as well. And that started an whole evidence of comments and interesting discussions and debates and everything. And from that, we now started ourselves and also identified a bunch of interesting studies of replication in humanities. And that is a theme of today. It's mainly about history and art history, but it stands for the humanity as a whole, maybe, and maybe not. We will debate that. We will reflect on the challenges each speaker will present for 12 minutes sharp. Then we have six minutes for Q&A after each presentation. And during the Q&A, it would be convenient for me, as the moderator, to have your questions in the chat noted down. And you can start noting down from the moment the speaker starts to speak. You don't need to wait until the presentation has ended. And please do it as clear and brief as you can, asking your questions. And then I try to take care that at least some of them get attention during the Q&A section. Without further ado, I'd like to pose a thought. Our first speaker is Rick Bales. I mentioned him already. He will speak about the possibility and the desirability of replications in humanities. He will, in that sense, introduce the webinar topic a little bit more brief in depth than I did briefly. And he will also comment on a few of the objections that came in our directions while we were launching this debate in a way. And we learned a lot from them. And I hope that you will learn a lot from that as well this afternoon. So without any further comments, Rick, I propose the floor is yours. Please share your screen. Thank you, Lex. Let me try to do that straight away. Let's see. There we go. Can you all see it now? Does it work? Yeah? OK, good. Yes, I can. So my way of start, then, 12 minutes or so about the very idea, the possibility, and also desirability or value of replication in the humanities. So briefly, by way of introduction, as Lex already pointed out, there's been a replication crisis, as some people noted back in the 2010s, 2012, particularly in the biomedical sciences and social sciences, also social psychology, trust crisis, some people also called it, where upon an attempt to replicate original studies, only a certain percentage of them was successful in doing so. So and sometimes the failure to successfully replicate was as high as 65, 70, 75, sometimes even 80%. And that was, of course, considered a problem. It is widely thought that there's enormous value to replication, whether or not that is actually acknowledged in practice. So among the values of replication are the fact that replication ate scientific progress because replicated results are more likely to be true. It prevents the waste of various resources, such as financial resources, time resources, energy, and so on, since non-replicable results are less likely to be true. Results that are not replicable are, if they are applied, more likely to cause harm to society, to individuals, to nature, to animals, and the like. And one final consideration is the fact that if too many results turn out not to be replicable, that might, particularly in the long run, erode trust in science or scholarship, academic scholarship among academics, but also among the larger public. So a few years ago, there was a symposium by the Royal Society of Science in the Netherlands and I think Lex kindly ensured that I would be one of the speakers, namely on replication in humanities. And I hadn't really thought about that, but when I first did, it struck me that given the nature of replication, it should be something possible and perhaps desirable in humanities. So I defended that on that occasion and then Lex and I joined forces, as he said, and we have defended that now on several occasions. And it has met with some agreement and with some slight disagreement, so to say. And we believe that there are important chances and opportunities to pursue replication in the humanities. So that is what I'm going to briefly speak about today. First on definitions, by replication study, I mean an independent repetition of an earlier study, answering the same study question and usually by using the same or similar methods under the same or similar circumstances. And by independent, we mean that it is not dependent, for instance, on the original results or the original agenda. Replicability is those joint features of a study that make replication possible in the first place. So replicability does not entail replication, but replication does entail replicability. Successful replication holds or occurs when a replication study delivers the same results or sufficiently similar results as the original study. And straight away, we see that that is a matter of controversy, of course, in certain cases. By the humanities, we mean a wide variety of different disciplines loosely related to one another, things such as anthropology, archeology, classics, history, linguistics, literary study, philosophy, the study of arts in theology. I'm not going to debate a definition today, but that gives you an impression of what we have in mind. Replication is something that comes in degrees, so it can be more or less successful. The study can be more or less replicable, so more or less lend itself well to replication. And finally, we often distinguish three kinds of replication. So reproduction, direct replication, and conceptual replication. And what we mean by these terms is when a reproduction takes place, there's a re-analysis of the original, so the existing data sets. Whereas with the direct replication, new data are collected, but using the same study protocols or the same method, for instance. And finally, with the conceptual replication, new data are collected with a somewhat modified study protocol, so that can involve new methods that were not employed in the original study. So paradigmatic replication involves independent researchers, even though that's changing nowadays, so more and more often the original researchers are actually on board with the new application study. With new data, with the original study protocol, and it explains the similarities and differences between the original study on the one hand and the replication study on the other. Now, as Lex already pointed out, our view that replication is something possible and desirable in the humanity has met with quite a few different kinds of objections. So let us address some of those objections here. Some would say that in the first place, we can't pursue replication in the humanities because the objects in the humanities are unique. In comparison with atoms or viruses or economic trends that you can see in multiple instances. So for instance, take the novel to the lighthouse by Virginia Woolf, or the Russian Revolution in 1917, there was only one of them. So I see that this is a bit of a red herring. The issue is not whether there's a unique object or whether we can derive, for instance, more data about that new object or reanalyze the original data. So unicity is, I don't think, a relevant issue. And in fact, it holds something similar, it holds for the sciences. So there's only one space time that's a unique object but we can still study it and do a replication study of the original study. There's a wide variety of methods. That's the second objection in the humanities, fair point, but of course the same holds for the sciences and in fact for specific sciences, like the biomedical sciences. Some others like Sarah de Rijk and Bart Benders and Britt Holbrook have pointed out that there is meaning beyond truth in the humanities. So often what matters is not so much. Whether a statement or a proposition is true but what the meaning of an object is or the meaning of a particular text or a statue. And I think that is actually often right. So indeed we are interested in meaning. But so two thoughts. On the one hand, you might think, well, there can be truths about the meaning of a particular text, for instance. So in analyzing a text of Shakespeare, we can debate what the text means or what the meaning of the text is or what the author meant to say. But that is an important consideration. And on the other hand, even if there are multiple meanings to a particular text, it could still be the case that multiple meanings are all equally valid, but some other meanings are not valid. And then studies as well as replication studies can be used to rule out certain meanings. So that is another consideration. A fourth objection that we've encountered is that this smells of scientism, if you like. So the sciences, the natural sciences in particular, impose their standards on the humanities, some sort of colonialism really in academia. And in fact, that has of course happened at times in the past, but we don't think that this is a particular instance of it. And in fact, we have argued against scientism on several other occasions. It's just that we believe that there is an intellectual disorder atom that has been pursued in the sciences that we believe is equally valuable in humanities. In the same way as pursuing knowledge and understanding is valuable both in the sciences and in humanities. Another objection is that humans are interactive entities. So it's different from atoms, for instance, or other inert objects, for instance. And we believe that is right, but of course the same holds for fields like biomedicine or sociology, where we study human beings in interaction with their environment, in interaction with other human beings, and sometimes even in interaction with the researcher. So we don't think that is a particularly convincing objection. A sixth objection is maybe more interesting. So I think a lot of scholars in humanities rightly point out that positionality of the researcher matters to the research. So the researchers values and principles and background ideas or paradigm or school or connoisseurship. So one may have studied renaissance statues in a particular area in Italy for decades, right? And thereby acquire a certain connoisseurship that others don't have. And this is indeed a challenging issue. And in fact, later on today, we will say a couple of more things about it. And the final objection is in a way the opposite. So some people have said, well, we are already doing this and we've been doing it for ages, for centuries, really. There's nothing new about doing replication in humanities. So why act as if this is something new and worthwhile? And we reply that this is, contrary to what some claim, indeed something new in the sense that it's a more systematic and more formalized way of pursuing replication. For instance, by way of pre-registration or making biases explicit that might play a role in research and ways to counter them. So something roughly similar has been going on but doing it in this more systematic way is of a value that the more informal way does not exemplify. All right, what then are the main arguments for the possibility of replication in humanities? I believe there are two categories of arguments for that. The first one is a priori. And the point here is that if you understand what a replication is, so an independent repetition of an earlier study using say the same data or new data. And if you understand what the humanities do, for instance, history or art history or linguistics, it just follows from the nature of those phenomena, that replication is possible, right? Because the humanities are concerned with statements, original findings, with data sets that could be re-analyzed or very often there's the possibility of collecting further data. So it just follows from what they are that it is something possible. And for those who aren't straight away convinced by the APAR argument, there are lots of examples actually out there in the humanities that show that something similar is going on again, even though not in this formalized way. So a lot of scholars have pointed out that the famous church father and theologian Augustine was heavily influenced by agnosticism, a particular school in philosophy. And as time progressed, people drew in new data, new writings, and time and again, this idea was confirmed. Or take the Rosetta Stone that was used to decipher hieroglyphic, then we find the Dematric, the hieroglyphic and ancient Greek texts. And here they would use different bodies of texts with languages that they already knew to decipher hieroglyphic. So that was in a way already a replication. And finally, in establishing that this painting for the unknown, the Sanse de Mont-Majous by Van Gogh, was indeed a Van Gogh. They used different methods. So they analyzed colors, the themes, the material, the wood, the diaries, and so on and time. And again, it was confirmed that it looked like this was indeed a Van Gogh. Finally, then, the value of replication in humanity. So we believe it's not just possible. It's actually also valuable, desirable, worthwhile pursuing. That has to do with the trustworthiness of the original findings. So for the same reason as in the sciences, the trustworthiness of the original finding actually increases once it is confirmed by reanalyzing the data sets, even more so when new data come in and even more so when a revised study protocol leads to the same or similar conclusions. But apart from the trustworthiness of the original findings, if you have your hesitations about that, we believe there are other gains out there for replication in humanities. So Rachel and Hans and Sherlock will share something about that in a minute, but it has given a lot of insight into the methods, or if you like, protocol of the original studies by seeing what authors have written down by conversing with authors, by exploring options that maybe they didn't use. So it has helped us enormously to better understand how historical research or in some case, specifically art historical research is carried out. It has also given us insight into the background assumptions for particular paradigms or schools that particular authors work in. It helps us to make future studies more transparent. So we learn what to pay attention to in setting up a study and writing it down. And finally, it teaches us things about the role value and meaning and connoisseurship. So the role of the scholar in both the original study and in potential replication studies. So again, not just the trustworthiness of the original findings, but there are these other gains as well. All right, that's it. Lex, I hand back the floor to you. Okay, thank you very much Rick for this interesting and self-provoking introduction. The thing is now open for discussion, but my problem I have is I try to change a lot of settings. You should be able to type questions in both the chat and the Q&A, the question module, I believe, but there is a technician on the background. Please check these things and enable it for all the participants. And while people are working on that and trying to type in their questions, I have one question for you, Rick. I know that you haven't claimed that it's always possible and it's always desirable within the humanities. But I'm still wondering, is it only for humanities where they have data, empirical data, or is replication also something to consider in non-empirical corners of the humanities maybe like your own trade philosophy, but there are different examples as well, of course. Well, what are your current views on that? Yeah, great question. So I had only 12 minutes. If I had 25, like it's in my package to share a few thoughts on that one. So the way I think I prefer to go about is to maybe first focus on the easy cases or relatively easy cases, which are the apostriori cases, so the empirical cases, the empirical studies where we do indeed collect data. Actually, I personally think that something similar holds for apriori studies such as in logic or epistemology or ethics where we don't usually work with data or collecting data or data sets, but with basic beliefs, primary intuitions, and then try to see what we can derive from that or try to establish an equilibrium in ethics, for instance. But I mean, at face value, you can of course reanalyze the way the original author or authors reasoned to how they came from those intuitions to their findings or conclusions, right? You can maybe draw in intuitions or principles that they didn't consider, so further data. So stretching the term a little bit, but analogous mechanisms can be found there. So, but I thought, you know, let's tackle the relatively easy cases first. Yeah, well, that's a sensible solution and I'm sorry for the difficult question. Now, the questions are starting flowing in, which is great. The first one by Harrison Derla, and it is a brief, so it might be a longer name, is when you can't rely on methods that test counterarguments, like randomized clinical trials, why should the focus be on replicating the reasoning rather than recklessly tracking the arguments and counterarguments? Right, so, let's see. Right, so we don't have randomized clinical trials, of course, in the humanities, but I do think that we can, we do have arguments and counterarguments, and there are ways to test those arguments. So one easy way to test an argument is to check it for its logical validity. Sometimes it's just a fallacy, for instance, right? So that's one way to rebut an argument. So I think what we should do is check the original reasoning, check the sources, where certain sources overlooked, how do we get from those sources, or at least the data from those sources to those findings, certain biases play a role, so all sorts of things can be reassessed, not just arguments and counterarguments. So the way it works is a bit different, so we don't use randomized controlled trials. So the way it works is a bit different, but the basic mechanisms are often pretty much the same. Yeah, final question, and then we move on, is a question that might sound familiar to you. P Singh, starting with an H, says, really appreciate your point about the scientism critique, and my few words matter. To that end, can you suggest ways to develop a more inclusive language when talking about open research that is sensitive to and considering the diversity of research disciplines and methods? So that's not natural scientists dictating what open research and meta research is. Well, that's a sensitivity we heard before, of course. Right, yeah. What did you think? Yeah, yeah, so we really need to think carefully about semantics here. Open science, open research, open scholarship, transparent scholarship, maybe. In certain languages, like German or Dutch, Wiesenschopf, Wiesenschopf is a term that catches all academic activity rather than just the sciences. So certain languages might be better fitted for this. So yeah, I agree. I agree, we need to look at this. Really important. It's always important to point this sensitive view, which is understandable. Out people in the science are usually not aware to this. They believe that the sciences include the humanities or forget about the humanities existing anyway. So let's move on. Thank you, Rick again. You're welcome. Our next speaker is Charlotte. Charlotte is trying to replicate a study on the attribution of two paintings depicting Rembrandt. She will now explain how she's doing that and which challenges she has to navigate and is still over her ears in navigating them. Charlotte, the foreshores. Thank you for this introduction. Today I will tell you about the replicating Rembrandt study. And this is a study that came out of the project that Rick just mentioned. And a study that explores the strengths and limitations of replication in the humanities by aiming to execute a replication study in the field of art history. And today I will first go into the art history level of the study. I will tell you a little bit more about the attribution ingredients, how an attribution generally works and about the original study we are trying to replicate here. Then I will go into the meta level of the study. Talk about definitions we use or approach for replication and some of the challenges we save. So let us see. We have a painting and we want to attribute it. When working on attribution in art history, you have sort of two pillars you work with. One is that you read the objects at hand. The other is that you read the context of these objects. And in these two pillars, there are different ingredients you look at and you study more thoroughly. In the object, it can be in this case when it's the painting, the support, the paint layers, material, the way the paint is handled, if there are signatures and whether the painting was changed over time. In the context, you can look at the provenance of the painting. You can look at, aside from that, the ufra that you suspect this artwork belongs to. By working this way, you have triangulation in your study. So different avenues you follow to come to your conclusion of an argument. And this is really a matter of integration of different aspects you study. And together, they lead you to insert an attribution. And attributions are these days, always teamwork. You need different kinds of specialisms within art history to really come to an attribution with a good foundation. The study that we're replicating here concerns these two paintings. These are two portraits of the young Rembrandt. The painting at the left is part of the collection of the malice house in The Hague. And for years and years and years, it was considered to be one of the finest examples of early self-portraiture by Rembrandt. The painting at the right is part of the collection of the Germanisches National Museum in Nuremberg and was considered to be a study of copy of the painting at the left, a contemporary one. Until the 1990s, when the malice house started technical research into the painting at the left and with a technique called infrared photography, they were able to look under the paint layers of this painting. And to everyone's surprise, an underdrawing was found. Here you see the image of these infrared reflectography. And here I indicate with blue arrows some of the lines which indicate this underdrawing. Well, underdrawing, upon that moment, were not found in the Oeuvre of Rembrandt yet. And in addition to that, the style of this underdrawing was considered not to be very Rembrandt-esque. And underdrawings furthermore are often used to transfer one image of one medium to another panel in this instance. These were all arguments that started specialists to doubt this attribution. Because of that, a gathering of the two paintings was organized and the paintings were studied by a group of experts. In the end, all experts that attended this expert meeting concluded that the attribution actually should be the other way around. The painting in Amalekai is a study of copy. And one in Nuremberg, hidden in the depot, actually was considered to be the painting made by Rembrandt himself. So apart from the underdrawing, and we go back to the ingredients now, the underdrawing is part of the paint layers, which is part of the argumentation. But the experts also look at the handling of the paint. There was an unexpected signature found on the Nuremberg version. And they looked at the oeuvre and thought the Nuremberg painting fitted more in the oeuvre of Rembrandt than the version in the Amalekai. So this was one of the, well, first example of a museum being the Amalekai de-attributing its own Rembrandt. And as you can imagine, it caused some attention at the time of this reversed attribution that was published. So we talked about original research. Now let's go into the meta level. The questions that were central at the original research was whether the de-hate version was painted by Rembrandt or not, whether the Nuremberg version was painted by Rembrandt or not, and what was the sequence of these paintings? Does one relay on the other or the other way around? And by, for answering this question, technical research was done into the both versions and a live comparison by experts was organized. And these are things that we're replicating at the moment. We're doing this replication according to the definition of replication as put forward by Pills and Bouter in 2018, being an independent repetition of an earlier study answering the same study question and using the same or similar methods under the same or similar circumstances. A reproduction being the reanalysis of the data sets that you already have, a direct replication being a collection of new data with original protocol and a conceptual replication being a collection of new data with a modified protocol. Since all these terms like data protocol, maybe protocol this, but are different terminology that are not traditionally used in art historical research or research in humanities, we first made a definition of the different terms that are used in this study to see what we actually think of when we talk about a source in this kind of research. And when we talk about the methods or data, and in this way, you really can make an exact or a view of what you're actually doing and in which of the categories of replication you're researching. So the source can be a painting in this instance or a colorful source, the methods can be a technical method or a literature study and the data can be a technical data, but also, for example, the description of naked eye observation. So here I have a summary again of these different kinds of replication. For this study, we decided to opt for a reproduction and a conceptual replication. We left the direct replication out because of times and means, but also because in this instance, it would mean that we would make all, use all the equipment, old infrared equipment, for example, to reproduce the old images and it would only filter out little mistakes made in these measurements. And we thought it was more interesting to go for a reproduction and a conceptual replication, furthermore so because if a reproduction would be successful, that would enable us to reflect more on the outcomes of the conceptual replication. This overview, you see the different stadiums of the reproduction and replication in this research and summarizing the different aspects you can or cannot change when working in a certain area. The reproduction of the study is entailing, looking at the paintings again with a naked eye, with infrared, with x-rays and dendro concerns the dating of the root, which we will repeat. Our conceptual replication is far more broader and that is because in the past 20 years, lots of techniques have been improved and there are way more opportunities to make an analysis of these both paintings. So the improvement of technical tools that you use is really another layer that is added to this replication case study. And as you move further away from the reproduction, at a certain point, you can ask yourself, when are we just carrying out an independent new study? And this border is kind of a gray area, really. So I wanted to give you a quick impression of how the research looks like. So we're in the middle of conducting the replication at this moment. And here you see the confirmation study of the mouse house with the painting on the table. And at the right, the image of the wood with a little measuring tape is showing a technique that is used to date the wood, so the panel, where the painting is made on. And here you see the backside of the painting under a microscope. This research is conducted in close collaboration with Sabrina Meloni, the conservator of the mouse house. So there are lots of challenges we face along the way by trying to replicate this study. One of the first challenges was to reconstruct your original study. The study was not set up according to the more traditional way you might write a protocol and conductive study, but it was caused by a coincidental discovery that was made during the preparation of an exhibition. So there were no protocols that we could rely on while replicating. Aside from that, there were practical and theoretical feasibility issues of replication of certain aspects of the study. Furthermore, we have the question, are we really replicating in the humanities here? As you've seen, this study is partially very technical. So I really see it as a hybrid case study and working in two kinds of realm. And another example is the example dealing with bias in replication. Since you're very aware of the original study and the outcomes of the study, because you need to study those before you can design your replication, you have to be very careful in the way you approach it. Another layer of dealing with bias in our study design was that we both do a reproduction and a conceptual replication which had, because of which we had to be very sharp on the order in which we conducted the research. So the research is done by a very interdisciplinary team of specialists, which also contains one of the original researchers, which is who is involved in the research. And the same goes for our advisory committee, which is represented by different specialists. And one of them, Professor Jürgen Radem, was also one of the original researchers of the study. For his study, we work with different partners which you can see in this slide. And you can read more about it in our pre-registration of the study, which is also quite a new thing in art history to pre-register a study, and a blog we wrote on the website for the Center of Open Science. I thank you very much for your attention and feel free to reach out if you have any questions or comments on this talk. Well, thank you very much, Charlotte, for this presenting this interesting study. It's one of the studies I can tell easily to all my friends and they all like it. So it's a very nice and neat example of replication in humanities. It's great. There are not yet questions in the chat, so that gives me the privilege to have my own question up front, Charlotte. Well, there is this interesting exhibition of Johannes Vermeer in the Rijksmuseum currently in Amsterdam. And it was a rumor that there were some fake paintings there as well. My question to you is, the methodology you have developed now for the conceptual replication, including the consensus meeting, would that be usable to study the Vermeer paintings as well, or would that be different when we talk about a different painting? Well, we're of course still in the middle of conducting the study, so we really have to see first if it really works what we try to do here. That's the right answer, of course, but now a bit deeper, please. Yes, so aside from that, I think that the certain steps we're undertaking and really the protocol we're following could be applicable on different attribution questions. So not in artistry research in general, but certain kinds of attributions. And a Vermeer attribution is one example of that you could use this methodology. Absolutely. Okay, thank you so much, Schavelts. Well, we have a program with many interesting highlights, so we have to move on to the next highlight. That is a double study done by Hans van Eegen and Rachel Peer. They do a replication on the same original study in a completely different way. We have two small slots for each of them and Rachel will then do the Q&A afterwards. And the circumstances that Hans was unable to attend due for unforeseen reasons, but he made a nice video clip for us. And if everything goes well, Rick is now about to start a video clip and then Rachel can take over with her story and the PowerPoint presentation. Sorry that I can't be here. I had some issue at home, but I hope that I can get the gist of what I've been doing this way as well. So we don't have much time. So I will give the briefest of introductions to the study we've been replicating and then some three lessons we've learned so far by protecting the replication. Okay, so the study we attempted to replicate was a study conducted by John Hadley Brooks in the early nineties. He investigated the thesis whether English Puritanism provided a fertile soil for the acceptance of science or practical science, that English Puritans would be more inclined to accept science or be it practical science and their non-Puritan Anglican counterparts at the same time. His conclusion is rather negative that there's no lack of evidence from the sources he studied that Puritanisms were more inclined to accept science than non-Puritans. So what I've been doing, I've read through his study thoroughly and I went through all of his sources again and see whether the conclusions he draw from them, they hold water, whether they are warranted. And I also included some new sources on Puritanism mainly from New England to assess his overall conclusion that Puritanism did not really contribute to acceptivity of science. And what did we find out? What one of the key, some of the key things we found out so far. One thing is like lack of documentation, lack of transparency as you may call it. Like Brooke is a thorough historian. He documents a lot or he explains a lot. He has specific bibliographical essay explaining his sources and where you can find more sources. But still it's not enough. I think there's no clear methodology section where he lays out his research protocol. There's no real documentation of why and how he selected his sources. There's no real documentation about certain background assumptions or certain schools of thought that sort of guide his historical endeavors. So in order to be fully replicable or have more replication studies, historical research will need to be thought with documentation. Like historians are not in the habit of adding like thorough methodology sections or really explaining how they came to certain conclusions, what sources they use. It's more narrative. It's also just not given often. So this is one step they will have to undertake in order to have more replicable research. Another key thing which makes history probably different from other disciplines like the social sciences. It's like the importance of expertise or connoisseurship. Like we consulted with John Hadley, Brooke, the original author. And he mentions part of his background assumptions or background thoughts are like inherited from like decades working on historical sources on reading stuff concerning the scientific evolution, which is somewhere in the back of his head. And he don't really have access to that. And he can't really explain how that all comes to the fore in his historiographical research. It's very hard to track his expertise. Also in the connoisseurship, historians who had decades of experience, they have easier ways of drawing conclusions or noting stuff that the rookies or beginners, like we don't really have, but this connoisseurship puts an extra burden on replicability. I mentioned other steps were documented. It's often not very clear how conclusions were drawn. So this is something that can be fixed more easily. And one thing which we did is like consulted the original author, which can give you some more information on these issues, but not always. Of course, these problems come to the fore in the social and biomedical sciences to some extent as well. There is expertise, there is connoisseurship, but it's often easier when you work with harder quantitative data. Lesson two, also something more specific for history or the humanities in general, the hermeneutics, like how can you draw a conclusion from a source and are multiple conclusions warranted? Like I mentioned, I sometimes draw different conclusions than the rook does, considering the same source. Here, for example, on the ratio of Puritans in the Royal Society, Brooke mainly focuses on the majority of non-Puritans. I mainly focus on a specific subclass or the proportion of Puritans given the proportionality of Puritans within the wider population. That sort of warrants different conclusions. Strays is a question, can those two conclusions exist alongside each other? Can both of them be warranted or is one more warranted than the other? What does it mean when you're replicating a study? Seems like the conclusion isn't all, isn't the only important thing, it also matters how you interpret sources and whether your conclusion is compatible with the original conclusion or not. A third lesson, like it's not even already clear what hypothesis is tested, like this issue of Puritanism and its relation to science, like I analyzed like different sources on the hypothesis and there are at least three interpretations, there are probably more. And all of them seem to be at work in Brooke's chapter in his original study. So historians don't always find it easy to pick one or stick to one or like interpret all sources in the light of one, they're more like holistic, more broader, which make it difficult to really see whether the original study is reliable or valid. These issues are also more of an issue in the humanities probably than they are in other scientific disciplines. These are the main lessons I really would like to bring to the fore here. There's a lot more to say on this study on how we did the replication as well. You can always contact me with questions if you want to know more. Sadly, I can't answer questions now or enter the discussion. So I hope you enjoyed the rest of the event and please let me know if you want to have more information. Thank you. Good job. Okay, great. I'm happy to continue the presentation. Just like Hans is doing the direct replication on the historical chapter written by John Hadley Brooke. I have been appointed to do the conceptual replication. So unlike Charlotte, we don't have to do both of these things. We're each focusing on a separate area of this replication process. And it's been a real pleasure working with the group. Sometimes I stay right from the start that the group was very open in embracing me as part of the group and that I wasn't a converted replications person from the start. I told them upfront, I'm a agnostic. It's an interesting subject. I'm intrigued and I want to learn more about it. And they said, great, that's what this is all about. And so I still remain, I'm sticking with my agnostic status. We'll see what happens at the end, but it's actually fun being the questioner and the staying undecided while hearing all these different perspectives. So I really have enjoyed it. One of the first questions we have is what is a conceptual replication in terms of what does the slightly revised protocol mean in terms of a historical study? So in our case, we were looking at, if Hans was doing the same thing as Brooke in terms of looking at Protestant and Catholic responses to Copernican thought, my study is looking at Jewish responses to Copernican thought. Is that just new data or is that a revised protocol? It actually also reminds me of the question that came up earlier when people said, how can we make this more user-friendly for humanity scholars? Does the word protocol even reverberate with humanity scholars, with historians? And that's something that really speaks to me of making this, having a lot of conversations with historians, making this come from the inside. So for this study, our group has decided that looking at a different religious tradition, in this case, Judaism is the definition of a revised protocol and not just new data. So that is the contours of my study. I did break it down into two sub-questions. So we're looking at both whether the Jewish historians and their source, they happen to be Jewish historians, but the historians who looked at the Jewish material, if they corroborate this qualification of the link that had previously been drawn between Protestants and greater openness to the novel scientific ideas and in this test case, Copernican thought. And then there was a second sub-question that we focused on as Hans was saying, this chapter is very rich, so there are lots of different aspects to it. But we really identified the second sub-question being that social factors are very important. And again, that it can't be deterministic in terms of the denomination that just the theology is the only thing that led to it. There are other issues at play here. So as I mentioned, the questions abound. The truth is, I thought I might take this opportunity that the first time I am presenting, I think Hans had the chance of presenting with the historians from the University in Utrecht. Pym is here today, thank you. We looked at your white paper very carefully because we found it to be the only replication, in your case a reproduction in the discipline of history. So that's very meaningful for us that we have someone to have conversations with. So there's really a very different stance that your project takes to our project in arguing only for reproductions and not for the relevance of direct or conceptual replications in history. And one of your arguments for that here, I put a quote up on the screen. If historiography is a deliberately subjective discipline in which the person and the background of the scholar, which obviously has been mentioned a number of times already, this is gonna be layers, everyone's touching on the same subjects, rather than a hindrance are a necessary precondition for acquiring knowledge than no two scholars, except perhaps identical twins can be expected to produce the same outcomes. And we're very curious and interested in this idea of what kind of replication can be expected or not. We believe it relates to the different categorizations that have already begun to be outlined by philosophers of science like Leoneli in 2018, where she identifies different categories of where expertise and observation is reproducible and where it's not. And we felt that your study took a particular stand on it and we weren't sure that we agreed with that particular stance, but we think that that's really great for further conversation and meeting it out further. There have been other schemes that have also been looking at these issues of where history fits in in terms of participant observation or describing other ways of doing work in different disciplines. And we feel this is a very fruitful area to continue the conversation. So just to give you a little bit about what I'm doing with the Jewish material on one foot. The first question was, is this comparing apples and oranges? That's why I have two apples there. Hopefully we're comparing two apples and not apples and oranges. But as Hans was talking about, Brooks chapter is wonderful, very readable and not boring in the sense of giving a very strict methodology, but also then we weren't sure what chronological period in particular we should be looking at. What geographical expanse in particular we should be focusing on. Something that came up in our conversation with the historians is the Jewish material is different that not all the rabbis we'd be looking at have university backgrounds and degrees. Is that relevant here? So these were all questions that were important for us as we were beginning to go through the sources and whether Brooks differentiation that he does talk about towards the end of the chapter of moderates versus more extreme positions if that was in fact relevant for the Jewish material. So what have we been doing? As Hans mentioned, the original author is on our advisory board as our other historians and also people we tried to search out on both advisory boards, people who have offered critiques of replication studies. So we have Britt Holberg also on our board who was one of the co-authors of one of the papers mentioned above. So we've been having a lot of conversations with them as well as historians that are not on the board. We are working with what it would mean to pre-register historical study. We actually have a revised version of the initial pre-registration that we're about to upload. And that's been an interesting process to think about and reflect on in and of itself. And then of course it's just jumping in and documenting the historians analysis and compiling all sorts of charts and different resolutions to try to analyze what would be important here. So very much on one, but if I would try to give the state I actually didn't press my timer to see where I am on my six minutes. But what would it mean to have a successful replication here? What are we looking for in the material? We have a fairly early acknowledgement of Copernican thought in the Jewish sources. We have fairly early praise and we even have a fairly early seeming embrace of Copernican thought. But then after that we also have thought leaders who reject Copernican thought. So is this enough to corroborate Brooks thesis that there's acceptance of Copernican thought on the Jewish sources, even if it's not linear even if it's not global to try to understand it. And my work has been a little bit complicated and by the fact that there is a current researcher really working on a lot of new materials that he hasn't yet released. But we're in touch with him all the time and always updating the study based on what everyone can let us know about their work. In terms of the second sub question I had mentioned above on the one hand this was an easier question the fact that social factors are important here and it's not a theological determinist perspective. This was universally embraced across the board on the one hand Brooks thesis was definitely corroborated but the question that emerged is is this actually an important corroboration or is this trivial? Is this just a development within the historiography of the field more generally that Brooke was part of a time period when he wrote the chapter that it was important to fight against this issue of thinking of things in this sort of closed deterministic perspective. And then there's just been this more general turn of understanding of the social issues and the importance. And so therefore the fact that it's been corroborated is not surprising because it just reflects the development of history. So on the one hand the second question is easier on the other hand it also raises I think interesting and important questions to think about. So I think my last slide is just a quote that shows how one of the historians really embraces or we have many quotes of how the historians embrace this new perspective. So he said Jewish discussion about demarcating spheres of physics and metaphysics reflected an emerging consensus of Protestant and Catholic thinkers. That was his parentheses, not mine about the appropriate structural relationship between scientific learning and Christian faith in the early modern era. So they're not focusing on the difference between Protestant and Catholic thinkers but rather how theological changes were shared between these communities and with Jews as well. So this was an example of how we see the second question really coming through strongly in the materials that I've been looking at. So happy to answer questions, try to answer them on Hans's materials as well but certainly on anything that we can related to our project. Thank you very much. All right, so together with Hans you displayed really in a small slot of time a lot of details and interesting elements of your studies. It's quite fascinating to me. There's one point I completely disagree as you can imagine. You said something about, Brooke was not boring readers with methodology and you mentioned that positive. That was not nice to a methodologies like me. I usually bought when I see a lot of theory in the paper but anyway. Well, I've heard that come up as a critique within the discipline. I think your colleague is smiling because that was one of their critiques if I'm not mistaken that it's actually a style in the humanities. So we're looking to be inclusive and not scientific in terms of being organic from within the disciplines also in terms of the terminology and also in terms of are we expecting certain stylistic commitments that are not in concert with what the discipline itself, how it views itself. It's an interesting question, I think. I got your point and of course I was just talking but anyway, I'm going to punish you for it by moving on to the next speaker because as a chair I have the duty to do timekeeping as well. And I really apologize but it was an impossible program and I want to give the opportunity to the last two speakers as well. So, Tim, he is the next speaking speaker. He is already mentioned by Rachel. He is also looking at the way replication goes in the history and said before he's a fan of reproduction and not so much of the other types of replication. Please go ahead, Tim, the floor is yours for your slots. Thank you very much. You can all hear me, I take it. I believe so. And it's not in the presentation mode yet, your slides, I guess. It is not. Okay, well, I'm afraid I cannot fix that on such short notice because it is in my screen. Okay, well, can you move from one slide to the other? I can. I can, so if you can do it, whether we can see it. Yeah, I think that will do. Yeah, thank you so much for having me in this symposium. It's been really interesting thus far and I've heard a lot of things that I will briefly touch upon as well. What I would like to do in this talk is to share with you some experiences from different, well, I would say experiments with replications in history that I did, particularly a project and also a course that I did together with my colleague at Utrecht University, Peter Huistra, who unfortunately couldn't make it today. I will, in this talk, tell you something about why we started these experiments, explain a little bit what we did and end with some observations. Now, as you have already heard in the previous talks, the default stance you could say towards replicability is that it's not really relevant for historical scholarship and that's particularly true for the field of cultural history where Peter and I are from. As historians, we are not out to discover once and for all how reality works, but rather add different interpretations about historical reality next to existing ones. Give to historians the exact same research question, the exact same data, the exact same method that you will get to, well, maybe not completely, but surely different answers. This is probably what Rachel in her talk referred to when it comes to my problems with conceptual replications. However, our starting point, and this is my next slide, if I don't hear anything, I take it, you can see it. Our starting point. I cannot, I'm sorry, Peter. You cannot, then I will try something different. This might work, try again. No, it goes big, it goes big. Okay. Yeah, this is okay. So do it with your mouse. I'll do it like this. If you can see it now, then we're fine. Sorry for that. Our starting point, that's where I left, is what happens if we try to do replications anyway. It was an open question to us, but we thought it was relevant because checking existing scholarship is becoming more and more easy. The more our data, which traditionally obviously is stored in often far away archives, if that data is being digitized and made available online, checking existing scholarship simply is becoming more and more easy. So you can imagine people starting to replicate existing scholarship, whether you want it or whether you believe in it or not. The other argument is our argument is that that our discipline has traditionally always had an open mind to replicability. Footnotes are open invitations really to check what information is based on. A peer review exists. And that's also our argument. We already do things like conceptual and also direct replications where research questions are approached with different kinds of data and our methods, but we just don't call it like that. This is what we call the historiographical debate. This is really what our field is based on. We believe, however, that there's also something to say for the strictest form of replications, reproductions where you follow an existing study as literally as possible. And the aim of doing this for us is not only to test robustness, to check for epistemic consolidation, do the claims hold or not, but more to see what is happening in historical argumentation. Is the argumentation sound? And this question, we believe, is as relevant for our field for history as it is for any. Now, this is at the same time a normative question, but also a descriptive one. Something simply to learn from. What is happening when historians are making claims for something? And this is seemingly a very obvious or maybe even a trivial question, but we've noticed that it can become a lot more difficult once you really take a closer look. And interestingly, our field lives by these different interpretations that I mentioned, but we also agree that not every interpretation is legitimate or allowed. So the question that we hope reproductions can answer is, yeah, what distinguishes legitimate? From illegitimate interpretations. So this is what we did. We took that closer look and we did that on two separate occasions. Once in a smaller pilot project that was called Once More with feeling a project in which three groups of two students replicated historical publications from three different subfields of history and once in a course for PSG students that we actually just finished. Both I stress were really quite experimental. And we've used the project and of course to learn how historians argue what sound scholarship looks like, but also how replications in history could actually be done in the first place. And to make it for us the most worth while the most interesting, we've gone to quite some length to pick studies of really of high quality. So that means relevant scholarship by established scholars in respectable publications. And while it's like I said, remained a true open question how replications in history can be done in the first place. One thing was clear to us soon enough that it makes sense to start from the end from the conclusions or the claims what we here call minimal reproduction. In this way we avoid exactly the problem that I mentioned earlier that if you start with the exact same research question, copy the exact same data or sources, the exact same method that you can of course still arrive at different conclusions. So now we and our students started with these conclusions with the claims from the original authors and try to trace these all the way back to the premises of the study. And we found this really a good way to assess the transparency of scholarship. Now in terms of observations, an interesting thing happened in most of the cases that we and our students studied. Most of these studies were compelling. They were relevant. They were interesting. All really good examples of historical argumentations. Still, once we took a detailed, a closer look, we often noticed some odd things, things that I've tried to put under the header of two main observations. First, historical scholarship could easily be more transparent than often is the case. We've heard that in the previous talks as well. Sometimes references are lacking why you would expect them or references are much vaguer than you would like. Well, there is this very strong tradition in history when it comes to source criticism but far less so in saying about how you went about methodologically exactly. And one of the things that were interesting that struck us was that often the more prominent a scholar is, the less strict they are or they seem to be in being transparent. It is as if they're saying, well, believe me, it is me who is claiming this. So I don't need any food notes. It is me, you know. And this observation alone already hints to the fact that these examples of intransperency, we think are not so much consequences of ignorance or fraud or something, but rather of implicit norms and codes. So that would mean that in spite of how we teach research practices to our students, there does not seem to be a single, formal or strict way of guaranteeing transparency in historical scholarship. Historical scholarship tends to be very much geared towards specific in-groups, specific audiences of experts and peers. But yeah, the question to us remains open whether this is a good thing or a bad thing or maybe simply an unavoidable thing, but it is something that really struck us. Second, what historians like to do is they like to evoke images of past realities. That is really what we do. Pieces of information from literature or examples from sources are then mobilized to fill in this particular image and the plausibility or soundness or the quality, if you will, of such interpretations, then depends on the usefulness of the picture of the past that is painted to interpret these sources or data. And this very often, at least in the examples that we scrutinized, leads to arguments that are not so much out to prove that something must be the case, but rather that something might be the case. And often arguments in this way of reasoning are replaced by examples. So this is what I claim. And here's an example that fits that claim. I have two examples here in this slide from that we came across during our experiments that are really illustrations of what I mean with invoking this image of the past. On the top, it's from a study that's on historical newspapers and the author writes that well, well in the early 20th century newspapers probably proceeded to sit on the kitchen table for most of the week to believe through at odd moments of the day underlining the importance of these newspapers. Yeah, of course, I can immediately imagine this to be the case. It can be, it doesn't necessarily need to be the case, of course. And that is not something that this author even tries or begins to argue for. The example below is from a comparison between two images where the argument is that the second image by the Hoge appeals much more to the senses than the other image. You can hear the cries, you can feel the heat, you can smell the odor. Yeah, it can be again, but it doesn't necessarily have to be, right? So there is no argument here. There's simply an invocation of an image. And this is very typical for historians we experienced. And yeah, in terms of replicability, sometimes problematic. Okay, I come to my conclusions. That is to say, I don't have any definite conclusions for that, both the project and the course that Peter and I have been doing were way too experimental. But based on these observations that I just shared with you and others, I can say that what I thought I knew about sound historical scholarship has become less certain while to us the place that replications in history could have has only become clearer. And definitely more interesting. We ourselves have learned a great deal from what we have been doing thus far. So we will definitely keep on working on this topic and keep spreading the word, for example, by way of the white paper that Rachel already mentioned that we have written that you can find through this link. And with that, I end my presentation. Well, thank you very much, Peter. And echoing your last sentiment, I'm becoming more agnostic about replication in humanities myself as well, I believe. It's quite intriguing what has been on the table so far during this meeting. And Rick Pales, he has a burning question and he asked me whether he would be allowed to put it on the floor. Rick, what is your question? Yeah, thank you, Pim. Wonderful presentation, I really liked it. I felt there was a bit of a tension or paradox like moving towards a contradiction at times in the presentation. So I felt at the outset, you were highly skeptical, saying things like, we don't want to make any once and for all statements. Well, who does? But statements or claims, we just add another interpretation, things like that. And then towards the end, you were rather happy about at least the reproductions. So here's my question, and wouldn't you agree that historians do make statements? They can be true or false at times, right? And the purpose is not just to add another interpretation, it's easy to add another interpretation. The purpose really is to add a better interpretation maybe or one that is more accurate, explains more, better fits the data and so on. And if you're enthusiastic about a reproduction, why not a conceptual replication as well? So the battle of Leipzig, how many people died? Well, let's look, not just at letters, but also results of excavations, for instance, or I don't know, other kinds of reports, right? So you use a slightly different method and you see what are the original results of, I don't know, a hundred thousand people died or so, where did that holds? Doesn't that make perfect sense? Yeah, thank you a little. Yeah, it definitely makes sense. So maybe I wasn't clear in my formulation, but I'm not against conceptual replication. What I'm saying is this is what history is all about. This is what we do all the time because, well, examples that you gave just now, yeah. This is exactly what historians do. Somebody studies a particular question based on particular source material and then a colleague comes in and says, well, yeah, I've studied the same question, but use this source material. And I come to similar, but also sometimes opposing conclusions, yeah? No harm in that. So that is, I think, perfectly fine. I just think that conceptual replication as a way to assess, to evaluate particular claims can be problematic because there can always be multiple interpretations of the same reality. So you can never say or, yeah, well, yeah. I think that is very typical for historical scholarship. You can look at the same reality from different angles. So who in the end, somebody makes a claim about reality, you approach the same research question in a different way, make an opposing claim. Who will ever decide who is correct? That's very difficult given that our reality isn't there anymore, cannot decide upon that. Yeah. Well, we're not debating, Rick. I'm sorry, I see you're waiting to respond. And but it's interesting, we need another media to hammer this out because this seems to be one of the real issues we are discussing together. So maybe we should organize a debate one day or the other. But my humble task today is timekeeping and I'm not too good at it, but I'm doing another attempt and I want to give Stephanie the floor. Stephanie Mirmans, she's doing ethnographic research about us, she's in a way to fly on our wall. You cannot see it on our screens, but that's what she's doing. And she will present, I've seen the slides already, and quite interesting get-tobox of reasons given to dual replication studies. And please, Stephanie, can you share your interesting work with us as well? Well, thanks, Lex, for the introduction. I try to be quick, but I'm not sure I can make it in time. So I will tell indeed about our project. And I want to emphasize I'm not doing this project alone, but I'm doing it together in a team with my colleagues, Yonah, Matt and Jeanette. Well, this doesn't work. Can you see that it switched? Yes, something happens, it's okay. Okay, so I wanted to take a step back and ask, what we'll do is to have a look at how do the humanities fit with other replication studies? And so if we take that step back and ask in the first place, why do we think it's important to replicate? I think historically it started with an instances of dubious papers, so fraud cases and the realization that maybe there are questionable research practices abundant in science. And then there were attempts to replicate. And somebody mentioned it already, there was this psychology studies that were on trying numerous of them to replicate them, but that didn't work. So there were lots of failures. So I think about 40% did not replicate, which was very alarming. And they were the recognition of replication crisis which is why a lot of people said, well, we should replicate more often, and not only because of that crisis, but also because it seems to be a such a thing to do in science with the intention to correct the scientific record. And well, Riko already mentioned it. So there was this paper that we should do the same in the humanities, it seems important to replicate there as well. And there were counter arguments and we already talked about that. But what I want to do here, these are the main questions for this talk. Are the humanities special when looking at replication and practice across fields? As for example, BART have been arguing. And then what is the meaning of replication? Does this differ between fields? And so in our project, we are investigating 28 replication studies in practice. So there has been the NWO, the Dutch research funder has been giving money for replication studies. And they have been funding 24 in total. And you can see in this table, 21 of the 24 agreed to participate in our project. So we are following them and seeing what happens in practice when you try to replicate. And what problems are they hitting? How did they try to solve this? What is the impact of their projects? And we have added seven other studies because the design was quite unbalanced as you can see. So we do interviews with replication researchers. We observe what they do in practice and we also gather documents like applications, reviews, paper drafts, et cetera. And I want to emphasize the pool that I have in the humanities. They're all historical studies. So I can't say much more beyond history for the humanities. So if we look back at, so we were following these projects and they are doing research along this empirical cycle of you start a study due to a certain motive, then you'll write an application, you have a certain design for your study, then you conduct a study and then it has a certain impact. If we look at sort of the assumptions for doing a replication, it seems to be that the motive is to detect coropies in original studies. Their replication design is hopefully or supposed to be an exact replication. So also NWO is asking for direct replications or reproductions only reasoning that the rest is normal science. Conducting the study is straightforward to do and to interpret and the impact is to correct previous literature. So we approach this of course with much more open stands. So we were just simply asking, what are your motives? What is your design? How do you conduct the study and what's the impact? So if it goes through this and I will put an emphasis on the humanities here. What is the motive? Is it really detecting coropies in original studies? Well, we already heard also this, it was more a pioneering stage. We simply want to try this out. So not ask how much sense does it make but simply experiment with replication and then see afterwards how much sense it makes. But we also heard that in PIMTUS also said this, that going over the same material, the same sources again is in fact often done in history. It is nothing new. It's just not called replication. So it's definitely not about detecting coropies per se. What about the medical and the social sciences if we compare this? Well, for a lot of psychology studies, this checking whether original study has been unresponsive is somewhere in the background. But if you ask them for the specific study they're replicating, this is not usually the reason. It's far more often the reason that they want to corroborate the original findings so that they say, well, this is an important effect. This is a landmark study. Everybody's blowing up on it. We really need to see whether this is true. It can also be a matter analysis in medical sciences. Some have actually used the funding rather pragmatically tweaking something that anyway wanted to do into a replication. So it's not really about detecting coropies in original studies. What about the design? Is it an exact replication in the humanities? One scholar said, a direct replication would be trivial. It would be weird to not take a new data that came on afterwards because I'm interested in the findings. Another one said, but when does it come a conceptual replication? So it's actually quite difficult to draw the lines. Another said, it took me some time to change what I wanted to do and fit it into the replication study emphasizing that it's actually quite difficult in the humanities. What about the medical and the social sciences? Is it so much easier? No, actually there's a continuous scale between doing something exactly as the original and doing something different from the original, which would be a conceptual replication. NWO was actually emphasizing certain changes itself. For example, you should have a larger sample size. You should do better statistics, but researchers also emphasized it does not make sense to use outdated technology or measurement devices. You do want to use the newest state of the art. And you want to correct, for example, for certain original papers, there have been some errors pointed out or shortcomings. You don't want to repeat them. And then medical researchers also several times highlighted, it's important to redo a study or to replicate a study and use another population. So in this case, not the country. But I also have been matter analysis. So throwing several replication studies and seeing what is the outcome. So triangulate or integrate them. What we see is there's a trade-off between doing an exact replication and doing something that you might call good or state-of-the-art science. So mostly what we see in practice is not really exact replications. Conducting the study, is it straightforward to do and to interpret in the humanities? This scholar says it takes much longer than expected. Another one says the hypotheses are not explicitly formulated as such. So you have to kind of delve them up. It's not so easy to know what to exactly do. How exactly do you need to trace the sources, the footnotes? What is enough and who decides this? And this one says it does show how important it is to actually talk to the original author. Otherwise we would not have known this. So because there was digging up extra information and far more information that was in the text. Do we see something different in the medical and the social sciences? Well, the time aspect of this takes a lot of time. It's a lot of effort. Yes, that's definitely the case. In psychology studies, they think it's sometimes even more effort than doing an original study. This, for example, also because a lot of methodologists want to have super huge sample sizes to make the interpretation easier. It's not often clear what actually has been done from just reading the text or reading the supplements. But often the original author needed to be asked. And then even then, there were lots of other decisions and changes to be made. For example, there was this one study that used a horror movie to stress out people, to participants. But actually that horror movie was not so much horrific anymore nowadays, even though it was available. They needed to redo the movie just because it wasn't having the same effect nowadays anymore. Of course, there were also all these changes that I talked before about that they were introduced and that made the interpretation actually incredibly difficult. And then of course everybody is working in a team. So teams might differ in what they think is important. And then some try to overcome certain problems with, so an exact replication makes it easier to compare with the original versus a different one is more to the state of the art. So they've sprung their study and did both. But that actually is another addition of time and effort. So straightforward to do an interpret, it's not usually. What about the impact? Is it about correcting the previous literature? In the humanities we heard, how can it be that we detected all these problems? So this is already indicating that there can be actually wrong things popping up. But simultaneously we were aware that somebody had made a career with this paper and that actually makes it very difficult. How do you handle that? And how do you actually, do you write that in a publication? What do you contact the original author? How do you actually handle this if you find something strange? It seems important to get into conversation with the original author. So this is the way of, instead of through a publication you get into conversation which seems to be maybe the better way. We got much more critical of our own writings and of others, this is a very broad fact. We do, what do we see in medical and social sciences? Actually the hesitations if you find something dubious to make accusations is the same in these fields. Getting into a conversation with original researchers and or diving more into the details is also something that we see. And then you can actually see that it's due to the details why there is a failure. Superceding the original. So the meta study, meta analysis emphasizes that it might be important to see this as new results and not correcting. The impact of publications variables. So not everybody is actually achieving anything with publishing a failure because the original study is still cited. A lot of medical and social science researchers told us that it became much more critical. They were looking at all the details. They saw the importance of transparency. They realized complexity in science, all the choices and decisions that they're making and how they impact the results. And then, so if I go back to my main questions, are the humanity special and looking at replication and practice across fields again, I'm talking about historical research. I actually don't think it's that special but they put an even stronger attention to contextuality, texts and the author as a person. What is the meaning of replication? Does this differ between fields? Well, I actually don't think it differs so much between fields but there seems to be a different meaning than typically assumed. So what, and this is I guess my take home message, we may need to rethink the meaning of replications what it is and what it does. I think it is our study shows it's more about gaining a richer understanding of the original study in its context and enabling triangulation with newer insights which then can lead to robustness or to conversation with the original authors. It's also about broadly to gain a deeper understanding of how we actually make knowledge and practice and that it's often local. Importantly, I think it requires time, care and respect to do replication work well. And the humanities could contribute in how to deal with contextuality. And I'm here at my last slide, I'm acknowledging and I'm thanking all the replication researchers involved in our project and that share with us their insights. And I want to put attention to that. We actually have a blog where we also post insights. Thank you. Well, thank you very much, Stephanie. This is really fascinating material. It shows that in clinical medical terms that replication might have some beneficial side effects which are really interesting. It's great. And by the way, in the chat, there is an interesting discussion developing starting by Rachel who kindly reminds everyone that we already organized a meeting to have a lengthy debate on our matter and that will be in June. The link is in the chat for everyone of you. And to you, Stephanie, personally I was quite fascinated by that you said the scientists can learn from humanities as well when we talk about replication. And I tend to agree. You mentioned point of contextuality. Can you give an example? And then maybe also please explain how that difference from what our preoccupation is mainly with generalizability. So I think that, for example, there is one study where there seemed to be a replication failure. And so that person then did not say it's a failure but went into the details, looking up, okay, where is this coming from? How can we explain this? And actually what occurred then was that something that appeared to be standard across this, in the field, wasn't as standard. So that actually generated the data in very different ways which could explain the difference. So that's the contextuality of, for example, how you generate data. Yeah, well, that clarifies it. Thank you so much. Well, ladies and gentlemen, and to all of you I apologize again to be rude and end the meeting but I'd like to thank the speakers really. It has been a very interesting event at least to me and I hope to all of you. It was great to have some interaction. The time was to brief to have a lot of interaction but still it made people think, at least it made me think about replication again. It's wonderful. Thank you for organizing this whole show, shout outs and all the technical people from the US behind it. It's really great to have the opportunity to do this. With that, I'd like to thank all the participants as well for being patient with our technical glitches but we survived and it went quite well altogether. No problem at all. So thank you so much and I hope to see you on other occasions on replication in the future. Thank you, goodbye.