 Okay, we are right at 4.05. So I'm going to get started. Hello and welcome everyone to a digital native publishing format research modules presented by Chris harboring from the liberate from liberate science. My name is Cynthia to Dangan and I'll be your moderator for the next 40 minutes. A quick reminder before we begin during the presentation portion, please keep your audio and video muted. Feel free to use the Q&A tab to post questions. And I will try to keep track of those. You can navigate to other sessions by clicking the desired session in the hop in schedule or click the left hand sidebar links. Thank you all for joining us this afternoon for Chris. It's this evening and we're delighted to have you here. I'm going to go ahead and turn it over to Chris. Thank you. Thank you, Cynthia. So good morning. Good afternoon or good evening, depending on where you are. Thanks for joining in this session here at the U.S. ETDA. I appreciate you taking the time to be here. I did turn on automated transcribing, but I'm not 100% sure if it's showing up in the screen share. Cynthia, do you see at the bottom, automated subtitles? Okay, seems like that's not coming through. So I'm just going to disable it. Yeah, I've heard that that's a little bit of a problem. Chris, I'm not seeing any transcription happening. Okay, then I'll leave that off. I'm very sorry for the people who are hard of hearing. I hope that I'll be sharing these slides afterwards also. So I hope that if there are any questions, feel free to ask my apologies for not having transcription with this. Oh, is it coming across? Sorry, it's showing up very badly here, my screen share on my end. Yeah, there's something wonky going on there. Okay, then I'm just going to stop it and I'm going to reshare. This is always a fantastic start. So thank you for your patience. So I'm Chris Harkrank as Cynthia kindly introduced me. I'm the founder and director of the Berlin based startup Liberate Science. And today I'm here to talk about research modules, a new publishing format. So before I get started, I want to share a bit more about myself because I definitely realized that I'm a bit out of my usual space. So I'm a researcher myself by training, and my background is in social psychology applied statistics. And I did my PhD in meta research or more commonly known as science and technology studies, where I really focus on how to improve how we do science and in order to really provide more reliable and reproducible results. And I really came of age during the so called reproducibility crisis. As these issues started coming to light in 2011, when I was just ending my my bachelor's and starting my master's degree and I used to be a research assistant to to a psychologist who ended up being one of the biggest frauds. So that was also definitely a realization moment that something was wrong, both at a local level and broader. And one of the real key findings of my dissertation was that the publishing system is really a massive bottleneck for any change that we want to that we want to create within the scholarly publishing or the reproducibility system. So if also after the talk, you know, some ideas that come up that that I from what I talk about here, please do not feel limited to just the Q&A right now, but also always feel free to reach out after with my handle chart grink or on Twitter or my email. And I really want to get started here by taking a step back for a moment. And this this graph from a from a recent publication. I think is a very nice one it talks a bit about all the various improvements in in scientific publishing that have come up through articles but also theses and dissertations and it talks about these things such as data sharing, code sharing conflicts of interest protocol registration replication and in my bubble. I don't know whether this is the same in your bubble, but it's very often that we think we're doing incredibly well on uptake for these practices. So that data sharing we keep seeing it more and more code sharing replication. So yes, we're doing really well. And sometimes my colleagues who are also statisticians who always like to look at the data we sometimes also forget that that we need to have empirical evidence and we would look at this. There are some domains where where it's promising things like funding disclosure disclosure and the conflict of interest disclosure. But when we look at something like the code sharing or the protocol registration, we're really only on track to maybe achieve this by the year 20, 2100. And when the world is also going to be many more in degrees hotter than it is now on average. So yes, we might see absolute numbers increasing, but we shouldn't forget that if for example we achieve a 5% increase in code sharing in 2020 compared to 2019 that the 8.9% increase in publications actually means that we are ending up with fewer publications that include code than the year before. So because the amount of publications is growing much faster than the amount of publications with code, it means that the gap is actually becoming bigger. And why am I sharing this right now is really to highlight this point that we're now in this exponential time of publishing. There's ever more information being produced. So that means that in order to really start changing the system, that change also needs to pick up the pace much, much faster. So the point there being that reform isn't really an option anymore in this exponential age, both for changing how and what we publish and as a side note also not for the climate crisis. So for substantial change, we need to substantially change what we do and what the system does and pretty much saying we shouldn't keep adding duct tape, but think about how to really pretty much revolt within the system. And to analyze this, to get a better perspective on how to actually look at these issues and to really assess where we can improve and how to improve it. I really like this framework from the Library and Information Sciences, which gives the scholarly communication or the publishing system five specific aims. And this is incredibly important to help understand where the work needs to go. So the current publishing system, what does it aim to do and how does it fail at that? So of these five functions, the first one is to register findings into the record. Then also to make sure those findings are archived to certify the quality of those findings, raise awareness about those findings and to incentivize or reward doing research. And at the moment, we know that the current publishing system fulfills these aims to a certain degree. The question is whether that is something we're satisfied with, also in light of the amount of money that goes around in the publishing space. And I for one would argue that the state we're in right now is rather miserable and also unacceptable, I would even say. So to sort of indicate a bit of these issues here specifically where we could say, okay, the system is fulfilling this function, but is it good enough? And so with registration, for example, we can take a very narrow view and we can say, well, the findings that are published are registered. So yes, it is fulfilling that function, but we also know from ample research that there is highly selective publishing and not for good reasons, but also highly arbitrary ones, where the quality is assessed to be worse, even though the methods are the same when a study is not statistically significant and the chances of publication are higher when it is statistically significant. So this selective publication creates an issue within the registration function. Then another issue here really with the archival that we see is that yes, publications are archived, but in very select places with very select access. So these are what are called dark archives. It's like a vault, only in certain events will those archives be triggered. And the idea there being yes, it fulfills this function, but if we take a wider interpretation of what is archival, one of the key fundamentals of any archival functional is to make a lot of copies. And one of those things is with these dark archives that it actually prevents making a lot of copies. And this you could argue either way, I'm not necessarily saying you should have one or the other opinion. My personal opinion is that a wider interpretation would be better for the publishing system. And then also with respect to certification, we see something similar. Yes, we have peer review in place as a certification procedure to make sure that there's a certain amount of quality. We can discuss the issues of peer review also. But I think one of the things I want to highlight is if we take a wider perspective on certification is also that we want provenance of results, provenance of research findings. So really to understand the origins of research findings. Then there's the issue of awareness. Yes, publishing helps to raise awareness, but there's the wider interpretation where publishing doesn't really promote access. It's increasing now, but also with that, it's going to take a while with the current trends before we actually reach complete access. And then with respect to rewards, yes, the current publishing system of articles has a reward function. Through beam counting of publications or citations. But we can also ask, is that narrow interpretation sufficient? Could we take a wider interpretation where we can actually reward and incentivize in a much more healthy manner to really nurture good research and good careers? And then we really come to this issue, this core issue that if anything within this publishing space right now, whenever we want to improve on any of these points, we have to go through ever more consolidated publishing spaces. So here you see a graph from a publication from 2016, I think, from La Rivière, where they analyzed how many publications get published by publishers. And we see that within the field of social sciences that there's actually almost around in some cases even more than 70% of all the publications come from five publishers. So and we see this drastic increase over the decades. So a researcher can improve their work all they want. But if the publications don't recognize that their career won't either. And they will either have to adapt to the rules of the game, which promotes worse quality research or leave academia altogether. And of course there are edge cases. But the research really indicates that if you play the game well, you do a lot of small sample size studies, get some nice innovative results that end up not replicating. That's how the game has been played over the past few decades. But we see that the past few decades has also come with this consolidation. So we would really have to go through these major publishers, these major institutions to make any change. And that's increasingly difficult also because big institutions are harder to change. And to sort of illustrate also this issue of selective publication, imagine 100 studies that are conducted by the researchers. So these are, this is all the work that we would want to register, that we would want to archive, certify, that we would want to raise awareness about, and that we would want to reward to a certain degree. You can also say negative rewards in that sense for insufficient quality. But because of the exclusivity of journals and the page limits and the innovativeness and other reasons, only a subset of these even gets published. So this is what we would be able to see at the other end. So a few studies might be of insufficient quality, but most of the work can be for now assumed to be at least of decent quality. So if we pick out one of these, this subset that has been published of this larger subset of papers that has been conducted, then we see that it's been certified by peer review, it has a DOI, and we have certain expectations of it. So we read this paper, there's this narrative in there, and we see this result. And then this question starts arising, right? Because this is the core of how we publish articles, theses, or dissertations, and dissertations, this might be a chapter in that sense. But then we get this question of, as I mentioned before, where do these findings come from? And behind every paper, there's a story in that sense of how the research was actually conducted. So we might, in an empirical setting, have the standard empirical cycle where you start out with your theory, you move on to your predictions, then you create study materials through, and you collect data around it, code results, and so on. And that might be perfectly represented, the paper might tell a story, and that might be the story, the origins of that story. But we cannot at the moment discern whether this happens or whether something else happens. So in my field, this is often called p-hacking or hypothesizing after results are known, where you go through the research cycle and by the time you get to the results, you figure out, oh, I forgot a certain covariate in my analyses. Of course that should have been part of my predictions. And it's very easy to then take this shortcut unconsciously and update the prediction, but that invalidates the results to a certain degree. If it's presented as a story that's nice and linear in that sense, we'll see later that this behavior in and of itself doesn't need to be problematic. But we don't know because within an article, like in this situation, we're sort of like trying to see through what the article actually is presenting, where the origins of these findings or that story comes from. But we can't see it because the paper isn't transparent. It's just that story. So that is why we're reintroducing research modules. So I might have been a bit exaggerated that we are introducing it in the abstract because it's actually not a new idea. It's from some researchers from Elsevier from the late 90s where they proposed, let's publish research in these independent units that can be used to construct larger pathways of research. So I like to compare it a bit to LEGO blocks of research. You have different bits and pieces and you can build findings from those. And those modules, these building blocks, they don't even need to be text per se because in this digital age, we can really conceptualize one module as simply being a container where we can put information in. And that information may take the shape of text in a theory, for example, but it can also take the shape of data or code or maybe even videos of a study protocol, for example. Or pretty much anything you can imagine. And taking this idea of research modules and creating those and constructing those within a digital space allows us to really create a vast array of these modules that contain all kinds of information that are created in the research process. And these modules can then become very helpful in deconstructing the article. So articles or theses or dissertations, they always present some studies, for example, or some findings. And each of these, I'll assume for the sake of the argument here that it's one quantitative study but you can also have other research processes. So each study represents a research process. Each process is composed of multiple steps in the research. And the order of those steps is incredibly important to understand the origins. And each step produces information in some form. So you probably already see where I'm going with this is that each step in the research gets its own module. So with the information we produce in each step, we have one output in that sense. And remember this, what I showed before, if we could see through the paper where the narrative is shared, where the story is told, and that we would be able to actually see the origins, the research process, it might look something like this. It might look like this neat empirical cycle. So we could then take a look on the left is this idealized version, but we could then subsequently in a research module sense, we could start out by publishing a theory on its own as a module, then move on to publish a prediction and link that back to the theory. So if we would now only like we would stop there, we would then be able to say, okay, these predictions, they originate from this theory, but we can keep building on these research modules. So in this scenario, we might add a research module for study materials, link that back to the prediction, and keep going with more and more modules all the way to, for example, a discussion here, but we could also keep adding additional ones as our research keeps going along, because there isn't a clear delineation between one research process and the next, because it's also very often one big research process. But so if we would then subsequently, for example, see these results, and we'd be interested, hey, where do these come from? We would be able to trace back the origins. We could see the code and keep going back up the path to see, hey, where does this come from? And with these research modules, each would get their own separate DOI. So you would actually be able to point very specifically to one step in a research process. And then if you would want, you could still collect all of these steps and then subsequently say, you know, I'm going to write a story, a narrative around this and publish that as a paper or a thesis or a dissertation. But then if we would move on to, for example, a bit more, a less idealized version of an empirical cycle, then we wouldn't be able to see this within the paper very easily. This would often be a nonlinear story in that sense, where we actually go from a theory to predictions and then we realize, hey, something's up with the theory. This doesn't make sense. So we actually want to backtrack and go and update the theory again and create new predictions and then create our study base on that. And that would be very difficult to share in a linear storytelling fashion, but within a research module space, that would be a continuation of the research process because that's in essence what it is. So we would, again, publish a theory, publish prediction, both get DOI, but then instead of immediately going to the materials as we did before, we actually create another updated theory module, which we can then neatly link back to the prediction. So the origins of this updated theory are actually also much clearer. And then we can continue through, for example, this need cycle. And this is very idealized, of course. But then instead of going one step further, where we previously saw this, what I called a shortcut or what's called p-hacking within the paper space because this update from the results to the hypotheses is actually, there's no space for that to happen within the article because it's such a linear narrative and this story that's being told. If we would represent this in a research module space, we would start publishing each step of our research as a module and the process could just continue up to the results. And then these updated predictions are also a next step in the research process. So we can actually understand what happens in this process because if we publish all of these steps as we go through the research process not just at the very end after the fact, we can see how the research develops, which contains also a lot of valuable information for other researchers trying to understand findings and where they come from. It might be a bit less neat in terms of story, but just like books provide us with a narrative across articles, articles can still provide us with a narrative across many research modules. So research modules can be really helpful to break free from the limitations of an article in that sense when it's not in this idealized form of a regular empirical cycle. And then in this specific scenario, we haven't even talked about who publishes these, so we might imagine that this is one person, but we can also imagine many modules being published by different people within one research process. So in this specific scenario, you see that three different people publish different bits and pieces of a research process. So with this, we would even be able to make research work more specialized if people would want to, and we could recognize contributions of various people across research modules within one larger research project or a research process, so that we could also really allow people, I, for example, love building big databases and analyzing these and providing the code for analysis, and that's really where also my expertise lies. I'm not that good in building up theories and predictions and creating study designs, so I'd be very grateful if I could collaborate with somebody and let that work also be recognized for them and that I would be recognized for the work with respect to building these databases and this code. And this would be incredibly helpful to reduce this pressure on researchers and especially early career researchers, I would argue, to be specialists in everything because that's quite the high demand. So in that sense, research modules might help to also improve the working conditions in that sense. So as a recap, research articles, they're a storytelling approach to publishing and what we're proposing based on this paper from the late 90s and later work is to take a process-based approach to publishing so that it's a research process instead of research storytelling. And these modules, they can be text, code, data or other information holders and that really has a sense of creating Lego blocks of research and the more blocks there are, the more creative also the outputs can be. So what now? This is all a nice story in and of itself, of course, but this is not just something that is a story. So we've been experimenting with researchers on a practical implementation of this over the last year after we've been developing these ideas for the past four years, partly funded by the Mozilla Science Lab and will be launching a publishing platform for this for research modules on February 1 where every individual can sign up and publish research modules open access for free. So if you would like to stay informed about this, I'll drop a link in the chat in a moment where you can sign up for the newsletter so you can stay up to date and also if this might be of interest to explore options to introduce this form of publishing with your department students or other constituents, I'm always happy to set up a meeting to discuss that further. I'll also drop a link in the chat where you can just slide into my calendar for a quick chat about that. So thank you very much for taking the time. I hope that it's a bit of a more amenable time for you but I'm still very grateful to get this time slot and I'm looking forward to hearing any questions, concerns or any other thoughts, to be honest. So thank you very much. Thank you, Chris. That was really, really interesting, compelling. I'm afraid that some of it was beyond me but I'm really excited to learn about what you're doing. It strikes me as I know cutting edge is kind of a pat thing to say but that really does strike me as what you're doing and I'm hearing that in the chat also. John wanted to make sure that you knew that this, your presentation will eventually be posted on YouTube with captioning. Thank you. That's fantastic. It will be available with captioning and also I have a few questions but there's also one from John who is no longer here. He had to go to another meeting. He's asking if you're familiar with the NDLTD. NDLTD. It's the Networked Digital Library of Theses and Dissertations. No, I'm not familiar with it. It's NDLTD. It's in the Q&A and there's a URL there. They're having a conference, a symposium coming up and John is recommending that maybe you consider presenting or publishing in their e-journal. That sounds like a good idea. I'll definitely check that out. Thanks, John, even though you're not here anymore. Right. So because I'm really a beginner at what you were talking about, I'm interested in a couple of things. One of them is what you talked about five publishing houses. The implications of that seem to me to be grave to creative research from diverse perspectives and worldviews and just having this really small, limited way to get your work out there. Can you talk a little bit more about that? Yeah, so I'm Dutch by origin. So I feel like just the colonial history is already very rich but the aspect of that these are British, Dutch or American houses that sort of quote, unquote, rule the world. So ruling the world. Yeah, that's definitely an issue. And I think that one of the key tenets for me is always researcher control. So that the researcher can decide and consent to what's happening. And my personal experiences also were that, you know, we had paper with some formulas in it and it was typeset and all of a sudden superscripts became subscripts and completely changing the meaning of things. And it happened to some of my colleagues also and it took away that autonomy. And I think that's one of the things that we see happening a lot also in terms of diversity and inclusion where, you know, the English language it's so familiar to so many of us in the western world but at the same time there's also so much more and this, the nurturing, not just allowing it and tolerating it but really nurturing these other ways of knowing is incredibly important. So autonomy for researchers to really publish what they want, when they want to is for me very important in the language they want because I think that, you know, it's amazing the kind of work that's being done and in the African continent that they don't get the opportunity to publish and be recognized for within their own institutions because the expectation is publish in high impact factor journals which are primarily in the English language. A lack of recognition and respect for indigenous knowledge and the ways that it gets reproduced and performed. Exactly. And we have this manifesto where we say it's all about knowledge trademark instead of like multiple forms of knowing. Thank you. That's really interesting. So you talked about publishing being arbitrary and selective. Can you tell me more about that? Tell us more about that. Yeah, so this is very famous paper in my field from this man named Mahoney where he did an experiment with results and it was exactly the same paper, the same methods except that the results were, you know, either they were, you know, yes, we found an effect or it said, no, we didn't find an effect. And it was the reviewers evaluated the studies that found effects as being of much higher quality than the ones that didn't find effects. And note, not finding an effect can be in that sense, we didn't find a difference between the placebo and the drug condition on side effects, which is a very positive thing. But yeah, so that kind of arbitrariness in terms of the human aspect of evaluating research is definitely something that comes to mind and it reminds me also of these studies that happen with respect to like resumes and job applications where they switch out only the name and then if it's a bit more of a foreign name, all of a sudden acceptance rates plummet and that kind of arbitrariness and how do we ensure that we don't fall prey to these kinds of human biases in that sense. And we're not often thinking about them. We have a question, Debra, would you like to ask the question yourself or would you like me to feel free to unmute and turn your video on if you'd like to ask your question. Okay, I will go ahead and she asks, I can see how this would be a great novel platform for established researchers. How would this type of publishing model translate to dissertations for our students? Great question. Yeah, and I think that from my own dissertation work, it would really be about being able to manage and track your own work better so that by the time that you have to create your chapters, all the documentation is already there and you can also point towards all the supporting materials that you've already sort of, the breadcrumbs you've already created for yourself. So in that sense, I think that this isn't a way to publish your whole dissertation per se, but it's more about collecting all the tidbits that go in there and having that path be available to you, so documentation in that sense. But this is definitely something where also the practice of this is incredibly important, which is also why I invite, if you have people who might want to be interested to work through these things to let me know because I think that it is completely different. So there's still much to learn, and this is not the end, because when something is introduced, new problems come up and we want to really be in this dialogue to continuously figure out, because we're building this. We're not a big publisher and we have the adaptability to really be like, you need X to do your research better. Well, we can work with you on actually making this happen within a few months, so to speak. Well, that sounds wonderful. We are down to just a couple of minutes left. Just continuing along Deborah's question, it seems like even though it's completely different from the way our students are trained to research and write and publish, that this is a great sort of model to start at the beginning of, as you said, an early career researcher. Yeah, and at the end you could sort of see your whole history evolve and how you're sort of creating your own tree or web of knowledge in that sense. Not to refer to web of knowledge at the database. Right. Wonderful. Well, we are just about out of time and we are out of questions. How would you like to, anything you'd like to close with? Actually, there is a question. There is another question, if you don't mind. No, that sounds good. Emily asks, data management and digital accessibility are important factors now with research. How are you addressing both of these? And I'm afraid you're not going to have a lot of time to do that. Well, in that sense, I think that these are new issues where also adding tools is very important. There's many ways how to interpret this, like various things. I think that having a dynamic approach might be able to solve some of these issues better. But Emily, I'm happy to chat more in DMs or something. So, Cynthia, thank you for moderating. That's something I want to say. Well, Chris, we're delighted to hear your presentation and I really look forward to hearing more from you and more from Liberate Science. So, thank you very much. Thank you, everyone, for attending. Bye-bye. See you around.