 Hi. So welcome everybody to this session on preregistration and also the uptake of preregistration in a couple of fields. So I'm super glad to have three experts on preregistration here with me and that and Celine Heinel particularly since she organized it that we can do this symposium here and thanks so much Celine for handing in the abstract and everything else. So this is not my doing. I'm just a bit of the MC tonight to or at least for me it's tonight so shout out to everybody else where this is a bit earlier also in the day and also for to the Australians where this is presumably even later than for me tonight. Okay so we're going to talk about preregistration and the uptake of preregistration in several fields. So we have we're starting off with medical sciences and here in particular clinical trials and I'm happy to have Nick Peter here who will talk about this uptake and then Marianne Bucker she will talk about psychology and particularly how pre-registrations has evolved in this field and I think she will tell us a lot about this because psychology is as we know one of the fields for preregistration has been picked up quite with quite some success. Then Celine Heinel from the Institute for Risk Assessment in Germany she will speak about pre-clinical research in particular in visual studies so animal studies and talk a bit also about the registry that they have I think now for two years two and a half years she'll tell you a bit more in detail and I will only very very briefly share some thoughts on preregistration in in vitro research about the challenges we face here because I can't talk about the uptake and you'll see why in in a couple of minutes. So we thought the session would be organized very well if we all give a five-minute introduction only and then we will discuss between the four of us a couple of challenges that we see and a couple of questions that we think are important so I will moderate this and after answering one or two questions that we think are important we have a little poll and this poll I will display after this and all the people here watching this you can then decide what would be a good idea for us to talk about and of course you can also post your questions into the questions and answers yeah so here in Zoom there's not only the chat but also the questions and answers I was confused of this also when I entered this conference I didn't know about this and in the questions and answers you can even upvote questions and answers so if somebody asks the question and a lot of people voted up you can be sure that I will hand over this question to the other three panelists and we will try also to answer your questions okay and with that I give to Nick who will start off with preregistration in clinical trials and give a short introduction Nick. Nick you're still muted okay can everyone see my slide yes excellent yes great so yeah hi everyone Nick DeVito I'm a PhD student at the University of Oxford and I studied clinical trial registration reporting and I just wanted to give a quick background on clinical trial registration and reporting for people who might not be familiar and sort of to form some of the discussion we're going to have so I'm starting briefly with the motivation of how clinical trial registries came about so in the late 80s Robert John Sands wrote this piece that's often cited as sort of foundational to the argument as to why we needed comprehensive global trial registration he framed it in a way to prevent publication bias right Simes was you know part of these evidence-based medicine movement that was emerging very much in the late 80s and the early 90s and it should be no surprise that the call for registries sort of grew alongside that movement he wanted to know what studies were being conducted so that he could better generate and identify by it but generate evidence and identify biases in the literature to about treatment effects mainly and he showed using a proto cancer registry at the time that had results compared to the published literature sort of what that looked like he showed real examples of publication bias so that was one big motivation and then the other which I think is a bit more unique to clinical to the clinical sphere is recruitment people wanted a way to connect potential patients to studies in which they could participate and I'd argue that like that it's not like psychology economics where you have undergrad you can pay $10 an hour to come participate in your studies right you need people who have the conditions there's harms involved there's risks involved that aren't there for lots of other types of research and I would argue that clinicaltrials.gov to this day right when you go there this is the front page and it is geared towards finding on studies that are recruiting for conditions and locations right that's very much geared towards participant matching and of course so there's a glow there's this global network of registries in now under the guise of the international clinical trial registration platform which is hosted in the WHO they set standards for registries and all these registries participate of which clinicaltrials.gov is the largest by far and oldest so um we don't won't cover all of these things this is merely for demonstration purposes but these are some main events that like led to registration um in the clinical field and um there were you know we can group these into regulatory and legal um milestones sort of statements from national multinational bodies promoting registration uh how to do it when to do it why to do it and the ethical imperative to do it um you had academics who were creating these statements to incorporate into the way we report research and you had industry um eventually coming on board to support it as well kicking and screaming but eventually becoming um you know full participants in it and uh pretty good at it which is what a lot of my research covers you can be slightly more reductivist though and say we had years of tireless advocacy and groundwork leading up to 2004 which was sort of the turning point for registration um people like Kay Dickerson and Ian Chalmers um Dickerson Ren these are people who beat who were beating the drum for years to promote registration and then you had a big scandal sort of an inciting incident for a whole bunch of action and that was the paroxetine scandal if people are familiar with that um if you're not very very briefly it was an it's an ethnic depressant that um GSK put on the market and they withheld and misrepresented evidence that showed that it actually increased the risk of suicide when given to children and adolescents and it caused a huge storm it led it to the international committee of medical journal editors to release their statement requiring prospective registration which was huge uh that is the biggest thing by far to ever happen to the field of registration clinical trials outside of maybe the founding of clinicaltrials.gov you can see 2004 that just shoots upwards um not all of these are prospective at last check you know trials covered under the U.S. law the FDN Memet Act which actually doesn't require prospective registration requires registration within 21 days about 80 percent of about 28 000 trials meet that standard but about 88 percent meet that standard about 80 percent are pre-registered so just some closing thoughts as we um to set the stage for what we're doing I think there's a lot of unique it's not unique things about the clinical landscape the motivations are a little bit different as I went over you have this high-level stakeholder buy-in that other areas I don't think have seen to the extent we have in clinical research we have laws and regulations which is something that are very difficult to do in other areas because human health and wellness is not sort of impacted and there's not an entire industry built up around it but um we have a lot of the same issues around using these to promote rigor and around the people complaining about the effort to do so the infrastructure exists for clinical research we have our own infrastructure but the OSF is an infrastructure for everyone else and um we have inciting incidents big scandals in these areas and we have plenty of staunch advocates uh on both sides and uh that is it for my introduction to clinical research yes excellent thank you so much Nick so you we're talking about infrastructure I think that's something that we'll definitely go back to in the discussion what kind of roles infrastructures play and and how they differ also across the different fields so next up is Marjan and she'll talk about psychology uh yeah I'll also start sharing my screen yeah so I will talk about pre-registration in psychology um yeah so um yeah in psychology it's somewhat less recent although we're uh I think also uh yeah 10 years ago I think it's it's almost um yeah it was probably around this time 10 years ago um I think some turbulent times in psychology started uh so one of these things was that we had uh this uh major fraud case uh of Biedrich Staples um but also some other things happened around the same time so uh we had this paper by Daryl Ben that was published in a high impact journal in which he proved that precognition was real and he did so by nine studies or something that all showed a significant effect um and of course we know that that cannot be true so it should be a no effect but how is it possible to get that published with so many significant studies and what does it say about other studies that are published um and also around the same time we had some uh big uh field replication studies um and uh of course later on we even found more and more uh field replication so that makes that um yeah a lot of people realized that something was not going well in psychology and um yeah that was uh yeah for a lot of researchers felt it that way and basically yeah what what the conclusion was is that we have all those decisions that we are making when we're analyzing or studies um and uh yeah if you combine these different things then it's quite easy to find a significant effect although it's not probably there um and then you can just build up a nice story around it and publish it or if it's really not working then uh you can put it in your file drawer and so we end up with all those studies that probably are for a large part no effects um and I think this was uh yeah because of all those different things that that went wrong or raised some red flags in psychology a lot of people realized that and also one of the uh main proposed solutions at that time was uh to do pre-registration because in that way you can then uh before you start analyzing your data you can decide on what you're going to do so you'll have your analysis plan and everything ready so I think it's a little bit more also focused on um preventing this p hacking or the use of questionable research practices and a little bit less on the uh publication bias the prevention of that um and um also because a lot of researchers realized that something went wrong um and and they started connecting with each other um and and that also led for example to the society for the improvement of psychological science which is a really action-oriented society and that's also resulted in things like uh different formats uh for different research designs so a pre-registration format for different types of research so um we have um many different formats from very extensive one to less uh more simple one uh and for specific things like qualitative research secondary data cognitive modeling and all those things are possible also of course because we have the open science framework which was also developed around um the same time um so you can see that there's a lot of uh work also from researchers in that so um many of these formats are also developed for example during meetings of this society for the improvement of psychological science so a lot of uh yeah bottom up um work from researchers helped to get this pre-registration from the ground uh but those are some other things of course so um we had a pre-registration challenge also to increase this uptake of pre-registration so for the first thousand uh researchers who pre-registered their study and got it published in uh some uh journal um they got a thousand dollar each um and there are now uh many journals who award pre-registration badges um so that might also um and yeah help people to uh to do these kind of things and what we see is that there are now a lot of uh registrations so I think now at the OSF there are even more than 77,000 registrations um and uh also um first chambers and others they uh they promoted registered reports so these are comparable to uh pre-registrations but then they're also peer-refute um before collecting the data and you will get a publication independent of your results yeah you should just follow your pre-specified plan and then you'll have this publication and that's now also offered by many different journals um and also now we see more and more research which also shows sometimes that that there are also definitely things that need to be improved or that are not optimal yet so I definitely don't think that we are here also because uh although pre-registration um has become more and more uh it is being done more and more it's of course still only a small part of all uh psychological research that's now pre-registered but this research for example showed that the specificity of pre-registrations could definitely be improved also the adherence to pre-registered plans is not always good or deviations are not transparently reported but also that it's often well valued the research quality um so we see more and more research also popping up into this which is really nice so um yeah I think that was what I wanted to say and other things will pop up during the discussion yes thank you so very much and um after psychology we now are heading back into more biomedical fields so um Céline will report now on in vivo particular animal studies uh okay okay perfect so I'm happy to be part of the session I have to say because I think yeah I will introduce preclinical research and we are really lagging behind so I think we can learn a lot from from other fields so um but I will shortly introduce um preclinical research and the background um why it's done so um similar to psychology and what Marianne already said is um we are we're also facing a strong reproducibility crisis within the biomedical sciences and in addition to this we also what was was observed already long time ago is that there's an insufficient translation from biomedical research into clinical science here's just one example meta analysis from stroke research where um they looked at treatment effects and experimental animal animal studies and up to a three clinical trials and they see a strong decrease in the rate of efficient effects and of treatment for acute stroke so we can see this across all kind of biomedical research fields and of course this has a lot of different underlying reasons so quality of research but also things which can be addressed by pre-registration like selective reporting questionable research practices and what I also want to add that this of course represents also an ethical issue if patients are really put at risk by being recruited in clinical trials and um only based on on I would say fragile results from from experimental studies but there's another ethical issue which is often forgotten is that most of the pre-clinical research is still based on animal research and animal experiments which are still very controversial in our society and we know already since long time and now there are really recent studies which show also the dimension of the problem and how many animal experiments are actually performed without contributing to any gain of knowledge and scientific progress so here too I won't go into detail just to quickly put some numbers on it is here two studies which looked at publication rates from animal study protocols so this is how every scientist in Europe would want to perform an animal experiments had to get a permission by applying with an animal study protocol and they looked how many of these protocols led to at least one publication in the end and the number in Germany which they found was 67 percent and something similar also in the Netherlands after seven years of the study protocol permission there were only a publication rate of 60 percent and if they looked at individual numbers of animals so the the numbers which were planned the animal which were planned which when were reported in the final publication these were only 26 percent so we don't know exactly the number but we know that a huge amount of animal experiments are conducted without contributing to any knowledge and this was also one motivation I think to or is one motivation to start pre-registering animal experiments so already yeah now 10 years ago there were there were voices which were raised and saying okay we could perhaps also pre-register preclinical research on the example also of clinical research and if this would make sense but it's only until 2018 where there was the first preclinical pre-registration platform for preclinical research the preclinicaltrides.eu an initiative by the Utrecht University so pre-registration specific to animal and vivo research and then in 2019 we followed with our pre-registration platform animalstudyregistry.org so also by the Federal Institute for Risk Assessment so it's a governmental initiative and both platforms aim the same so aim for pre-registration and preclinical research and offer a template also with a guided template to support scientists in designing their study and what they have to report and of course there's also the possibility to pre-register on the open science framework so a registry which is open to all kind of research and of course also animal research can be there but for the shortness of time I would just focus a little bit to show the number in the state of pre-registration and preclinical research on these both registries and so perfectly on time for our session today the preclinicaltrides.eu published one week ago a nice evaluation paper on the preclinicaltrides.eu so the experiences and so I took these numbers and added just our numbers to show a little bit how actually little is the pre-registration uptake still in preclinical research so we only have 171 pre-registrations both registries combined and most are still in an embargo and also I mean we're open to all countries over the world and we see that there's an uptake in all countries but what we also see is that we are still also working in our our little hosting countries still so we are more taking up in Germany and the Netherlands so we still have work to do to get it to get a broad uptake and so we talk to stakeholders to really also create incentives but we also talk to scientists because I think the main problem in biomedical research is still that it's unknown by most scientists they're simply not aware of possibility to pre-register and the ones they know they often also wait for a change of the system but I think we should take perhaps example on psychology and really try to get a bottom-up movement started so they can still already pre-register and make this visible so and by putting it into their publication or their applications for funding and with this I would like to end and perhaps just if you're interested also invite you to have a look at our website animalstudyregister.org and you can screen through the studies without creating any account and with this I would like to stop thank you and give to Ulf yes thank you so much Sillin yes this is a this including the the Dutch side so perhaps not all of you are aware of these so in the Netherlands it's preclinicaltrials.eu and of course also also but we'll talk about this later because there are now two platforms and we saw this in the talk by Nick as well the many platforms that that came up to to suddenly register clinical trials and Mayan also had an overview of what kind of options you have to register your study and the simplest one perhaps as predicted.com and we'll certainly talk about this later on but now I will very briefly look at pre-registration in pre-registration in in in vitro research so as I already said before there's not much going on in in vitro research and there are a couple of reasons for this and it starts with this so this is a typical in vitro paper so what you can see here it's highly relevant it's looking at the infection of human neuroprogenitor cells in and brain organoids by SARS-CoV-2 and SARS-CoV-1 and as you can see here in the upper panel if you are a bit familiar with this there is apparently an infection going on with SARS-CoV-2 but not with SARS-CoV-1 so apparently these these models are highly relevant to understanding the the disease and progression of disease because as you know SARS-CoV-2 has surprised us with a loss of taste and smell and this apparently in this model is is investigated so but now imagine pre-retro string such as study yeah so you have a lot of histology here you have a lot of graphs that show some kind of plug plug assay looking at how the growth of the virus in in the organoids and so forth it's highly relevant but still how will you pre-register perhaps this first hypothesis here up here you will perhaps be able to pre-register but the question now is how will you go about and and continue particularly and here's another example with the advent of single cell RNA sequencing how will you pre-register your thoughts about such results so here people are creating organoids that are apparently hard forming and it's about the development of these organoids and they do single cell sequencing to show that they recapitulate early on forkard development so they have a hypothesis that something should be going on in these in these organoids and they could have pre-registered but how could they quantitatively say okay this is going on in these kind of cell types here that you all see we expect a certain it's not cell types but certain genes that are upregulated in these in these cells and you see that they're doing this single cell sequencing and the question really is how much of this does need to be pre-registered and and as we can see there are many challenges here in pre-registration of individual studies so we have a plethora of laboratory techniques so we have semi-quantitative ones, wisdom gods, histochemistry, we have quantitative ones with QPUCR or aqua measuring proteins then we have very complex ones proteomics single cell RNA sequencing all the the whole toolbox of the wet lab is there and you don't even like animal experiments need to pre-register them perhaps you need some kind of ethical approval because you're using human donors as to create your IPSCs but that's everything that that you need to do so you don't even need an animal permit where you state what you're going to do with the animals but you just have IPSCs and with these IPSCs you can basically do everything that you want so and the challenges that I see here is of course that we have what kind of predictions will we make yeah what kind of our core predictions and do we want to predict everything that we do and we sometimes have this in animal experiments as well that we don't know exactly what kind of research at degrees of freedom do we really have because it's not only research at degrees of freedom and analysis but it's also in techniques so if you ran a QPCR on a certain cell type and there's this result does not well it's well it's not fit for the narrative that you have do you report it or not so we have strong problems here with reporting issues and it's pre-registration a way out of here out of this in the in vitro studies then it's about the knowledge claim so what are we claiming really in in vitro research that is helping us because it's not like in psychology or in evolutionary theory where you have an actual theory what is going on and from this theory you derive what you expect in animal research and also in in in vitro research you very often have just a rough idea what the what the disease is about think of alzheimer's think of stroke we very know well how stroke comes about but when it comes to the nitty gritty details of the mechanisms and if sound theory that makes us that helps us develop viable drugs is still missing then at the core i think is the description of the methods and protocols so methods and protocols are even more important than in psychology because here you already see in the next line ip so intellectual property is something where a lot of researchers say ah this is something that i want to guard yeah so some researchers and this is only if you talk to them privately say well we're omitting parts from the methods and results section on purpose so that nobody can recreate our organoid because we think this is a business model but we need to publish it first and then we have methods and analyses i mean a lot of psychologists are really really good at uh at quantitative methods um the wet lab researchers well i don't speak for all of them but there is at least some kind of room for improvement so confusions of biological units versus experimental units versus technical replicates it's um it's very hard sometimes even to distinguish those and if they cannot even distinguish those in a paper how will they distinguish those in the pre-registration so i can only point you to one of the projects that we're currently doing so it's situated at the einstein center for three r so we're also trying to reduce animal experiments and we have a project there that is adding to the three r three additional r the robustness registration and reporting of three r projects so we're working with five to six um organoid projects organ on a chip project to talk to the community what do they think should be registered for their experiments and what are viable ways to do this is really a platform something that they should use or it's an electronic lab notebook something that would be a lot easier for them to use to pre-register their experiments or register them at least in a larger community and make them available to review and i think we're going to talk a bit more about these problems that also apply to in vitro research to other to these other fields okay and with that i think we're finished with our presentations so i'm gonna stop presenting oh i'm doing this stop presenting have i stopped presenting already no no no there should be a little green bar up at the top of your screen for yeah i here stop okay yeah so thanks so apparently okay good so um i see there are already some questions in answer so some questions so you can vote for these questions already so um some people so just vote them up there's a little thumb up there and if you give these questions a thumb up i will ask them later but just as a just as a warm up question i want to give to the round is so what kind of concrete measures like in your field that you've seen have really made the difference for people to take up pre-registration and i would just want to ask nick first because you already hinted at some of those so what kind of concrete measures have really pushed it and um have pushed your field into taking up pre-registration yeah sure i really do think that the most important one as i like i you said i hinted at briefly in my presentation was the international committee of medical journal editors which is the top your jammas your Lancets your legends your your very top top journals in the medical field put put out a policy that many other journals in the field adopt and they said we will not publish your clinical trial starting next year unless it has a perspective perspective registration in a recognized clinical trial registry and like i said you saw you saw my chart that led to i think it was that and pretty much only that that led to a skyrocketing because you have this infrastructure you had clinicaltrails.gov in place and you even had a registration requirement from the FDA modernization act from the late 90s but uptake was very slow and there was no real enforcement and it wasn't until there were stakes for people who want their trials to be published that it really really took off and then um with the timing of it with the peroxitine all this other infrastructure then very quickly registries and other places came about as well and all these supporting efforts sort of built off of that but um i think that ICMGE can't be understated how important that was to to the process. Yeah and uh marion i think psychologists are very fond of badges but there's ambiguous evidence about about the badges so tell us what what what is this in your field? Yeah yeah so i think of course we are missing this kind of enforcement like what Nick is telling telling about that that we should should use it so but i think so and therefore it's also on on much lower levels of course the uptake than that for clinical trials but yeah i think for us the most important steps first was to create the opportunities to to do it so having the open science framework there having different templates to do it having a lot of papers that showed you how to do it in an easy way having this community of people that talk to each other and and try to really promote that and i think that set this this this first step and maybe the badges also helped and these things but uh yeah it might be even more important to first have the infrastructure and get it known everywhere and then uh yeah and then we probably need now somewhat more enforcement as well to to get it even at higher levels. Yes speaking of the infrastructure i want to um because Selene i'm skipping you here because i from your point i think the uptake is is not so big as i said in in vitro research and we come to the questions that you saw there are really good questions already in the chat but Selene um now you have two platforms in um in europe at least um or two platforms for registering these animal experiments and Mayan also hinted at and next showed that um these platforms are now kind of taken together by by a larger consortium do you think this would help because sometimes people either they don't know it and then they even have a choice and don't know which one is the best of course you'd say the the um your platform has some advantages but what do you think what how should this be done in the future do you think there will be some kind of merging of platforms and infrastructures? Yeah of course that's a good question yes we're at the moment two platforms but i think um like for the other field this is not actually a problem but of course we need to bundle the information i think it's good to have different platforms and different options so scientists can choose also and i think having just one platform puts a lot of complicated question like who should host it where should it be hosted and and it's a lot of also responsibility for for for one organization in the end but i think bundling the information so just um having having a platform which just um makes it findable it's searchable so one common platform for this or also several ones that which really bundle the information from the different registries will be important if we want to make pre-registration then really usable so um i agree that it's difficult to look at different platforms especially if at the beginning we just have so few pre-registrations and uh for your research questions you won't go to every platform and i think for the future it would be important to bundle the information but still having several platforms i think is actually good to have a choice yeah and i think at a at a conference like this where a lot of meta researchers are here all the meta researchers will be very very glad if they don't have to search through all the databases of course i mean nick and in the all trials you you had to you had to look through all the databases or how did you do this then or yeah i have um written many a scraper to scrape data off of clinical trial registries because there's so many but what's nice too and as you know the standard proliferate and different registries proliferate across different fields and if you get competing i say competing they don't really care where you register and often that decision is made for you in the regulatory schema um but the the the important part is that you have the ictrp which is sort of this overarching body that imposes a minimum standard on them of what registration should tell so you can guarantee or at least you should be able to guarantee whether you can is a different question but you should be able to guarantee that a minimum set of information about that trial will be available no matter which registry you go to now like the provision of that data and the quality of it will vary considerably but hypothetically these like 21 fields i think it is or something like that should be present on every registry you have that at least standardization and some registries go above and beyond that but i think that's a big um something to keep in mind as you know you think about a proliferation of standards across a field yes selen very briefly responding to this yeah that we're working with the other registries to define minimal standards at the moment i think that's very important you're absolutely right so that people can refer to if they want to recommend pre-registration yeah and now we're at the first step so you all show that they are platforms so people can register this is the first step but now the question is what should they register and i would refer here to a to a question by Olafur so we are so we apparently divided here between in vivo and in vitro research yes we did but of course a bit on purpose um and you already of course Olafur we know each other you already um graphs that there's more to it you asked um there's the isn't the important division to be drawn between exploratory research where multiple outcomes and degrees of freedom and analysis are desirable and confirmatory research where we are looking to restrict them now of course in vitro research will tend more to the exploratory side but there might be cases which we will have to truly confirmatory experiment so but before we go in this in vitro i think this is also a very um what can we learn from psychology here because i mean clinical trials are clearly super confirmatory nick i mean this is this is without doubt but in psychology we have the same thing um what can we learn from psychology there what can the basic biomedical sciences learn from psychology for this distinction and where would you cut cut this off mine yeah yeah i think this is a really good and important question and um i think what uh yeah what what one of the problems at least in psychology is is that we have a lot of research that is basically exploratory well we presented this confirmatory so it's also with how you report your results in a paper of course and we are really a lot of the psychology papers still follow this format of research questions hypothesis methods and so on and so on so it's it's really in this confirmatory format and and i think by doing these pre-registrations it makes this much more explicit this distinction because then you know at least that these pre-registered studies are confirmatory and also within studies you can then also um much better distinguish the confirmatory parts with the exploratory part so what you see often is quite a lot of uh even within pre-registration also an exploratory part because of course if you have the data there then it's also to go to look into um subgroup analysis or um yeah some other things um and also report that but then it's it's really clear uh this distinction and um i think that that's that's really helpful so i can imagine that that's in in in uh these other research fields it might also be good to maybe um pre-register some part to the the really uh confirmatory part but of course also do all the other more exploratory parts yeah so um a question so nick so if i look at clinical trials registries um and if i look at psychology at their pre-registrations and i haven't looked of call of course at all seventy seven thousand pre-registrations but sometimes it feels like it's even larger than the whole paper this these pre-registrations they're really like you can read read read read read the whole afternoon only pre-registrations but in clinical trials people have decided for a different route to to do this nick yeah so i think that's yeah that's an interesting distinction because in clinical trial registries they all have sort of gone this tabular format they go you know you're not so much specifying your hypothesis you're specifying your outcomes which infer a hypothesis but you're not quite being as direct right and you um and you specify basic design information but you don't get in the weeds but i think a lot of that probably comes from the fact that the vast majority of clinical trials are going to have a behind the scenes protocol a very very detailed protocol as well and we're starting to see movements in which um written regulators want you to share those as part of your clinical trial registration so under the FDA amendment act if you're covered under that law when you report your results they also want a copy of your statistical analysis plan and your protocol to be uploaded alongside them and people are doing that it's incredibly like 90% of all trials I examined um do that and I think similar discussions are happening right now in EU to about what extent and when potential protocols could be shared so I think well when you're uploading to the OSF you're uploading your protocol essentially that is your your pre-registration so I think that's the distinction there it's just these develop this sort of a tabular format but you have that other piece that's still there but it's a big movement to make that public earlier and earlier as well it's sort of a separate part of the pre-registration conversation yeah and I want to pick up here then um if on Michael's question so he thinks that it's time for enforcing the pre-registration of animal studies by rule and law and um well I mean we already at least in the EU we have a law on this that everything has to be pre-registered um in in animal studies because you you need to file in your animal study with a regulatory authority but um but it's then not not so open for everybody else so Celine do you think we should really have a mandatory pre-registration and also now connecting this if this becomes mandatory what would be your minimum requirement what would you have in there in these um because your your pre-registrations are already really really extensive so if you have to fill this out it takes some time so Celine yeah so of course a difficult question that we often uh get get this one because they are always they're the extreme the ones who don't want to do uh to pre-register at all and the ones who want to enforce it immediately but I think if we would enforce it now where it's not even known by most of the scientists and if we would be too early on this then we would really the the scientists would more react be yeah reluctant perhaps even and and just really meet the minimum requirements to just yeah get the regulation done but I think I mean Marianne has some really nice work on how the quality of pre-registration really also makes a difference and really can increase the efficiency of uh pre-registration and that's why I think if we would just enforce it now it would not uh it would not have the long-term effect of really improving science because scientists would not be so willing to to really start at least what I I believe but if we would see a mandatory so what you're saying I think um I mean I think in psychology it's it's a lot about statistics and about really preventing questionable research practices and and about making making really claims which are robust statistically but I think in biomedical research we would also what would be important is just to have uh have studies which leave a trace so so really just say okay there were some experiments and that's why if it would be mandatory it would probably be also already a step to to just know that there are some uh treatments tested and then perhaps um not make it as extensive but I think the the since most of the the the research is still exploratory also the whole statistical part is still difficult for for a lot of scientists and might still be a barrier so we are happy I mean we have an extensive template but we it's it's uh you're free to to fill in whatever you want so if you have an exploratory exploratory study you can just say okay I'm testing I don't know um opioids in mice and you don't need to to put a whole a whole statistical plan but then it's at least it's transparent that it was just an exploration so um yeah yeah and um and and then if we even if this is not mandatory I mean we have now pre-registrations even though if we only have 50 so Mayan I think this is also part of your research about the quality and I see that some some people who are attending this here are really interested in this so how we I as a reviewer I'm already I'm really overburdened even with a review process I mean I want to I want to do this very very I want to do this nicely for the for the people who wrote the paper but also be critical about it and then I even have to go back to the to the pre-registration and I remember a lot of conversations with Tom Hartwig for example also on this so how can we take away this burden also from the reviews I mean currently during the pandemic it's even hard to find reviews for your papers and then they have to look at the pre-registration to check this yeah yeah it's uh that's definitely definitely something um so uh it it's it can be indeed quite lengthy uh pre-registrations and um uh and and that's that might give some burden also to the reviewers uh and then I I've read also a lot of pre-registrations and they're also not always as as good as you hope that they are so even if they are there then it's often not really clear what they what they mean and then which variable they refer to um so yeah one thing is that I hope that if if this this the quality improves or that we are better trained in doing that that the pre-registrations are also better readable and maybe there could also be clear connections between the paper and a pre-registration I mean yeah we can also you do different types of papers of course in which these connections are are made more transparent between the pre-registered parts and the paper parts so that it's kind of more and and and yeah a nice and clear way uh yeah that we could also make it easier for the reviewers but yeah these are really steps that we still need to take and um yeah right now it's it's uh it might ask a little bit more from the from the reviewer on the other hand you could really start with this pre-registration and if you see that something is going wrong there then then you already know that that's uh yeah you can stop maybe your review and just give that back to the editor because yeah that's that's of course a major issue so Nick you want to reply to this yeah I just wanted to say I think this is a natural place for to talk about registered reports a very little bit and then how I and how I think that that is a um an area where clinical trials and are going to lag far behind the field um when you have people like Chris Chambers out there really promoting this in other areas and well uptake is I think relatively slow it is happening and it's out there but um I think clinical clinical research is going to be very resistant to this as a as a thing where and where um you know you're getting your protocol reviewed and and in acceptance because the journals aren't interested if someone bumps into David Meller they can ask how um the hackathon to send emails to all the medical journals went to try to get them to adopt registered reports but the couple emails I sent as part of that they both the they got back in touch and said we're not we're just not interested in this right now so that's an interesting area where I think clinical medicine really might lag behind some other areas and where um rigor and pre-registration could start to like eclipse it in other areas if you start to see increased uptake elsewhere yeah and um follow-up question um I see there a couple of more questions in the in the q and a and I'll come back to them we have eight minutes so very brief question to Celine so to to so to enable people to do this properly um what is kind of your what are the educational resources are there educational resources and or do you plan to develop some because I know in psychology there are some but of course this is not part of the of the general curriculum for for graduate students and um what are you doing in this respect because I'm very much interested in these educational issues Celine uh at the moment uh not we don't have any any uh resources I mean we're going to to universities and talking to scientists and also um um presenting pre-registration there and this is where I see that most are still not aware of it but of course it would be interesting to I mean especially when I see the example of psychology to really address also early career researchers and to really uh go there and at an early time point I think I mean we are planning a school together no if we also know but I don't have um uh resources at the moment we don't have but it would be nice of course to to create some I agree yeah and so um there's one question to me um by Anonymous um so in the in vitro studies um how do you think registered reports would help and I would take this as the um like um as a starting point because there's this registered report saying the pre-registration and Chris Chambers is certainly an advocate of these registered reports which I think are very well suited for what Olaf would told us before for this very confirmatory research um Mayan in your field is there any when would you when would you decide to pre-register something and then when would you decide to um to submit a registered report let's assume all journals including nature have registered reports is there some some boundary that you would set um yeah I uh I don't know so um yeah registered reports are of course then then um uh I think uh so what you see mostly in practice is that uh that that people um maybe it's an easier step to start with with pre-registration in a way that it's um you still can choose where to publish it for example or um uh you have more control over the the the time of your project because of course if you do a registered report then you uh have to submit your plan and then it will be away for some time by some reviewers so um uh so if I uh noticed that for example that that for some PhD students or people that have a limited time in a postdoc that that could be something uh so it in in that sense it seems to be easier to first do a pre-registration for people to do then immediately start with a registered report uh on the other hand I think when you really have a nice clear confirmatory study so when there is also a good theoretical base or some earlier evidence then uh then it's it's probably the very good situation to to do this registered report um but I don't know whether there yeah um whether there are other very clear distinctions in the the the research or the types of research or studies that you could do um although I think that pre-registration can be somewhat more flexible in a way that you can can can do it for different types of studies and these kind of things so it's it's I think it's somewhat more flexible and therefore maybe still more popular than the registered reports but I really yeah I would like to see more registered reports out there as well yeah yeah I I want to um as as the last question to all three of you so there are two questions that kind of align a bit and one is from Rose and the other one is from Gilbert um it's one is the social pressure that could be leveraged to um let people um have the pre-registration yes you have to do this because it's instills trust in your work and Gilbert thinks it's there should even be an intrinsic motivation to do it because it's it's kind of good for yourself I mean Mayan you're a psychologist and Mayan also did some work in on social psychology so this may perhaps even work it's better for your it's better for yourself because this is an intrinsic motivation to do it because you're just doing better signs do you think this will work we'll start with Nick then go to Celine and then to the psychologist Mayan who will wrap it up then uh whether this is possible short answers please because we only have three minutes left Nick yeah I I personally feel the intrinsic motivation should be a strong thing that's that's what draws a lot of what I do but um I don't I'm not confident it's enough for the entire field and now Gilbert and another thing also mentioned gamification and that's sort of something we've experimented with a bit true with our trials tracker project where which is for reporting not registration but um you know gamifying and trying to get sponsors to care about how they rank compared to other sponsors and how well they are doing on on a metric okay Celine yeah I also think it should be enough but I don't know if I trust the whole community also enough to do it but I think they first get to be aware that this exists to to to develop the intrinsic motivation and I also think that what Nick said with this gamification part would of course be perhaps interesting to test this and to speak to the or to to elicit intrinsic motivation yeah and now Mayan and please tell us also whether intrinsic motivation is still a thing in psychology or whether this has already been replaced I'm not so much into a real psychology anymore for some time but I think intrinsic motivation is really important on the other hand we have also a lot of other motivations that are not intrinsic so of course we have publication pressure and all those other things which might put us away from the good science and as long as they don't match with each other so if we're not awarded also for doing good science or get promoted by doing good science it will be really hard to only focus on the on the intrinsic motivation yes so thank you all three for contributing to a really nice discussion thank you for all those people attending and asking really interesting question and apologies to all of you where we couldn't answer the questions there were so many important questions here already in the chat we couldn't even bring up our poll because we had even more interesting questions there but thank you so much all for joining and I wish you a nice follow-up sessions to this conference thank you all three attendees here and from me I wish you a nice evening day or night wherever you are thank you very much bye bye thank you everyone thank you Shilin for organizing thank you thank you bye bye thanks very much to all our panelists that was a really fascinating talk and it just flew by the time and we'll be back in half an hour for the next session in the meantime please do go over to the Slack channel or the remote room for some other gatherings and see you again in half an hour thank you I could pop to the remote for a bit if anyone wants a follow-up yeah thank you so much okay I've just put the links in the chat for anybody who wants to uh to sign up if they're not already signed up so thanks very much again hey cheers