 Excellent. Well, welcome everyone. Thank you for sticking through, you know, second day end dish of the second day. So you are officially like the powerhouse of the conference. So thank you for being here. We are going to be talking about registered reports and diving deep into what they are, how to think about them, how to integrate them into your research process, etc. One thing that I'll point out is that the slides that we're using are available on OSF and there's a link to them on the bottom of every single slide so you don't have to like write it down really fast before I go. Betsy, if you wouldn't mind putting it in like the chat or something that could be really helpful. Yeah, so just a little overview of what we're going to be doing. First things first, we're going to introduce ourselves and kind of our perspectives where we're coming from and then talk a little bit about the replication crisis and linking it to why registered reports are needed and useful. We'll talk about some of the basics of registered reports as well as the benefits and a little bit of meta research on registered reports. We'll do a little compare and contrast between registered reports and pre-registration. Many of you were probably at the pre-registration session earlier today with Karen and Scott and so hopefully you can see some kind of parallels. And then we're going to spend quite a bit of time talking about practical advice of just like what, how to think about registered reports for different kinds of research and then we have a lot of resources and of course we'll have some Q&A at the end. Okay, so my name is Amanda Montoya. I'm an assistant professor of quantitative psychology at UCLA. My background is mostly in psychology and actually probably the reason I'm here is that I do a lot of meta research about registered reports. So part of what I do is is meta science and I actually have a grant from the National Science Foundation to study early adoption of registered reports and kind of barriers and benefits especially for early career researchers. And I'll turn it over to Betsy. Hi, I'm Betsy McCoach. I'm at the University of Connecticut. I'm a professor of research methods, measurement and evaluation in the Department of Educational Psychology which is in the Neag School of Education there. And I am an official registered report user. I have had my first phase one acceptance from exceptional children relatively recently and we're working on actually the submission, the final submission. So I think I'm newer to open science. I was thinking back on this and thinking I remember six years ago when Matt Makal and Matt McBee said to me that they thought that I was a journal editor at the time and they thought that we should do what I now know are registered reports but I had no idea what they were. And they explained the concept to me and I was like, what? Nobody, no journal is ever going to do that, right? And so now I think back on prior me and think, wow, I was all wet on that and I'm so glad that we are where we are now. But Amanda brings the season perspective and I bring the newly converted in the last let's say three, four years perspective. Awesome. All right. So we're going to start out just by talking a little bit about the replication crisis and kind of how it started and a lot of this is really motivating like why we think registered reports are really important. One of the things that I noticed as we were like preparing this presentation is a lot of the kind of context we're talking about the replication crisis comes from psychology. And so we'll talk a little bit about education specifically but I do think a lot of this kind of happened in psychology and then like boiled over into education. So there were a lot of things that happened kind of around 2011, 2012. So Daryl Bem published a paper in the journal of personality and social psychology, which is a really highly regarded journal in that area. And that paper essentially published a bunch of studies that all had very small sample sizes that supported the claim of extra sensory perception. And instead of convincing all of social psychology that extra sensory perception was real, the impact that this paper had was actually to cause people to kind of reflect on whether their own research practices that were common in the field were really the right things to be doing. So seeing these different studies called to question issues about were there studies that weren't reported that that he ran but maybe didn't report in the paper. So this is sometimes called the file draw problem. Some of the analyses between the different studies are slightly different, which might suggest that people are kind of adjusting the analysis procedure as they go, which could be called p hacking. And then some of the sample sizes were very even like 100 and some of them were kind of not very even like 92, which also called to question maybe they were analyzing the data and then stopping right when they found statistical significance. So there have been a lot of attempts to replicate these studies that have failed. I will note that there is one successful replication, but it includes the original author. So no kind of independent research team has been able to replicate this study. Around the same time, there was a paper that came out in psych science that talked about this idea. It's often called the false positive psychology paper, and it talked about specific research practices that are often used in psychology and education that can increase your likelihood of making type one errors. And this paper was actually like my introduction to open science or meta science or the replication crisis or whatever. I read it like right when it came out. I had a great methods instructor. This is like when I was an undergrad. And so they actually demonstrated this by running a study where they asked people to listen to the song when I'm 64. And then essentially used what they called researcher degrees of freedom to find evidence that listening to the song makes you bolder. So they tried a bunch of different TVs. They tried optional stopping. So collecting data, analyzing, deciding to collect more data or not, adding and excluding covariates, excluding cases, selecting conditions to report. So they actually ran more than just two conditions. But then ultimately, they only kind of reported two conditions. And this paper really had an impact on me because I saw a lot of these practices being used in the lab that I was working in at the time. I worked in a social psychology lab. I was doing my undergraduate honors thesis. And I was like, wow, this is just like par for the course for what I do. And I thought it was normal, but I wasn't thinking about the implications that it could have on finding effects that maybe aren't real. So this paper really, I think kind of opened a lot of people's eyes to some potential issues in the research practices that are used, especially for data analysis. In 2015, so this study actually started around the time that around 2011, but ultimately takes a long time to do a bunch of large-scale replications. But so this paper that came out in 2015 that was published in Science was a large-scale replication attempt that involved 36 research labs. They selected 100 studies from cognitive and social psychology. The evaluation of like what does it mean to replicate a study that's kind of still up for debate. So I'm reporting a lot of different metrics here. But one of the things that really surprised me was that the average effect size was about half of the original effect size. So just looking at what they found versus the original papers that effect sizes essentially get cut in half. 36% of the replications found statistical significance in the direction that was originally found. 47% of the the effect sizes are contained in the confidence intervals. There was a subjective rating of replication. And also they meta-analyzed the estimates. And in general, the major finding is like these replication rates are so much lower than we would hope for our field. Most of these are less than 50%. But this is psychology. So also around the same time, there were a couple really high-profile fraud cases. And this is, I think, one of the reasons why sometimes there's a little bit of confusion about like P-Hack, is P-Hack being fraud or like what counts as fraud because all of these things kind of boiled up in the field of psychology all at the same time. So there were a number high-profile fraud cases where they found people who just like legitimately faked their data. And all of this was kind of happening around the same time. One thing that I'll point out is that there were a lot of really strong proponents of a lot of these messages before 2011. This wasn't just like some magical time where I mean in some senses it was magical and everything came together. There are a lot of people who laid the groundwork for this. So Tony Greenwald has been writing about some of these issues since like the 70s. Jeff Loftus has been doing a lot of work in this area, especially emphasis on confidence intervals and data visualization. And then obviously Ian Edes in the medical field has been talking about this and Rosenthal as well. So I'm not to say that this like all started in 2011. There were a lot of people laying the groundwork before that. So one of the things that I was thinking about as we were preparing this is whether or not there's evidence that there is a replication crisis in education. Because there's not really the big OSF study that has replicated a bunch of studies in education yet they're working on it actually. But I think one of the pieces of information that we do have comes from actually very recent paper by Matt Makle asking education researchers about their different research practices. And so what they found in general is that researchers in education are using both questionable research practices and open research practices. So in general what they're finding is that even if we haven't done a lot of large scale replications of education research we are seeing that people in education are using the same practices that people in psychology were using that we think are kind of causing some of these issues. But we're also seeing people adopting open research practices which is really promising for progress towards the future. So thinking about kind of being in the replication crisis we want to think about like how do we come out of the replication crisis or how do we come out into the replication revolution or the credibility revolution I think. I think that's Simeon Vizier's term. And so the open science framework started which is just huge and really really important. There's kind of an increased focus on this issue of publication bias and the file drawer problem. And then a lot of these open science practices emerged. So we got the badges where we get a badge if you pre-register or use open data or open materials. A lot more encouragement of replication research. And actually specifically in education IES implemented these SEAR standards. I guess that's redundant SEAR some things. Where in these standards a lot of these open practices are recommended which is super great. So I want to talk a little bit about what we think of as kind of the symptoms of the replication crisis. So questionable research practices. So thinking about p-hacking where you do your analysis and it doesn't come out quite the way you expect. And so you're like well this variable is a little skewed. Maybe I should try a transformation or oh there's a couple cases that seem kind of funny. Maybe I should drop them. And a lot of this is stuff that we kind of do automatically that we're not really thinking about as like purposeful cheating. But it's kind of like you're doing your analysis and you didn't really have a concrete plan. And so you kind of wander around in your data for a while until you find something that looks good. And then you're like oh that must be the right way to do it. Another thing that is pretty common is called harking which is hypothesizing after results are known. And this is I love this figure of this guy kind of drawing the targets around the bullet holes. Because I think that that is just like so common is when you find results that don't align with your hypotheses are kind of automatic reaction in a single like okay what would explain what I found. You know so if you find results in the opposite direction of what you you thought you would start thinking about like well what's the theory behind that what would support that. And the issue is that when that goes into the introduction and makes it seem like that was your plan all along that's not really reflecting your scientific process. Another big issue is publication bias which is this issue that that researchers as well as journals and editors often prefer statistically significant results in publications. And so researchers won't submit publication publications if they don't necessarily have p-values that are lesson point oh five or don't support their original hypotheses. But then also even if you do submit that they might not make it through the peer review process. And then this is also related to the file drawer problem where we're just kind of keeping our research to ourselves sometimes if we think you know especially when we have non-significant results we will potentially just say well I don't think that I can publish this so I'm just going to put it in my file drawer. We should rename this to be like the Google Drive problem. Nobody has file drawers anymore. So one of the things that I think is a really kind of driving factor behind all of this is the culture of publication and the culture of academic research which is this publisher parish idea that you have to publish to get a job to keep your job and to continue progressing in your job. And the issue with this is that when publication bias is also a problem that means you essentially have to find significant results to keep your job or to fund your students or to you know to do any of these things. And so early career researchers so people who don't have as much job security seem to be a kind of higher risk of using these questionable research practices because they need to publish. I need to publish you know to keep my job. And a lot of research has shown that trainees and early career researchers can report this issue of feeling like they have to manipulate their results in order to publish so that they can move on get a tenure track job or even just keep their current job. So in general what this means is that the incentives don't really align with adopting ethical research practices because we have to publish which means we have to find significant results which means sometimes we have to pick hack which is kind of the the issue that we're struggling with. So in comes registered reports and this is what we're really here to talk about. So the idea behind a registered report is that instead of doing all of your work and then sending it to a journal what you're going to do is after you've designed your study and after you're kind of ready to rumble you actually submit a report to a journal and that's the journal that you intend to publish in like the final report. So typically what that means is submitting an intro methods an analysis plan or some journals ask for results section with like blanks where all the numbers would go and you just have to kind of look at what the journal requires but you're submitting that prior to conducting your research and this is a stage one peer review so the idea here is that actually these three sections are going to go out for peer review and the decision about whether or not your paper in full will be published is made before you have collected or analyzed your data and one thing that I'll mention we'll talk about secondary data analysis in particular but it is possible to do this with secondary data analysis which means maybe the data is collected but you haven't analyzed it yet. So the state one submission goes out for peer review there's often just like a normal kind of publication process there's a lot of iterative revisions with the reviewers and then ultimately the goal is to get what's called an in-principle acceptance which means that the journal is making a commitment to publish the paper so long as you kind of stick to what it is that you planned on doing and we'll talk about deviations away from your plan and what that means and what are those acceptable. So once you have an in-principle acceptance you will conduct your research as planned and then prepare your final manuscript you'll submit that to your stage two peer review which is typically there to just kind of check that you did what you said that you were going to do the reviewers can't at that point ask you to do anything other than what you said that you were going to do and they're really just there to kind of check that you were doing as you said and that your claims like in the discussion section are aligned with the findings that you found so there's two peer review stages in a registered report which is a little different than the traditional mechanism but still still kind of in line with with it's still attached to a journal so one of the things that we have to think about when we're doing a registered report is who needs to be involved and this is a little bit different from pre-registration if you're familiar with that which is that you are attached to a specific journal when you're doing the stage one process you're submitting to a specific journal the editor and reviewers for that journal are involved in this process whereas with pre-registration it's kind of something you do internally with your lab and one of the things that I think is super important is to get the right people involved up front so if you typically involve statistical consultants or librarians other kind of external people in your research it's really worthwhile to get those people involved as you're preparing the stage one submission so for example actually last week I had a consulting meeting where somebody said hey you know we're working on a stage one registered report and we have this really complex data analysis can you help us write the analysis plan so that's somebody who is like getting their team together very early which is ideal um planning and timing for registered reports can be kind of complicated because of that issue of the multiple stages of the of the review process and it's quite a deviation from I think what is for me and and many others kind of the norm which is that like you do all this planning and then you go straight into data collection personally I've actually found that having that kind of break in between the planning and the data collection actually is really helpful for me but I can understand this being a deviation away from whatever the norm is so one thing that you want to be thinking about is that the stage one should be time like a pre-registration it has to come before there is data collection and one of the difficulties is that you then have to account for the review period before collecting the data and so if you're planning on collecting data you know maybe you're going into schools in the fall you'll really have to be planning for that review period um before going into the schools um so for example actually that team that approached me they said we want to do this data collection in spring 2023 and I said plan for about eight months in the review period I think that's like a sufficiently long you know um window so that you're not going to like miss your time. Stage two peer review tends to be much shorter so oftentimes the stage one review process is a similar length to kind of the traditional process except oftentimes you have to shop around at fewer journals and so and then the stage two peer review process tends to be quite short and there's this nice paper by Hardwick in E&EDs that actually looked at kind of timing for publication so they looked at how long from in principle acceptance to publication and then how long from stage two to submission to publication and so one thing to notice is that this um in principle acceptance to publication includes the time to do the research right and so depending on how long it takes to do research in the field that you're in that time is going to be quite different. One of the benefits though like as an early career researcher I love putting that in principle acceptance on my CV very early you know it's like this is a guaranteed publication which is really really nice. So in education there are actually a lot of different journals that you can submit to that that now accept registered reports and so currently more than 250 journals can feel across many fields have adopted registered reports those who are kind of keeping track of the published registered reports there's about 450 maybe about 500 now and I wanted to give some examples of some journals within education that that published registered reports because this is often the thing that is such a barrier to folks is that if you don't know where to publish a registered report you just don't even know where to start so all of these journals that are listed here are education focused journals that publish registered reports there's also a number of different lists of journals if you're especially if you're kind of on the edge of education or you're looking for something that's like domain specific so the open science framework has a list of registered reports and then or journals that publish registered reports and then my lab also has a database that's kind of searchable and filterable and I'll show you an example of it later and I think that it's very useful for like finding the journal for you. So I just wanted to point to a few examples of registered reports in education I think it's really helpful to kind of imagine you know so to understand what do these ultimately look like. So AERA Open actually did a special collection of registered reports so these are seven registered reports in education and actually something that I really love that they did is that all of the authors wrote short reflections about the process and I think reading these can give you a really good sense of how people feel after doing a registered report which is like overwhelmingly positive. It's it's such a better process than the traditional process. I also pulled a couple other examples one from developmental science and one from learning and instruction and these are just if you're kind of going back from trying to look for examples in education I've actually found it to be kind of hard to like search for examples in a specific field so I wanted to give you a few that you can start with. Just in terms of a little bit of history of registered reports so registered reports started at two different journals in 2013 so Cortex which is a kind of cognitive and neuroscience journal in psychology was one of the first to to adopt this and then we've seen it adopted kind of more broadly within psychology as well as other STEM journals the Royal Society for Open Science started which was really great and then it's expanded into political science, clinical trials and just like kind of built out over and over and over and indeed what one thing that we're seeing is that there are a lot of registered reports now being published about the COVID-19 pandemic and thinking about how to include registered reports in clinical research as well. Okay this is just a little example of the database that our team has created so you can actually search based on a research area so all of the journals are classified based on their area so for example I work in research methods and statistics so I might want to look at journals that do do that and so this is a really nice way to just kind of filter out the journals that do things that are not related to what you do you can also search by impact factor whether or not they accept replications whether or not they accept secondary data whether or not they're doing a special issue and we're still building out this tool a lot so if there are certain fields that you would want a filter by feel free to reach out to me and I'm happy to to like expand out this tool because I love it. So yeah so a couple of the benefits of registered reports many of you are probably at the pre-registration section so pre-registration has a lot of benefits such as like reducing p-hacking and harking because you have kind of a record for yourself of what exactly you did it can help open the file drawer where you could see pre-registrations of studies and then maybe even if they don't get published you could actually see that you know oh lots of people have pre-registered studies looking at this effect and then never gotten them published those are probably no results and it's also really helpful for distinguishing between informing theories and testing theories in addition to pre-registration though there's a lot of benefits of registered reports so one thing that is really clear is it reduces publication bias in the file drawer problem because the decision to publish is made before your data is collected and before you have any p-values so we can't select on p-values that we don't have which is great. I've also found that that doing registered reports has improved my research design because the reviewers are giving feedback on your design before you've collected the data so there's actually a lot more things that you can change about the research that you're doing before you actually collect the data and so you have the ability to like adjust your design if it wasn't kind of ideal. A lot of journals also have power requirements so they're asking you to do some kind of power analysis to justify your sample size and that can also can improve the quality of your data. This can also be useful if you end up in a stage one submission and somebody says somebody already did that it's way better to learn that before you've collected the data than after and so you can cut ties with ideas if they're not working out for some reason but before you've invested a lot of resources in them and in general registered reports have been used to incentivize other open science practices so a lot of journals that publish registered reports will require or ask you to post your materials and data and a lot of these journals are encouraging replication research which is great so just in general I think registered reports have a lot of benefits there's this streamlined review process you are reducing wasted resources of studies that never make it out into the world and you're guaranteed a publication even if you get a no result which I think is just fab. So I wanted to talk a little bit about the the review process because I think a lot of people are feel anxious about the idea of oh I have to do two peer reviews instead of just one but I wanted to kind of demonstrate at least in my experience what kinds of comments I've gotten on research using these two different kinds of mechanisms so when I'm doing a non-registered report I can often get feedback that is pretty I would say demoralizing so you do all this work and you get peer review and then you get comments like you need to run more analyses you know this is not sufficient you need to do this crazy SEM or this other thing I did a paper recently where it wasn't a registered report and the reviewer was like you have to take this and look at moderation by all of these other variables and I was like why you can get feedback like your study is not adequately powered you need to run it again but with more people and so you're just like throwing away the data from the original people which sucks there's a confound in your original design so you have to run another study or this question is uninteresting and not worth exploring don't we already know this right so this is all pretty difficult feedback to get at this point when you've already collected data you've already like invested a lot of resources my experience with registered reports has been quite different where you can get the same feedback but at a different point in time and the impact of it is so different so when you're getting that kind of feedback at the stage one peer review the reviewers can suggest that you change your analysis plan and especially if they know something about you know doing a more effective test of your hypothesis then you're changing your analysis plan instead of like tacking on a bunch of additional analyses you can also just plan to collect more people the first time right instead of having to throw away all the all the initial people you can just collect more people the first time around if there's a confound in your design you can fix it before you collected any data which is awesome and also if your question is uninteresting at least you haven't you know collected all this data and used all these resources so the idea here is that after you've addressed all these things you can get an imprincible acceptance and then at the stage two peer review the comments that you can get from reviewers are very different so they're not focused on whether or not you did the study or proposed the study the right way but instead they'll ask questions like did you explore these questions and you could they can suggest that you add them as exploratory analyses but they would be labeled as such and you can also just say no to them at that point too if you're feeling bold and sometimes you know if your results don't align with your discussion section reviewers can also kind of correct you in that connection there to say like oh I think you're over claiming the results or you know especially like you accept the null hypothesis right these are things that reviewers can catch at stage two but again they're not criticizing the design of the study or any of the things that they've already approved so there's a growing literature of uh meta science on registered reports so I wanted to kind of highlight these in terms of talking about the benefits so there's a study that looked at the computational reproducibility of registered reports and found that registered reports are more computationally reproducible than traditional publications so these are papers that had open data and open code can we replicate the results and registered reports are more replicable or reproducible and there's quite a bit of research that suggests that there's less publication bias and less harking in registered reports registered reports are more likely to include replication research and they're particularly popular among grad students in postdocs the numbers drop off around assistant professors and that's like where I'm very focused on in my own research is like why why is there that difference um but I think it's actually kind of interesting because a lot of people I've talked to are like oh but the students the students won't want to do this and in my experience the grad students in postdocs love doing this um it's really really nice to just like have a guaranteed publication um and in general authors and reviewers overall report a really positive experience with registered reports um there's this other really great paper and I wanted to give it a whole slide um so this is a paper led by Courtney Sarberg uh who was was or is at osf um and they actually did this really neat study where they um had researchers peer reviewing pairs of papers and they compared registered reports to kind of really really nicely matched traditional papers and registered reports outperformed traditional papers on um 19 criteria so outperformed or matched um so sometimes people are concerned with registered reports that they won't be as creative or novel um and that that was not supported in this um finding and then actually a lot of things like quality of results overall quality of the paper um how important is the paper those are actually higher in registered reports than traditional papers which is very encouraging um okay Betsy is this where we switch or or is the next slide where we switch I think it's the next slide but okay okay so just to briefly kind of compare and contrast pre-registrations we haven't talked a lot about pre-registration but for those of you who were in the session earlier today um I thought it would be useful to kind of give a compare and contrast so um the the pre-registration is often a little bit shorter than a stage one registered report because you don't have to include like all the intro elements um but one thing that I find really cool is that for most journals um if you're doing a stage one registered report you can then just take that stage one and stick it in a pre-registration file and then that's your pre-registration which kind of doubly um seals you in which is nice um pre-registrations are not attached to a journal so registered reports are attached to a journal where you're submitting to a specific journal where you want to publish the timeline for pre-registrations is also a little bit more um flexible where you can pre-register and then immediately do your research so if you're in a situation where you need to do that pre-registration is a great um great alternative to registered reports so it doesn't have kind of all of the benefits but definitely many benefits which is always ideal um we talked about doing uh 20 20 percent of the effort can do 80 percent of the work right definitely like if you're in a situation where you're like I can't do a registered report for this you might as well just pre-register and then um peer review occurs twice for registered reports and then um for pre-registration because it's not attached to a journal um that will actually the peer review occurs after the full manuscript is written and submitted to the journal okay so I've talked a lot so I'm going to uh hand it over to Betsy who is going to talk to us about uh specific considerations great so um which one is right for your study pre-registration or registered reports and Amanda's already alluded to a couple of these points but I want to elaborate a little bit more um so pre-registrations are obviously can really have an advantage when you're under time constraints so you really don't have the time to send something out for review and you know get those reviews back um if you're just about to start data collection the end you can't delay it or you don't want to delay it then you don't want to be sending out a registered report right you really need to have the time to let it go out for peer review obviously a pre-registration is also a better option if you have a specific journal in mind and it's your dream journal and they don't accept registered reports because then you can't do a registered report having said that um I wouldn't be shy about asking a journal that doesn't actually already have a registered report policy about whether they might consider a registered report I know that um so I was the journal editor of get the child quarterly right before Jill Adelson and it was under Jill Adelson that they started to accept registered reports but I know for a fact that the very first registered report that was done for gifted child quarterly what was actually basically a pilot it was a it was um a researcher who came to Jill and asked if she would be willing to consider a registered report and uh Jill was already on the bandwagon for open science by the way but but so she so they were kind of already maybe headed in that direction but um they did that and it went well and everybody liked it and so then they developed a full policy about allowing registered reports to you know be one of the ways that um you know one of the ways that you can submit to the journal I would say you're probably more likely to have um success with the strategy I just described if you're talking about maybe a more a smaller niche journal I mean if it's the absolute biggest journal in the field and they've got like a five percent acceptance rate already for all their you know right then then maybe that's you know maybe maybe that's not the journal to try with but if there's a smaller journal in your field and you know you're you know sort of unfriendly terms with the editors I don't see why you couldn't try it and I do feel like Amanda said this from the perspective of the authors but uh when I've talked to editors who are using registered reports or who are accepting registered reports as part of their journal editors love it too actually um again when I I mentioned that you know six years ago when I heard about this idea I said oh my gosh that sounds crazy two sets of reviews oh no but but the reality is that it is it is actually good it's good for the journal too it's good for the journal and editors tend to really when they try it they tend to like it so okay um and then you know you can't change your design and if you can't change your design and methods at all in response to the stage one review so you are already locked in with what you're going to do then then you should consider preregist what you should preregister you should not submit for a registered report so what might be an example of that well let's say this is your dissertation and it's already gone through your dissertation committee and you know they you don't have the flexibility to actually change your design and methods because this is the thing that was agreed upon by your dissertation committee so that you know that might be an example of a time when you might not want to submit for a registered report um having said that well I mean I think that's true but I'll just say that I think that one of the reasons that I really like this idea of a registered report is that I think it is actually very analogous to the exact process that we use with doctoral students when we have them go through the dissertation process so the phase one or the stage one is very much like a dissertation proposal and the big difference though is instead of having your committee review and you know give you feedback and it's instead you have these you know three or four anonymous reviewers who are actually giving you feedback at that stage but I've always felt like um and I've been a major advisor on lots of different dissertation committees and you know I think I like to think of myself as a pretty good researcher a pretty smart person but I can honestly say that 99 plus percent of the time um when you get three really smart people in a room together you know when you get your committee or five however many there are when you get your committee in the room together usually the study actually gets better right as a result of lots of people being able to give some feedback about the study and obviously this this process is a little bit different you're not actually getting in a room with your reviewers but I think it it actually functions in very much the same way so it's it's you know getting more feedback and having external eyes be able to say at the design phase hey did you think about this hey you might not want to do it that way I think it's so helpful and so many times that I've been a reviewer you read a paper after it's already been done and you think oh gosh this would have been so much better except there's you know there's this one fatal flaw that they wouldn't have had to do but they did it now they have the data and now it's sunk right and and here you really have the opportunity to make that design better so okay so I was supposed to be talking about when preregistrations are better and I just went off and said all these great things about registered reports but um so so those are some times when you're more likely to want to use preregistrations registered reports are a better option obviously if you still have time to make changes to the study so how do you think about the timing of that a little bit you know I mentioned earlier that we had a we have a phase one acceptance at exceptional children and our stage one sorry same phase we just we have a stage one acceptance at exceptional children and it's actually for a secondary data analysis and we were getting data from multiple um multiple large school districts uh as part of as part of the study and so what we did and it's actually a grant funded project so once we had the grant funding the first thing that we did is we sat down and we worked on the registered report at the same time that we were working on recruiting the districts because it can take a really long time to recruit districts and so when we wrote the registered report we actually created a table that said you know we are reaching out to like these 10 districts here's what they look like you know here's what you know we gave um information about the the demographics of those districts and then said we will start and you know obviously after the registered report is accepted but we will start data analysis and start the study once we have secured permission from and data from at least five of those districts right so so we didn't know exactly which districts we're going to be in the study but we said you know we'll have at least five of them this is what the 10 look like you know sort of you you judge if you're comfortable with that and the reviewers the reviewers were and that was really nice because we were able to actually be you know sort of circulating back and forth with the registered report and we actually had a revise and resubmit to the registered report and so at that point we had actually really gotten much more clarity about which of the districts weren't going to be participating and so in our revise and resubmit for the for the stage one we revised that table and said you know which wasn't something they asked for necessarily but said okay you know when we first submitted this we thought it would be five of these 10 now we actually know it's going to be these five this is what we've got this is how we're going to move forward so that you know worked out really well for us and the timing was really good because we we kind of had dead time anyway because we were trying to get the data from the districts and as you know one or two districts came in we just put the data inside and said we're not going to look at any of these data right until we so we've got our acceptance so that that worked out really well so sometimes you can be I think a little bit maybe more clever about thinking about how you set the timing of your study so that it isn't such a long lag time that that becomes problematic um obviously there are advantages to both in the sense that we're making our hypotheses transparent people often think that you know once you do a pre-registration or a registered report that your hands are completely tied and you can't make any changes which obviously isn't isn't necessarily true I mean sometimes you do have to make changes uh for one reason or another for example you know let's just say we hadn't gotten the five you know what if we had only gotten four districts right instead of five you know that would have been something we would have had to change but um so you might have to you know you might have to make changes you don't I'm kind of getting ahead of myself because we've got something about deviations but um you can you can also add exploratory analysis in if you want to explore interesting findings if you have to make deviations then obviously you're going to make that transparent so I think the big thing here is whatever you're doing you're specifying up front you're being really transparent about your plans and if your plans change either because you want to dig deeper in some aspect or because something's happened that you didn't anticipate which by the way anybody does research in schools knows that unanticipated things can happen then you just make all of that really transparent and you explain the logic of it all right so next um yeah so we talked about time management the time really does operate a little bit differently with registered reports so you know the sooner you can start writing things up for your registered report the better and if you think about even when you're writing grant proposals the grant proposal itself is in some sense the start of a registered report there are lots of pieces of a registered report that are very similar to the pieces from a grant proposal so again when we were doing our registered report we went we didn't do it at the same time as our grant proposal we actually did it right after but we went back to our grant proposal and the verbiage from our grant proposal became the starting point for the registered report but what I really appreciated about the registered report process was that even though I felt like we had a lot of detail in our grant proposal and we did it really forced us to think through a few things that maybe hadn't made it into the original grant proposal so we got even more specific and and sitting down as a team and really working on that registered report together was so helpful we had this recipe we knew we had these hard conversations before we ever had data and then we sort of we all got on the same page and we knew exactly what we wanted to do so I can't really say enough about the process it was so so helpful um and I will also say this we found we said that we would pre-register you know we we specified in our grant these are the studies that we're going to pre-register and then these are the studies that where we're going to do registered reports and a lot of that had to do with the timing of it like we also have an experimental study going on as part of this large grant um and they're just they're just too many moving pieces and by the time we get everything figured out there isn't enough time to go back and forth with the peer review so we're going to pre-register that study instead of doing a registered report um but the IES which is our was our grant is with IES they really really liked the fact that we were promising to do registered reports or and or pre-registrations for our large studies so I think you know having these having initial phase and when we told them we had the initial phase one acceptance um they were super super excited so it really looked good on our grant progress report um so you know there are some things you can think about in terms of creative solutions for you know timing in terms of registered report um you know you might be for example maybe you're doing a study where you're you know collecting pilot data before you actually launch the larger study and so maybe you're writing a registry you know maybe you're writing registered report for the larger study while you're in the process of sort of developing the intervention or writing the pilot data um and you know you do need to think about you need to think about the timeline for review obviously I think that's important um I was in a session yesterday actually where the editors from exceptional children were talking about the fact that they're very sensitive to the fact that you know when you put something in for this for stage one that you are basically you know not going to do any any research until you've gotten that acceptance and so they said they a try really hard to kind of accelerate the timeframe as much as they can so they kind of push the reviewers to get those review reviews back in a really timely manner and be um the idea came up yesterday sort of in that session that maybe it's a good idea for the journal editors to actually be asking about the timeline that people have so that they can you know when those registered reports come in so that they can make sure that the timelines are being met okay next slide um so this is kind of just another whole issue in terms of timing which is the IRB and the registered report now every place has a different kind of research compliance office I don't know what yours is like where you are but ours is mighty tough and they really want every single detail specified and if one little teeny tiny thing changes it means that you need to go back and submit an amendment to the IRB obviously you know this is somewhat dependent on a the kind of research you're doing is it you know is it does it require full review or expedited review or is it exempt right so obviously there's a little bit maybe more leeway with some of the exempt studies than there are with other types of studies and it also depends on your research compliance office but if you're in a place where once you have an IRB in place that any tiny deviation is going to cause you to have to go back and make major amendments which is then going to add more time to the whole process then you might want to think about doing the registered report prior to submitting the IRB right and that might sound like oh my gosh now you're telling me that I've got to do this registered report and then the IRB and then the study and again it depends I guess on how long your IRB office takes to get results give me to get feedback to you because you know that could be your major sticking point but I think the good news is that what your registered report looks like is probably going to be very similar to what your IRB protocol looks like and again our IRB IRB office always appreciates extensive detail so I mean they I've never had the IRB come back to me and say I'm sending this protocol back because it's too long and you've given me too much information like that just doesn't happen right so once you've kind of thought through everything so the registered report then you can actually then you have something that you can submit to the IRB and with any luck all of that good thinking that happened in terms of you know really sort of getting this feedback and thinking through all of these issues will maybe translate into getting through the IRB more quickly again no promises depends on your IRB office so you but the point of this slide is really to say you really need to think through which comes first the IRB or the registered report if you've got you know as an example I'll keep going back to my example from our study in our study actually because we were getting deidentified data so we had no identifying information we did go through our IRB and our IRB said actually this doesn't qualify as human subjects research we don't need to see it again so in that case actually we we did the IRB first and that was great because we had a lot of leeway and so we went IRB and then registered reports so which which comes first the chicken or the egg really depends on your you know what you're doing what your design is what your IRB is like but you do want to think through the timing of this and be deliberate you don't you don't want to get your registered report accepted and then or you know you don't want to get your sorry you don't want to get your IRB accepted then have to make major changes because because of the registered report process and then have to go back to the IRB again if you can avoid I always have to go back to the IRB multiple times even without that step so all right so analysis planning oh my gosh it's 157 analysis planning is uh harder than you would think um because you know just because you plan an analysis in advance doesn't mean it's an appropriate analysis you really need to think through right what what's you know what you're doing this is more problematic for pre-registration by the way because you know now the onus is on the reviewer if you've come up with a really bad analysis and three reviewers say it's fine and you get an in-principle acceptance well then you're home free right that's the advantage of the registered report having said that we mean you know we don't necessarily want that to happen but um so so think about that and uh you know things that you definitely want to think about as you're planning your analyses in advance I think sometimes people are pretty good about thinking about the big pieces of the design but they're not necessarily so good at thinking about the little nitty gritty details like oh you know they might think about how their variables are measured in terms of like what subscale I'm using or what you know what question I'm asking but a lot of times people don't really think about how they're going to combine different items or different measures what they're going to do if they have low reliability how they're going to code certain types of items or variables and and all of those things I think are really important to be as explicit as you can and you know what do you do you know other things that can be trickier what do you try to plan in advance what are you going to do about attrition what are you going to do if people you know leave uh what are you going to do about careless responding and all those kinds of things all right next slide um so practical advice in terms again I think that generally speaking reviewers take a little bit more of a collaborative stance with the registered report because they know that the suggestions that they make can be implemented and so it feels a little bit more like there's a team there's a little bit more of a team effort in terms of the registered report rather than a sort of a I got you you should have done this and you didn't do it um so I think that that can be a real advantage again you know that doesn't mean that there isn't the occasional reviewer out there who wants you to do the study that they want you to do instead of the study that we've planned or the study that you're thinking of so you know this doesn't actually mean that it doesn't solve problems with reviewer number twos of the world right but in general I think the reviewers kind of come at this with a little bit more collaborative perspective we talked about the fact that conference proposals I mean grant proposals and conference papers can be a great a great way to start and you know if you're not sure if your study makes sense as a registered report it can be a really good idea to ask the editor of the journal that you're considering so what I well I know we're at whatever time of the hour it is now so I don't know yeah should we go a few more minutes or should we stop I think you can go ahead a few moments if you'd like okay I'll keep it I'll be less verbose now I will I will run through these slides quickly so we did a little bit of thinking about you know different types of studies and maybe how they lend themselves to registered reports and what might be some landmines along the way so I think it's obvious that experimental studies actually really lend themselves well to registered reports you lay out your experimental design you know from design to analysis prior to collecting data you do your power analysis right you lay that all out so I won't belabor that I think experiments really lend themselves nicely to register reports so next okay descriptive and survey research you might be thinking to yourself well why would I want to do a registered report if it's just a descriptive study but I mean I think you should right there's no reason not to and and even though it's a descriptive study you still really have a lot of thinking to do about your measures your surveys your instruments right what are those going to look like and also I think another really important thing if you're thinking about descriptive and survey research is your sampling methodology and your sampling plan right so how are you actually getting that sample and again you want to have a detailed data analysis plan next for secondary data analysis the whole hardest thing is if you already have the data no peeking right so you are going to do your registered report and you're going to get your in principle acceptance before you actually start to do any of your data analysis but I think secondary data analyses actually are very appropriate for registered reports and in some sense the timing issue is is a little bit easier if the data already exists versus if you're about to go into schools and collect data if you've used the data for some other study and this happens a lot with secondary data analysis I mean think about the ECLSK and other large you know large databases are used for hundreds of studies right but if you've used the data for another study then you're going to want to include an author statement about the data where it's been used before and for what purposes in the in the registered report at stage one and that will be really important for the reviewers to be able to decide things like is this actually a different enough study to merit another publication as well as you know have they done things that might compromise this actually being a registered report next okay longitudinal studies I mean obviously if you're going to do a study that's you know several years long you probably don't I mean you know could be rough to do a registered report and then wait for eight years to actually publish your results right so Amanda and I had a long talk about this actually yeah with any longitudinal study you're usually going to be publishing you know in phases anyway right so you could be putting together a registered report for some aspect of the longitudinal study for example you know maybe you've collected baseline data but you're going to be collecting you know some follow-up data at a couple of other time points right and you're you know you're going to write a registered report about this follow-up data things to think about if you've got longitudinal data if you're thinking about modeling longitudinal data you really need to think about the functional form of those data and anytime you're doing any kind of multi-way study you really have to think about attrition all right next replication studies well of course replication studies really lend themselves well to registered reports and one thing that I that we did want to mention is that there are also these things called registered replication reports so am pps advances and methods and practices and psychological science actually publishes registered replication reports and this is like a collection of independently conducted direct replications of an original study that share some shared predetermined protocol so often there are a whole slew of authors on these registered replication reports because there are multiple labs that are running multiple direct replications of the original study I feel like you see this less in education than you do in psychology because you know a lot of times what we're doing are more field-based or quasi-experimental studies that maybe are harder to do direct replications and harder to get many labs but I actually think that we should think a bit more in education about how we could take this idea and use it in education because because we do often have these really do need to run these large efficacy or effectiveness studies and the idea of sort of having you know having multiple labs sort of or not labs we usually form labs but having multiple groups work on replications I think is actually smart okay next qualitative studies well I don't have too much to say about this because I don't do a lot of qualitative research myself honestly but I think sometimes people think oh well if it's a qualitative study it can't possibly be pre-registered or it can't possibly be a registered report and I don't think that that's true because just like a quantitative study you can be thinking through your your design and your methods so the things you would want to include are things like the theoretical expectations you have the qualitative tradition you're coming from what's your sampling plan how are you going to get your I mean sampling plan is different here than it is in a quantitative study right but you still have some idea about how you're going to recruit participants for your study how are you going to collect the data what are those interview and observation protocols look like are there stopping criteria a lot of times in qualitative study you're looking for some you know right there you're thinking about when you have data saturation so you specify that in advance and then what kind of coding system are you going to use what kind of data analysis techniques so you can totally specify all of those things for qualitative studies and then the one little bullet point at the top is it's kind of interesting about less than half of the qualitative research presented at conferences eventually gets published so cloud drawer or google drive next meta analysis so again i think meta analyses are natural for registered reports but things that you want to make sure that you include in your registered report are the search strategies right how are you going to identify the studies for your meta analysis which and inclusion and exclusion criteria which obviously are two of the most important things that you think about in a meta analysis and then finally also you know how what are you going to go for how are you going to go and non registered reports so there might be studies that don't really lend themselves very well to registered reports one example that i you know i was thinking about are things like instrument validation so it's hard enough to get an instrument validation published anyway you know there are a lot more instruments that are validated than people that want to publish the eventual articles but like if you go through the whole instrument validation process and your instrument is not good like it doesn't right you don't have factors the reliability is poor you don't you know you have an idea about what you want to measure but the whole thing kind of falls apart in the validation process which happens all the time especially the first time around when you're creating an instrument nobody needs to read that right and so if you think about like a registered report yeah you could specify all the things you're going to do and maybe all those steps are right but if at the end of the day the instrument is crap like that's not really a publisher piece so that's i think a classic example of the kind of research that we might do that doesn't really lend itself to a registered report doesn't mean you couldn't pre-register the study though so remember that other things are things like pilot studies maybe you're doing small-scale development work right maybe that doesn't mean to you know that's not going to be registered before either all right done with that slide oops um so i think we've talked a lot about these you know how how long will it take and will it take longer i don't think it has to take longer overall really um but you're you're spreading the timing out in a very different way than you are with regular sort of a standard research project you talked about qualitative research are you more likely to get scooped i don't think so and remember that you you know you've got the ink on paper already in some sense i feel like you're even with pre-registration and with registered reports even more so i feel like you're less likely to get scooped because you've already got down on paper what you're planning to do whereas in the old days maybe you go to conference you have a couple drinks you talk to somebody you tell them your idea you don't have it pre-registered you don't have a registered report and next thing you know somebody has stolen your idea and they're running with it so i really think that both registered reports and pre-registrations are good insurance against getting scooped rather than the opposite um and how do you get your research team on board i don't know have them watch this have them watch this awesome webinar and then they'll they'll totally be enthralled um and then can you put multiple studies in one paper you know this is something i always struggle with in general like what's the right brain size for a paper often we have these larger studies that are so large you can't actually fit everything in one paper but i think when you think about registered reports think about the grain size of the registered report as being the same as the grain size for the eventual publication so you could actually have multiple registered reports coming out of one large research project all right um we've talked about this a bit um you know you want to make sure that everything that you've got at stage one is as detailed as possible it's really important that what you're you know i i i like to think of this as like if you if you were to just sort of disappear somebody else could pick up pick up the registered report and they could do the study the way that you intended it the other thing that i think makes a registered report different from a pre-registration is that you really need to pay more importance to the significance of the study the theoretical framework in the lit review you could do a pre-registration without giving a lot of um you know a lot of attention to sort of the front end of the manuscript but again this is going out for review and as Amanda said one of the things you need to do is convince the reviewers that this is a study that's important and worth doing and the only way to do that is to really be you know to really talk about the theoretical the significance of the study the framework and the lit review so don't give short shrift to that i think you we really spent as much time on the front end of the paper as we normally would for a publication the nice thing is though now it's written right i'm sure we'll probably revise a little bit um our stage one report ended up being as long as our eventual publication by the way because there's so much to say all right next um so if there are any deviations if something happens and you can't do what you planned you need to let the editor know as soon as possible some very minor deviations might be fine but you know there in the worst case scenario there's a really large deviation that might invalidate the stage one acceptance so i'll give an example maybe you say you're going to recruit you know 200 people and randomly assign them treatment and control and you give the power analysis and all that stuff and you were only able to get 20 people for your study well okay it may have been a phase one acceptance but that was predicated on the idea that you had 100 treatment and 100 control not 10 treatment and 10 control right so um you know you you you have to do what you say you're going to do and again if you end up with 198 people is the editor going to care probably not but if you end up with 20 I think that's going to be a huge problem next um you know a lot of people have said it comes up in those AERA open reflections and it it comes up it came up actually in the the editor's comments about that AERA open special issue that people feel a lot freer writing the discussion section once they have the phase one acceptance because they feel like they don't have to worry about whether the paper's going to be rejected or not so um you know there's this is a quote we felt the discussion became more honest and vivid as we could for example give sharper opinions so um so that's so that's kind of nice the other thing to keep in mind is if the phase one or the paper ends up being as long as a regular article sometimes what you have to do is sort of trim out some of the detail that you had in that in your first paper and some of those things might go into the online supplement so they still need to be available to everybody but sometimes there might be a level of detail that you have there that for space reasons ends up going online and then this question will the manuscript be peer reviewed again or will it go through editorial review only you should assume that it will be peer reviewed again there is the possibility that you know if things are really straightforward the editor could choose to make an editorial decision but you should just assume it's it's going to be peer reviewed twice but the second time around usually goes faster and we have lots of resources that you would be better off looking at the actual slides for rather than trying to copy down now especially given that we're way over time and I guess we don't have time for questions but I guess that's okay because we can't see anybody anyway so thank you so much yeah thank you I think if we don't get kicked out of this room I'm happy to like stick around and if people have them I don't know Leslie tell us if we will get kicked out no that's fine I don't think you'll get kicked out and also if you do if there's anybody in the audience who would like to ask a direct question if you raise your hand I can and I can let you speak there's a question in the Q&A Jonathan if you're still here I'm not quite sure I understand the questions I would love to hear you articulate it maybe okay go ahead Jonathan hi Amanda and Betsy thanks for the presentation yeah so I guess I didn't give a whole lot of context regarding what I was talking about with that I kind of mentioned my question the basis of it is still thinking about the public repairage paradigm when you know a lot of the the university hiring and funding agencies based heavily use agent decks and impact factor and such you know and obviously it's a whole kind of connected system where you know quote-unquote significant results are a really huge part of that entire conversation I guess I just like because it's not science is you know naturally fairly decentralized and you know you have the authors you have and then you know this huge support network to kind of like have the findings of the scientific community be communicated as well and funded etc etc I guess the idea is right now we're trying to fight this public repairage paradigm which is largely you know you know the poor incentives from you know just the inclusion criteria and the marketing kind of capabilities of the journals how do we kind of like also get the you know universities and the funding agencies more on our side and communicate to them so that they're actually on their checklist kind of pushing for this type of inclusion because like you know we have agent decks and impact factors just wondering whether or not you can summarize the activity of you know how people are using preregistration such that you can kind of get on board this process a little bit quicker yeah I mean I think one of the reasons why I think of registered reports is such a stronger solution than preregistration is because it still allows you to publish without perishing that like this idea that like oh if I pre-register all of my studies you still might not get published but registered reports that's not the case and so I find it to be a really optimal solution especially for early career researchers who maybe don't have the you know the pull to like change their system even given that though I have found myself like in my you know reports that I submit I'm an assistant professor I'm still like untenured I have like a footnote that I put in my in my statements that says like what a registered report is and what this like in principle acceptance is and why I've put it in this specific section on my bibliography but I think you know one of the things that that our group is hoping to do is actually do some trainings with like the tenure and promotion committees at universities to teach them like what registered reports are and why they should maybe get like a little bonus kind of view you know it's it's a more it's a more rigorous process so so yeah I mean I think part of the reason why I like registered reports is that you still publish but but I do agree I think that needs to be a lot more done at the system level to be recognizing all of the work that that people are doing to improve transparency and openness yeah thanks for addressing that I was actually mentioning earlier in a previous like I forgot to talk about like how you know we could talk to tenure committees because like it's easy it seems a lot easier to onboard the younger you know like graduate students and such because like but it's the more experienced researchers that seem to be a little more hesitant just because it's already so difficult and on top of it you know registered reports and such and open access and all that such like it doesn't necessarily help their odds to to make the cut in fact if they start to focus on you know the um polishing in these these these novel uh platforms that might even hurt their chances because they're yeah because the yeah criteria so um the competition is so stiff and it's yeah they need to work on those numbers yeah I think I think one of the things I think is really interesting we do a lot of trainings on registered reports and especially for those in the medical field there is this um question that's like well if I do a registered report and I find something like amazing can I withdraw the paper and submit to nature or like the land say I don't know good really good medical journals and I find it so funny because I I actually think about registered reports is like the opposite way like if I am ever going to get a nature publication it's going to be cut it's a registered report um and it's like I think actually way easier to get into some of these big fancy high impact journals through the registered reports mechanism because they are adopting them um all of the natures do it now and so like if you're ever going to get in at nature you might as well do it through a registered report um because otherwise you know you can submit but you'll never get um or maybe I don't know but like that's kind of my mentality is I think some people are worried they're gonna get in worse journals doing this but I actually think you'll get into better journals personally yeah cool thanks for the other questions I also have another question but like you know give someone else an idiot I'm muted okay so um I wanted to say something about your last question before you say ask another question which is you know I mean every tenure committee is different of course right um but I've been I've been on I've been around a long time I've been on PTR committees at the school level and at the department level um and you know I've mentored people through the process and here's what I'll say um I think that junior faculty often get really really really focused on numbers like they think that there's some you know like magic number I need 12 right I need 12 clubs by tenure but the reality is that what the PTR committee is looking for is evidence that you are going to continue to be a successful scholar that if I give you a job for life that you are not like going to disappear into the ether but that you are going to be successful and you are going to have a good career and so I mean from that perspective somebody who's got a whole bunch of registered reports with phase one acceptances right you know that's a good bet right somebody who because because that's a guarantee that that is coming out and so I think that um you know again there's no guarantees about what's going to work and not work for given tenure committees but but I think that it provides this evidence that you're on the right trajectory and that's ultimately actually what the committee is looking for yeah thanks that was uh yeah that's that's that's very good to know um yeah uh I have to admit that uh upon hearing what you had to say I totally forgot what I was going to ask so yeah maybe you'll come back to me but if not uh yeah we'll uh we'll just move on thanks thanks for answering my questions sure there are a couple questions and comments in the chat about kind of the the timeline and and one of the things that I've experienced is that it feels really different to do a registered report because there's kind of like I'm used to a framework where it's like hurry up and then collect the data and it's all kind of like this one thing and all of the waiting happens like after you've written the paper and you can kind of mentally move on from the project and so it feels like the registered report takes longer because that waiting period has moved to the middle of the project um and that's something that I've had to kind of like restructure my lab around like building in more redundancy with undergrad training because somebody who is around during the stage one preparation might not be around when we actually collect the data um and so it's it's I think it's worthwhile to like give it a try the first time let yourself sit with the uncomfortability but then like at the end it actually like it's so great to just like collect the data write it up and then it's like published so fast um you know so much faster than like shopping and around at all of these different journals and um our lab is actually trying to collect a little bit more information about timeline um because one of the things that's kind of difficult is we do have information about like when it's submitted to the final journal what was the timeline but we don't have a lot of information about you know our registered reports submitted to a lot of different journals before they find a home or is it that you know you kind of submit to one journal and it makes it in there um so our lab is trying to collect data on that actually now um as a registered report about registered reports um but it is it is all kind of open information and we're relying a lot on um kind of just uh people's stories I think I don't see any other question neither all right well given that we went right a bit over I guess we'll wrap it up here thanks everybody thank you both that was just brilliant I loved all your little cartoons that it made it very entertaining as well as informative so so thank you both relevant cartoons that's that's what we aim for that's right thank you thanks so much so much