 Well, we can get started with introductions as folks trickle in and Karen and I can do our origin stories. And then I'd love to, for those of you who are here attending, if you guys could write some brief introductions to who you are in the chat, so we can kind of take a look at those as we're, Karen and I are going to kind of trade off talking a little bit. Yeah, that would be wonderful. So I'll get started. My name's Amanda Montoya. I'm an assistant professor of quantitative psychology at UCLA. I like to say that I grew up in the in the replication crisis. So I learned about the replication crisis when I was in like my junior year of college was 2011 and my stats teacher at the time was really engaged in what was going on. And so in our in our advanced method seminar, we read the false positive psychology paper and a lot of the like scientific utopia papers and got really kind of into a lot of this content. And and I was one of those undergrads. I think that we have learned about in the plenary session that was just like, oh, isn't this how science is supposed to go? And I've never lost that in my kind of growing up through this system. So I've definitely seen the barriers to it and understand why sometimes we shift from that mentality of, oh, isn't this how science is supposed to be? To to kind of realizing the kind of realistic situation, situations under which science needs to be done. But but it's been kind of a pleasure for me. And so my studies have been in mostly quantitative psychology and thinking about meta science and how that how quantitative psychology and meta science are really kind of reaching towards the same goals in many senses. And then I will turn it over to my co-presenter, Karen. Sure, I'm Karen Rambo Hernandez. I am an associate professor at Texas A&M University. I'm in teaching, learning and culture in STEM education. And I have a joint appointment in educational psychology and research measurement and statistics. And I have to take a big breath after that because it's really long. Um, my intro into open science was much less noble. I was I had just finished getting a paper through the publication process at a particular journal journal, and I had gone through probably at least two or three rounds of reanalysis of the data to satisfy reviewers and the paper was better as a result. But I was really tired and I didn't like the reanalyzation process. And so on the heels of that, I was having a conversation with my friend and colleague, Matt Makle, about a new project that we were going to be starting. And I mentioned to him about this long process about reanalysis and the paper we were going to do is involved a huge data set. And so it was really kind of dreading the potential of having to reanalyze this massive data set multiple times. And he said, well, what if I could promise you you only had to analyze the data one time that was like I'm in. So it wasn't something noble. While it's more it's morphed into something a little bit more noble. I got into it simply because of the I only have to analyze the data one time instead of multiple times. And then all the benefits that come with only analyzing the data one time. So yeah, so we're going to talk about open science, specifically in pre-registration and registered reports. Amanda is going to do the lion's share of the talking and sort of getting into what pre-registration is and what registered reports are. What I would encourage you to do if you have questions as we're going along to use the chat and then I'll like throw them over to Amanda or if I'm talking she can throw them over my direction so that it's a little more interactive. So stop us if we're like off in some world that doesn't make sense. So all right. And with that, I'm going to pass it back to Amanda because she is going to get us started in kind of understanding the basics of the two. Awesome. Yeah. So we've done our intros. I will introduce some of the basics of pre-registration and registered reports. Karen's going to talk about a little bit of kind of comparing and contrasting deciding between the two. How do we think about that? And then we each have some kind of practical advice based on our own experiences, some resources. And then we're hoping to have a kind of extended Q&A with this whole group to kind of talk about what are some of the questions that come up in thinking about these. So I will get us started. So essentially for for both pre-registrations and registered reports, I'll go through the what, who, when, where, why. I put what first, because it seems kind of silly to talk about who and before what. So so what goes first this time? So pre-registration is essentially a process where prior to collecting your data or doing your research, whatever that process looks like, you create a public registration of your research plans. And when I say public, I don't necessarily mean that it has to be public. The moment that you publish it, there are processes for doing what's called embargoes where you essentially say, OK, this is time stamped, but but I want to keep it under wraps for a little while. And this helps with some concerns around maybe like scooping or something like that. But the main thing is that it's time stamped so that you can say, OK, I did this before I actually collected the data. And it typically includes things like your hypotheses, your plan for your sample size, the key variables and how they're measured and hopefully also how they're aggregated, any experimental conditions that you're planning on running, exclusion and inclusion criteria and the analysis plan. And so the the purpose of a pre-registration is to kind of in some ways, I don't want to say tie your hands, but also kind of tie your hands to keep yourself from doing very flexible things, which is one of the the biggest issues going on with the replication crisis is we think there's a lot of kind of research or degrees of freedom going on. And so what this plan does is essentially sets you up to kind of say, OK, this is my plan and now I'm just going to follow this plan. The who behind pre-registration is mostly you and your internal team. So the one of the benefits of pre-registration is it doesn't have to involve the journal that is involved. It doesn't you don't need special permission from anyone to do this. You could do this for anything that you want to do within your team. So it's all kind of an internal process. All of the authors who are involved in the project should approve the pre-registration before submitting just because you don't want to end up in a situation where one person wrote the pre-registration and then later one of the other authors is like, well, I think we should have done it this other way. So definitely run the pre-registration by everybody who's involved in the project at that point in time. And then because my experience is maybe a lot like Pam Davis Keans, I'm often the person who people come to later when they have data and they they want help. And one of the things that I would recommend is that if you tend to to reach out to a statistical analyst or a librarian or somebody else who works on your team, but typically after the data is collected, I would recommend getting that person involved early and involved in the pre-registration so that so that they can kind of weigh in on whether or not the plan kind of makes sense. And the when that goes on with this is this needs to happen prior to collecting or accessing your data. So for some of us, we do data collection in our labs. Some of us are working with secondary data analysis. Karen has a lot of experience thinking about pre-registration and registered reports for secondary data. So she's great for those questions. But basically the idea is that you want to make a plan before you do the plan. If you write the plan after, it doesn't really help you at all. And sometimes there are justifiable reasons to pre-register after data is collected, but before it's analyzed. So sometimes, and I was just talking to somebody this week where they pre-registered a study. Stuff went kind of wildly terribly wrong during data collection. It was COVID problems. So they had the data, but they read the data but they realized like they didn't follow their data collection pre-registration at all. And what that did is it changed the way that they would analyze their data. And I said, now what you can do is you can actually do a second pre-registration, tie it to the first one, explain what happened the first time and say, okay, this is our new analysis plan. We haven't looked at the data at all and then go from there. So there are situations in which this can happen, but I would recommend just in general trying to default to pre-registering before collecting data or accessing data if you're going for a secondary data analysis. And then one of the things that I recommend is to give yourself time to do a pre-registration. And I remember very early on for me going to workshops kind of like this and people said, oh, pre-registrations are great. You can do them in like a day. And I think that that's appealing and technically you can, but from my experience, I would say if you're not planning a study in a day, you should probably also not write your pre-registration in a day. And what I mean by that is that if writing your pre-registration is, I guess, it's really the thinking through the plan. If you do the thinking through the plan, like, well, before you do the pre-registration, then yeah, you can sit down and write it pretty quickly. They're not typically, sorry, they're not typically terribly long, but it's something that you want to put a lot of thought into. It's not something that you want to just kind of write out and then say, okay, because later you might regret the plan that you made. So what I would recommend is that you can, you can write it out, you can pass it around with your fellow authors and make sure that everybody agrees on this. And think of it as time saving for later. So you're really making an investment in future you, where later on when you are writing about these things, when you're thinking about how am I going to do the data analysis, you've essentially just already written out how you're going to do the data analysis, and then you can just follow that plan, which I really quite like. I'm one of those people, like, once the data's in the door, I'm like, okay, I want to analyze it right now. And so one of the things I really like about pre-registration is that if I've already kind of thought through how I'm going to do the data analysis, then I can actually do that. And sometimes I just write a whole script in advance so that I can just run that the day that I get the data, which is always very exciting. So where will you pre-register your research? There's two primary places, one of them is OSF and the other is aspredicted.org. I've done a little kind of comparing contrasts between the two of these. I would say OSF gives you a lot more freedom. There's different formats you can use and then you don't even have to use a format if you don't want to. There's no word count requirements. One of the things I really like is that you can connect it to an OSF project. So if you're really going full open science the whole way, you might have the pre-registration, later you have the data and materials attached to an OSF project, and you have the pre-print. And if you're an OSF user, then all of those things can be linked together, which is, I mean, one of the huge advantages to OSF. They will also allow you to embargo the pre-registration for up to four years. So that's kind of getting at this issue of if you don't want that plan to be public, the second that you publish it, or the second that you put out the pre-registration, you can set an embargo and you can set the date or it will just automatically come out four years after you pre-register. And this thing down here is just a screenshot of kind of all of the different ways that you can pre-register. One of them includes the pre-registration template from as predicted. So you can actually use the questions from as predicted by an OSF if you wanted to. As predicted is a little bit more aimed at experimental research. The questions are very kind of more applicable to those types of questions. One of the limitations that I found is that it does have a limited word count. For a long time, my default was as predicted, but then I ran into an issue with the word count because my analysis plans are often very detailed. And so I ran out of words fairly frequently. So I've started using the as predicted sheet in OSF as really kind of my system. And then as predicted doesn't put a limit on your embargo, so it's private until you release it publicly. And there are some issues with that, but just to know kind of what the differences are between these two. And then one of the things that I think is very cool about as predicted, I think OSF does something like this as well, that you can submit an anonymized version for peer review. And yeah, OSF does that too. And we have a couple of questions that have come up, Amanda, if you want to address those. And one of those is, do you have a recommended amount of time between preregistration and starting data collection? Oh, man, it depends. It depends so much on what you do. In theory, you should be able to, once your preregistration is published, you should be able to start doing your data collection immediately. If you're thinking about kind of major timelines, I would think about how long does it take you to plan a study and how long does it take you to plan the analyses for a study and then put those two together. And that's how long it's going to take you to think through your preregistration because that's really what you're doing is, you're planning the design, you're planning the data collection, you're planning all of these parts, and then you're also planning your analysis as well. One thing that I would say is that typically, if you're used to this process of getting data and then planning out your analyses as you have the data in hand, you may find that planning your analyses takes longer when you don't have the data at hand because you might ask things like, oh, is our outcome variable normally distributed? And then you just look. And then if it is, then you make a decision. And if it's not, then you make a decision. But when you're doing the preregistration, you might say something like, we're going to check if our outcome variable is normally distributed. And then if it is, this is what we're going to do. And if it's not, this is what we're going to do. So you kind of have to like garden or forking paths the whole thing if you're making a kind of strong plan. So another question that came up was, does the preregistration, what does it take for a preregistration to be official? What counts? Yeah, this is, it's a kind of funny question because like for you to say it's preregistered, I think the big thing is that it needs to be time stamped and it needs to be, oh gosh, there's a word for this, a persistent link to that. So sometimes people will say something that's on like your university website or your personal website doesn't count because you could just take it down. But, and so using these registries can be really helpful. One of the things I know is that journals, some journals, and we'll talk about this a little bit in the coming slides, some journals give badges for preregistration. And I don't know that it's immediately clear from those journals what their requirements are. But I think, yes, persistent identifier. And if you have, if you want like a badge, then you kind of have to go by whatever the journal says. And I think most of the journals will say that it needs to be on some kind of formal registry like OSF or as predicted. Some other options are like, if you're doing clinical trials, clinicaltrials.gov, you do like a, it's essentially a preregistration there as well. Great, so we had one other comment that was another preregistration website that's really useful is the one hosted by Shree. Yeah, so the Society for Research and Educational Effectiveness. And then the other, the only other question we have so far is, do you know anything about preregistration with meta-analyses? And is there a best, is there a repository for that? Yeah, I don't think that there is a specific repository for meta-analyses other than, I guess there's two kind of journals that do this, more like a registered reports process. And there is, one of our resources that we have is some guidance for pre-registering meta-analyses. And there's some really nice templates that are out there and some nice resources for that. So that's in our resources page and we'll chat about that then. But I think you can pre-register meta-analysis anywhere. I don't know if there's a specific place where they're housed. I know like meta-archive would be for pre-prints, but I can't think for preregistrations. Okay, cool. So maybe this should have gone earlier, I'm realizing, but why would you pre-register? One of the things that is beneficial about pre-registration is that it reduces p-hacking. So whether we think about it or not, whether it's purposeful or not, a lot of the time we're making decisions based on kind of what we see in our analyses, whether it's, oh, this p-value is significant and that one's not, or look, this effect size is larger when I do the analysis this way. There's a lot of analytical flexibility in the kind of current way that we're doing science. Well, I shouldn't say current because a lot of us here are probably not doing this this way. But a lot of the kind of norms now and in the past have been to kind of analyze your data and then pick the analyses that work and that's what you write up for your paper. And so pre-registration avoids that by essentially making a commitment to yourself. This is essentially kind of the equivalent of writing yourself a little letter and putting it in a box and then later you open up the letter, right? That's what my advisor was very much like. Why can't we just do that for pre-registration to just keep ourselves accountable? And there's some benefits to doing it online as well. Another thing that this helps with is harking or hypothesizing after results are known. This is just something that I think comes naturally to scientists. We want to explain why things turned out the way that they did. And oftentimes this comes with if things don't turn out the way that we expected, we start to start thinking about, well, are there reasons I should have expected what I saw? And harking can be detrimental because then when we write up our research, it might seem like we said, well, I hypothesized this all along when in fact the reality of the story is I thought it was going to go one way. I saw the results. I thought about it a little more and now I'm kind of convinced in the other direction. And when we're thinking about a transparent science, we want to be representing the whole kind of process. Another thing that I would say is minorly helpful about pre-registration is that in some ways it opens the file drawer. If someone is doing a meta-analysis or doing some type of research trying to understand what's the previous research that's done on the topic, they can look at available pre-registrations and try to see if there are studies that were pre-registered but never run or never published on a specific topic. How to then go about putting that in a meta-analysis is really, really complicated and hard. And I don't really think that we totally figured that out. But the basic idea is there is that it gives us some representation of what's in the file drawer. So if there's thousands and thousands of people who are out there trying to replicate ego depletion, for example, but none of those papers are getting published, then we can see that by looking at pre-registrations. And then kind of similar, one of the things that I find really valuable about pre-registration is it helps us distinguish between informing versus testing theory. And this comes down to the way that we think about our hypothesis is that if we're trying to change theory or build new theory versus testing a theory, then that will kind of show up in the way that we pre-register and write our hypotheses. And I just want to point to a couple places that will reward you for your hard work doing this pre-registration. And there's a lot of journals that do this. I tried to pull some of the ones that are specific to education. So emerging adulthood, exceptional children, gifted children, quarterly language learning and journal of research on education effectiveness all provide badges. So along with your paper, if your paper is pre-registered, then you get a little badge that says that your paper is pre-registered. And one of the things I want to point out is there are actually two badges for pre-registration. There's one that is essentially if you pre-registered the way that you're going to collect your data but not analyze it, you get the pre-registered badge. If you get essentially methods and analysis all at the same time, then you get a little plus on your pre-registration, which I think is very cool. And then I think you guys have access to these slides through the OSF. And so you can look at the full list of all of the journals in case these are not ones that you would typically publish in. Okay, so we're going to transition to registered reports. I see a couple questions in the chat. Yeah, so we had one question that says, how do we teach or learn the discipline of fully planning out the analysis for pre-registration, including consideration for the full garden of forking paths? Ethan, this is such a beautiful question. This is so lovely. It's so hard. And that's like a lot of my practical advice is like, this is so hard. This process of planning out your analyses correctly and thinking through all of these different options is really, really difficult. I think this would be something that would be great to have either a hackathon or an uncomference this week about. Because yeah, there's so much that goes into this and it differs based on the type of research that you do. So I mostly work in a quantitative field. If you're a qualitative researcher, you would think about this very differently. There's a lot to be considered here. I will give some practical advice on this later, I think, Ethan. But I would love to chat about this. This is one of my passions in life. Yes, thanks. Awesome. I'm going to write that down so I don't forget. Excellent. So let's talk about registered reports. Registered reports are, I kind of think about them as pre-registration on steroids, where it's essentially the pre-registration process but integrated into the journal publishing process. So essentially what you do is you develop and design your study. So you kind of go through this process and if you were doing a pre-registration at that point, you would pre-register for yourself, for your lab. But with a registered report, what you do is you then prepare your intro methods and analysis plan or alternatively a result section that just has blanks in it. And you submit that to a journal. And the journal sends that out for peer review. And the peer review process kind of works the same way that it does in a kind of traditional thing where it goes out to reviewers, they review, but they don't have your data. And you don't have your data. Nobody has your data. So they have to make a decision about whether or not this paper should be published based on the kind of merits of the method, the question, the interestingness of the question. And in particular, whether they think that the paper will be informative regardless of how the results turn out. And so there's a lot of kind of upfront work that goes on in this process where there often time is a lot of revision during this phase where a reviewer might say, well, there's this compound in your study. Can you measure that? And then you have to add a measure or change your sample size or change your analysis plan. But it's all kind of collaborative and with the reviewers and before data is collected. So in theory, you will only have to collect and analyze your data once, which I know Karen is a big fan of. So that's kind of advantageous. And then if you make it through the stage one peer review process, then you get what's called an imprincipal acceptance, which is a commitment from the journal to publish the results of this paper, as long as you can follow along with your plan. And so far I've seen very few documentations of things getting rejected later because people didn't follow their plan. And then essentially once you get imprincipal acceptance, then you will conduct your research as planned and then prepare the final manuscript. And then that will go back to the journal as called a stage two submission. And that goes back out for peer review. And at that point, the peer reviewers are not deciding whether or not this paper gets published or not. That decision has already been made. They're there to kind of say, okay, did they follow their plan? Did they make reasonable decisions based on the results of their studies? Are the recommendations clear? That type of thing. And then after you're approved at the stage two peer review process, then the paper gets published, which is very exciting and cool. So who is involved in this? There's a lot more people involved in this process than a pre-registration. So you're involving your lab. You're involving the specific journals. There's the editor and the reviewers. And in many senses, this is a very like collaborative experience because you are working with the reviewers to make sure that the research is up to snuff. And one of the things, again, I want to emphasize is if there are people who you tend to bring in after data is collected, I would get them involved from the get-go. I am consistently the person that people walk into my office and they say, we collected this data and we don't know how to analyze it. Or we made this design a little bit more complicated than we realized. How do we do for me mediation or moderation analysis with this? And sometimes my answer to that question is there's not a method developed. And you really don't want to end up in that situation when you have a registered report where you said, okay, we're going to analyze this data, but didn't realize that there's no method out there for multi-level mediation with survival outcomes or something wild like that. So get the people involved from the beginning and make sure that the plan is solid before submitting it to the journal. And when you do this, it's kind of a multi-stage process. So we're thinking about the process of thinking about the different stages of preregistration, kind of like in the preregistration, we want to do the stage one before we've collected data. I want to point out though that it's very, very common and often recommended to do pilot studies prior to submitting for a registered report. So most journals will allow for pilot data appropriately labeled to say, like, okay, we did this stuff before we did the registered report, but then this part is the registered report. And so a lot of the time that's really helpful for thinking through your analysis plan as well as making sure that your manipulation works and those types of things. So stage one, it should be timed like your preregistration. So this should happen before your data is collected. And then one thing that we'll talk about a little more later is that you have to kind of account for the review time, which is a little amorphous, as we many of us know through going through the peer review process. Knowing how long peer review is going to take can be tough. And so you want to take that new account in terms of planning for your data collection. So if your data collection needs to happen at a certain time, this is something that requires a lot of planning, especially if you're dealing with like cohorts in educational studies, you need to get the kids, you need to collect their data in spring or something, then you really want to be planning pretty far in advance to make sure that you have the in principle acceptance prior to collecting the data. And there have been cases where working with the journal editor, if you know that you have a specific timeline that you're trying to meet, making sure that that's clear to the journal editor to kind of make sure that the process moves pretty quickly. And then the stage two peer review process tends to take a lot less time than the stage one because really it should just be kind of a one pass. Did they go through and say the things that they were looking to say? And there's been some empirical work to look at this timeline. The information is a little bit limited, but Tom Hardwick and John Enides tried to look at this information from registered reports. And they found that median from in principle acceptance to publication is about 671 days, 761 days. And so that is essentially, once you've gotten the go ahead from the journal, that's kind of a measure of how long does it take to do the research. And not surprisingly, it takes about a year or two years. And the stage two submission of publication is pretty short. So it's a median of about 187 days. And if you, I think if you looked at stage two submission to like online first, then that would be even shorter. But oftentimes publication itself takes a long time of the printing presses and whatever goes on back there. So yeah, so I think registered reports are in general a little bit more kind of time complex than pre-registration. Yeah. And Amanda, we had a question about can a stage one go through multiple rounds of review similar to a traditional publication? Yeah. So that's very, very common just like similar publication or publication, a traditional publication, if you will. It's very common to go through multiple rounds of peer review and revise. And one of the things that I think is a huge advantage is you're revising something that hasn't been done yet instead of revising something that's already been done. And so there's a lot more degrees of freedom to fix problems as we foresee them occurring instead of well, the data is already collected and there's nothing else that we can do and trying to like fix everything on the back end can often be really hard. So I'll mention this a little bit later, but I think the request from reviewers can often be a lot more reasonable when we're dealing with a registered report because it's not go collect this data again better this time. So instead it's go collect it correctly the first time, which can be really helpful, I think. But yes, it's very common to go through multiple rounds of peer review, especially at stage one. So where do we publish these fabled registered reports? This is probably the most common question that I get when talking to people about this is where can I actually publish this? And it's increasingly common to see registered reports and in psychology and social, I think education has kind of lumped into a mix of psychology and social sciences here. But currently there's about 250 journals that have adopted this and about 450 published registered reports. And considering this started in 2013 and has really, really picked up over time, I think this is quite an accomplishment. A couple of journals listed here that publish registered reports in the kind of education area. And then one of the things that's very common is to put your stage one submission up on OSF registries so that it's there kind of like a pre-registration and it is independent of the journal so that someone could verify your stage one submission. Additionally, OSF has a list of journals. So if these six journals here are not the journals that you would typically publish in, that's totally okay. And there is a big list by OSF that lists all of the journals that accept registered reports. And I would say something that people might consider doing this week, if you're excited about this, is one of the first sips, which is kind of the psychology open science conferences. There was a big letter writing campaign to choose some journals that we thought, we want these journals to offer registered reports and then prep and create a way to contact those journals and reach out to them and say, hey, we want you to offer registered reports. So if there are journals that you want to offer registered reports, these next two days is really good opportunity to work with a number of like-minded people to work toward that even more. And then I wanted to share with this as tool, you guys have the link to this. This is in beta form, so we're still kind of playing around with this, but it's a new database for registered reports where you can filter based on a number of policies that the journal might have. So if you're looking to do secondary data analysis, this will give you access to which journals allow for secondary data analysis. There's some information about registered reports. You can look by each index or a number of different characteristics. And so this is a kind of different way for searching for a journal that matches kind of your needs if you already have a study in mind. And you guys have a link to this. Eventually this will be integrated with OSF and their list. So this is kind of, we're kind of in the middle phases there for that. Cool. So registered reports provide a lot of the same benefits as pre-registration, but they also in my mind reduce publication bias even more than a pre-registration because if your study is pre-registered and you find a null result, it's still pretty hard to convince a journal to publish it. Whereas a registered report, the commitment from the journal is made prior to actually seeing the results. And so a lot of research has shown that registered reports are publishing null results at higher rates than traditional publications. And similarly, this is helping handle the file drawer problem I think even more because the actual results get published rather than just looking at what has been pre-registered. And then in theory, if the review process is working correctly, then the quality of the research should be a little bit higher based on the feedback from the reviewers. In general, they found that these studies tend to have higher sample sizes than traditional studies. And I think there's a lot of aspects of that both like push from a reviewer but also buy-in from the researcher is like if you know that all of these participants that is going to result in a publication versus when you're running studies but you're not sure if they're going to get published, you're willing to essentially bet more money on the registered report. And then I find this process really helpful is that you can cut ties with an idea if it's not working out. So even before you're running participants, you can get feedback from our viewers that say like, hey, actually, somebody's already done something almost identical to this. Then you actually know that you don't have to do it. And registered reports have also been used to incentivize open science practices more generally. So things like open data, replications, etc. And I would say this streamlines the review process by not having to go to a journal, get rejected, submit to a different journal, get rejected. This whole process just looks so much easier when you're doing this in this two-stage system. It's different, which is scary, but it is a little bit, I think it's streamlined. And then again, you're reducing wasted resources by if you're consistently running studies but then not publishing them, then registered reports are a really great way to make sure that they get published. And you can even have a guaranteed publication even if you have no results. Okay, and then I just wanted to give a kind of sense of the difference between the review processes for these two types of publications. So if you have a non-registered report, you're going to develop your idea, design your study, collect your data, write the report, and then peer review happens right here. And at this point, you might get a lot of questions like you need to run more analyses, your study's not adequately powered, you have to run it again with more people, there's a confound in your original study, you need to run another study, or this question is uninteresting and not worth exploring. Don't we already know this? And these are like really, really common pieces that come out of the peer review process. But what I want to point out is that these things are awful feelings to have when we go through the peer review process, but if we could get these comments earlier, we could actually fix these problems without wasting resources. So for a registered report, you would develop and design your study and then it would go out for peer review. If your analyses aren't appropriate, hopefully your reviewers will catch that. They could recommend that you collect more people. And so this is just adding to your sample size, your initially planned sample size that you haven't yet collected. They could fix the confound in your study or tell you maybe this isn't worth exploring. And all of these three things would result in some level of in principle acceptance. And then you go for stage two peer review. There's not no comments at stage two. Oftentimes there are kind of suggestions for exploratory analyses, which you can often accept or decline. And you have to make it very clear that these are exploratory because they weren't recommended prior to doing the data collection. And then additionally, sometimes we are making kind of stronger claims than our data support. And so hopefully our peer reviewers will point that out. Yes. And then I think we will turn it over to Karen. Maybe you'll come. Okay. And I do want to go over a couple of questions that have gone into the comments. One, Amanda, you mentioned a repository for registered reports or database for registered reports. Could you say where that is again? Yes. There is a database for registered reports. The link is really kind of a mess. So if you look on the OSF slides, it's here, this Montoya RR journal database. And then OSF also has a list. And then also in the resources later, there is a Zotero list of published registered reports. And so that can be helpful if you want to look at some examples of registered reports as well. And another question that came up is, do you think it's worthwhile to include a discussion section in your stage one submission? Sort of saying, no matter what the results, it's important. I'm summarizing quite a bit. So yeah, I've seen it done different ways. There are a lot of really good examples of stage one submissions out there. And if you look on the registries, you can kind of see what people tend to do. It partially depends on what the journal asks for. So sometimes they say, okay, you can't like commit to a discussion section because you don't know the results yet. But what some authors do is they'll write some kind of parts of the discussion and leave some blanks for language so that they can say like, oh, well, we found more or less or no difference. And they'll like put in brackets kind of the different options of language. And it turns into like a little mad lib, which is very cool. So I think if you feel like you can commit to some language ahead of time, then you can definitely include that. I would double check the submission guidelines for the journal specifically because some of them might say, you do not include a discussion section yet. And some of them might say that you do. Yeah. All right, there are a couple of other questions that have jumped in. And I'm wondering some of them. Do you want to start, Karen? And then I can answer some questions in the chat as we go. Sure, sure. That sounds good. Okay, so just a quick kind of compare and contrast with preregistration and register reports. What is the typical length of the format for each? And a preregistration can be as short as a single page. It needs to include your data collection. Basically everything goes into a method section, data collection, measures. And I recommend also doing the analysis procedures. But it needs to have the most important pieces that it needs to have that timestamp or that date stamp before the analyses or the data collection happens. For register reports, the stage one register report is shorter than a full length manuscript. But I have also found that the stage one methods are more robust than a typical method section in a publication because you're having to explain what you're going to do without the advantage of having the data to do the explaining. So it's a little more effortful to do that explanation. And then a stage two is the full manuscript. So it's very much what you would see in a typical submission. So what journals accept them? For a preregistration, it doesn't matter. You can do a preregistration no matter what journal you plan to eventually submit your manuscript to. And kind of ironically, the journals that accept register reports or the journals that accept them. So those are increasing on a regular basis but you can only submit register reports to those who have those procedures in place. So the timeline, Amanda did a really nice job with this. With the preregistration, the first thing you do before you start the data collection analysis is submit the preregistration. Then you go do and conduct your research and write it up and then you submit for a peer review. A register report, you submit the stage one for peer review. You get the in principle acceptance after some back and forth potentially with your reviewers. Assuming that they want the study published, then you conduct your research and write it up and then you submit for a stage two review. And so peer review at the preregistration phase occurs just like a normal submission. So it occurs after the full manuscript is written. Peer review for register reports happens twice. One at the end of stage one when you've submitted and then again after the entire manuscript is written. So which one's right for your study? For me it really depends on time. If you have the time and there is a good journal to target then I think that the registered reports are the better option. But if you are right on the precipice of collecting data or it's in the very near future and it's a really time sensitive thing then a register report is probably gonna be a better option for you because you can't control how long that peer review of the stage one is gonna take. So if you expect that it might interfere with your data collection then you may want to just preregister instead of doing the registered report. Register reports are better if you still have time to make changes to the study. If you can go back and tweak and get that input but they both have similar advantages. Both of them make your hypotheses transparent and exploratory analyses transparent as a result. And neither of them completely tie your hands. If something happens your data don't behave like you expect them to and you have to look at it a little bit differently. You can include what's called a deviation in your stage two submission or a deviation in your register report for the full manuscript. But you just have to detail these different things happen. I've had that happen with all of the registered reports I've done. There's been something that didn't go as planned and so I had to say okay this was my original plan and this is what I did and this is why I did it but it was really clear and I've not had that be a barrier to publication. I'm plying through this pretty quickly because I want to make sure we can get to questions. So I'm going to pass it back to Amanda. Okay cool. This is my practical advice which is mostly that analysis planning is harder than you think. So to do this right this is related to Ethan's question. Thinking through these things takes time and is very hard and I'm watching my graduate student Tristan nod his head because he's gone through this process with me. So so claiming an analysis in advance does not always mean that it's an appropriate analysis and this is kind of an argument for registered reports over preregistration. So just because you plan to do it one way doesn't mean that it's the best test of that question and that's one of the reasons why I think registered reports can be really beneficial over preregistration is that preregistration is all internal right and so you're not getting that feedback from the peer reviewers about whether or not they think that this is the right test of the question. And what I would recommend is that when you're thinking about the analyses to think about the edge cases. So what happens if things don't turn out as expected or what happens if things don't look the way that we expect them to because oftentimes it's very easy to say like okay if all of the assumptions of linear regression are met then I'm going to do this this way and I'm going to test this question and when this comes out significant then I'm going to do this but there are a lot of extra steps that might occur. So one of the things I would recommend that you think through is how measures are measured and combined. So if you're in a situation where you tend to use scales you also want to think about like what happens if the reliability of your scale is low. Do you just plow forward or do you make some adjustments and do you come up with a process for making those adjustments? Think about missing data. How to detect what type of missing data you have how to handle the different types of missing data you have and how much missing data would be too much missing data for you to even make any claims based on what you have. Think about your exclusion criteria. One of the things that we realized while we were planning our one of our registered reports is that because we're on rolling recruitment if we have people who meet an exclusion criteria we can actually replace them because we're just continuously recruiting and so what we do is we check that people meet this exclusion criteria and then we still can make it to our final end rather than having to guess about how many people will we will have to exclude at the end of the study. Think about multiple comparisons and how many tests you're doing and how to correct for family wise error rate and then thinking about positive checks. So if you're doing something like randomization you still want to check that there is some success of randomization or failure of randomization having a manipulation check and then thinking about what you would do if you fail that manipulation check. So for example if I'm trying to manipulate some variable and then I have a measure of it and I don't see differences across groups on that then that might kind of change the way that I approach the study. Some of my other advice is thinking about time management for registered reports. So the planning process this whole process kind of operates differently. So one thing that I've found is that a good time to start thinking about this is if you're in the grant writing phase. So one having an in-principle acceptance for an initial study can indicate the quality of your research and the value to a grant panel and then reporting in-principle acceptances that you're kind of like yearly reporting times that can really show the progress that you're making instead of just having like a slew of publications right at the end of your period in your grant. There are a number of difficulties that can come up with the weirdness of the time. So you can have turnover in lab staff and you might want to think a little bit more about having redundancy in your lab staff. If you're doing this with student projects, they have deadlines, you have deadlines and all it's very hard to get all of these deadlines to line up. And then thinking also about grant deadlines which I know Karen has had experience with is if you're getting near the end of your grant and you need to produce something before the end of your grant, it might not be the right time for a registered report. There's a lot of different creative solutions. You can think about collecting pilot data and multiple rounds of data collection where you can collect some of the data and then use that as pilot data for a registered report and then plan to do kind of a second round as the kind of formal registered reports so you can get started without having to delay and then more redundancy if you have a staff turnover and thinking about working with the editors to make sure that the timeline for your review aligns with your plan for data collection. Yes, and Karen. Okay, I just wanted to note that I have found the registered report process to be much more how I want science to be. It's much more collaborative. The I feel like I'm working with the reviewers to improve the study instead of working with reviewers to defend what I already did. So I just I like the process a lot better. And anecdotally, I have had 100% of my registered reports have been accepted as a stage one, whereas not 100% of my traditional manuscripts get accepted the first place where I send them. So I don't know that that works for everybody, but I have had a much higher success rate with registered reports on that first place where I've sent it. Also, the stage two registered report or the stage two where everything's finally flushed out is really long. And so I'm running up against word limits for journals. So just kind of know that you're probably going to have to find creative ways to trim and move things into supplemental materials. And if you're not sure if your study makes sense for a registered report, ask the editor. I had one of the registered reports that's in a stage two review right now. I had previously analyzed some data from the study cross-sectionally, and I wanted to connect it longitudinally for the registered report, which it looks like that did not fit the guidelines of the journal. I emailed the editor, explained all the details, and she said, nope, this looks great. And so I just included that with my submission when I submitted the stage one report, and it was fine. I'm skipping over some stuff. So I'm going to go ahead and go to the next one. Now, all of my registered reports have been for secondary data analysis. So mine look a little bit different. The important thing is, is that you don't peek. But what I have found helpful is to try to do things like write up the full paper for a conference using a different data set. For example, I have a planned registered report that I want to submit using the TEMS 2019 data. Well, the TEMS 2019 data isn't available yet. And so I'm running all of the analyses on the 2015 so that I kind of work out all of the bugs on the 2015 data. And then when I submit the registered report, I just strip out the 2015 data and I have my empty tables and figures and say, I'm going to do this with 2019 whenever it becomes available sometime this month. I've also had a, in my registered reports, because I have seen portions of the data because my data sets are so huge, I include a paragraph that's really explicit about what I've seen, where I've published it, and I'm anonymized, of course, but trying to be as transparent as possible about the potential that I might have had to exposure to different parts of the data set. So it's 11 o'clock. We've been answering questions all the way along. We have some common concerns. I don't know about Amanda, but I think I'm happy to hang out here a little bit into the lunch break. But I know also this is, I think we're getting into the lunch break, so if people want to go off and eat, that's totally fine. But I can hang out here for 10 minutes or so. Yeah, I just want to say thanks, everybody, for joining us. And yes, I will be happy to stick around and answer questions. And also Karen and I will be around for the next two days. I'm also happy to chat with anyone who's interested in doing a hackathon or an conference about pre-registration and registered reports. And this is such a great opportunity to work with folks, not just talk at folks. So hopefully this is kind of the last session you go to that is mostly people talking at you. And from now on, we're all ready to work together, which is really, really wonderful. I'm seeing a question from Carrie about whether pre-registration can be used for exploratory studies. Karen, do you want to speak to that? Yeah, so yes, I don't see any reason why pre-registration couldn't be used. And in fact, I think it would be a good idea for exploratory studies, because it helps you to really flesh out what your research questions are. So just instead of registering your hypotheses, you would register the research questions and how you plan to address those research questions. Mm-hmm, yeah. And pre-registration isn't required for everything. I think that's one thing that maybe we didn't talk about. We were just kind of assuming that everyone wants to pre-register. But certain types of questions we don't need for pre-registration, and especially in exploratory situations where we're trying to generate more questions than we're trying to answer, then oftentimes we don't need a pre-registration because the whole goal is to start generating questions rather than providing concrete answers to those. But if you feel like you have a good sense of what your plan is, there's no reason not to, I feel like. Yeah, awesome. Other questions? We saw in the chat, there's so many thank yous. It's hard to... I know the question. Yes, excellent. I'd love to chat about the Forking Paths thing further or talk about unconference or hackathon on that topic. No. There's nothing else on the floor. Yeah, so I think... I mean, I'm doing an unconference session on sort of design analysis, but in some ways it kind of begs of the power analysis being a part of that. But in some sense it begs the question of how do we even kind of navigate or kind of map that kind of massive, you know, kind of possible space for where the study could go. I think one resource that we can point to, I actually realized that even though I'm an, you know, open science champion, I should actually read more registered reports to see how people did this and their Stage 1 submissions. So I guess that leads to one simple question. Normally when a journal publishes the imprints, when you get an imprincipal acceptance, because you said that the Stage 1 is typically more detailed than the publication, but journals publish the full kind of Stage 1 thing as well? Not always. And that's one of the arguments for pre-registering, even when you're doing a registered report. And a lot of journals will require this, is essentially that you take your Stage 1 and you submit it essentially as a pre-registration so that it's out there and available and people can check it. One of the, there are some journals though that will publish protocols. So the JMIRs of the world, of which there are many, they actually have a specific journal that's JMIR Research Protocols. And if you want to do a registered report with them, you submit to that journal and then essentially you have your choose of which of their journals you want the Stage 2 to get published in. So everything goes through JMIR protocols and the Stage 1 papers get published there, which I find really, really helpful. And I think the Zotero list has a group of, now I'm sharing with you Zotero, there's a folder for research protocols separate from the registered reports. So you can kind of check those out as seeing how people do this. And I find the process so, so helpful. So recently our lab has been preparing a registered report and I actually went through, I think it was something that David Mellor wrote, or somebody who was at OSF and like the way the OSF people, right, their registered reports is pristine. And was an example of one of those that had a discussion section pre-written. But actually, no, I'm realizing it was, it was the many labs five, which was done as a registered report. So I think that would have been early ever saw. And so they had essentially the whole discussion section written out and then like we'll add a paragraph here about such and such, depending on the results and that type of thing. And it's so detailed and so much work. But it's really, really good work to do up front. I think it just makes the back end of the research process so much easier. Yeah, in terms of the Garden of Forking Paths, there's so much to think about, even just like with one study, trying to think through all of the potentialities and what do you do? But I think it's so helpful to think about it in advance because you can come up with much better solutions. Like one of the things that we realized with exclusion criteria was like, well, okay, if we have to exclude people for these reasons, can we then recruit more? Because our study was on a rolling basis, so there's no reason we can't check if they meet exclusion criteria and then exclude them and obviously report how many people got excluded through those processes. And one of the things we had to think about is like, how many people, like if it's like 20% of our sample is getting excluded or something like, is that that would be a lot, right? And so at what point do you say like, maybe this is not the right kind of process or we're not sampling the right people or something along those lines?