 Okay, welcome. We are going to go through a deep dive and open scholarship with pre registration and just to give you a quick background. Hold on. Where are my. There we go. So here are lovely faces and you're going to get to see our lovely faces. One of the things I don't like about this particular format is I can't see your face and so please be use the chat or the Q&A so that we can engage with you as much as possible. This is kind of a strange format. I mean I like seeing Scott's face but this is a little bit odd so kind of bear with us and Yeah, Scott, you want to add anything. No thanks everyone for joining us. This is the deep dive session on pre registration. I'm Scott Peters. I'm from the University of Wisconsin Whitewater and the only other thing I thought maybe was worth mentioning is as you know Karen and I are presenting on pre registration, but I'll just speaking for myself. I only even heard about pre registration just a couple of years ago when I intended the CIP Society Improvement of Psychological Science conference and following that conference you know did my first pre registration so this is not something you know that either one of us have just known about for years and we're the OG people of pre registration. We are I guess relatively new in the last couple of years but also big time converts so we're not kind of presenting to you as the you know original theorists of pre registration but rather just people that have heard about it and have had started using it a lot more in our research. One thing I wanted to add to that is that I don't do research about pre registrations. I'm a user of pre registration so I don't know everything about it. I just sort of know what I've used and so this is going to be from a very practical perspective but there may be things that we just don't know that we can try to answer for you or find someone who knows the answer. But I'm not a an open science researcher I use open science in my research so it's a little different perspective. Also, I am at Texas A&M University, and I'm an associate professor there and Scott and I have worked on several projects together so you'll get to hear about some of the work we've done with pre registration. So. All right. Okay, so. Yeah, good coordination you've got two presenters you know only one thing control the slide so our outline for the day I'm going to take kind of the beginning of the session here talk about, especially about the why of pre registration and what problems exist that pre registration is supposed to help address pre registration as you'll find out does not inherently address all of them or really even any of them, but it makes a lot of these problems much more detectable and observable for the end user even the researcher themselves. They don't necessarily make problems disappear, but they make it much more obvious both to the researcher and to the end user when those problems are happening. So we're going to talk about what those problems are. We'll go through kind of the basic structure of what a pre registration looks like and what it includes we'll look at several templates that are online that you can follow. We will compare pre registration to registered reports, and there is a session later today after lunch about registered reports if you're interested more in those but we will compare them because, as Karen mentioned both of us have done registered reports and pre registrations before and they're they're similar, and we'll also just give you some practical advice in terms of logistics so in actually kind of making this work in the real world and then we've got a bunch of resource links at the end, and I imagine we'll end with time for questions as well. And then we'll get through the why what who went and where and we'll do that all with plenty of time to spare. That rhymes. Next slide. Okay, so oops, go back one. Sorry, too far. It's okay. So in terms of the why pre registration is a lot of kind of text on here but I want to talk through several of these. So I want you to think about how the published literature in your field. This is this big Venn diagram this big circle that published research is itself a bias sample of all research that has ever been conducted. So all the research conducted, you know, in an entire field say reading research reading instruction that not all that research has ended up in the published literature. It is in dissertations or theses or think tank papers or was never published at all, because of this thing called the file drawer effect where someone conducted a study, maybe they did their dissertation didn't like the results or only published part of it. And so everything that we might find in a journal or in a body of work is itself a bias sample of all the research that has ever been done. That's first of all. It is also a bias sample of all of the research that I give an author did. And I think our dissertations for those of you that are faculty or did a dissertation can probably empathize with this a bit. You might have had four or five research questions in your dissertation, but you didn't necessarily publish all of them in a given study, or maybe you publish several studies from your dissertation but didn't include everything that you did. So we have kind of this giant universe of research that has been done, but only part of that has ever been, you know, actually published, and even the part that's published only includes some of the research questions that were actually studied. So what we have is kind of this biased knowledge base, where the knowledge base doesn't actually represent the questions that have actually been asked in the field. There's this additional problem, which is that most are much I should say, of the published research in any given field this has been done in a number of different disciplines including an education which is the numbers I have up here. Matt Makle had a session yesterday about replication research, and this is just some numbers from a study he did back in 2014 so it's a little out of date now, but they went and looked at the top 100 journals. They looked for 10 years, if I'm remembering correctly, and found only 221 actual replication studies, which itself is very small. It's hard to know exactly what like an ideal number or percentage of studies doing replications would be. But 221 I think we can all agree is pretty low out of 164,000 in a 10 year period, and 67% of those were successful replications. Even if we just take that 67% and generalize it across that 164,000, we have a large number of studies in the field that are not able to be replicated. It doesn't mean those studies were wrong or they were flawed or fraudulent or anything like that, just means replications wouldn't have been successful. We have a lot more replication in other fields. Education, as kind of you can tell by the nature of this conference, is a little bit newer to the open science world. Psychology I think is a kind of maybe five, 10 years ahead of us. And I also have down here just at the bottom, this is with regard to registered reports, but I want to mention that most studies find support for their hypotheses. And so think about the studies you've ever read, you know the vast majority of them find statistically significant results. We'll talk about maybe why that is. But when you look at published registered reports, where the methods are peer reviewed before the study ever happens and we'll talk more about what those actually look like in a second. Only about 44% of those studies found support for their hypotheses. So what's the difference between those two. In a registered report the authors aren't able to change any of their analyses after the fact. So they're not able to see that a study did not find a statistically significant result, and then tweak the question or tweak the data in some way. Whereas in a regular study, you know, oh I my first analysis wasn't significant let me try this different variable let me exclude some part of the sample I'm going to be able to find a statistically significant result. So I'm going to add that the data will tell you whatever you want as long as you torture it long enough. And I think in the days of fancy computers and everything we can really torture the data for a long time. And just before I go on I have I see a question in the chat where someone asked well what is a bias sample, or I should say a bias sample is one that does not accurately represent the larger population. So we have a sample of all research all possible research that could be conducted. The sample of that of research has actually been conducted so it doesn't represent the whole universe, but then even a bias sample of that has been published. So we really only have this particular slice of the entire universe of research that's been conducted, which itself is a bias sample of all research that could be possible. It's getting a little metaphysical I suppose but we'll get to some more practical stuff here in a second. Okay, we can go on. This is again from the Matt Makle 2019 paper. Pretty recent paper in educational researcher. And these are some questionable research practices, and some of these I think are more questionable than others, you know, I don't think I have outright fraudulent data on here but I think most of us would agree, making up data is pretty bad. But some of these are less bad, like white white lies kind of thing. So these practices are really common so these are self reported questionable research practices so a survey was sent to some of the authors like major authors and major education journals again I think over a 10 year period. And this is how often people report using those practices. So 62% admitted omitting non significant analyses they did or omitting non significant variables like they were never there. So a quick example of how the published research is not a truly in the representative indicator of all the research that has been done. You know, 30% have, I call it generously rounded P value. So 0.058. Well, I'll just round that down to 0.05 that sounds fine or near statistical significance or something like that, or an excluded cases you know I'm going to leave some out of my analysis until it makes my results get better. And I, the ones that I have starred here are the ones that pre registration is really targeting I'm not going to say it makes them go away or addresses them, but it's really meant to target them. And the reason I say that is and as Karen's going to talk about in just a second pre registration kind of time sets your analyses and commits you to the things that you say you're going to do beforehand. So that you are not tempted to fudge the numbers later. I don't want to make it sound like it's any kind of malicious activity it's a lot of things that I was literally taught in my doctoral program, you know, omitting non significant findings is something I was quite literally taught in my doctoral program. But it does affect kind of the inferential value of a lot of the methods that we're talking about. So pre registration is meant to help address some of these on the researcher side, and also lend more confidence to the reader and the consumer on the other side once the research comes out. Thanks Karen for being the slide and this is just a visual representation of Matt Makles paper that he talked about yesterday a little bit. And this is just the prevalence of each of those quote q rp's. And so you can see, you know, omitting analyses is something that a lot of people kind of admit to doing again I was told to do that in my doctoral program. So omitting only non significant omitting non significant variables. Harking is hypothesizing after results are known. So, oh my original hypothesis didn't didn't turn out well so I'll just change my hypothesis in the published paper, and then claim the match and say oh I found what I wanted all along. And luckily you know you can think of this positively and say oh a lot of these are black which means people really never say or very rarely say they do them. But if you think about it we still have a good third of people, you know, a third of studies that are engaging in some of these. And I really want to remove kind of badness or you know evilness from this equation because that's not what I'm talking about. What I'm just talking about is a lot of these practices are going to harm the truth that is in the knowledge base that is in the published literature. So we are going to know less about how to best educate English language learners or something like that, because some of these practices, I'm not talking about like, oh, shame on these people for, you know, for doing those kinds of things. Okay, go on to the next one. Okay, so I think this is one of my last ones and, and then if you're still thinking well wait a minute we haven't even talked about what this is yet that's true that's coming up next. But first we're really setting the stage for why you should care about this. So pre registration has nothing to do with a journal, you can do pre registration for a study that you want to publish anywhere, or even a study you're never going to publish it could just be part of a dissertation, and you're never actually going to publish that in a journal that's fine. So it's kind of distinct and independent from a journal. Some journals will give you badges will recognize an article if it has been pre registered, but that's not, you know, that's not kind of necessary or to do pre registration. As we'll talk about a little bit registered reports are different those are actually connected to a specific journal. So importantly, because something that characterizes a pre registration is kind of a beforehand plan and agreed upon plan for what's going to happen with the study. You really need to get all of your co authors on the same page, because if you have something in your mind and they don't want to agree to that or they had a different plan. And work that out later, because then you're changing the registration you're changing the plan. So you really have to make sure everyone that's engaged in the study is in agreement for how things are going to proceed sample size, analytic methods, you know, timing, all that kind of stuff. And this is especially relevant if you are a collaborator so no Karen I collaborate on a lot of research and she knows a lot more about quantitative methods than I do. I want to make sure that if you have different expertise on your research team that everyone is kind of weighing in and making sure everything is insufficient detail, and is actually coherent. There's plenty of times and I've tried to write up something and and Karen has said no that's that's not actually how those words work so. All right, next slide. Okay, you're taking over. Okay, so I'm going to talk a little bit about the what. And one of the things I want you to think about when we're thinking about the what is you have probably done a pre registration. You have done a pre registration before and you just didn't know it. For example, if you submitted a grant application, you have done sort of a form of a pre registration where you've said this is what I want to do. This is how I plan to analyze the data and will you give me money to do that particular study. That is analogous to a pre registration, or if you have completed a dissertation, you essentially did a pre registration because you talked with your committee, you had a proposal and said okay this is what I want to do. This is how I plan to collect the data this is how I plan to analyze the data, and they all sort of gave you a thumbs up and said, yes this is good and you move forward so. So this is not a huge. There's not something so different here that we've never done it before it's just bringing that into sort of the regular rhythm of your research, instead of having it be just for dissertations or grant proposals or that kind of thing. So prior to collecting your data or conducting a research project, you want to create a public registration of your plan. We'll talk about some exceptions to this it doesn't have to always happen prior to data collection. I do work with secondary data analysis a lot. And so sometimes that's just not practical the data are available but I am still going to pre register my plan for the data. I just have to illustrate that I have not peaked and tried to examine what patterns are there before I do the pre registration. One of the really important pieces is that it is time stamped. And so you have this permanent record, you can go back and point to hey this is when the plan was pre registered. So what's really nice been working with secondary data analysis. And with one of my graduate students, he wanted to do some analyses with the new Tim's data that just came out. And so we work to do the pre registration, prior to the release of the actual Tim's data. So that is like, you know, doubly covering yourself that you've not looked at the data because the data weren't even available when in the secondary data. All right, so, typically a pre registration includes hypotheses. So, this would be what is it that you are testing. Why are you getting into the data in the first place. There are opportunities to do research questions but that's not the traditional approach for using a pre registration. But if you really do have some exploratory things there are some things you can do for including things that are not hypothesis based but are more research question based. Alright, the other thing that is typically included is your sample size plan. My sample size plan usually involves some kind of power analysis based on prior research. So based on previous research, this is the size of the group that I'm going to need in order to have the power to detect the effects that I expect to find. So you might also have some really practical limitations on your sample size, maybe you're planning to give everybody a $10 gift certificate for participating and you've got $500 so you're going to get 50 participants. So it is, where did you get that number that you come up with 50 because that's what the power analysis said or that's because the money you've had available to do the study. So you're just being really explicit about how you came up with a number that you're going to go out and recruit for your study. You also want to include your variables and how they're going to be measured. If you're using scales, for example, you want to think about the way that your scales are going to be included in your analyses. Are you going to use means? Are you going to do latent constructs? What is it that you're going to do? How are you actually going to include those variables and constructs in your analyses? If you have an experiment, then make sure that you've included all of your various experimental conditions. The inclusion, exclusion criteria is where I have run into things that I didn't expect. So this is where you start to think about outliers. How are you going to identify your outliers? What's the plan you're going to do? What about missing data? For example, in some of the secondary data analyses that I do, we discovered that in a particular time point, a student might have been observed three times when they were supposed to only have one observation. So I needed to think through how am I going to handle students having multiple observations within a time point. And so I pre-registered, okay, this is my plan if I have students with more than one time point. We also had issues with students' genders or ethnicities being recorded differently across time. And so how do I line that up? How do I choose which gender or which ethnicity to use in my longitudinal data analysis? And then the final piece, and this is what we're going to spend a little more time on later, is the plan of analysis and thinking through all of the steps, the different things that you're going to do. So next, we'll talk a little bit about when to do your pre-registration. So in a perfect world, you're going to want to do your pre-registration prior to collecting, or at least prior to accessing the data. It's also common for secondary data analysis to do your pre-registration before you have access to the data, although that doesn't mean you can't use it if you have seen parts of the data. So one register report that we did, we looked at data cross-sectionally, and then we wanted to later do a study that was longitudinal. And so I had seen one slice of the data, but I'd never connected it longitudinally. So I have a paragraph in some of our pre-registrations and register reports that talk about what specifically I had seen with the citations that went along with it. So also there could be, well, that goes to this third point. There may be reasons to pre-register after it's been collected, but before it's been analyzed. The thing you want to do here is just be super transparent. We're promoting openness, we're promoting trust and quality of what we're doing. And when we have seen it, it's okay, just make sure that you're letting people know that you have seen it. Now, doing a pre-registration takes time. A solid pre-registration could take many revisions. It might go through many rounds with your co-authors. But think of this as doing something good for your future self, because if you take the time now to do all the pre-registration pieces, then when you get to the data analysis stage, then you just start implementing. And it saves you time at the actual analysis part, but you do have to kind of front load the time in the pre-registration process. But I have seen in my work, and I think Scott seen this in his work, that the overall time is less with the pre-registration than when I did not do a pre-registration. But there's just a lot more thought that goes into it at the beginning. I was just going to add to that, Karen. If you've ever done a dissertation or you've ever written a grant or ever done an IRB, you've had to flesh out those things beforehand. Like you've had to say, this is what I'm going to say to people. This is how I'm going to recruit them. This is when I'm going to stop. And I don't know about you all, but I have copies of grants I wrote on my desk so I can go back and figure out what I said I was going to do. To Karen's point, it makes your life easier because you can go back and read that plan of attack you had before, before you're in the middle of the weeks. That way you're not changing things accidentally or changing things unknowingly to get particular results that you wanted to find. Yeah, and there's a couple of questions in the chat. One is about pilot studies and can they be pre-registered? And certainly you can pre-register a pilot study. One thing that I have done, and you'll see this later in some of our advice, is that sometimes it's helpful to intentionally do a pilot study to work out the details to make it easier for the larger study so you pre-register that. I've intentionally done that with some of my work where I took a slice of data that was not going to be included in the larger analyses and then I ran everything and sort of figured out where the bumps were and kept my code and then I use that code and that plan when I get to the bigger study. But I don't typically pre-register the pilot because I'm doing the pilot to help me do the larger study, if that makes sense. And I have also used conferences as a way to sort of test out and pilot, do many pilots for later full scale studies. And then there's another question. Who wants to do the pre-registration pay to those who spend the effort? I'm not, I'm not sure I follow that. So I might ask for clarification. Scott, do you have a. The way I'm reading it sounds a little bit like can someone be paid to do the pre-registration or, you know, and, you know, I think certainly that's not, that doesn't impact the effects of the pre-registration since the goal is just really to both prevent yourself from either intentionally or accidentally engaging in any of those questionable research practices like harking and changing hypotheses or adding data to get a significant result. Like it's just kind of holding yourself accountable. And then in the back end is providing trusts, you know, so that people can say, oh, this was pre-registered. So I know it wasn't, this finding wasn't due to that practice, which is something I see as a reviewer. So if that's not addressing the question, then by all means, throw another question or chime in at the end. So now talking about where to do the pre-registration, where I'm going to focus mainly on OSF, but there are other places out there. So I'll kind of hold that OSF and we'll look at that in a little more detail. There is as predicted, but that is really aimed for experimental studies. And so for me that doesn't always apply because I'm not always doing an experiment. So OSF is a better home for me whenever I'm doing pre-registrations. And also I'm just more comfortable in OSF because I've, I don't usually play over and as predicted. Now, kind of fun. This was a pre-print that was just released yesterday. So I dropped that in and it looks at some of the different places where people do pre-registrations and sort of the pros and cons and which ones might be, might fit your needs best. So GitHub, it was really not designed for pre-registration, but some people were sort of using it that way. It's a place to store your code and share information. But you can see that it really doesn't fit sort of the, what we would expect to see in a pre-registration space. As predicted is there, I'm not going to spend a lot of time talking about it. ZinMoto is not one that I have used either, but OSF has these two options for you, sort of two paths. And I want to kind of stop there and talk about the two different paths in OSF to see which one might be more appropriate for you. Both paths give you a timestamp, which is that critical that you have an official timestamp of when your pre-registration was done, finished and sealed. And then also both formats in OSF allow the pre-registrations to be Googled, so to speak, so you can search them. They're able to be found. They also persist. They're not going away. Those both websites, both options are going to give you something that is going to exist, you know, as long as we're around at least. You also have the option to do an anonymized pre-registration, and this is helpful when you are submitting to a journal and you want to submit the pre-registration for the actual study once it's been conducted and say, hey, this is what I did, and this is what I pre-registered. You can submit that in a way that is anonymous. And so the reviewers can see the pre-registration along with the timestamp without seeing who you are. Both of them allow you to add additional materials as needed. This sandbox is kind of a fun space because it allows you to go and play with the pre-registration without making it public. So if you've never done it before, you can go into the sandbox and sort of create a pre-registration and play around with it without creating a bunch of like junk or fake pre-registrations in your own profile. So I like using that sandbox when I'm teaching because if I'm using, you know, trying to show my students how to do a pre-registration, then the sandbox is a nice place to do that without creating all of these pre-registrations in my own personal profile. Okay, then this is where we start to get into differences between template and open. The template is literally a template. So you go through and you drop in the information, it gives you the questions that it wants you to answer, and you fill that information in. So it's a template, it's very guided, but because it's very guided, it's not terribly flexible. So you can't include within that template, you can't include images, for example. So if you have a picture, you don't have that flexibility within the template. But in the open side, it's like a word doc. So you can do whatever you want. You can include whatever you need to include. You have much more flexibility with what it looks like and what gets put in. But it is also a little bit harder because they're not going to step you through it as much as the template is. The flexibility in the template is pretty limited. The flexibility in the open is kind of whatever you want to do. The collaboration, and so going back to template, in the collaboration part, multiple people can be on an OSF template, but it's not like a Google doc. You can't be simultaneously in the template. So you have to think about, okay, who's going to be in it at what time doing what edits kind of thing. Because it can't handle multiple people being in it at once. Whereas the collaboration on the open side is however you want to do it. So if you wanted to do a Google doc, you could all be in it at the same time. And then for usability, the template is super easy to use. It's really nice because it makes you think about things that you might not have thought you needed to pre-register. Whereas the open is a little more difficult because, you know, if you give yourself enough rope, you can get into trouble. So, all right. So moving over to where I'm going to just briefly talk about the templates that OSF provides. I'm going to show you an example of a pre-registered study, an example of a completed pre-registered study, and then some resources. So the next four slides, I'm just going to go through really quickly to show you some screenshots of those different options. But you can also link to them in the slides and they're going to be publicly available for you so you can see them and play around with them at your leisure. And then after that, I'm going to show you how to actually initiate a real pre-registration using OSF. So first slide, this is just a screenshot of the registration forms and templates that are available to you through OSF. And you can see that there are lots of different types that they give you lots of different forms, templates, and all of the various options are over here. You can do Word, you can do Google Doc, whatever you want to do. This is a pre-registered study and I think the timestamp got cut off, but this was one that we pre-registered in July. My student, Blaine Peterson, is the one who did this pre-registration and we were looking at, so we have our hypotheses, we have the plan of analysis, and this study is now under review. And we have linked this pre-registration to the paper that is under review. And then this next one, I think this was Scott's first pre-registration, which is now published, which is amazing. So, so you can see that he timestamped it in April 2017, and then we are using that information, then he used that within the publication, and then you want to talk about how that worked. Yeah, so this was a really cool thing that I did after learning about pre-registration as an option for the first time, then I was working on this study and thought, hey, let's see, you know, let's do this, let's practice this, kind of like I think Karen has her students do now. And so what we did is my colleague Jennifer Jolly and I, you know, we thought through the data that we were going to have, we didn't know how many people we were going to get, so this wasn't secondary data analysis, this was a survey study in Australia. And so we came up with our hypotheses, we came up with our research plan, we came up with our plan of analysis, you know, use our power analysis and everything like Karen was already describing. Then we, as you can see, kind of timestamped that stamped that in April, and then I think it was actually in the fall, we started doing our data collection, and data came in and once we were done, then we implemented the methods as we said in our pre-registration, we submitted it to a journal, including the link to the pre-registration. And so, again, I mentioned this a couple of times, but the power, you know, what, what does the pre-registration do? Well, first, it kept me from going and fudging the numbers for any particular reason, maybe just to get a statistically significant result. Let's say, you know, this was about gifted education. Let's say I wanted to publish this in a gifted and talented journal, and I really wanted a publication so I could have 10 year. So I found ways to analyze the data such that it made gifted education, you know, look good. You know, I made sure to exclude certain data or just do supplementary analyses or whatever. I was prevented from doing that. I held myself accountable by doing that. So that was good. That way, the proposed research is actually more representative of the actual research that came out. On the other side, readers or reviewers, depending on how the journal works, if they wanted to, I'm guessing most didn't, but if they wanted to, could go and say, Okay, okay, let's see. You did this in the study. Is that what you actually promised to do before you saw the data? So it's making sure there was no funny business or flexibility in analysis or everything. So it made sure that what I actually did in the study was more true to what I actually plan to do, as opposed to being driven by any kind of other incentives that I might have. Yeah, Scott, did you get any feedback from others? Feedback after the pre-registration. Gosh, I feel like, you know, 2018 or 2017 was like 20 years ago in COVID years. I'm trying to think about after the study. I will say one thing that we published this study in, you know, we conducted it in Australia, and I'm based in Wisconsin, but after the fact people in the U.S. didn't know what to do because it was in the published literature and thought to themselves, Oh, I'd like to do that on my sample like or on my program or in the U.S. And all they have to do is go to the pre-registration and they can basically replicate the entire study because the point of the pre-registration is it has sufficient detail that people can do that. It's like the entire recipe, which as somebody else in the chat noted earlier, often we don't have enough space in journals to put the entire recipe. We don't have enough words. So I was able to say, Oh yeah, go right here. Here is everything you need to know. And as Karen mentioned earlier, you can keep that private at first. But then eventually you can see on the top left there this becomes public and is linked from the article so people can go and get that supplementary information. I don't. Yeah, I don't think we got any feedback after we pre-registered it but we definitely got feedback beforehand because we wanted to try to think through as many pitfalls as possible and make it like a really strong study design. Yeah. Okay. All right, so this is just another one of the OSF pages that has more resources for you about how to do pre-registrations it's got some templates it's got examples so that's just another link that I have for you in there. Then when you're actually in OSF and you are wanting to pre-register your first study you're ready to start putting pen to paper. There are, for some reason it wasn't intuitive for me so I wanted to give you like the literal like click here click here so the first thing you're going to do is you're going to log into OSF. And then you're going to go to my projects and click on that and when you do that it's going to open up the projects that you have in OSF. If it's a new project you can just add a new project. Oh, thank you. So, then after you click on the project that you either added or is already existing, then it's going to open it up and you'll see that that's this third window sort of kind of sitting on the bottom right. And then you click on the button registrations so your registrations live inside a project. And then when you click that registrations button, then you're going to get all of these options about the type of registration you want to do. So, we have the pre-registration, you can't see my cursor. We have the pre-registration and we have the open ended which are the two that we just talked about on that pre-print that just came out yesterday. You also have a qualitative pre-registration. I've not done this but I know it's, you have tried and are sort of in that space so there is a way to kind of help you think through qualitative pre-registration. You have secondary data pre-registration. Do you have all these various options that you can pick about which path you want to go? All right. Now, the analysis planning is harder than you would think it is because you're doing this in the absence of data for the most part. And so trying to think through all of the different problems that could come up or the ways that you're going to check the assumptions is it's not as straightforward. It requires a lot of thought and sort of careful planning. Now, planning your analysis does not actually mean it's an appropriate analysis. So, for example, Scott and I have a registered report under review. We'll talk about registered reports later, but we have a registered report under review. And the dependent variable has to meet some specific assumptions in order for the analysis that I plan to actually be appropriate. And so I have this really nice draw or I think it's nice at least I have this really thorough plan of what I want to do but if I don't meet the assumptions of the dependent variable, then I've got to sort of go back to the drawing board. It's a little more problematic in a pre-registration than it is in a registered report. But I did try to kind of detail what I would do if the, for example, my dependent variable I expected to be continuous but there's a possibility it actually might be more of an ordinal situation. So I had to sort of think through what would I do if my dependent variable actually ended up being more ordinal than continuous. You want to think about your edge cases. What about outliers? How are you going to detect them? What are the assumptions that you're going to go through and check? How are you going to examine your residuals? All that kind of good stuff. And think about how you're going to measure your variables, how you're going to combine your variables. What are you going to do if you have a low reliability on some of your scales? What's the plan? How are you going to handle your missing data? How are you going to find it? How are you going to handle it? How much missing data is going to be too much? Exclusion criteria. So if you're doing an experiment and people fail your manipulation check, what's the backup plan? How are you going to detect careless responding? Also, accounting for multiple comparisons. I'm all about nested data. So this is sort of in my space. But what are you going to do to correct the family wise test rate error? If you have the same data and you're going to conduct 10 different tests on it, what are you going to do? Are you going to keep your alpha level the same? What's that going to look like? And then you also want to think about what if a randomization doesn't actually truly randomize your groups and you have some differences between your two groups. Manipulation checks, that kind of thing. All right. I'm going to pass it back to you, Scott. Okay, you can probably pull all these up on the page. So we want to compare a traditional publication process to a pre-registration to a registered report. So these are timelines that I think OSF developed and then Karen very helpfully kind of put in I think some of the some of the arrows maybe it doesn't. So you kind of think of a study from idea development to design to collecting data to writing it up to publishing it. And the traditional approach is that we do all of that sitting at our desks. And then after the writing part we submit for publication. That's kind of the traditional approach. So in a pre-registration, after we've come up with the design, we stop, submit that pre-registration. I'll also note that's where we submit a dissertation proposal or an IRB in the US. So ethics approval. Then once that is all approved, then we collect the data and write the report or write the report in the study. So when a registered report is different, then instead of just submitting a pre-registration like to an OSF, you know, site like Karen mentioned, you are submitting that literature review and design to an actual journal that accepts registered reports. So in a pre-registration, you're taking everything Karen just said, uploading it to the internet so that it is freezed for all time. So in a pre-registration report, you can also do that. You can submit that to an OSF or wherever. But you're also taking the intro, the lit review on the methods, much like a dissertation proposal, hopefully shorter. And sending that to a journal that accepts registered reports. They then peer review that the same way they would peer review any journal. So in a pre-registration, they're not going to review it based on, say, the positivity or their particular interest in the results, because in truth, the results shouldn't matter. What matters is the quality of the science, which is not dictated by the results or the spin, you know, the discussion section, the discussion about it. So in a pre-registration, it's kind of the middle part. This is why I mentioned earlier, it's kind of independent from the journal, because you can pre-register a study that's going to get submitted anywhere or submitted nowhere. In a registered report, you really have to go to a list of journals and find out who accepts them. I will say, gift to child quarterly and also exceptional children are kind of special ed and gifted education journals, also AERA open, are kind of a three big general education, gifted education, special education journals that accept registered reports, and I know Karen and I have published registered reports and some of them. So hopefully that gives you a scope of like when things happen. The other thing I would just emphasize is, in a registered report, you also get that feedback from the reviewers. It really is much more of a collaborative process that you then revise and submit. And so when you finally submit any kind of final, you know, what is final methods can be a little bit more foggy in a registered report, because there are rounds of review that happen. It's just happening in a different place. Whereas a pre-registration, once that timestamp is online, you're not necessarily getting feedback on it before you go forward. Next slide. So this just kind of shows you, this is a really great visual I like, is so you can think of that big blue box is pre-registrations. So that's where you're just time stamping what you say you're going to do. It's publicly available. It's published before, it's put online before you do anything, but there's no in principle acceptance. You know, it doesn't influence publication at all. Then you've got RRs, which are registered reports. And you'll see that some of that RR circle goes outside pre-registration because there are registered reports that never have a separate pre-registration per se. It's kind of both ways, because in a way it's a little redundant. In theory, you've already kind of timestamped your methods by submitting to the journal, but there's also benefits to having it publicly online. So it's kind of, there's a little bit of debate about that. Then you got, so you've got those registered reports, which are part of a journal. And that's where you have your manuscript accepted before you ever even look at the data. It's kind of crazy. You can have a publication accepted before the data are even released, like Karen was mentioning with Tim's, or I did before, it was a pre-registration, but before you even collect data if you're doing primary data collection, which is nice because it can give you kind of confidence. Like you don't have to worry as much as long as you do what you said you were going to do, it's going to get published. That's all of those registered replication reports. Just to add one more term to confuse people. And that's just a replication study that has been done as a registered report. So it's just kind of a kind of registered report. It's not really anything, especially different from that. Okay. Again, just to compare pre-registration and registered reports because there are a lot of similarities. You can kind of see the biggest difference is really the format. Pre-registrations are more informal. There's a lot of different formats they can take, but they're not as rigid and structured as like a registered report because a registered report really is like the first half of a manuscript or even two thirds. Intro, lit review, theoretical basis, and then pretty detailed methods. They will expect more detailed methods in a registered report than in a typical paper. I've found that that's my biggest challenge with the registered report is I've submitted 11,000 word phase one registered reports. So that's 11,000 words of just lit review, intro, and methods. No result, no discussion, and we're already way over the journal page limits. That's I think the biggest challenge registered reports is then going back to the journal and saying, how do you feel about a 16,000 word manuscript? Timeline, I think we've kind of talked about the timeline a lot. I guess the only thing I'd say, we're talking a lot about registered reports, but there's stage one and stage two registered reports. Stage one is the pre-data analysis where they're reviewing things before you ever look at the data or analyze the data. So once you have the results and discussion written up, you resubmit, but in that resubmission, the reviewers are not checking the lit review and everything again. They're just checking how well what you did and what you wrote up matches with what you said. And I've had reviewers come back before that have tried to go outside those bounds. And then I say, ah, that's not part of what we said. Like, this is what was agreed on. We're doing this. We're not allowed to do any of this other phishing expedition stuff that you want. And every time they say, okay, fine, go on. So it does keep you from having to play this game of whack-a-mole of trying to please the reviewers after you've done the study. Okay, so thinking about which one's going to be right for your study, whether a pre-registration or a registered report. For me, it's really about time. If I have the time to go through the process with the reviewers, I've written the plan up and I've decided on a journal that takes them, then I find the registered reports to be the way to go. But because you get this back and forth with the reviewers before you actually do the analysis, and I like the guaranteed publication, if I do do what I said I was going to do, then I have a publication. Then the other thing is that if you are collecting data really soon, like I have a project where I'm collecting data in schools and I just don't have the time to go through and get a registered report done because the data are coming in within the next month or two. So in that case, a pre-registration is a much better option for me because I can still timestamp it, I can still talk about my plan, I can identify my hypotheses, and so it's a time issue. If you've got the time, you're not in a hurry, register reports, if the data collection is imminent, then a pre-registration is probably going to be better for you. Both are very similar, both are transparent, and neither of them completely tie your hands. In both cases, I have deviated some from my plan. Nothing has gone exactly as I thought it would. But so far I've been able to be really transparent about when I deviated, what I did because of the deviation, what was the rationale for it. And before I started doing register reports and pre-registrations, I wouldn't have made that transparent. I would have just presented the final product, not all of the paths that I took to get to that final product. So it's actually improving the quality of the, or just the explanation of how I got there. Alright, and then just some practical advice. It doesn't have to be perfect. Just get close to what you think you want to do. You're not completely bound, but if you change, you have to explain why. Pre-registration is really good for confirmatory analyses, hypothesis testing, but you can do them for exploratory. Scott and I have a register report under review right now where we have both hypotheses and research questions that we wanted to ask. The literature just wasn't mature enough for hypotheses and some of the things that we wanted to examine. It does save time in the long run, but you're going to put more work in at the beginning and use pilot data as much as possible to sort of think through and work out the bugs along the way. And it also gives you like the code that you might want to go ahead and include in your pre-registration. And commit. Don't use words like may or might like do write down what it is you plan to do, and then when that doesn't go well, then you'll explain why you made different choices. Okay. Common concerns that I hear. What about qualitative research? And so that's not my space, but we have linked you to some resources about that. And the one that I wanted to address is that fourth bullet. If you've got a research team or an advisor who may not be on board. And you may be like, like, if I can talk to just the graduate students for just a second. Generally speaking, those who are newer to educational research are more receptive to pre-registrations and registered reports. So if you have an advisor who has, you know, is a scholar and wonderful in the field, but they are kind of new to this space, keep in mind that they are still a gatekeeper for you. So I wouldn't, I wouldn't die on that sword, because you want to make sure that you can actually get through and get out so that you can start producing work in the pre-registration register report world. Okay. So, to kind of temper your pre-registration enthusiasm or your register, like pre-registration is generally acceptable. But if your advisor is not open to register reports just yet, just make it into the field and you can be one of those that makes a difference and it does more register reports. Are you more likely to get scooped? I don't think so, because you have a timestamp. And so that's kind of nice that you've got that timestamp to point back to. Multiple studies in one paper. As you can see at the OSF site, you can have lots of different pre-registrations within the same project. So that has not been a major issue. And we've talked about what to do if you need to make changes. All right, so before we do Q&A, I want to do a quick advertisement for a study just on register reports this afternoon with Betsy McCotch, my advisor, and Amanda Montoya. So if you want to get more detail about register reports, that is a good place to go. All right, so questions. See something popping up in the Q&A too. Yep, I've been kind of monitoring the Q&A. And some really great questions and very common kind of questions and concerns so very legitimate kind of questions and concerns. I asked a question about when something happens, you know, Karen was saying sometimes you have to deviate or, you know, the data are such that a particular analysis is no longer appropriate and you have to change that and oh my gosh what happens, totally, totally to be expected. You will have things that happen where the data are different. You don't get enough sample size. So when it happens and so your data collection is pushed down the road or you change schools, you know that you're collecting data from all of that is to be expected. Nothing wrong with that. You don't change your pre registration, because it still stays what it is. What you do though is in your study when you're writing it up when you're presenting at a conference, you make sure to clearly delineate unexpected things that happened stuff that happened. So in some places you might actually have a section that says unplanned analyses, and that can be just exploratory questions that you thought of after the fact or when it when seeing your data you thought, oh, we should really follow this up. You can do that even if it wasn't in your pre registration, just be transparent about that. Just be clear that's what you're doing. You know, that's that that's what this is all about. Because that way your reader can say, okay, this wasn't a planned analysis but his or her rationale for why they're doing it makes total sense that that's completely fine. And Matt has a Matt Makle just posted in the Q&A. There's a really nice way to remember the difference between pre registration and register reports. So pre registration is one word, and has only one submission. But registered report has two words and it goes through two submissions. So that can help you remember the distinction between a pre registration and a registered report. Thanks Matt. All right, I have somebody who raised their hands. How do I go to them. Oh, here we go. I can talk. It was. Let's say how do I do that one attendee raised there. I got to find. I can. Oh, there we go. Mark allow to talk. Okay, Mark, you should be on the air. Hi. Yes, my name is Mark. I'm actually one of the product owners for OSF registrations. And so I just kind of also wanted to highlight that we've actually released what's called updating for pre registrations and register reports that we released late last year. So if you do that, if you saw the images that you guys showed earlier, it was like a little update with a little drop down, you can click that, and you can update some of your responses. That way you can transparent transparently disclosed what changed and also get the explanation as to why, but you still keep that original registration so it's still that that time stamp for each kind of update because like you said earlier, research is messy. There's things that we don't. We need to be transparent about that. And it gives the research study more like a story of what you thought was going to happen and the story of the research of what actually happened. And that's a really a big benefit I think of a lot of these platforms and again I'll give a shout out to OSF because it's what I use is you can have a whole project web page. So you might have the pre registration link from the web page but then you also have other things on the web page. So if something did come up, you can include additional details about that you can also include a pre print which you can share later. It doesn't just have to be kind of a one and done thing of just the pre registration. And I think we're pretty close to being done unless we are. Yeah. Carrie I know I tried to answer your question in the chat but follow up with me if I didn't about not getting all your sample size and everything but otherwise the biggest things we'd recommend you know check out Betsy McCoach and I think it's Betsy and Amanda Montoya session later on registered reports. We're obviously big fans of that as well. There's a number of resources linked to in this session that you'll have available, and otherwise just a good place to go is just to OSF because you can just look at so many example templates of different registered reports from your field. And that's in my experience some of the best ways to see you know what are these look like what are you supposed to do. Very cool. All right, well thanks everyone. Thanks everybody for your time. Pleasure having you hope it was helpful.