 Welcome, everyone, to Love Methods Week. My name is David Meller, Director of Policy here at the Center for Open Science. I'm very happy to start today's webinar, a pre-registration, a plan on a prison. This is one of the events for holding for Love Methods Week, so thank you for everybody's been participating in Loving Methods this week. Let me go ahead and get started. I'm going to take a moment here and put a poll out. Actually, I'm not going to, and then the background is going to put a poll for everybody to fill out. This just gives us a little bit of information about your familiarity with the process. And we're curious to hear about what your background is in relation to pre-registration. So anytime, just please fill out that poll. Next slide. It's my pleasure to introduce everybody. We've got a panel here of COS staff members talking about the ins and outs of pre-registration. As I mentioned, my name is David Meller, Director of Policy here. I'll be getting us started and then happily handing it off to NOAA, our Principal Research Scientist, who will be going into a lot of the background about the rigor and benefits that the process can add to your research workflow. Mark will then, Product Owner here, give a demonstration of what the OSF registry includes and what the workflow looks like on this platform. Finally, Crystal will be wrapping it up with information about common pitfalls, questions that arise, and we'll be moderating the Q&A session. I'll just pop it up right here. Throughout the webinar, please feel free to use the Q&A box. If there are questions that arise that are about clarification of what's being presented at that moment, we'll interrupt our programming and try to answer that on the fly. Otherwise, we'll answer most of the questions at the end of the webinar. Let's go ahead and get started. Next slide. I wanted to give a little bit of a definition and a tiny bit of history about what registration is, just so we're all on the same page. When we define registration, we are asserting that it is a read-only version of a research plan that's posted in a public registry. Those have some qualifiers next, which I'll get into in a moment about those qualifiers. Sorry about that. I also wanted to clarify the fact that the same process is common across different disciplines, especially in clinical research and clinical medicine. The term is registration, or if emphasizing that's done before the study or before patients enrolled, perspective study registration. In a lot of disciplines, use the term pre-analysis plans to emphasize what type of information is created before the study takes place. Now next, I just wanted to emphasize the fact that most of the registries do allow for updates to the plans, although those have to be transparently and clearly described in a format that's easy for the reader to see when that happens. Also several registries allowed for those public entries to be embargoed for a period of time. Next, the brief history of pre-registration is that it has been proposed for a long time across different disciplines, but it only became standard practice especially in clinical medicine just about 24, 25 years ago. There was an ongoing problem of government-funding cutting-edge research in clinical medicine and constituents reaching out to Congress trying to find where these trials were taking place. Clinicaltrials.gov was created to solve a very specific use case to help connect patients with those cutting-edge treatments, but from there on out the benefits to the process accrued and were realized by methodologists and by others who realized that having a central repository of all empirical research taking place adds benefits to increasing transparency of the research process, improving understanding of how much research is actually conducted every year regardless of how much ends up getting published, and overall improved methods and rigors. Next, in addition to helping those benefits with clinicaltrials.gov, including it in basic or pre-clinical science, can add transparency to the whole research workflow, and it can help distinguish planned research from unplanned work. There are a couple of questions in the comments that we will get to, I promise, but just for a quick clarification because it does affect others, well, Registry Reports is a publication process by which that proposed research plan, what could become the pre-registration, itself undergoes peer review, and if accepted, a promise to publish regardless of what the results turn out to be. I'll mention that a little bit more, but it's a peer review plus a promise to publish that final result in addition to what we're going to be talking today specifically about pre-registration. Next, I'm going to start handing it off. The time to do pre-registration is right at the beginning of a study. Ideally, there will be one pre-registration for each quote study, unfortunately, that ideal is a little bit different from discipline to discipline with what individuals count as a study. Sometimes it's connected to a very specific data collection process. Sometimes it's connected to a particular effort. Next, in the complex reality, what we encourage folks to do is to create the pre-registration right before data access or data collection, and if there is going to be multiple papers resulting from the work that was specified in the pre-registration plan, think of a final reader being able to connect all the dots so that they can find where the results of the registered research plan end up getting disseminated or distributed. So if those are across multiple papers, those should reference each other, and links should be added to that original pre-registration indicating where results are being published. Finally, I'm happy to hand it off to Noah for you to the next slide about pre-registration enhancing significance in the research design. Noah. Thank you. Appreciate it. Some theory of why we do pre-registration, what we think it might actually impact in research. I want to start by a kind of very simple model of research that I think a lot of people think of, which is the straight line between the question and the answer, a question and a result, where we think of this as following a single path. We start the question and we might end up with people with X were found to be why times more likely to have Zp equals 0.044? This is a pretty common thing, but in reality, what actually is going on underneath the surface of it has actually happened is quite a bit more complicated. So next slide. Sometimes this is called the garden of forking paths. Along the way there are thousands or hundreds or however many different choices researchers could have made along the way. That could include things like what things were reported and what things were not reported, what outcomes were selected. Put data cleaning in quotes because data cleaning is not really a well-defined process. It's actually part of analysis. You can make many, many, many different choices for how you cleaned data. The model choices, the control sets, how you actually frame the questions, all of these things, all of these decisions go into a research process and there's not always a clear answer of what is the right way and what's the wrong way. And they're all kind of mixed in and all we usually see is kind of the one path at the end. Next slide, please. And another concept to really understand here is the idea of exploration versus confirmation. This is an overly simple model, right? There is no, this is not a clear dichotomy, but it's important to understand for one of the things that pre-registration might do is help distinguish these things. It doesn't favor one of them. It distinguishes between them conceptually. So on the sort of exploration side, you can think about something like the first time you look through a microscope onto a slide, you have no idea what's going to be on that slide, right? So we're letting this kind of what we see drive what happens. We're going to let the data drive the thing, what we're doing. We're going to be more sort of discovery-oriented areas that have truly new ideas and things that have never been seen before are inherently more exploratory. We will find things that are unexpected. That's kind of the point of this, is we don't know what to expect. And we can often drive model and theory, theory building from more exploratory analyses. Now that's opposed to confirmatory research, which is maybe a research that most of us are much more familiar with, with like news headlines and drug trials and those kinds of things. We start with a hypothesis, typically, or something like a hypothesis. And that drives what we're looking for. Usually when we think about significance testing for better or for worse, we are thinking about a confirmatory framework. We are saying that this is, there was one question, there was a P, you know, our P is meaningful, really generally only under this, this idea. P values and so on can be used many, many ways. But usually that's the way that we're, that are, they're being used or at least interpreted. We often think about confirmatory research as having a higher standard for rigorous evaluation for evidence. That's often the case again, things like drug trials and so on. And we want them to have a much more direct decision impact. We're doing, we're trying to find something much more solid, typically, in confirmatory research. We start with a specific question, we want to get a specific answer. Next slide, please. So if you're a research consumer and you see that result, that people with X result, right? Is that the whole story? And generally no, right? We don't know where this result came from often, right? Was this, as things planned, there were there changes that happened along the way? Was this P hacked? Was this part of a phishing exercise? Was that hypothesis that we saw at the beginning actually made after we got the data? That's often called herking. How do we actually use this data? How do we use the results if we don't know where they came from, right? The story of the data is in part the, you know, the meaning of the evidence. So for my research consumer, the pre-registration gives us that story, tells us what happened along the way so that we can interpret the results. Next slide, please. From the producer, pre-registration does all kinds of things. My personal favorite thing that pre-registration does, and it's, and this may be surprising, is it just lets you plan better. So if you start with an idea for a study, it lets you write out that whole comprehensive view of what is going on, right? I want to start with, here's the result that I want at the end. Now I have to work backwards through all of the methods. It helps me think about what it is that I'm going to do. Often, you know, things like protocols for IRBs don't contain those sections quite as much. So once you think about what the analysis and the plan and so on is actually going to do so that you're not in the situation which all of us have been in, which is you get to the end of the study, you've gathered all your data and it's too late, you've done something, you know, you've done something that can't be undone and you wish you had done it again. It helps prevent some of those problems. It also helps with enhanced feedback, right? So the earlier you get feedback, the better. Again, we don't want to get feedback for methods after the methods have already been done and we've already run all our tests and so on. It's nicer to have it at the beginning so we can have that comprehensive study design. And it also helps clarify what it is that's actually being critiqued, right? We're talking about the methods and not necessarily the results at this point when we get that kind of feedback. Often, it's hard to tell when somebody is critiquing something, whether or not they, you know, don't like the results and therefore it's the methods that are wrong or, you know, we're talking purely about kind of the methods and the plan at this point. And sometimes, not always and not inherently, we can improve the credibility of our research. So that's particularly true for confirmatory tests. Pre-registration does not mean credibility inherently. It just, you know, it could be a helpful sign towards that, particularly if it was well followed. And that's particularly through this kind of mechanism for limiting temptation for questionable research practices or QRPs. But we're setting kind of a default. Next slide, please. So the idea of this is that we have all of these many, many options that we go through in the research process, but we're saying, okay, we have chosen one particular pathway or a set of pathways. This is particularly for confirmatory tests or pre-registrations. And this will help limit us from, you know, the temptation to go into these other pathways if we don't see what we don't like or what's not publishable or something along those lines. It helps set a default. Doesn't mean you can't do those things, but it sets the default option as being this is what we're going to do. Now, often you will see different paths as you go along or realize the path you had planned on going on doesn't work for some reason. So this is not, you know, a prison. This is a planned pathway. And next slide, please. And of course, there are many, many, many types of pre-registration. Often we talk about them in terms of, you know, very highly confirmatory tests. But there are many times that you can pre-register. There's many styles of pre-registration you can absolutely pre-register and maybe should pre-register exploratory analyses in many cases. So I'm going to pre-register that I'm going to look into that microscope and document what is going to, what I'm going to see even if I have no idea. That is a very helpful thing. And depending on how you pre-register and when you pre-register and what the purpose of it is, and if it's going towards, you know, as someone mentioned earlier, a registered report, there are many, many, many different ways, orientations towards how these pre-registrations can work. And they might have different, you know, impacts on bias reduction or the confidence in what discovery, how much discovery we have. And that will all depend on how that, what we actually do during the pre-registration process. So it is very much not a binary. It's not even a spectrum. It's a multi-dimensional thing that you can explore and have some fun with. So at that, I will turn it over to the how for Mark, with Mark. Yep. So forget that. We're going to let me control screen for a little bit. Also fantastic. So we have spoke a bit on what pre-registration is, the benefits, etc. Well, the Center for Open Science also hosts the Open Science framework where it provides a place where you can upload your pre-registrations for free. So we're going to look at that for a quick second to see exactly what does that actually look like in the. So here is a demonstration of one of the registrations that actually exists on OSF. We're going to go through that together and we're going to start from left side over to the right. So on the left side panel, you can see that right now, we are on the overdue page, which you can see because it is highlighted. We have several other different tabs. I'm not going to go through all of these just because of time, but I want to touch a few that are important. One of them being metadata. Metadata is how you can add information to your registration and other objects on OSF to make sure that your research and your outputs, your materials are discoverable. So if you're ever wondering, how do I get this out on the road? Fill out the metadata. Not only does it make it easier for people to find yourself in OSF, but we connect with other different types of platforms, too. And it makes it easier to find that and access that on there as well. Another one is files, where anytime you submit a registration, if you have a project, it will archive on those files automatically for you. And that can include things such as you have a project, you've uploaded your pre-analysis plan, stimuli, code, all that's right there, it will automatically archive of that for you. If it's not on there and you decide to upload it for a specific question, it will do the same thing. So this way people can get all your materials and replicate your study using the same materials that you did, which streamlines scientific progress and shows replicability. The last one is components and links. Right now, we can see that this one both have zero. That just means that they don't have other components, which are our folders with permissions that are attached to this, and they don't link to other types of pre-registrations. So there was a one of those instances where David mentioned earlier that you tried to register individual types of projects, even though it might be a part of a bigger one. This is how you can link them and connect them so that way they can go between them as needed. And then the last thing I'm going to go over is going to be resources, which you can see here or by clicking here. So with this, this is where you can aggregate any other types of outputs, for example, data, publications, et cetera. So that way, you can say, hey, thunder, I did a pre-registration last. Here you go. Here's my original plan. Here's all updates that I'll chat with you guys about later. As well as anything that happened, all that. So a pre-registration helps create that foundation that you can continue to build upon and have all the information right there and send them a single link and have access to all your materials. So that is this left side panel. Now I'm going to go through the middle panel. So the middle panel is going to depend on the type of template that you're going to use. Earlier, we had a question of what is the difference of the OSF pre-registration template or the open-ended template? Well, there are several differences for that, for those different goals, and we are actually actively building those out. So if there is a template that is not available that might be helpful for your discipline or community, connect with me. I want to chat with you, because we want to make sure that all disciplines, all study designs are presented. And if you would like to learn more about the different templates, we'll go right here in the chat of one of our health guides. And you can go to, you can get access to full docs that you can copy or download and build off of as you're trying to explore or as you're exploring these different types of behaviors. So this one is based off of our OSF pre-registration template and is one of our most rigorous. It's designed to challenge you and your research team through all the different nuances of your study design, even things you may be not even considered. So with that, we have a structure where you have things such as the information. We have our heads, our main sections, as well as question labels. So that way it's easier for you to scan through here and see all the different pieces. So I can easily go through here and say, there's a hypothesis. I can learn about their study design if they include a randomization. You can also navigate them by clicking these three little hamburger signs, click there, and you can see that structure here as well and how it's separated from study information, design plan, sampling plan, analysis plan. It's easy to click through and get through all those different sections. And if you're a tech savvy and know how to use APIs, you can pull the same public information through our API as well. So, and then lastly is going to be our white panel. This is again, we're all out of that metadata where it lives. So you can see contributors, those that actually help participate with that pre-registration. A brief description that kind of says exactly what is the goal of that study and other information that may be relevant. Date register. This is really important because that's where you get the initial timestats saying this is where I submitted the registration, it's done, it's been archived, it's been either public or embargoed, which everyone you need. Embargo is just a fancy way of saying private for up to four years. And you can start data collection. So you can say that here's my plan. I started this date, data collection, this date reduces that bias. And a couple of other helpful links, like the project, if you want to continue building all those materials, a DOI, all registrations that are public are automatically given a DOI, which is incredibly helpful when you're doing your papers. And then as well as you can license your work. We have several different types of licenses and it really depends exactly what you need. If you want people to be able to build up on your work, to modify and edit it, but still credit you, we have a license for that. If you want to claim it and say, I don't want you to do anything with it, we have a license for that. So a bunch of different licenses that we can use. And those are also available under help guides. If there's one that's not there that you think is helpful, again, connect with me. My job is pretty much to make your life easier. And then lastly, we have tags. Tags are things where it helps, again, get to that discovery piece. What are some of the key words that are important for your study, as well as your discipline to help people find your information? And one more thing, citations. I'm horrible at them. I use APA. I don't use the other ones. So it's kind of really helpful for me just kind of going here and saying, what citation did they need for this type of journal? I can click it, copy it, paste it. Boom, done. So those are the main things about a registration. And then one other thing that I want to mention is registrations, like they said, it's not a prison, it is a plan. Things change. Great example is COVID. You can expect to collect a sample of 200 people another disease has. Hopefully not. You may go down to 20. You need to reflect that. And the way to do that is through updates. So when you have one, you'll have a little button that says, start an update. You can change the content, study design, et cetera, whatever you need, as well as the justification of why that change was needed. This person only has this one initial pre-registration, but if someone does update it, you will see all those various updates right through here. And you'll see something the very top saying this was changed and why it was changed. So all that is transparent for you to have access to. And so with that, I will stop sharing, giving that back over to my team and I'll pass it on to Crystal. All right. Hi, everyone. Actually, Mark grabbed the beginning, but I wanted to touch on, we do get a lot of questions about updating our pre-registrations. And so you might wonder, like why might somebody want to update their pre-registration? A lot of different things can happen. You could notice that you had an error, maybe you mislabeled a variable or you left something off. You can go in and you can change and update that. You also might, as you're going along, realize that you want to add an analysis. And so you can go in and add that. I don't know if this happened to you, but it's definitely happened to me where I might pre-register my study thinking that I'm going to analyze it in one way. And then I read a paper that changes my mind about the way I should be analyzing my data. And so maybe I want to go back and update according to what it seems like best practices are for the field now. So you might do that. And you might also, again, as Mark said, you might realize that you have to just make adjustments. Maybe you ran out of funding or maybe you've just had equipment break or something like that. You've had to adjust, maybe use a different agent, that kind of thing. You might need to make some adjustments and update that. There are three ways that you can do updates. Mark showed one. I think that's the one that's on the next slide. Nope, this is... If you go to your registrations when you log in, you will see a list of all of your registrations. Once it's been submitted, there will be an update button. And so you can click on that update button and it'll take you to the page where you can add an update. Another option is to do it on the OSF within your projects. You can click on there's a tab that says registrations. And so you can see all of your registrations for that project and you can update them through that way. And then the third way is the way that Mark showed. You can put in an update on the actual registration if you're in it. And as a reminder, any changes should be accompanied by a justification. So another question that you might get is there isn't a template for my kind of research. We do have a lot of different kinds of templates on the OSF, but we don't have all of them. And some of them are also community-made and there are some options for that as well. One of the best ways that you can still use the OSF to put in a registration is to use the open-ended registration. And so what that'll do is give you a slightly different layout than what Mark showed. If you go to the next slide, you can see it just has a summary box and it has a narrative summary of what's in the registration, how it differs from prior registrations if you have any. And then you actually have an ability to upload a document. So if there's not a template for the kind of research that you do, but you still want to make sure that you pre-register, you can use this open-ended option format your pre-registration with all the information that you think that you need to have in there and upload it and you can still register that way. And the rest of that submission process looks exactly the same. Another question that we... Oh yeah. And again, as Mark mentioned, if you're more familiar with pre-registration and you've done it more and you want to create a template for the kind of research that you're doing, you haven't been able to find one yet, please feel free to reach out to us, particularly Mark. And we can see if we can get your template added to the OSF. You might also be wondering, like, how do I convince other people? Like, maybe I'm really excited about pre-registration. I don't necessarily know how it's gonna get sold to other people. I might get some pushback. What does that look like? How do I have those conversations? And there's a strategy that I like to use that's on the next slide, where I start by just listening to what people have to say. So listening to understand rather than to respond and kind of try to hear what it is that they're saying. If they're like, well, I'm not really super interested in doing pre-registration. What is it about in pre-registration that isn't interesting? So they might say, well, I don't know that I have time to learn how to do this and then to get into all of this. You might strategize on some ways that you can save time. So maybe you can offer it to be the first one to fill out the draft. I've done that before on collaborations where I'm like, I'll fill out what I think is going on in this study. And then we can talk about it. And that's a good way for us to kind of hash out any differences. Maybe I'm understanding some parts of the study that differently than the way that you are. Also, as both Noah and David mentioned, a lot of times when you're going through the research process without pre-registration, you end up spending a lot of time in that data analysis section of your process because you're going back and forth. Maybe you don't actually remember what the hypotheses were or people are disagreeing on what the hypotheses were or what variables you should be including and all of that. And that can actually waste a lot of time. But when you've pre-registered, you're having those conversations up front. You're making sure that everybody is getting on the same page. And then that can actually save you time in the analysis part of this research process. And so what you can do is you can kind of listen to what it is that they are talking about and kind of see like, what is that underlying concern? Another recommendation that I might have is to start pre-registration with a new project. Sometimes with a project that's already going on, even though you can pre-register at various points in the research process, sometimes if it's going to be your first pre-registration, it can be really overwhelming to do that on a project that's already been started or a project that has like really, really tight deadlines because you're learning a couple of things at that point, you're hashing out your design and all of that. But you're also learning how to fill out this form and learning how to, you know, what these different questions mean and stuff. And so that can be pretty stressful. So starting with a project that's completely fresh, starting with one that has either deadlines that are far in the future or ones that don't have deadlines at all, starting with those smaller projects too, maybe pre-registering part of a study rather than the whole study can also be really helpful to just getting people on board. A lot of times it's, I mean, we're all doing a lot. We all have a lot of things that are pulling on our priorities. And so it can be really hard when someone says like, hey, come do this extra thing, then people can really feel a resistance to that. But if you can show the value of this extra thing might actually save us time in the long run, might actually help us to collaborate better and to better understand our study, then that can actually be really beneficial. So describing how pre-registration can help them address their concerns and also to meet their goals. It's also really helpful. Next question. Another question that we've been getting more often lately is can I pre-register in a different language? And so we have a couple of options for that. It gets a little bit complicated, but if you want to go to the next slide, I think our recommendation right now is pre-register in the language that you feel like you can be the most accurate in. If nothing else, it would be a good idea to include both that language and an English description and the metadata, especially the tags, because that'll help folks to find your study. And I think there's another slide. A couple of considerations to that, like some caveats to that advice, you might consider what language especially if your funder is requiring pre-registration. What languages does your funder accept pre-registrations in? So if they want all the documentation in English, you probably should write your pre-registration in English. It also is worth considering what journal you want to submit to. If that's mostly English language or all English language articles, the reviewers are likely also to be English language reviewers. And so if they're going to want to be able to check your pre-registration, it's easier for them to do it if it's in English. Some folks might opt to, in their pre-registration, have it both in the language that they're most comfortable with and in English. So just noting that if you do take that option, which is valid, make sure that if you do any updates that you're updating both of those, both the non-English and then the English description. So with that, I would love to open this up to Q&A. We've had some great questions come in and some of them have been answered throughout. Maybe we can start off with Mark. There are a couple of questions about the form that might be helpful to answer. The first is asking if metadata on private registration shows up on external sites, such as an item that cites a given publication, sorry. I don't think so. It's only public registrations. Anything that is embargoed, we make sure that that is state private just so that way we are aligning with, well, prophecy concerns. So no, that is not provided in your rails. It is just public information. Most of that is pulled through our API and our API only includes what's public. Great. Thank you. We've also got a question and I think maybe I'm the best person to answer this, but how does pre-registration apply in the observational as opposed to experimental sciences, especially exploratory studies where you don't have a hypothesis, but you're rather looking for one from the data? So other folks can jump in. I'm a mixed methods researcher. I do a lot of qual. So I can tell you about why I like doing it for my qualitative research, why I like pre-registrations for that. You're right that qualitative researchers and folks that are doing observational research don't often have a strict hypothesis that they're trying to test or confirm any kind of analyses. I would argue that one, even for exploratory research, more confirmatory maybe than we like to admit. Like we don't go into a study thinking like anything can happen. Like we generally have an idea of what we think or what we hope is going to happen. And so it's good to really think through what you think is going to happen because that'll also help you to figure out like what's surprising later. Another reason that I like pre-registration for qualitative research is so there's this concept of reflexivity, which is where you're reflecting on the process, you're reflecting on your perspectives and how they change over time while you're going through the study. And so having something that's documented and written down, I don't know about you guys but sometimes I forget what I had for breakfast yesterday. So like I'm definitely not going to remember what I thought I was going to find like six months ago, like after 30 interviews or you know whatever I've done. So like having an idea of what I was thinking going into the study, what I expected to find, what I expected were going to be important questions, how I thought about the variables or the concepts that I was exploring. Having that worked out and written down and documented is really helpful for the reflection process later and for me to figure out like, oh, actually I thought this was going to happen and actually people started talking about this other thing. And so it can really help you to figure out like what is surprising and interesting about a study. If you have follow-up questions on that, happy to but I don't know if other folks want to jump in with any thoughts on observational research and pre-registration. Yeah, I'll jump in a little bit quickly because the terms observational and experimental get are kind of, can get a little bit murky here. So observational usually refers to non-experimental data so you have some data set and you're working with that. That does not inherently mean that it is exploratory or confirmatory, right? So you can have very much, in fact, most I would argue observational research is confirmatory. So they're not necessarily broken down quite that way. But it's usually kind of a mix, right? It's not binary all the way one way or the other and what the pre-registrations allow you to do is also separate things into different parts. Like this part of this research that I might have is more sort towards the confirmatory side. We're also looking for this step in the exploration side. So it's not so clear cut as to experiments versus observations either. And I like to say all causal inferences is observational. So yeah, you can do kind of whatever. It's really kind of your use that to explore the space. Use that to explore like what part of this is you're defining in what way. Awesome, thanks. Any other thoughts on that that anybody wants to add to? The one thing I'll put a link in the chat here there is a whole registry on OSF dedicated to observational studies and the consortium behind that that's supporting that is really focused on making sure that real world evidence real world data from observational studies gets treated to the highest standards of rigor and quality as possible. So that's one of the motivating factors behind that. The desire to show confirmatory research in an observatory setting. So just to reinforce a couple of points that both of you have made as well. Awesome, thanks. There's a great question here about resistance to preregistration. The resistance to preregistration that I've been encountering is the PIs are afraid that their research question or hypothesis will be stolen or published first. How do I address that concern? Someone in the comments mentioned the embargo option but I'm curious maybe David and Noah if there are other thoughts that you have around around scoping basically. Yeah, so this is one of the most common questions about preregistration by far as people tend to be very concerned about scoping. It's less clear how much this actually happens. So it's not clear that putting out a preregistration might actually induce somebody to grab your whole thing, copy paste it and do it and rush it right before you. So it's not actually clear that scoping actually is kind of the problem that many people think it is. It's certainly real but it's maybe not quite as big. Importantly, it also allows people to invite others as collaborators. It actually might say this is an invitation if you're thinking about this same problem let's work together on this. And then in some cases it actually serves as a hey, I am doing this. This is kind of in some circumstances it might be sort of a stake of like this is what I am doing. Maybe it's not the best to do a very similar experiment in this space. You actually might save somebody a lot of time by saying, I'm going to do this thing and if they have a similar idea but they just don't get to it a month before you that's you know that'll actually make everybody's lives a little bit easier. So it's not super clear how much scoping is sort of the big thing. But it you know it is also a real sort of concern. I think talking about it is usually the best way to address it. David, more thoughts. I reinforce that yeah just by saying that once it's in the registry and of course the Raven Bargo that's a you know a public assertion that's your idea made on this date and time. That's a citable object gives you the sort of the same intellectual protection as any other sort of public declaration that that's an idea that was generated by you now of course finding results from that declaration is kind of almost even more important. But the act of creating a registration is obviously a several takes a while to do that. A lot of work goes into those as well. So that also sort of gives you a leg up. So yeah, that's a long thing. Awesome, thanks. We've got a question here that might be great for I'm thinking maybe Mark. I'm trying to create a pre-registration for a confirmatory study. My study plan is informed by a small pilot study. I think I should include these insights but I couldn't find how to do this on the COS pages What's a good way to include the pilot? Is there a template that accommodates this? Where would be a good spot for it? So it depends. I'm actually going to punt this over to David because there are a couple of variables to help with that. I think he's better addressing that. Yeah, you don't have to mention that there's a pilot study especially if the data that was collected for that which the point of a pilot study should be to demonstrate that the method works, that it's a worthwhile question and method to pursue. That doesn't have to be included anywhere in the pre-registration. If you do decide to mention it, you put it in as context or this project is being informed by pilot data or methods that were piloted at some point and you could cite the master's thesis or you could cite an OSF project or another publicly available DOI or URL to assert where that data is coming from and where it's tested. Again, ideally you should not include that pilot data in the next round of data collection and data analysis because that data was used for constructing the hypothesis. You were really filling around with the methods and the analyses and figuring out what works and what doesn't. And then that data doesn't have service purpose informing that next study. So best practice is to cite that pilot study in the pre-registration but it doesn't have to be formally included in any way. Now the OSF does offer a lot of features where if you wanted to link an OSF project as a pilot study to your next study that there are a couple of features that allow you to do that even more firmly to even create a better documentation of a whole suite of studies that occurred. But strictly speaking since the pilot data is not should not be used to the next round of data analysis it doesn't have to be included in any way beyond a citation. And I'll put in some help docs in the chat here to mention how you could link OSF projects together. I also think most if not all of the templates do have like an additional information area. So if there's anything that you feel like needs to be included in your pre-registration that you're not really sure if it fits in any of the other spots you can always like upload a description or type a description in of that information in that box too. That way it is there it's described but it's not you know it's just it's in there. Great here's a question if you have a pre-registration would it also be encouraged to publish a separate pre-print? Sure yeah I think this question actually links to another one that was asked elsewhere. So just want to a disambiguation of terms here since what we're actually talking about. So the the pre-registration that we're talking about here today is the plan for what you're going to do in a study. That is more or less all we are talking about. What you actually do with that might be different things you might when you actually do that plan or some variation of that plan you might make a publication out of it. Now you can link those things together in OSF and you would be encouraged to do so but the pre-print is not inherently part of the of a pre-registration. There's also another thing there's another question in the chat related to I think David mentioned earlier called registered reports and just to re-emphasize what that is a register I know these things sound it's all registered this and they're all ours it's terrible but a registered report is effectively a plan for what you're going to do that has an agreement with a journal based on peer review that says that I'm going to publish these these results when this is done. Effectively it is a you take a pre-registration you do peer a journal will do peer review with that pre-registration and make a different agreement on top of that. So there's a lot of things going on here but what we're talking about today is strictly the pre-registration itself that data analysis that plan for what you're going to do everything else kind of goes on on top of that. Yeah so definitely the relationship between those two is like in your pre-print in your write-up of your analysis you would reference that pre-registration right and you would outline this is what I said I was going to do this is what I actually did if I deviated from it here's why I deviated from it I added additional analyses or whatever you would you would kind of delineate all of that so and we definitely support pre-prints here so so please please please give us your pre-prints. All right let's see what other questions are there. Actually this is a kind of a tech question but an interesting one I'm a co-author of the COSRP CB study 20 and a big fan of pre-registration on the road a lot have limited internet access any plans of supporting a client offline app that would sync what's internet connection is available. So I guess that's this to me right? I think so. We have been asked about that a couple of times right now I can say that it's unfortunately not part of our roadmap for this year but it is something that we are trying to talk about internally and we're also building out several different ways that we can start chatting with other developers and other members of the community to help us streamline some of these processes and really address your needs. So stay tuned. Thanks Mark. Another question came in thinking about interesting or unexpected or unplanned results and they're often seen as exploratory findings so they're to be confirmed later in confirmatory studies even if it's a confirmatory pre-registered studies are people forbidden to report those findings and can authors update the pre-registration later on in the analysis stage? Absolutely not. You are not forbidden. This is not a prison. So you might have very good reason that you find something serendipitously and that is a very interesting and useful result right? Nothing about a pre-registration prevents you from doing that. It's a it just changes the default for what you're going to it just sets the default for what you're going to report. You can make different choices along the way depending on what you find that's an important function of science. What a pre-registration the pre-registration part does is lets you declare what parts of that were actually planned and what parts of that were unplanned or more exploratory. So no, there's nothing no there's nobody out there that's going to say oh no this is impossible to print now that you've pre-registered it. Some people might actually might complain you know might say oh you shouldn't because you did not pre-register this result but that's you know that'll sort of depend on where you're publishing and with whom. But yeah no this is not a prison that's awesome. Another question about licenses the cc license that are available on the osf are public domain cc buy and so they have not seen stricter options and csa and d etc. Are those options how can you pick a license that's more strict? This might be a question for Mark. So we usually have a process when it comes to licenses that's saying that that list is we're always evaluating and always reviewing so if there is one that is needed by your community and one you like to propose send me an email and I will send it over to our product team for further discussion and I know I've been saying a couple of things about just reaching out to me directly and go ahead and put into the chat my email look for you to reach out go ahead David I also add to that you can always upload any license you want if you just sort of copy the text of the creative commons license to a text file and add that into your osf project that'll take that license and those you know defaults meet me try to strongly support the most open licenses and for those or why those are the two that are readily available but osf does allow any license that you would like to add on as a free text response all right great question about what is the best way to know and explain to people what's important to pre-register and what's less important to pre-register might be good that's a thing couple of maybe thoughts on that there's I mean there's so many ways to interpret this I mean part of it is experience right so one of the knowing what's important and what isn't important will be field dependent right what is important for you to explain what methods and so on is going to really depend on what the question is and how that all works what I would typically say is start from pre-register even if you think like the the minimum thought that it might be useful to pre-register start with the pre-registration and then you're going to start developing good ideas for how much detail you want to get into what this does and what this doesn't do there's no real universal answer to what is important and what is not important ultimately I would say if it is important for the use of your conclusions to have something to have that trace back to the methods right how people actually interpret that then that becomes very important maybe to pre-register or more important to pre-register but it's really kind of up to to the specific questions in the fields there's no I don't think that there is a a broad grand advice that can be made here maybe I'm wrong what do you guys think I got distracted by looking into the license so Krista what oh no yeah I think yeah I think it's I try to think of it in terms of anything that I think could interpret or could affect how I collect my data how I analyze my data how I interpret my data how I explain my data so I try as I'm going through both the form and then also sometimes I have additional questions that I'm working through with my teams just kind of thinking through like what are the things that I'm the reason for pre-registration so you have a transparent document that that outlines what you were thinking at the beginning of the study it outlines the analysis you were planning and all that and the methods by which you were going to obtain and analyze your data so for me it's about like thinking about what is it that I'm I'm really going to want to remember like six months from now or eight months from now however long it takes to to have the data and what are some things that I think could potentially influence the way that I that I'm going to look at this data that I'm going to interpret this data and so those are the those are the aspects and I think that that varies from study to study and so it's a little hard to give like a hard and fast like these three things but that's how I tend to think about it like if you think about it from the from the base of like what is the purpose of pre-registration why do we do it I think it becomes a little bit easier to figure out like okay then it's probably really essential that I include like how I'm going to determine what outliers like how I'm going to determine like how I exclude data or include data or you know so what's getting in the data set and how am I going to figure out what what are my main analyses or how how those link back to my hypotheses like really thinking through that I think can be really helpful I don't David if no yeah that's the good all right um we've got um couple of different questions anybody see a question that they're really loving that they'd like to answer first one I can easily answer so microscopy data is highly dependent on visualization like in the open source software or mirrors when a great job of capturing all there is any future plans for integrating with them we are actively expanding our add-ons we're working on that now and what that looks like so it might look like that in the future I don't know within the next six months but I will be happy to connect with them in the future and see what else we can provide for you guys in the community any others standing out to folks we've got a question here about pre-registering a conceptual replication study which is a mix of confirmatory and exploratory any tips for filling a pre-registration out for them a hundred percent yeah so I'll hold off on the replication conceptual replication side we can enter that one separately that's a whole other kind of worms but for mix of confirmatory and exploratory research that I would say is most research has some mix of both of these things so the thing that I would I would I would encourage people to do is really is declare what parts of it are confirmatory and exploratory and you can if you're looking at hypotheses you can list all of your hypotheses and say this hypothesis is our main confirmatory test and these ones are sort of more exploratory as as you go down and then question whether the degree to which things are really exploratory and confirmatory take some time and think about them because a lot of times you'll actually try you're going to be thinking in one mode more than you think you might be right so a lot of times where we sort of want to say we're exploring but we're really sort of trying to do a confirmation and seeing that in person is really useful but yeah I think there I mean there are so many examples out there on on OSF for delineating you know what is more or less confirmatory and more or less exploratory you may want to think about often the language is sort of primary and secondary and tertiary research often falls under confirmatory and it's more exploratory sort of dynamics so that could be a good way to to think and talk about that the other thing is explore what your results might be in every possibility and then talk about how think about how you would actually interpret them and that will actually help you determine whether or not this is more exploratory or or more confirmatory so play it backwards a little bit and kind of listen to yourself and does that sound like a more confirmatory interpretation or does it sound like more exploratory that's that could be kind of helpful I think a good question for David would be do you see journals and academia taking more steps to incentivize pre-registration and registered reports we do so it's coming up more and more in publisher and journal conversations a lot of funders recommend it or encourage its inclusion in preliminary program or status reports we take a look at a lot of journal policies and the project known as top factor where we evaluate journal policies on their adherence to open science guidelines and there are several standards on pre-registration where journals require authors to state whether or not the work was pre-registered or if it was pre-registered they provide guidance for how to compare what was pre-registered to what was reported there are a small number of journals that require it but the most part most do not and there are over 300 journals that will take a look at the proposed research plan as part of that two-stage peer review process known as register reports and that's a direct incentive of course to to pre-register because that gives a real strong protection for the author to conduct the work as pre-specified if the editors or reviewers agree that the research question is important and the proposed methods are the appropriate way to go about it there comes that promise to publish the final results regardless of of what the final interpretation is as long as the authors and the research team follow those pre-specified and pre-agreed upon plans so those are all strong incentives to encourage steps to pre-register and we see a steady uptake of that over time and we expect that to that trend to generally continue so I actually have a question for everyone and just for sick of time my question is we have several other Q's and A's and I'm sure others will have questions as well where's the best place for them to engage with us at? Question maybe this could be an I used to be on Twitter but I'm sure I'm giving up X at the time I'm on LinkedIn now more these days the blue sky every now and then I'll put my email address in here yep and I think Amanda I think she was about to go off mute but she might also have an alternative where they can gauge with us too yeah we'll follow up from this email with recording and any type of materials and we'll provide an opportunity there but I think if you do have questions you can go to COS and go to contact and there'll be a form there where you can submit questions and we'll make sure those get triage to the appropriate people to answer those but we'll follow up to and you'll have a line to communicate with us awesome well we are right at I was going to say two o'clock it's two o'clock my time it may not be two o'clock where you are at we're right at our hour so I just wanted to take this time to thank you all for coming if there are any final thoughts from anybody I'll let them be but just thank you everyone for coming and if you have any questions definitely reach out