 All right. Well, I think we'll go ahead and get started today. Thanks again for joining us for this webinar. My name is Alex DeHaven. I'm a project manager here at the Center for Open Science, focusing a lot on policy initiatives and specifically for registration. I'm joined by Sarah Bowman. Sarah, I just want to say, here you are. Yeah, I'm Sarah Bowman. I'm a product manager here at COS and focusing on building tools to facilitate open science practices and OSF registries is one of my core products that I work on. Yeah. And so we wanted to get together to do a brief presentation with a lot of Q&A around pre-registration. So if you have any questions that you would like answered, we're gonna have plenty of time for that. So please come with your questions ready and make sure you add them to the Q&A function. If you see a question in the Q&A function that you wanted answered, but it's already written down, you can vote for that. And we will try to get to the most voted on questions first at the end of the presentation. And now go ahead and get started. So what is pre-registration? Pre-registration is the practice of distinguishing confirmatory exploratory modes of research. Now, both of these are very useful and they are integral. They play off one another for the process of science, but conflating them to get, conflating the two can get you in trouble when it comes to interpreting the results of these types of research. So a little more on confirmatory versus exploratory research. In the context of confirmation, we're talking about the traditional hypothesis testing where the results are held to the highest standards of rigor, and the goal of this mode is to minimize the false positives. And in this mode, the p-values are actually interpretable. On the other side, we have the context of exploration. And in this context, you are pushing your knowledge into new areas, generating these testable hypotheses, models or theories. And the goal here is to minimize false negatives. In this context, the p-values are meaningless. And pre-registration is simply the specification of what you are setting out to confirm versus what you are finding along the way or you uncover incidentally. And presenting exploratory results as confirmatory will increase the publishability of the work at the expense of its credibility. So it's important to delineate between the two. So what is a pre-registration? A pre-registration is a research plan that is time stamped, immutable or read-only, created before the study is done, and submitted to a public registry. Now, some pre-registrations may use existing data if precautions are taken to mitigate and mitigate any prior knowledge or biases the user may have before the study is created. So we want to make sure that we pre-specify or at least acknowledge any prior knowledge we have about that data so we can show if it did or did not influence the resulting statistical analyses and analytical decisions. And submitting something to a public registry does not necessarily mean that it has to be public immediate. You can embargo your pre-registrations on certain platforms. I know for OSF you can embargo for up to four years. And what we mean by that is you can time stamp it at that specific date but leave it private for only you and your fellow authors to see until you're ready to make it public once the work is done. That way you don't have to be open immediately in case you're worried about sharing your plans prematurely or worried about scooping or anything like that. And a pre-registration, in our view, has the following components. It has in the study plan, it has clear testable hypotheses, it has clear data collection procedures. So where you're going to get the data, what population you're going to be drawing from, any stopping rules that you may apply to your work. And then clearly defined manipulated and measured variables. Now in observational studies you wouldn't have a manipulated variable, but if you were manipulating something you want to be very clear on how you were doing that as well as very clear on how you are measuring certain phenomena. That way it's more interpretable to your readers. And then we also recommend having an analysis plan attached to your pre-registration that dives into your statistical models. We recommend one statistical analysis per hypothesis proposed. That way it's very clear to show, you know, here's my hypothesis and this is how I plan to address and analyze that hypothesis. As well as we recommend including inference criteria, so how you plan to approach the results and decide something is, you know, significant or not. So some of the benefits of pre-registration are it helps protect against natural biases and selective reporting in your work. It's a great tool for communicating with others, so it's really useful for showing your advisors or your colleagues or as an interim report to show what you're working on and how you plan to do the work you've been tasked to do. It's also a very useful exercise to generate more robust planning for your research. It does require a little more effort at the outset. You can't just dive straight in, but we see that as a good thing. It allows you to better test and check your assumptions around the study and better prepare for your study so it goes more smoothly and you can, you know, it just makes it easier at the end with a little more planning at the beginning. And then also it's a very helpful reminder to your future self. Research can go and evolve in many different ways and it's really helpful to have that documentation from where you've started to sort of bring you back to what you would plan to do, what you would set out to do when this whole thing began. So when talking about pre-registration, we are also often asked about register reports. Some say, you know, what's the difference? I'll do the same thing. Are they different? And we just get a lot of questions about them. So I wanted to briefly mention that here. A register report is a publication workflow that is centered around pre-registration. So it uses pre-registration, but it goes a little further. So here we have the traditional workflow in a study. You develop your idea, then you design a study to test that idea or hypothesis. You collect and analyze your data, write the report and publish that report. Now register reports differs in a few different ways. The first being how it is peer reviewed. It is the peer review process is split in two stages. One occurs before you've collected and analyzed your data. And the other occurs at the traditional stage after you've written your report. So how it works is authors will submit their stage one article to a journal. And that includes the introduction, the proposed methods and analyses and any pilot data if you have it. That is sent to the journal and that is passed out for peer review. And you can now get feedback on your proposed study from experts, which is very useful because this is also before you've invested in the work itself. And then once you get approved, once you pass that stage of peer review, you're granted what's known as an in principle acceptance at that journal, which means regardless of the outcomes of the study, you will get a publication in that journal as long as you adhere to the plan that has been peer reviewed and all agreed upon. And so you move on, you collect and analyze your data, you write up your report, and that gets you to the stage two article. And that seems probably a little more familiar. That'll have the introduction and methods from the first stage, but will also include those new results. So the registered confirmatory findings from your pre registered analyses, as well as any unregistered exploratory findings that you found along the way that are interesting and can help drive new discovery. And then finally, also including your discussion of your results. So pre registration register reports just a little quick reminder of what they both do what what similarities and differences. They both address unreported flexibility in conducting statistical analysis. They also make clear distinction between plan confirmatory research and any unplanned discovery research. However, register reports takes this one step further and addresses publication bias against non results by shifting the focus of peer review from how publishable are the results to how rigorous and dependable are the methods proposed. And this also has that two stage peer review, where you can actually get feedback before you've invested your resources in time. So you can make sure it's the best methods and the best path forward for addressing the hypothesis of interest. We have a variety of templates on the open science framework which Sarah may get into shortly, but I just wanted to highlight them quickly. So we have seven different templates to choose from and those vary in level of detail and rigor to cater to a range of experience or needs of our users. Additionally, we have community made templates that fill the take pre registration and fill in a specific niche or research area in a way that we could not do it from a general point of view. And for example, we have some for fMRI studies we have some qualitative studies one for secondary data analysis as well as one for model application. And we are currently. Oh, there we go. We are currently working to improve the pre registration process, helping guide users through choosing which template suits their needs best, as well as making the resources we have more easy to discover so it's you can have the support you need to make the best pre registration you can. And I just wanted to quickly mention some pre registration resources will be sending out the link to these these slides at the end so you'll be able to click on all these and see them by briefly the pre registration Revolution is an article that we published at in 2018 that discusses registration and some pragmatic, you know, common common concerns and solutions. We have a blog post by seven self called seven selfish reasons for registration that really highlights the individuals benefit from for privilege from registration, we have a variety of blogs on our blog. That highlight a variety of elements around pre registration I mentioned this one here. This one gets into this is registration a plan about a prison. This one gets into a lot of the common concerns of well what if I make a mistake in my pre registration what can I do. And that is really spelled out there. We also have a lot of good examples of thorough pre registrations from the pre registration challenge, which was an initiative, a competition that we held from 2016 to 2018 to encourage pre registration. So follow us at OSF pre reg on Twitter for the latest registration news and examples. And finally, you can also just check out the COS.io slash pre reg site for helpful FAQs and a variety of other resources. And with that, I'll turn it over to Sarah, and she will take you through a little demo of how to pre register on OSF. So I'm going to show some of the practical sides of how you do this on OSF. And then after that, we can get into some of your questions. Coming in fast and furious, which is great. I appreciate all see my screen now. So what we're the first thing we're going to do is log into OSF and I'm actually going to do this on our, our test server, because I don't want to create a real registration right now. And click sign in and then you can either log into your account with your email address and your password or sign in through your orchid credentials or through institutional credentials if you are at an institution that has partnered with with OSF. So I'm going to go ahead and create a new dashboard. The first thing you'll need is a project. Every registration is backed by a project on OSF. You might already have one created or you might need to create a new one. So I'm going to go ahead quickly create a new project. Really the only thing you need is the title. You can also, I know we've got several participants in the webinar from different locations around the world. You can choose your storage location and where your data will be stored. So if you happen to be in a country are working on some research where you need to store your data within your country. You can do that Canada, Germany, or Australia actually is another site that's available on our production server. So you go ahead and create your project. And you'll, the first thing you'll notice is up here on this right hand corner is that your project is private everything you make on the OSF is by default private when you make it and then later you can choose to make it public. It's also a good point to call out the difference between a project and a registrations public and private settings they have different settings so if you want to keep your project private make your registration public you could do that. Or vice versa. So I want to provide just a little bit more information to folks for when I make this project public. I want to put in a description so that people will understand a little bit more about what they're reading. You can also add a license which will indicate how your work can be used by others so I'm just going to take those two really simple steps on my project. And then you can also manage other authors on your project so Alex and I are working on this project together. So I want to go to the contributors page and add Alex to my project. So that he can also access the registration that we're going to work on. And this is when you can choose your permissions. Read read write or administrator and basically an administrator is the person that can control if your project is public or private, what the other authors on your project are and who can create that registration. So if you want your collaborators to be able to initiate a registration or complete a registration you'll need to give them administrative access. So we've got our basic project structure set up. Let's dive into creating a registration. You'll click on the registrations tab and start a new registration. Alex mentioned that we have several templates built into OSF to facilitate creating registration and I'm going to walk through two of them. But you should know that there are five others. We're going to first walk through the OSF pre registration. And there are some tool tips after each registration template title. There's a tool tip to give you a little bit of a blurb about what these what each of these registration forms kind of we're trying to get at what the goal is. And we have we'll send a link around places where you can look at each of the contents of each of these forms to help you decide what you what you want to use. So we're going to start with OSF pre registration, which is kind of the most comprehensive registration form. We have, you know, create a draft. Now you'll notice immediately the UI changes. Alex mentioned we're working on some improvements to the OSF as a whole but specifically starting with registries. So you'll start to see some new user interface new user experience elements as you go through some of these pages while we're working ahead on these. So the first thing that you'll need to do is provide some metadata about the registration. So you can give it a title that could be different from your project title. So maybe your product your registration title is a question does political ideology change with age. So you'll have a title, you can give a brief description. And this is where you lay out your policies. All the required elements for the registration are denoted with a red asterisk so you'll know what you need to complete for you're allowed to create your registration and what things are optional. So, as you're filling this out if you get to something where you say I'm not quite sure what an answer to that question might look like. There are some example texts. Click show example, and it will drop down and show you so all of the examples follow along with the effect of sugar and brownie tastiness. Thanks to David and Alex together. So you'll put that information in, and then you can go on to the next page, you'll notice on the right hand side it's telling us that it's your contents of your form have been auto saved. So as you're going along every few seconds when you stop typing, your draft is auto saved. So if you come to a question that you're just not quite sure you need to think about maybe you need to go and talk with your collaborators about. You can exit this workflow at any time, and then re enter it later, and your draft has been saved. So you go back into your registrations tab and pop into draft registrations, and there's your draft that you were working on. So you can pick that right back up where you left off. Let's choose some information about blinding. Now you get down to study design where you need to describe your study design. So maybe you've got some words to describe your study design or maybe you've got a file. And there are a number of different places on the swarm where you can upload a file that represents some aspect of your study. So you can upload that file right here to the form, it gets attached to the question that you've uploaded it to. If you're already working from an OSF project and you've already got files in it, those will appear here and you can just select them and they will be attached to that specific question. So this is a question about existing data. Are you registering a project that maybe the data already exists? So for this particular example, maybe we're using survey results that already exist. We haven't looked at them. We haven't collected that data ourselves. But we're going to get access to that data. So you might select that option. So how did you collect your data? This is coming from a survey that was done, sample size, a reason for your sample size, your stopping rule. So you pop through all of these questions. And then when you get to the end, you can review all of your responses and you'll see on the left hand side it's going to alert you if there's something that was required, but you didn't respond to it. So in this case, it's measured variables. I haven't provided a response. And then the next step is to create your registration, which is going to make this a frozen timestamped copy of your registration. And you have two options. You can either make this registration public immediately or you can embargo the registration for up to four years. And it's something that maybe you want to keep private for a little while until you're ready to publish your paper and then make it public. You can go ahead and do that. It's a simple date picker that pops up. If you embargo it for two years, and you really finish the work in 18 months, you can go ahead and choose to make that. And that embargo early, that's always an option. In this case, I'm going to make my registration public immediately. And it's giving me the option to create a DOI, which is a digital object identifier. And what this does is goes ahead and creates a persistent identifier and registers it with data site, which is a DOI registration agency, and we'll pass along some metadata about your registration to data site. And it helps pass along, make your make your work a little more discoverable. And again, creates a persistent identifier. So I'll say that, yes, I would like to do that. And now I'll submit. You're going to get this page that lets you know that your registration is percribing. So it's creating as another copy of your project for the registration. So any files that you uploaded as part of the form, along with any files that were already in your project are going to be copied over to a brand new registration. This should be pretty quick because we only had one or two files in there. So this is my registration overview page now. It's letting me know that this is pending registration. I've created the registration, but I haven't approved it yet. And this is where the authors on your registration become the permissions level become even a little more important. So any administrator on the registration needs to approve it. They'll get a link in their email to approve or reject it. And then once all of the administrators have approved it, or 48 hours has passed, the registration becomes permanent and public. So that's an important, an important thing to think about while you are choosing your permission. So I'm going to pop into my email and get my link in my registration. So now I've got a permanent public registration called political ideology and age, and you can see the contents of that form that I filled out right here in the middle of the page. And here's study design where I attached my file with my study design. So you can click on that and view the file. I have up here in the bar with the URL. A five character ID that's identifying my, my registration so it's FBA as nine is my registration identifier. It is linking back to my live project which has its own identifier. It's linked to my live project where I can keep working on my active research and the, the contents of this form in the middle are frozen and I can't edit them, but the metadata along the side is editable. For example, if you were to do this work and then publish it in a journal and you wanted to link this registration to the published work, you could come down here and edit your metadata and put in the DOI for your journal article. That metadata also gets sent off to data site where if you have a DOI for your registration. So it's help helping connect various pieces of the research lifecycle. You can also add subjects and this will help improve discoverability of your work. So this is about political ideology. So I'm going to put this in political science. That helps people who might want to look for other work in specific subject areas. And then you can also add tags, which will help discoverability. So if I am using the survey data, I might want to point that out. So if someone's searching for those keywords, they'll find my registration. So that is your registration of the OSF pre-reg form. And I am going to go ahead and show you one other form on OSF that is much less detailed, a little more lightweight. It's called the open-ended registration and it is one simple question, one simple test field. So I go back to my registrations tab on my project. I can see my completed registrations, and anyone who is a member of the public that might be looking at your public project will see your public registrations listed here. So if you have created multiple registrations of the same project, they'll be able to see those different registrations and when they were registered trying to kind of track the evolution of your work. So I'm going to go to the open-ended registration, which has one simple question, which is a narrative summary of why you're making this registration. And there's lots of different reasons how you might use this form. You might be creating an update to your existing registration and maybe we created that OSF pre-reg registration and we realized there was an error in it or we've made some type of change to the plan for whatever reason. You might want to create a new registration and put in some information about that here. Or maybe you are using a different type of registration form than is not yet on the OSF, something more specific to your discipline. Then you can attach that as a file. So I'm going to do that. And then I'm going to, that'll attach right here and I can say registration details are included in the file that I've attached. That is the really quick registration form. We've got all the registration forms from the OSF. Alex will point these out in his slides, I'm sure. All of those, all of our registration forms on the OSF as a Word doc or as Google doc or in various other formats that you might want to download those and work on those offline. And then you can either come back and put them in the form or you can upload the doc. So that are, those are two types of registration forms on OSF. And then the last thing that I kind of want to just show you, I'm going to pop over to our production server and show you the search and discovery interface. So it's osf.io slash registries. And it's pulling in not only registrations that are on the OSF but also registrations that come from clinicaltrials.gov, or from research registry so it's aggregating search results. And you can filter by provider. If you are looking at a registration that's on a service that is not, it's not on the OSF it's on, for example, clinicaltrials.gov. Then it's actually going to take you out to that external service to show you the results. If you are searching on within OSF registries, then you can filter by the form type that the users have filled out. So if you are particularly interested in replications that might have completed the replication registration form, you can filter on those things. I showed you, Alex alluded to this and I showed some pieces of this where we're redesigning this workflow. So I wanted to tell you about some things that are coming. One is filtering by subject area. So I showed you how to add that to your registration, but it is not yet included on the discovery interface. We need to get a critical mass of folks to complete that information before we can surface that. So that is one thing that's coming. Alex mentioned a workflow to help you choose what type of form is right for you. And so that is something that is in the works. We're also improving the workflow to allow you to create a registration from scratch so that you wouldn't need to have a project first take off that first step. And that is something that will be completed in Q1 or Q2 of this year. I feel like there are lots of other exciting things on the way. Lots of improved UI, so you probably saw on some of these things when you're popping back between the project and the registration. Not all of that UI has been updated yet, but it will be. So those are exciting things to look forward to. And I see we have lots and lots of questions. Let's get to it. Yeah, I will start with the top one. The question is, can I use the same wording in a pre-reg and then later the later paper or do I have to be careful because of self plagiarism? I've actually never thought of it that way. We encouraged people to use their registration verbatim. But if that does cause concern for self plagiarism, then we've also seen people use the pre-registration information. The information doesn't change. Just the tense. It goes from future because it's a plan to past. You will do this versus you have done that. We encourage you to use similar language because it draws a parallel and keeps it very clear and concise. What you plan to do is actually what you ended up doing and it makes it easier on you when you go to write the whole thing up because you've already written your methods. You've already written your analysis plan. You just need to get it over into the final document. Okay. The second question is, is it okay to change authors adding slash deleting after having pre-registered? Can that be seen as a disadvantage for publication when the authors don't match anywhere? Oftentimes the authors on a pre-registration and the final publication do not match because a lot of the published research out there right now that includes pre-registration includes pre-registered work and also unregistered work, which is totally fine. You just need to make sure that it's clearly demarked what was pre-registered and what is not. We encourage you to leave authors on the pre-registration if they are contributing to that pre-registered work because they deserve that credit. But if additional researchers are involved in the project and they come in after the pre-registration or add an additional feature or facet to that work, that's okay. It's okay for them not to match up as long as it's clear who did what and proper attribution is given. Moving right along, can you talk more about pre-registration of studies using existing data? What precautions need to be taken? What assurances need to be made in the pre-registration? So when using existing data, it becomes a little bit of a slippery slope. If the data previously exists, then the biases you may have from that data are very minimal if not non-existent. With existing data, it's impossible for the reader to know how much you had known prior to creating the pre-registration. If you know the data very intimately and you very much understand how the data are going to lay and what the results are going to be, then the pre-registration is very diminished in its power. But if you can't avoid it, some people are studying things from the past that cannot be recreated and we also don't want to duplicate data collection efforts if there's good data out there. Simply pre-registering and specifying what you know about that data helps your reader better assess what you knew before you went into that project. And that's the best you can do in situations like that is just disclose. That way it's transparent. It's transparent what you knew prior to creating that pre-registration. And then it's up to the reader in the community to assess how much if any bias may have crept in, which is just a useful thing. Okay, next question. It's a long one. Okay, so I recently reviewed a paper and pointed out the discrepancies between the pre-registered analysis plan and the actual analysis reported in the manuscript. Okay, did not mention or justify the departures. Then someone mentioned on Twitter that there can be innocuous discrepancies due to information reduction on OSF. How should a reviewer judge such discrepancies then? I thought the most important point of pre-registration is to differentiate analyses according to the pre-reg and those that differ from it. So I agree. It is strange that the discrepancies were not listed. Now this could be just a learning curve where people, you know, they're trying pre-registration and they have failed to do that thing. And I'm happy to hear that you pointed out those discrepancies. And it's strange to me, it's surprising that they were not addressed. I'm not sure what is entirely meant by the information reduction on OSF. If you want to follow up with me offline to discuss this in greater detail, I'm going to try to answer it in general. But I think you did the right thing by acknowledging the discrepancies pointing those out because it's okay to have those discrepancies. Things happen, changes are just part of life. But if we don't disclose those changes or at least acknowledge the change, the change happened and why it happened, then it really does undermine the power and the validity of the pre-registration. So, yeah, I think that was good practice. I think as a reviewer, you should judge it as you did. And I'm a little confused. I'm a little confused about what is meant by the information reduction on OSF. So if you want to follow up after this, that'd be very helpful. We have another one. Should I pre-register an exploratory study? Right. When there's no hypothesis, how can you? So if it's so exploratory that you don't have a clear hypothesis, you can pre-register if you'd like. You can pre-register things like a decision tree to show how you plan to explore. But that is helpful but not necessarily required. When we talk about pre-registration, we reserve it more for that confirmation element trying to confirm any interesting hypothesis that you may have. You can, yeah, you can pre-specify if you have any prior information you think would be very valuable to your future readers and reviewers for your exploratory study. You can include that in a pre-registration just to show, you know, this was my thought at this point in time. I don't think it is necessarily required. It's really up to you. Okay. So should I submit a link to my pre-registration project when sending my paper to a journal for publication? What about, and the anonymity during the review process and there's a follow-up with the co-editor of a journal. They include a place to provide information but require that the pre-registration be masked when we do not share pre-registration links, the author is identified. I think that's probably good practice to submit that information when you are submitting your journal publication. And we, the OSF offers some solutions for how to manage anonymity if you are having a blinded review. As long as you don't, I think this is the most important part is that you don't include your identifying information in any of the files that you upload or any of your responses to questions. But as long as your only reference to your name is in that list of authors and contributors, then you can create an anonymous view only link that will strip your identifying information out. And then anyone that's looking at it is just looking at the contents of your registrations. Yeah, and I want to sort of reiterate there, if you have any identifying information in a file, that can't be changed once the registration has been finalized because now it is a timestamp and mutable page. So oftentimes we'll get it, it's just a simple mistake, it's an accident, you accidentally have your name and the title of the file, and now you want to submit it to anonymous peer review. Unfortunately, we can't change it after that fact so make sure if you know you're going to submit it to blinded peer review that the files you upload and the questions you answer do not have overly identifying information for you. That way you can keep that blind, that blindness. Okay, another question we have how detailed should the methods be in a pre-read should a person be able to replicate using only the pre-read. In an ideal world, yes, a person should be able to replicate your results using only the pre-registration. That's sort of what we encourage you to strive for providing enough detail that someone can an interested reader can take what you did and extend upon it or at least replicate it. Now, there's always the question of should I provide more or less oftentimes if you like, you know, oversaturate with information that's not necessarily relevant to the research you can sort of muddy the waters and sort of confuse and add to the link which reduces the accessibility. So you want to find that sweet spot of clarity where you can very clearly specify what's being done as a sort of a recipe that way your readers can better use that work because, you know, the pre-registration can be thought of as a tool. It's a tool that others can use to extend the research you did or at least try to replicate the research you did. And you want to make sure that tool is as useful as possible to the reader, so I would try to think about that when you go into the method. So, you know, one example I'm just off the top of my head is say you're pre-registrating your analysis plan and all you say is you're going to do an ANOVA. Well, you could better specify that by using the, you know, stating the hypothesis that you're going to be testing using ANOVA and also the variables that you'll be using in the statistical test. That way it's very clear that this is the ANOVA. These are the variables that I'll be using and this is the hypothesis that it maps back to. That way there's just a little more clarity and continuity between your hypothesis and your analysis. Can we have another question? What is a good length? Would you say that a lot of length and detail is important for a good registration plan accounting for all the ifs and possible sub-research questions? Or short plans only listing the most important research questions hypothesis method? I usually use the as-predictive template, which is about one page. So the length kind of gets to the question about, you know, what's the good, you know, how much detail is enough detail. I would make sure it's not the tone because then it really does reduce that accessibility. I'm trying to focus on one or a few key hypotheses of interest and focus on including the variables related to those specific hypotheses and the analyses related to those. You don't need to pre-register everything under the sun in a specific discipline because that or a specific research area because that's just too much. But, you know, at the end of the day, you'll be doing a specific research project related to specific questions and generating specific data. I would try to include all of that. And if you've got related studies, I would make that known in subsequent registrations for subsequent studies. I hope that makes sense. When submitting a pre-reg study for publication, can you add a link to the manuscript? Oh, I think we entered this one. Oh wait, can you add it without showing author's names? Yeah, Sarah, you just answered that. Yeah, so if you, and we'll send more, you know, some more things to send out afterwards. But if you go, you'll want to look at the anonymous view only link is how you would accomplish that. We can send some more links out to help guys about specific steps to accomplish that. Another question. For exploratory and confirmatory studies, is there any difference between the two and pre-registration process? Yeah, I would say there is in that in an exploratory study, we really mean, you know, you're just getting out there into the unknown trying to uncover any interesting phenomena that could then later be confirmed through follow-up studies. So in an exploratory study, we mean one that doesn't really have a testable hypothesis or direction. It's just a high level research question that you want to just get a general understanding of that you can then follow up on in a more rigorous way and confirm it and create a test that will directly assess the question of interest. Right. So I list all their measures, measured variables from the study also in the pre-reg even if they're not part of the pre-registered hypothesis analysis. That is, I'd say, solidly optional. I would recommend you stick to if the variables are being used in a pre-registered hypothesis or analysis to include them. But if you think they're very useful to include just because you're going to be collecting them anyway and they're going to be used in a, you know, sort of a follow-up for exploratory manner. You can include it if you want. More detail is often useful. But if you worry that it's going to blow up the pre-registration, I would at a bare minimum keep all the measured variables, manipulative variables that are directly related to the pre-registered hypotheses and analyses in the pre-registration and then maybe save the other variables like the total list for the project or for subsequent registrations that actually directly use those measurements. Okay. So what are recommended practices for reporting pre-registration processes in completed studies that are being written up for publication, e.g. referencing the pre-registration method, connecting the pre-registration. Okay, so I definitely think you want to reference your pre-registration in the method as well as maybe in the results section. So in the method, you can say, you know, this work was pre-registered. Here's the link to the pre-registration. It outlines how, you know, my hypotheses that outlines my study design and my data collection. It maps out my variables. And I'm going to go into this a little more detail here in the method section. I would also mention it in your analyses because if you have any exploratory results that you want to make mention of, you want to make sure you clearly define what you set out to do, what's included in the pre-registration and the pre-registered results of any pre-registered work and separate that from any exploratory work. That you've done and uncovered along the way. Okay, next question I've got, is there any evidence that pre-registration improves study quality? So there is some beginning evidence. It's still a pretty early process and it's, we're still generating enough of a sample to then test it against the control that isn't pre-registered. I will say that pre-registration so far has improved the publication. It makes more of a space for publishing null or negative results. So that's good. It helps set the publication record straight or straighter by allowing you to publish work that is negative or null. That way, that's very useful. Also, we've gotten some anecdotal evidence from participants in the registration process, the register report process, who say that they really enjoy getting that feedback early. Remember, register reports, you pre-register what you're going to do and then you send out for peer review and that definitely has helped. And also the reviewers of register reports have really enjoyed helping make the study better when you can actually do that instead of after the study's been run. So it improves the quality that way, but there is ongoing research and efforts to critically evaluate. Does pre-registration actually improve study quality? What does that mean? How do we define that? But so far, the clear evidence is it does definitely make space for negative and null results in the publication record, which is a good thing. Okay, so now we've got a long, oh, this might have been answered by a fellow participant. It's great to see. What is the incentive for journals to agree to register reports? They may be agreeing to publish null slash negative results. And so we kind of already got to that. Jill Adelson mentioned, she says that if it's worthwhile question and solid methodology, then it's important to publish negative null results. I agree with that 100%. That is one of the benefits of register reports. As a journal editor, I appreciate that my reviewers can focus on theory ideas and methods and not be biased against negative and null results. This helps combat publication bias. Yeah, that gets into the question previous. And Jesse, the question asker followed up. I agree. I don't read many articles published with negative and null results. So I'm curious if journals and editors value theory ideas and methods in the same way. And so, yeah, this kind of gets to the reason, the motivation for register reports and for making it more okay to publish negative and null results. As you don't see them often, there is a positive publication bias and we think that register reports is a very useful publication method that will help mitigate that and make space for negative and null results because it focuses on theory and the how the science was done instead of what happened to be uncovered, which is ultimately not up to you. You can improve the quality of your study. You can improve the design, but you can't guarantee that it's positive and novel. So it's focusing on things that you, the researcher can control instead of focusing on things that you can't control, which is the outcome. Okay, no question. Does registration apply to fields like computer science where papers often present new tools and techniques rather than test hypotheses. So, if you're not testing a hypothesis, if it's not hypothesis testing work and it's more just characterization and description of a new cool tool. I think pre-registration has less utility in that sort of preservation of statistical validity, but it can still be useful if you want to pre-specify, you know, how you were going about creating this. If there's, if that record will help improve users' use of the new tool or technique, then pre-registration could be useful that way. But it is probably, at the minute, it's probably less useful than I'd say a traditional hypothesis testing framework. Okay. When my data, e.g. surveys and interviews are collected in a language other than English, should I translate to English before deposition? I think this is ultimately up to you, your resources and your target audience. If you want to leave it in the native language, you feel free to do so. Now, that may cause a barrier when it comes to the review of those data. If you want to provide, we often recommend providing both if you have the resources and ability to provide a translated in law. That way, native speakers can engage with it and also non-native speakers can see what the translation would be. So, really, that's up to you, your abilities and your bandwidth. Okay, another question. Why should I register my study at OSF instead of clinicaltrials.gov? So, clinicaltrials.gov is a space explicitly for clinical trials. OSF, you can register any study that is not a clinical trial. That way, you can add the benefits of pre-registration and the rigor and transparency without it having to be a clinical trial. So, that's why I would do that at OSF instead of clinical trials because clinical trials is a very specific selective repository for a clinical trial. And for non-clinical work, you can definitely do that at OSF. I also personally think that the registration schema and information is more discoverable than OSF. That may be because I work here and I've read thousands of them. But I've always found that like clinical trials, it's harder to find the important things that I'm looking for. Whereas OSF, it feels much cleaner and easier to get to because you can link straight to that pre-registration. Whereas in clinical trials, you provide that identifier, you have to search it, and then you get this big web page that has a variety of links and characteristics. But that may just be a learning curve for me. Okay, when would you not pre-register? Okay, in what scenario slash situation? So, we generally recommend holding off on pre-registration. If you're more in that exploratory mode, I'm still trying to figure out where I want to go, what questions I want to really test directly and explicitly. If you're just trying to cast a wide net to see what is interesting, if you're in that sort of context, you probably don't need to pre-register that. You can, like I said earlier, you can pre-register some decisions of like how you plan to explore, but that's optional and that's really up to you. Really, we recommend pre-registration when you are in that context of confirmation. When you've found something interesting and you want to see if it holds up to that additional scrutiny, add it at a pre-registration to it and go from there. I'm going to give Alex's voice a rest and pop down to a question we have about creating our registration form built on Prisma P, the reporting guidelines for systematic review protocol, and if so, how should the community show that it's interested? This is a great question. We're very much interested in working with different communities of folks that have worked in different disciplines, have different needs, and we want to learn more about those needs and about the norms and practices in your community. So definitely reach out. You can email me directly. You can email support at osf.io and those fine folks will get you in touch with me. We're interested in partnering with these groups and finding ways to meet your needs and grow community around open science practices and around registration. So definitely reach out. We still have 10 questions. We're bumping up on time. So I think we've got a record of all of these questions and we can follow up with these questions after the webinar. We're also going to send out a link to the recording as well as a link to our slides from today and some other resources that you all can use to keep this conversation going. And then I think we're also going to turn some of these questions into a blog post so we can follow up with you individually or follow up with you in mass so that everybody can get the answers to these great questions. Well, I was there was one that I was going to answer but we can save that later but there was one question was what types of errors in a prereg are you allowed to correct using the short registration process. So these are errors that may be you know clerical it may just be a lot of typos that really confuse the language or maybe you are a grad student and you're in a methods class and you pre registered one type of statistical analysis but you realize another one better suited for you or you get feedback from an advisor or something like that. You want to update the methods before you've actually started to do the work before you started to analyze the study. That would be the type of thing you'd want to update your pre registration using that short registration process where you would you know just briefly define the change you want to make why you want to make those changes in the state of the project at the time that decision was made and then also just for visibility sake a link to the original pre registration because you know often and also another example just while I'm thinking about it is say you're having issues with data collection and you're not getting the number of participants you need you can start to if you need to add an additional recruitment method or something like that you can include it there but again we encourage you not to update the pre registration after you know the results because now it becomes an it enters that gray area of are you making these changes because you've seen the results and you're looking for positive and novel or are you changing it because it's a methodological issue. And I think the rest of these things will follow up like Sarah was saying offline. Thanks everybody for coming out and joining us today. I hope this was informative and useful if you have any other questions please feel free to follow up an email. We will be in touch with our website following up with a link to the slides a link to answers for these questions and additional helpful resources that we didn't get to in this webinar. Yeah, and that have a great day. Thank you all.