 All right, I'm going to go ahead and get started. I want to welcome everybody to today's webinar. I'm very happy to introduce Tamarinda Hevin, who is presenting the results of work that's been ongoing for, I think, over a year now. Tamarinda is finishing up her doctoral research at BU University in Amsterdam and is starting a postdoc at the Quest Center in Berlin. She was a scholar in residence here at the Center for Open Science right at the beginning of the pandemic. And of course, it was very unfortunate. She left us in person at that time. But the work has been ongoing, the work that she'll be presenting today on the Delphi study and the community development of these pre-registration recommendations for qualitative research. She'll be talking about the development of that template and the process. I'll come in again at the end to talk a little bit about some tips for writing up the results of pre-registrar work and some community activity around journals that are engaging with the process and a few other aspects as we go forward with this. Tamarinda Hevin is, as I mentioned, finishing up her doctoral work at BU University Amsterdam. She has a background in epidemiology, psychology, and philosophy of science. And her current research interests are all about responsible conduct of research, research, and equity, and open science and meta science. Tamarinda, welcome. We're really glad to have you here today. I will stop sharing my screen right now and let you present. And I want to let everyone know that slides will be made available to everyone who registered for the webinar. We'll post a link to that so that you can get access to that. And a recording will also be made available. Please go ahead and submit questions throughout the webinar. I'll be monitoring the questions. And if it's a question that deserves some little bit of interruption for clarification, I'll jump in and do that. Otherwise, we'll make sure there's time at the end for questions as well. Tamarinda. Thanks, David. That was a very clear introduction of my work. So well done. So as David said, from January to April, I was officially a scholar-in-residence at the Center for Open Science. But I want to say that the work I'll be presenting today is not something I did alone. I benefited from a very notable and helpful steering committee that I briefly want to introduce to you. So on the top, if we're going to have to write, you see Raphael Pinero and Fernando Rosentlat. They wrote a very interesting paper on pre-analysis plans for qualitative research. Then if we go to the right, there's Florian Ken and Christian Gladich. They disseminate the pre-print on pre-registration in qualitative research and pre-analysis plans. On the bottom left, the man with the glasses is Annan Jacobs. And he wrote a really interesting chapter on publication bias and pre-registration in qualitative research. And then to his right, it's Lea-Anne Marchotl, she was my co-author on Elizabeth more of a conceptual paper on qualitative pre-registration. Then I think the man in the middle will be familiar to most of you, and at least to David, because that's Brian Nosek, who was also a steering committee member and is the executive director of the Center of Open Science. To your right, we have Tim Arrington, director of research. And all the way in the right, bottom row is Meenika Mocking, who was a Delphi expert. Now, I already said that Lea-Anne and I wrote this paper together called Pre-Rex and Qualitative Research, where we did a little bit more of a conceptual exercise, trying to see if even though that pre-registration is primarily used in the quantitative field, would there be a way to meaningfully extend it or revise it so that it could also be used to boost the credibility in qualitative research? And basically our conclusion was, well, yes, but the format would be subject to rather substantial modifications. And that's what I'll be talking about for the rest of this talk. So I'll briefly give a bit of background, clarify what I mean with pre-registration, qualitative research, just so we're all on the same page. I'll explain why I believe that pre-registration as a practice may be relevant to the qualitative research. I'll review the Delphi that we carried out, how we've recruited suitable panelists, how the first questionnaire was created, what our initial results were, and then we'll have a little bit of overall discussion and I'll also show you the template that we actually developed. But first things first, what do I mean when I talk about pre-registration? And here I've just taken a quote from a paper by Nosek et al in 2017, where they explained that scientific progress is driven at least partially by generating hypotheses with existing observations, but also testing hypotheses with new observations. And is this distinction between post-diction and prediction that they argue is, we all know this, we appreciate conceptually, but it's not always a tier two in practice. And they say that it's because, mistaking generation of post-dictions with testing predictions, that would reduce the credibility of the research findings. And an effective solution hearing would be to define the research questions and analysis plan beforehand, that means before observing the research outcomes, and that is a process we call pre-registration. But it made it interesting to look a little bit more into what then is post-diction, because that's the unfamiliar part, at least for many of us. And they have defined post-diction as, in this type of research, the analytic decisions are influenced by the observations. And these observations in the data, those are the ones that create the fork in parts. Like the researcher is exploring what is possible. And the data here is helping to generate not necessarily to test or verify new questions and hypotheses. And this, you may say, is already a little bit resembling of qualitative research. I understand quantitative research or how I present it here very broadly is that it aims to answer the how and the why or the what questions of a phenomenon. It tries to reveal the perspectives of subjects or patients. And it has a couple of, well, at least relatively unique characteristics. First, the research process is flexible. Flexibility really is an invaluable asset here. The researcher has the freedom to engage in a cyclic process of data collection, data analysis. So things like the number of participants is not fixed beforehand. Like if need be, the researcher can go back into the field and recruit more participants when saturation has not been reached, for example. Another thing is that qualitative research really embraces subjectivity. And with that, I mean that, you know, the qualitative research actually, you can say functions as part of the measurement instrument itself. The quantitative research has a great say in generating findings from the data. So during the data analysis procedure, the data transformed into descriptions of maybe themes, patterns or theoretical models by means of the research going through several stages of data interpretation. So every result in the qualitative design is one that is made possible by this interpretation. So it is, in a way, subjective. And lastly, qualitative research, very analogous to the post-exon research you just presented, can be used to generate theory and hypothesis. But the can is really important here because there is also evidence from recent reviews among others by Alan Jacobs that if you look, for example, at leading political science journals, about 40% of the qualitative work that's presented there does explicitly aim to test hypotheses. Now, why do I consider qualitative research a candidate for preregistration or why would preregistration be helpful as a practice potentially? And here I've just put that I think it can strengthen the credibility and transparency, but I really have two things in mind here. First, the openness of this information about the study, the fact that it's published on this open platform, that can really encourage the researcher to carefully reflect on different study aspects and to systematically report on the design and analysis choices, including those that are made as the study progresses. And secondly, these records about the study design analysis plan can help the reviewer, the reader, or the user of the study in assessing the study's quality because they get a structured insight into how is this study set up? How is it designed? Now, in the current landscape, the importance of preregistration is increasingly recognized by different stakeholders. And here you can think of funders like the Arnold Foundation that now require the work they fund to be preregistered. But you could also think of journals as David will be talking about and disciplinary organizations that encourage the practice like, for example, the EPA. But if we take a little bit of a step back, you may say, but hold on, isn't preregistered in qualitative work inherently counterintuitive? Is something so subjective and so flexible? You know, what's even left to preregister? And to that, I want to say a few things. First, well, yes, a precise specification of the internet analysis where you basically fix all the parameters beforehand, that is unrealistic for qualitative research. But the usefulness of preregistration for qualitative research makes that more widely than is immediately apparent. So, you know, first, we already talked about these forms of qualitative research that do involve an element of testing. That is why preregistration may come in helpful. But also for those types of studies that operate on a more explorative or iterative logic, even when you're exploring, scholars usually bring some prior knowledge or some theoretical preconceptions to the study. And it's the preregistration and that can help distinguish, you know, which aspects from a study's findings drew on and which diverged from these initial beliefs or expectations. And as I already alluded to before, different authors, the steering committee members to pretty broadly put forth ideas about what should be included in a qualitative preregistration format. But, you know, these investigations were not really done empirically. These were just conceptual ideas. And it was at least my understanding and that of the steering committee members that if we want to make something for a community to potentially benefit, we need to involve that community in designing the form. So the goal of my stay really was to gauge and understand, you know, which parts of a preregistration template would qualitative researchers find helpful or informative? And for that we use Delphi study, which is appropriate because they're, you know, as I said, there are a few papers on the topic, most are conceptual rather than empirical. And Delphi is really a structured group communication method. So it's a helpful tool to find a set of items that a community agrees upon. And just in a nutshell, a Delphi consists of consecutive questionnaires with feedback reports in between. And the idea is that, you know, like surveys, you can reach out to an otherwise dispersed group of very busy people. They can complete this in their own time. And in contrast with maybe other consensus or agreement-making methods, because these feedback reports are sent to everyone in an anonymous fashion, the idea is that there's not one person that can really dominate the debate. And you can see the actual design of our study here. So as you can see, we had a proprietary phase. Our first round really focused on making sure the items that we created were relevant, whether they were comprehensible and whether we had been comprehensive. And the second round was much more focused on refining the phrasings and both collectively led to the final form I would be presenting. Now, we put together a pool of potential panelists using a four-fold strategy. So first we asked, I asked basically, the steering committee members to put four suitable colleagues, people that they knew to be, you know, good and knowledgeable qualitative researchers or qualitative methodologists. I've also invited all the authors of qualitative research reporting guidelines because as you can maybe already see, there may be some resemblance or some also, you know, desirable overlap between reporting guidelines for qualitative research and a preregistration format for qualitative research. And the third party here is what I've called active qualitative researchers. And with that, I mean that we did a very broad search in both scopers and web of science to filter out those authors that have published at least four papers within the last five years using qualitative methodology, reasoning that that's a good proxy to get a sense of, you know, who has been active in the field. And finally, of course, people could recommend colleagues to take part, which led to a total pool of 294 research that were invited. Now, we also used the four-fold methods to create this questionnaire. So first, we integrated the existing works by the steering committee members. And secondly, we did a systematic search in both PubMed and PsychInfo, reasoning that those are two databases that contain quite a lot of quantitative research. And we used quantitative research as a term, preregistration. We also screened the 21 existing reporting guidelines for qualitative research on the equator and have hand-searched additionally for some other reporting guidelines that maybe it is but less biomedical oriented. And that led to a rather large pool of potential items, namely 36. And we discussed those with the steering committee members and they commented that suggested added, just things to be merged. So our first questionnaire was really based on these 21 proposals. And just to be fully clear, with a proposal, I mean a suggested term and then an elaboration. Now, this is an example of what our first demo looked like. You can see the term here, the term is study type. And then the elaboration is below, which starts with please describe. And we also wanted this form to be intuitive. So on the left column, you'll see the different headings and people in the first round were also asked to, indicate is this the heading where you would expect to find this term because we wanted the template to be intuitive to navigate. Now, our panelists also saw this demo and all the different proposals. And then we asked them to rate on a scale of five, being strongly agreed to one, being strongly disagree, to what extent do you agree with the suggested term, the suggested elaboration? Where do you think this item is actually relevant to the qualitative research? And finally, I do agree with the heading where we have placed the term. And as you could see, the headings were things like study info, design plan, data collection and analysis plan. And here comes the really important point of Adelphi. We also asked for arguments. So if people rated that an item was really irrelevant in their view, was that because it needed substantial revisions? Was that because they expected it elsewhere? Did we just need to come up with a new term altogether? And I can really say I've learned a lot from the different arguments that panelists provided. Now, Adelphi seeks to establish agreement. So it is key that you define up from, what percentage do you take to be agreement? And we set our agreement cut off at 68%. So that meant that 68% of the panel members had to either strongly agree or somewhat agree with our suggested proposal. Now, in the first round, there were about 39, sorry, researchers that completes the questionnaire. And of our initial 21 proposals, 14 of those were considered relevant. And we've merged them into 11 revised proposals because we merged them because panel members sometimes suggest that KD's things actually tap into very similar issues. You might be better of combining them. And also 21 proposals still arguably quite long. There were two proposals that had conflicting arguments. And they had scores that just below I cut off criterion. And with conflicting, I mean that there were some panelists that said, this is the most important item that I've seen thus far, but other people that said, you should leave this out altogether. So we did our best to rephrase those as best we could. And to present them again to check, what should we really do with these? Here you can see a demo of our second iteration. You can again see the revised term. This time we talk about study design and the revised elaboration that starts with please indicate. And this time we only asked panel members to rate the extent to which they agreed with the revised term. And they saw that we presented them the all term, then we gave them an overview of their arguments, a formulation of how we worked their arguments into the revised term, and then they saw the actual revised term. And we did the same for the revised elaborations. And again, we asked them, please help us, help us make this better, give us your arguments or where we could we push potentially improve. In the second round, there were about 35 researchers that completed a questionnaire. And of our 11 proposals that were left, 10 of those were already above our agreement criterion, so we only need to slightly revise those. One, we had to revise it a bit substantially. And we see a very similar picture for the elaborations. So we have still revised all of them, but some of course, a lot more than others. Any two optional terms that I just presented a few slides ago, they were actually considered to be relevant. So they've been added to the form in their revised version. Now, I thought it was nice to briefly sketch what this would look like. So here you can see in the sort of bottom row, if you will, you can see the term. So in our first round of the Delphi, we presented a term called sampling strategy. And then we got various forms of feedback. That's the sort of, that's the row above, if you will. And I've made both the specific parts of the feedback that the panellists gave that we really worked with. So people said, look, you shouldn't maybe talk about sampling. It's very confusing. It's a statistical thing. It doesn't really matter here. So in our second round, we used a term called case selection strategy. And then people were a little bit happier, but they still thought, you know, you may be mixing two things up. And there is a way in which we use sampling to define a process of recruiting participants. So here you can see that our final term reads sampling and case selection strategy with also a final elaboration where we try to work with those remarks. Now to conclude, we developed an agreement-based form for pre-writing qualitative research. And I think at this point, it's really important that I say it is pivotal to assure that the strength of qualitative research, which is precisely its flexibility to adapt, to adjust, to respond to what is found, is not lost in pre-registration. And it need not be, because analogous to quantitative pre-registration, a pre-registration is a plan. It's not a prison. I also want to emphasize that the use of these pre-registration, of this pre-registration form is voluntary. And it should provide a systematic starting point. And it really needs to be, especially in qualitative research, updated as the study evolves. I also want to say that, you know, qualitative research is a hugely diverse area. And the form that we developed, that is based on a sort of overall agreement, may not be optimally suited for every study. And that's why we would encourage scholars to propose what have labeled here extra modules. And there I'm thinking of, you know, bits and pieces that could be added to this form that may be relevant for their specific type of inquiry. So for example, approaches like process tracing may be able to define much more precise evidence criteria before actually commanding a data collection. And that could be added because it may be relevant to some parts, but not to sort of all researchers engaged in qualitative research. And finally, and maybe this is no brainer, but I do want to emphasize it, there is at this point, no empirical evidence that pre-registering a qualitative study would lead to a higher study quality, whatever that may be. And I think what we now need mostly is good examples that people can use for inspiration and to get a sense of, you know, what's the level of detail that you're expecting me to provide here. Now, let me end with a little bit of next steps. So as you'll see in a minute, the template has been integrated into the OSAP workflow meaning that when you want to pre-register a study, the work that I just presented is listed on the menu. And the other step, of course, would be to build a community or a qualitative pre-registration enthusiast. People want to promote the practice, people may want to educate others or who could in some future time evaluate its potential effects. Now, our paper that describes what I've just described to you has been accepted in the International Journal of Qualitative Methods, could basically be released every day or any day now. I think David and Claire are going to send a link to this work to all of you later so you can read into it a little bit more. I do want to take a brief moment to thank all of the panel members. Here you can just see the names that have agreed to be disclosed, but there's also lots of anonymous panel members. And this would really not have been possible without them. They gave me, you know, voluntarily and without any compensation, an hour or sometimes one and a half hours of their time. That's how long these questionnaires take. And of course, I also want to thank CUS. And I'll be giving a short demonstration of what the form looks like. But in case you want to browse for yourself a little bit later without necessarily having to pre-register already, you can have a look at the OAS Earth Wiki where the form is also available as a Google Doc. Maybe if you first want to look into the questions yourself a little bit more. Now, I wanted to give a brief demo and I'm just looking at David to see if you can see that my screen has changed into OAS Earth Test. Yeah, perfect, thanks, David. So here I've just made a task project. You can see task project called Pre-Rect. And then I've went to the registrations tab and now I'll do a new registration. So you can see the manual I've just talked about. So we'll go to qualitative pre-registration. And first is going to give it a little bit of metadata. To this, of course, where you give the title. I've already given that for you. We'll give a brief description that this is a demo. You can see me. I've been one contributor. You could of course add others, a license, add your subjects. Maybe you're working in arts and humanities. I am partially, but that's all the sort of less informative stuff for us now. So the first part I already alluded to is the study information. So first we're interested here in your study aims. That's really just sort of what's the overall objective or purpose of your research. As you can see here, if you think that it is different across different domains, then you may also want to specify it. There may be different types of aims that you may have. You may want to really evaluate a theory. You may be much more exploring or an understanding. So if you want to, you can select the type of aim you want to work with. And then of course, very closely related is your research question or research questions, as they will be guiding your research. And if you may be doing hypothesis testing work, this would also be the place to disclose it. And here we've also given a little bit of an example. So as you can see, oh, you cannot see. I'm sorry. Should be, yeah, there we go. This is just one random example I've given here, but I do think it's informative so you get a sense of depth what sort of thing we're after. So here I've set, you know, looking for your aim to explore the attitude of caregivers towards our time in patients. Maybe coronavirus patients would have been a much more relevant example, but please bear with me. In a local board, then your research questions could then specify, you know, what do you plan to study? Maybe you want to look into how the board stuff still tries to treat patients with dignity that may be especially challenging if they have to wear all these face masks and basically have a sort of like moon-like suits. What sort of things do they still do to show the patient that they can, that they're trying to treat them with their utmost respect? Or you may want to look into, you know, how is the relationship between the patient and the family or the loved ones has changed into patient was admitted? So that's just to give a somewhat overview of the type of questions. You can indicate the anticipated duration. So that's just to give a sort of overview of when this study is started and also when people find your work, they can know if it's still ongoing. And of course this may be difficult to indicate, that's why we only take months and years, but it may nevertheless be informative, especially for other people that may work on similar research. Now moving on to design plan, the overall study design here we're really looking for sort of a brief label or a very brief description. Examples here I think that case study or ethnography that are relatively broadly understood types of design. Of course you may have a much more sophisticated design here, including a mix of quantitative and qualitative methods. So that's again something you can indicate here. And then we're interested in learning, you know, how are you going to recruit your study participants? What sort of strategies will you be using? And also then from those participants, how are you going to select what you'll be focusing on? How are you going to really select your cases? And for a couple of these items, you will see that we've added an additional prompt to ask you to think about, you know, why do you think this type of strategy is appropriate? Maybe also given the type of aim that you just indicated or given the type of research question. Moving on to data collection, this just very descriptive. We would like to know the type of data you'll be using. Of course, a lot of this, as you will see, will be language, written in oral, but it could also be other types of maybe behavioral recordings or secondary literature. And here we're also interested in your data collection methods. So here we've just summed up a couple of examples that may easily come to mind. So interviews are a very common one, participative observation, maybe in the Alzheimer example we just used, it might be much more informative to observe interactions than to interview people that may already be very busy. And again here, we're asking to provide just a short rationale for why is this particular method appropriate given your study aim and your research questions. And here's the point where we're asking, okay, how are you going to collect this? So if you would be doing interviews, this may be the point where you want to upload your topic guide or if you're using this bit more creative tools like enabling techniques, you could also add those here as you can see, you can add up to five files even or you can just give a brief description here. Then there is the analysis plan. Here we're first interested in, what data analysis approach do you take? And especially because some of these terms are understood rather differently by different scholars. So I hope it works now, right. If we think about things like phenomenological analysis, there's still a rather broad variety therein. So just saying, I'm going to use thematic analysis or phenomenological analysis may not be very helpful. So here we've given examples where we refer back to, okay, I'm using this approach as was defined by this scholar, so that others that work in your niche in your field would know or would be able to find out, okay, what is it exactly that you're doing? And then we would like to know a little bit more about the process of data analysis, maybe people that will be involved, what sort of procedures will you be using to turn raw data into a more analyzable form. And here, if you did specify any hypotheses a few sort of taps ago, then what sort of evidence will you take to be consistent or inconsistent with that proposition? And if you think it's informative, what sort of software and tools will you be using? And then we close with some, what we've labeled here, credibility strategy. So these are really measures that you can employ or things that you can do to try to assure good methodological integrity. So a couple of examples here will be familiar to the most of you. Member checking is usually the process where say the people that you have interviewed receive a summary or the full interview transcript so they can actually check in a way, like, did you catch the right point or did they maybe mean something else entirely? Peer debriefing is also something that many of you may know. But we've included a rather large array of examples, as you can see, but we also added other because there are so many credibility strategies people may be using in a few, we couldn't sort of actually capture them all, but this is intended to give you maybe some prompts to think about. And again, we want to know why you have used those. Why do you and your team think that these particular strategies are going to be most helpful? And then finally, this is an item we had a lot of debate about and I personally think it's very important, but opinions differ in this regard, of course. This is the reflection on your positionality. So here we are interested in learning what is your relation with the phenomenon you're going to study. This may be clearer when I show an example. So here we've taken a case of, if you studied the lives of detained immigrants, we may find it helpful to know a little bit about your political viewpoint, like how is your lens shaped with which you're going to look at this data? Maybe you've been doing a lot of relevant policy work. Maybe you have a personal background as a detained immigrant, because that can help us to better understand how you got to particular interpretations of the data. And then of course, it will close with an overall review, but I don't think that's informative for us here. So I'll hand it back over to David. Thank you so much for that overview. I'm always struck by how collaborative this process has been, both the iterative feedback of the Delta workflow overall and the community input and of course, I wanna give thanks to all the folks at COS who are working to sort of put this online as part of the registry. A couple of good, great questions have come in. I think now is the time to tackle those two questions that have come in before I'll finish up with pointing to a couple of relevant resources and then there'll be a little bit of time at the end for more questions. So keep those questions coming in. We'll start with Razia has a great question. Razia is a PhD student doing qualitative research. Can I use this at the beginning stage or who's gonna be the user of this? And Cameron, I'm wondering about your response to that or I'm happy to tackle that as well. Yeah, that is a great question indeed. And it can be very, very broad so it also depends a little bit on what you would be looking at. So of course, you know, you could be using this but I think what your question is referring to who would potentially look at this? And if I'm correct there, then that depends a little bit. So first of all, there is the option to embargo your preregistration. So sometimes you may be doing very sensitive work and it would be really unhelpful for you for a lot of people to already know about your work or maybe for the people that you're going to interview to find out about what it is that you want to find out. Then you can embargo your preregistration. In that case, the other users may be your team members or the people that you will confidentially confide this information with and they will be able to give you feedback about the sort of design decisions that you made or they may be able to ask you constructive questions to better shape your research, especially at this early stage. If we disregard that embargoing option altogether then we will be looking into a sort of broader community and it depends a little bit on you. If people go to the OSF and they may be interested in a specific question or they may work in similar work to you they may just simply look up sort of key terms to find out if someone else is working in the same area as they do. Then they may run into your preregistration. They could then contact you for questions maybe they want to collaborate and so on. It could also be that you actively seek out feedback from other people by sharing your preregistration. Then you of course, unless a bit more in control it will still be public and open but you will explicitly attract others to come in and comment on your work maybe or to just show them what sort of thing you have been doing so far. And I can especially if I think of my first year at my PhD remember departmental presentations where I've just given a sort of overview of my work. This may be a great tool to then leave to after. So it really depends and you have a variety of options and it also very much depends on to what extent you would like others to come and find your work. There are of course the hypothetical scenario that I want to be honest about that you put this online that nobody will ever look at it but it depends a lot on what it is that you want to attain with preregistering your work. And now we just add to that the one guaranteed user or audience of that will be kind of your future self kind of sort of helping to remind precisely what your frame of mind was or your rationale or the justification was for one particular analytical strategy was. Sometimes they depending on the data collection methods or the length of the work could be several years from when a pre-registration is created to when kind of writing up really takes place. I think it could just be a useful reminder for understanding the justification and the rationale for one particular decision that might have occurred quite a while ago. And every department, you sort of mentioned your committee early on. Every department runs things a little bit differently. Oftentimes there'll be a couple of years in there'll be sort of a mid exam or a proposal stage where committee feedback kind of simplifies some of the PhD research plans that will take place over the next couple of years. And that's often where we see pre-registration being used in doctoral research programs. And that's a great opportunity for that. Crystal asks, and this is something that we can get into a little bit more also, but there's the question about mixed methods. If Crystal's doing a mixed methods program, basically should one choose one of the quantitative pre-registration forms, this qualitative form, or two different pre-registrations? I have an opinion on that. Tamarind, I'm curious if you do too. Yes, I do have an opinion. And my opinion would be that we deliberately in creating this form has kept that, we have kept that sort of option open. So there are various prompts where you will see, maybe we just talked about study design. This will typically play out in your sort of study design that you're using mixed methods. So we've explicitly added a small sort of part to the elaboration, saying, if you may be doing a combination of quantitative and qualitative work, this is where you could feed that in. So in that sense, I think that the qualitative pre-registration form that we developed can be to some extent modified to accommodate your mixed methods. But to some extent that last sentence is very, very important. It will of course depend a lot on what has the amplitude in your work and whether you think it's ultimately suitable because it may also end up in sort of really awkward situation where you may be missing items from the more quantitative part. And actually, David, I don't know if we can sort of mix these forms in a way. So what do you think? Yeah, I think that could be a possibility for the time being. I think the best strategy is to take a look at both of them, take a look at OSF pre-reg, take a look at the template adapted from the work from as predicted, take a look at the qualitative form and kind of see which one makes the most sense to you given the work that you're about to endeavor on. And I know this qualitative form, as you mentioned, I think it's flexible enough to be useful for many mixed method studies if the work that you're about to endeavor, you're about to take up slightly more quantitative or qualitative focused, that could influence the decision to use one or the other. I would generally not recommend pre-registering twice unless you're approaching this as two different studies. If you're approaching this as you think of it as two different studies, a qualitative and a quantitative work, then that would, of course, I would, of course, recommend two different registrations. In general, I would recommend trying to stick to one pre-registration per study, however you as a researcher define it. That's not a hard and fast rule, that's just kind of like an organizational recommendation to try to keep the so-called file drawer as tidy as possible. Of course, it's not too tidy because there's a lot of ongoing work that changes over time, but it can be a way to try to make that at least a little bit more interpretable as time goes on. I'm going to point to a couple of resources now and we'll keep your questions coming. I think we'll have some more time to answer that. So I'll pause on answering more questions right now and point to a couple of bits of ongoing work, please. Bear with me as I take over your screen another time. All right, I wanted to focus on tips for writing up the results of pre-registered work. I want to show you a couple of ways to use the OSF registries for, as Timuranda mentioned, sort of discovering ongoing work that's been pre-registered. And then take a very short mention about what register reports are and how journals are engaging with this. And this gets to one of the questions that's coming in through the Q&A, but once work that is pre-registered is conducted, of course, that the next step is to write up and submit for publication. And we've worked with a lot of researchers going through this process. The number one rule to make sure is to declare, to include a link to the pre-registration. And there are different ways to do this. You could cite it, you can include in a footnote at the top of the method section, remind readers that this work was pre-registered. And each item on the OSF, whether it's an uploaded file or a project or a pre-registration, includes a citable link. And so it's a short, persistent URL. You can create a DOI, a digital object identifier to make sure that these are more easily discovered that way. Including that link will allow the reader to take a look or reviewers or editors to take a look at what that pre-registration included. Importantly, the number two recommendation is to report the results of everything that was in that pre-registration. So if you have four different avenues of work in that study, include the results of all four of those. If one of them, the number three, didn't pan out for it or every reason, still describe it even if it doesn't make a major part of the narrative, include a reference to it, say what was attempted and give a brief explanation of why it wasn't continued. This gets to one of the main points of rationale, one of the main justifications for a pre-registration. And that's to address sort of the publication bias or the known tendency to only point to the stuff that worked especially well. It makes us happy, it creates the cleanest narrative. But understanding all the work that was attempted, but it didn't pan out as expected is an important way to give a more complete picture of all the work that is undertaken. This gets into the distinction between planned and unplanned or maybe confirmatory and exploratory work or discovery work. Anything that was not part of the pre-registered plan, go ahead and include that in any write-ups of the work, but make a note that this was not part of the pre-registered plan. Halfway through, you discovered that people were responding one way versus the other and that took a different route. That's absolutely appropriate, but it's important for the reader to understand the context that the justification for that sidetrack occurred based on incoming data or was not part of the original plan. That provides the most complete context for the reader to understand the work. And finally, this is a little bit of an extension on that first one, but it deserves its own emphasis is that any sort of any changes that occurred in the methodology or the analytical strategy need to be documented. So those transparent changes are to be encouraged. This is a way, again, to give the most complete context for what changes occurred and why. We've got some recommended language or templates for how to do that. Again, these slides will be made available if you want to take a screenshot of the screen right now just to capture that URL, please do so. But keeping those transparent changes is just a way to, again, provide the most complete context for the work. There are a couple of search terms on the OSF registries that I'll encourage you to look at. Again, there's a bitly link at the top, but if you go to OSF and the top left corner just click on registries, you'll get to a couple of different search filters to look through this open registries network. The OSF registries lets you filter by different type of form. And so, of course, this new qualitative pre-registration template was just released just very shortly, last week or within the past few weeks. And there are already nine publicly available registrations available on that. So those are openly available to Peruse. You can take a look at how people are using this for different types of studies right now. There are presumably several more that are under embargo right now and then the work is under array. So this will continue to grow as time goes on, but there are already several examples to look through to see how this work is being, to see how people are engaging with this process right now. This is just a search term from the generic OSF registries across all of the forms. We've had templates that are, of course, derived more directly for quantitative work, but scholars have been engaging in this for quite some time. So this is just a very basic search term. We have 290 projects that include in their titles or their descriptions qualitative methods. So take a look at that search string. Again, there's a bit.ly link at the top if you want to take a search of it or just navigate to osf.io slash registries and you can do your own searches to see how people have been engaging with this. And so those are all the available OSF, publicly available OSF registrations that engage with some of these methods. I wanna take the last bit of just a few minutes to give a little bit of a distinction between the process of pre-registration and engaging directly with journals in a format called register reports. The disambiguation between these can be a little bit confusing, but they complement each other very nicely and often they involve each other. So a pre-registration is, of course, specifying in advance, submitting to a registry how a study is planned to occur. The work then takes place and it is written up and submitted to a journal. With register reports, I'll give a workflow diagram in a minute, it's a two-stage process where the proposed, that pre-registration, that proposed plan is submitted to a journal before the work is conducted. That plan then undergoes peer review and can be granted in principle acceptance, a promise to publish regardless of how the study turns out. Both of them, they have complementary benefits, both pre-registrations and engaging a journal with a register report addresses unreported flexibility in how studies take place and help make clear distinction between planned and unplanned work. Registrar reports are particularly useful for addressing publication bias against null findings. And that register report process includes that two-stage peer review, which I'll get into right now. So that plan study looks essentially like a regular journal article submission with the introduction and justification and proposed methods, but of course it does not include any results, except perhaps some prior work or pilot work. That stage one peer review asks the authors to determine if the study has big enough sample size, are the proposed questions and methods feasible and detailed, and is there some sort of quality control step included in the proposed study design? If editors and reviewers agree that the answer to those questions are yes, it's large enough study that they'll promise to publish the results regardless of outcome. They can be given that in principle acceptance for the final results. That stage two peer review then occurs after the study occurs. This is often much quicker review process, simply asking if the authors followed the approved protocol and are the conclusions justified by the data? So this format has been around, for several years it's been independently developed by a couple of groups over time. The particular initiatives that we support a lot of journal editors with has been around since around 2013, and there are 275-ish journals that accept this format, the Register Report peer review format. There's a comprehensive resource page available on the COS website, COS.io slash rr. And I particularly wanna focus on a policy table under the participating journals tab. This has a comprehensive list of journal policies and requirements for each of the journals that accept this format as a publication option. And several journals have indicated that they'll accept this format for a qualitative research designs. Several in the education field and several in psychology accept mixed methods or qualitative studies with the Register Report format. This can be a great way to get early peer review and that very enticing, in principle acceptance. So a way to guarantee that the work will be publishable no matter how the study turns out. So take a look, here's another bit.ly link. All these links will be made available if you wanna take a screenshot, please do so now. We know of about 270 journals, at least 14 of them specifically state that they will accept qualitative studies as part of these Register Reports. 52 of them said that they'll consider it, so the decisions to be announced, but that does indicate that they're at least considering it and would be receptive to an inquiry. So take a look at that table for journals that are potentially relevant and some updates to this table are to make it more searchable or coming. So be on the lookout for that as we work on that. Perhaps the most common, the biggest most frequently asked question in a lot of these, and we've already touched on it a little bit with the embargo question, is whether or not one can be scooped based on submitting work to a registry. The simple answer is no, these are citable documents. This is a very concrete piece of evidence that work is yours. It's time stamped, it includes your name and it dictates when this idea was developed and when it was put into a publicly citable database. If the work is particularly sensitive for whatever reason, as I already mentioned, please do use the embargo feature. Maximum of four years gives a good body of time, good amount of time to make sure the work can be conducted and only made publicly available when it's appropriate to do so. All right. I think that is, yeah, that's it for my slides. I'll leave this up for a minute and I'm going to start going through some of the questions that have come in. If you're heading out, thank you very much for attending and let's get through some of these questions. If you have questions that have popped up, please put them in the Q&A window. So I'm going to minimize my screen right now and so you'll just see our lovely faces mostly. All right. I'll go through a couple of these questions. Some of them have already been answered but it's okay to be a little bit repetitive. Just a quick one to this one. If you're submitting for a publication, you should include that link or a DOI in state either in the cover later or at least in the method section. The work was registered and here's a link to that. No matter how much you deviate from the plan or decide that the work deserves to be changed based on lessons learned over the course of the study, including that is best practice. Laurie asks, you mentioned the mixed methods. Do you think a project with quant and qual? Yeah, I think we've already answered this one. Generally don't preregister them separately. If you think of it as a single study, try to keep it in one pre-registration. It's not a hard and fast rule. It's just a guideline, but it's kind of up to you to decide if you think that they're two different but related studies than two registrations, that's fine. I would err on the side of putting them into one pre-registration if you think of this as a single study. Ruler writes, the template looks good, but what's the envisioned process for how to update a registration over time as you move through the research process where you may change data collection and strategies, the shape of your questions? And for context, Euler is an early career researcher in human-computer interaction. Tim, Miranda, do you want to answer that or...? I think it was also already a little bit in the slides that you just provided. Well, at least the slides, the link that was in there will give a sort of short overview of how to do that in practice. And my sort of first answer would be that the current way is a way, but it may not be the sort of easiest way. So, of course, this sort of whole pre-registration thing was developed in a sort of mindset that it should be this frozen, non-advertible version only. So what you will see is that this frozen, non-advertible version will forever remain, which is one of the reasons why you just saw that there was a review option before you would actually pre-register your work because that is going to be findable until a very long time at least. And then, indeed, especially for the sort of quality of case we're talking about, there may be various bits and pieces where you want to update along the way. And actually, whereas in quantitative research, that may give people this bit of an underbelly feeling, which is very often not justified, but it will still happen, I would say that in qualitative research, the sort of opposite should happen. If there have been no deviations whatsoever, then a lot of qualitative research would be a little bit puzzled. The ideal way to do that now, and David needs to help me a little bit here with how we're doing that in practice, but I believe that you can sort of actual summary reports where you can then specify what has changed and why. And it sort of gives you a little bit of an open text box to just describe what has been altered and for what reason, is that correct? Yeah, there are two ways to do it on the OSF platform right now. There is the description field. Every registration has some editable fields we call the editable, it's hard to say, metadata. And that description in those fields can be updated at any time, and that's a great way to note any changes that need to be noted. The other is, and I'm sorry I don't have time to demo this fully, but every time you create a registration on OSF, you have a project. And that project gives you a lot of flexibility to upload a document that you can then version control. You can use the Wiki as a narrative description. And those places give an ability to structure in any way that you think is appropriate. Changes over time, so changes to your data collection strategies, changes to your methods, changes to your analytical approach. And keeping a change log, keeping a single document with documenting those changes can be helpful to you, to your readers, but mostly to you, it's just a good way to keep the track of what's going on. As we get close to time, if there are questions that are unanswered, what we'll do is we'll write those down and submit them to everybody. I want to, Sabrina asks, there is a particular template on the OSF registry for specific for register reports. Yes, if you have in principle acceptance, if the journal has said we'll publish this work, regardless of outcome, you'll have a PDF or a Word document of that. Go ahead and use that very simple form as a way to upload that document, and you can submit that to the registry if that's what you have. And I would recommend using the qualitative template that Tamarind has been talking about as sort of a basis for creating that manuscript, but not necessarily that template directly on the OSF registry if you have already been given a promise to publish the results of this work, regardless of outcome from a journal. But if you're using the qualitative form and you've registered it, no harm, no foul, it's not a, nobody will penalize you for that. So we will, let's see, we are at time now, so I wanna respect everybody's time. What we'll do is we'll copy out these questions, write up a couple of responses to, if any of your questions didn't get answered. Thank you very much for participating, Tamarind. And thank you very much for being a scholar residence at the Center for Open Science for this great project where we are genuinely really keen to have this, have the results of all this work up online. Thanks, all.