 Hello and welcome to this ASMR conference 2003 tutorial. I'm Gjeldron Peters and I will explain a bit in this tutorial about the pre-record package and about the Inclusive Systematic Review Registration Form, which is actually where this package started and probably the pre-registration form most of interest to delegates at this conference. These slides will be available, the QR code and the link are in the bottom left corner and this slide will also return at the end. And this is a recording so you can always just go back and get it if you're missing it. So this started with me supervising a student who was doing a scoping review about something soccer related. Don't ask me why, I'm not into soccer, but he found it very interesting. And then we wanted to pre-register the scoping review, as one does, and it turned out that there's actually no service that allows this. So I went to Twitter and asked whether anybody knows the service that I didn't know about yet, where you could pre-register such a form. It turned out that actually there wasn't yet, but in the process as it got retweeted and a bit more people responded, I met Nicky who had the same situation. He does studies into veterinary science and he also couldn't pre-register his studies and work with students anywhere. So we had the idea to develop one and to get in touch with the Open Science Framework if they were into this so that we could add one to the Open Science Framework. At some point Brian Nosex responded and that got the ball rolling and then we got together a group of people which kind of snowballed into a larger group over time and we started working on this form. And the idea was of this form that it would be as inclusive as possible so that you could pre-register any kind of systematic review, whether it was a systematic review into case law, because you're like a legal scholar or anthropology or psychology or really anything, and also not specific to any review type. So you can use it for meta-analyses, but also for scoping reviews or qualitative reviews or whatever. As a consequence, we didn't have any items that are mandatory and there are quite a lot of items that are for example relevant for meta-analyses, but not relevant for scoping reviews. So the idea is kind of that you don't use this form. If there is a better form that better fits your specific use case, but because often there isn't, you can use this as a kind of generic full-back form. So we posted it on MetaArchive pre-print. You can get it at this Shoredoy G5FJ. This link will also return later on if you want. And that was that. In this process, I kind of started thinking about this process and the fact that it was necessary and possible. And I realized that this is actually one of the problems of the current preregistration landscape. As you can see here, this is of course the preregistration landscape. And in this preregistration landscape, there are some preregistration providers. For example, clinical trials.gov for RCTs, randomized clinical trials, of course Prospero for health-related systematic reviews. There is the open science framework for lots of different preregistration forms. And because of this landscape, the typical preregistration workflow is that researchers select a provider, they go to their website, they create a new form, they fill out the items that are there, they submit it. And then it's stored in the provider's database. It's published maybe after an embargo on the website of the preregistration provider and then it's shared in an article or a website or some other means. I realized that in this process, there's some room for improvement. One of those, I think the most important ones is to democratize science and to make it easier for everybody, regardless of their ontological, epistemological, methodological perspectives, to create a preregistration form that fits their use case well. In addition, I think it's important that preregistrations as much as possible are machine-readable and can easily be improved or imported and updated or reused. And since I use R, and I expect a lot of people at this conference also use R, I thought it would be nice if you could integrate preregistration in your R Markdown-based workflow. So that's why I started creating Pre-Record. So starting from the beginning, this is an example of two preregistration forms into qualitative research. For qualitative research, there's the concept of saturation. When a researcher thinks they have enough data and they will stop data collection. This is one preregistration form for qualitative research. And here they kind of enforce, they kind of mandate a specific definition of saturation. They follow Fouzhennaise of 2005 and they say, that is what saturation is. But there are different definitions of saturation as always. And this other preregistration form for qualitative and quantitative ethnographic studies allows the researcher to specify their own definition for data saturation. This means that it can make quite a difference which form you use as a researcher. We know that actually this diversity in ontological, epistemological and methodological approaches is actually beneficial. Science approaches or proceeds progresses faster if there is more heterogeneity in how people study things because there is no one single correct way. From this perspective, of course, it becomes a bit problematic that there are only a few forms available. UNESCO also recognizes this in the Oprescience recommendation which was adopted by all member states in 2021. So in principle, we all already believed it. And then they argued that actually it's important that their infrastructure we use and repositories we use are adapted to the user needs. So that basically whatever your ideas are about how things should be studied, you can actually work with this infrastructure. And currently, because there are only a few providers, the form items that they choose to include in the forms and how they're phrased and how the instructions are added change, they influence how people do their research. And this decreases the epistemic and methodological diversity which is a problem because it kind of slows down science. And in addition, ideally, we have open infrastructure that's as much as possible under the control of the whole scientific community rather than a few actors who happen to host pre-registration forms. As I said, that was the first reason for developing pre-regure and I think the most important one. Second reason is to make pre-registrations machine-readable. This is also something that UNESCO recognizes as important. It's also something that's recognized more widely as important. And I think especially at this conference where we do systematic reviews, I don't think I really have to argue very much that it would be great if stuff would just be machine-readable rather than having to code stuff and making mistakes. And then finally, being an R user, it would be nice if you don't actually have to go to this separate website and type stuff in, but you can just have the pre-registration as a part of your R Markdown-based workflow which in my case also means that you can use Git and you can easily go back and see what was changed when, by who, etc. So now I'll give some brief demonstrations of pre-regure using this inclusive systematic review form. And we'll start with creating and importing a form. If you want to create a form, you don't actually do it in R. You can do it in R, but it's usually easier also because you often collaborate with others who might not use R to use a spreadsheet. I have an empty template. Again, the link will be provided later. And this template allows you to specify the pre-registration form's content. So not the content for a study. It's not, you're not creating a pre-registration. You're creating a pre-registration form. So you have some metadata, the title of the form. For example, you could say there's an S Mark pre-registration form, the name, the date, et cetera. Then there are some instructions you can add, which are available to people who complete the form. There are the form items, of course. They have a section that they're placed into. You can see there's here in the third worksheet. So the sections have like an identifier, a label and a description. And items have referred to their identifier to their section with the identifier. They also have an identifier themselves. And they have a human-readable label and a description. So basically, this is quite easy for most people, also not two IT savvy people and not two R-minded people, to edit, and they can actually specify their form's content here. Then once you did this, you can import it into R. So this URL is just the URL from Google. If you click on the address bar, you can copy it. You have to make sure though that it's available to everybody. So it has to be, you have to use the share button to make sure that it is viewable by everybody. Then once you made it viewable by everybody, you can import it like this. So now we're importing it, and now we have it available in R. And we can view the content here. As you can see, there's not actually any content yet. If we would now edit the spreadsheet, I won't do that because of time, but if we would edit the spreadsheet and import it again, then whatever we change would be visible here. This object that we just created can be used to add content. But before we start moving on to adding content, let's think about how you actually get form content, or maybe I should have called it form structure. Because of course, before you can start putting stuff in the spreadsheet, you have to know what you want there to be in your form. So there are four or five ways to do this. The fifth way is just combining more than one of the first four. The duration list approach, the expert consensus approach, the Delphi method or scoping reviews, which I think will appeal to this audience especially. Before I go into that, I want to draw attention to the fact that there is no consensus on what should be in a preregistration form. And it depends on what you think preregistration is for. So John McVeethries wrote about preregistrations and he argues that it should be used for all social science research activities. And you could extend this to saying that it should be used for all systematic, all research science activities. And here the idea is that it's not limited to situations where hypothesis is tested or pre-value is reported because the goal is mostly to restrict research degrees of freedom. Basically to make sure that anywhere in the process you don't either consciously or unconsciously introduce bias. And if you do, it becomes more visible to yourself so that you can try to correct for it later and visible to others, of course. And if you deviate, you can always argue why you thought it was useful to deviate, which is seems the least we can do as researchers. However, Daniel Larkin's disagrees and here argues that preregistration is useful in a context where you test predictions, where you test hypothesis. And therefore, if you don't study a hypothesis, you just do descriptive research and you just want to describe a bit of reality, Daniel Larkin's would argue you shouldn't use preregistration. My personal view aligns more with that of John McFeetries. But my point is that that doesn't matter because there are different views and again, like with the other, like with basically doing science in general, there is no one correct way. It's actually beneficial if people have different ideas about what should be in a preregistration form. And it's beneficial if we have a plethora of preregistration forms. So with the rationalist approach, that's the simplest. Just you or you and a group of colleagues sit together and develop a form. That's it. The benefits are that it's efficient and quick and you have full control of the degree of specificity. So if you have a certain study design that you use a lot with your lab, you can just create a form for that. That's fine. Drawbacks are of course that your procedure is not systematic and reproducible. And it might only be applicable to rare use cases. You can make it a bit more systematic by using expert consensus. Of course, you'd have to think about how you define expert, but that's a different problem. And then they basically just use the rationalist approach. They also just think about what should be in the form. The benefit is that you immediately have consensus with this group. And that helps with adopting the form. The drawbacks are that the process is often still not super well documented and the selection of the experts determines what you end up with. Then you can use the Delphi method, which is a more systematic version of this expert consensus. So it has the benefits of the expert consensus method, but it's more systematic. The nice thing is that the recruitment is often more formalized like emails are sent to listservs and stuff like that. But the selection remains central and still, of course, who you want to include determines what you end up with. And of course, you know, if there's no one correct way to do science, there's also no one correct way to determine what an expert is. Scoping reviews are probably the most transparent approach you can use. Not in the least place because you can pre-register your scoping reviews. And I don't think I have to explain it too much about this to this audience. So it's very systematic and reproducible, but you will always look to the past because you will base it on the existing literature. So you will look at the practices that people applied and then those will form the basis for your form. And of course, you can combine this in any way you want. That, of course, makes it more time-consuming and expensive, but also better. Okay, so imagine that you chose a way to get your form structure and item contents and stuff. And then you created the form or you want to reuse somebody else's form and now you're doing a preregisteration. So we just saw that we loaded a form specification directly from a spreadsheet, but now we'll use one, the inclusive systematic review form, that's included with pre-regger. So then you can load it very quickly and then pre-regger can tell you what the next item is to fill. In this case, we haven't actually completed anything yet. So the next item is just the first one, which is the target discipline. Now we can look at the content that has been specified in the meta-metadata section. We can see that actually nothing has been specified. And then we can specify the target discipline, the first item, using pre-regger, pre-reg specify. Basically, you pass the object that holds the preregisteration and then you use the item identifier and the content to specify what you want to add. And then you store the result again in an object, usually the same object. You can include validation rules in your preregisteration form specification. In this case, what I passed, passed the validation. And then it's stored. So now we can look at whether this helped. And it did help. So now here we have the psychology as a value. Like this, you can specify everything. Often I think you won't do this in the console, but you will want to do this in an R Markdown file. You can also knit the results immediately. And that's often what you will want to do with the R Markdown file as well, so that you can actually include the entire form contents. Okay, so that's how you initialize and fill a form. As I said, that's usually not what you want to do. Usually you'll want to write an R Markdown template. And for that, I'll show some examples from Nitro. Nitro is actually from a bit of a different R package that's not completely done yet, Meta before. Meta before is mostly intended to help you with extraction. It helps you specify how information should be extracted and then scalably and transparently extract data in a way that can be combined with data from other studies. This is accompanied by this Nitro, which is a narrated illustration of a review outline, which is an R Markdown file. And in this R Markdown file, there are already the commands to generate a preregistration template using the Inclusive Systematic Review template, which then writes it an R Markdown template to preregistration auto-generated, which you then rename to something else. In this case, we use preregistration.RMD, because of course, otherwise, if you accidentally run this again, you override your specified preregistration stuff. So if you run this command, this R Markdown file is created, which is created, which look like this. And this R Markdown file contains, that's the first chunk here, contains the entire form specification. That's all the way at the bottom, so that is not in the way. But it's loaded first. And then it initializes this form that has been defined in the R code at the bottom that's loaded at the top. Then here, it shows the information that was specified in the preregistration specification. For example, these are the instructions that are added in the instructions worksheet that I showed earlier. And then it has a sequence of chunks for every item. So basically, you can just fill those out. Here is the target discipline, so you can add the target discipline, you can add the title, you can add the alters, et cetera. And here are the name, the human readable name, the label, and the instructions for every item. So for tasks and roles, describe the expected tasks and roles, blah, blah, blah. So you can give this to students or to collaborators and they can just fill this out. This can be rendered, of course, to PDF, but it can also be rendered to HTML. And if you render it to HTML, or you include it in our markdown file, then it also includes the form and the, so the form specification and the data as JSON. So you can look at that at the preregistration, preregur package download website. You have this where you can see how you can import the preregistration or a preregistration form from embedded JSON from a URL. This is, for example, the vignette for creating a preregistration form, which is also here, that's this one. And that also includes an example of the created preregistration form, which includes the preregistration form specification as well as the data as JSON, which is imported here. And then here you can see that the content has been imported properly. These are the links that I promised earlier. I'll move on from this slide because this is a video. And that was actually it in terms of what I can do in terms of showing, showing preregar a bit and explaining a bit why it was created and the form itself. So I would say, go visit either preregar.openScience or the doi for the form, the inclusive systematic review form and then go play around with it. And if you have any questions or issues, you can mostly find me at Mestodon. I am technically still at Twitter, but you can also email me although Mestodon probably worked best. Thank you very much.