 So thanks again very much for joining us today for this webinar on the global flourishing study pre-registration and registered reports. My name is Leslie Markham, the program manager for this study at the Centre for Oaken Science and I'm joined by Teresa Stankov, the data manager and Mark Call, the product owner, who will provide details of the data and the registry respectively. And we also have Amanda Stoller from our communications team ready to help with any technical difficulties and to help with the questions in the Q&A. Please add any questions you may have to the Q&A. We'll try to answer some directly in the response box and we'll have plenty of time at the end of the presentation to answer more questions. And we'll also follow up with any that we can't get to today. The webinar recording will be sent to all people who are registered for this webinar along with some slides and the Q&A. This is the second part of an introduction to the Global Flourishing Study. If you didn't attend the first part, you can find the recording along with the slides and the Q&A on our website. And I encourage you to view that recording if you're intending to register for the access to the Global Flourishing Study data. For some reason I can't find my slides, there we go. So I'm just going to do a very quick overview. We did cover this in the first webinar so it's going to be very quick. But just as a reminder, this is a five-year longitudinal data collection on aspects of flourishing. Data is being collected in 22 countries and we will have over 200,000 participants. This is a big collaboration between Baylor University Institute for Studies of Religion, the Harvard Human Flourishing Program and Gallup Handling on the data collection. And we're also grateful for the number of funders that we have involved in this effort. Just also just to say right at the start, we only have Wave 1 sample data, which is the Sweden data set available at the moment. Please don't publish using this data because it has been perturbed and randomized. The Wave 1 global data will be available in the coming months once Gallup has wrapped up their collection in the field. There are three ways to access the data. You can prepare a registered report, which we'll talk about in this webinar. You can prepare a pre-registration or you can wait for the data to be public released approximately one year after the early access through registered reports and pre-registration. You can find a lot of information on our website, www.cos.io forward slash gfs. This is the website that's specifically providing access to data and tools for researchers. So please do take a look at that. So what are the benefits of pre-registration? When you pre-register your research, you're simply specifying your research plan in advance of your study and submitting it to a registry. Pre-registration separates planned or confirmatory hypothesis-generating research and unplanned exploratory hypothesis-generating research, and both are important. There are clear benefits of pre-registering your study. If you value reducing the impact of confirmation bias on your scientific claims, you want to prevent peer reviewers from being skeptical about your analysis choices and you want to increase the confidence researchers have in your claims. Confirmation bias describes the human tendency to seek evidence that supports rather than rejects existing beliefs, expectations or hypotheses, and this makes researchers more likely to make analysis choices that support their original claims. Analyzing secondary data can reinforce confirmation bias since it's tempting to focus on the areas that match existing beliefs and report positive results over impartial analysis of the data. Access to altered sample data and a rigorous analysis plan may help. In this picture, an archer with a bow contently looks at a wall with arrows, while an assistant is busy painting targets around the location where the arrows hit the wall. The problem is that the sharpshooter did not demonstrate accuracy. Hypothesis are only tested if they're made beforehand. Isn't it still possible that the sharpshooter is a good shot? It is indeed. The problem is that nothing has been done to challenge the claim that the sharpshooter is a good shot. They might be a good shot or they might not be, but based on the end result, i.e. a wall with some targets and arrows, we can't evaluate how good a shot they are. The same holds when testing scientific claims. If a claim was based on a hypothesis generating after looking at the data, the claim might be correct or not, but we can't evaluate how much confidence we should put in the claim. Another practice that some researchers admit to using is to perform many slightly different tests on the same dataset and then to selectively report only those that support their prediction. This practice is called selective reporting. By only reporting tests that support predictions and hiding tests that do not support predictions, researchers can make it look like there is clear support for a theory when this conclusion does not follow from all tests that are performed. Just because a researcher has selectively reported results does not mean their conclusion is wrong. However, it means a claim is more likely to be wrong even when it's supported by a statistical test. Researchers often have great flexibility while analysing their data. They can use this freedom to freely explore the data they have collected as long as they transparently report what they have done. Researchers can deviate from a planned analysis. Sometimes this is even an improvement and as they might learn based on the data that their original analysis plan was suboptimal. With a good justification, the new analysis plan might bring researchers closer to the truth than the original analysis plan. Evaluating whether an updated analysis is an improvement or opportunistic requires careful thought. So now let's turn to the benefits of registered reports. Registered reports aim to change the reward system for earning publication. In the standard model, researchers design their study, collect and analyse their data, write a paper and send it to a journal for peer review. Authors decide whether and when where to send their papers and editors and reviewers decide on the value of the research based on the results. Publication bias produces published literature looking more beautiful than reality. In step one of a registered report, authors submit a stage one registered report to a journal as a pre-registration in which they provide the motivation for the research question, preliminary evidence and the proposed methodology. They write down this study as a pre-registration. With registered reports, the initial phase of peer review moves to before the results are known. In step two, the pre-registration undergoes a stage one peer review. The data has not yet been analysed at that point. The editors and reviewers evaluate the submission with two considerations in mind. Firstly, is the question important? And secondly, is the proposed methodology an effective investigation of that question? The journal can decide to reject the submission, ask for amendments or provide an in-principle acceptance, which we just shortened to IPA. Acceptance means that if researchers follow the plan, their study will be published, regardless of whether the results are statistically positive or negative. After the in-principle acceptance, researchers collect or analyse the data following the pre-registered plan and resubmit a stage two registered report. Deviations are possible and should be clearly documented and if the deviations from the plan are substantial, the journal editor should be notified to prevent a rejection of the stage two manuscript. At the last step, the researchers submit the final manuscript for peer review. This is called stage two review. Now the manuscript includes the data and the analysis conclusions and the conclusions. It's not possible that the data is rejected after a negative result is observed except under rare circumstances where quality checks indicate that methodological problems led to an uninformative study. The editor and reviewers only assess whether authors followed the planned methodology and interpreted the results responsibly. The results themselves are not a basis for deciding whether to publish the paper or not. Registered reports have the benefit of moving peer review forward in the research process such that it is still possible to incorporate feedback from peers in the study design, improving any weaknesses that peers identify before data collection rather than after the fact. For the global flourishing study, you will have access to the sample data a few months before the full data set. You can use this time to develop your preregistration and submit it to a journal that accepts registered reports. In this figure from Chambers and Zabella, we see the rise of the number of journals and the spread of registered reports to different disciplines over time. Today, more than 300 journals have adopted registered reports. The figure shows wide adoption from the social sciences through the biological and medical sciences and economics. Since 2020, some notable advances include adoption by large multidisciplinary journals such as PLOS One and this year, Nature has adopted the format. You can find a list of journals that accept registered reports and may be applicable to this study on the GFS website. You can also find a full list of the 300-plus journals offering registered reports on the COS website. Don't worry about trying to get down all these links. We will be sharing the slides when we send out the recording of this webinar. Registered reports should eliminate publication bias that favors positive results over negative results and she and her colleagues compared registered reports and standard articles. For each paper, they identified the first hypothesis tested and coded whether the results supported the hypothesis or not. Standard reports almost always reported positive results for the hypothesis, 96%. When the authors and journal pre-committed the hypothesis and reported the result, just 44% were supported. So these are the ones that have done registered reports. So making the decision to publish prior to knowing the results reduced publication bias and ensured that the published results reflected what occurred rather than what emerged through selective reporting or questionable research practices. In this study, the authors gathered a sample of published registered reports that tested novel research questions and then identified two matched standard model papers for each registered report. One published by the same lead author or authors and the other published in the same journal on a similar topic at about the same point, time point. 353 researchers participated in a structured peer review process. They were matched by expertise to a pair of papers, one registered report and one of the matched control papers. We see all judgments are in the positive direction, i.e. a higher score from registered reports. The black horizontal lines indicate 80% of thick and 95% of thin credible intervals. So the more positive evaluation is distinguishable from random variation for the first four aspects of studies. And they are methods and rigour, methods rigour, quality of methods, amount learned from the study, quality of the research question and alignment between the question and method. Visit the peer community in or what we've shortened as PCIRR website to learn more about their initiative. PCIRR is a nonprofit that creates communities of researchers who review and recommend unpublished preprints in their field. This is a free service. Authors of registered reports that are positively recommended by the reviewers have the option to publish their articles in the growing list of PCIRR friendly journals that have committed to accepting PCIRR recommendations without further peer review. You can email PCIRR in advance when you plan to submit a registered report and they will prepare for your submission by already selecting the reviewers and asking the reviewers to block time in advance to quickly return reviews. Okay, I'm going to stop sharing my screen and I'm going to pass this over to Mark, who is going to walk us through the registry. Over to you, Mark. Thank you, Alice. Make sure that I'm sharing the correct screen. You should be seeing our GFS registry. Am I good to go, Leslie? Fantastic. View to myself. Yes. All right, so welcome everyone. This is the Google Flourishing Study Registry. This registry is built on the OSF Registry's infrastructure in a way where you're able to submit your study plans and able to gain access to the secondary data. And click Overview of the discovery page that you've seen right in front of you. If you are interested in a short little excerpt from the study, we have that at the very top. We also have some very helpful links such as being able to learn more about the study. You can go to the COS website, as well as the email address, as well as some documentation to help you submit your study design, your plans. We strongly recommend that you open up those different resources because they are going to be incredibly helpful and they will more than likely be used by us whenever you have kind of questions. They'll give you to the right answer, et cetera. So strongly recommend that you dig into that. Get familiar with it as you're coming up with your study designs. Now, how would you be able to submit your study design or be able to access or request access that data? To do that, first you need to create a OSF account and be able to log in, which I have already done, and go to the GFS Discover page. There is a way you can do that through OSF. I'm not going to go through that, but you can also just type in this link and it's also available in all of the other different materials. Now, to submit a request or proposal study design, et cetera, you will click Add New and you'll be asked these two questions. Do you have content that's already in an OSF project? I'll be happy to share your links. I'm not going to get into that for this session, but you do have the option to say no, completely brand new. I will add it as I go. But if you do have some of those materials in a project, you can click Yes and select it from there. And then the second question is what type of registration you want to create? For our purposes, this is very simple. The pre-registration for sample and wave 1 data and it's the only one that you can select. So super simple for that. Once you have all that set up, you will create a draft. And this is where you will start building out some of the information that is going to be imperative for, to be able to see if you're able to access that data. I'm not going to go through each individual item, but I'm going to touch on some of the key important aspects. First of all, it's going to be this registration metadata. Anything that is with a brand abstract is going to be required. And I strongly encourage that you take time on this page to make sure that everything is accurate and correct. Because once you submit it, most of the information that is available on this page cannot be changed. So once you think of a title, make sure it's a title that you and your researchers are comfortable with. Same thing with description. Do a high-level description of your study. Think almost like an abstract, but since you really don't need, you don't know the results, two or three sentences about what does your study entail. Also go through contributors. These are the people that are going to be affiliated with your registration or need access to it. Category, license, subjects, all this information is going to be imperative. And mainly because this helps other people discover your research. So once it is approved, once it is registered, this is how people are going to be able to locate your study. Each one of these things and possible recommendations of what to select is going to be and all those materials on that front discovery page. It's the reason I said, please look into them, explore them, get familiar with them, from a chat if you have two screens, have them open as you fill this out. Once you're done with the metadata, you will go through several different pieces of the workflow, these different sections. The main things you will be interacting with are text boxes where you can enter in your information. And these are expandable and collapsible so you can easily expand it to see everything that you need. Same thing with radio buttons. We also have checkboxes and also options to select different files. So if there is a file that you already have all this information in such as one of the questions is a manuscript, all you have to do is click this little plus button and pop up or show one up. You can either upload it and be able to enter it through that kind of away. Again, I'm not going to go through any of these. The only thing that I would also say is anything that you see in red means that there is missing information. So we strongly recommend that you go through it and see what have I not filled on it. If things that are completely filled in or optional, you would have a green lettering with a checkbox saying everything that is basic means is filled in. Now, this is just saying that there is some type of input. Does not mean that it's accurate. It does not mean that it is changing to make sure that the text and the spell checker is correct, all that stuff that you have to go on in. But it's just saying that there's something in that box. You will go through the rest of these. At the very end, you will have data request agreements. It's important that you read through these and to make sure that you discuss with your team and click I agree. Some of these are required. I think all of these are required actually. Yes, all of these are going to be required with your response and the moderation team is going to look for these to make sure that everything is filled out appropriately. Once you are done, you can review all the information. As you can see, most of the amount is blank. But it's another way for you to go through all the information that you currently have and make sure that nothing is out of place, not anything is incorrect. It gives you an opportunity to go through it and double check all the answers. A few other things to highlight is as you fill these out, the system does auto save every couple of seconds and it will also save every time you go to a new section. And once everything is filled out, you will have the option to register. Once you click register, a couple of things will happen. One is you will have the opportunity to say if you want your registration to be public, meaning that once it's accepted, anyone can see it or embargoed, which is a fancy term to say that it will be private for up to four years. Once that is done and you select that, you can select register again and an email will be sent out to all the different contributors that are associated with that project. If they have admin permissions, which you will set by contributors right through here, they will have the opportunity to review it and decide to say, yes, I approve it or no, I reject it because it needs additional edits. They need to reject it within 48 hours. After that, the system thinks that everything is good to go and it will automatically submit it to the moderation team where they will review your responses and to say, yes, we approve this and be able to give you access to that secondary data. Or they will say we did not approve it because we're missing this information or we don't understand this. So that is the quick little workflow. There is also another resource that I want to provide you guys, which is our help guides. Our help guides have a lot of information in them and a lot of step by steps and how you can do these different types of workflows and how to enter certain types of information. So I strongly recommend that you also be able to use this. If you have any questions, you're more than welcome to reach out to us and email us at contact at OSF.io. And so with that, I will stop sharing my screen and send it back over to, I think it's Teresa's turn. It is Teresa's turn, but I'm going to share my screen again and Teresa's slides. So just bear with me one moment. Just goes back into slide show. Okay, over to you, Teresa. All right, thank you. I'm going to be focusing on the global flourishing data set and some of the open data practices we're using for both the sample data and also the final data. So next slide, please. Yeah, just as a reminder, the purpose of the sample data set we're providing is to allow you to start to plan your analysis before the real one wave one data is released. So this sample data consists of just the wave one data from Sweden that we have made various transformations to such that it is informative, but it doesn't contain the real values. So it is very important to understand that this is not the real wave one data and is therefore unsuitable for analysis. And there are those, again, those two sample data sets available, one for researchers who submit a pre-registration for their study that contains data that has been perturbed from its original values and one available for the global flourishing website that is both perturbed and randomized. Next slide. So we want to encourage pre-registration and registered reports as much as possible. So we have for the final data release, we have two different release dates for the full data in each wave, depending on whether or not a pre-registration or registered report was submitted for a study. So for projects that have submitted a pre-registration, the full data will be available as soon as it's released. So we'll get the data, we'll get it ready, we'll put it on the website and it'll immediately be available to those researchers who have put together one of these pre-registration plans or registered reports. For anyone not submitting a pre-registration, the data will become freely available to anyone regardless of if they've done a pre-registration or whatever, one year after the data is made available for those with a pre-registration. So again, just to reiterate, if you do a pre-registration, you'll get it immediately. If you don't submit a pre-registration, it'll be freely available one year later. Next slide. So for all data releases, data will be shared in CSV format. So CSV is simple, machine readable, and broadly reusable data file format, and it doesn't require any special software to open or use. So since it's not tied to any particular software, you don't need to purchase any special anything to use it or worry about issues opening the file in the future due to software updates. So that's why we preferentially use these non-proprietary data file formats. That being said, we will also share the data either as an SPSS file, which is the format that we receive it in from Gallup, or an RDS file for convenience. So these are rich data formats that allow us to attach some of the metadata, such as variable and value labels, to the data directly. So this is available purely for convenience to prevent kind of issues between translating stuff from the codebook into the data. But the sample data, or for both the datasets, the sample and the real full data, there will be the CSV version with accompanying metadata. So the sample data that we have here is currently available in both CSV format and as an RDS file. So that's what's available now. Next slide. So rich data formats are not necessarily accessible to everyone and don't contain all the metadata relevant to the dataset. So we're also providing separate metadata files. So for the sample data release, we have put out a simple code book, which is available now. And it contains all of the questions, variable labels, value labels, some information on if the question was asked in the intake survey or as a part of the annual survey. And that information is available for not just the Sweden dataset, but for all of the countries. So there are some questions that were asked in countries that weren't Sweden. And some questions had answers that weren't relevant to Sweden. So primarily think questions about political parties that are relevant to some countries, but not others. All that information could be found in the code book, which is online now. For the full data release, we're intending on releasing an XML metadata file using the data documentation or initiative or DDI standard for metadata, which is designed to accommodate survey data. Next slide. So again, just to go over the the code book for that's available right now for the sample data that anyone can see and use to help create some of these analysis plans. It's available in a spreadsheet with one of the tabs describing information about the data. So information about what variables are available, some more descriptive variable labels, question text in English, etc. As well as a tab that is the code book for the code book describing what each of those columns actually describe. And so this will help you interpret that data in the CSV format and provides a little bit more context than what's available in those RDS files that we have also provided. So I think that wraps up my section. Thank you. Thank you very much, Theresa. That's great. So we have some opportunities if you'd like to participate in some research with us. So the Centre for Open Science is expanding the knowledge base on the effects of registered reports and pre-registration with a randomized control trial and a survey in the Global Flourishing Study. The study will measure effects such as the impact of registered reports on timelines from conception to publication, publication outcomes, in other words, which studies are ultimately published or rejected, and researchers' beliefs and experiences with registered reports and pre-registration. So this is the participant workflow. There are four arms so participants can agree to be randomly assigned a GFS data access pathway, which would be either pre-registration or registered reports, and be surveyed on their experience, or they can opt in to be surveyed on their experience with either mechanism after they've completed their GFS registered report or pre-registration submission. Anyone can enroll to be randomly assigned a pathway prior to uploading a submission to the GFS registry, and the main outcomes will be comparing the two randomized participant arms while the self-selected participants will help us better understand selection processes. Pre-registration and registered reports could be potentially transformative to how research is conducted, and so there are great examples of people we would love to encourage to enroll, and these groups are researchers who want early access to the GFS data set, but are undecided on whether to do a pre-registration or a registered report, researchers who want to try out a registered report but aren't quite sure how to start, and researchers who are willing to be part of a growing evidence collection on pre-commitment devices like registered reports and pre-registration. There is a lot more information on the COS website and there is an enrolment button there, and again you will be able to get that when we send these slides out. So how do you get some help? So you can read more about the materials presented here today by accessing our full suite of training resources for the Global Flourishing Study on the website, so I encourage you to do that. In terms of the pre-registration that Mark went through, we have a guide to pre-registration. We have a pre-registration template which looks exactly the same as the one that he demonstrated on the registry, but this is a downloadable Google Dock, so you can collaborate on that, get your pre-registration right and then just cut and paste it into the online version if that's a way that you would prefer to work. The code book as Teresa has just explained, and we also have a lot of FAQs on the website as well. I'm sorry, I just lost my space there. Okay. So you can also email me at globalflourishing.io with any questions you have, and I can point you in the right direction. Mark is very happy to have one-on-one calls with anyone who wants to walk through entering your pre-registration. This is great for you because if you have any in the moment questions, he can help you with that, but it's also great for us because we can then identify whether any issues or something that's unclear and we can go back and amend the workflow to make it easier in the future for all of you. Please join our email list to receive updates. Given that you've actually signed up for this webinar, you will get updates, but if you know other people who want to receive information as well, please join that email list. You can find the link on the website. The other key thing is please go and visit the research website. There are five tabs there, all filled with lots of information.