 Hello and welcome to my talk what if a very short primer on conducting multi-verse meta analysis in R. A multi-verse meta analysis is nothing more than conducting all possible meta-analyses for a given research question. What that exactly means will be covered later. Finally we will look at why one would even want to go through the hassle of conducting all possible meta-analyses. So why one would want to conduct a multi-verse meta-analyses. Then we will talk about the basic idea behind multi-verse meta-analyses and all the plots and all the analyses you can see here are prepared on my Github repository. So you can calculate and plot everything you see here. And then I will show you plots a lot of plots. So now let's have a talk about the motivation behind conducting a multi-verse meta-analyses. Imagine a scenario where there are multiple meta-analyses on the same research question but with diverging results. We have meta-analysis A that actually does not find an effect. On the research question of interest meta-analysis B is inconclusive and meta-analysis C actually finds an effect. So you might suspect different reasons for these diverging results. For example the individual meta-analyses might have used different methods. Fixed effect, random effects, multi-level modeling. So this might be the reason for those diverging results. Or they might have used different criteria for removing outliers. They did not remove any outliers. Or they used a specific cutoff, a fuzzy cutoff and so on. Or they might have used different inclusion criteria. And the list can go on and on and on. But it is really important to understand why those different diverging results emerge and this can be very tedious and also frustrating. This is where multi-verse meta-analyses come into play because they are perfectly suited if you have the same research question but different summary results. The basic idea and origin of multi-verse meta-analyses began with this paper which data to meta-analyze and how by Martin Voracek and colleagues. The question you should ask yourself when conducting a multi-verse meta-analysis is actually the title of the paper. You have to think about which data, which subsets of studies are eligible for your multi-verse meta-analysis and also how you could analyze those. So all the different meta-analytical models that could be used and are reasonable should be defined beforehand. If you want to dive deeper into this topic, I can recommend all of those three papers where you get the background that is needed to fully grasp multi-verse meta-analyses. So which meta-analysis or meta-analyses should I compute? Multi-verse meta-analyses would suggest why not all of them. So the multi-verse of meta-analyses for a given research question, the multi-verse meta-analysis, is basically the combination of a all possible study subsets and b all available statistical and meta-analytical methods. Let me break this down for you a little further. First we have to decide on the which factors that is which data or study subsets should we analyze. Imagine we have a research question how effective are psychological interventions for individuals with depressive symptoms. Here we could decide to include studies based on age groups, different sexes or different therapies. And this could lead to different paths. For example, we could include only adults and only male participants and only studies that investigated therapy B. And you can also see that there are a lot of different resulting study subsets that could be analyzed. In our simulated example there are 36 subsets of studies that could be analyzed in such a way. Secondly, we have to decide on the how factors that is which statistical and meta-analytical methods could we use to analyze our data. And here again we could have three different factors that are interesting to us. The first one how we handle effect size dependency. So when we have multiple effect sizes per study, what are we going to do with this? Are we going to keep all of those effect sizes or are we going to keep only one effect size per study based on some criterion or are we going to average those effect sizes? Those can all be valid methods but lead to different paths that could be taken. We could also handle outliers in a different way. We could either remove them or we could keep them. And then there are different meta-analytical methods that we could use. For example, we could when we kept all effect sizes, we could use a three-level model or robust variance estimation. If we only kept one effect size per study, we could use P-uniform selection models or if we averaged our effect size, we could use a random effect model. And there are many, many more but in our example, we have 36 different methods that could be used. And in total, when we multiply our which factors with our how factors, we have a lot of potential meta-analyses that could be run. And now, of course, it is important what are we going to do when we have so much information and so many meta-analyses, then we have to visualize them. And to do this, we can either use descriptive specification curve plots or inferential specification curve plots. Let's have a closer look. Here you can see our descriptive specification curve plot with simulated data. Let me walk you through this in more detail. We have our upper panel with the descriptive specification curve and our lower panel with the which and how factor combinations. Each of those vertical lines represents the confidence interval of a single meta-analysis. Down here on the x-axis, you can see that in total, we have 160 meta-analyses included just in this one graph in this one picture. On the y-axis, you can see our summary effect size, in this case, Hedges G. And this black line represents the effect size estimates ordered by magnitude from our smallest effect size to our largest effect size. Those colors represent the amount of included primary studies, warmer colors include more studies and cooler colors include less studies. You can also see a black dotted line representing the null effect. So all the confidence intervals that cross this line are statistically not significant and a red dotted line representing our smallest effect size of interest. And now you can investigate why meta-analysis A and B and C diverged. We can have a closer look at the which and how factor combinations that are the reason or could be the reason for different results. Our meta-analysis A included all age group, only male participants and all therapy types. They did not remove any outliers and they used p-uniform to analyze the data and estimate the effect. And meta-analysis C, they made some different choices, they took different paths, but maybe even more importantly than pinpointing why single studies that worked is we can look at the overall picture from a bird's eye perspective and actually identify patterns. So for instance we can see that female participants produced much larger effect size estimates than male participants. Therapy B produced much smaller effect size estimates than therapy A or therapy C. We can also see that removing outliers leads to smaller effect size estimates and overall we can have a very nice look at the robustness and overall evidence based on our which and how factor combinations. And here you can see the same plot but simulated under a null effect. So all those confidence intervals actually cross the null line. So those studies or those meta-analysis would not be statistically significant and report quite small effect sizes but here you have some outliers that are quite large in comparison. So it can be helpful to look at the descriptive specification curve to see how the overall evidence looks. Here is an example from an ongoing research project where we actually plot over 5000 meta-analyses in a single plot to answer questions that are relevant for psychotherapy research. So it becomes quite overwhelming but this plot helps us a lot in finding patterns and understanding how robust our evidence actually is. Here you can see an inferential specification curve plot with simulated data and I simulated a real effect so you can see that our red descriptive specification curve plot is different from a null scenario which is represented by this gray line. To accomplish this gray line I simulated data under a null effect and did some bootstrapping to get the 95% confidence intervals and this plot is not in this area so we could be pretty sure that the effect is different from a null effect. But in this example I simulated a null effect so here we already know that there's no true effect and in this case the descriptive specification curve is in our gray area. I highly recommend pre-registering or publishing a protocol for your multiverse meta-analysis because it is quite funny that when you want to look at flexibility in data analysis you can also fall victim to flexibility in data analysis so it is a good idea to be very clear upfront what you are going to investigate and how you plan to do so. Here you can see some readings that I suggest you already saw those two papers but Julia Rohrer wrote a very nice blog post on dangers and pitfalls of multiverse analysis in general and I can recommend this a lot. Thank you so much for your attention if you are interested in the slides you can find them at my open science framework profile if you're interested in the code for all the plots you have seen you can find it on my Github repository and if you are interested more generally and related topics follow me on Twitter where I post on a regular basis. Thank you bye