 Hello and welcome to my tutorial where I'll be telling you about some tools, namely our packages and our Shiny apps that you can use to account for within-study and across-study biases when conducting a meta-analysis. All of this work is joined with Maya, and both of us are at the Quantitative Sciences Unit at Stanford University. MetaBias.io is the central hub for this project. This is where you can find all of the hosted Shiny apps and links to all of the packages and their documentation. So why do we need the tools that I'll be talking about today? The results for meta-analyses can be biased because of statistical bias in the underlying studies that they're using. These biases can include publication bias, where studies that are statistically significant are more likely to be published than studies with non-significant results. P-hacking, where researchers may manipulate their data or their methods to obtain statistically significant results, and multiple biases operating together, such as when publication bias interacts with the internal bias to studies, such as unmeasured confounding. Many varying methods exist to correct for these biases, and I don't have time to give a detailed overview of them, but what I'm going to be describing is a complementary approach of conducting a straightforward sensitivity analysis. These analyses let you evaluate how robust is a given meta-analysis to different levels of these biases. For each of these three types of biases, we have developed an R package for conducting these analysis analyses in a corresponding Shiny app where you can do the same analyses interactively. So for each of the three, I'll be showing you some examples of function calls using this package and then showing a demo of using the Shiny app. And note that each of the calls that I'll show is just a representative example. Each function in each package has many options that you may wish to explore when conducting your own analysis. First of all, let's look at publication bias. Here's an example meta-analysis that we'll look at. This meta-analysis examined whether playing violent video games is associated with increased aggressive behavior. As you can see here in green, the meta-analytic estimate of this association was 0.16. But the potential impact of publication bias on these results has been highly debated. Here's a sensitivity analysis approach to the issue. So we'll consider results to be affirmative if they're significant and positive, and results to be non-affirmative if they're non-significant or negative. And we consider a model of publication bias where there's some fixed selection ratio, and affirmative results are more likely to be published than non-affirmative results by this unknown ratio. Then given the actual observed affirmative and non-affirmative studies in the meta-analysis, we can ask how severe would publication bias have to be, i.e., how large would the selection ratio need to be, for the meta-analytic effect to be explained in a way, i.e., for the observed point estimate to be shifted to the null. The larger this value, the more robust the meta-analysis is to publication bias. Let's look at this sensitivity analysis in action. In the publication bias R package, you can use the function PubBiasSValue to find the value we were talking about, the selection ratio that would need to be present to attenuate the estimates of the null. All you need to pass to it are the y-values, the estimates from your meta-analysis, and the v-values, the variances from your meta-analysis. In this case, passing the data from the video game and aggression study, there's actually no amount of publication bias that could entirely explain away the point estimate, nor shift the confidence interval to include the null. Something we can do is change the value to which we are shifting. Instead of shifting to the null, we can find the s-value for shifting the point estimate to, for example, 0.1. Here we see that to shift the point estimate to 0.1, the selection ratio would need to be around 30. Affirmative studies would need to be around 30 times more likely to be published than non-affirmative studies. And to shift the confidence interval to 0.1, the selection ratio would need to be around 5. Under the same model of publication bias, we can also look at the complementary analysis of taking some fixed selection ratio, let's say 10, and seeing how much the estimate would be changed if that selection ratio were the case. So here for selection ratio of 10, the point estimate would decrease from the original 0.16 to 0.11. In the worst case, if affirmative studies were infinitely more likely to be published than non-affirmative studies, the estimate decreases to 0.09. Now I'll show you how to do a parallel analysis in the corresponding Shiny app. When you load the app, you'll see a place to browse to upload your own meta-analysis data. This needs to be a CSV file where each row corresponds to one estimate and there's a column for estimates and a column for variances. We'll use this built-in example for demonstration purposes. The inputs that appear at the top auto-populate and let you select which column in your data has the point estimates, which column in your data has the variances. If you need to switch the favored direction, assuming that publication bias favors negative effects rather than positive ones, you can change the direction and there's multiple different model types you can fit. You can fit robust random effects or fixed effects models and if you're fitting a robust random effects model, you can optionally add a column of clusters. At the bottom, you have your outputs. The left panel shows the corrected meta-analysis. So you pick what selection ratio you want to examine and for that selection ratio, you see what the corrected estimate and what the worst-case estimate would be. For example, if we set it to 10, like we were in the code, we see the same corrected estimate of 0.11. In the middle is the main sensitivity analysis, the S value, where you see how severe the publication bias would need to be to explain away the analysis. So as we saw, for shifting effect to the null, there's no publication bias that will achieve that. But if we set it to say 0.1, then we'll have 30-fold publication bias shifting the point estimate to 0.1. And finally, on the right, we have a modified funnel plot where you can visualize the distribution of affirmative and non-affirmative studies and you can compare the uncorrected pooled estimate to the worst-case estimate. Next, let's turn to p-hacking. The example meta-analysis we're going to use looked at studies of money priming, which investigate whether exposure to money-related stimuli can affect people's thoughts, feelings, motivations, and behaviors. Among all of the studies in the meta-analysis, the estimated effect was a hedge's G of 0.25. But on the subset of studies that had been pre-registered, the effect was just 0.02. So this difference raises suspicions that maybe non-pre-registered studies could have been subject to p-hacking. What is p-hacking? P-hacking can be thought of as manipulating results within studies to attain statistical significance. For example, through fitting multiple models, various other researcher degrees of freedom. P-hacking is often talked about interchangeably with publication bias, but there's important distinctions in how they operate and what the implications of those are. For illustration purposes, let's say that this is a set of p-values, each coming out of one study. The effect of publication bias is to filter which of these studies' results are published. So for example, for maximal publication bias, here we can say that the studies with p-values below 0.05 get published. We can think of p-hacking as a process where researchers iteratively obtain p-values using whatever manipulations they're doing, and finally arrive at a p-value that is then published or non-published filtered through publication bias. The resulting distribution of p-values is going to be biased very differently than the distribution from just publication bias. How do we use the actual distributions observed in meta-analysis to recover some sort of correction? We can get an intuition by thinking about the cases under which a study can get published. So we'll say that a study is hacked if it's biased due to p-hacking and unhacked if it isn't, and look at studies that are affirmative and non-affirmative. Let's assume stringent selection, meaning that hackers will always hack until they obtain an affirmative result or give up and withhold a result from publication. In this case, published affirmative studies can be hacked or unhacked, and their distribution is biased in a complex way and depends on the details of hacking mechanisms. But the published non-affirmative results can only come from unhacked studies leaving their distribution pristine. And critically, this is true regardless of what hackers might do specifically to obtain affirmative results. So whatever the distribution of p-values or z-scores of all studies is, the published non-affirmative studies follow a tractable distribution, a right-truncated normal distribution. We can get consistent meta-analytic estimates by modeling them as such and fitting a right-truncated meta-analysis just to publish non-affirmative studies. You might wonder what if there's less stringent selection and some non-affirmative studies do include hacked studies? And this could happen if some hackers give up before obtaining an affirmative result and end up publishing a non-affirmative result. It turns out that published non-affirmatives from hacked studies are usually downward biased, and so while the right-truncated meta-analysis is mis-specified, it will generally be conservative. Using the p-hacking package, it's straightforward to fit this right-truncated meta-analysis and see the results of the meta-analysis corrected for potential p-hacking. So you pass p-hacking meta just the estimates and variances, and you get estimates of both the mean, mu, and heterogeneity tau of the underlying distribution. And so we can see that in this example the estimate corrected for potential p-hacking is quite close to the estimate among only pre-registered studies, far smaller than the estimate among all studies. As before, I'm now going to show you how to do a similar analysis in the corresponding Shiny app. Like before, you have a place here to upload your own meta-analysis data or look at an example data set. And note that under the hood, this model is fit in the STAN probabilistic programming language because it's fitting this custom right-truncated meta-analysis model, and so it can take a bit longer to fit. Just as in the publication bias app, you can select which columns in your data have your estimates and your variances, and you can switch the favored direction for publication bias if needed. The left output panel has the analysis we were just looking at in code with the uncorrected estimate, the corrected estimate, and the worst case estimate. The other two panels contain some diagnostic plots. The center one is a QQ plot for the published non-affirmative studies, and these should cluster closely around the 45-degree line if the assumptions of right-truncated meta-analysis are not violated. And on the right, there's a density plot of the Z-scores of all published estimates, and this helps you see where Z-scores might be disproportionately concentrated. The implications of that are somewhat complicated, and you can read more about them on the app. Finally, let's look at multiple biases operating in concert. So in addition to a cross-study biases like publication bias, meta-analysis can be compromised by within-study biases, for example, unmeasured confounding and non-randomized studies. Critically, these biases often operate non-additively. For example, because publication bias that favors affirmative results selects indirectly for studies with more internal bias. So the sensitivity analysis builds on the method for publication bias by additionally considering distribution of internal biases across studies. This lets us ask a parallel question of, for a given severity of publication bias, so for a certain selection ratio, how severe would internal bias have to be, on average, across studies, to shift the point estimate to the null or to some other value? The multi-biased meta-package lets you conduct this analysis. So saying for a given selection ratio, for example, 2, this is the bias that would have to be present for the estimate and the confidence interval bound to shift to 0, respectively. Or if we change the selection ratio to 10, the amount of bias that would be required to shift the point estimate or CI bound to 0 changes. There's not a shiny app for doing this analysis soon, but it's on its way. It'll be available coming soon. To sum up, this tutorial has been about analyses that let you evaluate the credibility of a meta-analysis by determining how robust is it to a cross-study and within-study biases and their interaction. You can use the three R packages that I've talked about today to do these analyses, all three packages are available both on CRAN and on GitHub. And you can go to metabias.io to conduct all these analyses on the web through the shiny apps. Feel free to email me with any follow-up questions. Thank you for listening.