 Hi, my name is James Prztyewski from the University of Wisconsin-Madison. Today, I'll be giving you a short intro to how you can synthesize dependent effect sizes using a combination of the R packages metaphor and club sandwich. This talk is drawn from joint work with my colleague Beth Tipton from Northwestern University and you can get our working paper on the topic from the link listed in title slide. If you're conducting a research synthesis, it is exceedingly common to encounter studies that provide multiple dependent effect size estimates. This might come from having multiple outcomes measured on the common set of participants, from having one outcome that's measured at multiple follow-up times, a study with multiple treatment conditions compared to a common control condition, or if you're interested in using multiple correlations that are all drawn from a common sample. If you've ever extracted data or if you've been involved in the analysis of a large-scale systematic review, you've probably encountered scenarios like this. It's easy enough to extract data to get these multiple effect sizes from your source studies. But the problem is that it's often difficult to get information about the extent of the dependence between effect sizes, such as the degree of correlation between multiple outcomes measured on the same set of participants. Not having that information makes it more challenging to do a defensible synthesis. One increasingly popular approach for handling dependent effect sizes is called robust variance estimation, or RVE. RVE uses formulas called sandwich estimators to construct standard errors, hypothesis tests, and confidence intervals that are robust to the modeling assumptions you make about the extent of dependence between effect sizes. In other words, you can get standard errors and confidence intervals that still work even if certain of your modeling assumptions are mis-specified. It's kind of like buying extra insurance in case your model is flawed in some ways. RVE involves what is called a working model that approximates the dependence structure of your effect sizes. It's called a working model because it doesn't have to be correct. That's the whole point of using RVE in the first place. But using a working model that's close to the truth, that approximates the true structure of the dependence in the effect sizes, helps because it gives you more precise estimates of average effect sizes or meta-regression coefficients if you're conducting meta-regression. The most popular implementation of RVE is from the RoboMeta package for RVE. The package has two built-in working models, but as I'll explain, both of these working models are limited and a bit inflexible. The default working model is called the correlated effects model. It allows for there to be correlation between effect size estimates drawn from the same study. But it only includes a single random effect per study. So that amounts to assuming that all of the heterogeneity in the effect sizes exists between studies, that there's no within-study heterogeneity except for what's due to sampling it. Another available working model is called the hierarchical effects model. It does allow for there to be within-study heterogeneity, but it doesn't allow for there to be correlation between the effect size estimates. My main message today is that you don't actually have to choose between two rigid working models, neither of which might capture all of the important features of your data. A different strategy involves estimating a working model using the very popular and very flexible metaphor package. The package includes a function called rma.mv, and it provides a very versatile set of models, including multilevel models and multivariate models that can capture all sorts of nuance in your data. I'll demonstrate that you can treat these models as working models and combine them with RVE. One leading candidate for a working model combines both the features of the correlated effects model and the hierarchical effects model. Also we'll call it the correlated and hierarchical effects model. In the paper we call it CHE. This model allows for correlation between effect size estimates. It also allows for within-study heterogeneity in the true effects. So we don't have to choose or assume away some of these features. We can use a working model that we think captures the major aspects of our data. Now if you estimate this model in metaphor, the standard errors and tests and confidence evals that come out will be based on all of the modeling assumptions. So if you get some of those modeling assumptions wrong, the results will be wrong potentially misguided. But we can combine this working model with robust variance estimation. There's a package that I wrote called club sandwich, which also provides an implementation of robust variance estimation. It gives you robust standard errors, hypothesis tests, confidence evals for many different types of models, including RMA.MV models estimated using metaphor. You just feed in the model, you get out robust standard errors and confidence evals. And one advantage of using the club sandwich package for RVE is that it includes built-in small sample corrections so that you can trust that the results will remain accurate even if you're working with a fairly small set of studies. Here's a very short illustration of what this workflow looks like. It involves really three steps. After reading in the data, you first have to create a sampling variance covariance matrix that describes tentative assumptions about how much correlation there is between the effect size estimates from a given sample. Once you've got that sampling variance covariance matrix, you can feed it in and fit a working model using the RMA.MV function from metaphor. I store that estimated model in an object called mod, and then you can feed that into the conf int function from club sandwich, conf int for confidence interval, and this will spit out clustered standard errors and confidence intervals. The package also includes functions for running hypothesis tests, constraints on your model, and so forth. So to sum up, why would I recommend that you consider moving to this metaphor plus club sandwich approach? First, a more flexible model that better approximates the real dependence structure of your data will give you more precise estimates of average effects or meta regression coefficients. In simulation work, we found that there are some circumstances where the precision gains can really be quite substantial, like standard errors that are half the size of what you get from the standard approach with RoboMeta. A second advantage is that using more flexible working models gives you better descriptions of heterogeneity, such as estimates of both the between study variation and within study variation, and that's useful diagnostically, and it's also potentially interesting in its own right. Rather than just assuming a way within study heterogeneity, you can use this metaphor working model to assess and investigate it, at least tentatively. Then finally, combining metaphor with robust variance estimation provides protection against model mis-specification. So you don't have to stake all of your inferences on having identified a perfect model. More details about all of this, an extensive example, code demos, simulation evidence are all available in the working paper that I've linked to here. Thanks very much and feel free to reach out with questions.