 Hi everyone, I am Mega. I'm a quantitative researcher at the American Institutes for Research. Today I'm doing a tutorial on my wild meta package written with James Busevsky and Pierce Cappelli. The package implements claustrophobic strapping to handle dependent effect sizes in meta-analysis. So typical meta-analytic techniques like meta regression assumes that effect sizes are independent. However, dependent effect sizes are very common in meta-analysis. Primary analysis can yield more than one effect size or effect sizes can be nested in some way, creating dependence. For example, there's this meta-analytic data in James' club sandwich package that contains effects of SAT coaching on student achievement. The data has 47 studies and 67 effect sizes with some primary studies yielding more than one effect size. There are effects from the same study for math and verbal sections of the test. So here's the example data and we have the study, the effect size, the variance of the effect size, the type of test and the number of hours study. And we see some studies, that's the same study with two effect sizes, one for verbal and one for the math sections of the test. So one way to handle dependence is to ignore dependence. But if we ignore dependence, we get incorrect standard errors and therefore incorrect inferences from hypothesis tests. The standard method to handle dependence is to use robust variance estimation with small sample corrections. But in our paper in Joshi Posayevsky-Bretva's 2022, we showed that such corrections can lead to low power, especially for multiple contrast hypothesis tests. An example of multiple contrast hypothesis tests is to what extent to do effects of SAT coaching vary across math and verbal sections of the test. So in our paper, we examine an alternative method called cluster wild bootstrapping. Regular bootstrapping involves estimating unknown quantities by resampling from the original data many times. Cluster wild bootstrapping in particular involves resampling residuals by multiplying them by random cluster level weights. In our paper, we showed that cluster wild bootstrapping maintains type one error rates adequately and has more power than RVE with small sample corrections. So this is the algorithm for cluster wild bootstrapping. First, we fit a known model, which excludes the variable that you want to test in a full model, which includes the whole meta regression model on the original data. We then obtain residuals from the known model. Then we generate an auxiliary random variable that has a mean of zero and variance of one and multiply the residuals by the random variable set to be constant within clusters. We then obtain new outcome scores by adding the transform residuals to the predicted values from the known model fit on the original data. We then reestimate the full model with the new calculated outcome scores and obtain the test statistics. And then we repeat steps three to five R times and R is the number of bootstrap replicates. And then we calculate the p-value as the number of bootstrapped F statistics that are greater than the original F statistics. This is a lot, but you don't have to program this yourself because Wild Meta does this for you. And here's our wild hex sticker designed by my friend Riz. So I'm going to go over Wild Meta now. And the main function in the package is called Wild Test CWB. And it works with meta regression model fit using the Ruby Meta and Meta4 packages. And for Meta4, it works with both RMA-MV and RMA-UNI models. The function takes in a full model, a meta regression model fit using Ruby Meta or Meta4, constraints, number of bootstraps, and some other information. And most users won't have to mess with these this information. And then we also have the seed argument for reproducibility. And now I'm going to demonstrate how to use the function with this example data set that I went over before, the SAT coaching data set with data set with dependent back sizes and for math and verbal sections of SAT tests. So first I fit a meta regression model using the Ruby Meta package. Here I fit a meta regression model with zero intercept and with the test and ours as moderators. And then I specify the study of the variance of the back size and the data. And here are the results from the Ruby Meta model. The estimates for math and verbal are the average effect for each test type because we fit a zero intercept model. And we want to test whether these two effects differ from each other. Now here I use the wall test CWB function from Wild Meta. I specify the full model as a Ruby model. I use the constraint equal function from a cop sandwich package to set up a constraint matrix. I set the number of bootstraps to 999 and I set the seed for reproducibility. And the results indicate that the effects likely don't differ between math and verbal sections of the SAT test. Here I fit a model using the RMA-MV function for Meta for package, similar model and I input the RMA model in the Walt CWB function and I get similar results for the Meta for model. So the new version of Wild Meta supports parallel processing to make bitstrapping a bit faster. We use the future package to support parallel processing. And all you have to do is run this code to get cluster set up on your computer network and then run Walt test CWB. And here I'm running the function with 1,999 bootstraps and it took approximately 40 seconds to run and here the results which is similar story from earlier results. So I used different number of bootstraps 999 and 1999. So how many should you use in application? Generally the higher the number of bootstraps the better precision and power. But also if the number of bootstraps are higher it's going to be computationally intensive and it'll take more time. Generally 1,999 or higher number of bootstraps is recommended. But please see this paper Davidson and McKinnon 2000 for guidance on the number of bootstraps to use. Thank you so much for listening. Here are some links. The first one is for the Wild Meta package documentation and vignette and all kinds of examples. And the second one is for my website where you can find my contact information. Thank you so much. Thank you.