 Hi, all thanks very much for watching this presentation and for attending this conference. I'm mega and I work as a quantitative researcher at the American Institute for Research. And I'll be presenting on my package while meta, which implements cluster while bootstrapping for men analysis. This package is co-authored with my advisor James Pacific scheme. And we also have a wild hex logo designed by my talented friend rice. The package provides functions to handle dependent effect sizes and men analysis. Typical meta analytic techniques like meta aggression involves the assumption that exercises are independent. However, in social science and education research is common for each primary study to yield more than one effect size for for studies to be nested in some way, creating dependence. For example, Tanner Smith and lipstick 2015 is a meta analysis, examining the effects of brief alcohol interventions. The men analysis consisted of 185 studies. And in those studies 100 1446 effect sizes. The meta analysis included primary studies which had multiple correlated outcome measures, for example, the alcohol consumption outcome was measured by frequency of consumption quantity consume and blood alcohol concentration. The studies also included repeated repeated measures of the outcome and multiple comparison groups creating sort of correlated data structure related effects data structure. There are three ways to handle dependence. One is to ignore dependence, but doing so can result in incorrect standard errors and incorrect inferences from hypothesis tests. Some ad hoc methods include selecting one of excise randomly per study or analyzing subsets of data separately. However, such methods result in loss of information. The ideal way to handle dependence is to use multivariate models, but to do so requires information on covariance or correlations between effect sizes, which are really hard to obtain from information provided in primary studies. Hedges Tipton and Johnson 2010 introduced another method called robust variance estimation RVE, which doesn't require the knowledge of correlations between effect sizes, but uses sandwich estimators to estimate the variance. However, studies have shown that RVE only works well when the number of studies is large. So Tipton and Johnson 2010 suggested over 40 studies are needed and Tipton 2015 also showed that the performance of RVE also depends on the characteristics of the design matrix. Meta analysis and social science research, however, typically have smaller number of studies, over half of them have less than 40 studies. And there are smaller studies using RVE results in type one error inflation, and therefore meta-analysts can conclude that some effect is present when it is actually not present. Tipton 2015 and Tipton and Pusevsky 2015 examines several small sample corrections for a single coefficient test and for multiple contrast hypothesis test, and both recommended a method called HTZ test, which is CR2 correction for RVE, plus saturated degrees of freedom, an extension of that for multiple contrast hypothesis test. The HTZ test was shown to control type one error rates adequately, but it may have possibly have low power, especially for multiple contrast hypothesis tests. In my dissertation, I examined an alternative method called claustrophobic strapping, which has been studied in the econometric literature, but not in the meta-analytic framework. General bootstrapping is used to estimate unknown quantities by re-sampling many times from the original data. And claustrophobic strapping involves re-sampling residuals by multiplying them by claustrophobic, claustrophobic, claustrophobic random weights. This is the algorithm for claustrophobic strapping. First, we fit a null model and a full model on the original data. The full model consists of all the variables of interest in the meta regression model. The full model consists of variables except for the ones being tested in single coefficient test or multiple contrast hypothesis test. We obtain residuals from the null model, and then we generate an auxiliary random variable and multiply the residuals by the random variable, which is set within clusters. We can also multiply the residuals by CR2 matrices before multiplying by the weights. We then obtain the new outcome scores by adding the transform residuals to the predictive values from the null model fit on the original data. We re-estimate the full model with the new calculated outcome scores and obtain the test statistic. And we repeat steps three to five for r times, which is the number of bootstrap replications, and we calculate the p-value as the proportion of bootstrap test statistics that were greater than the original test statistic. In my dissertation, I ran two massive simulations to compare the claustrophobic strapping test against the HTC test. And I found that the claustrophobic strapping test maintained type 1 error rates adequately and provided huge gains in power than the HTC tests, especially for multiple contrast hypothesis tests. Dependent effect sizes are common in meta-analysis and social sciences. If we use the original RVE as suggested by Hedges-Tipton and Johnson, it can lead to type 1 error inflation, meaning high false discovery rate. If we use the HTC test recommended by Tipton and Tipton for CFC 2015, it may result in low power, especially when you're doing multiple contrast hypothesis tests, which means that we may miss the effects that are present. So we recommend the use of claustrophobic strapping test, which balances type 1 error rates and also provides more power than existing small sample corrections. So claustrophobic strapping algorithm is implemented in our package while meta. The main function of the package is called wall test CWB. And the function works with meta regression models that using the Ruby function from the meta package and the RMA and V function from the metaphor package. And these are the arguments required for the function. The full model is the meta regression model using Ruby or RMA and V with all the variables of interest. The constraints are like the contrast to be tested. R is the number of bootstrap applications that you want to run. And we recommend a high number like 1,999 or higher, like higher bootstrap being replication results in higher power. For the rest of the arguments, you can please read our documentation online. This is the example data that I'll be using to show the functions of wall meta. It's called SAT coaching and it's available in co-op sandwich package. It is a meta analytic data setting the effect of SAT coaching on verbal and math SAT scores. It contains the study type variable, which indicates whether groups were matched non equivalent or randomized hours of coaching done and the type of test verbal or math. It also contains the text size and the variants associated with the text size. And as you can note like each one study can have multiple effect sizes, because they have multiple outcome measures correlated outcome measures. So there's a correlated correlated effects dependent structure. I'm running a model with zero interest out the study type variable, the hours and the test variable, and running a correlated effects model using review from a meta. And this is the, this is the result from the Ruby model. So these three coefficients are the average effect for each study type controlling for hours and the test type. And for multiple contrast hypothesis test. I want to study whether the effects of coaching differs based on study type whether it's the same for matched non equivalent randomize or face or if it's different. To run the multiple contrast hypothesis test to examine whether treatment effects differ by study type. I'm using the wall test CWB function for more meta. I'm using the Ruby model. I use the constraint equal function from the club sandwich package to create a constraint matrix, setting the first three coefficients equal to each other. I use 999 bootstrap replications and I set a seed. And this is the result that we get. We get the p value from possible bootstrapping. And if it's greater than 0.5. For example, if the if your nominal alpha is 0.5. And conclude that there's no statistically significant difference in the effects of the city coaching across the three different study types. You also get information on what tests you're in any adjustment that you use, and the CR corrections and statistics used to conduct the wall test. And here I'm fitting the same model, but using metaphor or MAMB functions. I'm estimating a multi level meta analysis model with study type nested within study using our MAMB and the wall test CWB works the same way the inputs are very similar. And you get the p value for the metaphor model. We also have a function called plot in one meta, which plots the bootstrap distribution bootstrap test statistics distribution so these are the f statistics from each of the bootstrap applications. And this dash line indicates the original f statistics from the original model, and the proportion of f statistics bootstrap statistics that are greater than that original f statistic is the p value from the cluster of bootstrapping. Thank you very much again for listening to this presentation. We have a website for this package, which has instructions on how to download the package from CRAN, please download, and or from Github Github. And it has examples on how to use the package with the Ruby models and our MAMB models. We have documentation on the functions and what all the arguments are. And we also have a vignette, vignette detailing what cluster wall bootstrapping is, and how to use it with Ruby meta models and metaphor models. Thank you very much. If you have any questions, please let me know.