 Hi, I'm Witold Wiencek. I'm a statistician. I work on applied research in health and in development economics. I'm a consulting director at the Development Innovation Lab at University of Chicago. And this is a short talk where I'll be showcasing Bagger, which is an R package for Bayesian meta-analysis, which is a stand, the most popular Bayesian modeling platform. In this talk, I want to talk a little bit about motivation behind creating the package, show a few simple examples of how it's used, and along the way, highlight why meta-analysis and Bayesian inference make for a good combination. Before moving on, I want to give credit to my friend and co-author, Rachel Meager. She's an assistant professor at London School of Economics. In fact, the package was motivated by our shared interest in real-world applications of Bayesian methods, especially meta-analysis. As Bayesian statisticians know very well, all kinds of meta-analytic methods have already been implemented and implemented very well even 20 years ago, for example, in Weedbox. So I personally think that the market gap in terms of Bayesian tools for meta-analysis is less in building new capabilities and more in avoiding errors and increasing access. What I mean by this is that we live in a world, I think, where meta-analysis is more and more used as a kind of a simple plug-and-play tool. And this means that there are more users who will have wrong interpretations of models, overconfidence in models. And generally, we end up seeing all kinds of garbage in, garbage out problems. And because Bayesian models are more technically sophisticated, this problem compounds when we move to Bayesian meta-analysis. So this package is built for a particular group of people with particular goals in mind. People who understand the advantages of using Bayesian models, but they might not have technical skills. And they might not be statisticians. They might be casual R users. The four goals, therefore, are to implement all of the basic models, focus on making them accessible and allowing people to criticize models, help them avoid Bayesian mistakes when they're likely to occur, and still keep this framework flexible enough so that we can extend it to as many models as feasible. However, the goal is not to build a power tool for people who would already be able to build their models by hand, as it were. So moving on to an example, I will use a very simple one. Bagger also has methods for working with binary event data or quantiles or zero-inflated data, although some of those things are still work in progress. But I want to use a canonical Bayesian example from Donald Rubin's 1981 paper on SAT improvement in eight schools. And this is a nice way of highlighting how a meta-analysis model is also a form of Bayesian hierarchical model. So as is typical, we only have aggregate data. Each study measures mean and a standard error, Y-I-S-E-I. And then at the upper level, though, the true effects, tau-i's, may come from a common distribution. How they relate to each other is regulated through sigma parameter heterogeneity. So sigma equals zero, Bayesians would call full pooling. That is the fixed effects case of meta-analysis. Sigma equals infinity means no pooling. That's no meta-analysis, no borrowing of information across studies, and anything in between is partial pooling or random effects. Our objective as Bayesian statisticians would be to derive posterior distributions for tau and sigma, sometimes also for distributions for tau-i's effects in individual studies, which means that we always need to specify a prior on sigma and tau. Before showing this in action, a quick recap for why we might choose to go Bayesian in this meta-analysis. There are some typical arguments for going Bayesian instead of frequencies. I won't cover those here, but few specific ones for meta-analysis. First, incorporating prior beliefs I think is a unique strength for meta-analysis because they are always conducted in data-rich environments. So for example, for a medical RCT, one can borrow information from thousands of other meta-analysis to have a prior on how much heterogeneity across studies there might be in a particular domain, for example, in an oncology trial. Secondly, Bayesian methods avoid problems with underestimating variance across studies when there are a few studies especially. And while doing that, they also emphasize uncertainty in parameters. So it's not that heterogeneity is often big or small, but it's also that very often meta-analysis it's very imprecisely determined. So Bayesian models bring this fact to the forefront. In general, Bayesian models allow us to create predictive distributions, so the distributions that tell us what would happen in another study that wasn't observed much more easily than other methods. And in general, they allow us to create any kind of derived quantity and describe them probabilistically. So for example, what is the probability that out of every study in the meta-analysis each of them had a true effect greater than X. Those are very relevant to real-world decision-making. And lastly, they also allow for easy and formalized model comparison, for example, choice between fixed and random effects. The cost of this is of course higher technical sophistication and we should be clear about this. And this is exactly what we're trying to solve here. One last advantage, which is not covered by the package, in some sense, is that Bayesian models are much more flexible in terms of meta-analysis models that we can specify, but then one would have to do this by hand. OK, so now quickly on to an example. We load in the package, take a quick peek at data. You only need R standard bugger packages in R. No extra software is needed, but you do have to have R tools installed alongside R. This is the aggregate data in eight schools. If our input data was one line per pupil in each classroom, the model would be minimally different because we have to estimate the treatment effects and standard errors jointly in this meta-analysis model. But the code that I'm going to present would look practically the same. And now this is the entire analysis in bugger. So quickly, segment by segment, we fit some kind of a default model in the first couple of lines. That's the meta-analysis random effects partial pooling model. Then typically, an analyst might consider some alternative modeling assumptions. So here, I just made up some extra assumptions about the amount of heterogeneity in data using different priors and also some completely made up example on where the mean effect might be. Once we have the models, we typically plot them. That's also one liner. We use a very plain syntax everywhere. So we use generic functions like plot and print. The specification of prior distributions, Bayesian prior distributions is done using plain text like Cauchy, Normal, Uniform, Student, et cetera. And then once we have a number of candidate models, we can compare them using specialized functions. This one is called bugger compare. We can plot or print those comparisons. We can also compare the model conclusions. So those predictive distributions I mentioned a second ago, those are also encoded through one line commands. And a very typical comparison that meta-analysts will decide to do is to compare fixed versus random effects. So one line comparison like this is done here at the end. And to formally choose between partial and full-pulling random effects and fixed effects, in other words, models we can also conduct leave one out cross-validation, which I'll explain right at the end. So now let's take a peek at the results and what the outputs look like. This is the default output from a fitted model. I didn't show the fitting process, but it only takes a second. And the user would get all kinds of automated prompts about the decisions made by the algorithm when necessary. And what we see here are group-specific estimates at the bottom. And then at the top, we have summary statistics. So where is the true effect? What is the amount of heterogeneity? What is the amount of pooling? A pooling statistic is a complement of I squared statistic that is known to anyone, probably in evidence synthesis. And alongside each statistic, we see not just the mean, but also how precise each of those estimates are. So as I mentioned, it is very typical in smaller meta-analyses to for heterogeneity to be imprecisely estimated. And this is to some extent true here as well. If we look at the 95% interval for this default analysis. Here is one of the examples of plots that we have in bugger. It could be a forest plot just as well. This is just a slightly different visual flare. Here we can see that, in fact, in this random effects model, all of the estimates are pooled very strongly toward the hyper mean, which is at the bottom of this plot. If we have different candidate models, we can compare them using all kinds of visual styles as well. This is just one of them. So here we can see that in two liners, once we have several models, we can just name them and try to plot them alongside or print them alongside each other and see how they differ in the results. Another type of comparison is to compare the predictive distributions. And here you can, by the way, see that all of the syntax for the plots is done using ggplot. So there is a lot of interfacing between our package and quite standard R tools for working with data. And here we see that there are actually some practical differences to those made up models. More practical example of model selection, I think, or the most typical one is when we consider differences between fixed effects and random effects, or again, the full and partial pooling. So here is an example of how we can do this using a one liner in the package. It's a bugger compare function again. We just say what equals pooling. And when I plot this, I can see the thing that I've seen a couple of slides back that for this simple example, random effects look quite similar to the fixed effects model, especially if we consider the bottom row here, the pooled estimate. And here, as a gratis, we also have plots of what happens under no pooling model, so no meta-analysis. But it also makes sense to test for this formally. So here are three lines where we build leave one out cross validation models, which are metrics of predictive performance. They're closely related asymptotically to information criteria that you might know from other modeling tools and from outside of meta-analysis. In this case, I measure leave one out cross validation for partial pooling model and full pooling model, and then I do a one liner comparison, which tells me that there is indeed a slight preference for fixed effects model, which we just so visually. This model has fewer parameters and fits similarly to data. There is, of course, much more, and all of it is written in pretty plain syntax. So the things that I could have been doing to examine my model here are comments such as treatment effect, group effects, draw effect draw, which is draw from predictive distribution, forest plots, other pooling and heterogeneity statistics using pooling or heterogeneity command, et cetera, et cetera. But that's the basic outline of the package. The package in its beta version is available on GitHub. And yeah, thanks for listening.