 Hello and welcome to the presentation. My name is Josh Plannan. I'm a principal researcher at the American Institutes for Research, and I'm coming to you today from downtown Washington DC. Thanks for joining me. And so today we're going to discuss an evidence gap map shiny application for effect size and summary level data. So like I said, this is a collaboration with numerous individuals here at our Chi Zhang, Joe Taylor, Ryan Williams, mega Joshi and Lauren Burr mega in particular, I'd like to give a big shout out to mega was our lead developer on the shiny application, and she really helped us rethink how to process this information in the our environment so a big thank you to mega I know she's watching. So, a huge congratulations to her as well. I'm super miss if I didn't think our sponsors. So the project was originally sponsored by an Institute, a US Department of Education Institute of Education Sciences, and ended a little while ago and so the methods of synthesis and integration center or mosaic, as we call it, and our has picked up some of the slack and helped us get this shiny application to the finish line. So let's take both of those sponsors before we go further. So let's then turn to creating one of these gap maps with meta analytic data in particular with our shiny applications. Okay, so first things first, where can you find this shiny application so has a has a shiny application server that where we host various shiny applications and mosaic actually has a couple of different shiny applications that you can take a look at while you're there but if you're just interested in taking a look at the EGM shiny app you go to our shiny apps dot shiny apps dot IO backslash mosaic underscore EGM actually the mouthful but relatively simple URL to get to and you know you're in the right spots. So if you find a landing page that looks something similar to this. Now of course we're going to keep updating this page but, but this, this landing page will probably look pretty familiar to you. Once you click on it, regardless of when you come to it. So before we dive into it. This one I mentioned a bit about the philosophy and the features of the shiny app. The first thing is we wanted it to be compatible with a typical meta analytic data set. We wanted to, we wanted you to be able to say okay I've already done my meta analysis, make probably an R, and I have a row level data effect size level data on the rose. So we wanted to have columns that indicate the effect size and the variants and, and different characteristics of the programs, and we wanted you to be able to take that data set and uploaded into the shiny app and have it work. So that's what it does. We also just wanted it to be easy. But as it features uploading effect or summary level data sets so that means so again, you can have a row level where each row is an effect size or each row can be a cell within your EGM I'll show you what that means in a minute. You can actually have to so a traditional to level EGM is just x axis and y axis with multiple categories within those two axes. You can, and you can produce that very easily with this shiny app, or you can actually produce a three level EGM which I'll show you what that means in a minute. I'm just going to give you some really nice summary outputs. And then, like I said, an easy to copy and paste our code in case you want to do some more fiddling with the the gg plots or anything else within the plot itself. Okay, so the live demonstration. Finally, so two examples one using effect size level data that comes to us from a review that I completed a little about a year ago on the effects of programs on cyber bullying outcomes. So we're going to look at at that so that's the effect size level data sets, and then summary level data set comes to us from that motivational intervention review that I mentioned and that had 14 different motivational interventions across three different sites. And I do want to mention, right off the bat when it comes to loading the shiny app in Chrome that there are that there are several dependencies and so if it takes a minute the first time that you go to the the site. So give it a minute let it load and it'll I'm sure you it'll it'll eventually initialize. Okay first things first put a load data upload my own data there is an example already built in here I'm not going to show you that right now you can you're free to play with that if you'd like but it has all the same features but we're going to use our own data to start off with going to effect size level data and then you can actually upload a CSV or an Excel file we're going to start with a CSV file. Browse. We're going to use this example yes level CSV open. Just takes a second upload complete. Now we've got several dropdowns here. The first one says specify the first factor for the EGM. This first factor is your x axis. So what do you want on your x axis to be. It really is your preference you can go. You can you can put the outcome down there on the access you can put the outcome on the y axis. We're going to go with design on the x axis design is the the RCT QED individual level not school level that sort of thing so the research design. Then on the y axis we're going to go with the outcome. The outcome type so we've got two different outcome types here we got cyber bullying or aggression or traditional in person bullying. Now if we want to specify a third factor we could hear I'm not going to for this round but I'll come back to it and show you what we do with that. This effect size so which variable in your data sets is the effect size variable so just it's easy one it's yes. Then we've made this easy for you if you happen to have a column that's got standard errors you can click on that here and will transform standard errors for you, or if you've got variances for your effects go and click on variance. And then tell it where the column of variances is. And then finally, which column is your study level study identifier now this is important if you have nested effect size data. And so this would be like if you have correlated effects or hierarchical effects, so multiple effects per study. But even if you don't, you probably have a study identifier just go ahead and click on that. And that is it. That's 12345 clicks. And you click on over to create summary data. And the last click you're going to, you're going to push is has to do with the assumed correlation among the effects. And when there are multiple effects per study point eight is is generally the correlation that we say is about right so we're going to use that for now. Click summary data. I'm going to expand this to show 25 rows and now what we see here are our basic summary data. So one of the cells that's estimated. Let's walk through this for a minute. The first column here is the factor one the x axis. So this is our design column. So we've got non randomized class, class level assignment non randomized individual level assignment non randomized school level assignment, randomized class randomized individual randomized school, so on and so forth. The y axis is our outcome. So we've got two outcomes here we've got aggression or traditional bullying or the cyber bullying. The estimation method here I'm going to sort by it just so we can see this easily. There are actually three different types of estimation methods that EGM will select for you. The most basic one we actually don't have here occurs when you have one study and one effect. When that happens. And that analysis is happening within that cell. It just gives you the basic effect size. If you have two or less studies and for less effect sizes you're going to get the univariate random effects model this is just a conservative model that doesn't assume any correlated effects even though there are effects within study. And then the simple averages within the studies to create the average effect per study, and then the correlated effects estimation model is the robust variance estimation method and and that is why we need this little scroll over here. And it accounts for the correlated errors within study. That's the estimation method. Then the average effect size is exactly what you might imagine it is the average effect for that cell. And so in this particular example, negative effects are a good thing because it means that there's a decrease in bullying or cyber bullying for the intervention group compared to the comparison group. And we've got the number of studies and the number of effects. Studies within a cell is for the randomized individual looking at cyber bullying we've got 12 studies and 27 effects, and the most effects is 61 actually. And so this is randomized at the school level. And there's 61 effects there. Okay, enough of that. How do we get to an EGM click on EGM. There's a lot of dropdowns here. This says the first one says do you want to map average effect size onto a continuous color. You want to overlay anything on the dots this gives you allows you to say the number of studies or the average effect I'm going to click average effects so we can see what that looks like. And then the access access is the design. And the y axis is the outcome that allows you to put any labels that you want in there. And when you click click create plots. And then we get an EGM. Okay. So we look here one more time. We can see our typical EGM that we think of we've got the x axis is the design. We've got the six different cells, six different categories for design, two different categories for outcome on the why and then crossed in there is our average effects for each one of these cells. The average effect corresponds to this color code down here with a negative for being the largest in a dark purple. This I believe is set up for people who have color blindness or issues with processing colors this should be good for those folks. And we've overlaid the average effect within each of the cells. If you want to download the plot quick on this download plot, you can give it a different name. We'll just say patient. You can do some stuff with the width of it, but if we download the plot. And then we click on it. You can see it. Last but not least is the R syntax. You can copy and paste this right there like that. And this will tell you this will give you. Everything you need to reproduce that plots using your own data. Really, that's it. I'd like to thank you all for joining us today. I appreciate any input that you have. I'd like to thank again my collaborators and co authors, Chi, Joe, Ryan mega and Lauren. If you have any questions, feel free to reach out to me. There's my email I'm also on Twitter. You can find all my contact information at Joshua are planning.com. And we look forward to having you all use the program. Have a great rest of the conference. Thanks for having me have a great rest of your day wherever you're calling from.