 Hello and welcome to ESMCON for 2022 and special session five other quantitative synthesis as always this session is being live streamed to YouTube and the individual presentations have been pre-recorded and published there as well. Subtitles have been verified and can be auto translated for the for those individual talks automatic subtitles will be available shortly for this live stream. If you have any questions for our presenters you can ask them by the presenter's individual tweet from the EF has clicked on Twitter account, see them on our feed. Presenters may have time after their talks to answer some of these questions or at the end of the session if time allows will endeavor to answer all questions soon after the event. We'd like to draw your attention to our code of conduct which is available on the ESMCON website at ESMCON.github.io and I have the pleasure of now introducing Eves Plesson from VU Amsterdam. Hello and welcome to my talk what if a very short primer on conducting multi-verse meta-analyses in R. A multi-verse meta-analysis is nothing more than conducting all possible meta-analyses for a given research question what that exactly means will be covered later. Firstly we will look at why one would even want to go through the hassle of conducting all possible meta-analyses. So why one would want to conduct a multi-verse meta-analyses. Then we will talk about the basic idea behind multi-verse meta-analyses and all the plots and all the analysis you can see here are prepared on my guitar repository so you can calculate and plot everything you see here. And then I will show you plots a lot of plots. So now let's have a talk about the motivation behind conducting a multi-verse meta-analyses. Imagine a scenario where there are multiple meta-analyses on the same research question but with diverging results. We have meta-analysis A that actually does not find an effect. On the research question of interest meta-analysis B is inconclusive and meta-analysis C actually finds an effect. So you might suspect different reasons for these diverging results. For example the individual meta-analyses might have used different methods. Fixed effect, random effects, multi-level modeling. So this might be the reason for those diverging results. Or they might have used different criteria for removing outliers. They did not remove any outliers or they used a specific cut-off, a fuzzy cut-off and so on. Or they might have used different inclusion criteria and the list can go on and on and on. But it is really important to understand why those different diverging results emerge and this can be very tedious and also frustrating. This is where multi-verse meta-analyses come into play because they are perfectly suited if you have the same research question but different summary results. The basic idea and origin of multi-verse meta-analyses began with this paper which data to meta-analyze and how by Martin Voracek and colleagues. The question you should ask yourself when conducting a multi-verse meta-analyses is actually the title of the paper. You have to think about which data, which subsets of studies are eligible for your multi-verse meta-analyses and also how you could analyze those. So all the different meta-analytical models that could be used and are reasonable should be defined beforehand. If you want to dive deeper into this topic I can recommend all of those three papers where you get the background that is needed to fully grasp multi-verse meta-analyses. So which meta-analyses or meta-analyses should I compute? Multi-verse meta-analyses would suggest why not all of them. So the multi-verse of meta-analyses for a given research question, the multi-verse meta-analyses, is basically the combination of A all possible study subsets and B all available statistical and meta-analytical methods. Let me break this down for you a little further. First we have to decide on the which factors, that is which data or study subsets should we analyze. Imagine we have a research question how effective are psychological interventions for individuals with depressive symptoms. Here we could decide to include studies based on different age groups, different sexes or different therapies. And this could lead to different paths. For example we could include only adults and only male participants and only studies that investigated therapy B. And you can also see that there are a lot of different resulting study subsets that could be analyzed. In our simulated example there are 36 subsets of studies that could be analyzed in such a way. Secondly we have to decide on the how factors, that is which statistical and meta-analytical methods could we use to analyze our data. And here again we could have three different factors that are interesting to us. The first one, how we handle effect size dependency. So when we have multiple effect sizes per study what are we going to do with this? Are we going to keep all of those effect sizes or are we going to keep only one effect size per study based on some criterion? Or are we going to average those effect sizes? Those can all be valid methods but lead to different paths that could be taken. We could also handle outliers in a different way, we could either remove them or we could keep them. And then there are different meta-analytical methods that we could use. For example when we kept all effect sizes we could use a three level model or robust variance estimation. If we only kept one effect size per study we could use p-uniform, selection models. Or if we averaged our effect size we could use a random effect model. And there are many many more but in our example we have 36 different methods that could be used. And in total when we multiply our which factors with our how factors we have a lot of potential meta-analyses that could be run. And now of course it is important what are we going to do when we have so much information and so many meta-analyses. Then we have to visualize them. And to do this we can either use descriptive specification curve plots or inferential specification curve plots. Let's have a closer look. Here you can see our descriptive specification curve plot with simulated data. Let me walk you through this in more detail. We have our upper panel with the descriptive specification curve and our lower panel with the which and how factor combinations. Each of those vertical lines represents the confidence interval of a single meta-analysis. Down here on the x-axis you can see that in total we have 160 meta-analyses included just in this one graph, in this one picture. On the y-axis you can see our summary effect size in this case HG and this black line represents the effect size estimates ordered by magnitude from our smallest effect size to our largest effect size. Those colors represent the amount of included primary studies, warmer colors include more studies and cooler colors include less studies. You can also see a black dotted line representing the null effect so all the confidence intervals that cross this line are statistically not significant and a red dotted line representing our smallest effect size of interest. And now you can investigate why meta-analysis A and B and C diverged. We can have a closer look at the which and how factor combinations that are the reason or could be the reason for different results. Our meta-analysis A included all age group only male participants and all therapy types. They did not remove any outliers and they used p-uniform to analyze the data and estimate the effect. And meta-analysis C they made some different choices, they took different paths but maybe even more importantly than pinpointing why single studies that worked is we can look at the overall picture from a bird's eye perspective and actually identify patterns. So for instance we can see that female participants produced much larger effect size estimates than male participants. Therapy B produced much smaller effect size estimates than therapy A or therapy C. We can also see that removing outliers leads to smaller effect size estimates and overall we can have a very nice look at the robustness and overall evidence based on our which and how factor combinations. And here you can see the same plot but simulated under a null effect. So all those confidence intervals actually cross the null line. So those studies or those meta-analysis would not be statistically significant and report quite small effect sizes. But here you have some outliers that are quite large in comparison. So it can be helpful to look at the descriptive specification curve to see how the overall evidence looks. Here is an example from an ongoing research project where we actually plot over 5,000 meta-analysis in a single plot to answer questions that are relevant for psychotherapy research. So it becomes quite overwhelming but this plot helps us a lot in finding patterns and understanding how robust our evidence actually is. Here you can see an inferential specification curve plot with simulated data. And I simulated a real effect so you can see that our red descriptive specification curve plot is different from a null scenario which is represented by this gray line. To accomplish this gray line I simulated data under a null effect and did some bootstraping to get the 95% confidence intervals. And this plot is not in this area so we could be pretty sure that the effect is different from a null effect. But in this example I simulated a null effect so here we already know that there is no true effect and in this case the descriptive specification curve is in our gray area. I highly recommend pre-registering or publishing a protocol for your multi-verse meta-analysis because it is quite funny that when you want to look at flexibility in data analysis you can also fall victim to flexibility in data analysis. So it is a good idea to be very clear upfront what you are going to investigate and how you plan to do so. Here you can see some readings that I suggest. You already saw those two papers but Julia Rohrer wrote a very nice blog post on dangers and pitfalls of multi-verse analysis in general and I can recommend this a lot. Thank you so much for your attention. If you are interested in the slides you can find them at my open science framework profile. If you are interested in the code for all the plots you have seen you can find it on my Github repository. And if you are interested more generally in related topics follow me on Twitter where I post on a regular basis. Thank you, bye. Thank you Eves for that and just to remind you if you do have any questions feel free to post them on Twitter underneath the post relating to this session and we will get them answered as soon as we can. Next up we have got Megha Joshi from the University of Texas at Austin. Hi all, thanks very much for watching this presentation and for attending this conference. I am Megha and I work as a quantitative researcher at the American Institutes for Research and I will be presenting on my package Wild Meta which implements cluster wild bootstrapping for meta-analysis. This package is co-authored by my advisor James Paseevsky and I also have a wild HEX logo designed by my talented friend Wes and the package provides functions to handle dependent effect sizes and meta-analysis. Typical meta-analytic techniques like meta-aggression involves the assumption that effect sizes are independent however in social science and education research it's common for each primary study to yield more than one effect size or for studies to be nested in some way creating dependence. For example, Tanner Smith and Lipsy 2015 is a meta-analysis examining the effects of brief alcohol interventions. The meta-analysis consisted of 185 studies and in those studies 1,446 effect sizes. The meta-analysis included primary studies which had multiple correlated outcome measures for example the alcohol consumption outcome was measured by frequency of consumption, quantity of consume and blood alcohol concentration. The studies also included repeated measures of the outcome and multiple comparison groups creating sort of correlated data structure, correlated effects data structure. There are several ways to handle dependence. One is to ignore dependence but doing so can result in incorrect standard errors and incorrect inferences from hypothesis tests. Some ad-hoc methods include selecting one effect size randomly per study or analyzing subsets of data separately however such methods result in loss of information. The ideal way to handle dependence is to use multivariate models but to do so requires information on covariance or correlations between effect sizes which are really hard to obtain from information provided in primary studies. Hedges, Tipton and Johnson in 2010 introduced another method called robust variance estimation RVE which doesn't require the knowledge of correlations between effect sizes but uses sandwich estimators to estimate the variance. However studies have shown that RVE only works well when the number of studies is large. Hedges, Tipton and Johnson in 2010 suggested over 40 studies are needed and Tipton 2015 also showed that the performance of RVE also depends on the characteristics of the design matrix. Meta analysis and social science research however typically have smaller number of studies over half of them have less than 40 studies and when there are smaller studies using RVE results and type 1 error inflation and therefore meta analysts can conclude that some effect is present when it is actually not present. Tipton 2015 and Tipton and Pusevsky 2015 examines several small sample corrections for a single coefficient test and for multiple contrast hypothesis test and both recommended a method called HTZ test which is CR2 correction for RVE plus saturated degrees of freedom and extension of that for multiple contrast hypothesis test. The HTZ test was shown to control type 1 error rates adequately but it may possibly have low power especially for multiple contrast hypothesis test. In my dissertation I examined an alternative method called claustrophile bitstrapping which has been studied in the econometrics literature but not in the meta-analytic framework. General bitstrapping is used to estimate unknown quantities by resampling many times from the original data and claustrophile bitstrapping involves resampling residuals by multiplying them by claustrophile level random weights. This is the algorithm for claustrophile bitstrapping. First we fit a null model and a full model on the original data. The full model consists of all the variables of interest in the meta regression model and the null model consists of variables except for the ones being tested in single coefficient test or multiple contrast hypothesis test. We obtain residuals from the null model and then we generate an auxiliary random variable and multiply the residuals by the random variable which is set within clusters. We can also multiply the residuals by CR2 matrices before multiplying by the weights. We then obtain the new outcome scores by adding the transformer residuals to the predictive values from the null model fit on the original data. We re-estimate the full model with the new calculated outcome scores and obtain the test statistic. And we repeat steps three to five for R times which is the number of bootstrap replications and we calculate the p-value as the portion of bootstrap test statistics that were greater than the original test statistic. In my dissertation, I ran two massive simulations to compare the claustrophile bitstrapping test against the HTZ test and I found that the claustrophile bitstrapping test maintained type 1 error rates adequately and provided huge gains in power than the HTZ test, especially for multiple contrast hypothesis tests. Dependent effect sizes are common in meta-analysis and social sciences. If we use the original RVE as suggested by Hedges-Titton and Johnson, it can lead to type 1 error inflation, meaning high false discovery rate. If we use the HTZ test recommended by Titton and Tostoyevsky 2015, it may result in low power, especially when you're doing multiple contrast hypothesis tests, which means that we may miss effects that are present. So we recommend the use of claustrophile bitstrapping test, which balances type 1 error rates and also provides more power than existing small sample corrections. So claustrophile bitstrapping algorithm is implemented in our package while meta. The main function of the package is called wall test CWB. And the function works with meta regression models for using the Ruby function from the Ruby meta package and the RMA-MV function from the Meta-4 package. And these are the arguments required for the function. The full model is the meta regression model for using Ruby or RMA-MV with all the variables of interest. The constraints are like the contrast to be tested. R is the number of bootstrap replications that you want to run. And we recommend a high number like 10999 or higher. Well, like higher bootstrap being replication results and higher power. For the rest of the arguments, you can please read our documentation online. This is the example data that I'll be using to show the functions of wall meta. It's called SAT coaching and it's available in co-op sandwich package. It is a meta-analytic data setting the effect of SAT coaching on verbal and math SAT scores. It contains the study type variable, which indicates whether groups were matched, non-equivalent or randomized, the hours of coaching done, and the type of test verbal or math. It also contains the effect size and the variance associated with the effect size. And as you can note, like each one study can have multiple effect sizes because they have multiple outcome measures, correlated outcome measures. So there's a correlated effect dependent structure. So this is the Ruby meta model. I'm running a model with zero intercept, the study type variable, the hours and the test variable, and running a correlated effects model using review from a meta. And this is the result from the Ruby model. The three coefficients are the average effect for each study type controlling for hours and the test type. And for multiple contrast hypothesis test, I want to study whether the effects of coaching differs based on study type, whether it's the same for matched non-equivalent, randomized, or if it's different. I'm using multiple contrast hypothesis tests to examine whether treatment effects differ by study type. I'm using the wild test CWB function from a meta. I input the review model. I use the constraint equal function from the club sandwich package to create a constraint matrix setting the first three coefficients equal to each other. I use 999 bootstrap replications and I set a seed. And this is the result that we get. You get the p value from possible bootstrapping. And if it's greater than 0.5, for example, if the if your nominal alpha is 0.5, you can conclude that there's no statistically significant difference in the effects of this SCT coaching across the three different study types. And here's the information on what test you're in any adjustment that you use, and the CR corrections and statistics used to conduct the wall test. And here I'm fitting the same model, but using metaphor or MB function. I'm estimating a multi-level meta analysis model with study type nested within study using our Miami and the wall test CWB works the same way the inputs are very similar. And you get the p value for the metaphor model. We also have a function called plot in one meta, which plots the big strap distribution bootstrap test statistics distribution. So these are the f statistics from each of the bootstrap applications. And this dash line indicates the original f statistics from the original model. And the proportion of f statistics bootstrap statistics that are greater than that original f statistic is the p value from the cluster with the stripping. Thank you very much again for listening to this presentation. We have a website for this package, which has instructions on how to download the package from cram, please download, and, or from get up, and it has examples on how to use the package with the review models and army and be models. We have documentation on the functions and what all the arguments are. And we also have a vignette, vignette detailing what cluster wild bootstrapping is, and how to use it with rubber meta models and metaphor models. Thank you very much. If you have any questions, please let me know. Excellent. And thank you very much for that mega. So the next talk is from Alex Nicole Harper from Southampton University. Thank you. Yes, it is. Great. Thank you. So thank you very much for tuning in. I will be talking about how we have used sub meta analyses to maintain independence among spatio temporary replicated demographic data sets. So this is where it's done with Professor Patrick Doncaster and Professor Thomas Ezard at the University of Southampton, and Dr Kevin Wood and Dr Jeff Hilton at the Wildfire and Weapons Trust. I'm just going to kick off with some acknowledgments. This is part of my PhD, which is through the NERC Spitfire Doctoral Training Program. The lovely photo of an IDA that you can see on the slides is by Kate Evans on behalf of the Wildfire and Weapons Trust. And of course a big thank you to my supervisor team. So a little bit of preamble. I'd like to point out that our meta analysis is perhaps unusual in that we are considering mean values, rather than a form of effect size. So this could perhaps be termed an informal meta analysis in the sense of morocene colleagues, since we are reporting on a distribution of values. Specifically, but we are using meta analyst calls. Meta analysis in those terms. And another bit of preamble is the method that we're using. So it's an established method from Doncaster and Steak 2018. And it's inverse variance with adjustment for small samples. And what that means is that we can account for estimates that have a small sample size, that's the weighting accounts for that. And also it means we can include studies which don't have a associated variance. So what we're doing with a random effects error structure, the meta estimate is the sum of the weightings multiplied by the estimates by the sum of weightings for the study level means as weighted by one over bi plus T squared, where T squared is where T squared is Cropkins estimator, the hedges and Hawkins method specifically and the study level error variances vi take the mean across the variances of all the studies divided by each sample size. So to reiterate that's the mean which is allowing us to use mean variances, rather than having one per study. And then we can include estimates. We can't include estimates without some size they're excluded because each size needs to be accounted for. And then the meta variance is one over the sum of weightings. So to get into it properly, meta analysis is useful tool in demography, which is the study of the components of populations. The meta analysis is that for the spotted owl of boys and colleagues, where they say the meta analysis can be used to combine results from multiple demographic studies replicating time and space to obtain estimates of vital rates. In their case they're looking at juvenile survival, adult survival and fertility. Another example is the black bear study by best and from here, the plot shows cub survival values, and they also found that survival of fertility values, buried across eastern and western North America so that meant that while they couldn't perhaps use meta analysis to derive content level values the process has uncovered new roots for analysis looking to the reasons behind that variation. However, demographic data sets tend to be messy other than those which are typically subjected to matter analysis. This is the two reasons. So firstly because ecological data sets in general encompass a lot of natural variation and the presence of covariates with a large range of And then demographic data sets in particular tend to include lots of observational rather than experimental data, which might therefore be structured So the demographic meta analysis need to have like careful application of accepted national analysis methodologies, for example in relation to avoiding non independence which is what we're here. So we use population modeling to investigate breeding ecology and form conservation for the common either which is the sea dark you can see here, which is a species of the second polar northern hemisphere. So the geographic models are designed to be relevant to the species as a whole, and hence we wanted to parameterise them with some kind of global mean values of vital rates. So we have a data paper which describes our population of these vital rate estimates, and we had over 20 independent estimates for adult annual survival clutch size which is the number of eggs laid and hatching in the portion of eggs producing young. And so for these we wanted to do some form of metronautics we had enough studies. However, we had a lot of internal replication. What I mean by this is where the researchers are going out into the field to gather data often they're trying to answer a particular question. So for example how does predator control affect adult survival. Does clutch size vary with age, or how is hatching success affected by the presence of a predator such as mink. So we've been collecting data across multiple years and or locations. So for example, adult survival across three columns, as shown here in this example by would it out clutch size on different islands for example, egg and stump island. In this case, but it might also be different types of islands of open or woodage or hatching success across multiple years in this case. So we have this sort of internal replication for 7% of adult survival studies, 33% of clutch size studies and 12% of hatching success studies. While these are themselves non trivial values. It also shows that any type of full data analysis might have had a sort of intractable number of studies so we need to be making best use of what we have. So how to make use of as much available information as possible while maintaining the prevalence of wedging non independence. So ideally a matter analysis involves single less single effect size estimates derived for each study in order to maintain statistical independence and avoid pseudo application. And so what we thought is that we could apply our overarching metronome methodology to those cases where we had multiple estimates but for one study. We didn't use the same use of the terms of metronome analysis as in Zubman et al 2015. But it aligns with the suggestions of Megasyn and colleagues in terms of adjusting sample size variants of waiting to represent information content, or had a way in colleagues in that it's preferable to have metronome is a reduced data set them both larger. So this is what we actually did. I'll take the example of that would I tell study that I mentioned with the adult survival, different colonies and this is one that fed into our study. So this is the data that you had you have a different estimate with associated standard errors and sample sizes for three different locations within the study. So you essentially the same methodological methodological steps as you would in the main method analysis, so you convert standard errors to variances. You use those calculate that mean variance, which then calculates the bias mean variance divided by the sample sizes. T squared, hence the weightings. And from there you can calculate your meta estimate and the matter variance. Then when you're sending those three to your overarching metronome analysis. So this estimate is just the matter estimate, but the variants that you send forward is the meta variants multiplied by the number of contribution estimates so here there's three, there's three studies. That allows compatibility with the calculate variants from the other non sub meta analyze studies. So to get show how this affected our meta analysis adult survival, we've coded the sub meta analysis with the asterisks and I'll have it in here. So in the case of a crucer towel 2012, the sub meta analysis across two replicates gives 0.72 as shown, you would actually get the same mean. If you did just take a simple mean of the values. However, if you were trying to send one single estimate forward. If you've chosen the most precise as a disorder standard error, that would have been 0.761 which is significantly larger. Similarly with the widow tell study. Sub meta analysis gives not point nine one eight as shown the mean across three are considered similar at not point nine one six. However, if you've taken the estimate which was either the most precise or the largest size. That would be not point nine four so again, quite significantly larger. So the summit analysis is hopefully balancing information content against risk of bias so with the widow tell example again, we have these three adults live or estimates across the three sites. As I said if we selected yesterday with the largest size that would likely overestimate. If we sent all three estimates through to the main meta analysis. That will be lending false equivalents with the estimate from other studies which are themselves usually means across areas of years. And so take your mean, whether unweighted or weighted as in this case with the sub meta analysis reduces the risk of bias and ensures that comparison. And this seems to be similar to senior retails second order meta analysis, but it's theoretically close to solutions and on the dependence, which are. So to do that comparison of the second order meta analysis that's referring to the meta analysis of previously published meta analysis. And ours is similar that it's two levels of meta analysis. Three meta analysis one for each by the way, or very numbers of studies of which some provide a single estimate self mean, and others provide the multiple estimates on which we are conducting the sub meta analysis to show a sensitivity analysis for adult survival. So if we just took a mean with all the estimates analyzed equivalently we'd have not point eight five six. We took a mean with the sub means taken across those studies that we've discussed so we started to analyze. You get open eight seven. If you do a weighted mean where you're sending forward most precise single estimate from other studies. You get a lot higher not point six one, whereas a weighted mean either with simple means or weighted means for the sub studies gives not point eight five seven. And while these numbers are not that different from the grand scheme of things we're talking about survival which is a probability so ranging from not one. And it's a very important one for the species in the life cycle and so actually small differences will make a big difference to population growth rate for example. I'll point out we did explore meta regression but there were no obvious relationships, so we were stunning clutch size which had lots of estimates. We expected it might vary geographically. There was no obvious pattern for clutch size with last shoot as we might expect, but there was a hint of reduce clutch size within the Arctic Circle. You wonder whether the latitude was too simplistic there are very different climate regimes across similar last year so for example 57 degrees north in Scotland versus Hudson Bay. You have a average January temperature of three degrees versus something like the minus 25 with you know completely frozen. And so that's really different and perhaps we could estimate that I'll represent that through continentality continentality and oceanic indices. So the former represents the cool effect of land masses, the latter the one effect of oceans. We found both of these to be marginally significant predictors of clutch size on the left. Clutch size increase decreases with increasing continentality so harsher winters short breeding seasons on the right you can see there's a hint that perhaps oceanity has an effect within the Arctic Circle but not below or that interaction is actually not. So this sort of thing can be used for extension, but we had reason to focus on the single estimates for our modeling. So to summarize our submatter analysis addressing on independence within studies to help us get our global mean vital rate, where there's evidence that there are, where there is more interest in underlying drivers, or we have evidence of strong subspecies which we didn't find here. Multi level modeling would be the ultimate obvious method as recently developed by our colleagues. The associated code with that study implies that it would require a more balanced data set than ours, because we have studies providing single estimates across a single site year single estimates where they've performed some kind of mean across multiple site year combinations, and those providing multiple lessons where we've conducted the submatter assets. I'll also point out again the messiness demographic data, some have no sample size and so we're rejected due to our method, some had some size but no variance and included with the mean blind all and a few provide both. Thank you very much for listening, I hope I've spoken to the importance of tailoring matter analysis methodologies to non standard situations and hopefully it's accessible to ecologists and those sorts of users as well as matter analysis specialists and I'll just put the references up there. Thank you very much. Brilliant, thanks very much, Alex. So just a reminder, if you do have questions, please, please, please feel free to post them on Twitter or as a comment on the YouTube live video. I'm going to hand over now to Maria Lambrish from the University of Rivera and Virgili for our last talk. Okay, hello, can you see my screen. Yes. Good afternoon, and my name is Maria Jambri from University of Rivera, Virgili and I will present here a manida on a method analysis using overall results. Here they start with a brief introduction. So method analysis is a combination of the results in a single estimate and is used in clinical applications, mostly but we need a lot of academic findings to obtain the final clinical application. These academic findings can come from different files. The first one that is emerging nowadays is metabolomics. Metabolomics is the study of endogenous and exogenous metabolites in biological systems. These are small chemical compounds. And the aim is to provide information about all these chemical compounds in a system by the diversity and the possible combination that exists, the number of chemical compounds is really high. For example, in humans, to date we have identified more than 200,000 compounds and this number is increasing day by day. A common workflow in metabolomics studies is you start with a sample and then through high throughput technologies like mass spectrometry or nuclear magnetic resonance, you obtain a list of compounds that are present on that sample. And then you can perform the statistics on this list. Usually these are these control studies that you have a disease like cancer, then you compare with a healthy control group. What do you do to estimate that? You compare the populations by the statistical significance to know the difference between these two populations and also you calculate the fault change, which is the ratio of fault change these two populations. But there are not a standard method to report the results, so you need to study the results reported may differ. When you want to do a traditional meta-analysis, first you need some statistics that are present in all the studies. The most common are the mean, the standard deviation and the number of participants. What happens in metabolomics is that the standard deviation or the error is not present in the studies, so the number of studies that present the error is very few. So this don't allow you to perform a complete meta-analysis of all the studies from the same question. The second option will be make the meta-analysis with the raw data, but the size of the analysis using the raw data is very huge and this is not feasible with the metabolomics information. For that, we develop an approach that uses these two single estimates, the p-value and the fault change. To perform a meta-analysis, when you cannot perform a meta-analysis with all the estimates, this is an approach to use this data and have an information. The values are combined using a weighted Fisher method and the weights comes from the study size. So it's not the same meta-study with 100 participants than a study with 20 participants, for example. Then the fault change is combined logarithmically. This reduces the skewness that comes from the different techniques and methodologies to extract the components and it's then averaged and weighted for the sample size. All this implementation of the meta-analysis is implemented in a grand package called a MANIDA. This includes the data involved from common text files, the meta-analysis using the MANIDA approach. Also another qualitative analysis that is about content that I'm going to explain in a few moments and visualization of the results and also the data that you upload. First, to use the MANIDA, you can import your data from the component we are talking about. Maria, I'm really sorry, you cut out for about five seconds there, so I don't know if you can just go back and repeat what you said. Yeah, from here? Yeah, so you managed to introduce the package and then it sort of cut out a little bit when you talked about what it included. Okay, so this is the grand package MANIDA implemented and this includes the upload of the data from simple text files. You can perform the MANIDA meta-analysis approach. It also includes a qualitative estimate that is about counting, I will explain in a moment deeper, and then visualizations of the loads of the results and also the data that you upload. So MANIDA starts with the upload of your data from a simple text file. We're here with talking about metabolomics, about chemical compounds, so the data in this case are the component, and then you need the p-value, the full change, the sample size, and if you want you can include the reference where this data comes from. Then with the function computer MANIDA, you obtain the meta-analysis results, so the p-value combination, the full change combination and the sum of the sample size. In case of metabolomics, another disadvantage is that not all the studies disclose the same components, so the sample size will differ from one study to other and from one component to others in the total results of the meta-analysis. Talking about chemical results, you can complement the results of the MANIDA meta-analysis with information about the chemical component like the molecular mass or the molecular weight. You can also publicize from the databases of these components. Then you can visualize the results in a classical volcano plot. Volcano plot plots the local technical full change versus the local technical p-value, so significant results would be the ones that are in top right and top left. And here you can see by the dashed line the values that are over the cutoff establishment. Biology usually full change over two is considered significant. It means that have some effect in the body. Then in some cases the full change is not disclosed numerically, so we only have the trend of the change. We know that a component is upregulated or downregulated. In this case, the options for meta-analysis are less. So the only option is to do a qualitative estimate. This is the vote counting. So we assign a vote for each trend. Component is upregulated will have a plus one and a component downregulated will have a minus one and then you compute the sum of all these votes. As before, you can complete all the information about the qualitative analysis by the public databases information. And also visualize the results. Here you can see a bar plot with the total sum of the votes for each component present in the literature. What happened is that we realized when we were doing a meta-analysis with urinary compounds for colorectal cancer that in some studies the trends were not consistent. For example, we can see here that citric acid was found upregulated into studies, but in other three studies was found unregulated. But by the different sample size of the studies, the total combination of the p-value was significant. So for this, we developed the spread plot. In this case, when we have a significant result, we can confirm if this significant result is consistent across all the literature in our first glance. All these functions are also implemented in a Shiny app for non-air users with the same functions that we have explained. So you can upload your data at text file, then you choose if you want the quantitative or qualitative analysis depending on the data that you have, and select the columns to do this analysis. In case some rows have missing data, this won't be used to perform the analysis, the magnitude or the vote counting either. Then this, you go to the quantitative analysis and see the results. By the volcano plot first, when you can choose the p-value cutoff and also you can choose the cutoff for the full change. Then in blue, you have the significant compounds that are non-regulated and in red, the significant compounds that are upregulated with the p-values that you have selected. This is the same table that you obtain when using the air package. If you move to the qualitative analysis, you'll find the plots for the votes and also the exploded plot to confirm your results. All these functions and results can be downloaded in a report. This is an HTML file where you have all the information with the tables and also the plots. You obtain the results for the amnida, the approach, and the vote counting also. So thanks to all to hear me and thanks to my supervisors and collaborators that have worked with me. If you have any questions, I will add to answer it. Brilliant. Thanks very much, Maria. That was great from everybody. We haven't had any questions yet via YouTube or Twitter, so just encouraging those who are watching to please feel free to submit their questions. We do have a few minutes left, so I wanted to ask, I guess, if anyone who's here had any questions for any of the other presenters on any of their presentations. We've got some stunned silence there. Excellent. So I thought all the presentations were excellent, so thank you so much for giving up your time to talk about your work. It was really fascinating seeing the developments that are happening in this field. If we don't have any questions for each other and we haven't had any questions from Twitter or YouTube yet, I think we can let you all go a few minutes early. Thank you just once again for all of your contributions today. It's been really, really interesting.