 I'm just setting it up now. Should be the meetings live streaming now. All right. So I believe we are on. We are streaming. We are live. And it's time to start this workshop on Test Adjust for and Report Publication Bias. Before we start, let's go through some little housekeeping. So welcome to the SmartConf 2023 and to this workshop, which I believe is the first workshop. And this workshop is on Test Adjust for and Report Publication Bias. This workshop is being live streamed to YouTube and has a group of participants taking part live. I'm very warm welcome to you all again. If you have any questions for any of us three presenters, because we will be talking in turns, you can ask them either through the Twitter account. So you have the tag there at ES Hackathon. Or that you have to comment on the tweet that is specifically for this workshop. So just reply there and create a threat yourself. And then if you register for the workshop, you can ask questions here also in the Q&A facility. So down there somewhere, there is a Q&A We'll keep an eye on there to answer questions. And you can also comment and chat with other participants on the Slack channel. And you should have got a link to that Slack channel in one of the latest emails from the conference. So we will do our best to answer your questions. We will try not to get dizzy because it's like three places where to look at. But if there are some questions that we don't answer now, we will come back to them right after the workshop. And again, if you have any questions, lay down. You can also contact us. We will put our information there. So before we start, please make sure that you have read the Code of Conduct, which is available at the website of SmartCom. This is very, very important. All right, so let us introduce ourselves. So my name is Alfredo Sanchez Tocha. I'm based in BFL University in Germany. I'm an evolutionary ecologist. I'm not a mythologist by training, but by now I consider myself to be 95% reconverted into an evidence synthesis and a meta researcher. And you'll see a little bit of a pattern here. Would you like to do it or shall I do the presentation? I can say it. So hello, my name is Małgorzata Lagisz. People call me Losia, so you may see that name somewhere. I'm evolutionary ecologist and ecotoxicologist by training and surprise, surprise, 98% reconverted into presbyt baton. Evidence synthesis and meta researcher. Good to see you around. Thank you. And last but not least, Okay, hello, everyone. My name is Jianfeng Yang. Now I'm working at the University of Sydney. I was a agriculture engineer, but now I 95% reconverted into evidence synthesis and meta research. All right, so the three of us are gonna be taking turns for the presentation. I will start with a little presentation on publication bias and some examples drawn from ecology and evolution, which is my field. That's why it was easier for me to kind of present this part with those examples. I hope that's okay, even if you're from other fields. And then we will move on into the presentation from Losia. She's gonna talk about the common practices for publication bias test. Later on, your phone will talk about the new method that we are suggesting here that you should all use because we believe this one. It is the strongest and most robust method to test publication bias for the time being, or so we believe. And we will after, so the last part of today's workshop is gonna be hands-on. You have to get your hands dirty into how to actually do it. And for that, we will use R. No surprise, also, even the conference. All right, so I'll start. I'll start by talking about publication bias, as I said, then with some examples from ecology and evolution. So if you wanna switch off already, the take home message is that publication bias is commonplace. So you've got to test for it. And now I'm gonna try to convince you why that's the case with a few examples. We are gonna talk about two types of publication bias. One could define more. And these two are rather common and so it's the client effects, also known as a time-lapse bias, and then small study effects, which often are known as the file joint problem. And it might be also what you have in mind when you think of publication bias, really. So for the client effects, in ecology and evolution, one of the first examples, if not the first, came from this study and published in the 2000, where the author realized, well, looking at the studies on testing whether parasites can manipulate the behavior of their host. The author realized that there seemed to be a pattern with effect sizes becoming smaller over time. And this is what you can see here with this correlation coefficient. There is a negative correlation between the mean effect size of these studies that were included in the meta-analysis in the year of publication. And this is precisely what the client's effects are. They can be due to multiple reasons, but the main idea is that we would observe a pattern where effect sizes are becoming smaller over time. So they are approaching zero. And in many cases, it could be that those effect sizes over time become zero. This study was soon followed up by a much bigger study, a more of a meta-research study that combined meta-analysis from different meta-analysis from the field. And as you can see, they also found evidence for this kind of negative trend. So by looking at 44 meta-analysis, which included more than 200 studies, you can see that the correlation was negative between this correlation between effect size and year of publication. The size of the effect is smaller, but it was still present. And this is not something that was happening in the past, in the 2000s, in the 2000 and 2002. This is something that still happens today. So there are many examples these days where we still find these relationships. So this is one study that comes from my PhD, which I finished a few years ago. And we also found that there was a correlation between effect size and year of publication precisely for this question, which is a question that you may have even seen on TV because it was featured by one of the BBC documentary narrated by David Attenborough. And in this hypothesis, what we are testing is whether this black patch that the house powers common bird that you may have seen around, whether this patch reflects or signals the dominant status of the individual. And as you can see, we detected that there is a pattern for effect size is becoming smaller to the point that in recent years, there seemed to not be strong or any evidence for this hypothesis. And a very good example just published the last year, and actually even this year, is this one on whether ocean acidification affects the behavior of fish. This is a very particular and interesting study because you can clearly see that there is a strong decline in the size of these effects. And basically, the effect has become very, very small over time. The interesting bit is that if you extract or exclude effect sizes coming from a couple of labs that seem to be the ones that suggested this hypothesis, what happens is that there is actually no trend, meaning that it seems that the trend is driven by a couple of labs in the world that they were producing very strong and statistically significant effects. No surprise, one of the authors of one of these labs wrote a reply to this paper, trying to kind of debate that there is anything wrong there. But even with the reanalysis that this author did, that you can still see that there is a tendency for effect size, not a tendency, there is an effect for these effect sizes becoming smaller over time. And actually on the reply to their reply, what the authors of the original paper are suggesting is that these effect sizes, which are lock response ratios are almost impossibly true. So they've run some randomizations and simulations to show that for these effect sizes to be true, I mean, it's quite difficult. And there is some allegations of fraud involved, but long discussion, yeah, sure. All right, so that's all about the client effects where the time being. Let's move on to small study effects. So this is really what you may have, if you haven't really tested for publication bias or you haven't heard of it before, this is probably what you have in mind when you think of publication bias. So imagine a finite plot, a typical finite plot that you may find in any meta-analysis on the literature where you have on the y-axis the correlation coefficient, in this case, it could be another FX size. And on the x-axis, you have a measure of precision. In this case, I use sample size for simplicity. And this is all run a simulated data, just for the fan of it. So 1,200 FX sizes, a huge amount analysis. And I simulated a mean value of around 0.2 for the correlation coefficient. And small study effects mean that there are certain effects, precisely those of small size that we do not find for whatever reason in the literature. So when you collect your data, even if the data should look like this, like a perfectly symmetric funnel, and that's why this is called funnel plot. When you collect the data, you realize that there seem to be some asymmetry in this funnel plot. And precisely the effects that are missing are those that are small in size and that either are statistically non-significant or they might even be statistically significant but against the original hypothesis. And we don't know where these are. And the idea is that these are somewhere hidden in the file drawer of our research. So for us to test if there is evidence of these small study effects, we need to try to identify these asymmetries in the funnel plots. And we will talk about this later on. That's precisely what we want to learn how to do this in a robust manner. So some examples, one that I like a lot from Ecology and Evolution because it's a really comprehensive study is this one where the author and this came in this case, Tim Parker, tries to summarize all what we know about coloration in this wonderful beautiful bird, the blue tit. And there have been multiple hypotheses suggested for the coloration that these birds present, whether they may reflect the age of the individual, whether they might be used in sexual selection. So for females to choose males, et cetera, et cetera. The conclusion after this super comprehensive study is that for most of the hypotheses, there is evidence of publication bias. So what we actually really have certainty about this, blue-match coloration in this species is that males and females are different. That's about it. For the rest, there is a lot of potential for publication bias to be affecting what we think we knew about this coloration in birds. Another example that I wanted to highlight is a recent study that we did in our lab. Led by a master student, now a PhD student, where we tried to answer whether the size of males in this group of species of fish would predict the reproductive performance of the male. So basically the idea that has been suggested is that small males in this species might be the ones getting most of the kids in the population, because they are very good at being involved in sneaky populations. And we wanted to test this. So we collected data from a studies that tested these hypotheses precisely. And this is the final plot from this studies that explicitly tested these hypotheses. So again, in the y-axis, we have a correlation coefficient as our FX size. And in the x-axis for simplicity, we have the study sample size. And as you can see, there seem to be some asymmetry. And when we tested statistically for this asymmetry, which we will show you how to do later again, you can see that there is a negative correlation between FX size and study sample size. And this is precisely what is telling us that there seems to be some evidence for small study effects, because what's happening is that the larger the sample size, the smaller the effect. And if what we would like to see, so if we don't want to find publication bias, which is hopefully what we all want, that there is no publication bias, is a straight line. That no matter what happens with the sample size, the evidence is the same. That would indicate that the funnel is symmetrical. A funny thing that we did with this study is that many studies measure the two variables of interest for us, which were the size of the males and their reproductive performance, despite that they are not testing this hypothesis. So what we did is that we located those studies, we tried to get access to their data either because the data was openly available or we contacted authors and we got a lot of data that way. That in principle, in principle was never used to test this hypothesis. And then we added it all to our dataset. And when we include it and do again the test, this is what we observed. What we observed is that there seems to be still a little bit of an effect of evidence for small study effects. So you can see still that trend for effect sizes becoming smaller with sample size, but this trend at least is not statistically significant. I mean, it is there, but it's not that strong anymore. Meaning that by doing this approach of analyzing open data sets and data from authors, it seems that we managed to kind of reduce the impact of small study effects on our conclusions. And well, those are the client effects and small study effects. Those are the two types that we will focus on today, but just keep in mind that you can find evidence of publication bias by doing other type of tests. And just briefly, I wanted to show that in this case, for example, you could compare if you have access to effect sizes that are published versus those that are unpublished and see if those differ. So in this plot, what you can see is that if we would have based our hypothesis on published effect sizes only, our conclusion would have been stronger. So this is the overall effect with the 95% confidence interval for the published and this is for the unpublished. Another thing that you may want to keep in mind is whether the experiments are blinded. So basically the experimenters are blind to the hypothesis, meaning that they do not know which group is the control and which is the experimental. And this has been shown to lead to some inflation of effect sizes. So here you have effect sizes from non-blinded experiments leading to a larger overall effect size than those from blinded experiments. This is something that you can try to test if you find any study that is blinded. That's sometimes a challenge in ecology and evolution. And another example is comparing studies that were fully reported to those that were partially and non-reported. For this, what I mean actually is that this overall effect size is based on effect sizes that we got directly from authors. Yeah, so it's very similar to the published versus unpublished. But you can also see that for those studies that partially reported effect sizes, so they did not report all of them, a part of them. There's even a little bit of a tendency for the effect to be stronger, meaning that there might have been some selective reporting that leads to this actual bias and the potentially small study effects. And with that, I think it's time for Lucia to go ahead. Let's breathe one minute, one second to get some air and refresh some nice bubbles. And whenever you tell me, Lucia, I start taking care of the slides. Thank you. Thank you. Yeah, I always find it very impressive like always evidence for publication bias in published studies. So I think the question here is, do people test for publication bias in the meta-analysis? And nowadays we have hundreds of thousands even meta-analysis published in Ecology Evolution and few orders of magnitude more in other disciplines. If we take like medicine and psychology, meta-analysis is very, very popular. So we wanted to look, we are ecologists, evolutionary biologists by training. So we wanted to see how often people test for publication bias and what tools from the available tools they use. So we've conducted a survey which was published almost two years ago. And I will put a link to the article. So you can look at the details and methods and also there is an article, a correction which just fixes a small mistake in one of the equation. So I will put it in the chart, okay? Yeah, so you can see the actual paper. So what we did, we had a data set of over 100 meta-analysis in Ecology and Evolution and we surveyed the methods they use for testing publication bias. And let's move to the next slide. And this study was led by Shinichi Nakagawa and we had lots of good collaboration. There, we look at two types of publication bias that you are already familiar with because Alfredo introduced both. So outcome reporting bias, it's selective reporting. So small study effect, this is the most commonly known type of publication bias and also time lag bias, you are already familiar. So positive results are published earlier and large results effect sizes are published earlier than the negative or a non-significant one. So, okay, let's move on. We still go straight into our results but if you look at the get arrow, it's non-reported. That means that almost 20% out of 100 meta-analysis did not do any test for publication bias. So they don't know if the data set is potentially biased by small studies effects or time lag bias. Okay, and if they don't know, they cannot correct for it either. But the good news is 32% use funnel plots. It's a simplest and most common type of publication bias. The good thing about this, it's really easy. It's implemented with lots of software but think about it, it's quite weak and not very precise and not very effective, especially if you have few studies in your meta-analysis. But still, it's better to use this one when nothing. And we have around 10% using either correlation based or regression based methods. So this is a kind of a single group of type of tests. For descriptions, I suggest you just look at the paper. We have nice section for each type of publication bias test describing how it works and what are the benefits and drawbacks of each. And other common names of publication bias tests that you will see in published meta-analysis is phase-safe number and dream and field test. And yeah, and if you look at yellow arrow, you will see that just 5% of meta-analysis tested for time lag bias. Okay, so this is a very visual summary from the paper. On the left side, you have same types of publication bias test in the previous slide, but the columns kind of characterize the test. And if it's yellow, if it's green, it's a good sign, okay? It can do some of the things that are on top. If it's orange, it's not good, okay? So just giving you a preview of what your phone will be talking next. If you look at the very bottom, you will see multi-level meta regression and that's a new type of test, which is not done by visual inspection of funnel plot. So it's not a type of funnel plot. You need to actually do some stuff, but it's really powerful and it deals with all sorts of weaknesses that other methods may have. And so I think we can move to the next slide before we move to the phone's part where he will introduce this new, more powerful method. Oh, something is messed up in this slide. Okay, that's okay. I think it's just different computer. So I just wanted to give a quick reminder that publication bias test, those small study effects and time lag bias are not the only biases that you can observe in your data. And all those biases stem kind of from human psychology and our publishing system. And you can potentially also test for those using methods like your phone will be presenting where you can include moderators in your analysis. And some of moderators can be related to those other biases. If you can extract data, code it, you can include it in your models and try to test for other types of biases. So for example, not just preference for novel surprising or positive results, but also results supporting parent consensus for views and results similar to the person doing the test, sexy, fashionable topics effect, but also maybe looking at the institutions and type of countries that do the meta analysis or primary studies, language of the studies and the ranking of the institutions and all of those. There's lots of things to explore in the future, lots of other potential biases. So that kind of goes into more bibliometric and other studies, but so this is just an interesting point. And it's time to move to the next section where you need to look at some formulas and learn a new and more powerful method and first theory and after that there will be a practical part of this workshop where you can actually use the R code and try out this new test. Thank you. And maybe 30 seconds of relaxing before we move on. Thanks. Alfredo, so I want to share my own screen. Okay. Hello, everyone. So after Alfredo and Lucia's presentations, I think you got a very good impression of what is Publication Bows, why we need to detect it and the kind of practice or detection methods. Now I'm going to take you through a new method that can be used to properly test and adjust for Publication Bows in your own dataset. You might, oh, sorry, it's my voice here. You might still remember. Lucia's story showed that Final Plot is the most common way to detect the Publication Bows. Final Plot is a very easy way to use and very intuitive. The principle is if you see symmetry in your Final Plot, this means your dataset is safe, there is no Publication Bows. But if you see a symmetry as it's showing in this panel, in your Final Plot, this means there might be Publication Bows in your dataset. So you need to interpret your meta-analysis results with caution. But you might wonder what is symmetry and what is symmetry. This is, to be honest, this is quite a subjective at some point for a given Final Plot. Some people might think this is symmetry, but others might think this is a symmetry. So what we really need is a more objective way to detect the Publication Bows. Agile regression or Agile test is this kind of quantitative method that can be used to statistically test the symmetry of Final Plot. Many people might know this method. Let's recall this a bit. This is a mathematical representation of the Agile regression. In the left hand is the effect size estimate. The principle is if we regress effect size over the sampling error, we can get the statistical relationship between effect size and the sampling error. That is the slope beta one here. So in your dataset, if your slope beta one is positive and statistically significant, this means your dataset might have Publication Bows because large sample size means small sample size and low precision. Studies with small sample size and low precision report large effects. This means there is a high likelihood of selective reporting, or at least the Final Plot is symmetric. Although the Agile regression is intuitive and very common, there are at least three issues to be resolved for this method. The first issue is if your dataset has a high degree of heterogeneity, which is very common in Manningfield, for example, in ecology evolution, the Agile regression will have a non-nominal for comparative rate. That is the type one error rate The second issue is when your dataset has a high degree of data non-independence, which is also very common in some fields, like environmental sciences and ecology evolution, Agile regression will not perform nominally. It will have a non-nominal type of error rate. The third issue is the Agile regression, including all the existing methods is our low power to detect publication bouts. So this means your detection method will have a high degree of force negative rate. So how to resolve the issues? We propose a method to resolve the issues to help properly detect the publication bouts. This formula is mathematical representation of our method. Left hand again is the effect size. We still regress effect size over the sampling error because this is the only way we can get the relationship between effect size and the sampling error. But meanwhile, we need to account for heterogeneity by including other important covariance or predictors or some people call it modulator variables because as we said, the high degree of heterogeneity can lead to false positives. And then we need to organize one multilevel random structure to account for the data not independence. If the dataset also has a correlation in sampling errors, you might still need to impute with this study covariance metrics account for the correlation in your sampling error. Within this framework, we can get the accurate correlation between effect size and the sampling error to reflect the small study effect. The elegant, I mean, this solution is very elegant because you can detect the other very common form of publication bouts that is the time like a bouts by simply adding the publication here as a predictor. The slope beta two can reflect the relationship between the publication here and effect size. By doing so, you can examine whether your evidence is temporarily stable or not. I mean, whether your effect size is decline over time or not. One point it should be noted is that in essence, the error regression including our extended error regression is to investigate the relationship between the effect size and its sampling error. But for some effect size measures, for example, standardized mean differences, SMD, there is an inherent or natural relationship between effect size and its sampling error. So this will cause for spurtive because even though there is no publication bouts, there is the existing correlation between effect size and its sampling error. As we can see in this panel, if we keep the sample size constant, if we increase the point estimate of the effect size, the sampling error will increase accordingly. So this will lead to spurtive. So what we can do to resolve this, so a simple issue is to get rid of the point estimate of the effect size when calculating the sampling variance. The common way is to use the modified measure of sampling error. The effective sample size based on sampling error is a very common solution and works very well. I will show you this point in the later hands-on session. By using this method, we investigate the publication bouts in quality evolution using a very big data set. If you read this paper, you will find the impact of publication bouts if you neglect it in your data set. This paper now gets published in BMC biology in a very special publication form, registered report, which is pretty cool. I think this publication form will be adopted by many publishers in future. In this registered report, we found that publication bouts is very common and persistent across ecology evolution. This is the first very large scale meta-research evidence showing that publication bouts is a common and general phenomenon in ecology evolution. So everyone needs to pay enough attention on it to detect it and transparently report it. We also found that publication bouts can need to low-power our primary study and the exaggeration of effect size and even need to the wrong direction of your effect size estimate. You might wonder if we find publication bouts, what we can do to remedy its impact. Here we provide a solution. This is the formula which can be used to correct for the publication bouts. The parameter of interest here is the intercept. This intercept can be interpreted as the true overall effect or the population level effect after correction for the publication bouts. Or more accurately, you might call it a bouts correction effect. The principle is in your dataset, if your sample size is big enough, let's say you have an infinite sample size, then the precision is big enough, infinite. So that the sampling error is negligible and even near to zero, let's say we send sample variance here, this term to zero. At the same time, we send the publication year to zero to account for the time like a bouts and also account for other important predictor variables. Then you will get the intercept. This intercept can be interpreted as the true effect because there is no sampling error. So you will get the true effect. By using this method, we investigate the impact with the publication bouts in collusion. We found that around 24% reduction in the magnitude of the effect size after correcting for the publication bouts. Amazingly, we found 60% originally statistically significant effect became non-significant. So this is really not a good sign to this field. But we have to acknowledge our method is very general and can be extended to many publication bouts detection, but it also has disadvantages. One of the disadvantages is our paper, including all the existing deduction methods are low power. The average power of the detection method is around 20%. So given the low power of the existing method, the publication bouts detection method, what we need to do is to shift our interpretation philosophy. Conventionally, in our metallocids, we will report a publication bouts as a biocardomous way. Let's say there is a publication bouts or there is no publication bouts. But given the low power, we really need to shift from the statistical significance for cause to more biological significance for cause. Let's say we need to focus more on the market share and the precision of the publication bouts estimate. I will show you how to properly report publication bouts to report test results in the next section. Okay, so Alfredo, shall we take a break? We can take a few seconds brief before we enter into the world of R. Are there any questions so far? I mean, there were quite a few questions there. I get dizzy, myself looking at the questions. Also, I start dizzy looking at three places for questions. But if you have any questions, perhaps this is, we can have a little small break trying to answer them. Yeah. Perhaps the Q and A here session, if you can. Otherwise, I'm also checking the tweet there and the slack, so far there's nothing there. As far as I can see, we're not seem to see anything on slack or two thirds. Yeah, so I just wonder, is it time to put link to GitHub or we do it after presenting HTML? Yeah, that's a good idea, I think. Yeah, let's share it already. Yeah, so the GitHub web app contains copies of two of the papers we've been talking about, like FPDF files and also a few other things that will be useful and code, of course, and data. So we can share it already now. You mind taking care of that, Losia? Yeah, yeah, I will put it on slack because I think we cannot use chat here. So I just put in the question for presenters. The link is also on the Q and A as a reply to Wolfgang's question. Well, comment. There's a github.com link that I believe everybody should be able to see. Okay, I'm just putting now the link to GitHub. Okay, it's on. Brilliant. If there are some questions, okay. Wolfgang has a question, I'll try to answer, and Carolina. Wolfgang, I guess, by extrapolation, you mean that it is basically for a hypothetical scenario where the standard error is zero. Is that what you mean? That's already, so the question is I can read it. I mean, I'm not sure everybody has access so the intercept is an extrapolation and its standard error can be quite large due to that. Hence, it isn't surprising that one might lose statistical significance when one looks at this. Any comments on this? Yeah, so there is a little bit of an assumption when we run these models and we look at the intercept, which is the one that we consider to be the effect size adjusted for publication bias and is that that intercept assumes that the standard error is zero, which is a hypothetical scenario that cannot be. And Wolfgang has a good point that maybe that's also because of the standard error being quite large in that it might not be surprising that that's where we lose this statistical significance. In my experience so far, I mean, I cannot say it from the mathematical point of view. I'm sure Wolfgang, you can say that better, but from my experience, it's not also that common that that intercepts loses the statistical significance. So it is often the case that it gets smaller, that they adjust the effect size, but quite often it remains statistical significance, at least with the examples that I've used it so far. I don't know if you've found and Losia have any other comments on that. So I'm not sure I can answer really your question well enough. I'll move on to Carolina, sir. Okay, thank you for offering us a big supply. The next question, and probably I can explain the next question in the hands-on section, because the way we also go through the formula. It's okay. Yeah. But Wolfgang is saying that many of the examples you showed, your phone, the statistical significance was lost. In the, yeah, that's true. So I suggest, yeah, we answer Carolina's question and Wolfgang's, we can look at it during the tutorial. There's still one question from Peter. Given finite resources, we need guidance on the best compromise methods. I am not sure exactly what you mean by that. What methods are you referring to? Contacting authors and trying to obtain and publish data plus unreported standard deviations or what exactly do you mean by that? I mean, we face this problem also. It's always a trade-off. You have to, there are some practical decisions we have to deal with when we are doing evidence synthesis. I'm sure it's not something that only happens in ecology and evolution where resources are comparatively low with other fields like perhaps medicine. So, yeah, I mean, it is an important and difficult issue to deal with. And yeah, as I'm saying, we normally have to take some practical decisions based on whether we can do it or not. So, if really you have so many authors that you cannot really contact them all to try and recover unreported standard deviations, you're going to have to give up and do your meta-analysis in this case with whatever is reported entirely, which is of course a kitty and also can lead to some biases. But you can also reflect that in the discussion of your methods. So you can, for example, provide how many missing data or how many of the percentage of studies that do not report it entirely and things like that and include that information when you are interpreting your values, your results. It is really a very difficult thing to deal with. So I don't have a clear answer other than, yeah, we do what we can. And as you say, sometimes resources don't allow us to do a perfect job. That's why many times comprehensiveness, even though it is desired, it can be a little bit too topic, in my opinion. There are no other questions. There is another one in the Slack. What kind of bump in sample size would be required to have adequate power to use the modified version of Agar's test? Geofon, would you like to try to answer that one, please? Yeah, it's on the Slack channel. I can copy it into our chat so that you can read the monies. Yeah, if you can copy it, it's great. In the chat so that you have easy access to it. I'm not sure what Carolina means by bump in sample size. She's working. Did you circulate? What kind of increase in sample size would be required to have adequate power to use the modified version of Agar's test? We do not have a very clear kind of number of the sample size you used. But according to our, this is a very picky thing to deal with. Because in terms of multi-level version of Agar's regression, it's harder to say how many sample size you used to have enough power to detect the publication bubbles. But the more the better. We do not have a clear cutoff on this. But in principle, you can use it no matter the sample size. But we're going to be smaller, but it's going to be the same or worse for other methods. I think we have one more answer question from Peter Stewart about not being able to contact every author. What's the best we can do with limited time and money? And my take on this would be to publication by ourselves. But before that, if you cannot contact every author, focus on the most recent publication. So from our experience, trying to recover data from people is that if paper is older than five years in publication date, it's very unlikely anybody would even reply to your email. So we would only contact within the last five years. And of course, the more recent paper, the higher chance of success. But the emails will still work. And more likely that people will have raw data and they will remember actually the details of their study. So I think that would be the most effective way really. Just focus on the recent and get whatever you can and do a test for publication biases. I don't know if you're fine. I would like to add anything to that. All good. All right. I would suggest. Sorry. There's a follow up to that. How do I cost out my synthesis time money to understand the trade-offs between cost effectiveness versus completeness? Do I have enough resources to do anything sensible at all? You can always do a pilot before you start and try to assess how many papers will be included in your meta-analysis. If they are very cute, but you feel you can do more, then you can just broaden the scope of your meta-analysis. And if they are too many, narrow it down, get the question more focused. You can, let's say, focus on a specific taxon or ecosystem, geographic region. So there are many ways of changing the scope of your meta-analysis to keep it within your budget and time frame. So it can go up. It can go down. But it's best to be checked before you start. So we always do a round of piloting and scoping for every meta-analysis or systematic review. You can link to a paper that might help with this piloting. OK. All right. OK. If there are no more questions, OK. I mean, I'm really a little bit busy checking the three places for questions. If there are more questions, I suggest we move on to the tutorial you form. And you lead that. Lozi and I can continue looking at taking care of the questions. And we can also answer Caroline's question, original question with what everything means in the question, which I totally understand. And also we can try to tackle Wolfgang's question about the standard error and the intercept. All right. Let's get going then. Yeah. Welcome to the exciting hands-on sensing. In this sensing, everyone should be able to have practice using the data we provided here. So what you see is HTML file, which can be used to provide with your hands-on guard and code. And you can modify our code to your own data sets for your own publication by the test. Our tutorial heavily relies on two papers. The first is coming from Sni-Shinaka Kawa's paper. The second is coming from Wolfgang, is the author of the metaphor package. So if our tutorial have helped you some bit, please credit the original author properly. The main packages used in this tutorial will be two packages. The first is the metaphor package. I think many of you already heard about this and already used it. This package can be used for, in our tutorial, can be used for effect size calculation and the model fitting. The second package we use is the O-Chat plot, which can be used to visualize the model results by the metaphor package, which is really, very nice. And we randomly collected when publicly available data set our example to showcase the method to test and adjust for publication bounds. This is where our data set come from. So the first thing we need to do is load the necessary packages and the process data from the example paper. So we carved our data in the GitHub repository. You should be able to find all the data and code in the GitHub repository. After loading the data set, we usually will have a look at the data set. But for this data set, you do not need to worry about the meaning of each column, because this doesn't matter in terms of this tutorial. Before conducting the publication bounds, a necessary step is to calculate the effect size and the sampling variance. Here we use the standardized mean differences as our effect size measures. We will use this function from metaphor, which is really powerful. This function can be used to calculate almost every very common effect size measure. Here we specify SMD to the argument of measure. And then you run all the syntax. Then you will have all the effect size and sampling variance. And this column is calculated effect size. And this column is corresponding sampling variance. These two variables are the key quantitative variables used for publication bounds test. So the first thing we need to do is test for the publication bounds. There are two publication bounds form. The first is the small study effects. The second is the decline effects. So let's recall the mathematical formulas used for testing publication bounds. So regarding the one question in the last session, I will explain what is M and what is mu. The M is the sampling variance used to account for the sampling variance. I mean, due to the limited sample size in your study, you have to account for the sampling variance. And this term is used to account for the within study random effect. In Manlyfield, for example, ecology evolution, when studied probably report more than one effect size. So this will cause the issue of data non-independence. When testing publication bounds, you need to account for such data non-independence. Otherwise, you will have non-nominal type of an error. Hope I answered your question. And although this formula looks complex, but it is not difficult to contrast this formula using existing knowledge, especially we can use the powerful function from metaphor package. This RMM refunction is really powerful. You can use this function to fit very complex mathematical models, including this formula we used to test the publication bounds. The first thing you need to do is to add a unique identifier for each rule or each effect size. This is used to, this is aligned to this term, which is used to account for the data non-independence. And then you need to send all the continuous variable to use the interpretation of the intercept. The first continuous variable you need to center is the publication year, because this will be the predictor to test for the decline effect of the time lag bounds. This variable is aligned to this term. And then you need to center the other important modulator variables. In our case, we have two other important variables. The first is the latitude, and the second is the longitude. These terms are aligned with this term in the formula, which is used to account for the heterogeneity. And the last variable you need to create is something standard error, which is used to test for the small study effect. This variable is aligned to this term. And then you can construct the model using this syntax. You first need to specify the effect size, sampling, and variance. And then you organize your random effect structure, which is aligned with this term. And then you specify all the predictors, including the sampling error. Publication year and other important predictors. After running this model, you will get the model output. The output is very long. But in terms of publication by detection, the most important part is here. This section provides all the model coefficients of the predictors you fit to the model, and the corresponding hypothesis testing. This through represents the model results of the small study effect. And this through represents the model results of the small study time like-abouts of the time effect. This is the point estimate. This is the standard error. This is the test statistic of degree of freedom, p-value, and the confidence interval. As we said in our presentation, a more appropriate way is to replace the original sampling error by the modified sampling error. A typical one is the effective sample size-based sampling error. And the first thing we need to do is to calculate the effective sample size and then calculate the effective sample size-based sampling error and replace it by the original sampling error. Then we write a help function to calculate the effective sample size-based sampling error. Then we calculate it. And then we take the square root to get the sampling error. And then we specify it to the one of the predictor in the model. And this variable is aligned with this term. And after running this model, you will get the model output. This is the new test of the small study effect. After testing the propagation mouse, the next step we need to do is properly report propagation mouse results. We see that in the model output, we see the model slope beta 2, which is the effect of the propagation error, is not statistically significant. This indicates there is no strong evidence of decline effects. If we visualize the model results, we can find some different results. Here we use the bubble plot function from each other plot to visualize it. And this is the syntax to visualize the propagation mouse test result. Then we will get this bubble plot showing the relationship between propagation error and the effect size magnitude. We can see there is a slightly negative temporal trait in effect size magnitude. But remember, in our propagation mouse test results, we did not find any statistical significance of the propagation error. But actually the point estimate of the slope beta 2 is minus 0.058. As the time is going after 10 years, the magnitude of the effect size will decrease to 0.058, which is quite large. It is equivalent to a median size of effect size. If we only focus on the dichotomous reporting of the propagation mouse test, you will neglect the importance of evidence for the existing or decline effect. Actually, in this case, in this example, the effect size is declined over time. A similar interpretation can be applied to the test of a small study effect. The model slope of the sampling error is statistical significant. I can show you here. We can see the p-value is statistical significant. And the point estimate is quite large. Once we use the bubble plot function to visualize it, we can see there is a clear evidence of a small study effect. This is a kind of visual check with a confirmation of the propagation mouse test. So if you find a small study effect in a dataset, you really need to interpret your results and conclusion with caution. You also can use some traditional funnel plot to examine the small study effect or the asymmetry of the funnel plot. We can use the funnel function from a metaphor package. This is a typical funnel plot, which can be made by the metaphor package. You can see, previously, this funnel plot is asymmetric. Here is empty. After finding the propagation mouse, the next step you need to do is to correct for such a propagation mouse to see whether your result is robust and whether you need to be careful about your evidence. This is the formula just for the propagation mouse, which is very similar to the above one. The only difference is to replace the sampling error by the sampling variance. This is the whole syntax to contrast to this model. After fitting this model, let's have a look at the model output. You might still remember, for this formula, the parameter of interest is intercepted because it can be interpreted as the true effect or we call it both corrected overall effect. When we look at this intercept here, we see the true effect or population level effect is still statistically significant, although the magnitude decreased a little bit. From this perspective, although this dataset has some sort of smart study effect, but the evidence is still robust. If we compare the both corrected overall effect to the original effect, we can see this is the table showing the comparison between the original one, the naive one, and the both corrected one. Actually, the magnitude indeed decreased, but it is still robust in terms of the statistical significance. Next step, I think you might want to try your own analysis using our code provided in the GitHub repository. If you want to learn more about the population bias test, you can read our published methodological paper in methodical evolution. In this paper, we summarized the cousin practice of the propagation bias test and a deeper explanation of the method we proposed and as we illustrated here in this tutorial. Okay, so Alfredo, next show we start the practice. It's actually a very good question from Peter on the UNI that I think you could try to answer your phone. If you can find it. In the Q&A. Okay, Peter. In this one, we use effective sample size. Do you mean this question? Very good point, by Peter. Does the user mean one does not equal? Okay, so using the effective sample size to replace the traditional sampling error is the way to, I mean, for some type of effect size, when you calculate the sampling error, in the formula of the sampling error, there is a point estimate of the effect size. So this means there is a natural relationship between the sampling error and the point estimate of the effect size. So this will increase the possibility of your propagation bias test. But for some other effect size, there is no this kind of natural relationship. In such a case, you can still use standard error as the predictor to detect the propagation bias. But if in the formula of the sampling error, there is a point estimate of the effect size in the formula of the sampling error, it is better to use the effective sample size based on the sampling error as the predictor to test the propagation bias. Otherwise, you will get a non-nominal type of error rate. Hope I answered your question. Maybe just a small clarification. In principle, yes. But it depends on what effect size you are using. Yeah. So if you are using correlation, in principle, we don't advise you to use the effective sample size. But if you are using the standardized mean different, standardized means difference or the low response ratio, then it's when we recommend to use the effective sample size. And you can see there that in principle, you would get away without the standard deviations. Yeah. But not for the estimation of the effect size, or say perhaps, for the low response ratio, you don't use any correction. So if you don't use any correction for the low response ratio, sorry, let me try to replace it. You don't use any correction for the low response ratio and a correction for a small sample size. And then you go ahead and use the effective sample size. I believe you could get away by not using the standard error. I haven't thought about this question. So not 100% whether that's why I also added the paper that whether it's, it will be the case, but I think you could get away with it. So it's a good point that I would like to think far than myself. Hope that helps. Please make sure to add any follow up. We are using low response ratio. I think it could be, but I mean, that requires that you're not using any correction for the low response ratio. Yeah. So you're just using really the response of the two ratios. In the log, which I'm not sure. Yeah. Yeah. Yeah. I cannot really say more than what I just said. But certainly something I'm going to think about myself. Any more questions in the slide? Nothing. Twitter. And it doesn't seem to be very popular. Just good. And we use the other ones. Yeah, that channels. Any more questions so far? I mean, this is a lot of information, so it can be a little bit overwhelming. We totally understand, but please do not hesitate to try to say what is that? What is this? And then we try to clarify because later on, you can use all the material. But if we can clarify some potentially small things that are really not clear now, it's going to be beneficial. So, yeah, just go ahead with whatever random question you may have for something that is not clear, please. Or you can have practice and if you have any questions, we can address your question. Wolf, if you're still there, and you would like to give yourself some information about your question, I'm sure you have an answer better than us, please feel free to do it. Then we clarify and we all profit. You said we were going to talk during the tutorial, but otherwise I think our sales are going to remain quiet and wait for your questions, hoping you're working on your own data or our data. Okay, I see. Welcome. So the standard error is actually smaller for the bias adjustment estimate. So we can see here, yeah, true. So it goes from OO 22 to OO 20. Is this surprising? Perhaps not given Taylor's law, but maybe I'm going too far there. Because the mean is going down, the variance is also bad. It probably doesn't apply here. Maybe your phone, you could look while the folks are working on their data. Maybe you could look into if you can have access to our recent publication and see if you can see a pattern or whether standard errors become smaller for the adjusted. You mean, I mean, Wolfgang's question. Exactly. Exactly. If you could look into your data and the data we use for the BMC biology paper and look at the original estimates with their standard errors and then see if the bias correct ones tend to have the standard errors. If they tend to be larger or smaller, then maybe we can at least have a practical answer here. Yeah, I will definitely check whether this is the case. Yeah, this is a good point, Wolfgang. Yes, yes. Is the original estimate, the model from where you get the original estimate, does it include the same moderators or is a unimoderator model? Yeah, our BMC biology paper, we do not have any other moderators because we don't know which moderators were used by the original authors. So we only include the publication year and some error as predictors when adjusting for the publication. True, true, but per default those are two moderators that are absorbing heterogeneity and reducing the standard error as Wolfgang suggests. So it's actually a very good point, but you cannot have, basically we are comparing two models, one that has at least two moderators, two estimates less. So that could be the reason why the standard error goes down. Okay, I see Wolfgang's points, it's very, very interesting. Yeah, yeah, I will take a note on how we investigate it later. I'm going to take note of that. And there are more questions, open questions. Yeah, there are questions on Slack as well, regarding the formula. Slack, okay. Yeah. Is this like a supercharged version of the pet PEC? I don't know how you pronounce that. Okay, so I can tackle the second question. So I cannot really say if it is a supercharged version of the ETPC. This is basically like a regression version of the agar regression in a sense that we are making use of multi-level meta analysis or multi-level meta regression to do it here. So that allows us to account for heterogeneity and to model it so that for data sets that are heterogeneous as it is pretty much all the data sets in economics and evolution and I believe in other fields you also find what good levels of heterogeneity. We are trying to create this method that is robust to the present of this heterogeneity. For the second question, why are we using VI instead of SEI? The idea is that at the beginning you're going to be using the standard error. So the sampling standard error as your predictor to test if there is any evidence of asymmetry, any evidence of small study effects. Shall you find statistically significant, a statistically significant effect of this moderator, so the standard error. It has been shown by simulations that then the best way to not lead to bias estimates for the intercept is that you run the model with the sampling variance instead. That's why we are suggesting you have to use first one and then the other one. This is based on simulations. It's actually in our original paper and the methods in ecology and evolution, we cite the, I believe, two papers that suggest or show that if you don't do that, when the effect is statistically significant with the standard error, you will get, I believe, a down bias. Yeah, down bias or underestimate basically. Yeah, yeah, underestimate. So that's simply based on simulation data. And that's why we suggest you first do it with the standard error, the sampling standard error, and then you move on to the sampling variance if you actually find evidence. Hopefully that helps someone. There is a harder, a harder is happening. I don't even know what that is to be honest. I think somebody started by mistake. Okay, if you have any questions, I think it's better to write them here to go add another extra layer of complexity for answering the questions. Twitter, no questions. I was just checking our BMC biology paper because we have DIY for it but it doesn't link anywhere so it's been accepted that doesn't seem to be published. Yeah. Yeah, so there is a preprint on Echo ever archive with flings to GitHub and everything if somebody's interested in looking at that paper so we can post that link to the preprint for now. If people are interested checking it out as well. Is it okay. You found out there though. Yeah, I was sorry I was answering a question so I could not pay. I, sorry, should we post a link to preprint of a BMC biology paper because the paper is not out on the live on on the journal website. I cannot find it. I think it's a good idea. Echo archive I just put it on the questions on Slack and also share if I can post. Peter has a very good question. Sorry, I kind of answered it so it disappeared now. Okay, so the link to the preprint is now available at Slack. Yeah, I just try to type as I said one of the questions. In the question and answer on Zoom. Anybody actually trying to run the tutorial. If you are and you get stuck. Drop questions. And we will do our best. Peter's question is really good. We need to consult with you. We'll see whether the imputed assembly error can be the predictor with a small study effect. Is someone is writing a question on Slack perhaps. Can you say that I did not find a question. Good question or time follow ups as moderators. Yeah, so would this work with meta analysis where we use those a short time follow ups as moderators that's a samples question and select. I am not sure why it could not work. As long as they are moderators, you could include them in the model to have for eternity and see. There is still evidence of publication bias, both the client effects and small study effects after you've explained some of the eternity that those two moderators might explain. Given that I have never used something like those a short time follow ups, I don't know if they have any special feature that would make this different. I don't know if lotion on your phone have any other comments or if you want to reward or clarify that. I suppose it should be possible with any mother interest as long as I can see so long as moderators that you think that might explain some of the eternity you find in the original meta intercept on the meta analysis. Can you pass this question to me security this question. Yeah, it's on the slack I've read it on the on the chat here in the zoom for you. I mean for all the continuous variable, you just need to center them and put them as a predict one with a predictor in the meta analysis model. It doesn't matter whether it is knowledge or time follow up. As long as it is continuous variable. I think the complication here is that they, they are likely to be like auto correlated. So, you know, both are not independent point like it's not like you have, let's say, different level of proteins in the diet or different drug levels but both are with time at least yes, they are like subsequent measurements so you have kind of extra level of non independent that you need to model. Yes, the people is if you mean worry about the not independence between different dosage or time follow up, you can use a multivariate method analysis model to model this kind of data and not independence at the top but you still can add the sampling error as well as a predictor to test the smart study fact. So this doesn't matter. As long as you think the variable interest is the, the important a moderate, you should put them into the model. Another interesting question that I would like to answer actually is, do you have an explanation for why effect sizes can decrease over time so you found I'm just going to stop your sharing and I'm going to share some slides that I have for this. Yeah. If I can. Here. So anonymous attendee. I can see my slides now. We can see. Okay, I'm kind of trying to find my way on my screens. So, there are multiple reasons why we can find this decline effects. And here I'm going to just show three examples, what would be a true change over time imagine imagine you're measuring something like antibiotic resistance and what you observe is that there is like a resistance is increasing over time. And if you, if you wouldn't have thought that that can happen, then you would see a decline in effect sizes that is just driven by this resistance. So this is like a true change biologic of biological interest in this case. This is rarely the case but it's a possibility. Another one could be simply due to changes in methodology that you might or might not be aware. So, I think I had a, I don't know, I don't have the example here. Imagine, imagine a topic for when after a few years of study. Yeah, I think I have an example like imagine animal studies like when we look at biomedical studies from like 50 years ago, they did really horrible things to animals like you know chopping off legs and other parts of animals and other things, but over time because of ethical concern. There were lots of restrictions what you can do to animals so there was also technological improvements what you can do. So all those manipulation became much more subtle precise and subtle. So, I think because of that, you will also see smaller effects because just what we can do in terms of ethics and technology. So that's like a practical example. Yeah, that's it. I mean I had another example with the methodology where basically they changed. It's actually in this call each other and cool it cool in sky where they basically started studying animals in islands. So the first studies were on mainland and then they study animals and islands and there was actually a different between the hypothesis in the islands and the in the mainland. And I think that with having a decline in the effect that was driven simply because they change what they were studying. But yeah, I think it's a very good example of this is much better. And then, what we normally assume with these decline effects is that is because of systematic bias as you and I mean assume because sometimes we cannot really know for true for real. So to explore multiple potential explanations, let's say methodological or consider true changes, and then realize that it's like they're going to be a systematic bias so often it is the case what we call the winners course. The authors are the first published test of the hypothesis are simply the consequence of somebody finding by chance, the honeypot. I don't know if you say like that in English, where they by chance find support for the hypothesis normally that's going to be based on small sample sizes and then they get it published. It's a good journal or at least in the past in a good journal and then people after that try to replicate that hypothesis and they don't find evidence or the evidence is not as strong as it was originally. Other times it could be that publishing negative effects or statistically non significant effects may take longer either because you don't actually make the effort of trying to publish it. And you take a longer time to put time on trying to publish it or because journals, meaning editors and reviewers are going to push backwards like you're going to take a longer time to convince them so your test, even if you did then let's say in 2005 you may not publish them until 2010 and there's a delay and that's why we could potentially see these decline effects, etc. So there are multiple reasons basically that's what I wanted to kind of say I don't know if I lost and you found you have any more examples or reasons that those are the three I had in. Actually, we have a common peppery. On this topic, but let me try to find. Okay, there are more questions. Okay, the question from Peter. Shall I read it so in our data set we have long term agronomic trials, perhaps crop yield is reported across multiple publication but it is all from the same experiment. Does this create non independence that give me another headache. Yes. It does. Whether it gives you a headache. I mean it depends on the structure of your data set, you could potentially account for that. As long as you are able to know if the effects sizes come from the same population or experiment you could try to model that through the multi level meta analysis or having as a random effects population or experiment, and then you could even potentially go fancy and rather than using sampling variance a vector of sampling variance to wait effect sizes you could even model a matrix that does not assume that the random effects are independent or all the effect sizes come in from each population or experiment. I don't think it's a too much of a big headache as long as you are comfortable with multi level meta analysis. But yeah, it's a level of complexity you have to account for. So we are actually some meta analysis also methodological paper using lab ID, as you know, one level of non independence and sometimes it doesn't matter sometimes it does. If you have like one lab dominating most of your data set you certainly want to take it into account when interpreting your results or if possible statistically as well controlling for. And they are likely to use similar method similar side so definitely results from the same lab are not going to be independent. It's always good to use multi level structure to account for. I mean potential non independence in a data set, even though your data set doesn't have any non independence result will be similar to the normal random effects model so you will lose nothing. So you can find out that it doesn't matter you know you account for this or not it doesn't make a difference but it may make a difference. So it's good to check. And it's good to keep kind of you know be careful when you're extracting data and spot those patterns or you have a lot of paper from this place and this place. Actually, another thing it's good to extract data together from the same lab in same area. It, you can kind of spot inconsistencies and also kind of fill in the details you see okay this was definitely done around the same time, same people. And they miss this detail in this paper but they reported in the other you can cross cross fill the gaps between fabrication just because people usually don't report everything. That's also another question and slack that I'm trying to answer. But I will leave it on slack I think we don't need to really potentially. Someone let us know if that kind of helps and if not we will try to tackle it. There are more questions so do they're still empty slack I believe we tackle everything soon and so far. Okay, so we are still here for any questions that may arise. Particularly if you're playing with the data and you have any. Any questions we are, we are here. Another question on the Q&A. Are you also partially adjusting for positive publication bias by using the method the symmetry mentioned, or did I miss something. What exactly do you mean by positive publication bias. Can copy the question in the chat. Yeah, it's Q&A. You're on this on this zoom. I did copy it. Okay, sorry. By using. Okay, I mean the positive sign. What is the positive publication bar. I'm assuming you just mean. Public like small study effects. The fact that there's there are missing. Not statistically significant effects or even effects that go. In such a case. Our convention is still to adjust for this kind of. Because the constant method to detect a publication about is, I mean the power is very low. You, you, you didn't find the publication about, but there is still may be publication about because of the low power of the detection method. You're going to find them. So it is always good to correct for such about if possible. Yeah, but in principle that's precisely what we, you will be adjusting for. That's the idea that you, the, the effect size this adjusted effect size which corresponds to normally the intercept of this model will be adjusted by the small study effects after you have included in the in your model. Either the sampling standard error, or the sampling variance. So yeah, principle that's what we suggest. I actually there is a comment on Twitter by Gavin steward, he has a very good point. He says, I wish I could join test and report definitely adjust for is more controversial. But for include potential for distortion distortion in decision making. And it is true that I mean, whenever we try to adjust for publication bias. We have to keep in mind that it is a difficult task. It is an estimate because we actually don't know what happened. Yeah, when we find the evidence for publication bias we just find potential evidence for publication bias we cannot really know how many effect sizes are somewhere unpublished and the reasons why so you should always keep in mind that this adjusted effect sizes are estimates. It doesn't mean that the, the, the true effect size, once we've adjusted for publication bias is that. This is just a potential estimate that accounts for the evidence that we found for it, and it kind of normally brings down the amount of evidence for the hypothesis sort of say. Yeah, I don't know if that helps. It's one more question from Peter. Is there a off the rack function or vignette regarding how to create labs from our seed or author names. Do you your phone and loss, you know, if there's some kind of our package that allows you to, I'm assuming, having a data set reference with all the reference. You might analysis to try to fetch or scrap or, or see or org ID websites in so that you can automatically generate the lab ID variable as a random effects. Yeah, yeah, that's a brilliant question. So, one thing to do would be at the moment like I think what's most are friendly, I find a lens. Database which is not lens story. Open Alex. It's a new one. It has an app package. It doesn't have web interface, but it's our friendly. So you can connect the search for the papers, and it has quite a lot of geometric information about the authors. Institutional information is usually very messy so it's probably best just to do a network analysis of overnames and see if you can see any clusters, like people that appear repeatedly and connect publications into a clusters. So, so co authorship network basically, and then use those co authorship clusters or where they co offer many papers together, which means it's likely to be the same lab and try to use it as one of the moderators. Yeah, so it's something that people have. I think people started thinking about, but usually they do it manually. I guess for small data that it's okay, but if you have a really big one automating it will be, yeah, will be something good to try. Yeah, actually, I can show you something that indeed one paper on this topic, but this paper did not provide any article to identify the course of clusters. This paper used the math lab. I think, yeah, my lab. So, at the moment that I could a solution we did not have. So we only can do it manually to collect the author ID or something. Any more questions we still have a couple of minutes before we have to say bye bye. Today I feel like I would profit from having four screens rather than two. Thanks, Peter for all the questions. We're very active. Sure now my screen again. Lack empty. I'm assuming with their tool. Can you provide the slack channel in the chat because some Peter is asking. Okay. Thank you. Okay. Yeah, you need Peter you need to browse on slack. If you go tunnel and go to manage the open the little like triangle on the right from the tunnel, go to manage it. And then you will see browse channels. And you should join a tunnel questions for presenters. They are only five channels all together. But you need to choose that one questions for presenters. And this is the one used for all sessions. Where you can ask questions. There are more questions. Wolfgang. Okay. Why did you not use the alternative method of computing. Since we do this. Do you get a restart? I guess Wolfgang is talking about the example, the specific example that you use your phone in your tutorial. Did you use the effective sample size? I'm assuming. Okay. So, actually, we showed this method we used the alternative method to computer the same people in the tutorial. I think we use both. I cannot remember from the top of my head. Okay, I can show. I'm not in figure two, at least figure two. Actually, I had this. But I deleted this actually. In previous version I had this in figure two alternative method to computer the sampling error for the figure two. But I want to put this tutorial as simple as possible. So I did each of that figure actually. Yeah, but I think Wolfgang has a good point because also for the adjusted effect, it seems that we did not use the effective sample size. So that's something we should perhaps update. If that's the case. Yeah, yeah. Maybe something we should update. Yeah, good spot. And everybody with the GitHub link will get access to this updated one. So that they know how to do it actually with the effective sample size, which is the method we suggest. Thanks Wolfgang. Yeah, thank you. We had this in Paris version, but I deleted it. Good spotting. Thank you. I think we have to get going. Are there any last questions? I see. So Wolfgang is already doing the updated GitHub. Very cool request and we have updated. Yeah, actually we borrowed a lot of stuff from Wolfgang's metaphor package. Yeah, but he's finding that the adjusted effect then becomes bigger. Yeah, I mean, I cannot really say why then. But Wolfgang, is there a statistically significant effect or a statistically significant evidence of small study effects when using the effective sample size? Yes, this can happen because we did not account for enough heterogeneity because we did not get the full list of the modulated variables. So the beta 0 can go up, also can go up and also can go down. And I think about it now without looking at the data. So this actually is because there are lots of the categorical modulated variables we need to put them into the model. Then we will estimate the mechanized mean. I mean it's about the character effect. But since we are not able to collect the full list of the modulated variables so the beta 0 probably is not as a real effect size. I mean the true effect or population effect size. So it's just approximate. It can go up and also can go down. I think we should, because we have to leave it here, I think we should update the GitHub, the tutorial and then provide a bit of an explanation, read an explanation on the Slack channel for everybody so that they can read it too. I think that would be the best. And then Wolfgang can also comment afterwards. We can have a chat about it. I think it's a very, very important point. And now we are running out of time. I hope that's okay Wolfgang and all. We gotta leave it here before we leave it. Yeah. Just to close the session. That's it for this workshop. So I hope you enjoyed it. I hope it was useful. We did our best to answer your questions. I'm sure there are more questions. If you need any extra help, so I think we can continue talking on the Slack channel and we'll do our best within our possibilities, time constraints involved, and to answer them. And we'll see you at the next session. We have an amazing week ahead. The schedule is packed and full of exciting talks and workshops and webinars and what