 And we're live now. All right, welcome to ESMACON 2023 and controlling for publication bias challenges and future directions. This session is being live streamed to YouTube. Automatic subtitles should be available shortly after this event, and we work hard to get these mainly verified as soon as possible. If you have any questions for our presenters, you can ask them by the ad ES hackathon Twitter account by commenting on the tweet about this session. If you registered for the conference, you can also comment and chat with other participants on our dedicated Slack channel. We'll try to answer all questions as soon as possible. We would like to take time to draw your attention to our Code of Conduct, which is available on the ESMACON website at esmaconf.org. Now, without further ado, let's get to the topic of our panel. So, typically our goal in systematic reviews and meta-analyses is to provide a comprehensive, unbiased synthesis of all the available evidence that we have in a research field. And I think that most people would agree that this goal is seriously threatened if we have reasons to believe and we often have reasons to believe that some results are systematically missing or underrepresented or maybe even distorted in the published literature. In practice, controlling adequately for such broadly speaking publication biases plural in meta-analyses remains challenging. And our goal was therefore to bring highly experienced field expert together in this panel who would help us shed light on the state of the art in controlling for publication bias and associated problems such as questionable research practices and how we can implement these methods in practice using R. This is why I'm super, super happy to introduce our panel for today. First, Maya Mathur from Stanford University. Due to time zone conflicts, Maya has nicely agreed to pre-record her answers which we will be playing first whenever we introduce a new question. The entire interview with Maya has also been uploaded on a supplementary webpage for this discussion, which can be found at protaglab.org slash as Macon 23 will also post that link. And this webpage also showcases some representative work of the other panelists today. Now to the other panelists who are with us here today which are Pranishek Bartosz from the University of Amsterdam. Nice to have you here. Met page from Monash University, Robbie Van Aert, Tilburg University and Wolfgang Fichtbauer University of Maastricht. Again, we are super happy that you could all join us today. Lastly, I also want to introduce my co-moderator for today which is Eve Blesen from Shakhty Berlin. And Eve, why don't you start us up with the first question? Hello, everyone. Hello, Matthias. As Matthias said, we have five distinguished panelists today who will discuss an unknown number of questions on my of, I could say one of my biggest nightmares publication bias. I would like to open the panel with a very broad question on the different flavors of publication bias. Namely, how is publication bias typically conceptualized? All right, we'll start with Matthias answer. So a typical conception is fairly broad. There's a lot of different flavors of publication bias. So I think the most general definition we might give is something like selection where studies are more likely to be published and make it into the meta-analysis if they favor a desired hypothesis. Now, we can already see that there's kind of a lot of more specific conceptions that are within that umbrella. So first, what do we mean by a study favoring a given hypothesis? Do we mean it has a point estimate that's positive, let's say? That would be, for example, what funnel plots are looking for? Does it mean the study is not only has a positive estimate, but also it's significant? That would be more like certain selection models or maybe does it mean that the result is significant in either direction? So what we mean by favoring a desired hypothesis depends on sort of the context of how we think publication bias operates and this really does have important implications for what methods are going to be unbiased in any given setting. And then a second source of ambiguity with that definition is when we say selection, what do we mean by that? So sort of a traditional conception of publication bias is one where there's sort of a collection of studies that investigators produce and then the publication process, let's say journal editors choose among those results which ones are gonna get published and hence get into the meta analysis. That's a form of very clean kind of missing data where we have a well-defined population of results that are candidates to be published and then among those only some get published depending on their results. I think especially in relatively recent years it's become increasingly clear that's not the only form of selection that happens in real life. There's also potentially p-hacking or other forms of selective reporting that are happening within studies. So we can think of it as investigators will perhaps fit multiple models or let's say analyze different subsets of the data within their study to produce a single, let's say like candidate result to publish or multiple. This is, it sounds similar to publication bias but it actually manifests very differently and it's actually a much harder bias to deal with because we no longer have a sort of clean like missing us at random setting where what's getting published is just a sample of what's available. Rather any given study if there's also p-hacking will be biased even for its own true effect. So it's not just sort of a, it's not just that the published studies are not representative. It's that the individual studies themselves are now also biased. So just to say that the umbrella of publication bias is quite broad and there's sort of advanced multiple definitions within it. Thanks so much. I was just gonna say, great, we can move on to the next question, right? Yeah, excellent answer. But maybe does someone have something to add or footnotes to that really great answer? I would say, I don't know if it's the case with everybody on the panel, but I think at least in medicine traditionally we've tended to use the more broader or encompassing term of reporting bias or dissemination bias to capture all those types of bias that Maya was referring to and tends to restrict publication bias to just the simple example of the publication of a study versus the non-publication of a study. And I think another helpful distinction that sometimes we've been making is what effectively Maya outlined was two main scenarios. One is which information is missing and that can either occur because the whole journal article is not published or because the article is published but some of the outcomes that were measured are just not reported at all versus the other scenario where the results that are reported have been cherry-picked based on running multiple analyses, taking multiple measures and only picking the most favorable results. So at least in my circles, we've sort of tried to encourage the use of the term non-reporting bias but it hasn't really caught on. We haven't been that vocal about it but we kind of use non-reporting bias to refer to the non-publication of studies and the non-reporting of study outcomes and keep that separate to the cases where you do have a result available that might have arisen by a pHacking another in the various activities. Along those lines, I mean, if one wants to make things even more complicated, one could just ask about the representativeness of the studies or the evidence more broadly that you have included in a meta-analysis or systematic review. So it doesn't even have to just pertain to the effect sizes or the statistical significance or which ones are being reported but you can even ask about more broadly are the types of participants in the studies that I have representative for all of the research that has been conducted on a particular topic or even more broadly to the population that I want to generalize my findings to. So I mean, that opens up a whole different kind of worms but one could even just look at publication bias just as one small piece of this much broader issue. I think this is definitely true also because pHacking also has different effects on, for example, the results of the meta-analysis. So you have a lot of different types of pHacking. For example, you have optional stopping or also called data peaking where you repeatedly add participants to your primary study till you find a statistically significant result and this hardly affects the effect size of the study. So this will also not be a problem for the meta-analysis itself, only it only has problem for the standard errors because the standard errors will be affected by this procedure and then it can in turn also bias the meta-analysis to some extent, because usually the inverse of the variance is used as ways for this meta-analysis. So this is one type of pHacking that doesn't affect, mainly does affect the results of the meta-analysis whereas on the other hand, there are also examples like, for example, outcome reporting bias, what definitely affects the result of the meta-analysis because then outcomes are selectively reported, probably depending on whether they are statistically significant or the effect size. And this will definitely have a consequence for the results of the meta-analysis. So therefore I think it's important to draw the station to on the one hand, publication bias and these pHacking behaviors. Yeah, but when you connect back to it when people want to conduct a meta-analysis and correct for the publication bias, I think they kind of also expect to correct for pHacking because they just want to get unbiased estimates right. And I don't think they are really like trying to focus on representing of the population when they do publication bias adjustment because that's more of a qualitative assessment that they can do when they read the articles. But I would say they somehow try like, want to get an estimate that's representative of the effect sizes of the underlying true effect in the literature. So I think it kind of then boils back to publication bias, pHacking, et cetera, together to some degree. Although defining what do we really mean by the true effect? I mean that in itself, right? That would require a very clear conceptualization of who are you talking about under which circumstances, right? So I mean that in itself is super difficult, right? So. Okay. Anything else I know? I mean, it's an incredibly difficult question in a way to give a really succinct definition. Anything else you want to add? Otherwise we would proceed to the next question. One more point that I think is important to emphasize because what I quite often see in the literature is that people interpret methods like, the results of methods like Aga's regression test or the event correlation test or the term of film methods. They interpret these results as evidence for publication bias. If they observe some difference between, for example, the traditional meta-analysis and the, if you apply the term of film method, the estimate corrected for bias. Well, I think it's very important to emphasize that these methods that are all based on the film plot, that they are actually assessing whether there are small study effects. A small study effect implies that there is, that you are studying whether there's a relationship between the effect size of the studies and the precision of the studies in the meta-analysis. And this can be caused by publication bias, but there are also other causes for small study effects. So it's very good, I think, to be aware of the fact that if you observe, let's say a small study effect, that this can be publication bias, but it can also be something else. And then just to make it even more confusing that the issue, I think that most people tend to ignore is that they're using a lot of these methods to detect evidence of some pattern observed in the data suggesting that there might be publication bias, but few of them go the next step to then consider, does the extent and amount of sort of missing studies and missing outcomes actually, is it going to influence my meta-analysis at all? Because there can be some cases where you find some studies that don't report an outcome that they said that they were going to and it looks like they probably didn't report that outcome because it was non-significant. But if it was, say, a trial of 10 participants, including that in your meta-analysis of 10,000 patients, is probably not going to have any impact on the pulled results. So I don't think we do the best job of dealing with that. And so we often see people saying in the literature, oh, I've got a funnel plug that suggests that this meta-analysis is biased, but they don't necessarily consider, well, is it enough to have shifted the meta-analytic effect? So that makes it even harder, unfortunately. Thank you so much for laying the groundwork for our entire discussion. We will come back to some of those themes when some tools useful, which context might be dangerous for these interpretations, whether this skews the entire knowledge we believe to have. But due to time constraints, I would suggest that we move to the next question. Matthias, please go ahead. Yeah, so the next question was a little bit, or is a little bit more practical. So of course, as meta-analysis, we also kind of want to know how to control or adjust for publication bias in R. So the question was, what methods to adjust for publication bias are there in R currently and which approaches can you recommend to novice, experienced meta-analists? And again, Maya has also commented on that, so I'm gonna play that first. Yeah, so there is a whole really great suite of publication bias methods in R. Many of my co-panelists have produced some really excellent ones, and so I'll sort of let them speak about their own packages, but so I'll kind of focus on the ones that I've been involved with. So my group has put out a few, one is called publication bias, and this is an approach that is sort of conceptually similar to a selection model that is considering publication bias that operates on statistical significance. But what it does differently is it's a sensitivity analysis rather than an estimation method. So it asks, how severe would publication bias have to be in order to explain away the results of a meta-analysis? And the objective here was to kind of obviate some of the more challenging assumptions that are common in publication bias methods, such as needing to have a large number of studies, needing to have independent point estimates, needing to have normality. Not all methods assume those, but those have been common methods and common stumbling points when we have meta-analyses that do not fulfill that. We also have a two more recent packages, which I'll touch on a little bit later around the concepts there, but one is called p-hacking. And so this does sensitivity analyses for p-hacking, which has been sort of a really challenging issue. And the second one is called multi, or I guess it's the third one now, is called multi-bias meta. And this one does sensitivity analyses that are conceptually similar to publication bias, the package, but also accommodate the possibility that studies have internal bias. So for example, some of the studies are confounding because they're observational, but then also there's publication bias on top of that. Again, I'll touch on why that's a challenging case later. Thanks again to Maya. Any other approaches? I mean, Maya has mentioned it. Many people here have contributed packages or functions. Any other approaches did you might recommend or want to mention that maybe meta-analyses can use in our in practice or shouldn't use? Well, I think at least to mention trim and fill is a technique even though it has received quite a bit of a bad reputation in recent years. So that's the sort of kind of an imputation technique where you sort of look at the funnel plant or the method looks at the funnel plant and tries to sort of impute the missing studies on one on the two sides. Then you have the pet-peasy methods which take this regression test where you sort of fit a regression line through the funnel plant to see if it's asymmetric and you sort of look at the intercept of that regression model, which sort of extrapolates to a study of infinite sample size where the standard error, for example, is zero. So that's sort of considered then a corrected effect. And then you have selection models and P-curve, P-uniform, while Robbie can say maybe a bit more about that. Those you could sort of also consider a special cases of these selection models. When you try to model the selection process in itself, well, sort of this idea that studies sort of not representative due to some process happening out there in the world. And you can try to capture that process if you have enough data and based on that adjust the effect estimate and try to de-biasing. So I think those are three general types of methods that are quite popular. Yeah, and maybe I can build on top of the answer because as it's clear, it's like so many different methods that one can use to adjust for publication bias. It can often also yield kind of different estimates. And it might be difficult to like pick and decide which one of them you should use at the given scenario and maybe we will get to that later too. But it's also relates to what we are trying to do here in at UWA and we are trying, we build this ROBMA package from my package when we try to combine the different publication bias adjustment methods together with Bayesian model averaging. So we fit all of the, all like we did the different types of models, for example, the selection models, the pet piece and we look how well are the data described by those different models and we combine the models together based on their prior predictive performance. So if the models predict the data well, then we give them more wave and then we base the overall inference on all of the models together. So that kind of allows us to kind of get the inference back together based on how well the data are described by the different publication bias methods because in different settings, different methods might perform better and we might don't know the true underlying conditions for those methods. So we can try to assess it based on the observed data themselves. Maybe one method that is not often really considered a method in itself, but I think that should be mentioned here is if you actually have the, so if you're really focused on this type of bias due to published studies being not representative because of an overabundance of statistically significant findings or positive effects, whatever, I think a method that should always be considered if possible is to compare the effect sizes from the published studies with if you are able to get them some of these unpublished studies from thesis, dissertations, et cetera, right? So, I mean, this is not directly an adjustment for publication bias, although well, if you find a difference there, you could sort of use that to maybe unbiased the published effect sizes, but in any case, I think that is a very straightforward technique that if you have unpublished studies that you can always use. And what I really like about this is that many of these other techniques, well, they are based on sort of an extrapolation, like Pat Peezy, where you're extrapolating to a study of infinite sample size or you try through really fancy statistical machinery, capture, selection processes, right? So, I mean, it's nice that we have these methods, but it's all based on sort of an unobservable process happening out there. And if you have published and unpublished studies, you can compare them and you can see what the difference is and that is much more manifest than while using these fancy statistical methods. And that's me saying this as a statistician, right? So. Yeah, maybe if I quickly jump, you can also say similar thing, I will treat these reports, right? You can also look at the studies that have very high quality of reporting, pre-registration and compare between those studies and the rest in a similar manner, which also ties back that I believe you would have no publication bias because all of the studies would be conducted in such a clear manner, pre-registered, published with clear pre-analysis funded. You wouldn't need to do all of this statistical archeology to recover the missing studies and extrapolation, right? I think another thing about these methods that Wolfgang's talking about, similar type of approaches is if say you're doing an assessment for published studies, even you don't even have to, sometimes you don't even have to look for, sorry, I think you should always try and find a registered report or a protocol or a trial registration for a study, but even some study authors themselves can essentially tell you that they've selectively reported their data because they'll say what outcomes they plan to measure in the method section and then lower the hold, some of them just don't appear at all in the results. So that raises your suspicion, okay, they'll probably select or they'll report it in a way that's not really amenable to meta-analysis, sort of just saying there was no difference between groups, not telling you the direction of the effect or the exact p-value. But the thing I like about those methods too is that I don't know if others have noticed this, but at least by focusing on how systematic reviewers deal with publication or reporting biases, I feel like they're tense, not in all authors, but there's sometimes there's a tendency in some authors to want to not claim evidence of publication bias because that would go against their theory that something works. And so it's somewhat easier to say, or maybe it's not, but it feels like it's somewhat easier to run various statistical methods that account for publication bias, report only the one that says there's no problem or report a final plot that says there's no problem to prove that they have no concerns about reporting bias. Whereas if you've sort of done this story, I guess that sort of person probably is not going to do the heavy detective work of actually identifying evidence of reporting bias, but sometimes if they have done that, then that's kind of right in their face that there's something missing here and maybe that's a bit more objective evidence for them. So who is going to look for publication bias and publication bias test results, right? So that's a nice project right there. Well, actually we did some project like that when we tried to analyze a very big piece of meat analysis and look at the results of publication bias test. And actually I have a meme like this from my class with like publication bias, detecting publication bias, publication bias and publication bias. So to comment on what Natchez said, I've seen that, but I've also seen a quite opposite phenomenon where it's almost, well, almost on vogue to actually make sure that you can demonstrate that there might be a publication bias going on. So I've seen both. I've seen also really kind of strong attempts to if anything suggests that there might be publication bias of all of these different techniques, then okay, that already calls into question the results. So I've seen this go both ways. And I think this is a practical problem, right? Because we have this wide variety just from a practitioner's perspective and it makes it easier. You have so many research degrees of freedoms to sort of start weaponizing these methods that are based on untestable assumptions. And I think this is also a little related to another question that we have, of course. I know it's a tough one, but which of these methods performs best, because this is something that I also get asked a lot. So we have these methods, so which one should I use? Which one is the best? Even maybe you've prepared this question and introduced it properly, but I think it fits you nicely. Yeah, especially because we opened up this discussion, performs best for what? To demonstrate that there is publication bias or to demonstrate that there's no publication bias or to actually find an answer to what I would say, normal meta-analysis that don't use these tools as weapons would like to know. I would suggest that we also get Maya's perspective and we ask her to answer which of these methods typically performs best or might be best suited for which specific context. So which method will perform best is gonna be very contextually dependent, unfortunately. And this is because the methods generally make just different assumptions. And so, ideally if we knew sort of the mechanism of the publication process and we knew sort of statistical characteristics of our meta-analysis, like are the true effects normal? Are the point estimates independent? What's the sample size, things like that? If we knew all of that, then we probably could choose a method relatively easily. Now, the challenge is that in general, especially when it comes to the mechanism of the publication process, we can make it educated guess, but we don't usually know for sure. And so, just as one example, if you think that selection primarily is about whether a study is significant or not, and I think that's quite plausible in many scientific contexts, rather than operating on sort of favoring larger point estimates, regardless of significance. And if you think that publication bias affects all studies regardless of size, then you are in a context where funnel plot methods will probably not work well. Not necessarily not work well, but you're outside the kind of classical case where those methods are appropriate. And so that the situation I described is one where something like a stepwise selection model, so something that assumes that the publication probability increases once a study becomes significant would probably be more appropriate. A student of mine, Maximilian Meyer, led a tutorial paper that kind of flushes out, especially these differences and assumptions between selection models and funnel plots with some sort of applied recommendations for that. And I do actually think that for this reason, there's probably way too much emphasis on funnel plots alone. In that paper, my student found that, essentially funnel plots remain almost the only method that's used in at least medical meta-analyses. I do think psychology has been a bit more sort of forward thinking with methods, but selection models themselves have assumptions. They typically require pretty large sample sizes to work well, and they have distributional assumptions, which if violated might cause the results to not work. So again, it's sort of, I think ideally we would be fitting multiple methods with an awareness of their differences and assumptions and not just blindly fitting multiple methods and comparing them, but fitting multiple methods and then thinking critically about, am I in a context where this method's assumptions makes sense, for example? I think it can also be wise to try to include some methods that are either more statistically conservative or that use fewer assumptions, often at the expense of precision. So an example of that is the sensitivity analysis. I mentioned where what you're losing is the ability to estimate how severe publication bias is. What you're gaining is a reduction in assumptions. And so doing a method like that, in addition to something like a selection model or maybe in some context a funnel plot would help sort of characterize what you can say under fewer assumptions, not no assumptions, but fewer. And I know one of our panelists also will be talking about some cool methods that do sort of Bayesian model averaging over multiple types of models. This is sort of a similar idea of being more like epistemically conservative about what mechanism we're willing to assume for publication bias. And in practice, multiple methods may well disagree. If you want some concrete examples, the paper by Maximilian Meyer is one to check out. I also have a preprint about p-hacking that kind of discusses sort of how badly things could go when we have p-hacking in addition to publication bias and how that can produce differences among methods that can be explained in terms of differences and assumptions. Okay, thank you, Maya. Same question goes to you. What would be best practices in a world where we have so many different tools and so many different specific contexts? First of all, this is a very good question but also very difficult to answer because I think, yeah, what Maya said, it really depends on the characteristics of the meta-analysis. So what I would do, and that's basically what she also suggests is first check which methods you think perform well for the characteristics of your meta-analysis and then apply these methods. And also very important, of course, is to fully report all the results of these methods and not be selective in the reporting of these results. And what you might also do is to just write a preregistration where you already say, okay, I'm going to apply these publication bias methods and then you have already, yeah, at least carefully thought about what methods you are going to apply and then if you're also really going to report these methods, I would say that this is also an important and good thing to do. Okay, I'm not a statistician so I have no idea how to answer this question but what I do want to ask is, are there sort of decision aids for meta-analysis to help them sort of, so is there sort of like, if you could feel out if you answer a couple of questions about the characteristics of your meta-analysis that it would at least funnel down the list of possible methods one could use and ones that would be considered appropriate for their meta-analysis that someone has maybe developed or are developing because I think, yeah, back in 2013, when I know there was a paper that published, sorry, a systematic review of these type of methods found that there was 100 of them. I'm sure there's probably double that now. Or maybe not double, but there's already a lot but if there's a way to help sort of, I guess funnel us in the right direction that would be helpful or maybe the answer is don't use any of them. I'm not sure, I'm sure that's not the answer but I'm curious to know if people have any advice on that. Maybe I can answer the first question. Rather there's some applications you can use to determine different methods. There's paper by Coutret all in cycle, advances in methods in psychological science that has some application like this where you can specify what's the expected effect size, what's the expected heterogeneity, what's the degree of publication by us, what's the number of studies and whether you are interested in bias, written in square error, et cetera, a slightly improved version of this a bit more simulation is by Tonkin Reed published in RSM in 2021, 2022. And they also updated this application to do that. The problem that I see with this kind of is that you need to put in the expected heterogeneity, the expected effect size and the expected degree of publication bias. And I don't really think you are able to do that before you analyze the data. And when you analyze the data with some of the methods, then all of these estimates are going to be biased based on characteristics of those methods. So you run some catch 22 problem when you are making your specification on your best estimates. And maybe you can do a few rounds but you have no idea where you will end up. So I don't really think this is, I think it's a good attempt but I'm not sure how feasible it is to actually perform this to get the actually best properties of those methods. And of course you can say I'm biased because I have my own methods in play that I try to develop. They try to deal with this based on the Bayesian model leveraging as Maya mentioned before. So I personally think this is probably like one of the better ways how to deal with this. But that's of course my opinion. So you can disagree with that. We have any other recommendations for us practitioners who feel a little bit lost in the woods? Yeah, really difficult to say. Matt, I was going to say I am a statistician and I have no idea how to answer that question. So it's just really tricky. It's very difficult to answer. So if we're going with our own pet methods I personally do like selection models. I think they're really nice. They have some sort of nice statistical properties. I think they fit quite well with what we are often worried about. And it's one of the methods that is also considered of course in the model averaging that Funtichek mentioned. So I think that's one of the methods that is part of this whole package, right? So the only difficulty with selection models is that there are also again many of them. There isn't just the selection model. People often consider a relatively simple one. There's sort of three parameter selection model where you just make a distinction between the likelihood of selection for studies where the P value is below 0.05 or above. But you can have many other types of selection models with more smoother curves or many more steps in this step function. And so then the difficulty again becomes well, which one do you choose? Or which one do you apply? Or do you apply all of them? And what do you do with this multitude of different results? Well, we actually specify a few of them in the package itself because you're right that there's multiplets different weight functions you can do. And actually like in practice, what we found that what usually plays some role is also direction of the effects that effects in the supportive directions are also more likely to be published. So basically if you have one set of P values then on P value of 0.5, that's additional step. And I think the current implementation has six different weight functions that differentiate between significant, marginally significant, the correct direction or not and different combinations of those assumptions. But yeah, I also like selection models. I like the statistical model behind them. And one thing that Maya mentioned was that usually it's a lot of studies to fit them. I don't really think it's that true if you go to the, if use Bayesian statistics because you can do some restrictions on the parameters which I think are quite sensible and that help with convergence a lot. For example, that if you have statistically significant, statistically non-significant studies, there's can be ordinal restrictions of the publication properties. It's quite unlikely that non-significant studies are gonna be much likely to be published than statistically significant studies. And these few restrictions that you can implement on the parameter space makes those models much easier to converge as well as some very small regularizations that you can input with the parameter distributions on the parameters. So there are quite big gains you can get from Bayesian statistics, especially in those cases when there is very little data by regularizing some of the parameters or some of the models which helps with the convergence and makes the behavior much more stable. And I think we also show this in the simulation studies that we run that the root mean square error of the methods that we show is usually one of the smallest. I mean, this can be interpreted as a playful selection models, quite broadly speaking. I just want to ask, so we also talked, of course, about small study methods, funnel plot-based methods would be the takeaway that these are done too often that we shouldn't use them as regularly as they're still quite often done at least in the literature that I know or that I have some kind of dim expertise in or do we really not know? It again depends on the characteristics of the meta analysis. So for example, I've also seen situations where this pet piece method performs quite well. So therefore I would definitely not say, do not use any of these methods because this pet piece method and also I think also the agus regression test if you have a large number of studies in your meta analysis then also this test then for small study effects is actually also I think a good way to go because generally has only if you have less than 10 studies then it is especially problematic with respect to statistical power. But if you have a large number of studies then this should be okay. Of course, depending on the true amount of publication by small study effects. So this is all hard to say but I would definitely not say that do not use these kind of methods. And these methods generalize quite straightforwardly to more complicated models. So while conceptually you can generalize the selection model to a multilevel multivariate meta analysis in practice that is extremely difficult actually how to implement this but these sort of yeah, pet to PZ regression type techniques you have to think about what kind of selection are you trying to pick up here? Are you thinking about missing studies or missing effect sizes within studies? So you have to start making sort of distinctions between that but they do generalize quite easily to this to these more complex models and that is one of their great strengths. Yeah, that's definitely an issue for selection models and not only how to implement it but also how to estimate them if they are implemented because for example, we tried implementing them and we have a implementation for multilevel selection models but we are unable to run it in reasonable human time to estimate any studies with more than five estimates per cluster. So that's another limitations in these cases. But yes, I think pet piece is quite good in many cases and we also included actually in the Bayesian model averaging ensemble because those methods generalized very well and they have usually small bias although very high variance of the estimators and we also run some simulations with only some Bayesian restrictions on the pet piece regressions. For example, restricting the coefficients to be only positive on the regression of standard errors which also seems to help a lot with the variance of the estimators. So yeah, there's also some improvements that can be made just in applying those methods even if you don't wanna go straight to the full Bayesian model averaging and combining all of those methods together. Wonderful, thank you so much. I'm afraid we're almost out of time. Thank you so much for the wonderful discussion. During the discussion I was reminded of this saying about missing data. The best way to deal with missing data is to have none. And maybe that's also true for publication bias. It is a difficult topic but I think maybe it was a, at least I found it very illuminating. And yeah, so again, that's it for this session. We hope you enjoyed it as much as we did. Thanks to all of our panelists. Again, we've also posted a link to the supplement with a few more references on some of the work and see you at the next session. Thank you so much. Thank you.