 Hi everyone, my name is Martin Schumi and I'd like to tell you about the Evidence Synthesis Package, specifically how it supports combining effect estimates in a distributed research network without sharing individual person data. This work was a joint effort by me, Yong Chen, David Madigan and Marc Souchart. So let's talk about the context where you'd use the package. Nowadays it is very common to see distributed research networks where there are multiple sites that have data we wish to analyze. In my work, these are, for example, hospital electronic health records or insurance claims data. For various reasons, including patient privacy, patient-level data cannot be shared. At each site, the data is behind a firewall and is not allowed to leave. Fortunately, each site has converted their data to a common data model, so the structure is standardized. Now, if we still want to perform a study across these data, one of the sites can take the lead. They develop the code needed to execute the study. These studies can be quite complicated, for example, using complicated logic to define the exposures and outcomes and using propensity scores to adjust for confounding. The code is distributed to the participating sites. And each site then runs this code locally. The output is aggregated data, so no longer patient-level data and can be shared with the lead site where the results can be synthesized. Just to give you an example, we performed such a study a while ago where we compared the effect of two drugs on a potential adverse effect. You ran the study across a network of 10 databases. Only six had enough data for this question to produce an estimate, and we combined these estimates using standard meta-analysis, assuming random effects. One question we'd like to answer in this context is, what information can we share? We can't share patient-level data. And because we typically have a somewhat complicated model, such as a propensity score stratified time-to-event or conditional porcelain regression, we can simply share 2x2 tables. In the example I showed earlier, we simply shared point estimates to standard errors. But is that the right thing to do? Well, one rather big problem with that approach is that it assumes the likelihood is normally distributed, and this assumption can be violated when counts are low. And by counts I mean the number of outcomes, not the number of subjects. In this real-world example, as you can see in the table on the right, we have about 150,000 subjects in total, but only 28 have the outcome and only 3 in one of the treatment arms. In this case, the likelihood is already not normally distributed. The dashed line indicates the normal approximation applied by a most statistical software that produces a point estimate and standard error, while the solid line indicates the actual likelihood. And this problem can get much worse when there are zero counts. Again, in this example, we have lots of subjects, but there are zero events in one treatment arm. In this case, there is no point estimate and no standard error that makes sense, but there is still information about the effect size. And so the solution we propose is to communicate the actual shape of the likelihood function. We describe and thoroughly evaluate this approach in the paper I'm showing here. Our current best practice is to describe the shape of this curve using an adaptive grid, which means we simply provide a bunch of x, y coordinates. Because I tried to show in this plot, the adaptive bit means we try to space these grid points so that they are closer together when there is more curvature and fewer when the curve is almost straight, thus minimizing the number of points at which we need to evaluate the likelihood. So each site approximates the shape of their likelihood function and shares it with the lead site. Because this is just a curve, no patient-level data is communicated. Now the evidence synthesis package implements both a fixed effect and a random effect model using these approximations. The random effects model uses a Bayesian approach made possible by the piece Markov chain Monte Carlo engine. So let's walk through the steps in R. For this example, we'll simulate some data using these functions in the package. We simulate 10 sites. As you can see, at each site we have the usual variables for a stratified Cox model. A unique row ID per person. Each person belongs to a propensity score stratum. We have x for the exposure and y for the outcome, both binary in this case, and time to either the outcome or to censoring. If we look at the 2x2 table, we see we have lots of people. But in one of the treatment groups, only two persons have the outcome. Now imagine we're at site 1. We can use this cyclops package to fit the conditional Cox regression in the usual way and estimate the hazard ratio and confidence interval. Now, suppose we want to estimate the likelihood curve. We can use a normal approximation, which is expressed just as the point estimate and standard error. But as we can see in the plot on the right, the normal approximation denoted by the green dash line is not a good fit for the real likelihood denoted by the gray solid line. Instead, we can use the approximate likelihood function to create an adaptive grid approximation. As you can see, the function requires a cyclops regression object as input to learn the actual shape of the likelihood function. The approximation is just a set of points on the log hazard ratio scale with the corresponding log likelihood values. As you can see in the plot on the right, this approximation does fit the true likelihood very well. Just to show you all the code in the example, here I'm using LAPLI to do this at all sites. Normally, of course, each site would do their own approximation and share it with the lead site. At the lead site, we can now combine these likelihood approximations. Here, we're using a fixed-effects model. In this example, I'm also computing a gold standard by pooling the data. Normally, we wouldn't be allowed to do that. I'm just doing it here to show you how well the method works. If we use the normal approximations, in other words, standard meta-analysis, we see we get a warning that some of the sites did not have a valid estimate, probably because there were zero counts. We also see that the point estimate is quite far from the gold standard point estimate and an effect size of 1.6 when using a normal approximation compared to the gold standard of 2.4. If we use the adaptive grid approximations, we use data from all sites, including those that have zero counts. We see we now get an estimate very close to the ground truth. Probably more realistic would be a random-effect model. Again, we pool the data to obtain a gold standard estimate. Again, the normal approximation is quite far from this gold standard. The adaptive grid approximation on the other hand is again close to the gold standard. When we visualize this, it may become clearer what is happening. Here, I'm showing four plots. The bluish curves show the partial likelihood distributions. On the left, these distributions are assumed normal. And for sites 2, 5, 8 and 9, we have zero counts, so no normal approximations. On the right, we're using the adaptive grid approximations. Not only do we see that all sites missing on the left show greater likelihood for large effect sizes on the right, but we also see that the sites without zero counts show a skew distribution with more probability mass for higher effect sizes. As a result, the meta-analytic estimates are very different. The evidence synthesis package currently supports Cox, Poisson and logistic regressions, either conditions on some strata or not. Even though we can currently combine evidence for one parameter only, the models themselves can have more than one parameter. These other parameters would be nuisance parameters that would be re-optimized at each point in the adaptive grid. The evidence synthesis package is available on CRAN and we also have a package website with documentation and a vignette. Thank you for listening.