 In this tutorial, Alex Seto and myself, Nicola Cooper, will introduce MetaDTA. MetaDTA is an interactive web-based shiny app that conducts meta-analysis of diagnostic test accuracy data through a point-and-click interface and creates novel data visualizations. I would like to start by acknowledging the researchers that have worked on the development of MetaDTA as well as all members of the NIHR Complex Review Support Unit who have tested the app and provided valuable feedback throughout the development process. I would also like to acknowledge the funder, the National Institute for Health and Care Research, in the UK. MetaDTA is part of a suite of online evidence synthesis apps developed by the Complex Review Support Unit. Separate EsmarConf 2023 tutorials are available on MetaBaseDTA, which is an app for conducting Bayesian meta-analysis for diagnostic test accuracy and meta-insight and app for conducting network meta-analysis. A new app focused on evidence-based research and meta-impact is currently in development. The main principles we adhered to when developing these apps were that they must be free to use and open source, where possible utilize existing bar packages, have a point-and-click interface, and have an emphasis on visualizations and methods for sensitivity analysis. Before introducing MetaDTA, I thought it would be useful to present a brief overview of diagnostic tests. Diagnostic tests are routinely used in healthcare setting to confirm the presence or absence of disease, for example, the presence or absence of COVID-19. Unfortunately, diagnostic tests are rarely 100% accurate, and those that are known as perfect gold standards may be too expensive, invasive, and or take a long time to process. Therefore, cheaper and quicker to administer imperfect diagnostic tests are developed, and their accuracy is assessed against the gold standard. For an interactive, explorable explanation of diagnostic test accuracy, please see our DTA primer. Our motivation for developing MetaDTA was driven by the added complexity of meta-analysis models for synthesizing diagnostic test accuracy data, compared to meta-analysis models for effectiveness data, and the lack of user-friendly software in which to undertake such analyses. The added complexity is due to diagnostic test accuracy being a measure of two dependent variables. Sensitivity, the proportion of people with the disease who are correctly diagnosed as positive by the test, and specificity, the proportion of people without the disease who are correctly diagnosed as negative by the test. This requires the fitting of relatively complex bivariate testical models, which can be a barrier for some researchers. The two main aims in developing MetaDTA were to develop a freely available user-friendly web-based, point-and-click interactive tool, which allows the user to input their own diagnostic test accuracy study data, and conducts a meta-analysis for diagnostic test accuracy reviews, including the ability to incorporate quality assessment and conduct sensitivity analysis, and to develop interactive graphical displays to facilitate exploration of the diagnostic test data, including the meta-analysis, and effectively communicate the results to non-technical experts, as well as the ability to customize, explore, and export the plots for publication. MetaDTA uses existing R packages Shiny and LME4. Shiny allows creation of web application with interactive user interfaces, and LME4, a package that fits generalized linear mixed effect models, is used to fit the bivariate meta-analysis models. MetaDTA is hosted on the Shiny app server and is available to any users with a web browser without requiring any specialist statistical software. However, MetaDTA is not designed to replace statistical expertise, and non-expert users are encouraged to consult a technical expert. I will now pass over to Alex Sutton, who will present a demonstration of MetaDTA. Okay, thanks, Nicholas. I can never remember the exact web address of the apps. So, if you type the RSU apps into Google, it should be the first link, and that shows all the apps that the Complex Review Support Unit offer. And if we go down to DTA, MA, we get to the one we're using today. Clicking on the link, and that should load the app up with a bit of luck. And you are presented with the opening screen, legal things there, just about data protection. Read those at your own time. First things are a couple of papers that we asked people to cite. Did they describe the app if they use the app in a publication? That way, we can measure how much use it's getting, and hopefully it'll help funding it to come. Right, there is a user guide you can download which has more details than I will be talking about today. You can get the app itself off GitHub if you want to modify or run it locally. I think that's it there, just showing the changes in the versions. So, all the action happens at the top here using this menu system here, we should get you through all the bits of the app. The first thing you need to do is load data. Now, there is some data loaded in the app, and I'm going to use that first, and I recommend you do as well to get used to the app and make sure you understand what's going on before trying your own data. Some example data sets. I'm going to use the one which has added to it both quality assessment information and covariates, so we can see all the features of the app. I'll click that there. This page gives you some information about the data set. It's a test to measure dementia and call the IQ code test. It's a questionnaire. If we click through, we actually get the data that the file contains. 13 studies here. We have the author the year, then the outcome data, true positive, false negative, false positive, true negative. Anyone familiar with these sorts of analysis will know that's the typical format of the outcome data, and they're essentially in any analysis. After this, we have extra columns giving further information that you can use. As I said, we have these risk of bias and applicability measures seven in total across both from the bias assessment tool, Cochrane Collaboration Recommend. Then we have some covariates as well. We have the actual threshold used to determine whether somebody is deemed diseased or not by the test country the study was conducted in, and these are different versions of the IQ code with different kinds of questions in it. Okay, so that is the data for analysis that will go straight on and look at the analysis. The first tab here, we have the data again, but we also have the first things that the app has calculated, the total number of patients in each study, and then we have the sensitivity and specificity of the test eye, the test outcomes, and the weighting that you get in the analysis there. So we're giving the data there. We can move on to the next tab. This is the SROC plots. That's a summary ROC plot, receiver operating characteristic plot. Very briefly, it's a plot of sensitivity against one minus specificity. And on the plot here, we have hollow circles indicating individual studies. There are 13 of those, one for each of the 13 studies in our meta-analysis. The blue dot is the pooled overall meta-analysed estimate of accuracy. The smaller of the two regions by the large dashes here, that is the uncertainty, the 95 percent confidence interval around the confidence region, I should say, around that estimate. And the larger one is the prediction interval, which reflects the amount of variability between studies as well. And there is heterogeneity in this data set beyond that you would expect by chance alone. So this region is considerably bigger than the confidence region, because of variability between studies. Okay, there's quite a lot you can modify on this plot. I won't go into it in detail, but you can plot a curve as well. It's called the SROC curve. And you can, we'll just take that off for a minute, and you can look at the study weights or the prevalence of disease in these studies as well. So the study weights, they're just an ellipse showing you the size in each dimension reflects how much weight the studies get, and that's the prevalence. Sometimes it's concerned the test are a function of the prevalence and will change performance depending on how common the disease is. You can look at the individual uncertainty and the individual estimates as well. I'm just going to go to the bottom and reset all that for speed. That takes you to how it was when you came into the app. And then we can look at these covariates and the quality scores. You can look at individual ratings of quality. Here we go. You can either have low, higher, and clear quality. And this is on the patient selection dimension. And we see there are a number of studies, which are red, which is saying how I risk bias for that particular dimension. And we can identify which studies are which, if we click on the points, if I click on the red dot up there is rather extreme estimate of sensitivity, we find that's the Mulligan study and it gives its estimate of sensitivity and specificity there. Okay, you can look at all those seven separately, or you can put them all in the same time using a little symbol with seven segments, each one representing one of the dimensions. If you click on one of these circles, you'll get a blow up below. So this has gone, claves that study and it's got the risk of bias is low in all but one dimension, the RS dimension here where it's high. And that's a nice way of summarizing the impact of quality on of the studies while looking at the results as well. And we'll just take that off and then move and look at the covariates briefly again. So you could look and see whether the threshold used to diagnose somebody is having dementia using the test, if that's varying between studies. And we see it is a little bit that might not be very clear to see we can color code it. And we see one of the smallest sensitivities is the study with the highest threshold. And here the ones with the lowest threshold have the highest sensitivity up here, which is interesting and the yellow somewhere in the middle. So it might be the test sensitivity is being influenced by the threshold used in a particular study. Similarly, look at country, the categorization doesn't really help there. We have the countries plotted on any of the covariates you included in the data set you can look at in this way. And then we can move on to the statistics, which underlie that plot to give you the pooled results. And there are others as well. I'm not going to go and explain what these are, but if you can certainly look those up and there's some links in the end of the app, which help with the technical details. These are parameter estimates for the bivariate model, which are basically the untransformed estimates that we were looking at. We were looking at you might find this panel useful if you are handing these the estimates over and just somebody who may be using those in further modeling like an economic decision model where you want the estimates and the uncertainty around them. And in a similar way, if you're a Cochrane user and you want to plot your SROC in a Cochrane authoring tool, then here are the estimates you need to type into that tool. So this is a panel for particularly for Cochrane users to reproduce the SROC in there. And finally, this may look more familiar to people that don't do diagnostic test reviews, but know about metronalysis. We have forest plots separately for the two outcomes sensitivity and specificity. Again, always output anything you can download as a PNG or PDF or any of the tables as well can be outputted if you want to. So that's it for the metronalysis tab, but we have a sensitivity analysis tab as well. And what this allows you to do it looks almost identical to the previous tab, but it allows you to remove studies from the analysis. So if we move on to the SROC plot here and say we're concerned with these studies up here with very high sensitivity, which studies are there? Well, they're Harwood and Mulligan. And if I wish to remove those just to see how much impact they're having on the analysis, we can see that the analysis updates, the old analysis, including all studies is still included as a faint gray line, but the updated analysis is presented in the blue lines. And you can see how much the pool estimate has changed as a result of excluding those two studies, not a huge amount in this case. And obviously you can include or exclude studies as you wish to explore the dataset. And all these other tabs are as they were in the previous in the metronalysis tab up here, only it gives it for the selected studies only as well and allows you to make a comparison. I'm not going to show all those there. This tab is here as people may find it helpful as a way of presenting the results and imply that the help to put in interpret what those sensitivities and specificities mean, which are quite abstract concepts. So the idea is you think about a thousand patients, you set how common the disease is in the population you're interested in by default, it sets us to what the prevalence is on average across all the studies in the metronalysis, but you can change it. So if you think prevalence in your local area is less, you can change that. And then it will tell you how many people are diseased and test positive and are not diseased and test negative. And then the people who are misdiagnosed either not as disease but test positive or are disease but test negative. And out of the thousand, we have the numbers 272, 235, 28, and 465 here and we have intervals around those representing the uncertainty in the performance of the test. This can be a nice way of summarizing results. There's a slightly different view there as well you can use. And yes, you can do the same for the sensitivity analysis as well. So that gives us an overview of the basic functionality of the app. We have a couple of other things. These are references to some of the ideas in the app so you can look up and find out more information about some of the methods used and we're keen to acknowledge all the packages that we have utilized. They are packages utilized in creating the app and all those are used here and we'd like to acknowledge obviously all the hard work that's gone into making those as a longer notice about data. Again, you can read it at your leisure if you use the app. So the only other thing I really want to show is how you get your own data in here. And what we recommend is you choose the format that you want. I'm going to go for the simplest here for speed and then you download the example data set and that should download and I'm going to open that in Excel. There we go, opened in Excel. So that's the data that we were using. I'm just going to just show how it works. I had a different new study at the end and I'll try and give it some extreme results. Whoops, so if I've just added a study just an example and if you want to add a complete different data set which you probably will just delete the data but keep the headings the same and that way the program is very exact thing but you'll make sure you don't okay you'll get the right labels for the headers which is important and I'll just put this into documents here and call that demo update. There we go save that there it's saving as com with the limited which is what it came in as into Excel and you know that's going to be the right format. If we jump back to the app again I'm just going to here you go here to select the file I would deny and download but in documents there we go and that should be there yes and there we've updated the data loaded our own Zs included and we do a meta analysis here we should see this very bad performance and write down here skewing the results just silly example but just showing you how easy it is to get your own data in as well. So I think that's about all I need to show you as I say there is a manual you might say what if I want to include the covariates into the analysis what if I've got imperfect gold standard once if I want to compare tests well our other app the meta base DTA is the more advanced one allows those things and we'd recommend you looking at that that we think this is compact and simple to use and does what a lot of people want and yeah we'd love to hear from you if you use it please give it a try if you do diagnostic test accuracy reviews thank you for watching back to Nicola. In summary the main advantages of meta DTA are that it is a user-friendly interactive tool that allows researchers to conduct binary meta analysis using their own diagnostic test accuracy data and no coding knowledge required it allows the inclusion of risk of bias data and facilitate sensitivity analysis it allows data exploration through novel interactive graphics and it facilitates effective communication results through interactive displays which allow the user to input the prevalence for their population of interest. Meta DTA is an established app currently used by researchers worldwide with approximately 400 users hours per month features not currently available in meta DTA include allowing for an imperfect gold standard different test thresholds comparative analysis of two candidate tests and subgroup analysis and meta regression however if you're interested in a platform that does include these additional features then please check out our meta base DTA Esmart Comp 2023 tutorial. Finally this slide provides links to the meta DTA app and GitHub as well as key references we welcome user comments and feedback and our contact details are available on the front page of the meta DTA app. Thank you for listening.