 Hello everybody, I'm Carice Neville and I work as part of the Complex Review Support Unit based at the University of Leicester in the UK. Today I'll be sharing my work around developing a novel multi-faceted graphical visualisation for treatment ranking within an interactive network meta-analysis web application. So treatment ranking within network meta-analysis is a really powerful tool but the results can often be misinterpreted or used or presented inappropriately. As a consequence we as a team wanted to improve the ranking output within our NMA web app Meta Insight. Therefore we had two aims one ascertain the current methods and visualisations for treatment ranking within NMA and two develop a novel graphical visualisation for Meta Insight. To make sure we're all starting in the same place I'm going to quickly introduce NMAs. So firstly the standard method for gathering evidence systematically whilst also appraising and synthesising the evidence is called a systematic review. To then obtain a quantitative pooled estimate of the outcome of interest in your review one can run a meta-analysis to quantitatively compare healthcare interventions or treatments. The next step is to simultaneously compare multiple treatments which can be done with a network meta-analysis. This is done by forming a network of studies and treatments and then the NMA uses both direct and indirect information to estimate relative treatment effects and it is through NMAs that treatments can then be ranked. Our first step was conducting a targeted review of the literature since January 2011 looking at papers that introduced, discussed or compared ranking methodologies of visualisations. This gave us in total 29 academic papers and articles. Within that 29 there were two articles that were particularly useful. As part of their paper Veronica and others authors produced a wonderfully concise summary of common ranking statistics and their primary characteristics. The paper by Cosmire and authors is a brilliant detailed record of graphical visualisations currently used within the field of earthen and synthesis. So from these articles what did we learn about treatment ranking statistics and which ones are important? I'm going to bring forward four key points I found. So firstly most ranking statistics are based on Bayesian methods where multiple simulations are run then giving rank probabilities based on the rank distribution outputted from these simulations. Secondly a popular option appeared was the surface under the cumulative ranking curve aka the sucra and this does what it says on the tin. It gives a single value but incorporates the entire ranking distribution but naturally still has limitations. There does exist a frequentist alternative for the p-school however some authors have indicated that its interpretation can be challenging. While simpler methods may be easier to understand such as probability best they are often more unstable and don't encompass the entire analysis. The opposite is also true i.e whilst more complex methods are more rigorous they can be harder to understand. And finally interpreting ranking in isolation is not advised there are many things to consider alongside ranking results and not doing so can lead to exaggeration of results or misunderstanding. The literature also introduced us to lots of visualizations there are too many to mention in this presentation so I'll focus on those that I felt worked well and fitted within the remit of a single outcome as this is what meta insight works with. So first we have rank grams which are a popular choice with lots of variations and they essentially plot all ranking probabilities for all treatment and rank combinations. They can present the results on the same or separate axes, can use lines or bars and can present absolute or cumulative probabilities. Cumulative rank grams are also known as sucra plots due to their direct relation to the sucra statistic by definition. Generally rank grams were found to provide informative balanced summaries of the distribution of ranks. However some people have stated that comparing treatments can become difficult. An alternative that is less commonly used is a radial plot. Generally radial plots have an outcome measure for example sucra marked radially around a circle. Some variations use a donut shape and use coloured coded sections to indicate the value. The benefits of this layout include the ability to present concentric circles to consider multiple outcomes or subgroups and including other information nested in the middle such as a network plot. However a limitation of working on the radial scale is that generally humans struggle to discriminate between radio lengths. Next the ability to display multiple elements of an analysis in the same space, a multifaceted display, allows researchers to present the larger picture and this is great for treatment ranking as it allows the users to form robust and sensible inferences by drawing conclusions from a combination of sources. However one does need to be careful to not overload the reader with large amounts of information. And finally with advances in technology a natural movement in the field of data viz is incorporating interactivity. Interactivity allows users to drill down further into the data or analysis as they wish and can even involve users choosing their own settings. For example this interactive visualization allows users to specify the relative importance of outcomes regarding treatment choices. So that was just a snippet of the literature that we found and it provided great direction and knowledge when developing our own graphical tools for treatment ranking within Meta Insight. So three things to quickly mention regarding how we developed these tools. One we decided to start with just Bayesian analyses due to its popularity in the treatment ranking field. Two designing the tools very much an iterative process of creating designs, sharing them, getting feedback and going back to the designs. Starting with pencil and paper and moving on to mock-ups with different software. And thirdly the final designs were created in R which is what Meta Insight runs on and the ggplot2 package. Now before sharing what we came up with let me quickly introduce the example our dataset for this presentation. This is the inbuilt example dataset in Meta Insight for continuous outcomes. It compares pharmacological interventions for the treatment of obesity. The outcome of interest is BMI loss three months from baseline and the NMA was on 24 studies including five treatments plus a placebo. So our first plot we're presenting we've called the litmus ranking gram. The base is a rank gram which we've chose due to its popularity and easy to understand nature. To aid interpretation we decided to plot cumulative probabilities and have all curves on the same axes and with cumulative probabilities one can then say that the nearer the curve to the top left corner the better the treatment. It was reported that comparing curves can be difficult so to further aid comparison we added a litmus strip of sucre values which has two functions. One to act as a key through the colors and two gives the sucre which is easier to compare. So for this example one can see that the ranking analysis indicates that rim on band performs well as we can see the green coloring high sucre value and the probability curve being near the top left. For the converse reasons placebo performed worse and you can see that it's hard to discern between the two curves we're met forming and all is that subatrimon which may seem like a limitation however I believe it actually emphasizes the point that they performed equally well and next we have the radial sucre plot. We decided to develop a second plot for two reasons. One as the number of treatments increase a rank gram like the one I showed earlier can easily become crowded and two we really liked the radial plot with the nested network plot that I showed earlier and so we created this at the price of losing some of the granularity that the litmus rank gram gives. So for each treatment its respective sucre value is plotted radially in descending order. We stuck with the sucre to not overload users with various statistics and a sucre considers the entire ranking distribution whilst outputting a single value. To keep things consistent between the plots the same color scale was used with the color of the nodes also indicating the sucre value. We are aware that sucre is now represented in two manners and thus could be seen as wasted ink however we believe it further aids communication and impact. So we took the plot by side at et al and took it a step further by overlaying the network plot rather than nesting it. Therefore the size of the nodes represent the number of participants in each treatment arm and the thickness and presence of collecting lines indicate the number of trials that directly compared the respective treatments and we felt that overlaying the network plot meant less work for the users and more impact. The benefit of including the network plot in this way is easy to see if I unpack this example so one can see that rim and bound formed the best due to its high sucre and green color. However it's actually quite hard to see rim and bound due to its small node size from having few participants in treatment arms. Furthermore one can see that it isn't very well connected to the network. These immediate factors should tell the reader that the result from rim and bound should be interpreted with caution. On the other end of the scale the large red node for placebo immediately shows that there is a good amount of evidence supporting that placebo did not perform well against the other treatments and that we can be fairly confident with that result. It has been noted that having the network plot overlaying like this could become messy and hard to read for large amounts of treatments or connections. So in such cases we created an alternative version of the plot which goes back to the original nested layout. However we've kept the nodes colored according to their sucre value. As I mentioned earlier a key message that is present in the literature is that we should avoid presenting ranking results in isolation. In this spirit the final ranking graphic tool was designed to be a ranking panel. So here's the design. The central pane contains the ranking plots that I've just presented as these are the focus the headline act. Then on the left we present the relative treatment effects. This will aid interpretation of the ranking and reduce potential exaggeration as an inherent risk of ranking is accidentally putting undue emphasis on treatments performing differently purely due to being ranking differently. Then the aim of the right pane is to give users the opportunity to explore and interrogate the data behind the ranking analysis. Currently this is just a network plot but we have other plans that I'll mention later. Finally here is a video illustrated how the ranking panel looks and performs within Meta Insight. Our NMA app created with ARP and the Shiny package. Firstly we run our Bayesian analysis and then click on tab 3c to access the ranking panel and this is how the panel initially looks. The user can switch between the two new ranking plots as they wish including the simplified radial sucre plot. There is also an option to have the plot in a colorblind friendly color scale. All of the plots can be downloaded and the ranking results exported as a CSV file. Furthermore as is the case throughout Meta Insight there is functionality to run a sensitivity analysis where you can remove studies and compare the results. So this new treatment ranking panel is currently live in a beta version of Meta Insight so please do have a look and play around. We do have future plans and ideas including the ones that I alluded to earlier in this talk. So firstly there are plans to extend the contents of the left and right panes within the ranking panel. Regarding the left pane we plan to allow the reader to switch between relative and absolute effects. With the right pane we very much intend to increase the number of visualizations available to the user. Current plans include incorporating some form of risk of bias or grade visualization so users can see the quality of the studies being analyzed. We also anticipate including David Philopo's bias interval plot where users can ascertain what level of bias could change the results and thus how robust they are. We have lots of ideas for making the panel more interactive including options to easily change the reference treatment where applicable. We plan to look into using the plotly package to enable further information via hovering. Unfortunately it isn't simple to just quickly convert ggplot objects to plotly objects for these plots as they have multiple layers. And finally we plan to wrap the newly developed plots into an R package so that other researchers can create them with their own data but outside of meta insight. So thank you for your time today. I just want to acknowledge Will Stahl Timmins, the biostatistics research group at the University of Leicester and the rest of the CSU group for their input into developing these designs. And you can have the link again to the app so you can explore it yourself. Thank you.