 Hi everyone. Welcome to my lightning talk today at R-Medicine. My name is Jie Cao. I am a PhD student in the Department of Computational Medicine and Bioinformatics at the University of Michigan. Today I am going to present a new R-Package, Runway, developed by our lab. It is a package that can help researchers visualize their prediction model performance. Why is this package called Runway? When you have your models and examine them, sometimes your models look quite different, but other times your models look the same, especially if you are just staring at numbers. Therefore, Runway is the tool we developed with the hope that it can help users visualize model performance and make the comparison if multiple models exist. In Runway, we provide the three core visualizations of two different aspects of model performance. Model discrimination refers to the ability of the model to discriminate or separate samples with the outcome from those who do not have the outcome. To evaluate the model discrimination, we provide the traditional receiver operating characteristic ROC curve and a new plot, stressful performance plot. Model calibration refers to the agreement between the predicted probabilities and observed probabilities. We provide a calibration plot as a visualization tool for model calibration. All plots drawn by Runway are publication-ready graphics. To use Runway, users are only asked to provide their model predictions in two outcomes. It can either be output from a single model or predictions by multiple models. With the ROC plot or ROC plot multi-function and the code shown here, users can draw a traditional ROC curve with 95 confidence interval ribbon for single or multiple models. Although the ROC curve and the area under the ROC curve are the most popular plot and metric to report for prediction model performance, there are some limitations of the ROC curve. First, it does not show the threshold information. It is important to know the model performance at different thresholds to assist making a decision on choice of threshold. Second, there is limited ability to compare models given that the ROC curve only shows sensitivity and specificity conditional on thresholds. Using the ROC curve only to evaluate the model and compare models can potentially give misleading information. So, threshold performance plot is what we provide in the Runway package to address these problems. It carries all information of an ROC curve and also carries all information of a precision recall curve which is also popular in clinical predictive modeling. The plot also shows thresholds and the distribution of predictions. It is simple to run our code and make a threshold performance plot by running threshold plot or threshold plot multi. Users just need to specify the true outcome column and the prediction column. If necessary, specify the column storing the model names. We present four metrics, sensitivity, specificity, positive, predictive value, and the negative predictive value across a range of probability thresholds on the x-axis. The plot shows confidence intervals for each line. The shaded gray areas show the proportions of cases that would be categorized as positive or negative at the given threshold. The distribution of predictions is shown as the bottom of the figure. Model calibration is another important aspect of model evaluation. Calibration examines the relationship between the predicted probabilities and the observed probabilities. With the well-calibrated model, for a certain class label, we have the confidence to further use the class probabilities in the interpreted as the probability of certain events happening. Non-linear machine learning algorithms are becoming popular in clinical predictive modeling, but they often predict uncalibrated class probabilities. That's why we also include calibration plot for users to evaluate their model calibration. Calibration plot can be drawn by calling calplot function. The default calibration plot uses 10 bins. Users have the option to change the number of bins using n bins. Like the threshold performance plot, we put the prediction distribution at the bottom of the figure. Wrong way can also show lowest smooth the calibration estimates. Using n bins equals to zero will remove the dots for each bin and show a smoother line only. Same as our C-curve in the threshold performance plot, calibration plot also allows comparison of different models by calling the calplot multifunction. The left figure shows a non-smoother comparison. The right figure shows the comparison of smoother lines. We have used plots produced with wrong way in some publications and the usage is also extending beyond our research group. In this paper, we evaluated a widely implemented proprietary deterioration index model among hospitalized COVID-19 patients. We used the standard threshold performance plots to determine appropriate thresholds to identify high-risk and low-risk patients. The yellow and the green areas here are simple additions to the standard threshold performance plot to show patients that would be put to high-risk or low-risk group as given threshold. The calibration plot shows the model over or over predict the risks. In another paper, we used the standard threshold performance plot to evaluate a widely implemented proprietary sepsis prediction model in hospitalized patients. The published figure, which is shown here, is in a different style because it was reformatted by the journal. Our original plot was generated in the runway style and can be exported as a vector file, which meets the requirements for journal submission and is also easy for journal editors to reformat it. Here's an abstract showing how threshold performance plots are used to evaluate the epic sepsis prediction model for three different outcomes. The authors plotted their model performance based on our standard threshold performance plots with some modification. At the end, I'd like to acknowledge the team that put this package runway together, Pan Gibson, Sean Meyer, Kira Sakira. The package is currently available on GitHub. We also have a documentation page. Please feel free to contact me or Dr. Singh for any questions, comments, suggestions on runway. Thank you for your time.