 Hi, welcome to this USAR 2020 presentation, a risk-based assessment for our package accuracy. My name is Andy Nichols, I am the lead for something called the R-validation hub. I will be introducing that hub today before handing over to Julianne Manitz to talk about the risk-based assessment framework that the hub has jointly helped develop. We are here representing both the hub and the executive committee, the names of which you can see on the slide. So that's Doug Kelcoff, Yolong Zhang, Lynn Taylor, Joe Rickert, Min Lee, Kevin Amerson and Julianne and myself. We have included an abstract for our talk. I won't read that out. It's a short presentation and we'll be covering it in the next couple of slides. But the format of this presentation will be such that I'll just introduce the hub and what we do and then I'll hand over to Julianne to introduce the particular challenge that we face within the pharmaceutical industry and to talk a bit more about the framework that we've developed for our packages. So the R-validation hub has been in existence for around about two years, coming up to two years now. We are an R-consortium-funded working group and over the two years that we've been in existence we've grown to the point where we have around about 100 members from 50 different organizations. We work within the pharmaceutical industry and therefore most of those organizations are made up of pharmaceutical companies but it also includes independents, contact research organizations and regulatory bodies as well. This is a large group of people. Our mission within the biopharmaceutical industry is to enable the use of R, in particular in a regulatory setting so where the output may be used in submissions to regulatory agencies. This is something that has been a challenge for a long time and it's not an area where R is actually used very much today. So before I hand over to Julianne I just want to let you know a little bit more about the R-validation hub and give you some information should you wish to find out more. So we have a website www.pharma.org. From that website you will find blog posts giving you up-to-date information about what the various streams are working on within the R-validation hub. You'll find this presentation and previous presentations that we've delivered over the last couple of years and crucially you'll find the white paper upon which all of this work is based. So if you want a bit more detail around some of the things that Julianne is going to be talking about please navigate to the website and have a look at the white paper. Julianne is also going to be introducing a couple of tools. In particular she's going to talk about an R package called risk metric. If you want to have a look at that package then you can navigate to our github page which you can get to from the from the main website and on that github page you'll also find various other resources and things that we've developed and soon you'll be able to find a risk assessment application. So that's a shiny app that implements the risk metric package and allows you to add comments to that and generate reports for documenting the reviews that you performed. All of which Julianne will talk about in the next few slides. So at this point I'm going to end and hand over to Julianne to deliver the more interesting part of the presentation. Thank you very much. Thanks a lot Andy for the introduction. My name is Julianne Mannis. I'm a member of the R-Validation Hub Executive Committee and I help with the communication. Working with R in Pharma means working in a regulated industry. We are facing a number of different regulations and here I just want to mention two of them. The first one is the good clinical practice guideline by the International Conference on Harmonization of Technical Requirements for Registration of Pharmaceuticals for Human Use, short ICH. This is an international quality standard for conducting clinical trials in general. They state that software use should be reliable and the documentation of appropriate software testing procedures should be available. On top of that in the United States FDA, the Food and Drug Administration, has similar guidelines and further specifies validation as establishing documented evidence which provides a high degree of assurance that a specific process consistently produces a product meaning it's the predetermined specification and quality attributes. In other words software validation needs to establish three things, accuracy, reproducibility and traceability. The R-Validation Hub focuses on accuracy because reproducibility and traceability refer to environment in which the statistical software is executed means no difference between R and SAS. Validating R can refer to different things so when talking about R we can refer to Core R which is the base product owned by the R Foundation. This includes the base and recommended packages that include packages such as Survival, MGCV and R-PAD. So you can do already a lot with just a base Core R packages. However to make real good use of R it's essential to use community contributed packages which may have different owners varying quality and varying popularity. Guarding the validity of Core R the R Foundation fortunately provides a guidance document on regulatory compliance and validation issues. This document provides details on their software development life cycle and good practices that ensure accuracy. For instance hiring of highly qualified individuals, proper maintenance of R source code and control of releases, testing of each R release against known data and known results as well as the identification of issues and resolution prior to release. It can be therefore concluded that there is a minimal risk in using Core R for regulatory analysis and reporting. That is great news. However for contributed packages things get a little bit more tricky. We know that for each package on CRAN that it has been had to pass a series of technical checks but that does not necessarily guarantee the accuracy of that package. One way to assure accuracy would be to develop your own tests. However we already know that the maintainer of the package provides their tests and in addition popular packages go through some type of community-driven peer review testing and that considering the other Lidation app is suggesting a risk-based approach to evaluate the accuracy of contributed R packages. Let me walk through such a suggested framework. In the first step we would qualify whether a package is intended for use, means the user is directly loading it into the R section or whether it is a dependency and indirect import through these packages. If it is a dependency maybe minimum checks for suitability are enough. If it is intended to use we further process the package in a risk assessment. In the first step we figure out whether we want to make use of a statistical method implemented that also includes algorithms and things like that or if it is just non-statistical such as data manipulation, application interface, communication or plotting. If we have a non-statistical package the establishment of various metrics the measurement through various metric might be sufficient. So one bucket would be checking whether good practice of maintenance has been is performed and that can include existence of vignettes, websites, newsfeeds, form a mechanism of bug tracking or whether the source code has been publicly maintained. In addition the community usage should be assessed in a risk metric. We assume that the more exposure a package has by the user community the more ad hoc testing has been performed. Therefore the better the quality. If the requirements are not met we can always go back and check the test coverage by the maintainer or add even more testing by ourselves. Based on that risk assessment we should be able to determine whether a package of interest fulfills the requirement accuracy for a given use case or not. You may require higher standards for submission relevant work by accepting lower standards for exploratory past-hoc analysis. So this is all very theoretical. For practical purposes they are fortunately already tools which are implementing those particular metrics we have been talking about. Let me introduce you to the R package risk metric which has been led whose development has been led by colleagues co-authoring this presentation. I selected a few packages with different popularity in order to illustrate how it works. Our risk metric is itself not yet on CRAN but can be also included. U-Terts is an R core package. Ggplot2 is a very popular tidyverse package. HMS is a little bit more old school. ServeMiner is less popular but well established in pharma and Cox robust has been the author's package on CRAN at the time of selection. First risk metric is referencing the selected packages and with the call of the function package assess the package metadata is assessed and the package metrics are stored. This information is then converted into a numeric score value between zero and one where zero refers to poor and one to great quality. Finally each packages risk is summarized in a weighted sum of assessment scores. The rates for each metric can be determined by the organization. As an output we get a number of metrics for evaluating good practice for maintenance, the number of downloads for community usage and test coverage as well as the overall risk. We'll leave you with a short introduction. Hope that you can answer the question whether it's time to integrate R properly, integrate R into pharma and want to strongly encourage you to get more information from our website pharma.org. Please feel free to sign up for the mailing list to stay up to date and any feedback and experience on the implementation of the framework in your organization are very welcome. Thanks a lot for your interest and attention. Bye.