 Hi everyone. Happy Thursday. Thank you for joining us today. It's good to be here discussing the learnings and reflection from our Validation Hub case studies. I'm Maya from Procogia and I'll be hosting the webinar today. A little bit about me. Procogia engages extensively with our consortium so I get some fun time set aside to volunteer and help out with our consortium and our adoption series specifically while I work full-time at Procogia. We're heavily involved in the art community and we're a data science consulting firm that's based in all the work but primarily in the U.S. and in Canada and we help out companies on projects on all things related to data. But more specifically we really enjoy promoting the adoption of R. Very passionate about that and I really love working here because I get to do cool things like this on Thursday morning at 8am. We're all insured for this though so no complaints. Please feel free to reach out to me to stay in touch if you want to get involved or join us. It's something we're very passionate about. The other part of this is I'm also interested in this on selfish reasons because very fascinated by our validation so I'm kind of like strapped in and ready to see this presentation just as the rest of you. The event is recorded so you can always come back and refer to it if you missed any parts or if you had a meeting at 8.30 and had to leave. I would like to go ahead and introduce our speakers and contributors, panelists. They'll be helping out with the breakout rooms today after Julianne presents. We'll also be uploading a PDF if any of you haven't used Zoom breakout rooms before. It will have a simple instruction on how to use it and how to hop in and out of rooms. Introducing you to the main presenter, Julianne Manitz. She's a statistician in pharmaceutical industry who currently works in the field of immuno-oncology at EMD Serrano in Massachusetts. In this role, she works on statistical methodology, innovation, and technology. She has contributed to various R packages and a member of the R validation hub executive committee working on software validation for the usage in pharmaceutical trials. You also have other contributors who will help us out with the breakout rooms, Preetam. Preetam is a standards lead at Mark providing leadership to develop and maintain global standards for atom implementation, R package development, open source package qualification, computing platform enhancements, and compliance management tools. He's also a member of the R validation hub executive committee. He actively promotes the use of open source software and clinical trial data analysis via platforms and paper publications. He has a PhD in bioengineering from Temple University. We also have Doug. Doug Kalkov has been supporting the adoption of R in the pharmaceutical space for the past six years. During his time at Roche, he's pushed the adoption of R through pilot clinical trials, showcasing the benefits of using R by crafting internal tools and building services that embed the R validation hub's guidance as a part of their software development cycle. He's passionate about making the use of open source tools in a regulated setting as a viable path, not just in large pharmaceutical companies, but in lean startups and public sector. He wants to do this by addressing challenges through open initiatives. Now we have Eric Nolan from Biogen. He's a data scientist in the pharmaceutical industry who currently supports adoption of R and other open source tools at Biogen. During his time at Biogen, he's pushed adoption of R and open source tools by showcasing internal tools for Biogen's analytics environment and software development lifecycle. He's contributed to various internal and external R packages, and he's also part of the R validation executive committee. He's currently the lead developer of the RISC metric package which is sponsored by R validation hub. Let's get started. I'll let Julian take the stage. Thank you. All right, let me do the screen sharing. I hope this works for you. Yes, we can see your screen. This is a good time to yell at me. I'm obviously presenting on behalf of the R validation hub. This is a group of people and partly we presented here helping out with the breakout rooms. Before we get started, I actually have the honor to make a really good announcement because we have a little bit of a shift in leadership and it has been leading us in a really beautiful way for the last couple of years. He is stepping back and Doug is stepping up. Doug, do you want to say just one, two words? Yeah, sure. Thanks Julian for letting me, for announcing this and for letting me say a little something. There's not a whole lot that I can add since my blurb was just read off second before, but this is a good time to thank Andy for being our forming leader and for really setting this up to have a bunch of success stories in the past few years. I'm excited to be carrying that torch and so, yeah, looking forward to seeing what comes. What that means for people in attendance is if you want to see things change in the way that we've been operating, now's the time to reach out because we know we're going through a little transformation ourselves and that allows us to reflect on how we've been operating and if you want to see anything happen differently, if you want to see more of these types of outreach activities, now's the time to let us know. So thanks Julianne, I'll head it back over to you. Yeah, speaking about reflection, so this presentation is a little bit of basically actually taking a step back and seeing what the evaluation have has been doing over the years but also reflecting on how the things were implemented in the different companies and I think it's a fantastic opportunity to actually realize how much impact we could have in such a short amount of time and also how helpful we were maybe in like setting some guidances in one places and then maybe opening up other opportunities for further development, for how to improve the risk assessment for our packages in pharmaceutical industry. So having said that, that's a little bit of an outline just keeping the recap, we are trying to recap like real quick just in case you haven't been too much in touch in like what this evaluation has been about in the first place, the white paper and then actually going into the case studies because there are quite a few companies who have adopted it and we had a comprehensive conversation around how that was implemented and then it's time to stop leaning back and we want to have your opinion and actually have an active discussion as active as possible in the breakout rooms. So yeah, this is only for the first part along with folding kind of webinar and then for the second part, please join us. So in 2018 there was our pharma and we basically started out with it was time to integrate our into pharma and the question at that time it was actually a question is it even possible, is it a feasible thing to do and a few very passionate people came together and I think our pharma really helped out there to bring people together in Harvard here in Massachusetts and it was a wonderful venue and that was a really good kickoff to actually prove the point that it is not only possible, it is actually bringing a lot of new improvements. So just we recap what is our validation hub. It was initially an PSIA special interest group and now it's also an our consortium working group. We have approximately a hundred members and over 50 organizations and a mission. So we started out in June 2018, we got that our consortium funding as well to create that online repository for our package validation. Our mission is to have a cross industry initiative so that not everyone is doing their own thing and have some sort of alignment to enable the use of our in the biopharmaceutical industry not only in the big pharma companies but also having the learnings from that into smaller companies as well and basically having the on ramping there and making it easier. So having said that, there are a bunch of resources available. There's a website called pharma.org. On this website it's worth watching out for the white paper, for blog posts, there has been other publication last fall, presentations and various conferences, case studies, all these kind of things, recordings. We try to have that as updated as possible and publish blog posts on a sort of regular basis. But you know this is all theory but for practice they're actually also tools available which is the risk metric package and both are Github, Samhita and Krann and Eric is leading that. So he's here, he's joining us so that's great. And then the second tool is actually a visual interface, a shiny app for the risk metric package so that no programming requirements are necessary and that is led by Aaron. So this is how it looks like. It's actually very straightforward. It's just very straightforward way on how to use the risk metric package. So you have like just six by examples of different packages that they get into a package reference first and you collect that information and as soon you have that package table basically you can do the assessment and based on the assessment you can assign scoring and summarize the scores across the different metrics. So for the different packages, the six packages below each other here is the different versions. Obviously every output will be version specific. License might be available, it might not be available, but that can be scraped and then there are like some metrics like whether help was exported, vignettes are available, bug URL that would be Github and status is basically the percentage of closing and that is news a little bit just like some, you know, good maintenance metrics, meta information that can help to assess whether packages like develop is a good software development cycle or not. So these two are supposed tools are supposed to support an assessment rather than having to manually go into each of the packages description file whatsoever. And just to have a look at the app, I think this is not how it looks anymore. I'm sorry for not updating that screenshot, but I'm sorry, but in a nutshell, we have here the package that is selected and you can just have a drop down manual, you can extend the package list and also the different versions and on the lower part will be basically something that can be filled out by the user and they're different. There can be either an admin or user user for this app and accordingly, you would have like the packages here for example is under review has a fairly low risk of 4.14. And then based on that the overall risk can be assigned manually and the decision and you could create a validation report that could be used for the documentation. Yes, so this is basically already the result and that result can be like informed by different maintenance metrics here we have vignettes news, source control and GitHub yes no and always kind of maintenance metrics that may be more or less important for your decision making to assess the risk. There are other things available like community usage and and here you also have a report preview and kind of bear these things. So just to let you know how this looks like there is a lot of additional information available as I said on the website that guides you more specifically and more extensively into how to use these tools. And the our validation hub is obviously not existing like in a in a vacuum there are a lot of partner initiatives that we want to highlight here just a few of them. There's for instance the our table for our tables group for regulatory submissions working group to create standard tables that meet the requirements that are needed for FDA submission standards. There has been our submission pilot working group which has been focusing on IT and platform questions around to make a all regulatory submission and just basically proving the point that it is possible and I think I mean I personally think so if you if we can make an all regulatory submission that is actually also saying that it is very feasible to maybe do a mixed hybrid or whatever it's feasible for you. Then we have a partner initiatives clinical statistical reporting in multilingual world seeking to assess like those sort of differences that may come up and using different programming languages in pharma very typically between SAS and our other open source languages and then obviously our pharma kind of a little bit of a plate birthplace of the validation hub sort of where this group really initiated it in the first place and it's an annual conference it's a we really recommend it it's a lot of fun and it's one of the few conferences which really took off online and just made it really made it a fun way to join from home. All right so so many resources but what how do we actually assess the risk so the white paper basically differentiates the between core packages means base and recommended packages and contributed packages and there are various arguments why we can assume there's many minimal risk in the core packages for regulatory analysis specifically the R Foundation has provided a very extensive guidance document on the regulatory compliance and validation issues so it really goes into the details of their software development cycles and based on those details the minimal risk can be recommended so for the contributed packages however the outer parts we are using them a lot there is a huge variety and quality robustness and trustworthiness right so when we have a package on CRUN we have some basic technical checks that again that are basically run but it does not necessarily guarantee that a package is accurate or the package out the result is the result is accurate that you create so based on that the R Validation Hub defines a framework to assess the accuracy based on multiple steps intended to use we are going into that on the next slide so you basically have a new package you ask your question whether you intend to use that package or if it is an indirect import if it is an indirect import you can the white paper recommends minimal checks for suitability otherwise if it is intended to use the risk assessment should kick off and there was the idea of differentiating between a statistical package and non-statistical package means statistical packages having things that like actually like implement algorithms and specific calculations that basically result in the final hazard ratio being accurately derived yes or no versus maybe less strong like we can accept a higher risk maybe for a package that does data wangling or plotting because these things are easier to just check visually for instance um then the pack the another step of questioning could be with a package is maintains so this is like a lot of like like things that you basically get out of the description whether do you have a good maintenance do you have a news file do you have do you get quick results on github kind of these things and that can be obviously helping you to understand how fast a back if they have discovered one would be resolved for instance another step whether how much the package is used because not only the testing in the background maybe of interest but also the community usage is kind of an informal testing even though the testing is not like directly implemented into the package and then you come to the final decision whether it meets the requirements you include it into the regulatory environment or you reject it and at all these steps before you reject you can always have extra remediation and testing to actually come back and give it the opportunity to meet the requirements so this is kind of the pipeline that was suggested in the white paper a couple of years ago and last year we had a three-part presentation series on case series and eight pharma companies actually participated and we had like a call for participation eight car pharma companies came back and said like yes we would like to share how we are implementing this in the different experiences as well in different gxp frameworks and we were asking people to highlight what was easy to implement and what was more challenging and the opportunity we thought it would be a great opportunity on the one hand to bring together to see what do what does do other people do what do other companies do but also to kind of waterproof your own framework so if you're if you're deciding as a company to make an all-out submission and a certain point and that's the first time you're actually like exposing your framework to the outside there might be things be found from someone from the outside that you haven't considered when you're developing the framework so this was a little bit of an opportunity to actually like expose the framework expose it to discussion and get also some more input from other industry collaborators and i think it went really nicely we have the recordings available on the our validation half minutes page here so on pharma.org and there is also a little bit of discussion and exchange in github and we obviously encourage everyone to contribute and if we have another group of people who want to update their framework and maybe we do another part of or new companies who will be interested to show their concepts we would be obviously very happy to accommodate another episode so for this presentation we want to summarize some common themes and some differences but also some challenges and where we maybe have further ideas so for the common themes i mean we are we the pipeline was suggested by the other validation happened and we are calling for case studies so it's maybe a little bit of a selection bias but all case studies surprisingly use the risk based validation approach so people who said that's something that we don't use at all for some reason didn't reach out to us and so that could be a selection bias but we say like yay so there was a there was a large number of companies who actually implemented it as outlined by the white paper there were different classification of the package qualities either in a three-level high medium low or a binary high low categorization but the assessment how the different companies got to this assessment varied in various settings and there was in all approaches there were was a high importance of test coverage as an assessment metric and for trusted resources so they are like different metrics were chosen and getting to that and the differences but a common theme was that the test coverage was rated quite high as importance and for trusted resources so that's one part of the white paper that I didn't really touch on yet so the white paper opens the opportunity to create and classify certain how let's say maintenance groups let's understand the art consortium who produce core and base the core packages so the base and recommended packages as a trusted resource because they have been opening up their software development cycles and then a typical question would be is our studio also a trusted resource because for some of their packages they actually have created a similar document also star developers have for instance have done that having said that that idea of creating trusted resources has been actually adopted so we have seen that the core packages base and recommended were treated as collective as low risk and some organizations also trusted our studio developments like the tidiers and other packages and then another common theme that we have discovered is the majority of the approaches focused on this assessing the intended to use packages only and because by the end of the day you have your tests running on the up on the highest level and you can rely that you indirectly test the dependency the parts of the dependency that are well that are relevant for you but yeah a few companies also ran the matrix on the inputs just to be on the safe side all right coming to the differences in the approaches there were various degrees of automation in the risk qualification and so the some packages were classified almost automatically at low risk if with if certain metrics were approved approving that some if certain metrics were fulfilled some of the packages were automatically at low risk with little or no human intervention and for higher risk packages there were typically some sort of pathways to have additional human assessment so that packages are not being thrown out without double checking first other companies however had were relying highly on human guided assessments throughout which is a little bit more a little bit more way more labor intensive we also found that there were different weights that were assigned to the test coverage and various suggested meta data metrics so for software development cycles I have mentioned them before and then there was not like a common threshold but for what kind of test coverage would be required to create a low to have a low risk package but there was a range of like 50 to 80 percent minimum test coverage in order to to rate a package as low risk and then different risk mediation strategies were applied some organizations immediately introduced their own unit tests others that restrict the package use only to a subset of the package functionalities there were different interesting approaches definitely the opportunity to learn from and then some common challenges because having this adapted is one thing but also where where is the where is the pathway where do we continue next also where can our validation have actually help out in the future and we have identified some common themes and these are actually the breakout rooms so I hope you're almost almost done with reporting the laundry now so these are the different challenges that we would like to discuss in the breakout rooms so the first thing is so I'm just speaking first about the common challenges and then we speak about the breakout rooms so the common challenge one piece of a common challenge is the time intensity of the resource and time intensity of the package risk assessment so R is for free but it is not for free that kind of a theme I think became pretty clear to everyone and buying a licensed software and buying like oh someone has looked over that is something that we that we kind of have to find better ways for R and that's why we are doing this right so it can can be really a considerable challenge to go through a large number of packages especially when the versions have changed and etc etc at the same time you're going to make sure that the reviewers of the R package actually do have the right technical expertise and also you have a good set of different contributors from different areas because not every package can be assessed by everyone so you want to have good contributors across the organization for my tea quality assurance statistics data science programming lines etc etc so you're there that's like who is doing the R package assessment and how much time should that person spend on and that obviously depends on the level on automation that specific companies are implementing another challenge was finding appropriate test data sets test cases and expected model outputs and that was a common challenge and there that's definitely a topic in the breakout rooms I happen to know that and the another common challenge is a long-term management and maintenance as well as the oversight of this package assessment process specifically for those approaches that are highly dependent on to one things automatically so what's a good timing to actually make sure that the decisions that are automatically taken are still the right ones in a couple of years and how long are you going to wait and things like that so having said that like these things are still in the little footsteps and this would be maybe before we go into breakout rooms we may be going to have like questions first and then we'll go back out once what do you guys think I'll jump in quickly so we we have a little bit of an audible to call in terms of how the breakout rooms are going to go so it turns out because we're set up today as a webinar and not a meeting we can't do breakout rooms so what we're going to do instead is we're going to use the question answer style for people to contribute and we're going to go down the list of breakout room topics and we'll each present kind of a brief summary on the maybe some of the considerations that we would like input on or some of the topic areas that you might consider discussing we'd appreciate if you're in attendance if you have questions or thoughts on the topic this is kind of meant to be an open forum as much as it can be in the webinar style meeting so also if you'd like to chime in you're welcome to you can raise your hand and our hosts here will give you privileges that you can ask the question directly if you'd prefer that otherwise you're welcome to ask it in chat and we'll read it out as we go through so if Eric's ready maybe we'll go down the list and just kind of introduce the topics sorry to put everyone on the spot to kind of briefly do a summary of these these four different topics but are you ready Eric yeah cool thank you great thank you so I'm one of the developers on the risk metric package and my compatriot Erin Clark who is the lead developer on the risk assessment app we work sort of hand in hand developing sort of a you know a suite to help with this initial risk assessment the sort of validation process and one of the things that we have are aiming to figure out are how people in the how users end up implementing sort of package scores whether it's some sort of thresholding maybe you know like we talked about some people might classify packages as low medium high or low high or accepted rejected and specifically like maybe what those thresholds are or approximately what they might be so as a score of zero point three or lower high or is it zero point five sort of how those thresholds might be drawn and then also I think if you're not aware the risk assess risk metric package and app allow you to reweight individuals Patrick assessment scores to calculate that final risk score and we're curious to know if users use this functionality and if so how they might sort of reweight various assessments as Julian mentioned like test coverage is heavily relied on you know and so we're interested to know how much you might reweight something like that assessment we actually have a survey made up for this it doesn't look like I can send it to the entire webinar but maybe Maya or a waiter could send it I just dropped it in the panelists chat so I would welcome any we would welcome any feedback and sort of how you maybe tune the scoring outputs for your organization or your use cases all right maybe I can take over from there I'm seeing a couple questions trickle in but if you want Eric maybe I'll ask this one to you but if you it might pertain to everyone so there's a question about asking whether anyone's companies have or have put this into practice and whether their it functions or quality assurance functions have been involved in reviewing the process that they've adopted is that something that you want to answer Eric or should we kind of collectively answer that after we've all introduced the topics I can give at least my experience at Biogen and then if others want to chime in so we have presented this to our IT and IT quality organizations and gotten their buy-in but we have at Biogen kept the process in the business so we got them to understand like how packages work and how it's not really software so that they don't necessarily need to worry about as much but here's our process to make sure we're you know doing things right and and that seems to work well reduces overhead on their part and increases sort of response time on for the business. Yeah the same happened in our part of the company as well EMD as well. And just kind of reflecting on the case study discussions I feel like that was a little bit of a common theme is that a lot of the process that people adopted started with the white paper that the our validation hub put out and then a conversation with their quality assurance or IT folks to kind of see what that middle ground looked like and who should own the process and it sounds like then from there a common theme seems to be the business takes over the actual evaluation part and I can speak for on Rocha's behalf to say that's also kind of how we have adopted that that's what our trajectory looked like as well so. Yeah at Merc I mean sorry I had answered the question directly in the chat but yeah at Merc we made sure that when we're doing the package quality process we had the buy-in from IT and quality QA teams all the way through and it's an active process we keep on going through them for different changes and at Merc IT is actually you know we are actually both partners because everything we develop they also are part of it because we made sure that we are the business owners for it but a lot of the work is also done by them in terms of library updates and things like that so it's it's a continuous you know partnership that we made sure that it's actually documented and we got buy-in from QA teams. And then I'm seeing another question that might relate directly to your topic area Eric I'll read it out just so it's on the call and then if you don't mind answering the question reads high test coverage indicates that most lines in the code base have been run through tests but this may not necessarily be indicative of the quality of the tests how do people address this problem. Great question I we don't have a way to address this I mean at some point we have to assume best intentions right I'd like to think that our package developers are not malicious people because like that seems like an interesting use of your time if that were the case but certainly a real concern and I would welcome you know ideas on how to maybe get at this I've always wondered if if there is a way to measure sort of multiplicity of testing lines so do you test it once do you maybe test it and that might indicate sort of the thoroughness of your thought process do you know you could test to make sure your output is a number but do you also test for errors when you know something else comes out or something goes in and maybe that's a way to sort of get at that intention and quality of tests but I don't think we have a good answer at this point. If I remember there was I remember there were like one or two companies we actually went and had under some circumstances they would end up in the and actually review the test code. I would need to go back who exactly that was and say etc etc or everyone could go back and check that out in github and recording obviously but it's really it's a concern that was brought up I think in every company and some there you know we are not only we have to at some point quantify and put numbers on things and say like this is a number that we build as a composite of different elements right and the test coverage is one thing and then there might be other things that like check if there's like good software development practices and at some point we got to drill it down to one number and say like okay yes or no and or yes no maybe in some cases and this is just the nature of it and also I want to keep in mind that you know validate that's why we kind of moved away from validation even though it's still the name of the validation hub and actually went for risk assessment because we can just do so much in mitigating the risk that we are taking by using certain software and by the end of the day we also and that's also requirement by the FDA beyond testing our making sure we're documenting the quality of our software is we also have to document that the people who are executing the software having the appropriate education and therefore we have a whole process that is required by FDA so the risk assessment of the software is just one piece I want to keep that in mind as well. Yeah I just want to chime in also that you know this question has been flipped on its head and questions raised I'd say a package that says low test coverage is that not a good package to be included you know so that's something that we have also tried to address as Julianne was mentioning and Eric was mentioning that test coverage is just one part of the metrics that when we are doing the evaluation for the risk assessment so it's just it's not necessarily the major part it just plays a part in identifying that and as as Julianne mentioned the reviewers also play an important part so you know completely automating it would not actually be a good path that is at least Mark philosophy right now is that we need to have some human element so that we can you know look at these fringe cases also look at some issues that would pop up when you're doing a full automation. Maybe just to make sure that we hit all the topics before we hit the hour I'll continue with a second what would be a would have been a breakout room so one of the follow-on topics to what Julianne had mentioned is that there's a big amount of overhead that goes into each company kind of investing this effort initially and one of the long saying goals within the our validation hub has been to provide resources so that this is a collective effort so that this isn't something where we just provide guidance and there's an overwhelming amount of activity that each institution has to do independently in order to allow for people to do analysis in our and so part of that initiative is how do we provide packages to people in a way that's kind of communal and you know in the our world we have resources that already kind of satisfied satisfy these goals and we have CRAN and bio conductor we have our universe we have you know if you want to go with a product you can use like a positive package manager you know to manage this stuff even in the public sphere and we're thinking now how do we take one of those products or build something ourselves that satisfies all these different needs that we might have from a validation perspective or whether we need to go in that direction at all is also you know totally an open question some of the considerations that we have around this topic are what are the things that give us confidence or concerns when using a package and can we surface those things so that they're made more visible when you when you go to choose a package could a package repository or a service or a you know off-the-shelf solution that's public offer this information more transparently how do we improve the reproducibility of polling packages so you know on CRAN you know packages can be archived you know that's an issue that we have to address and if you go and try to install that package you know you're going to be hit with a package on available message so the reproducibility is a little bit sacrificed but with this in the spirit of me making sure that all the packages available are always of high quality in an industry where reproducibility is really critical you know that poses some challenges so is there improvements that we can make there to help make sure that we can continue to provide reproducible analytics especially for you know high high importance deliverables can we if we have varying levels of quality you know that's an open question for us too you know sometimes you have a high-profile phase three study or sometimes you have some exploratory analysis you still probably want them both to be reproducible if they're going to be used for really like meaningful decision-making to your development process but they might have different tolerances for the quality of code that you want to make sure you're using so how do we is that something we need to bake into the process and then thinking about this this kind of important information let's say that we discover that someone has 100 test coverage but they're all kind of these spurious tests that you know are kind of a false positive is that something that we want to be able to flag so is there like a community engagement part where we want to leverage the fact that we have you know thousands of eyes on these packages and we want to be able to assess the the collective we want collectively be able to assess the quality and maybe share those findings with one another so is there a community aspect that we want to be able to roll into this so those are all open questions that we have in terms of what is what does an ideal product look like in terms of how we jointly to deliver packages and if you have questions or suggestions or other thoughts on that topic I'd love to hear them so feel free to drop those in the Q&A otherwise I can hand it over to Julianne to introduce the test data and cases. Yeah Colleen also put into the chat repo working group so that's obviously also a really good place if if this is not a good time for you to speak up and have to think about it a little bit like this is the place to go and these our breakout rooms have been also like set up in this kind of perspective because we do not only want to discuss it and then like you know okay now we are done going home it's time for lunch kind of thing we actually want to like continue working on this so another piece that is kind of right now a little bit the question we that it should be part of the repo working group on that is to sharing test data and test cases because as pointed out like the test coverage is crucial for so many for so many assessments and the quality of the test cases might not be good enough whatever defines good or not good in this place right but having output sharing like I think a lot of company do use the CDIS data for instance and it seems like we all independently reproduce some set of test cases using the CDIS data so why are we doing this why aren't we producing this once and instead of everyone using the same CDIS data another company could use a different data set let's say we use some statistical standard statistical literature where we have the output and the calculation with R or with another software and we can we can prove to be producibility and make sure that whatever version the package comes it's always the same number that's going to be the hazard ratio when you compare the survivor so having said that like having those standard data sets and building the test cases and good literature that is peer reviewed is like something that we may all do internally in the company and it would be worth sharing because it's just creating a lot of overhead a lot of work and the question would be here like what are the test data sources that you're using do you have test cases and is that something that should be introduced actually into the repository or is it something that we could kind of as a collective try to channel way channel back to the package maintainers so they actually become test cases so the next time they immediately get integrated and we do not have to externally run them anyways so yeah if you have any good ideas for this this is a good time to speak or bring it up later I'm not seeing new questions but yes as Julian said please throw them in I'll ask a question so one of the resources that's I think really cool in this space is our open-size statistical review rocklets which is kind of an annotation scheme that implements their statistical review guidelines and allows you to kind of like annotate that as part of your documentation and I was kind of curious if anyone on the call has experience working with those and what their experience has been because I think that could be a really cool way of starting to annotate you know these types of test cases and then that gives us a mechanism to introduce tests or documenting you know standard data sets that we've used and implemented using packages and contribute that back into kind of the repository itself so really interested to see how that continues to grow if anyone's used it before I can say not yet yeah last time I checked in on it it was still in in development but I haven't in a couple years now so I'm really intrigued to see if it's you know started to grow some legs and be put into action I think it might be time for us also to revisit that so yeah interested if anyone on the call happens to have experience with it but as there's no other questions trickling in maybe I can pass it off then to pre them to introduce our last topic thank you Doug again this topic is about our package reviews and it's it's a recurring theme and one of the challenges that each of our organizations has been facing as we know we saw from Julie and stock you know reviewing the packages for their accuracy or even their risk levels is something that has to be done in order to be operating in a qualified environment and it's one of the things that the QA teams are particularly interested about and is something that they keep bringing up in all of our discussions now with respect to the package reviewers the main important thing is the qualification of the reviewers as Julianne has mentioned and which we are probably all aware of you know on an yearly basis we are all of our organizations probably collect our resumes updated resumes to be made sure that they are stored because we're operating in a regulated environment and that's one of the things that the regulatory authorities would probably need when they're auditing so in terms of qualification of reviewers that's something that we're also trying to make sure that we have some sort of document or we have documented what kind of qualifications they possess so it's not just about the education it's also about they've used that package that they're reviewing before or if they have experience with respect to that package that's some of the questions that have been raised and that we're trying to answer as well now as Julianne and others mentioned also is that just because you're a reviewer of an R package does not mean you're not you're qualified to review every single package so could be that you're not qualified or equipped to actually review packages that are highly statistical complex modeling and that would probably require a few more statistical help so maybe you know a package that is highly statistical nature should be reviewed by more statisticians and also a lesser you know more less packages that involve less of statistical modeling more of utilities maybe could be reviewed by a reviewer who's more qualified to review them so that's some type of you know package versus reviewer questions that we have also been raised in our organization and others as well another topic is also that about standard workflow or guidance or procedures I don't think there is any literature out there that actually guides us as we do for the actual qualification now that we have started doing these package qualifications and validations for open software there's a lot of literature coming out regarding that but with respect to specifics about the reviews themselves there's very little guidance and so you know that is another area of topic that we're trying to see if we can look at some guidances but at this point we might have to actually do another white paper regarding this particular topic and this brings us to the actual question is what kind of a documentation you know we would need and things like that when we are actually doing the we are asking the reviewers to review a package for the qualification purposes and you know we have a checklist you know somehow this is my template checklist about what what needs to be done what needs to be checked with respect to the package and say it could be the metrics from the respiratory cap or you know we could be some other in-house qualification app that's generating metrics they might review that or they might go to the source of the package themselves and check out what what the different changes have been over the years and things like that also you know some basic questionnaires with respect to what the package is doing so for example is that you know package you know the results from the package are they reproducible especially if it's using some site of random number generator or you know things like that are you know is there is there some other imports that that package is using and is that import you know qualified to be actually useful useful for our purposes and also you know is there any periodic publications that these packages have been you know demonstrated and so that's something that is also useful so a questionnaire for the reviewers as well to answer and guide them and also another important part is the timing or the frequency of this review so the packages as we all know could be updated you know sometimes in a within a month or so and so would you actually approach your reviewers each time they're reviewed they're updated and ask them to re-review them and if they do are they focusing on just the changes themselves are they actually focusing on the entire package again and so these are the questions that we are trying to answer as part of a reviewer and you know that's something that would be really helpful for all the organizations so you know if you had some you know some ideas some questions about this you know this this would be a really wonderful forum to start working on the I mean asking these and maybe we can get answers together I'm also realizing that we are almost we are at the hour yes um so yeah these conversations are always very interesting and we are running way too fast out of time but it doesn't mean that we are tabling it and are not coming back we really encourage people to join the all hands meetings that we are trying to make happen more frequently and yes we are always we are doing this an hour free time so we are we are we are always promising to get better but we already have like some plan for this year so please sign up for the our validation hub mailing list to stay in touch with these and we will have all hands meetings coming up over the course of the year and I'm very sure that at least two or three topics from that list will be extensively discussed in those all hands meetings amazing thank you so much for your questions and we really appreciate you for taking the time to tune in to your curiosity on our validation I've also posted a link to the webinar recording in case you missed parts of it and you came in later or if any of your colleagues want to watch it it's on the our consortium website on your webinars you can also see the old the past webinars there as well thank you Julianne, Doug, Prithem and Eric your passion shows and I loved getting an insight into the work you all do so I appreciate it thank you Maya and Agita for amazing this thank you yeah thanks everyone and thanks for being flexible as we had to make some coordination changes with how things ran so thank you everyone for your willingness to float with that thank you thank you bye bye have a good rest of your day you too