 and kick things off before we get into the main presentation, but give it another minute. I'll save it for people to come in late. Just see, you can see this. Are you seeing that in presented mode or are you seeing that in the, it isn't- The main screen. We can see the main screen. Excellent. And then the presenting mode again. Oh, good. Fantastic. Okay. Well, let's slowly get going. And then if people are a little bit late in and they're a little bit late, I'm sure people will pick up what's going on very, very quickly. As we've got a few presentations to go through, I want to allow a better time for that discussion at the end. So thanks, everyone. It's been a while since we had our last Our Validation Hub meeting. I changed the name for it because realistically it's not been quarterly. It's probably been about six months since we last had last had a meeting or more. But that doesn't mean that we haven't been active as a hub. And if you're not signed up to the mailing list, I'd encourage you to sign up to that because we do share when we put our new blog posts and so on. And if you want to sign up for that and we'll keep checking the website as well, you'll see some of the latest activities that we've been working on as a hub. What we're going to do is because we've got this very focused session today and there'll be another one on May 24th that we're going to follow up with. So part two on the same theme will occur then. I'll send an invite out afterwards. That hasn't gone out yet. And so if you're on the distribution list, if you're signed up to that, which you can do through the website on the Contact Us page. If you're signed up to that, then you'll get an invite later today from me to join that next meeting. And these two meetings really are with kind of our theme for the year. So those of you who've been a member for a long time will have seen we've put roadmaps in the past and talked about what we're trying to do. There's an initial focus on a white paper and what I kind of thinking our methodology was. That was followed by a focus on tools and we're still building some tools and you'll get some updates via the distributionist following this meeting separately from the main thing today. But the main thing today is around how people and the main theme for the year with the group is how people are using the framework. So what are people using that's working well? What are where are people adapting our framework? And so really it's about communication again, but it's about making sure we understand how people are using what we've put out and where we might need to provide some more information potentially through future white papers and hand or tools. So that is essentially the majority of my intro. We've got four talks today, Roshnav artists, Merck and GSK and we'll get the, he presents to quickly introduce themselves in their case studies. So they're gonna walk through case studies at their company about what they're trying to do with some sort of highlights of things that have gone well with implementing the framework and areas where they've adapted or changed or feel that there should be improvement. At the end, we'll bring it together for a sort of full discussion Q&A. So some of the presenters may give a little bit of time the Q&A during their presentation, we'll have a big Q&A session at the end, which I'll try and manage for everyone. So just before we get into the main thing, so the first one will be Colleen Zevolos from Roshnav, so I'll hand over to Colleen in a second to present, but just whilst I have your, hopefully have some of your attention, just to note that this Thursday, so April 28th, this Thursday that our consortium are running a session called reporting table creation in R, and you can see the advert for it there on the left. You can sign up for the R consortium homepage as a webinars link that you can quite easily get to from the top menu there. If you're not aware of this, not signed up. It's not a R validation hub activity, but we are part of the R consortium. And if you're here, you're probably in some way involved or interested in R adoption within your organizations, and the theme for those webinars is very much about R adoption. So the next one is on tables, and if you may be aware that there is an RTRS, tables focused working group through the R consortium as well that runs in parallel to this one, so we consider those one of our partners. So if you wanna hear about that, Gabe is going to be leading presentation talking about something called R tables, which many of you probably be aware of as an open source package from not just R tables, but our whole ecosystem around that. And the other thing I've already mentioned today is the R validation hub, all hands part two. So 24th of May, you'll get that invite soon. You shouldn't, you won't have it yet, so you'll get that before the end of the day. And we hope for more companies to do their case studies. Okay, well with that, I will stop sharing and hopefully Kalina's online and ready to, ready to, Kalina's not. I'm jumping in, Andy. This is Doug. We have, I'm Shaman, representing Roche today who's on the call. Fantastic. I love it when problems solve themselves very quickly. So that's fantastic. So Shaman, I'll let you take over sharing. I tried to do it. I first have to allow my Zoom to be taken. I will have to restart the application due to like my new mag does not allow me to share. I have to restart Zoom. So we'll be back in 30 seconds. You know, Keith, if it's longer than 30 seconds, I'll switch the order. So second on the list is Matt Fiddler from Novartis. So Matt, if you could be on standby, just in case. Otherwise, I will try and give it 20 or 30 seconds more of a fill until he's back. Are there any questions that anyone has immediately kicking off before we start? I'm not expecting it, but someone wants to bring from my voice, you're welcome to say something. Sorry for that one. But fortunately it was that another problem that's also instantly, so on the nicer part. Okay, so are we able to see my slides right now? Yes, and the presentation is good. Cool, thank you. So as we already know, my name is Shuman Maximruk and today I will present our package validation by Rosh Farma validation team. Led by the Colin Zebius and Doug Cahillhoff and the members of the team are also Ranzo Brasci and myself. So first I would like to unite some design priorities we put in front of us while preparing the process. So first is leveraging best practices for our package development. We believe that those practices are some kind of guidelines that adhering to each will allow us to minimize the risk of potential flaws in the package. Then we wanted to automate the process of getting those checks done, but at the very same time avoid special cases for particular class of packages. So if we ever decide to allow some exceptions, we want to allow those exceptions for every possible package, not just for particular subclass. Then moving forward, we want to minimize the packages scope. So as you are aware, a lot of packages in R does not necessarily provide a unique functionality. They are rather interfaces for other functionalities like databases, APIs, and so on and other. In that case, the team that is writing those packages are responsible only for the R language part of this package. So we are not interested in the system they talk to. This will be handled by another validation process. Then important part in our process in the human and we want the human to be in the middle of it. So for instance, if some automated checks are failing, we want the human to have the last decision what to do with such package. Thanks to that, we believe that we balance also risk mitigation against automation because for instance, if a package is failing for some like false positive even problem or a minor RCM V-check node that is not applicable to any regulatory use case, then we want this package to be permitted. And thanks to human eye in the middle of this process, we can leverage that one. Going forward with our validation, we target this for systems, mostly containers. It's because containers can be shipped for once device to another, saving all of the system settings. And this ensures that a validated package that was that was valid in a container will work the same way regardless of the system it's running on because of that container. And last but not least, we want the whole process to be transparent, to encourage in-house development and iterations. Here's a small graph that presents the sprints of our process. I will briefly go through it and then explain each step in detail. So everything starts with a validation, with a request. So package has to be requested to us and it can be either external or internal package. Externally developed package can be CRUN packages, BIOS packages, public GitHub, public GitHub, whatever. While internal packages are those developed within Roche and those are usually submitted via CI-CD. Then we run our to that first automatically gathers package metadata and just after triggers automated set of checks. The later steps depend on the results of those checks. If the checks passed and everything is fine, then we run reverse dependency check against other validated packages to make sure that the upgrade or a new package will not crush anything. And if that package is also passing, then we are publishing the package to a validated package repository and generated a report. If the automated check failed, then we have that human in the middle, I've mentioned previously and that human eye represented by two, by a pair of eyes on this graph actually assess the severity of the failure and whoever it can potentially inflict some of the regulatory work. If it's not severe, we justify those failures, why we can permit to happen and proceed with the later validation as it were passing, as all of the checks are passing. So reverse dependency check and so forth. As for the submission, the submission really has only two prerequisites. The first prerequisite is our dependencies. We require all of the dependencies to be validated first and every unmet dependency will immediately stop the execution and the execution of the validation process won't resume until all of the hard dependencies are validated. And the second prerequisite is that the source code of the package has to be reproducible. So it has to be either a Git repository, a Git hash or a checksum associated with a parable. Later, the submission triggers a CID in the workflow managed through merge requests. This actually allows requesters, developer but also a support team member to see and monitor every part of the process, every part of the validation process. So every step is recorded and can be witnessed from inside this merge request. This also provides a grant for discussion. So developer or a person that requests the package can ask questions, raise concerns, basically tell whatever they have in mind regarding this validation inside of the merge request associated with the validation. So everything is one place and it's kind of public. Public actually are obviously to the internal extent. And as a result of that one obviously, submissions are overseen by the support team member and we are ready every step to help with the submission. And the next step is actually the package method data collection. So as I said, we have an automated tool that collects data about the package that we later used to populate the report. And we start with a basic method data. So name, version, alpha, description, documentation, basically everything that can be used to present this package in a later report. Then we execute RCMD check and record those results. We execute tests and calculate coverage. Then we map unit tests to the documentation objects building a traceability matrix. And last but not least, we record a system information about basically about the system where all those checks happened. And answering the question of how they are collected. So they are first collected, all of the checks are executed in a control to a producible analytic environment. So basically in like similar words, an environment that already has an R validated in it. That's simple as that. And secondly, we strictly manage dependencies. So we allow only dependencies that were later violated prior and they can be installed only from a validated R package repository. And here's some table that summarized those and the data we collect. Source control and documentation, our thing that's clear. If it comes to RCMD check, we, this is our hard requirement that it has to pass without any errors. Ideally it will pass also without warning and notes, but this can be later addressed. As for tests, they all have to pass. The coverage has to be above 80%. And as for traceability, every exported functionality has to be evaluated at least by one unit test. But it can happen. Sorry, warning in progress. Okay, never mind. But obviously it can happen that a package may have some gaps that has to be addressed. And it can be, those can be really minor. So like coverage is for instance, 75 instead of 80% or some small note about enhanced package not available for checking. In such case, we can address those gaps. As I said, using that human in the middle, I will repeat it over and over again, but I simply think it's very important for our process. And those addressing gaps can go into paths. It can be either CRAN comments, margin files, so shot out to CRAN or remediation procedure. Unfortunately, I don't have much time today. So I cannot explain those processes in detail, but feel free to visit our pharma case studies GitHub. There will be this exact presentation, but with slides added about both of those sections. And answering the last basic question. So what comes after validation? So the divided packages are uploaded to a validated R package repository. And we think that it's also, although it seems like it's end of it, but we think that's an important part of our process because it actually allows us to leverage CRAN style rolling package release. So we can use snapshot, we can use archive, we can use simple remotes install version to sometimes roll back if the upgrade is not sufficient for us. We can also, it also helps users because users, we embed the appropriate option repos to the image. So then users using it does not, don't even have to know whether they are using a validated package or not. They simply will always get a validated package. They will, they can focus on the purely on work instead of thinking, whether I got a validated package or not. Oh my God, what to do? They don't have to do it. They just get a valid package using install.packages and everything works. And the second step after validation is report generation. So report, our report is a mix of static and dynamic document. So static parts are basically part of the process, extensive vocabulary used during the check process and so on and so forth. Why dynamic part is those are simply made data collected during the check embedded in the report and rendered together. And all of this is uploaded to the document management service. And to summarize all of this, we believe that we created automated strict set of rules, but a set of rules that only highlights gaps to the support team members. And important note is that validation applies only to a specific version of the package run on the specific system. So if we like upgrade the R version or change the architecture, the validation does not holds. And to say it at the end, we believe that the process with developed assures high quality of the packages for regulatory analysis reinforces internal developers familiarity with best R programming practices because of those possible reiterations of the package after addressing gaps and reduces risk by making our analysts and developers more capable contributors. And I think that will be it and I really appreciate your time and thank you for that. Thank you very much. I already have at least five questions for so, but we're gonna save questions for the end because we're on the sort of 10 minute mark. So we'll go to the next question, the next presentation, then we'll come back and some of those questions will probably be generic to many of the presentations. So thank you for that. Next up, we've got Matt Bidler from Novartis, I believe. Are you here Matt? And is it you that he's presenting for us? I'm here, can you hear me? Yes, you can, hi Matt. Great, let me see if I can share my screen. I don't know if I need special permissions or not. I think you should be seeing my screen. Yep, we can see that. Awesome, okay. I'm not a Zoom user, so I'm trying to move things around here. Okay, my name is Matt Bidler and I'm presenting on behalf of the R Governance Group and Dave is also here with me. What we're talking about is our package risk assessment and validation that we're currently using at Novartis. It's a little bit different than some of the other approaches I've seen here. So I think there's some discussion. So the basic approach here is a risk-based setup with pre-installed packages that allow ad hoc install packages as well. So there are four different types of risks that we have. We have high-risk packages, medium-risk packages, low-risk packages, and then intolerable risk profile packages. And we'll talk about what goes on in each of those cases. And for the user, we have two different use cases. We have one where everything's pre-installed and that's always available to the users as they start up their R session. And then we have an ad hoc installation to address a specific analysis need as long as it meets certain requirements for regulatory work. So as we talked about here, we have three different types of risks. First is the low-risk types of packages. In the low-risk, we wanna make sure that it's from a high-trusted source, for example, BaseR, recommended software from the BaseR distribution like NLME, for instance, or the Tidyverse dependencies. If it's something that's from a reputable source, we believe it's also a low-risk that includes something like RStudio or Microsoft. The other thing we assess for reputable source is if there's a large amount of community usage, which includes a very high download rate or a large number of reverse dependencies. And then also some packages just by their nature are low-risk. For example, a display package where one of the display packages in R is praise. It's used for test that. But its only purpose is to praise the user for doing a great job. There's not really any risk to analysis by telling them they did a good job. Or for instance, data packages where they only have data within that package. There's no risk to analysis for those particular types of packages. So there are low-risk package. So if all those are false, we go to the next tier of risk. This would include coverages that are queried from the package. Our current threshold is 60%. If you're above 60% coverage, we assume that you're a medium-risk package. The other risk assessment that we look at here is this package in a peer-reviewed journal with credible citations. If that's the case, we also consider a medium-risk package. High-risk packages must maintain some certain criteria. They have to have documentation, they have to have releases, they have to have frequencies of updates, but that's not sufficient to be in our system. We also have to have a user to not just installation qualification, but additional PQ tests. So these additional tests and what's used in testing in our environment at Novars are categorized based on use case. So for the use case one, the pre-installed packages, all packages undergo formal IQ and OQ testing by IT. Low-risk and medium-risk packages do not require a business unit testing or no PQ testing. We still believe there's adequate evidence to consider them sufficiently trustworthy to use without the standard PQ. All high-risk packages require business testing to or the PQ testing to verify the most important functions that are used in analysis. The documentation for all this process is done with standard lifecycle and management tool and with QA approval. For the ad hoc usage, we make sure that the user installs the package and runs all the package tests, vignettes and examples. We make sure the user does a risk assessment. All that information is stored in whatever project location they have their data in and their analysis in. For high-risk packages, business user asked to store additional evidence has proved the credibility of the package, review the evidence and the evidence has to be reviewed with the project team and the project team ultimately has the accountability to sign off on any high-risk package used in an actual analysis. We also have a tool to help with this ad hoc installation and risk assessment. This tool is an R package that ensures the proper ad hoc package installation. By default, it uses whatever snapshot date we have in our system. Currently, we're using M-RAN, although we could use RStudio in the future. But once it has that M-RAN date, we have the user install whatever package version that existed at that M-RAN date to ensure maximum compatibility. We also install all the suggested dependencies and the tests to be able to run the examples, the tests and the vignettes. All those are run during the time of the ad hoc install and all that information is stored into a zip file. The zip file is saved for evidence of that successful installation was performed and you can look at it and see the review, the data to make sure that it makes sense. It also stores the package version information if you're using a different version or a GitHub or GitLab or a Bitbucket type of source to make sure that whatever you use is reproducible. It also conducts the risk assessment that we just talked about using metadata from the R package ecosystem and uses that to infer what the risk is based on our current practices. And that is stored in a Word document with embedded R information in it. And with that, that's everything I have for this brief overview of what we do at Novars. Is there any questions? Please feel free to ask. Thank you. Timing-wise, I'm gonna move on with the questions but I think there's already been one question in the chat. I think that's another thing. If people wanna put the questions in the chat at least then we've got there. I think we're gonna run out of time to be honest to get through like a ton of questions at the end, because there will be lots. So put the questions down and if we've got them documented and this couldn't, as we mentioned, a couple of the presenters have mentioned already, we are looking to capture this as well as the meeting here today. We're looking to capture this on the site in GitHub. So we will have questions that we can address through that in kind of formal written case studies as well. So all the questions you've got will be captured and answered in some capacity, whether we send our answers after the meeting or whether we address them in the meeting or if we do them within the papers that will follow. So please do document the questions, put them out just in case you don't get a chance to answer the question at the end. With that, I'll move us on to our third presentation, which is Pretherm, if you are online. Can you hear me? Yes, we can. Okay, and can you see my screen? Yes, we're not seeing the presenters, you were seeing the start, now we are. Yeah, that's for you. All right, thank you for the opportunity to present at the meeting and I'm Pretherm, I represent BARTs from Mark. We are being trying to implement the R package qualification based on the different frameworks that we've been exploring. And so this is just a simple case study that we are gonna be demonstrating using GGLE package. My co-authors are Paul Bernanke, Jane Leo and Elon Zhang. So starting off with respect to the package qualification implementation, we just looked at first the different guidance that's available from both regulatory authorities as well as cross-industry initiatives. So we focused on both the FDA guidance as well as the ICH guidelines for the different validation requirements. And also the different frameworks that are available from the cost industry initiatives, both our validation hub and transfer rate MSA initiative. So we have taken elements from both our validation hub and transfer rate framework to implement a qualification framework that minimizes both risk and increases the contents in the package being qualified. So that just a quick overview of what the qualification framework at Mark is being implemented as a risk-based package, qualification which includes a risk-based strategy which looks at the different types of analysis and reporting deliverables that are being used, that are being delivered using the package. And the goal is to create the documentation that contains qualification details related to that package based on some pre-specified criteria. So there's some examples that are available that can be used to look at the different risk levels is the deliverable. So if that package is being used for some sort of an external deliverable, such as an ECTD required deliverable, including CSR requests and submission packages, then that package has to be qualified under the low risk. If it's non-ECTD or other requests such as data monitoring committee requests or manuscripts and publication requests, it could be either in moderate or low risk. And if it's purely for data exploration or exploratory analysis, then it could be in the open risk or any of the other categories. Based on this, we have also defined the different criteria that are required for the package qualification. So we have specified about five different criteria that are required for the different levels of classification. So a low risk package needs to meet at least the first four criteria listed here, C1 to C4. And one criteria is sufficient to qualify it within a low risk category. A moderate risk can be qualified using any of those C1 to C5 criteria. And an open risk is any package that is available on PrAN or Bioconductor and is not qualified under either the low risk or moderate risk. And we have guidance in space within the organization that open risk packages cannot be directly used for regulatory purposes. So that is something that is already in place and it's a part of the standard operating procedure. Now, within this context, we also have an implementation of the global R library with the mark, which is a library that contains a set of directories with all the R packages and the dependencies installed. So the goal of this R library is to establish the shared baseline approach that RStudio approaches within the R package management and helps with providing all of the users a uniform and a shared baseline environment we can start with. And they don't have to worry about different dependencies or any of the other issues that pop up with package management. This global library is updated approximately every three months or as per business needs required. And in order to update this and also maintain the R library, we are using the RStudio package manager, the RSPM for the package repository server and also to install the packages in the global library. Within the global library, we have the three risk categories as I specified before, the low, moderate and open and it is a nested structure. So with all the low risk packages are part of the moderate risk category and all of the moderate risk are part of the open risk. The open risk category packages are anything that a user installs or downloads within internally within work from either from the RSPM as the RSPM is already synced with the CRAN or by a conductor. So they have access to most of the packages that are available online. So when we come to the package qualification workflow that we're implementing, this is a quick overview about how the qualification work from works out is we use a snapshot date within RSPM to pick a particular date where all the packages are frozen and then use, then also have a document about which packages need to be updated within the different risk categories. This package list only contains moderate and low risk packages that need to be qualified because the open risk are already part of the library and they don't need to qualify separately. Then we create a request form that can be used to inform IT about the installation of the different packages with risk categories. Before we send it off to IT, we do a driver installation of these packages to check if there's any issues with respect to installation or any dependency issues. Once that is complete, we send the request to IT and they start the installation process and once that is complete and simultaneously, we also generate the qualification documents that are required to qualify that package into a low or moderate risk category. With that in mind, the GGLE package was a case study that we would like to present. The GGLE is our package that extends GG plot to functionality and is requested by some of our internal teams for a moderate risk category for publication purposes. And so at the time of qualification, this package was in the open risk and so the process is highlighted here. It started with us reviewing the package documentation to determine which qualifying criteria the package could be qualified under and then we performed a driver installation for the global library update. With this in mind, I'd also like to point that we have an internally developed R package that does automate most of this process and it's not a manual process. We just use the functionals from that package to automate most of these steps. So once the driver is complete, we also do a check of installation log for any errors or warnings regarding this package installation and then also do a package code coverage and associated STLC documentation check. So you can see here two screenshots about the different STLC documentation. The one on the left is from the GitHub of the GG plot too and that is replicated within the RSPM because it's synchronized with the CRAN and the Bioconductor sources. Once this package coverage and associated STLC documentation is complete, we also cross check against an internal database which contains a list of R package authors that are in the trusted category and we call a white list. So these are authors that have had a history of package publications or whose have had package downloads more than 1,000 within the last two years. So we have a set criteria which includes these trusted authors. Once this checks are complete, the qualification document is generated. That qualification document I would like to point out if I can open it really quick is a HTML file that can be used to... And this is the qualification document that is actually generated for the different qualifications and it includes a complete documentation of what risk levels including the qualification date and the criteria. So it has information about the packages including the STLC as well as the different dependencies for the different packages that are dependent on this and as well as the qualification details in detail about how this package is qualified into moderate risk. Go back here. So in conclusion, we have implemented a risk-based qualification process at Mark to classify the different packages based on the deliverables that are generated using that package. And we demonstrated the package framework qualification framework using the GGLE package which was qualified under the moderate risk. There is still work in progress in the internally developed R package for automate more of the processes including checking for the different installation errors as well as dependency checks. And there was also ongoing work into defining the pre-specified criteria to qualify an organization or group of vendors as trusted source. This was brought up during a recent internal QA team discussion where there was more feedback given about how we could be using a specified criteria to qualify an organization as a trusted source or trusted vendor. So we're still working on that particular part of the process. So with that, any questions or feedback? Here's my email and that's pretty much my presentation. Thanks for being done. We're a little bit behind on time so I'm gonna quickly get on to Ellis and Becker. We'll probably have about 10 to 15 minutes maybe depending on how quick you guys are for questions at the end. Remember, all of this is feeding into documents afterwards. So I'm just gonna keep filling guys until you've got your slides up on screen. But everything will be shared through those talks afterwards. If you don't get a chance to ask a question today, you'll get a chance to do that for them. And in May 24th, when we repeat this exercise, we'll do some number of presentations as we go on a good half hour safe for the discussion. All right, with that, I'll hand over to Ellis or Becker whoever is going first. Well, thanks, Andy. Ellis, do you wanna lead us off? Sure thing, yeah, so today we'll be talking about our package assessment and the general ideas behind the way that we're approaching assessing various R packages. My name's Ellis Hughes. I'm a data science lead and the innovation team at GSK. Becker, you wanna introduce yourself? Sure, yep, I'm Becker Kraus. I work with Ellis. I also wanna give a shout out to Teal O'Blank who's also on the call and he's working with us on this and advising us as well, so. Okay, let's go on to the next slide. So the way that we're approaching package assessment is that we're really trying to do risk mitigation and it really is along a spectrum. So we're trying to balance case specific, time and effort involved with package assessment with how quickly can we get this done and how much information can we gather quickly? So there's two sides of this. It really depends on the risk the organization is willing to take on and their interpretation of the documentation that exists that's been presented in prior presentations like the ICH E9 which is you need proof and documentation that the software is appropriately tested. And so we're kind of taking into account advice from the RValidation Hub and just generally observing how the pharma community is approaching this. So there's one end of the spectrum which is complete reliance on automated assessments which is really only trying to use measurables really produced a lot by like risk metric or things like that like test coverage and whatnot. You execute the included tests. You look at general package development criteria. So does it pass the R command check that's included with CRAN? Is it on CRAN? If it is, there's usually a lot of trust already with the package itself but really we're trying to gain empirical evidence to prove why we should trust a package. And the value in this is that it's really scalable it grows really quickly. You can assess a lot of packages with minimal effort but it does have the possibility of you just start generating data without looking at what does this mean? What is the implication of this assessment? And there's not as much nuance in it. And then if there is any situations where if you've completely automated everything and something crashes, adding a human back into the loop can really gum things up and really make this slow. And so your fallback methods for complete reliance on automated processes needs to be really robust. The other side of the spectrum is complete reliance on expert opinion. So this is where you have an expert go in, look at the package, look at the functions, look at how you're planning on using the package. What is the package doing? Is it statistical in nature? Or is it data manipulation? Is it used for utilities? And that's really focusing on what is the package doing and relying on your experts to do this? And they can look at additional things like what is the development plan? What is the software development lifecycle? How robust is the code? How robust are the tests? The issue with this is that you're really tied by how much time your expert has to be reviewing these packages. And there's a limited number of experts, just there's a limited number of people in the world that have this amount of expertise. And then you don't really want them spending all their time just looking at packages because the reason you hired them you want them to be spending their expertise not on does this package work but actually using these packages. And so our goal is to hit a middle ground if you want to click space. And this is to leverage both of these ideas where we can use our automated assessments to try to target our experts so they're not wasting their time on things that's not valuable where they're not adding value to the review. So they don't need to worry at looking at like what is the test coverage? Does it need a minimum bar? Those sorts of things. It's given the way this package is behaving. Let's look at that and look at the way we intend on using the package and through the lens of the information we gather through this automated assessment what is the final conclusion of the package? Do we trust it? Do we not? Do we want to push it forward? And so, yeah. So we're trying to combine these two and reduce wasted time. So we do this by establishing a baseline level of trust. So we're using the ideas that exist that have been compiled by the ARM validation hub using risk metric, using tools that already exist. Test coverage, you can look at CodeCov and whatnot but then you also look at other things like the documentation quality. So that's a little bit outside of the risk metric world but it is still pretty straightforward to do where you start looking at vignettes. Are they helpful? Do they answer the basic questions? Can they get a user really going and using the package? The help pages themselves, are they instructive? Do they provide good examples? What are the main functions that exist inside of the package? And then you can look at the community itself. Is there a paper that they wrote that was submitted and was not accepted? Do the rate blog posts, do they respond to issues on the GitHub page? Things like that to really help derive the initial assessment of how they look at the package and that then drives this next piece that Becca's gonna talk about. Yeah, so when we have this baseline level of trust we're equipped with it, that informs really the next step in the process. And for us, that is really focused on testing and test quality. So with that baseline level of how much we trust it that dictates how big of an investment we're going to do in assessing tests and really doing a deep dive there. And so that involves figuring out test quality. It's a little bit different concept from test coverage and whether there's tests or not which we gain in that first part of the assessment. So figuring out test quality really does require us to look inside the code itself, inside the code base. So this step involves actually looking in the files and we need to either obtain the tar ball or the go to the GitHub repo, click around and see what we need. At this point, we've already identified key pieces of functionality that we're concerned with. So the most important functions are the ones we would focus on. And from there, we're trying to navigate the different files, you know, the source code and the documentation and the tests and so forth. A package and the package directory is not really meant for doing this easily. So what we've done, what we're working towards is having a nice interface for doing this more easily. So we have a prototype that we've built that allows you to whatever package you're assessing allows you to actually peek inside and navigate these pieces a little easier. So I just have a series of screen screenshots for the app here, for this prototype. For this example, we're looking at the read Excel package. We have a dropdown here on the left of the various exported functions we can toggle between. Let's say I'm interested in looking at the read underscore Excel function specifically. So that one's selected and I have three tabs here for some key pieces. Test code, source code and the help page. With the source code, it's literally just the code file for that. So we can see, you know, exactly how it's being programmed. Perhaps of a little bit more interest for this exercise is the help page and the test code. So the help page allows us to see exactly what the function is meant to do. It's inputs and outputs and really the different scenarios, different use cases that can come up. And then from there, we can actually look at tests and the test files and see how well what we can see when I clicked over to the test code tab, there's a little dynamic portion of the UI that pops up. All of the test that files, all the test that scripts that call our function of interest, some place in there. So you have access to all the places it's tested, not just the test read Excel function. So another thing that we make this process easier is if we can direct whoever's doing the assessment, if we can have a visual, a little more visual direction, visual cues to make this quicker so they can more easily find the places that the function's being tested. So here you can see the code highlighting that helps with that. So I wanna step back and say that the risk assessment app that Marley's spearheading is really a nice way to take the risk metric package and add this human component. So we're talking with Marley about adding this in potentially to the risk assessment app, this ability to manually navigate the actual package contents and be able to perform this exercise in the same place as the rest of the assessment that we mentioned earlier as the rest of the assessment that involves more of those automated pieces. So that's all we have as part of this presentation but we're happy to take any questions in the next few minutes. Thank you. Thanks, Becker. Thanks, Alison. Thank you to all of the presenters today. We were perhaps a little bit ambitious with the four presentations to take as we've gone to a point where we've only got nine minutes left of the call. So what we're gonna do is pivot slightly with this. And so next time when we meet on the 24th, we will focus that more on the discussion side. So for now, if there are kind of clarifying questions for the presenters, it'd be good to hear those. Otherwise, as I say, please use the chat and document those and hopefully we can get an export of that chat later so that we can address those within the discussion because there are no doubt plenty of questions. Julianne's just posted in there. Can you please send the questions to her email address there? So if you can post the questions to Julianne and thank you before we do run out of time, I just wanna say thank you to Julianne for coordinating this and getting all the presenters together as well for today. So thank you, Julianne for sorting that out and thank you again for the presenters today. So are there any quick sort of qualifying questions that anyone has based on the presentations today? I'm not sure if everyone can unmute themselves. I think people can self unmute. If not, I will ask one slight, one question that verges on discussion. And that is with respect to when packages enter in this process. This question for all presenters. Who governs what packages are allowed to enter the process? So can any package be put through your processes or is there someone kind of, is there a pre-gateway before you even get into the process decide whether a package is allowed to go through that or not? And this is freedom. So within Moraq, we have a process. I mean, we have some of our governance committee that actually looks at the different use cases for these packages before they're qualified. And also during the qualification, we do have a review panel of comprising of one programmer and as well as statistician that do the review of the different qualification documents and make sure that the packages are properly qualified. So we do have some gateways that actually govern these processes implementation as well. For Rosh, so internally developed packages. So those packages that were developed for Rosh employees and the source codes available on the either Rosh GitLab or GitHub, so any like source code platform. So those packages can be valid free. So basically on demand. So whenever the maintainer decides the package is in a place when it can be validated, the process starts. As for the crown packages and exterminate packages, they also can be requested by anyone at the very every moment but those are required acceptance by the team, by the validation team. From my point of view, this is Matt at Navarra. So we actually have our automated risk assessment forms look at all the packages that are evaluated for the ones that are high risk. That's where we do the most reviewing because we just wanna make sure that these high risks are not an entire risk category. We also make sure that we review the PQs in that point. So in general, we're kind of sorting this out as well. We're gonna have a team of individuals that are experts that will be reviewing the packages just for use case. And then there's an additional team of people that will actually be doing the evaluation of the individual packages that'll be giving feedback as to whether we'll be accepting those packages. So it's kind of like a two-fold approach. At least that's a general thought right now. Thanks everyone. Just to give a flavor for the sorts of things that we'll continue in this discussion. There's been a couple of other suggestions around taking discussions of Slack and so on. We do have a Slack group. We've not been heavily used by this group historically but that's another area where you can take some questions to potentially as well. So we'll try and work out the best way to get that conversation kind of flowing continuously with the presenters. So my idea was that these things shared on GitHub and there will be a public record of like all the quick Q&A that will be there. So if you're not able to make a discussion for example on the 24th of May, then you will be able to access some of the information, the questions that have been asked in some other way. I can at least promise that. Other questions that are coming in, things around like how have people worked with their quality teams around this process? Do people have like new SOPs and the likes to govern those? So that one's come in. We don't have to answer all these now just to give you an idea of the source things are coming up. How do you look at, if people are using combinations of packages that are not part of your testing process, if people are doing things that, you can only test so much. So if people are using things in an ad hoc way, does that matter? How do you account for that in the process? The question that hasn't come on by I wanted to ask was around if let's say a study team is looking to use a package or you're looking to use a function that's in a package that isn't in the current set, how would you accommodate that? Would it have to go through the process or do you have a means for teams to kind of all by self certifying it? It's okay, I know this one function is fine or would they have to kind of write that up, write the function themselves in their own code? So how would you handle packages that are not part of your base set? Because obviously if you install a new package, for example, if you had, I think it was a Roche presentation, we talked about containers. So if you had a new package to a container, you kind of changed what's being tested, mandated, qualified. So how do you account for that? How do you account for that? And there's a bunch of other questions on there already. So with that, with three minutes left, I'll give everyone about time back in their calendars. We will pick this conversation up. Thank you again to the presenters and we'll see you guys all soon. Fantastic turnout today. So obviously a lot of interest in seeing these case studies. So we look forward to a couple more cases and a big discussion next time. Just a quick question, Andy. Will the case studies that we sent out, we posted on the GitHub? Julian, do you wanna talk a little bit about the process? Yes, so we have a GitHub repo and you should be able to create a pull request and we should go through and actually approve them. So we put that on our to-do list as well. Sure. Okay. Thank you, everyone. Clearly a lot of interest in this topic. Nice to see everyone back again and we'll see you all in a month's time. Thank you. Have a good rest of your day. Thank you. Thank you, everyone. Thanks, Andy. Thanks, everybody. Bye-bye. Thank you. Thanks, everyone.