 Okay, obviously we will have people join a little bit late or a couple of minutes in now. So we'll get going with the meeting for time. Maybe we run out of time last time, so we're going to waste too much now. Right, if I can get on to the next slide. Before we move on to the main topic, just to mention that we're, for those of you who are also following the Our Adoption series by the Arkansas Gym, we have provisionally agreed the title of the next one in that series. So Colleen and Ning will be joined by some guests at some unnamed time and features. So have a look out on the Our Consortium website for that. I'll mention it in an email update soon as well. And if you're interested in that with the topic of around submissions in art, if you have any questions that you would like to ask for regulators in general, but specifically for the FDA, I'd appreciate if you could send me those questions, I'll pass on to Colleen and Ning for that session. So that might tell you something about what we've got planned for the session, but it is a little bit cryptic, I appreciate at this point. But if you have any questions or anything you want to learn about submissions, we'll be sharing that in the Our Adoption series. You can navigate to the Arkansas Gym homepage and under webinars from one of the tabs at the top, you can get to webinars and you can find all the historical webinars there. And this one will be advertised there in due course. Okay, so for today to get quickly onto that, this is part two of at least a three part series now we've had an unprecedented amount of interest in people coming forward to share case studies of how they're implementing either the directly the thinking of the Our Validation Hub or as if you caught part one, you'd have seen slight variations on what we're what we propose. So if you missed part one, there's a link here that I will send out again afterwards as a reminder you can go on there now and watch the video and we're part way through a website update for the Our Validation Hub. So that will be updated on the website soon as well so you'll be able to link directly from that and go and watch the video of that. And the video of this should you have to leave leave earlier and think like that. So, from there. I did too much in terms of recap we had four presentations last time. And we've got two more this time. And that should give us a good 30 minutes or so to launch into a sort of general Q&A slash panel discussion so we've got almost all or almost all of the presenters from last time here as well so if you've got questions on the talks last time or the two talks a day or just general questions of people who are implementing implementing GXP after GXP kind of approaches within their companies, then we've got a good 30 minutes or so to make sure that we discuss that today. Okay, unless there's any immediate questions coming up I will hand over to, hopefully, to Damian. I didn't do a check if there you are. Yeah, I'm here. Excellent. So I'll stop sharing. And, Damian, hopefully you've got the rights to share. Let me check. Can you see my screen. Yes, we can. And it is now in slideshow we're seeing that fine that's great. Perfect. So welcome everyone and really happy about the last session that we had. We saw four really good examples of how R is used in the GXP environments, especially our validation half approach on the processes. So today I would like to spend the 10 minutes that I have on talking not only about the validation itself but also on the broader context because I saw that in the questions you asked a lot about what happens next and some additional questions. We have a lot of experience in working in GXP environments, so I would like to give you a peel off experience that we currently have and would be happy to discuss this further during the discussion session. So after a short introduction, I will go through the three steps which are the accuracy, reproducibility and traceability. The introduction. My name is Damian or Davish. Hi, everyone. I'm the president of Ceylon and one of the founders and up Ceylon is a company that provides shiny consulting. So we work with other with our clients, helping them build shiny dashboards on top of the R models that they have developed, although we also work very often in R directly. We had a chance to work and we are currently working with many different Fortune 500 companies in restricted environments, especially in Fortune 100 pharmaceutical companies where we actually got to see different approaches that are used to using R in GXP. So let's take a look quickly on the GXP itself, which is here for ensuring a safe and reliable product. For accuracy, we have the violated packages part the process and the distribution of the packages. Then there is the whole SDLC process, how to actually allow using those results and using the application to actually move forward. Then there's programming practices that you should apply and from the reproducibility, how you can actually make sure that you can reconstruct the full environment used to produce the results. The last part that I'm going to show is traceability. So how to reconstruct the development history of the drug or medical device and how to have this accountability so the ability to resolve who actually has contributed what and when. Let's start with the accuracy. When it comes to validated packages and the process, we focused a lot on the testing and approval part. And as you have seen already in the past presentations, this process is quite similar. You need to set the right rules that you abide by and you agree with the IT team on that one. You need to test the meta information, you need to test the source code, and sometimes you actually need to write the test yourselves if you really need to use a functionality that isn't well tested. The next step is the distribution and infrastructure, you need to ensure that the data scientists have the right access to the packages and updating and adjusting how to track versions updates the repository and how to write new tests. Shortly about testing and approval. I highly recommend you to keep using the risk metric package and the assessment application. These are great open source solutions that are developed by our validation hub, and they already provide you all of the meta information that this needed to actually make your assessment and of course you can enhance it further and contribute back. So this is really, really great solution to have. When it comes to distribution infrastructure, I have seen many different ways of actually deploying the packages to the data scientists, what I wanted to share with you is the one that I like the most. So there are different approaches, providing Docker image that is available to the users. There could be an ability to have a full image within AWS, for example, that reconstructs the whole machine. I like the lean approach of using RStudio package manager that allows you to have your own copy of CRUN. However, within your internal infrastructure, that can be actually also used offline. So once you do the validation on anything that comes from CRUN GitHub internal, you can install this directly in package manager, and you can set up spaces there so that anything that you validate is directly accessible by data scientists in the development environment, but also in the production environments. If you want to use Docker, you can do it as well because you just pull those packages, not from CRUN, but from your internal URL that is provided. And this is great because there is no chance for a sleep. Your data scientists do not even have access to CRUN, but they actually pull everything directly from package manager. So what happens if you want to update or adjust your packages? You saw in the rush presentation and you haven't seen the presentations last time, definitely please go through all of them to automate as much as possible in the process. This is going to take a lot of waves from your bug and will allow you to quickly check and validate a new package. Once this is validated, you can push it directly to our studio package manager so that this is automatically available to your users. And here you can actually organize those packages into different groups that are called sources. So CRUN, for example, is one source of packages. There could be a local source or internal. You can have pre-compiled packages there as well. And then you can also group the sources into repositories. So for example, you can give URL to your users for the production repository and it could contain a subset of CRUN, some additional local packages, but you have full control over what the users are using in your environments. So one question that I often get is what happens when the package is very big or doesn't pass the validation and you really want to use the functionality. So first of all, there was one of the answers that you just asked the requester to add unit tests and to merge them back into the package. And this is a good solution, especially if the functionality is small and you can write those tests within a couple of hours. However, one of the interesting things that you can do, if you have a big package to use, you can actually create another package that subsets the functionalities from the big one. So imagine you have a deep layer and you just want to use the first function. You could create a new package that exposes only this first function and you can write tests to this. And of course, all of the dependencies that are used by this particular functionality. So let's jump into the SDLC process and the IQOQ and PQ. These are reflected by three main areas, the documentation, installation and testing. And this is very important because once you set it upright, you wouldn't have problem with deploying your application within the GXP environment. Otherwise, the process can be very lengthy and can involve a lot of people. So for the documentation, of course, writing this from scratch is problematic and it takes time. It is important to start just as your, or even of course before you start even doing any implementation. But the important parts to keep in mind is that you can generate parts of the documentation directly from the code and read me. So if during your development process you learn to document all of the functionalities to write a good read me, this is going to benefit you a lot during the SDLC process once you are actually going to production. For installation, I propose to use the infrastructure as a code approach where everything that you install as a library or any dependency is fully coded. Of course, we know RN, we know ACTRAT and we know Docker, but if you don't use Docker within your organization, you can actually use Terraform and that symbol that I'm going to show you in a couple of seconds. Then when it comes to testing, the more you focus on building units tests in your application that you later wants to promote GXP, the easier it's going to be within the process because then you just export your unit test and this already shows that your application is tested. And you can even lease the pull request. So please make sure that you do the code review within your company and then you can just lease the pull request that you have done. And this already confirms that there was a set additional set of eyes that have looked through any code that has been implemented. Also for more complex processes, you can even automate end to end tests. I have been in projects where it takes weeks to actually go through the whole process of testing, which is done by a completely separate testing that follows the screenshots and scenarios that are described for this particular application. So also updating this is problematic because you've changed one small functionality and then you have to change all of the documentation. You then have to process this to the test team and they have to go through the whole set of scenarios after any change of the code. If you have this automated, this can be done in a very, very fast way. So what are the programming practices that you should employ from the day one. You should set the coding styles, coding and styling standards in your organization from the very beginning, create a differential of them. What does it mean that you actually merge the code into the main branch, definitely use deep or any other sources version control. This is a must. And you should document all of the level rules, even the simple ones like date format, which is important, that it is not ambiguous. Sorry. You need to make sure that your code is well documented and testable. So extract the functions, avoid code duplication, write the unit test and to document everything. I have seen projects where the amount of code actually grows into one huge pile of lines without any additional functions and this is problematic. Once you actually want to go to production, we've already a great solution that provides a lot of value. So you can set up continuous integration pipelines, use linter and run tests automatically. And this will allow you to very quickly iterate on your application or any solution that you're building with our first shiny applications. There are some end to end solutions that allow you to go through a scenario and automatically make a screenshot of the state of the solution and compare the screenshots to the screenshots that you are expecting to see. And this can be a tremendous help in the testing process. Okay, quickly going into attributability and taking a look at the time. If you cannot use Docker in your organization Terraform will automate setting up servers. So it can set up many servers with given specification that you provide and even firewalls, and then Ansible will automate installing all of the libraries that you have. Important thing to know is that if you use Docker and you want to use RStudio Connect for example, RStudio will enable running Dockerized applications soon. So this is very good news, especially that I know that this is very important for many IT teams to make sure that we comply with other standards that are in the organization. So if you're going through the last part that face ability, the Git itself is actually providing you a very good history of how your solution has been implemented. But you should keep in mind that you should be logging a lot of information directly from your application or your R Markdown file, or any other R solution. You need to set up a format and there are really good packages that you can just use to provide logs, and you need to set up a separate infrastructure to collect them and to ensure that they are stored. You need to remember to include all of the meta information in any report or application that is generated. So date, author sources system version environment, and many others. And also one important thing to remember is that whenever you're building R Markdown reports in a Word document version, you should render and store a PDF version of it as well, so that you can retain it for for the historic purposes. Okay, so hopefully this was useful to you and hopefully I managed to do it in time. Thank you very much and would be happy to discuss it more through the questions. Thank you Damien. What we're going to do is put questions to the end, just to make sure we get to the point where we can open up the, open up the floor. So if you have questions, specifically for Damien please do ask them at the end when we open it up. Otherwise, for now I will hand over to Nicholas and Satish. Thanks. Thanks. Thanks Andy. I'll share this is Nick and there has been a slight adjustment so Satish is not joining us. We do have Jessica Higgins and Eli Miller with us to cover that. So, and can you see my screen okay. Yeah, perfect. Okay, so thanks for having us. Wonderful presentations. I'm very happy to have the opportunity and to attend all of them and I'll reiterate if you missed the previous session. Highly recommend going back and checking that out as well as the questions listed as issues so experienced building a GXP framework within our. So this is J&J specifically within our statistical decision science and clinical and statistical programming group. So, we took this very literally the question is, you know, how have we been implementing the white paper and what are the challenges. So this was kind of intended to set the stage we all know why we're here. Who's covering what so myself will cover package validation section and this is section four in the paper section five is system qualifications which Eli will cover. Jessica covering risk assessment framework I'll touch a bit on some of some of the challenges. We did put together a panel to try to be able to address any question we thought we would expect. But with the teach not being able to join and shown not being able to join as well. Some of those we might log as as the issues or will be log this issues and we'll follow back up and get answers to those. Okay, so hopping right into my so section four hits a lot on our package validation and similar to Damien's workflow that we just had. It's really about accuracy reproducibility traceability. The key part around accuracy. I mean the paper touches on base and recommended contributed packages popular packages splitting those into sort of different risk buckets. And going essentially different workflows applied based off of the bucket of package falls in. We completely aligned with this. There's really not much to add based off of the other discussions that have been done here. So between the risk metrics and whether or not take a an automated approach to defining the risk of a package and its acceptability for use versus a more expert opinion that requires much more time and view that I guess we'll touch on a bit later. But accuracy. Yeah, we've got three or four buckets. We pretty much accept base aren't recommended without much additional testing. The popular packages along with the tidy verse in our studio documentation kind of reduces a level of risk and then we've got a separate risk assessment that occurs for basically everything else on the reproducibility front. We are using. RM we are using Docker. The R studio package manager actually the slide that Damien showed is essentially our setup. And then we use an internal package for called in the setup to keep consistencies between like interactive execution versus like a containerized batch execution. So we have development happening interactively in our studio are validated container is separate from that. And then code would be executed just on that validated container and we want to make sure that interactive development area matches the back to execution area which is locked down and immutable. So in the setup covers that aspect of it. And then timber is one I just tacked on here right before actually the presentation. It covers the reproducibility aspect from a logging of systems packages and functions used errors warnings and all of that. But it actually is a nice tie it to trace the ability. I dumped in this one paragraph directly from the paper because I thought it was it was interesting and timber I think actually covers this. So, you know, we validate packages, or maybe even we validate specific functions within a package, but you get all of those imports with it. So having that risk of being able to trace what what can you use versus what you cannot use. Being able to provide that back and log what you've done for users to understand what they can and can't use but also understand if they're using something they can't use what's the process that they should be following now. If they'd like to use that moving forward. So timber hats added this functionality there so I'm sure there's I know there's other logging packages out there as well but I think in terms of the package hitting these few highlighted sentences. It does a great job handling that traceability aspect. Okay. With that I'll pass it over to Eli to speak to our system qualification. Yeah, thanks Nick. So, per the white paper, the primary goal of system qualification is to more or less prove that the our environment is working as expected, and doing what we intended to do. So that's the primary goal of the system qualification. The primary goal was to build the qualification process into the existing pipeline to kind of close that loop and require as little intervention as possible, in terms of, you know, human review or anything like that. So we decided to leverage virtualization and the fact that you know container images are immutable, and they're not going to change. And once you validate that image, any container that is being spun up from that, you know you can, you can kind of assume that container is validated there. It also leverages our existing pipelines for constructing these environments. And so we don't really need any new compute infrastructure to create the environments. It just gets run through the regular pipeline with some additional scripts. And the idea of how we use it was we leveraged VAL tools, which was a package that was developed through the fuse organization. And the, the idea there was we made one unified process of designing requirements tests and test code into one documentation framework so really you say validate the container, and it will validate the image rather, and it will run all tests and output a validation report that is then stored in the container itself. Each package that is intended for use, intended for use, and the R environment can get tested which is basic usage. So one of the things that we did was we installed the package tests into the environment so let's say you have deep layer. So one of the tests that we would run would be test all of the native deep layer tests just to make sure that they, you know they return what deep layer expects them to return. We also test to make sure packages can load cleanly and unload cleanly. In our statistical packages, we verify the results independently so we have reviewed literature that we have converted into our code so we will actually start testing the statistical results that the packages are outputting to make sure we're getting valid results. And the sum of all of these processes is a validation report that you know gets embedded straight into the image. So the report can then be reviewed by stakeholders, any issues with the validation changes need to the environment, anything like that if we're adding a new package. It's a pretty quick process to just add the requirements, you know, add the tests for it, and then send it to the pipeline and have it reviewed, and then actually be getting used by, you know, end users or at least tested by end users. And then if it was going through a manual QA development and release cycle. And that is all I had for that. All right, I'll pass it to Jessica to cover our risk assessment framework. Jessica, are you with us. You're you're on mute. I think Jessica you double muted. We can see you and we can see you're talking, but we can't hear you. Sorry, give it. I think it might have a solution. If not, yeah, if not, I can, I can hop in. Yeah. I think we might have to go with you Nick. So it's all right. I'm just thinking if you can cut you maybe if you can, if you can find a way around it, then you can help summarize when we get into the question Q&A. Sounds good. Yeah, so I'll do my best and just go if you can, if you can hear me try not to cringe too hard about completely botched some of this. You know, section section 6162. So purpose and intent. Intent is really what we wanted to focus. Hey, there she is. Hey, sorry, I called in my phone. I'm unsure why my computer is not working. Perfect. Thank you. Thanks for everyone for being patient. And so, yeah, I just want to sort of jump in right when Nick was talking about the. One of the interesting parts about the risk assessment was running through all this framework defining to us was very important to define the purpose and intended use of the package. It, it were coming from a case where we want to be deliberate about what is being included in the validated environment and that this is something that we're going to that is going to be used. And that we're using the right the right package for the right job. So understanding the use case is definitely something that we focus on a lot spent a lot of time talking to the statisticians figuring out which package you need. Why do you need it? What are you going to use this for? And then once we had, you know, setting that up with the very first step of what is we're going to include when we next we're trying to determine the risk as, you know, this is the difficult. I think a difficult part of the risk assessment is just where does everything fall in this level. We definitely utilize risk metric as a tool. And specifically to gather all the needed metrics sort of how often this is what are the number of downloads. What does it include documentation? What does that look like? And one of the things that that risk is a great tool to assist, but it doesn't hit on the quality of the documentation. So that part really still needs sort of human eyes and a human assessment. And as we are, you know, then we determine the risk and make make an assessment and that sort of we create the requirements of the testing required based on the level of risk. But I wanted to follow up and talking about responding to the risk. We sort of attack this in two different ways. What risk can we be, can be mitigated by testing of this package by testing the functions to make sure that they're responding to what we, to when doing what we expect them to do. And then additionally, what risk can be mitigated by the RQC QA process itself. So relying on that as well to know that, okay, this package, this function does what it says it does. But if you still, if you given it an incorrect input, and it's going to give you an incorrect output, but that's, you still need to rely on your QC QA process. So our testing really revolves around making sure that it operated on how it could. And we also mentioned and documented what the mitigation efforts would be for these types of packages. And I'll tell you a little bit about trust resources. We really did agree that that whitelisted packages can be added with minimal testing and including the documentation that that comes with those packages and comes with the sort of the base our documentation and as well as some of the documentation from our studio about the tidy verse and the Arlib suite of packages. So, yeah. There. All right. And then I know we're got just kind of over a bit so I'll try to wrap up the challenges quickly. Thank you Jessica and so the challenges are the agility and scope GSK. And I believe Alice might have presented the slide that had the two arrows with speed and automation on one side. Slowness and experts on the other. This is this is a huge challenge for us. I'd say we lean. Not completely on the expert side. We have some automations in place, but that review. We're really putting on the expert. So the process is slower than. Most would probably hope, but it gives that extra level of confidence as this is something new. So we hope to slide down that. That arrow closer to to automation as we learn and progress here. Another huge challenge education enforcement and change management so that intended use of a package versus dependency. Having everyone with the skill sets and understanding what this means why they can use something and not another thing. I'm answering the question if I need this package, why does it take so long to get it. So these are are mostly our biggest challenges really are just change management people understanding why it takes as much effort as it does actually doing the testing. So probably everyone has this issue or maybe not data and finding the right published results to use for your testing. And one interesting thing we found which is kind of specific to our implementation with with valve tools and PDF rendering in our containers is the rendering took longer than anything else. So if it's a route you're looking to go and you might have some policies or something around container builds, maybe early in the process talking to QA to find out if HTML is acceptable during the build process. And then you convert it to PDF after just as far as having that actual document that you might actually then put signatures and things on. So the HTML sped up the process considerably and also made it a little bit easier for us to then compare expected reports versus like what's already been a validated report across our Dev and QA prod environments to make sure. In fact, we can confirm everything is operating the same. Okay, so that's it. I will pass it back to you and D. Thank you very much. So at this point we're going to go to kind of an open forum for for questions, possibly the easiest thing is to kind of type the questions into the chat maybe. Well we'll see try try calling them out if everybody wants to kind of ask a question, we can we can go to the chat and try and manage it that way. But I will kick things off and use my privileges host to ask one of the questions which has been already captured in the in GitHub so there will be a reminder was a reminder I think last time I'll send a reminder again about the questions are Julian has already done it you've read my mind Thank you. So there's some questions from last time that are in the issues within GitHub where some of these case studies are now being written up. And one of the questions I like related to something you said then around experts and you referred to Ellis's diagram with the, you know, the automation versus using experts time. How does everyone determine what qualifies an expert, like who's who's doing that in each of your companies and how do you determine whether someone is sufficiently an expert or not and do you record that do you write down. What are the skills someone must have from an audit perspective. Well that's open to all the question I think it was with Ellis and the team of GSK, but definitely applicable to you, Nick and probably most, if not all of the presenters. Yeah, I'm happy to take a stab at it. I'll follow back up on the issues with with some colleagues feedback as well because really showing Lee is probably better suited to answer this question. Within the SDS or statistical organization, he did a survey across all of our therapeutic areas of who knew what what are what are the common methods that they're using within each TA and then use that to then identify within each group. Who has that expertise who's been using those packages for years from from a stats perspective. Validating things that we've been delivering in SAS forever because we never have to do anything with the QC results so they've just that that's just something they've been doing for decades already using those packages. So we felt pretty confident with with who we have there and that vetting process. But we are going to run into packages where we don't know, like, you know, someone's going to make a request and no one in the group, we have like a core team that we've decided are making those decisions. We might see the package request and we're here vetting that package and throw our hands up in the end say we're not qualified to do this and I'd probably start back with the requester themselves and see what they know about the package, why they're requesting it. What's their experience. But that's still, there's still some gray areas for us there. Yeah, what you touched on the end is similar to what we're doing at GS case I'll let back come in on that but pre thermo so you know, no bit. Oh yeah so Andy I mean at Mark actually what we do have is, we have constituted a panel of one programmer and one statistician who's available at all times to actually review a package, and get feedback on the qualification again at Mark the the qualification is based on the deliverables that are being used by that package. So, if that is a publication related or an external agency reported deliverable then that actually goes into a lower and a moderate risk. When the package goes through that threshold, then the review panel is automatically activated and we actually go through the testing case scenarios and so if it's the big issue right now at least at Mark is defining, you know, what level of, you know, review should be done, especially for packages that are statistics oriented mostly inferential statistics oriented so that's where we're thinking of actually implementing an additional review in form of a statistician who's well versed with the different in our processes the analysis and reporting process that's more of the, you know, the focus that Mark is actually looking at right now so we do have a panel of both a statistician and a programming programmer that actually looks at the different packages in order to qualify them as well. Thank you. Did it back or did anyone else want to come in with anything different or added to that. And I can. I agree with everything. You know, some of the points mentioned about it's it's going to be a little bit of figuring out as we go I think as we encounter a new and more complicated use cases but I think this is particularly relevant for the statistical applications right. And so we envision a process where it is a partnership with the requester. Ideally, that is a person who has experience and knows what they will be using it for knows what the, you know, the functions are going to be most frequently used from it so that our assessment can be more targeted and we know you know which tests are going to be used and hopefully they have the requester has the subject matter expertise that they can better evaluate the test. Thank you. And that leads us on to another question I had but I'm not going to hog the floor does anyone else want to jump in with a question for any of the presenters. I'm wondering how people staff this in general, if it's like a community of expertise and people do this in addition to their day jobs, or if it's outsourced, or if people have dedicated fts just for maintaining our packages. I know it's kind of a combination but I'm curious if the presenters could kind of weigh in and talk a little bit about that. So I can start at least at work we started off using a dedicated fts by starting a working group and then now it's start by dedicated fts as well as, you know, other colleagues that were doing it at their own time outside of their responsibilities so it's a combination of both but we do have a dedicated group that is actually using different processes to do the package qualification as well as system qualifications. Yeah, and I'll hop in next so all of the above actually so yeah we've got a few minimal dedicated most of most of the headcount comes from, you know, I'll get you 20% of someone's time or maybe 25% of someone's time and they continue to do their portfolio work. And it's a mix of internal and vendor support just depending on the skillset needed for the task really finding the right fit and skillset it can be difficult so. And just speaking to GSK we have within Andy team that dedicated resources for this, you know, focusing on the packages. And, you know, we're thinking about how do you scale up and how can you decrease the burden I think those are all really important questions. And thinking about when you have people come in and request new packages how you can, you know, get them to as I said before a partnership so so figuring out how this process is scalable is something we're actively working on but at this point, you know it's starting with the smaller group putting our heads together and tackling the early use cases. I don't know Andy if you have anything to add on. No, I think the general answer I'm hearing from everyone is pretty similar in that there is either a either a small team or individuals with a small amount of time who are able to kind of focus on this for now. But scalability is kind of acknowledged as a problem where we're going to have to get people from the wider business I hope there was a question similar to this as well. And I hope that we're able to come together as a community before long to start sharing examples of tests we write of assessments that we write and so on. I appreciate. I think at the moment there's there's probably a bit of reluctance to share here are the heroes are assessments because you're making a quality assessment of a package but there may be other, there may be things that we can do in the future. And once it you know it's one of those things once it starts. It will be easier for other people to kind of come in and chip in and so hopefully as a community you can get to a point where we reduce the burden overall and sort of crowdsource that way. Other other questions coming in again I've got I've got loads and there's a few that are in GitHub pages but I want to make sure that those who have questions here have got a chance to ask and there's one from Steven I can see in the chat which I'll read out unless anyone has anything else that you want to chip in quickly. So the one from Steven in the chat is who collates the test cases compiled from literature. So I guess generally, we could broaden that slightly as to how, how are those test cases compiled from literature where are people finding them, and how is that how is that done. For those that are doing that I think not everybody was good grabbing test cases from literature. Well, I can jump in. This is a big challenge. I think the idea of you. You know, it's easy to say here here are the things we want to use with this package. Now go forth test it test that it does what it does. So finding the the correct data to use and see the published literature or it comes from you know a textbook or something like that is is more challenging. And then also the, you know, sometimes the person coming up with those test cases and going on that literature hunt is is not the person that has been writing those those tests in in our. I think that you need to have someone with an understanding of the in the statistical package case of the statistics being done to make sure that they're hitting all of all of what's important. And what we found is that it's a bit of a back and forth of, you know, ready talking with this, the sort of the stakeholders, the statisticians of okay, we have this requirement. Is this, you know, this is this test work, this is what we're going to test and then sort of we've had a bit of a back and forth on coming up with what that looks like right now. Hopefully that'll be more streamlined as things go forward and we all work together more on this. But it's definitely I think this is one of the bigger challenges is getting this information together and then actually make sure that that's sufficient and and that you could find the data that needed for that kind of testing. Any other thoughts on that question. I mean, it's in a tangent but some of the test cases that we do use for the testing and the package called question is that we pull directly from the package themselves, especially if they're, you know, the authors have included them and then try to run them ourselves in our environment just to make sure that you know the testing cases are the testing scores are actually correct and we're getting the correct scores as you know indicated by the author of the package. So that's kind of validation that we also do but this is something that we're starting to look at work because of you know quality issues that were raised by the QA teams that said that we would have to actually look at testing more closely, especially for packages that we deem as low risk. And so that's something that you're trying to get together and then see if you can do that. And I don't really know though but I just wanted to ask all the presenters if there's any standardized data that you're actually using to do all of the testing or is it something that is, you know, all over the place based on the package. So I think I think for very basic things everyone's on board with using the XBT files from the CDIS pilot is it seems to show up in all the presentations, but then it's much more difficult once you get down using the understand methods so for that I don't know if any standard data out there. So that's really the challenge. Yeah, that's something that we're trying to figure out as well. And kind of running into roadblocks there. And one thing to note there is also, if you use internal data, even if it's like gone through data transparency, anonymization and that to allow you to use it for other purposes, then you can't share anything about it. So it's like, you're doing the work, it's probably less effort, but then you're not going to be able to share it. And then you're kind of losing creating potentially this community that I think some people are chatting about potentially having to do where you can share your data and test cases and people can build up on that, rather than everyone reinventing that. Yeah, this is Jeremy I've got an internal package that I want to open source and by far the biggest hurdle is getting our de identified test data approved for opens for open source. So I either have to decouple everything from the test data, which is a pain, or go through like an additional bunch of review like everybody's okay open sourcing the code but the data. Yeah. If you if you figure that out I would love to hear how how you made that happen. But one thing I'll add actually is a little while ago when we first started exploring this internally we we wrote some tests based on publicly available data that supported published published books. And often like when people write these particularly statistical books they will separately share their data. There are some conditions around that data, but with a view of sharing that publicly our contact to the authors and one of the books we're using like a couple of years back even now maybe 18 months ago. And they were certainly willing, I guess as references for their book to say first to use the test and share the test and that so you know one thing might be useful for us to do as a community is pull together different literature resources that we're using and so on where there are where there is supporting data that we can use and make that list available maybe that's a page for the our validation help website that we can we can put up. So yeah, if you if you are speaking to book book authors paper authors and do you have lists of data that you're you've been allowed to share. Yeah, please share with me and Julian after this and we can think about how we best surface that information for everybody because that will certainly that will certainly help as a first step until hopefully we were able to start sharing some tests at some point in the future. Okay, I think that Joe was bringing up is if anyone is using simulated data that could be shared as well. Absolutely. Yeah, any data any data that we can share publicly for whatever purpose tests obviously is the ultimate goal but yeah all of that would be helpful. And on that note one of the things we wanted to ask today. And our validation have perspective was, are there, are there any issues that you've faced that that would not. What were the biggest challenges that would be best to solve through our validation hub collaborations. So the things that we haven't you know people we've mentioned risk metric and risk assessments and so on that we put out there in the white paper itself is obviously providing guidance what other things are there out there that you think a big issues that we still need to solve. I can go first I mean that's something that's already in the chat and be just me that's lost pretend it's also free on my side. There was something in the chat. I'm trying to flick back quickly to the chat to see if I can work out what it was pretend was referring to in the chat does anyone else want to jump in whilst we wait. Oh, pretend we can hear me. Yeah, I just stopped my video maybe there's some issues with the bandwidth so you know just a quick, you know, the question with respect to what Damien has been referring in the chat is a common repository for you know validated packages or at least qualified packages that we want to take a look at I think that would be a very good step forward as well because that's something a lot of organizations I think are also struggling with this defining that and you know if you can get some sort of a validation from organizations that you know some of these packages have been used for submissions and there's okay or you know we got go ahead from regulatory as well so that I think that that would be a good exploration point as well. I don't know how others feel about this. I agree I think it'd be fantastic that there's, of course challenges but what doesn't have those challenges. And as Andy kind of touched on if there's just a way to get it started. Then, I would think it would pick up steam. I do want to make one comment though. We, we don't validate our packages were validating our entire environment and those packages reside within that environment so that might act that that is also another challenge it's like what does it mean when we say that that's validated and we don't want them people to go change they said it's validated I'm going to use it for whatever went in for us it's really the process of executing all of those tests in the environment that return and expected document with everything passing. That's really what we're saying. You can now use the player, but not before that. Yeah, I've, we've talked a lot about this as a, as a steering kind of committee for the validation hub as the exact and it's difficult because no one really wants to say yeah we approve this and put that stamp approval on the public domain. But you can do other things like for example, if you write tests for a particular package or like, like I was mentioning with the literature, if you're able to say, here is an example showing how this package works with this literature and here's the test and you put that out and everyone can use that and then that can become the resource and like in the risk assessment you can say well, I can see that there are tests here with public data already I don't need to rewrite my own test, which also said, you know, simplifies the internal execution time because you sort of front load it in your upfront risk assessment rather than have it in as part of your own process so with that we can get to, you know, you can almost imagine like a risk metric like has tests that are available on some our validation hub resource but it's we get we've got deep into this conversation before it's very difficult to get I think, again, to the point I made earlier a new reference Nick to get that ball rolling. And once we've got something there we can look and tweak it and see what we can do is more. And if there are volunteers, by the way, on this call, the one thing that they are validation hub survives on is people's activity people spare time. There are people who are interested in these topics and we'd be looking to potentially lead any of these initiatives, and we'd certainly welcome that that input from anybody on this call. Okay, we've got about four minutes left there was one more playful question, which was asked on GitHub, which I think is interesting to ask, which is has anybody rejected a package. I mean rejected means different things to different companies, but so some companies will say, well something that is deemed high risk and then you write test but you could still I guess theoretically get to the point where you write some tests and they fail. So does that has anyone come across anything whether they're not accepting at the end of the day. There was one instance where we had one package that we had qualified and put in moderate risk category at least at mark where we had said that this could be used for moderate risk use cases but the issue was that that package was not back out then older version of our that was being used and so we had to actually reject that package from the moderate use case. We had progress to a more, you know, latest later versions of our so that's one use one scenario where we had to reject the package for that purpose. But happy to say in the latest round of qualification it is now back into the moderate use case so. I can double down on this one. And also, what Nicholas said, the versioning is very important, the whole environment that you're running the package on, because there were some cases when for example the package was being upgraded. It was using some other underlying the libraries, and there was some assumption made that could make the package actually crash on the same functionality that was working before. Or actually there was a functionality changed like the name of the function that hasn't changed the description hasn't changed but the behavior for some edge cases was changed. And this could make a difference and make the package to be actually not approved. So we haven't rejected anything but we're so early in the process. It's mostly been our core group of experts deciding what packages were going to be allowed for use. So we're not there yet, but I fully expect as the ball gets rolling and more people begin adopting the use of SAS and R at the same time, that request will come up that we'll need to but nothing at this point. Okay. So I will close the meeting out there to at least give you some time to get to your next meetings if you have them. Thank you everybody this meeting has been recorded so we'll release that along with the recording from last time. Thank you again to Damien and Nick and co for presenting today, and thanks again for the presenters from last time for coming back and sharing some of your, some of your thoughts, we have a next, a part three to this on June 14. I don't think the invites gone out yet, but I will send that out afterwards. So if you want to come back and continue this discussion drill into some of those areas, ask new questions and new presenters then June 14 will follow up with another one. All right, thank you very much everybody and hopefully I'll see you in a month. Thank you. Thank you.