 Can you let me know when it's okay, June? Yes, it has already started. Go ahead. Perfect. All right, so welcome again. My name is Karim. I'm part of the executive committee of the R-Validation Hub. And in particular, I lead the regulatory R repository work stream. Today we'll be wrapping up 2023 with updates of what happened during this year in the R-Validation Hub. Then we'll do a deep dive. I can show you the agenda. Then we'll do a deep dive and focus on the regulatory R repository work stream. And then we'll open up for discussion. And there we go. So while we were waiting for people to join, I asked you to share in the chat who you are, where you're from, who are you joining us today, and what your cat mood is. I cannot see your comments. So it's just for you guys to read between yourselves and for everybody to know who's here. Does anybody want to share your cat mood today before we kick in? This is a quick icebreaker. If not, it's fine. We can continue, but for those who are excited for seconds. Okay. I think a bunch of people are like in between five and six cat mood wise. Perfect. Thanks Julian. Yeah, I'm rather five. It's six p.m. here in Switzerland. This is my last meeting of the day and then I'll go and have dinner. So finishing up the day. Hope everybody's okay. So looking back on 2023, I tried to summarize a couple of events that happened in chronologically. We welcomed Doug as our new chair. Took you, taking over from Andy. So congrats again, Doug. And immediately Doug took a couple of actions. For example, creating official communication work stream. I know there was a communication activities already before, but this was some work done around consolidating the communication activity in a dedicated communications work stream led by Julian with, for example, Aaron, there's Jackson, Anuja, and some more people. So that is something that Doug kicked off when he started. There is also a new executive committee composed of the following people here. So Doug is chairing us from the art consortium since the art validation hub is supported by the art consortium. Joe on the call, we heard from you. We heard you Joe before, Prietham. I don't know if I can see, I cannot see who was on the call, but Prietham also is part of the committee. We have Julian leading the communication work stream myself, leading the repository work stream. Eric and the risk metric and Aaron with the risk assessments work stream. So some changes in the executive committee since this year. A couple of events we participated in because it's good to remind ourselves also our presence because part of what we do is about, is to make an impact. What we do is to make an impact for our community. So part of our job is also to reach out and communicate and speak about what we do. So I here have the use R, it was renamed R in R Basel this year that was on the Rosh campus. That was in July and there I was there to represent the regulatory R repository work stream. I know we also had members of the risk metric team representing the R validation hub. So that was one of the events. We had the Posit R and Pharma Roundtable in Chicago organized by Posit just before the R and Pharma conference where Doug was there to lead a workshop. So it was not a conference this time, it was a workshop. So I remember Doug organizing workshop is different than simply giving a talk, there's more logistics to it involving sticky notes, which is honestly sometimes quite a nightmare, but it was a very interesting, so well led workshop and with interesting output from it. So that was another one. And finally we were present at the R and Pharma conference in October where Julianne and I represented the R validation hub and the different work streams. So I think we can say we've been present at the main events around R this year again, I guess. So this is I think a sign, a healthy sign of us having content to share with our community. Otherwise we would not be invited because we're not invited just for saying we're not invited by defaults to those events and conferences, right? So we propose topics and usually there's a committee of each conference selecting who goes and who doesn't. So based on the interest of the community. So that was something else in 2023. And now I'll do a little work stream updates very, not very briefly, but I will do a quick, still quick because then we'll have a really deep dive into one of the work streams that I would like to give you an overview of all the work streams. So as a work stream, I put here still the R consortium. Even though it's not a work stream per se, but R consortium has a strong presence in the R validation hub. So the first news that happened this year is that in August, the R consortium announced that Joe had been appointed to the position of executive director reporting to the board of directors. That was a new position that was created in the R consortium to ensure that there was going to be long-term and consistent oversight of day-to-day activity. And Joe had demonstrated this in the past years. So again, Joe, congratulations for this and it's worth reiterating this news that happened this year. Also the R validation hub worked in bringing its communication activities under the umbrella of the R consortium. So for this, we thank, of course, our communication work streams led by Julian, but also on the R consortium side, Jesse, Aelina, Agedda, Maya, and others from the R consortium marketing committee who creates great material for our comms. So that's another update of this year. And worth mentioning also that the R validation hub and the R consortium started, still early days, it started a collaboration between two of its work streams. So the regulatory R repository work stream in the R validation hub and the R consortium repository working group that are, even though they have repository in their title, both of them, they do focus on different types of repositories. But for the moment, basically our R validation of work stream is attending the R consortium repository's working group to make sure that there is a link between those two topics. So that's an example of a collaboration that started this year. An overview of the different work streams. So on the communication side, as we said, so there was an effort to consolidate this work stream and to put several people together to lead this. Examples of the topics that were tackled this year are how to structure the R adoption series seminar, how to structure the advertisement efforts, how to make the Canvas project more visible on our websites and revisiting our websites as an ongoing theme as there are always improvements to do. So we thank the communication work streams, work stream members. I skipped the repositories work stream here because I will deep dive into it in a couple of slides. On the risk metric side that by Eric, so for those, the few I hope who don't know risk metric, it's a framework to quantify in our package risk of use, right? And developments in the risk metrics include, for example, creating prettier outputs, ease of use, have more complete metrics, for example, for as many package sources as possible. Modular additions, for example, to allow custom assessments based on community package, for example, as Oyster, SRR package test, for example, and also focusing on the metrics and scoring. So making custom waiting more robust and convenient. So thanks Eric for this work this year and the team on the risk assessment side, led by Aaron. So risk assessment and risk metric work together as risk assessment is actually an R package that contains a shiny front end to improve the use of risk metric. And actually it was mentioned in the RN Pharma conference, but for those who didn't attend this app, risk assessment received an award, the title of best app at the SHINee conference 2023. So that's an achievement worth celebrating. For example, things that have been going on in this work stream are improved report builder, database view, better dependency inspection, automatic risk assessment, and many more. So congrats to Aaron for this work this year. And finally, a new one. It's one of the kicked off recently, risk score led by Aaron as well. So it's a data package for cataloging risk metric results across public repositories. Little disclaimer right now, the risk score is in the POC stage. So it's not fully operational, but since it was kicked off this year, I wanted to give also an update on this one. Before I have this little gif here to thank all of the work streams, I'm aware that I gave a very high level updates of all the work streams. So this is why before I close this section and go to the deep dive of the regulatory, our repository work stream, I would like to open the floor to my colleagues who are on the call and the executive committee or those of you who lead those work streams if you'd like to add anything that I didn't mention that you'd like to share with the community. You can take a couple of minutes to do this now if you would like. Does anybody want to add anything? Now's the moment. Sure, this is Aaron. I help lead the risk assessment and risk score projects. So I'll just say a few words. Risk score. To say that I lead this is, I use the word lead gently, but I did push some initial commits to sort of get the repo started. But yeah, I think we're currently soliciting interested parties who want to contribute on this project. But yeah, essentially like Colleen said, just a way of capturing risk metric, scores and outputs for all packages on Cran. So that's really exciting work. But yeah, just it's really in a tentative experimental stage right now. So if anyone wants to contribute, I'll share a link in the chat to the repo if you want to check it out. And yeah, for the risk assessment application, I would just say a couple of the big things is our friends over at GSK contributed some code recently that I think has really launched the application forward a ton in terms of the way we can manually review package risks. So if you haven't seen those, I think we pushed a release, I think it was just a month ago. It wasn't very long. It was around the time our environment was happening. And so you can now actually explore files for any package. So we actually are downloading like the source file so that you can explore like the description file or the test files, whatever you want to view. It's really helpful. And then we also have a function explorer that GSK also contributed. That's really awesome. For any exported function, you can view the source code, you can view the rendered help docs and you can even view the test for those functions. So it's just a really standardized way to sort of evaluate your package or the package that you're reviewing in just a consistent way for the whole organization. So that's my two minute plug for some of the recent features. So go check them out. I'll share some links in the chat so that you can follow up with if you have any interest. Thanks. Aaron. Anybody else? Julian, Doug, Eric, can I continue? Maybe I'll just add a small note about the communications work stream. So sorry I don't have video today. I'm still kind of recovering from about with COVID. So I'm keeping camera off while I'm isolating and look like a wreck. But one thing about the communications work stream is that they're still kind of finding their footing and starting to think about revamping some of our traditional communication streams like the website and how we post content. But they're also thinking about how we want the community to look in the future. And part of that is building kind of more like grassroots communities at organizations and being more connected with those. So how do we give materials to like pharmaceutical partners so that they can build smaller like local communities within their own organizations and how do we kind of like feed that information back to a broader community so that there's more kind of knowledge sharing happening across companies. So I think that's a really cool idea. Another way of looking at how we like broaden our communities and build more inroads across our organizations. So just be on the lookout for that kind of material if that's something that you're looking to embed more within your organizations. Thank you, Doug. Wish you a good recovery. Yeah. Yeah, also, if you want the other validation hub to have like within like if you have an internal user group and you want to have a conversation internally in your company, like just feel free to reach out to us. We are definitely looking for opportunities like this in particular next year. So and if you want to help out, there's a ton of work to do. So that's also always an opportunity to help out with this initiative. So Eric, any word if not we'll continue. All right. Thank you guys for complimenting this summary of the 2023. So we'll now do a deep dive of the particular work stream called the regulatory R-REPEL simply because we, it's a new work stream this year and we didn't have the opportunity to share further information in the community meeting before. So we decided to dedicate a bit of time in this one. And then I'll open up for discussion. So the regulatory R-REPEL, hopefully most of you have heard about it. So the idea is that we build a repository of R packages that are considered validated. So it's a centralized place. As it says here, open dynamic cross industry repository of R packages are validated, validated R packages. So to distinguish ourselves from CRAN, it's not another CRAN that we're trying to build. This is what the R Consortium's working group called Repositories is looking at partially. But here it's really another need that we're trying to meet. It's right now the fact that every pharmaceutical company is setting up their own validation process and team to meet the needs of our industry, which is regulated. And when we submit results and outputs of our internal analysis based on our clinical trial data to a health authority, we need to show that this code was validated. So this is a new challenge, new since a couple of years now, but new challenge since the pharmaceutical industry started to use open source programming languages like R for its clinical trial data, clinical trial data analysis, compared to SAS in the past, which was a licensed software. So we didn't have this question of validation. At least it was not a concern for sponsored companies, pharmaceutical companies. It was for the company SAS. Anyway, so this is not a new topic. But since we are hearing more and more about the need for validation, each company is developing its own solution. And we thought that it would be a good solution to have a public centralized place where to avoid duplicate work, basically, in each individual company. So we have taken time to reach out to different stakeholders in the industry. I write here, validation leads, health authorities also, developers, of course, people who maintain also infrastructure and companies, and so on. And we basically concluded that this would be a solution that is of interest to all the stakeholders. So this is why we created this work stream at the beginning of the year. Basically, how we got here is that our validation have had its white paper widely adopted. But implementing it was a bit difficult, as each company has its slight variation. Sharing software, in this case, with health authorities, is sometimes a challenge. They are overwhelmed by technical inconsistencies. If each sponsor company shares results with a different version of R, a different environment, which was developed then, the health authorities, when they need to replicate our work, our results, they then face some challenges. And as is written here in this last bullet point, so we felt that the most productive path forward was a shared ecosystem, a shared and central place, hence the spreadsheet. So this year, we've been working on a couple of things. So it is, of course, important to build some type of consensus around what is a package, evaluated package, what does it mean to evaluate a package, how do we distribute those packages. So we've split the work in different parts. So we started with stakeholder engagement. So we interviewed a lot of people, stakeholders, as I mentioned before, roles like validation leads, health authorities, developers, system admins, and so on, to make sure we were speaking about the same thing and actually grasping, sensing a need that was actually the case. Then we split the work into several proofs of concepts. And then we started developing the proofs of concepts. So it was all of this is voluntary work. I have at the end a list of all the members. You will see how well we are represented across the industry. But it's really incredible we were able to achieve with voluntary work. And just it was great to see from the start the confirmation of our initial assessment that this was needed. So what a solution looks like is, first of all, to have some standard and public assessment of packages. This is probably the most difficult piece to agree on. But because each company, as I said, has a different way of validating, we've seen it with the case studies at the R-Validation Hub together last year, asking Roche, how do you guys do validation? And the artists and so on. So each have a different and slightly different approach. And agreeing on the one that fits all, at least to start with, is a difficulty, but is needed. Thinking about how we communicate, how we contribute to the repository, how we then consume from the repository, how bugs can be solved, how this would be maintained, and so on are pieces of work that are key to solving that we've been tackling or come to this unit. So the proposal so far is not concrete, but we know that we need a central place where there is some type of assessment of packages. That's the source of those packages are probably going to come from a source like CRAN or our universe where we've been exploring also other avenues, that there will be a validation pipeline, that there will be also user interface, and some type of forum where people can discuss the assessment of packages and so on. Another way of representing the future solution is like this. It's very less elegant than this one, but it still speaks, I think, where we have in the middle the repository setup, what we call, I wrote here in the yellow sticky notes, as one of our pilots, the possible repo setups. So what are the pieces of the puzzle that we need to put together so that this so-called repository works, and it includes a repository, a reference image against which we validate a UI, a user interface, and some type of database of validation metadata that we did have here, with contributors on the left contributing to the repository, consumers on the other end, obviously, consuming the validated packages, maintainers, or we hope this would be as automated as possible, so it could be a robot's maintainer, or a robot at least, an automated way of maintaining the validated state of those packages, and something that would look like a governance board, making sure that the repository still meets the need of its users. That's all the different stakeholders and people engaging with the repository can do so, and that we still meet the needs that it's meant to do. A couple of items that we have thought about is, so I mentioned this reference image. This is something that we ended up concluding that we needed, otherwise we could have probably several reference images, but at least we need to define, at the beginning, a golden image or a reference image that we validated against, because a package is only considered validating in a specific environment. We cannot say package X and version XYZ is validated. It has to be validated in a specific environment. So this would be defined here with the reference image. Obviously, the part of the defining requirements and testing and matching those automatically is part of the validation. So this is also another point of discussion. Expectations of public communication, so how do we, communication channels for the different stakeholders of this repository, what do we use, industry standards, communication channels is also another topic of discussion, and we, so here edge cases are better handled by transferring community engagement, as this would be a central place with validated code. We want to have, as I mentioned before, a place where the community can contribute and discuss, this cannot be a black box, so it needs to be transparent, otherwise it will not work. And I'll finish with this one. So there are two last updates I wanted to share. So first of all, the one, the second one, I'll start with the second bullet point. We have put together a benefits of joining us PDF. It's a one pager where we summarize the reasons why we think it's important to join, and this is a document that we ended up creating because in our members meeting all hands, you were asked, well, I have my day-to-day job, so if I were to ask my manager to join this working group, I need to explain why it's important, and so on. So we decided to put this one pager together so that anybody can pick it up, share with your colleagues in your company, or share it with other people you think are talented and should join the working group. Hopefully that's useful. We'll share the slides after the call, so you'll have this afterwards. And the last piece of update I wanted to share is that we have last month submitted a grant proposal to get a bit of funding to my idea as lead, and I think that the rest of the team also was interested to see if we could get some small amount of money to hire a couple of people full-time dedicated to building the repository. Since, as some of you know, when you have work done purely on volunteer work, it can be sometimes not necessarily very constant. People give the time they can, so when you meet on a weekly basis, you don't necessarily have everybody there all the time, so sometimes it's a challenge. It's also, I think, important to say the truth, to bring people together and to focus on one vision. But this time we were, so the funding was not granted, but it was a good opportunity to get some feedback on what was expected so that we are successful in the next round. And also something I will always remember, the discussion we had a couple of weeks ago with the executive committee. And Joe told me, well, there's a lot of you can do with volunteer work, and that's an opportunity to make it grow, make the momentum, surf on the momentum, create the momentum, and see what other working groups are doing based on also only volunteer work, and a lot can be done. So this kind of reassured me, because at the same time, the infrastructure steering committee was actually in support of our work stream, but needs a little bit more concrete results. So this will be working on for the next round of grant proposal in March. So yeah, this is what I have for the, I just want to thank all the repository members. And as you can see here, I'm really proud of the variety of companies and the various stakeholders we have on this work stream. We have sponsor companies, as you can see here, a variety of them. We have CROs also represented. We have health authorities with the FDA, and we're working on getting somebody from the European medical authority. And we have also service providers who are key to collaborate with since they own, they own sometimes part of the infrastructure we that are part of the R ecosystem that we would like to base ourselves on. And some of them also have some expertise that we want to leverage and that we count on to make this working group successful. So really another little fun gif here for Christmas Eve, Christmas Eve gif. But really to say to all of you, I hope some of you, I think, I know most of you were going to attend this call today. So thank you for your time. We couldn't be where we are today without all of you. So it's an opportunity for me to thank you in front of this audience. Now it's the time to open for discussion. We have about 20 minutes. Thanks for listening. And yeah, I'll open up for questions, discussions. And Julian, if there are some questions in the chat, then I cannot see. Not at this point, but it doesn't mean that people cannot come up with questions right now. Great presentation, Julian. Thank you. Quick question. Are you going to provide these slides to us so that we can maybe share internally with our, in our institutions, that's one question. Yes, we will. Julian, you know better than me, how and where we share those slides, but of course we will. Yeah, I can totally, like if you send them all, send them out to everyone who has been included and it's called, we can send it out to the list. Can you go to your, I think it's the MVP, like the, it's the slide 15, I think? That's what you have, some proposals so far. Question, documentation is critical for any implementation in anywhere, right? So if we were to use this documentation of everything that is being done, yes, it's part of it, but it's, I just don't see it pointed out there as part of the MVP or the proposal. So I, in my opinion, it should be part of it, right? Because that's how our IT colleagues and everybody else is going to see and consume. Granted, this can be done through a, through a communication site, you know, external for the, but even if it's that, there has to be a part of it that is, that is documentation. If it's not, I just don't see it there. So that's what I'm saying. It's a good point. Yes, we, it's not represented here, but it's obviously part of the outputs of this. I take notes to add it in the, in this diagram. I'll just maybe touch on that a little bit. So I know that like the documentation has been, I know it's part of our early discussion. And I think part of the challenge is that the way that people document and the context that they want to add specifically to that documentation is very company specific sometimes. But what we can definitely provide and what we've focused on so far is like the data that would make its way into the documentation. So how do we, how can we support it? So you have to do as little footwork as possible to collect all the, all the underlying qualities of a package that you would then want to document. And from there, maybe there's this intermediate where things still end up being a little redundant across companies, but that gives us the, the foothold to then start establishing this like common template of what that documentation could look like. I know that we've had discussions about having a, template like that from the start. So it's just a matter of like, where do we put that effort as we're driving things forward? And I think right now the primary questions are around like the technical components. There's not really that many open questions about how do we pull it into a report when it's all there. So if we're thinking about what things we need to, what questions we need to answer to know that this is viable, most of those end up being, I think more the technicalities of hosting the software and pulling those in a more standard way, the metadata that we'd want to pull into a report. So yeah, unfortunately you can't foil the whole ocean all at once, but that's definitely in the backs of our minds is something that eventually we need to cross that bridge. Thank you, Dr. Thur. Any other comments? I mean, this isn't a perfect common opportunity to bring up vicious opportunities, ideas without having to, having to join another working group, right? So use that opportunity. Maybe I will just toss up a question while we're on this slide. So I see the governance board called out here and I'm wondering what kinds of decisions you see them playing a prominent role trying to address. I think in any project like the one we're trying to build like I imagine there's one for CRAN and one for our universe and one for GitHub, GitHub, whatever tool and system you put together by experience, I think it's important to have a group of people who are not with their head in the wheel, that makes sense, I don't know if that's a French expression, but some people who have a more helicopter view of what's happening with the system in question or in this case the repository to make sure that things are happening as planned, that the purpose in this case, the repository is still up to date, that if there's a new need arising that this governance board brings it back to the people maintaining the repository or I think it's important to have always a group of people who are independent from the day-to-day activities and who are able to give feedback and improve the tool during its life span. So that would be my mission for us. Let me add to that. So it's particularly important to jump off based on what Kamin just said. The repository, it also has to have governance that's transparent and open. So any endeavor like this has to be in compliance with the antitrust laws. We have to make sure that there are, let's say democratic and open mechanisms for the governance of which packages are included in the repository, how people interact. So this is important concern for the art consortium and I'm very pleased to see that people are thinking in this direction. Yeah, thanks, Carol. I think it's not a, it would not be an alien to have this, it's actually quite a good practice. Okay. So how often do you lead with the, just if anyone is in the background who wants to join, how often do you mean and do you have like a smaller circle, bigger circle, like how does the organization work? Yeah, good question, Julian. So we have a core team and an extended team. So the core team members are the ones with the little asterisks here on the screen. So it's a smaller group compared to the extended team, obviously, but still a big team. I think we're about 12 to 15 people meeting weekly every Thursday afternoon, Swiss time, morning, Eastern time. And the, on a weekly basis for 45 minutes, at the moment, what we do is we give updates on the yellow sticky notes here, which we called our themes. So there's a group focusing on the validation pipeline, another one on the possible repository setups, and another one on the reference image. So, and then we'll put those three pieces of the puzzle together. So that was, that's what we're doing on a weekly basis. There's a GitHub issue at RiftDoug, you could share in the chat while I speak and you can dig it up. There's an issue where you can sign up in one of the themes. So if any of you wants to join. So this is for the core team. And then we have an extended team that meets approximately once a quarter. So the extended team consists of people who cannot put one hour per week in this work group who are interested to follow its development. So this is similar, we do similar sessions to this one where we update our community on what's happening. And I'll add that the only distinction between the core team and the extended team is like time commitment that people are willing to offer. So we have a bunch of people, our core team that attends regularly and are consistently committed to the working group. But understandably, we have a bunch of volunteers that don't have that kind of bandwidth to commit all the time. So they prefer to kind of sit kind of on call, you could say. So we kind of tap into those resources when we have a specific task that we're aiming to deliver. And we'll see then who's willing, who has kind of bandwidth at that particular time to jump onto it. So to paint on your availability, if this is something that you're interested in participating in, you're welcome to participate in kind of either capacity. Yeah. Thank you, Doug. I think in the meeting invitation, you will find my email address. So if she would like, if it's too complicated to find the GitHub page of this work stream and to find where you can sign up, because there is a place where you can sign up, but so free to just email me and I will give you, I'll make sure you have an onboarding session and then you can let us know which team you wanna participate in or if you just wanna be part of the extended team, which is also fine because the extended team for me is almost as important as the core team because it shows the support we have from different companies. So I trust that everybody gives the time they can and if you're extended member, it's also fine. I have posted the email address also in the chat. So there are no other questions. This is the last call. Okay, seven seconds of silence is about the time that people get. So I appreciate you all signing in and taking this time out of your day and evening and thank you, Colleen, for presenting. And I think it's time to say happy end of the year and let's see where we pick up next year. We'll definitely look out for those emails that are coming from our Validation Hub email list and if you're not on it yet, I highly recommend you to get onto it. We will definitely make more use of it as the communication works from all the pieces together. Having said that, happy end of the year. Enjoy your end of November, December and see you sometime in January, February. Yes, thank you all. Same to you all. Take care, bye bye. Thanks everyone. Thank you. Bye bye.