 Okay, well, let's start. So let me welcome everyone welcome to this which is the first meeting of what we're referring to right now as the our submissions working group with the art consortium. And I'd like to start out by just a couple of concept comments on the various our consortium working group so as many of you know, we have had a working group for quite some time devoted to our validation the our validation hub. And we also have a working group with focused on making tables in our calling the RT RS working group. And now we have this working group, which the intention is to focus on infrastructure issues, it issues and the like that have to do with moving towards all our submissions. So some of you may wonder why so many working groups. Well loosely, what I have in my head is the model that the IEEE used in the early days of computer networking. So for instance, the Ethernet project was divided by where you were on the protocol stack, you know, the physical layer. The MAC layer, the media access control layer, the networking layer, etc, all the way up to the presentation layer. So loosely speaking, I think that's what we have here is a project where there are overlapping and not independent projects. But I'm hoping it makes it easier for people to not become overwhelmed and to get focused within the working groups by having a kind of like a layer on the stack where we're focused. So, so this is the infrastructure one. According to the, you know, analogy, we'd be down there at the bottom physical layer, the media access control, right. And for other things like the tables would be up at the presentation layer, how to, what happens at the certain levels of interface. So if this works, it'll be great. If it doesn't work, we'll try something else. Now, what I'd like to do, Paul, would you like to make a few comments on, you know, what the, what the environments like. Do you want me to show the slides that I got prepared? Sure. Okay, so let's see if I haven't done this with zoom. So this is the first for me. Okay. And can everyone confirm they can see the slides. Yes, we can. That's good. Okay, let me go to the slide show format. So I thought I'd give sort of a brief overview of some things. You may be familiar with this already. And if so, I apologize for going over other ground. So one part, if you're folks are doing a submission to FDA is there's what's called the electronic common technical document. And this sets out the overall framework that a submission would need to take. And it's fairly complicated. For those who have to look at it the first time, but there's, let me point out where some things are available. We are through the ECTD technical conformance guide, which is getting into some of the nitty gritty and then there's actually an FDA guidance on the topic. So this part in and of itself is actually handled by the Office of Business Informatics under the Office of Strategic Programs. And I think that there's something on the order of over 100 staff who are working with this type of aspect. You may be familiar with this, but let me go ahead and just mention that there's an overall aggregate file structure, the M1 through M5 format. So folders and naming conventions are supposed to be set up in a standard way. Actually all lowercase letters, which this doesn't quite follow, but administrative information prescribing is M1 summaries M2 quality is M3 non clinical M4. And what I usually work with is M5. So here's the study data resources. So the ECTD sets out sort of overall framework, then the study data actually sets out how actual clinical trial data and other sorts of studies should be submitted. There is a entire website for study resources. And we have the study data technical conformance guide. And this is the part that is worth looking at on a regular basis. It's usually updated twice a year. So their ongoing requirements or changes tend to be reflected in the technical conformance guide. And let's just see if we can click on that. Let's see. I'm sharing the application so you're probably not seeing this. So let me just skip that. That was probably, I don't think you guys can see that, can you. No, no, okay. Let me not confuse the issue then. Sorry about that. So we can make these. All public links in this particular one so I can share the slides with you later. CDISC is probably something that everyone is familiar with. It stands for the clinical data interchange standards consortium. And it comes in a variety of different forms. We have the study data tabulation model or SDTM. The idea of an analysis data model or Adam, and then there's the standard for exchange of non clinical data or send. So, typically the last bullet will be things for like animal data. Most of the statisticians and data scientists will be looking at SDTM and Adam. And most statisticians mostly deal with Adam directly in theory. The idea was that SDTM would be close to the raw data. Adam would be derived directly from SDTM. And at one stage, the idea was that it was one procedure or our functional way from the actual results. Although I don't think that particular model has worked. So this is the basic setup. See if I included any of the other technical aspects. SDTM has some different types of variables listed. There are the required things like the study ID domain use subject ID. Those that are expected, they should be present, but they could be missing. So for example, a death flag should be present. But if no one dies in the study, then there would be no point in including a death flag. And then there are things that go by the term permissible. We want the data if it's collected, but it may not be. So some of this would be things like ethnicity investigator name and so forth. Let's see timing variables. This may be something you're familiar with, but SDTM timing uses ISO 8601. And essentially, there's actually it's discussed in on Wikipedia. The, what it does is it's sort of a reverse. It starts with the highest level year, month, date, hour, minute, second, and then even I think it's hundreds of a second or it can go to thousands. Usually expressed in local time, but it can be in Greenwich meantime. The part about this that can be somewhat challenging is that essentially this is an interval. It's a point estimate. If you only know the day on which something occurred, you'll lose the last part the hours, minutes and seconds. So it could be anytime during the day. And most of our software programs treat that as a point estimate and give it a start time say at the beginning of the day. What that can do is create a problem is that an adverse event can be registered as ending before it actually started, for example. So that's something that can occur. So without going into the details, I thought it would be appropriate just to point out there's all of these references and resources that specify how things are to be submitted. Most people are familiar with the fact that XPT data sets are used. What may not be quite as familiar, for example, that's embedded in this study data technical conformance guide is the idea that data sets are supposed to be limited currently to five gig per data set. So there may need to be splitting of data sets, say for labs and questionnaire data. So those are some of the trickier parts of this. So there are are literally hundreds of pages of documentation that I've very glibly just put up links to. On that note, let me do a stop sharing. And I guess the next thing is to turn it back to you Joe and ask if there are questions and comments so far and details that you'd like to see further. Thank you. So, questions. All right, so I guess I would have one question. It took me a while to unmute but thank you for the summary answer that part of the infrastructure is not what do you expect to receive from the health authority site. How flexible do you think this is for negotiation or for improvement as we, if we were to move towards another mission. Because I particularly would like to see that what we submit is runnable code so we give you essentially a collection of packages that have all the data that the analysis code the data derivation codes. And for that our packages are great format. That's certainly something that can be negotiated. The one thing that's embedded in all of that are some of the naming conventions. So, for example, one downside right now is that the systems are set up that file names are supposed to be including the dot suffixes are all supposed to be lower case. Okay. So, the standard convention for our code is dot are with a uppercase are, whereas this would require a lower case are. And I think that also would make it impossible to submit in our package that gives you the documentation because they have like namespace has to be uppercase and description. So that might be something that we would have to negotiate. Well, if I could comment Paul and thanks for a great summary and hi everybody. It's great to see all of you again. How maybe take this a step further. To me, I think the real nugget that would be this is really pine the sky kind of thought is if we can pass not just a code and not just say the packages that we use to do the analysis and the submission, the actual analysis execution environment itself, then you have everything baked in and it's literally just running in like a container or some kind of, you know, our service, choose your adventure and cloud computing or whatever. But then there's like no ambiguity on like what's being transferred it's like the exact same run times or packages are versions as what we in the in the companies are using. We know that we've been pushing the envelope over and some initiatives, especially with CID. I wonder if that's something that we can take into this at some point. I know it'll take a lot to get there though. So the, that's been a mood in some other venues such as fuse is can we can say a sponsor just say, say they have an instance up on a in the cloud, say AWS or Azure and could they just give us access to that environment. I don't think that will quite meet with regulatory requirements because what happens if well there needs to be the ability to go back and look at the data sets and the environments, you know, basically work with the actual data at a future point in time in some So if there's basically a say serious adverse event that occurs in the future for a drug, there may be some desire to go back and look at the totality of the data. So I think in that sense, having a an environment that's just say there and then is say closed when the sponsor obtains per approval, definitely will not meet long term FDA requirements. Yeah, that makes a lot of sense. I was thinking it'd be something where and we even talk about this in the project you're not cooperating on the past. If we give you the instructions for the point that you all do that at your infrastructures women that's obviously another kind of runs of having your own infrastructure to support it if we give you like the recipes for it. Yeah, and if it's done the right way which, by the way you did. Anyway, we try. Well, we got something up and running. It may not have been optimal. Eric and I worked on a CID project. Eric did most of it. I just did a little bit of facilitation. Don't sell yourself short it was very helpful. Well, yeah, we're, there were still issues. So the we ended up having to kind of make up our own rule book and go around some rules which may have worked okay for a CID submission where we're a little bit more free leeway. Right, but probably would not work for a standard regulatory submission. Yeah, I realize it's a different mindset and it's a different set of rules mean not to sound harsh but it's definitely a more higher set of rules I would think in terms of what's been established already. And trying to get a docker environment and everything set up and passed into FDA through the submission gateway has been a challenge to put it mildly. Basically the containers I can stand up, for example, aren't allowed access to the local hard drive. So there's various sorts of other rules. In principle, I think docker containers and something along those lines make a lot of sense. I think that would be the way to go. It's trying to get things to work. A lot of things that are set up for example, to take advantage of Linux, tend not to work as well with our available machines because our IT folks, with the exception of some servers are almost exclusively windows based. Interesting. Okay. So trying to get those environments and different operating systems synced up has proven to be. Let's just say challenging is perhaps an understatement. Well, a quick follow up on that Paul. So is every reviewer's desktop environment potentially different in terms of our Yes. That's something that maybe we're some consideration in the future. That's a challenge. Okay, so it turns out I wear many hats. You just don't see them all. One of the hats I wear is, and I probably need to get out of it is actually manage a cluster or a group of workstations. So those I can control, and I can control the environment and the packages and how that set up. And actually what we did with and working with Eric is when it was specified we knew this are with this and actually used I think you use pack rat to get all the packages. So in that case, since I could control that environment I could set it up. We have had discussions of setting up what we are calling a statisticians desktop. But that's more of a concept that we're, you know, still in progress. So, in that case one could standardize it. If we have sort of like a VM everyone logged into and it had a standard suite of tools. But that's going to be a while away, just because just negotiating the licenses to be able to do that with our standard software products is going to be problematic. So we can do that. What that does suggest though is use of appropriate package managers, whether it be pack rat or RENV will be one of the major steps we need to consider as part of the submission process. Yeah, I agree with some of the things that we can control better first and then you know obviously the bigger items, I think we'll hopefully fall into place later on but package management. By far is one of the biggest things I try to champion around the projects I work with. So, so Paul, this is Sean with our studio I guess one of the questions I have. Maybe you could help shed some light on this. It seems like from our perspective as a software provider. We're talking about problems that have been widely solved outside of the realm of FDA submissions. So could you help me understand more like it sounds like you're sort of on an island with very little support from an organization that claims to like want to innovate and we'd love to help with that. So how do we go about like this group at the Arkansas shum, like approaching the FDA internal like it structures or whatever it might be that at some point are going to have to change, because like this conversation about dot lower case are it's almost absurd. So like I appreciate what you're saying that we have this, you know, that you've gone out and done these things with Eric. Well, what can you expand beyond kind of the island of influence that that you have access to. That's a good question. I wish I had the solution. Essentially, we have. I won't say we've set up a second it shadow network. But we almost have. We're calling it scientific computing. So one of the problems that as a government organization we encounter is there's sort of the business government type of side of things. And then there's this need for a computing infrastructure that will support all of the science that's there. And so that's why we've actually developed something called scientific computing. We have a scientific computing board. And that tends to be where we do a fair amount of work. Our high performance computing centers are actually outside of our office of information technology and management. And actually information management technology. So, you there, yeah, you're, you, you might say island or sort of parallel system that has been developed over time to actually facilitate and enable the science to be done. So what we can do is try to engage with some of the key partners. Say, with the submissions process itself to see if we can do some. We can overturn everything but I think having a pilot that would be an actual our submission and see if we can obtain some waivers without having to break the entire system. Yeah, would be a next step. Within work, we also try to build a pilot study that have part of analysis using R to submit to FDA in next year. And we summarize some of our understanding in a slide deck. Would you mind I go through that to share with the team here to see if that's a common understanding based on the current infrastructure we had. That's good to me. Let me share my screen. So, basically, what we're trying to do probably as a pilot in next year is to submit submit one analysis using our based on the SDTM and Adam data that already being created by sauce. So we need to focus more focus on the analysis side to generate table listing and figure. And always go with more focus on the primary or secondary ethics analysis, because as a typical leader part that require a sponsor team to submit to the code to FDA. This is in line with I think the document just mentioned for the FDA study data technical conformance guide. And those part highlighted is more of related to analysis side, because from the guidance I think the minimal requirements for primary and secondary ethics analysis. And we also need to mention what software to be using over a DRG. And we also understand that the code itself based on current FDA guidance is not to say it's actually executable from FDA reviewer. It's more like a source to help reviewer to understand what's a key variable and algorithm we use to perform the analysis. Actually as part of the committee that wrote this. So, yes, the idea was that .exe for example is will be rejected outright. So somebody writes a specific executable code. It will not be allowed to be submitted. And part of that's just essentially the guard against viruses worms Trojan horses. The. So, other parts. Yeah, so essentially it's supposed to be an ASCII text which is what the typical our data set. Excuse me, not our data set but our program is submitted. So we should restrict the code return all in the ASCII text. So those special correct should be avoid. Correct. And that's partially where things are with their. There is a before we even get the programs and data into the past the gateway and into electronic document room there is a staging area that actually will scan everything and ensure that we're not being hacked or corrupted data files are being submitted. I see. And so that's how we've tried to build up over tentative solution for the pilot study. And this is actually the action item we trying to proceed. So internally we perform over analysis in our markdown document. So what we're trying to do is to extract the code from our markdown and put into a TXT file. And we also do certain components check to ensure the code only content as a factor. And also we will have a read me file to summarize the running environment we had. ADRG or ARM. So we will provide step by step instruction to select what's our version and K package and it's our version that to be used to rerun the analysis. And another part to ensure if a reviewer really need to run the analysis, we try to give a public available our studio package manager pass that have the same snapshot as we had to run the analysis. So, if a user for them being a Windows computer they want to run the same environment they can install from this repo to get the same version they need. So that's one way we think that's probably lower the environment running environment to reproduce the results. I'm not sure that we have the commercial version of the package manager widely available yet. So this way, we don't need a commercial one install. So this is the community addition. Okay. Right. This is a public available link, as long as, yeah, so you can link outside of the internet, they should be able to download the package. Okay. That should work. And we're actually attempting to do something along those lines internally. Okay. Well, there a way that you're aware of that we would be able to deliver the exact set of source code of the packages that we use along with the submission, instead of relying on a public package manager. The short answer is I don't know. We could that might be something we can explore. So you're basically saying you would have all of the Are you talking about using like a say a package manager to have a all the packages. Compile together and then What exactly did you have in mind open for discussion because in any validated instance that you Are putting together you're going to need to have snapshots of your packages available on your your local system somewhere and like our studios package manager project or product is one solution to that. But the gist is that at any point we're able to recreate our environment ourselves with whatever qualified environment that we're using for those submission has been set aside. So it sounds like a big piece of this would be understanding the best way that we can give you those components so that you are able to reconstruct it yourself. And I'm hearing the limitations that you have there. I'm kind of having a public access point for this is one one piece, but I mean, with organizations are going to have their own individual packages that they're going to need to hand to some funds well so there's different pieces of that puzzle. If they're, if it's a internally developed package that say not on cram, or say widely available via GitHub. I think there would be some concern about validating that package. So let's just assume for a second that every package that we're using has undergone that validation process, and let's say that we can provide documentation or supporting pieces for that. One potentially could submit all of this as part of a submission. I'm not quite sure how. What type of validation will be considered sufficient. I think that's still something one would have to consider and negotiate. So, I guess when one area that we have done, for example, and maybe even can comment on this is, I know GS design has been externally validated. And actually is part of the Moderna SAP back submission that is actually going up for an EUA. Yeah, we are one of the Merck alumnus, I guess is statistician at Moderna. And I was very pleased to see that that was there. We've also been going through some of the testing and we should have by the end of the year so that code coverage for the package is over 80%. You know, for any new package we do, we're doing it according to our system development life cycle which includes specification and validation. So we're prepared for an audit of any package. I think that we would submit for any critical analysis. If that's helpful. And also, you know, do you want to mention the potential way of submitting a package, according to the current. Yes. So, while we discussed the question Mike mentioned to submit internal package we developed to FDA. And we're trying to say select to zip the package into a TXT file. So the general idea is that we leverage the key component of source coding our package. And then use a special correct to separate each file and put into one TXT file. And then we can unzip that into the original our package if statistician from FDA run a code from a function. So you can see one example here. So this MK serve is one of the package. We try to develop internally to for some survival analysis method. And we put all source code into the TXT file and using another function to unzip into original package for the structure. So, instead of to having a FDA statistician to go through a lot of files in our package. So that can be combined or in a one TXT file structure. So as long as the idea to put that conversion function into a package that could be available on cram. Right, so we have a colleague currently developing this kind of feature and trying to public that package. Definitely in a validated way, and then FDA statistician once install that package we can provide instruction to on the former TXT to a package that can be loaded into the the Windows machine from FDA side. So, well that sounds interesting. I think it is a, yeah, it is a complicated. Well, I was thinking that Sean would probably say that's kind of a complicated work around to get through this. The simple part is you don't have to go through the FDA bureaucracy. So, sometimes the technical solution that avoids having to tangle with the bureaucracy is, it looks like it's the long way around but it's actually the shortest path. Yeah, I can definitely see that. I can see that but at the same time as like a enthusiast of all these enabling technologies that I get to use every day at my job with obviously containers and HPC and cloud deployments. The technical side of me just admittedly feels kind of sad to hear all that you have to jump to just to get there. There's more term solutions and long term solutions. Right, right. Progress on the interim, you know, towards more sophisticated methods in the future. Also, our packages and the structure of our packages historically have been proven to be quite beneficial. I mean, the FDA probably is also interested in like having, you know, all the features that come with it, having the help files and the other things. So, this is true but the, we have our own, well, external folks have issues with our bureaucracy, we have our own internal issues with our bureaucracy. So the gateway, the office that manages the gateway is actually separated from the actual end users, and that sometimes can lead to issues and problems. I mean, we're still using XP T files, which is a 20 year old standard. So, and I've seen two attempts to replace those with XML, but it still hasn't taken place. So kudos to our colleagues for their ingenuity with some of their workarounds but I would agree that it would be good if we can revisit some things and see if we can work with folks but it's probably going to just from what I know of how their gateway systems works. It's probably going to take a while to a build up a justification for why it needs to change and then actually be implementing an actual change. So, if I think that I realized from the outside that that looks kind of idiotic at times, but yeah, Paul, another thought is that it takes a lot of change. If you need any encouragement from, you know, say pharma, they are active in this area with with transcelerate. So, if you need advocacy from the group like that, in addition to say this, you know, we can certainly work on that. That would be helpful. I know that the current OIMT head Amy Abernathy has worked with actual clinical trial data and understand some of the issues that are involved with her experience with flat iron. So I think trying to say, okay, let's actually sit down and figure out how to do a better process. One of the issues is that nobody would do things the way we do them today. If we had to design it today. A lot of it is left over from various legacy systems, starting from well over 30 years ago, and then basically things, a creed knowledge, and other things rather than basically being redesigned from scratch. So at some stage it would be probably helpful to consider how to update and modernize the submission gateway. But you know that that's easy for me to say because I'm not a part of the office that handles that. So, it's kind of like saying to your neighbor she would be nice if you could paint your house. So Paul is that office that owns the gateway is it really one office or is it really like another some group that starts with security that they defer to and another group that you know has various other responsibilities. The gateway is handled by the Office of Business Informatics, which is actually under the Office of Strategic Programs. And the group that actually does the gateway is mostly contractors. So the experience and working with contractors in a federal setting. They're not necessarily interested in improving the process. For say they're interested in fulfilling the terms of their contracts so they get paid. And I say it's the easiest step to make a small change like how do we get to uppercase fine names. I think that's the first. Very, very, very nice suggestion, maybe. But I'm sure that the our consultant has some routines in place to make sure that the submission that come to the current servers are legit, and not some malware or something. Maybe there's some possibility to benefit from the procedures. I mean, I can see that, you know, you need some checks to not being getting stuck and stuff. And like trying to renew things is very difficult. I hate to say it, but the simplest way. Well, okay, so part of it is I don't. Well, it sounds good from outside if a government organization could borrow from the foundation, such as they are foundation. The likelihood of that being carried out is close to probability zero. There would have to be a contract and have to be led there's a specifications. It's a government bureaucratic nightmare. I can think of one way. And you, you're not going to like this, but it's the work around system is to submit everything in lower case, and then you write a script that then takes all those lower case things and attaches uppercase and then it lets you work with your project. So then you're back to a long solution. So that is, believe it or not, that's probably the easiest step. I had a kind of another question for you, which goes all the way back to some of what Eric was saying at the beginning, which is that in some of these pilots. The FDA reviewer has kind of come to the pharma environment, as opposed to the farm was environment coming to the FDA. And you express some concern there that you know, a sponsor may need to keep an environment around you know indefinitely. I understand I mean that that actually seems very feasible compared to some of these other things we're talking about a shelving a Docker container or an AMI or whatever. It's certainly easier than writing a script that messes with the case of my file names. Is there any like appetite there. I guess, in the FDA for that type of model where a sponsors environment becomes the source of truth or are we really in a world where it has to kind of be thrown over a wall into the FDA. An interesting question. My own concern is that FDA should always be able to look at the data in its environment. And part of it is, I may trust a company's statisticians, I may trust the IT folks. I'll tell you what I don't really typically trust the marketing folks. So, I could see where at some point in the future, marketing folks might get control over some of the environments at least partially and say slow down things as a way of thwarting investigation. I can see fighting, you know, investment and banking functions, you know, within a company. Right. And then, you know, I could also see bad players within a company, even saying, Okay, we can, we can basically track. If we control the environment we can track everything that FDA is looking at in this. And we can anticipate. Essentially, you can see the cards that are being dealt to the FDA. And we can anticipate how to game that. I'm not saying that your companies are necessarily ones that would do so. But if that power exists within a company, I am not sure I trust everybody in all commercial enterprises to behave in an ethical manner. So can I ask you, I've been, I've been told by a senior person or senior in age that the pharma companies used to give actual PCs like computing physical boxes to the health authorities and they at some point said, you have to stop it. You have to be dealing halls with your computer. But it seems like almost the easiest ways to give you a PC without the network card or laptop nuts. Essentially on your own term. Yeah, our environment at one time because Steve Wilson probably has reminded me that back in the 80s. This was done that and or government procurement and it was so lagging that an environment was physically given to enable this but at the same time there was still the paper submission. Which go back to that too. Yeah, if you've seen the paper submissions. There's like truckloads of them I've seen pictures. And I can tell you I've seen, I've dealt with a submission that I looked at. I may have actually once hit print accidentally and I realized that there were 10,000 pages in this submission. And then I had to hurriedly go cancel the print job. So, I think part of it is, let's identify what needs to change. It might be something that can even be advanced through various channels. Transcelerate is one even Padufa is a possibility. If company say when there's time at the Padufa negotiations that we would like FDA to modernize its gateway. I know what FDA negotiators will say is that, okay, we'll FD. Let's negotiate what that will cost. And how that will proceed. Most likely won't be prohibitive. True. Yes, I would guess so too, but those are some of the aspects that occur. So industry has its has some leverage as well as FDA. Well, I mean, especially if you don't follow the accretion model you mentioned Paul. But I don't know. Maybe I'm wrong. So we're running out of time here. This, this has been, I think, a fascinating and help productive discussion. Some administrative things. I will work. I'm a little bit behind, but I will work to get a GitHub repo setup. I think that will include the video recording and minutes from the meeting. And please if you long and Paul, if you could send me your slides, I will make them available to in the repo. And then we should decide on the next meeting. And maybe the cadence of meetings. You would like. If this time slot or perhaps a little bit earlier would be, would be better. So how about some time in January is the week of maybe the second week in January is that a good time to meet next to the week of the Monday the 11th. Friday the 15th, nine or 10 a.m. Pacific time. Anybody have any. Looks fine for me. Works for me. Yep. Yeah, I'm fine. 915 9am Pacific. And I'll make sure everybody gets an invitation. And we can, once the GitHub setup, we can use the issues and, you know, and whatever to continue the discussion in between time. So, thank you, everyone. And let me know if I can do anything further to, you know, facilitate communication. Thanks for setting this up. Thank you. Thank you. Thank you, John. Take care. Goodbye. Bye.