 Stephen, do you want to join the panel? Thanks. Thanks, both of you, Stephen, for describing your process and Paula for kind of walking us through the FDA regulations and thought process. If anyone has any questions, if they can come to the aisle, you'll have to turn on the mic. But I wanted to, I had a quick question that I wanted to kind of start with. So apparently 2014 is way outdated from Stephen's presentation. And I was wondering, have you had to update your application as your device changed, or how did that work? That's a good question. So one of the benefits of perspective study is that we did lock down the device while we were enrolling. However, that enrollment in that study is now finished. And we've moved to a new institute, a new lab, and the technologies have advanced amazingly. And so our previous methods are considerably outdated versus the specifications today. So we, in all likelihood, will be contacting you pretty soon to update our device description. The intended use is pretty much the same, but maybe slight differences. Also, the other regulatory environment in terms of specialized bodies, such as the ACMG, are increasingly expecting us to return additional types of findings besides diagnostic results. And that's an area that we also will need to think carefully about. That's a new area for us. And what is the requirement for confirmatory assays for those? In general, this is a troublesome area. I can very much understand the need to make sure that a diagnostic test result is accurate. But as we get into the myriad other things that we can see in a genome, at some point it becomes impossible to confirm these other findings. And so will this be a substantive limitation of the types of additional information that patients may want to get from a genome, such as their pharmacogenomics status? And then I had one for Paula. So you said that even if the device subsequently is being learned about, if that's not the primary purpose of your study, if you're still learning about the device, then it is a device. And that just kind of brought up the question. How do you balance safety and innovation? As Steven pointed out, 2014 the methods are already so outdated. And how do you keep up with having to file these device exemptions and move just this balance of moving research forward but still allowing for safety? We are committed to being very interactive in this process and doing the most that we can to be timely. And it is our regulatory authority. It's our responsibility to ensure the safety of subjects. And we have developed processes like the pre-submissions that we try to give informal information as quickly as possible. And also our IDE turnaround times are the fastest that we have. We have 30 days. And we have to make a decision in that time, whether your study is exempt or if you are NSR or if you're approved. And we can do that through the IDE. NSR being not significant risk. I think we had a question in the back first, and then, yeah. I have a question for David and Paula, mainly. My name is Yelena Berglund. I'm director of regulatory affairs at Duke University. So one of the things that our group does is review all PI-initiated studies before going to RB and making sure that all regulatory or some regulatory questions are addressed. Starting from simple one, is there a device? Are there drugs? Do you need to do submission to the FDA or not? So the question that we, especially in the last few years, have quite often is use of IVD as a companion diagnostic. So in a situations like in which you are using IVD to randomize the patients, so you're using as a companion diagnostic to non FDA approved drugs, obviously the study will have an IND because drugs are not FDA approved. If study is run under an IND and there is a companion diagnostic development, can we safely assume that CDR rage been reviewing those protocols as well and some issues regarding device being addressed? Or does every investigator need to do also separate ID submission? Because often those studies, I mean those devices itself simply have inherited the risk associated with the drug. So therefore device itself will be significant trace device because you're randomizing patients to non FDA approved drugs. So what are your thoughts on that? You've been getting some mixed messages. So we will work both ways. So if the device comes in with the IND, CDR reaches out to us and we review the device under the IND as a consult. Or it can come directly to us as an IDE, as a separate IDE. But I think we've seen both. And David, if I don't work too much on the companion diagnostics, but if you have anything to add? Yeah, so just to address that a little more, because this is an area of change. We do see many, and you'll hear more about this later on today, but we do see many IVDs because they come in through an IND and CDR notices it and sends a consult. And we will look at it through there. We are changing our policy. So we have accepted in the past device information and sort of the formal review through that mechanism. But it turns out nobody really, the regulations are difficult enough that we're going to now require a separate IDE if it's a significant risk device. And that is because CEDR does not have experience with the device regulations. CDRH doesn't have experience with the drug regulations. So it sort of works out better for everybody in the end to keep those separate. So we plan on releasing guidance sometime in the next few months that we'll describe this more thoroughly. But now what will happen, though, is through the console process, we'll send language back to the sponsor saying, we need a separate x, y, or z. So you'll get that specific type of guidance. Thank you very much. So I'm Jonathan Berg from UNC Chapel Hall. I'll be speaking later today. I had a question about devices because I think we got some conflicting information about what constitutes a device when we went through our own process. And without getting too much into the details, it wasn't just limited to the sequencing device. It was inclusive of all the possible results that might be returned and the parental decision-making process. So it was really a very broad definition of device in our particular case. And I just want to note that something that's evolved to a changing and thinking much more about this specific device that generates the data, or was there just some kind of blurring of the lines in our particular case? I mean, we do work on a very case by case. We look at everything independently. I mean, sometimes we ask questions because we want to understand the risk of the study. The device is whatever gets you from sample to whatever goes on the test result, on the report. So that is including the molecular geneticists clinical assessment of pathogenicity. That is a gray area. We may need to understand what the process is to understand the risk. So sometimes it gets blurred more because we want to understand what you're doing. So we can figure out if you have a study that is significant risk or not significant risk. As you know, we do not have the authority to regulate medical practice. And that part of it is very complicated. So no, that shouldn't be part of the device. But sometimes it's a part of our review process, for example, including submitting, allowing the FDA to review the decision aid that we were preparing for parents. That was part of the device. Anyway, I think maybe the communication was that it's part of the study. So we need to review that. It's part of the study. Whereas the device was really just the. So it may be some clearer language for investigation where the device ends and the study begins. Right. And maybe we're not so clear with that as to why we're asking for something not because we're trying to figure out the device, but the downstream parts of it do impact our risk determination. And a lot of the times we are trying to get to an NSR determination if we can. Because it reduces our burden, our workload, as well as yours. So sometimes we ask questions that we're to get a better understanding of the study. But from our perspective, the device is whatever is going from sample to the test result that is given to the patient. I'm Jarvis Kern from the NCI Intramural Program, CCR. I'm going to go back to the first question that was asked. So we at the CCR have also developed a clinical sequencing platform, Ex-Home for patients with cancer. And the advice that the FDA gave us is that there should be a central umbrella protocol where the device, we produce a device master file. And essentially all the other principal investigators who are then going to use that for clinical trials are then going to actually submit an IDE exemption. And the question to you is I think we actually thought that's quite a good way of doing it, actually. Because you spend a lot of time evaluating the device. And then it should speed up the process for all the other PIs when they submit the clinical trial. So can you make a comment on that? And then secondly, who actually determines for those principal investigators who then submit the risk? Can the IRB do that? Or do they have to always submit to the FDA for a decision? I will take the second. And I will defer to maybe David, who may have more experience with this specific topic on the master file. So the investigator makes the risk determination and the IRB. FDA helps if you want. If you're looking for help, if you submit an ID, then we will make the determination. So that's how that is done. So sometimes we get questions from the IRBs. They come to us and they say, we're not sure if this is a non-significant risk. But the initial responsibility is with the investigator and the IRB. And we can help. And then David, if you can talk to the other or Sharon. So I have a question for Dr. Kingsmore. I just wanted to ask about the modifications part of what you talked about about changing the test during the course of the study. I was wondering if you could illuminate, you could describe a little more what the thought process is. Like what triggers the decision to actually change the test and what do you do then as you're changing the test to make sure it's still working the way, giving similar results or adequately results? Sure. Thank you. So new technologies will come along, David. Either a new algorithm, a software algorithm, let's say for alignment or variant calling, or a new tool for waiting variants with regard to pathogenicity, or a new sequencing chemistry, like going from version 3 to version 4, where it's believed the quality is improved or the yield, or indeed an entirely new sequencing instrument, such as the HighSeq 4000. And so as those become available, generally there's some evidence in the literature that supports their performance, analytic performance, generally relative to their predecessors, some kind of benchmarking studies. Some groups are faster than others in terms of adoption of these. We have always been keen to adopt new technologies quickly. And so we will often contact companies who have technologies at the beta test stage when they are still developing their analytic validation information and work with them to define that analytic validation in a particular medical application. Because often it's very much application-specific. And the analytic validation absent its linkage to a specific application area doesn't always make a lot of sense. So we'll often work with them to find the application for the technology. Pacific Bioscience is long-read sequencing. On the SQL instrument is a brand new technology this year that potentially is transformative in terms of our ability to pick up structural variation in a clinical sense. We can see 20,000 structural variations routinely in a human genome with that technology. Our best current approved technology, the chromosomal microwave picks up tens. And so clearly this is going to very rapidly need to be examined in terms of what are the analytic metrics those studies have started to come out. But so far there have been no clinical applications of that technology. And so really we have to think long and hard about how to do that. And it's really a matter that's somewhat guided by our CLIA CAP type regulations where anything that we're going to introduce as part of an LDT needs to have a formal change order associated with it that's documented. It says exactly what has changed from what to what, when, those types of things. And then it has some information related to performance testing in the old method versus the new method. And typically that will speak to sensitivity and specificity and precision. But it depends somewhat on the actual thing. And again, that needs to be part of the document file that we would keep for subsequent CLIA CAP inspections. We have had a lot of discussion and I would love to get feedback because it's not been clear to me when those changes become material enough that we need to go back to FDA and say, we really think this is a fresh investigational device that that was not an incremental change. That was a material change. That's not been quite clear to us. As I say, once you've locked down a study to a specific protocol, you're good for a while. But nevertheless, the pace of technology change is such that even within protocols, we just can't do testing that we now know is, let's say, 2% less sensitive. That doesn't make sense. So this is a difficult question for us to understand is how frequently might you want us to come back to you in this era? Thank you. Cappuccino. I mean, I think there are going to be two types of situations to answer your question. There are labs who are developing methods and deploying them in clear cap regulated environments and who are used to thinking of, I would say, 90% of the things that FDA are concerned about and appropriately so. And in that environment, it's not a huge amount of extra work. It definitely is a learning curve to understand, for example, risk. That's not a clear cap type thing at all. There are some aspects of the device, as Jonathan has pointed out, that also fall beyond the remit of what we think is an LDT. So that matching doesn't necessarily fit that well. So there may be additional things there that we need to think carefully about, and are they in or out? I think if you're doing this type of genomic research outside of a clear cap approved laboratory, you may be in a very, very, very tough spot and may find it extremely difficult to continue. So that's a big change for genomic research, I think, is that really, you need to saddle up with a clear cap lab, get a board certified lab director, make sure that that individual is empowered, and putting in place all of the aspects of training and documentation related to the testing that's done. I definitely think second time round, we have a much clearer sense of what the FDA is looking for. That will help us to focus in on providing that information in an accurate and succinct manner and likely in our first application. We didn't quite get that. We weren't sure why we were providing information, or even whether it was relevant or not, and we have a much clearer sense of that. I would say my personal experience with the FDA was positive, but we did find that what Paula's been saying was true, that this was a team of people who were working alongside us. It was something which actually I benefited from and I think the project did. And I think also for IRBs who are somewhat new to this area, it can lead the IRB to have an additional level of comfort to know that a federal agency has specifically looked at a protocol. You know, I do have to say that I've been at FDA now for about seven years. And I do see that we've gotten better with IDEs, because we've gotten more familiar with them. And yeah, I think that we're learning as well. And our goal is to be, if you have questions, call the lead reviewer. And we benefit a lot from having the information that we need. So if we have a pre-sub and we can understand what you're doing, it's much better than if we have something where we're sending you back a lot of questions, because we can't figure out at all what you're trying to get done. With the IDEs, we do just have the 30 days. We are interactive. A few years back, we wouldn't do that, because we didn't have felt that we had enough time. So now we will ask questions and try to get to a decision in the 30 days without having a disapproval and any subsequent IDEs. I don't know about David is better versed in the resources that we have coming out to maybe help people. The pre-sub guidance document does have some good information. And we do have resources to try to help you put together the best submission you can put together for us to be able to start quickly with the file. I have one more question. Maybe it's a little bit random. So when you are assessing risk analysis for IVD, you're going through these potential exemption criteria. And one of them is if the same sampling is invasive and presents significant risk. So what we often see is that investigators are using samples that already been biobanked. However, at the point when that biobank was created, sampling was invasive, let's say it was a liver biopsy, that if done in present time would constitute four significant risk based on your guidance and the study could not be exempt. It would go into abbreviated ID. How do you reconcile these things? That's again another question that we were getting a little bit different answers from when we submitted different protocols. So I was wondering what is your opinion on that? Obviously when biobank is created, there is no device. So 812 does not even apply. But then once when you're taking samples for biobank, there is no risk to the patient at that particular moment. Then again, we don't want to create a loop in which investigator biobanks it for two days and then use it. So how do you see that process? I think we try to understand what happened and how much of that was a decision based on the study. So we try to understand how much of, so if you have patients that get biopsies as a standard of care, that may not raise something to a significant risk. But if the investigation is requiring that you get this procedure, that may be different. So this is why we need to understand what is happening, how much of it is being dictated by the investigation, how much of it is being driven by the protocol and how much of it are you leveraging information that you already have on a patient. So those might be, you know, those biobanks are because people got these biopsies. So just to clarify, sometimes we do have a clinical research protocol. So it's not part of clinical care or leftover samples. So there is a clinical protocol going through the RB in which it's clear objective is to take samples for research that will be used for some future unspecified research. And that goes into Biobank. So it's for research. But then again, like two months later, there is another protocol which gonna use those samples that been classified before. So it was not leftovers and it's not part of standard of care because I see what you mean there. It wouldn't be considered invasive in that case. So I haven't dealt with that. I'm not, do you guys have any thoughts on it? Yeah, it's, we may address this. I mean, I don't think it would occur to me that people would Biobank them for two days to avoid something, but. We have this question and then after you, we have a question from online. Rajeev Agrawal from ORWH and IHOD. So my question is if I'm using Dr. King's more technology, he already has ID exception. Do I need to again get an exception because I'm using his platform, which is already cleared or do I have to cross file to FDA to get exemption on how exactly I'm going to proceed? So you would come in independently because. So there is no exception to that because he already has approval from FDA and I'm going to use his technology in my lab that I can just use his exception or approval and move forward. You might be able to leverage some of that information to say that this is being, this is part of an investigation the same, but this is not an approved technology. You know, your study is NSR, I'm not familiar with it. So you would be able to leverage some of that, but this applies to each study. So every study is viewed separately. Sure, but like as far as IND is concerned, if a company has an IND and you want to use the same drug, you don't have to do the whole process. You can just do a cross filing against the original IND and get very quick review. So it's not the same here or similar. Okay. Well, we will address that later, yeah, for the use, yeah. And then I think we'll do our online question and move to break. Okay, this is a question from online. If a study searching for genetic variants that increases risk for disease does not return results to participants or their physicians, does it need this approval process? It seems to be something that would be NSR, but I think we'll get into that when we talk about risk determination a little bit more. That scenario is consistent with an NSR, but without really understanding the specifics of it, you know, we'd have to figure it out a little bit more. Thank you to both of our presenters and for answering questions, and we will take a break and get caffeine and get some energy in this room and meet back at 10 o'clock for analytical validation. Thank you.