 I hope everyone had an enjoyable lunch. Session four is risk assessment case studies, and Dr. Laura Lyman Rodriguez, Anastasia Wise, Kelly Kalman, and Jeffrey Seidman are going to walk us through three case studies and hypothetical case studies involving risk assessment for IDEs. And now we have Dr. Rodriguez. Thank you, Christina. And if I can invite the panelists to go ahead and come on up to the table. So this is the interactive portion of our programming. So the success and value of the next hour or so is going to be highly dependent on everyone's involvement and from online as well. Please send in questions. We have worked to construct some case studies to help us think about some of the things that we've heard about this morning and do our best to put our own thinking caps on and think about how we would make different decisions and then talk about the issues that that raises and where we might have come to different conclusions and thinking about the individual scenarios. So I guess one thing that I guess it's crazy, and I'm not sure how many people we have on the web now, but I'm hoping that everybody that was on beforehand also came back from lunch and we will just get going. So the task here is to not to make predictions of here's a set of facts and this is exactly what will happen. As we heard about this morning from FDA, the determinations are very much based on the device and the specific ways that the device is being used. And so the case studies that we're going to talk about are hypothetical and they are meant to be a thought exercise and just an exercise. So we do hope that they'll be useful, but they are something that NHGRI worked to put together and so they represent just that, our thought exercise. We made some assumptions about different things that we put forward as study design. Some of you may think that they're not quite realistic or that in reality they would look a little bit different. Obviously we tried to make them pretty realistic, but we were also trying to pull out certain points. And so sometimes we let that creativity be a little bit more of our guide than what we might expect to find in the laboratory. So just to get started we'll move on to the first one. I guess I can use this will be a little bit easier. So protocol X. I'll just walk through first what the basic parameters are for the study design. So 100 healthy newborns enrolled in a NGS screening program using whole genome sequencing. Pathogenic results are confirmed by Sanger in a CLIA certified laboratory because the results are going to go back to the participants. Results that will be shared include non-medically actionable conditions with a childhood onset that will be reported to the parents. Also the study design calls for returning medically actionable adult onset conditions to the parents of these children as well. Also within this study there is an aspect of it that includes doing sequence analysis of the parents of the newborns who are being screened. So there's a trio analysis to this looking and in those trios the pathogenic variants that are de novo will be returned if the parents consented to receiving the findings based on the ACMG guidelines with ACMG 56. If so genetic counselor will be available and can provide services before and after testing and that is where we stop so I can go back to this. So before we get going based on this you can take a few more minutes to look at it. I actually read through it to give everybody time to read faster than I can talk but I want to just start our interactive session by asking everybody just at the outset before we go through any of these further how many people based on what you've heard this morning based on what you bring into the room think that this would be a non-significant risk determination. If you were making a determination would would do make a non-significant risk determination for this study and how many would make a significant risk determination and how many are abstaining or something that's true I did not give the choice of exam. Thank you Sarah. How many would market exempt? Okay you'll go with exempt. Okay. Thank you. And I hope everyone online also voted for yourself to have this in mind as we walk through some of the case. What we're going to do now is sort of highlight some of the aspects of the study design that we think are relevant to the considerations that we've heard about this morning. So first of all the device here is the of course next generation sequencing testing that's not a surprise that's what we're here to talk about. And so again as we heard this morning just to reiterate this the device is everything to go from the specimens to the results. So it includes reagents, consumables, instruments, software everything that we've talked about as part of that pipeline. Another point is that the results here in this particular case we have chosen to put forward this orthogonal confirmation by Sanger for pathogenic results and in doing this as we heard this morning this can be a way to have an alternative way to confirm the next generation sequencing information but we still have issues to think about with regard to false positives and false negatives. It's also then important as well to think about the analytical validation data that would be important to come in to support this going forward if the ID request was submitted to the FDA. They would certainly want to see that kind of data along with the submission. Other factors that we thought were relevant to be considered here were the fact that here as you'll recall we're talking about screening healthy newborns. Also in doing the screening they're being screened for disorders that they wouldn't normally be screened for so this is a difference from standard practice. Also of course genomics is struggling with generally outside of the context of just conversations about IDEs but there are risks associated with returning results here and as we heard a lot about this morning the FDA in particular is concerned about the false results so false positives or false negatives and what the consequences might be of returning either one of those forms of errors in the data and then in particular thinking about what the consequences of returning information that may be a false positive or false negative could be so that's something that's relevant to thinking about the risk attached again to the specific trial in terms of what's specifically being tested for and what the specific implications could be in that scenario if someone either received a false positive and pursued treatment accordingly or did not receive information based on a false negative and again maybe perhaps didn't pursue pursue treatment that might have been recommended. Another aspect that is included within the study design is something that can be relevant to thinking about the overall submission for an IDE would be the fact that the inclusion of a genetic counselor in the course of returning the results to be a resource for the patient participants. So let's see we have a few other things that we wanted to highlight and then we'll go back to the interactive discussions. Again thinking about the aspect here including the testing of trios and the return to parents of if assuming that they consent to receive the information of incidental findings relating to the ACMG 56 and what implications might be for that. So now based on this information our posit was like the majority of the room that this protocol would likely be significant risk and so based on that I'd like to ask the our FDA colleagues to see if they would agree with what we put forward and if they wanted to speak to any of the particular reasons why or issues to call out and we can have any questions or discussion about the things that we raised here as well. So I guess I'll start. I mean I think our concern is the return of this information for healthy newborns. You know I think it's obviously not very typical right now for you to sequence healthy people and return results. You know most of the experience that we have is obviously with someone who has signs and symptoms family history that is being sequenced for some purpose and then you know that's initiating that there's that phenotype initiating a sequencing and we understand obviously there's recommendations from ACMG on incidental findings but I think our concern is the return of results for a healthy study population. Did you want to add anything to that? No I have the same concerns. Someone from the audience want to ask a question and we are just to clarify we have a few minutes to do this so we'll go back and forth with the dialogue and then there's some other points that we might draw out. The way how the scenario was described is that the sequencing will be confirmed by another process or another sequencing. So not being familiar with the disease in question if you look at the exemption criteria in order for study to be exempt it has to be non-invasive device which this would be the case, non-invasive sampling I assume it's just blood draw does not introduce energy which would be assuming case here and does not lead the therapy or cannot be confirmed or can be confirmed with another approved method or device. So the second bullet point was kind of indicating that you do have available method for confirming that diagnosis of that disease in question. So the regulation actually says confirmation by medically established diagnostic procedure. Sorry by what? By? Medically established diagnostic procedure. So I understand that Sanger is an analytically valid way to confirm it but the question is whether or not there's also the clinically valid piece so it's you know is that a medically established procedure to return results to healthy people for genetics? If what is the standard of care currently? What is the approved method currently how you diagnose those patients? If you will return those results from whatever that other method was then you are confirming them before giving parent diagnosis. Correct but you're not that's not the only piece it's not just an analytical confirmation medically established diagnostic procedure the question is would you normally sequence a healthy person? I mean there's an ethical piece that the ID regulations also compel us to consider the ethical piece to it as well in that medically established diagnostic procedure. So yeah I mean I'm no yes it is no the protocol is NGS which is the National Children's Study. No I don't want to take more time from the others but that would be the method for confirming any IVD such as you have another glucometer and you're confirming in healthy or diabetic people you're just trying to confirm the blood glucose so I don't see and we did have examples not with this particular thing but we did have examples in which if we're using to develop in we are developing IVD and those results might or might not be given to a patient but before medical diagnosis is made or any kind of medical decision is made will be confirmed with another medical approved procedure or device that your study will be exempt I mean we had that situation so that's why I'm trying to understand why is this different. As I said I think what we what we're trying to describe is that piece isn't just an analytical piece it's the clinical validity as well and so obviously glucometer is indicated for people with diabetes not for healthy people so it's sort of the same thing you wouldn't use a you know a glucometer on a healthy person obviously we have procedures for evaluating gestational diabetes and other things but you know that's our concern is obviously the use of sequencing and healthy population where that's not right now a typical diagnostic procedure. So you're saying that when we analyze those type of IRBs that we should be focusing or is it healthy people or healthy volunteers or patient population that that should make a difference in that fourth criteria because you seem to be making the that is the reason why example with development of glucometer would be different than this one because with glucometer we presumably will recruit patients with diabetes while here we are generally screening so healthy versus disease is the difference that we should go for. I think I can fill in the disconnect here I think the problem is when you're talking about medically medically relevant procedures right for a healthy individual there's nothing that is necessarily a medically relevant procedure to like normally healthy individuals don't have any treatment have any standard treatment that's used on them and so you're basically saying that doing the sequencing if you do it if you would normally do it in healthy individuals that's one thing but it's not normally done in healthy individuals so it's not not the standard of care for healthy individuals and I think that's where the disconnect is. When it comes to the meeting the regulation in terms of now I've lost the words again medically established diagnostic procedure I mean I think you have to consider the study and who is in the study and whether or not that what you're doing in the study is a medically established diagnostic procedure for the people in your study. So I guess I heard the same theme coming back and I think it was Jeff earlier in your conversation talking about it's risk is not based on the device alone or on the trial alone but on the combination of the two and so that I at least to me I'm wondering Kelly if that gets to your point that it's that while they're saying or sequencing it's saying or unhealthy and you wouldn't normally be testing these healthy individuals for these tests and so there's still some risk that you all would perceive from that because of the population. Well and that's why I'm saying it wouldn't meet the definition of exempt right which is actually what you're trying to. Well I was just going to tell you how I thought of it and I actually don't know if we're even we are thinking of it quite the same way but my take on this the saying or an exemption because I think it's been different say in oncology again than in the sort of genomic study we're talking about is that you know there may be certain variants for even rich for in a healthy population you might know what the medical significance of that one so BRCA and risk right I mean that's something that you know saying detecting that by saying or would this I would I personally would consider that medic you know probably medically established but you will be detecting many variants for which it's not the reporting of that would not be a stand those variants would not be standard for that population right so if you got a variant for you know in the CFTR2 but you're healthy maybe that's not really the reporting of that is not medically established so even if you confirm it analytically by saying or you don't have you don't have any clinical meaning to that and I I don't know if that's exactly a true statement but that's you know I'm just trying to think of an example off the top of my head so I don't know if that's exactly what you're trying to say Kelly but you know that's how I thought I guess it's similar I mean even BRCA one and two we have a lot of information on on these in people with family history or who actually have disease I mean I still think you know we're we're getting a lot of information on what these mean in people and healthy people that have no family history and no disease and does it actually have the same level of risk so I think you've got to be careful in those cases of saying it's always going to be the same for for example healthy population versus the fact that we know a lot more about it in people with you know strong family histories or people who already had breast cancer or other so just to kind of put a slight twist on this and I maybe it comes out in a future case I apologize but if if instead of a healthy study population you had a study population of you know kids with intellectual disability and undiagnosed diseases those sorts of things and in that setting right now it is sort of established medical care to do exome sequencing it's offered by clinical labs it's been up you know taken up by a lot of physicians so in that case if you were looking at a clinical study with regard to exome sequencing in children with a likely genetic diagnosis then maybe that wouldn't rise to the risk if what you're saying is that that would have been the standard medical test to do anyway would that be appropriate to say I mean I think that makes more sense and obviously we I don't know as much about what's going on that field I mean obviously you know we just wanted to hear about what you're doing with the results and and all of that and adult onset conditions to children who let's say in the hypothetical that was just given would with that in and of itself affect your decision about risk level I can tell you that we've struggled with that one and we had a lot of discussion around that and involved you know emphasis to just to talk about it I realized that the ACMG recommendations actually didn't define age and they thought there was a benefit to children but I still think the question is whether or not that's really we understand that's recommendations and we're learning about that that's emerging but I guess the question is whether or not I think I think that was one thing that we struggled with so I can't say that that's black or white so I guess this is example that you gave you the BRAC mutation and stuff is also something that we had and that is and we did have the little bit of that argument with the other parties like investigators like oh but these are already diagnosed as a cancer patient so we are not to read diagnosing them but I see that differences a little bit more subtle value didn't diagnose them with BRAC one positive whatever mutations so that's a little bit different I'm just trying to to now search and see yes they were all cancer patients none of them were healthy although I can clearly see that a lot of studies would simply have a control group that because you need to have a control group on CRO out to for glucometer or anything else so that I would make a distinction at least from what we saw between your example and the one that was given here which process I'm not very familiar I think what I was gonna have seen follow up to actually all of this now that I think about it is you know how do we even define medically established I mean you know in the example you gave Jonathan there's you know certainly it is a common thing to do like in oncology we have the advantage of having NCC and guidelines and you know professional guidelines that say exactly what you want to do in various scenarios it's pretty easy to decide what sort of standard care I'd say it's challenging here when we're sort of in the middle of uptake to say well what's really medically accepted I mean I would think that you know so I think that's why it could be hard to define right now because we're seeing this type of transition I mean I expect at some point say for your case with whole exome for individual disability at some point it will be standard of care I mean that's the way it's going I don't know what date yet or when we would call that but you know I think you know this is going to be a challenging conversation so and that actually goes to the second bullet we had some other things we wanted to raise for discussion of things that to think through not to necessarily have an answer for but to think through about whether or not some of these elements of the study that were added in to address risk how they affected that that determination or how they might be used in the consideration of risk and so obviously we've talked about Sanger and now the group also came to this discussion about does the decision to return the ACMG 56 as a professional guideline at this point how does that mitigate or affect risk determinations and just that question I had as well of there are they are professional guidelines that are out there I'm not sure at this point whether they are commonly considered to be clinical guidelines or standard of care or if that's still a debate and so I think these are useful things to think about and and do highlight some of the areas where it does come down to really looking at the individual cases we've we've heard this morning from FDA the other thing that we noted in the course of walking through the case as an issue that addressed potentially some of the risks posed by the return of results was to include a genetic counselor in those discussions with the patients and I know we often hear about that is as being an answer because they can make sure that that either physicians who are receiving results or patients who are receiving results have a have a better shot I guess I would say at understanding the information I just wondered again to ask our FDA colleagues and you know how subjective is that is it is it still context specific how does that affect if someone is trying to decide should we integrate genetic counseling into our protocol what are the things to think about in terms of what it can add besides obviously the capacity of the genetic counselor to communicate and the knowledge that they would be able to convey well I don't think that I think genetic counseling is obviously we think it's very important we know a lot of the professional guidelines for a lot of the you know genetic testing actually states that you should include pre and post test counseling I think if you think about just the risk of NSR versus SR I don't think you know having genetic counseling would would change that risk but I think that in terms of it for example adding to sort of the safety and that piece would actually just really help us for example in the IDE and approving it because there was that piece conveying the information and I guess those were my thoughts do you have anything else no I would simply agree that having a genetic counselor involved would be a risk mitigator but it wouldn't necessarily change the overall decision was there a question from the web or okay are there other questions or or issues that we haven't highlighted that anyone that this is raised for anybody or questions that anyone would like to discuss before we move on to a different case one of the bullet points was pathogenic mutations are being returned there was no discussion obviously you know what is pathogenic in the terms of when you guys are looking at it when the FDA is looking at what is being returned do you have a cutoff that you all would prefer to use or do you defer to the lab that's putting on the it's doing the experiment or doing the the protocol to decide what is likely pathogenic what is highly pathogenic you know that 90 versus 95 percent question that that was talked about earlier I haven't had a protocol yet do we have any none the context of an IDE but I think the way you know we've thought about that is you know show us how you're interpreting it ought to be fairly you know the the rule should make sense you know I mean we certainly looked at that in the context of pre-market submissions for you know actual tests for marketing where there would be new variants that would be called and we we've said we've looked at the process of interpretation rather than the actual variance but you know it's one place where I know outside of just looking at this the study like this though you know we've had some new drug trials come through where they want you know that a lot of these people who have conditions the you know apparently you know many cases these patients will have been sequenced for with a targeted panel if you will and we do struggle there because often they want to leave it up to whatever lab they use and that is problematic for the trial because labs can define them differently and so you may actually have a problem with your trial if for example one lab treats one as benign in the other one you know so that's that is a problem that we find is that with labs defining variants differently especially when you want to use different labs their own targeted panels and they can differ in their interpretation it winds up causing problems in those kinds of trials but then we cedar that's their problem that are okay okay these are comments from the web someone's commented that when clinical labs report the ACMG recommended secondary findings this is similar to returning results to a healthy population since these people are not symptomatic there any comment on that yeah I mean I still think the concern here is the healthy study population and whether or not I mean you normally would not you know ask to have them sequenced right and so whether it's ACMG 56 or any results I mean that's not currently standard of care okay and then a question from online was to take the bracket example finding a known pathogenic variant in an infant might save the parents lives standard of care dictates return it may benefit healthy child is this still significant risk and why well I'll weigh in you know I think you know in the sense that if you had a incorrect result you know if we go back to the sort of core question of where are the consequences of an incorrect result I I would tend to be on the significant risk side that you know you don't want to falsely report a BRCA mutation because the even the parents well well I mean you have the backup with the parents of they probably would get tested themselves so that might mitigate it but you know certainly if that it would depend on the workflow but I do you know obviously we haven't encountered this before and I think it brings up an interesting question in looking at risk and it'd be interesting to hear from IRBs and things like that about you know this is a situation where the testing impacts not just the patient but it opens up the you know possibility of posing risk to other family members who may or may not have consented to be a part of this and so I think that's you know an interesting aspect of this is what you know how far down the sort of path do you go of of causing you know of people who may be sort of peripherally involved initially who haven't consented to themselves be tested but that the return of the result and a family member would possibly indicate that they may themselves have a condition okay and this will be our last comment on this one and then we'll move on to the second I'm not clear why there is so much emphasis on this false negative and false positive because it's not a significant population you will have with this false negative and false positive unless you say okay NGS is completely screwed up you have 90% false negative and false positive but there is so much better answer you are going to get and but still I'm not hearing those comments but rather more on the false negative false negative false positive false positive and the other thing is for the trio design why will it pose a significant risk because you did find a mutation in the child and you are sort of like confirming whether the parents have it or not so I'm still not clear why this will pose a significant risk you have to remember that we're considering here that the the device is investigational and you've done some analytical validation but it's not to the level of it being an FDA cleared approved device where we would ask for extremely robust validation that means that we're fairly confident that it's accurate and reliable so obviously that is just a piece of it where the potential false positive false negative that's that's the question that's there in terms of risk and one of the other mitigating factors that we review when your ID comes in is the analytical validation and obviously if it turns out that your NGS pipeline is only correct 50% of the time we actually may disapprove that IDE anyway but you know so the accuracy winds up being a mitigating factor in the risk but I think that's you know you almost have to assume that the investigational device will be wrong what's the risk that happens to the people in the study when when the answer is wrong and that's how we are tasked to evaluate risk in a study and then once it comes in we evaluate accuracy reliability and then a lot of these other things like the trio may actually help you once again it may be way to mitigate the risk of a false negative false positive because you have that information I mean as long as obviously there's no found-out question or issue of paternity which apparently happens also a lot more often than people were expecting but you know I think those are all one it would potentially be mitigating factors so and I have to just remind you that significant risk doesn't mean that you can't do the study it just means that you have to have an IDE that we are approving and most the time you know we're not making a significant change you know we may ask you for example more information in your consent document or consider some vulnerable populations but you know we're not trying to get anybody to change the study to avoid a significant risk determination we just would need an IDE yeah I would just reiterate that we when making a risk determination we do look at worst-case scenario we do worry about what could happen to anybody with a false result false positive or false negative okay so we are going to move on to the next case which is very creatively called protocol why so in this particular scenario we have a phase three clinical trial 500 patients with relapse colorectal adenocarcinoma they're randomized to standard treatment versus targeted therapy arms based on the next generation sequencing tumor sequencing there's an oncopenal analyses of both the tumor and the germline genomes in a Clio certified laboratory the primary analysis for the tumors they're analyzed for somatic variants that are targetable based on a literature search there's also a secondary analysis component where germline variants known to predispose to inherited susceptibility to colon cancer will be assessed as a primary outcome for the trial those participants identified to have a druggable somatic variants are treated with a therapy the particular therapeutic and the survival time and or recurrence are compared to standard treatment and a secondary outcome to be measured in this trial is that participants who receive molecular diagnostic reports of both the somatic and the germline variants those with germline variants will be offered genetic counseling so this is our setup and again we'll just take the the straw poll here and see from everyone our three categories of different options if anyone thinks that this particular if this were to come in for an IDE submission whether or not it would be assessed to be actually first exempt from the IDE regulations or how about non-significant risk determination okay and how about significant risk okay and again I hope everyone at home slash in the office is playing along on the web and keeping those things in mind as we start to walk through okay so first thing to think about of course again as before and as we've talked about all day is what is the device here again this would be next generation sequencing the entire pipeline we also have the onco panels that are looking specifically at particular genes a difference in this proposed study design or this case is that NGS is standing on its own there is not any orthogonal confirmation of any of the variants identified another thing to keep in mind is the study population again we've heard this morning about it's not just the device but it's who and for what purpose the device is being used and here we're talking about individuals with relapse colorectal adenocarcinoma so these are this is not a diagnostic population but these are folks who have relapsed and so that can have issues to think about with regard to what is the standard of care for this population and what are the expectations for this population coming into the study again thinking about what the investigators are looking for through this design we have a case where we're looking at somatic variants and that is dictating which therapy arm the participants will be directed into and so this would be something again to think about with regard to something that was raised earlier that we heard about that that makes a difference with regard to risk is what are the consequences then of those decisions and again coming back to what we've heard and talked about even last time the emphasis on thinking about false positives and false negatives especially when those different results can make a difference in terms of which direction a participant goes in terms of their participation in the study again there's a return of results issue just a basic return of results issue thinking about looking at the germline variants and what that could mean and the fact that if we thought about an example of an individual was tested positive for Lynch syndrome in this case what are the risks and responsibilities related to returning that information within the trial and again thinking about what the consequences are for false positive or false negative with regard to this particular result moving on to the secondary outcome something else to think about is again also the fact that those that do receive the germline variant information will be offered genetic counseling it will be a choice in this particular case and so that's something that we've already talked about is is an issue that could be considered with regard to the overall trial and the overall ethics and I guess soundness of the study design when FDA is considering the IDE in particular not necessarily about the risk itself okay so with that actually before I put up the answer that we've guessed I wanted to invite the FDA participants on the panel to share their thoughts on whether or not that first question of this protocol would be non-significant risk or significant risk well I'd like to make a little more general point that this case we have patients with recurrent colon cancer and having recurrent or incurable cancer although that may be a risk mitigator there are there are for a variety of different tumor types there are standard of care therapies for recurrence that do have some effectiveness so that the fact that a patient has incurable cancer certainly does not automatically make it NSR so I guess the drugs that they're mentioned there should we assume that they are FDA approved drugs and we are just or there might be investigational drugs in it okay I think so I'll speak to our assumption in writing this particular study design is that they were FDA approved but I think your point is incredibly important because that makes a big difference in terms of the layers of risk that are presented and so that's a really valid point to be thinking about and so again just for the purposes of conversation we've already heard from Jeff that it depends on the particular cancer type that even having recurrent cancer or incurable cancer in and of itself doesn't change the implications in terms of different directions that participants might move through a study and so it really is based on the standard of care for the particular disease in question so we based on this one and some other information we were thinking that this would be non-significant risk but again it will depend we'll go ahead and go to your question sure sort of a point of clarity can you put back up the the list I'm sorry so when you say NGS tumor sequencing we're in the very top bullet I actually in my head combined that with the second bullet where it said on the panel analysis I was thinking when I first read this that you were talking about a targeted sequencing panel and that was the only sequencing that was being done on these patients but when you say NGS sequencing at the top you're actually talking about like exome sequencing separate from a targeted cancer panel is that correct it's it's it's a question again I think as to what we're talking about a Rebecca but Rebecca is the author wants to okay so you're saying that the only sequencing in this protocol is the Onco panel then I would say that absolutely there's no risk why would there why would there be any risk if you're only if you're only subjecting cancer patients to cancer related sequencing well there could there could be risk if you're depriving some of them of standard of care which you are in you know half the patients are randomized to standard of care and the other half get something else okay and so that's the other question I had was so when you say randomized at this at the start so it doesn't matter what somatic variants you find in the patients that are randomized to the standard treatment or are they not even being sequenced basically because I think all all these little these little things like that make a huge difference in in the risk and if it's not clear that the patients are being randomized and then only the people only the patients in one in the in the one arm are being sequenced and not the other one then that makes a huge difference on you know if you find us a targetable variant in a person who was randomized into the standard treatment you should be using targeted treatment on that patient that would be the standard of care at that point right just go make a question about anchor panels so the anchor panels can be NGS obviously we in the NCI we do 470 genes and those are you know it's not as big as the next one but still NGS so so so anyway but just just go back just going back to your low-risk not significant risk assessment but what about the same question if you sequence you're returning germ lines mutations and soon as you sequence the germ line you're in fact sequencing the family so it's not it's not so what what's your comment about that in terms of risk do you not think isn't it the same rules apply to the first case where in fact you are you know you're you know you are finding something that may be incidental or secondary that actually puts risk to the to the family right well I think I think David addressed that a few minutes ago about the fact that there could be risk to other people family members just to spark the discussion we again had that example and that's why I ask are the drugs FDA approved or not because we had a each way you want in whole genomic department is doing study like that and when we submitted study very similar to this drugs were approved of course panel that they were testing is novel and couldn't be confirmed with anything else it was the insignificant risk which we assumed and for the reason that you just mentioned then which I want to just underline and that is that some patients even if the drug is approved first line of therapy will see splot in and second line of therapy was something else so you might be depriving some patients of first line of therapy by randomizing them based on then their genomic profile that could not be confirmed with anything else currently available right and I would point out that when you say a drug is approved it has to be approved for that intended yes no no in this case it was non FDA approved drug whatsoever it couldn't fit the exemption criteria so on the second bullet are you offering people the opportunity to opt out of the germline results and if so are they being offered genetic counseling before they make that decision because I think that definitely affects the risk so I think that you're right that that would affect the risk and I'm gonna let FDA talk about that but I'm gonna say we didn't think about that so we were not in this imaginary scenario had not contemplated that so I don't know Kelly if you had a comment but if not I have another reaction to well if I recall from the bullets it actually the germline sequencing was only of variants that would make you susceptible to color right so it's not the full panel and I guess it all depends I mean I would get your protocol and if it for example had to opt out but you know we're looking at the worst-case scenario so I mean I think that depending on the information you provided I mean if it was standard of care for people with colorectal cancer to have you know assessments of germline force I mean that's the question that would have for you is because I'm not in I don't I don't do the oncology things for example very much I don't at all but obviously if you know you provide us information that and you know not just your site but many that this was standard of care I mean that would be information that would help us assess the risk so I also hadn't remembered to to flash up here we we did also have some points in here that we had integrated that to try and address the management for some of the risks so again thinking about the study population that this affected it that it was not that it was relapsed cancer we had been assuming that that the standard of care was no better than the other things in this particular scenario but again that's not going to be true in all cases and then again the addition of the option for genetic counseling for the germline variants the reaction that I wanted to raise and just because it's it struck me is that all of these different questions that now you all are posing back to ourselves as we think through this are the same kinds of things that as we've been talking to FDA over the past few months constantly come up in terms of all of the what ifs well what if this particular issue has changed or what if this particular situation is present and why I think it in this situation the details really matter and all of these different caveats of different experimental designs different scenarios of your study population different implications in different populations of what the device is going to tell you really makes them some big differences and even the typical things are what we thought were typical that you could do to try to mitigate risk still our context specific and so it is complicated to think through in the absence of specific information and so again well I hope not too many of you but some of you may be frustrated that you know we're not getting precise clear-cut black and white answers in some of these cases I think it is helping us this thought to go through this thought exercise and think about all of the different ways that we can identify how we could change a scenario and what the implications might be of those changes so I want to before we move on and we're making pretty good track to get to the next one right on time but I think we have some online comments or questions for this one yes so don't shoot the messenger but so any remain there's a comment that says any remaining risks resides with the determination of pathogenicity for a variant that is the practice of medicine and not a function of the device I feel I need to represent our web viewers yes no I think that's a do we have reactions to that or thoughts from the audience as well on on that point that actually kind of raises a point that came up on our lab meeting last week I'm in less be seekers lab as I talked about earlier and he and our lab manager were the one of our people in the lab that does a lot of the sequencing for the confirmation sequencing for the exome results that we get back they were arguing back and forth about whether returning a specific result was made any sense and basically it came down to less the you know the MD was saying statistically it there's there's little risk there and our scientist who was who is actually doing the confirmation was saying I don't care how little risk there is there is risk I don't want to return this and that's that's sort of where like it's it's an interesting like difference between the way that that people in the medical profession versus research scientists think about things I obviously think that you just need to think about how you're defining things I mean because you're going to be you know it's a study and you're obviously have endpoints in mind that you want to see what the outcomes are you know in many cases you know what we're asking for example an IDE is how you're defining these things and obviously making sure that at least they make sense and we rely a little you know mostly on in this case probably some of the definitions that are out there and so you would obviously have to make the argument as to why you're defining pathogenic in that way or likely and all those things and then I mean in part in a lot of these studies I mean those things are being evaluated since I think that it still is a little bit of a you know highly dependent on the person who's doing it you know it's it's going to be interesting to see how that comes out in your study but I don't know if we necessarily tell people how to to change those things what we are asking is obviously it's probably in your best interest to define how you're going to put variants in those bins in a consistent manner and that that makes sense and then you know that would probably be evaluated in your study you're going to see how that falls out in the end. Okay so I think Kelly may have just gone to a point that I was just thinking about a sort of it's again not an answer for today but a place where I think we have continued dialogue to have a discussion about this gray zone not a gray zone but this transition that we have as we're learning knowledge and and the Jonathan talked about it as well earlier where the you quickly get into the clinical validity issues in the practice of medicine and these are not automated procedures with clear-cut answers at this point there's a lot of interpretation that's going on and that needs to go on as we try to learn through this and so I was wondering you know how do we continue this dialogue with the FDA from the research community to work through this time frame as we as we learn and what are constructive thoughts that anyone has on on how best to do that and I think Kelly you were pointing to the fact that something we've heard before is that just because there's significant risk doesn't mean it can't go forward and that in the IDE submission we're looking for is what are the parameters that you as the researcher or the clinical researcher who may be doing some practice of medicine in the context of your research study to answer some questions being specific about how you plan to interpret things so that FDA can see what that is is that a fair now I can repeat that and know that I'm not a clinician and I don't actually know how much you can put that out in advance and I think that's something else that I've I've heard that sometimes is hard to say how concrete can you be upfront about those things just to kind of comment on the the issue that just because you have to submit an ID doesn't mean you can't go forward with the search right that and that's an important point but there's also a substantial amount of burden associated with that for a researcher and so the you know sort of cost-benefit analysis of you know what would be the harm of having IRBs oversee this and what do we gain by having the FDA oversee it you know and sort of how do we balance that that need because yes you can go ahead and and do your research you just have to have an IDE but that means different reporting structures it means having to you know sort of be locked into a specific protocol and if there's changes to make notifications and so forth so it isn't a completely insignificant additional regulatory burden on researchers I can only you know as I said when these regulations were written 40 years ago you know the decision by Congress and the people who are writing the regulations what was that if if studies met this bar that it was important for the FDA to review them and weigh in on safety so I mean I think that we understand that and I think that's why even you know David's talk talked about how research and IDs used to be here and now they're here and so we are trying our best to you know especially be here and you know help as much as we can you know now that we find a lot more of the research is actually intersecting with how the regulations were written 40 years ago another point that I'd like to raise kind of jumping off the previous comment is are we at all concerned about the chilling effect that IDEs might have on genomic research in terms of building some of the evidence base that we need to more accurately determine risk how people are impacted over the long term psychosocial income or outcomes and things like that are we worried about researchers who might be discouraged from doing the research that we need to gather the data to assess risk just a point for comment we acknowledge that and we hear that about all forms of FDA regulation and all we can speak to is obviously the mission that we have the regulations that were tasked to follow and that we're providing obviously you know some evaluation of safety for studies safety and effectiveness for devices and obviously we've heard that also you know with regards to you know the discussion about regulating LDTs so I mean we hear both sides we obviously as we said we've tried to point to how our timelines we're trying to move forward as quickly as we can interact with people as quickly as we can so we can move forward with the decision and not hold things up you know that's obviously not our intent but our our our side is actually to look at the safety of the participants and that's what we're tasked to do and obviously you know innovation in some cases it's safe in some cases it's not and that's where we're sort of the watchdog for that okay so we're gonna move on now to our next protocol which has even more questions and open-ended paths we could go down than the first two so and again cleverly called protocol Z as we move forward so in this particular case we have a large 800,000 person nationwide cohort for a longitudinal study going forward the cohort includes individuals recruited directly from health care provider networks and health care systems pharmacogenomic arrays as well as exome sequencing through next generation sequencing platforms will be performed in Clio certified laboratories participants may download uninterpreted sequence data to look at if they wish to look at the information incidental findings will be reported according to the ACMG 56 as the guide the guiding principle sequence data will be deposited in electronic health records and will be shared with providers upon participants requests though I would have to say if it's in their electronic health record those parties they're going to see it so perhaps this one was that if participants agree the information will be deposited into their electronic health record in addition a resource will be created so that de-identified individual level data can be accessible to secondary investigators through a controlled access process to do additional analyses on the accumulated and generated data these secondary investigators may also return individual level results to the participants who will be able to have access to them if they would like okay and that's as complicated as we got so who wants to say exempt for this one non-significant risk significant risk okay see we're coming together we're starting to agree on some things I have a quick question short what's the primary outcome what's the primary analysis who's doing the primary analysis the endpoint is the resource I'm going to start interpolating here so it is to create a resource and to follow these people over time and to see what is possible to be learned by connecting the all of the data that can be collected a genomic otherwise looking at their health outcomes it is to create a resource and follow people long-term and generate hypotheses that can be examined okay so first question that we've talked about before what is the device here I have another question how many devices might be embedded within this particular I think you end up as if there's many devices as there are clear certified labs that are doing the the sequencing basically like if you're whoever's whoever's doing the pharmacogenetics array and the NGS sequencing for that specific individual would be that they would have their device and they might have a cohort of a hundred thousand of those 800,000 but that would be a device and then another all the different labs would have to be a separate device correct a follow-up to add on to that for some of the FDA folks to ask if then all of those secondary investigators who might be doing follow-up analysis there would that also be included in the device or be a separate device yeah I think I think the hard thing is obviously the any additional I think the hard thing here is that it's almost impossible to assess risk because you're not there's so many unknown risks here I mean generally you know the way that we do is we need to have some quantifiable risk and I think the difficult thing is when you have almost unlimited risk it's significant risk so and then obviously I mean the issue is and I think you know the secondary analysis themselves would each have to be individually assessed going forward because I think in some cases maybe they would be NSR in some cases they may be SR even you know as part of a significant risk study you can still probably have an arm that's like that never seen it this is creative yeah I mean the interesting thing is you know and I noticed one of your bullets even was for example allowing that the actual data itself was not an issue or not a result but I think you know if it's an investigational device you just letting somebody download their DNA the question still is you know how well validated that is the analytical validity piece is still important because if somebody downloads it and takes it for truth and takes it and does their own own analysis I mean that's still in in question there you know without knowing how well it's been analytically validated but I don't know if anybody else's question so as we counted it at NHGRI we we did similarly come to at least two as the answer to the devices for the pharmacogenomics arrays and the exome sequencing and then we're also in thinking about the secondary investigators especially as I listened to it this morning I was thinking that they depending on their analyses and again assuming the intent to return the information to participants that they would be individual devices but that perhaps to simplify trying to submit the information a master file could be created on the core information and all of the analytic validation data for how this this the pharmacogenomic information and the exome information were generated so that that could be something that secondary investigators could just refer back to and then only need to talk about what they were doing in their individual laboratories at least confining the types of things that they might have to do so that again my hypothesis on that is that a reasonable way to think about the use of a master file or I can't say for sure but I'm not sure that a master file can be used by those by people other than those who actually submitted it well figure permission okay we always accept right of reference letters from the original creator of the file to allow other people to refer to it okay also thinking about the fact that these were individuals recruited in large numbers through their health care provider networks asking a question and this is just a question if increasing as I'm trying to look at the notes if increasing the number of people in the cohort increases risk and what does it matter if the cohort is composed of individuals with very diverse health statuses that we've talked before about healthy versus individuals or populations with a disorder and here we're going to have a recruitment mechanism that's not based on that and we assume if you recruit this many people we all have something that's not quite healthy going on and so how would you think about the risk here and I again I don't know if this goes back to Kelly's comment earlier about with there's you know infinite ways to think about the risk is it just significant risk or I don't know Jeff if you'd like to well just as far as the number of people in the trial as I mentioned in my talk we don't look at the numbers so if there's one patient I mean there is such a thing as a one person ID I mean I think the problem with as you said I mean yes it's gonna be people all sorts of different problems I mean obviously though then you're not you know the study obviously is in for example focus to answer a question that would help that population or you know that where something might become NSR because it's you know if we know the oncology one is a little bit easier in terms of you know the risks in that group you can define them very well and I think the problem is obviously here not really being able to define one specific risk and assess it it's just there's it's unlimited so I have a two question I have two questions so that when a patient is in stage four or a stage three cancer insurance company does cover the genomic sequencing or NGS so you can use a targeted therapy so in those cases I mean that data is available in electronic record so do they have to get or they already have IDEs or how exactly happens there so IDEs for an investigation or trial if you're actually not in a study you know but if a test is being offered and it's an LDT I mean right now you know and if it's validated you know and not investigational where it needs an ID then you know right now the FDA has been you know there's been enforcement discretion for pre-market review so that's the difference I mean we're what we've been talking about today is investigation so trials you know we're asking questions versus when you know a test is offered and used by a physician that's a laboratory developed test not in trial so it means the NCI recently has launched this match trial if people are aware or let me go back a little bit when NCI and NHGRI started this TCGA study where like they sequenced thousands of patients and the first one was glioblastoma and they found okay these are the targets and then okay it is giving to the scientific community there are those targets you can develop there are targeted therapies or therapeutic intervention so they do they also classify in the IDE category or not I'm Tracy Lively from NCI and there were there were two very significant differences between the TCGA study where the specimens were de-identified and no information was transmitted back to the people who donated the tissue so it's purely research no clinical significance whatsoever to the individuals who donated the tissue where the match trial of course the information is transmitted back to the person to let them know whether they're eligible and we went through an extensive consultation with CDRH it was actually determined to be an NSR but it is being conducted under abbreviated IDE with very extensive monitoring very extensive characterization of the analytical pipeline that we're using to do the sequencing we had to do a lot of work even under an abbreviated IDE to demonstrate that for instance all four labs that are doing the sequencing are as concordant as it is humanly possible to be so that any specimen that goes to any one of those four labs is going to return the same result for that patient so there's a two very different research projects so if you're NSR then there's abbreviated IDE documentation that's so an abbreviated IDE is just if you're NSR you have to I believe was it call it I forgot who slides so there's labeling requirements there's still documents that you have to keep abreast of but if you look in the regulation actually says if you're NSR and abbreviated then you know these three things you still need to keep but you don't come into FDA you don't have an IDE you're not providing those documents to FDA but the idea is that you should have them on hand okay so we have just a few minutes left and so I'm going to go through a couple other points but just wanted to jog our online viewers to see if you if you do have any questions or comments that you'd like to make to please go ahead and send them in and we have a couple more minutes we can try to get to them so again I think we've covered the discussion of some of these issues in terms of places where clear risk is is introduced potentially thinking about the fact that participants can download uninterpreted sequence data so you know and and do then with it what they might which is something that again from an ethics perspective as Sarah mentioned that is becoming a norm in terms of thinking about the participants right to their information and so study designs like this yeah no that's not exactly what you said but I don't mean to put words exact in your mouth but I mean this this the interest from participants to receive their data and to not have a gatekeeper or feel the need to have a gatekeeper if if they would like to get it in uninterpreted ways is something that is going on and an active conversation and dialogue in the community and so but thinking about that from the IDE perspective as we go forward and the risks and responsibilities again to address or manage some of those risks and then of course depositing it into the EHR so that it can be shared again assuming participant consent to share it with their providers through their EHR's and what are the responsibilities or implications even of doing this in terms of thinking about the risk to participants in in this particular study I can think of lots of things that would come up that that may not be applicable but are things that we at NHGRI think a lot about in terms of just even providers knowing what to do with the information and are there risks from providers having information that they don't know what to do with or doing the wrong thing with the information because they don't know what to make of it and how do we again integrate all of those particular issues into how the study might be able to go forward. We've already talked about ACMG guidelines and and how to interpret or or I guess did different ways to navigate that question I'll just say and then we mentioned earlier this and already raised this issue about secondary investigators having the ability to through controlled access have access to the data resource that is created through this study conduct secondary analyses and also return those secondary analyses directly to participants. So I think there's an issue with those two bullets in that you're saying that the information is de-identified so yeah how would they return individual level results if they found them to the participants I mean you can so I think in this particular study to work it's not de-identified I mean the investigators so it's de-identified and the secondary investigators will not know that this is Laura Rodriguez's is information but would have a unique identifier and a link so that the information could come back into my file somewhere so that I could see it. I mean this actually raised a question I didn't even thought about before which is this idea of patients having access to their raw data and would it be accurate to say that any study that gave participants access to their raw data would be a significant risk study it's unlimited risk right. You cannot possibly I mean for an 800,000 person study how would you possibly monitor the adverse events of that. I mean I think that's where we would have to find out a lot from you about how well you've analytically validated it and all the information you're providing and and I think I don't want to assume that would be significant risk but I think we would just have lots of questions and that's you know where you can come in and tell us about what you've done. Assume that the analytic validity was perfect and you were giving the research persistent a BAM file of perfectly called variance or whatever. So if the data was perfect. The data was perfect and they and it was just free access to their personal genomic data is that a significant risk study. No uninterpreted just raw variant calls perfect variant calls analytically perfect. Well if it were if the data were perfect then the IDE would be easily approved and you wouldn't have to worry about it. But you'd stop to get an IDE it would be a significant risk study. I'm just trying to clarify here. I don't know. Do we have to know on that one for a little bit. Please do. This is exactly well I do think there is this question of whether you know I mean this data has no clinical meaning associated with it. It's not interpreted. So I think this is one of those things that would I am sure if it came in in a pre sub would provoke a very heated discussion but I do think this is one of those areas where we have to work to develop a risk approach. We have not faced the situation before. And I'm thinking of also the for DTC companies ability to return things like you know results that aren't you know basically aren't interpreted or have you know including you know I mean I think you can go either way on this because you could say it's significant risk because a patient could go somewhere and get it interpreted and then make a decision but on the other hand it's you know the variant list in and of itself or even the raw data file that they can go learn bioinformatics and get interpreted may not have any diagnostic meaning in and of itself. So I think that's one of those things where you know we fortunately have not had to decide that yet. So I think you know it's another place that I can think of. It's hard for me and I see a lot of risks depending on for example which even which variants are looking at and for what. So you know I do worry because a lot of cases you know people get DNA file. I mean I'm just going to say that no DNA testing right now is perfect 100% accurate but so in many cases someone's not going to go out and redo it unless they actually find out something is wrong and by then something's happened and gone wrong. So maybe they actually think that they are you know perfectly normal in terms of a drug metabolism by you know this DNA and it turns out that it's wrong and they want to be put on that drug and something goes horribly wrong and so I think sometimes the hard thing with us is that you know when you're talking about being able to value anywhere anything where there's no phenotype associated with it and the problem is that you won't find out it's wrong until you have a phenotype and something does go wrong. So I think that's where you know as Steve had said I think it would be really interesting discussion because I think that's where we struggle where you know these kinds of studies have that question that will be there that someone may actually you know their DNA testing might be wrong and maybe they have a BRCA1 or the fact is that science changes and they may not come back to it and they are moving forward with life with some you know wrong information in their minds either because the DNA testing sequence is wrong or because their interpretation was wrong so I don't know. So I know that at the NCI when you're going back to question when patients some patients have said to me look can we have our exome data and we want Vogelstein's group to analyze it. In that case we just ask our IRB and in most cases they give permission they say the patient has a right to their own data and we just give it to them with our IRB's approval. So we're going to have that be the last comment because we are five minutes over and I know we have some of our speakers left this afternoon that need to get out of here. So I guess I'll just close and I don't mean this to be on a negative note but a challenge for us to think about in particular with this one as we have a situation where we've talked about there's unlimited risk posed that's hard to think about and when we're thinking about as I heard a couple of times our FDA colleagues come back to you and they're saying that what you all think about is the worst case scenario and that's what you're trying to allow for in the IDE submission and make counters for and again I think here the worst case scenario is in an 800,000 person study the opportunities for that are rampant and so it is something that will be interesting and fun to think through should something like this come to you. Thank you all and thank you all for bearing with us through this thought exercise which I know was very far-reaching and open-ended but I hope helpful. As my staff unfortunately knows I love a good pun and we have some speakers who face the significant risk of DC traffic so that's where we're moving into session five is steps after determining risk and we have Dr. Yellenham Berglund who serves as director of regulatory affairs and head of regulatory training at the Duke Translational Medicine Institute and you provide regulatory guidance and support to PIs and other members of the research community in determining regulatory requirements relevant to their study activities and we appreciate you for coming today.