 for his institution, which is TGen, so he can't be here. But David really led all of the calls. And so I want to give David credit for this as well as the rest of the team. We had six conference calls. We had a very aggressive agenda and a large number of topics that we were trying to discuss. I think what we can summarize, and I'll go through some issues that were put together by the team, is there are six calls. And the recurrent themes that kept coming back and forth, which was to look at the need for standards, and not just from a research point of view. There's a lot of issues that we really have to shape around this, and we'll talk some more about that. There's also a recognition that there's, we don't need to be repetitive. In fact, many of the team members are part of other groups. And so one of the challenges is how do we not duplicate effort and take really, really busy people and center their focus onto some productive activities? Not that this isn't, but it's just everybody's quite busy that's on these calls. And the other issue then is for sure that kept coming up was a need for central data repositories. I think we've heard that several times here. That there's a real need to have data to make comparators against for making diagnoses and quality control and so forth. But in using those, that's an easy thing to say. What are the pieces we need to have to put into play? So some of the aspects that we've taken on is to really look at what lab best practices has just a high level before I dive into these. A really important piece here is that the technology space is changing very, very quickly. And one of the challenges then is, although we don't like the terms whole genome sequencing, we don't like the terms whole exome sequencing, one of the issues around that is, is how do we know when we have that? And how do we know if the next technology is any better or any worse, as we all drive to having better, faster, smarter, cheaper, there needs to be some standards around that. So that's the high level of this. And really the key issues is that the laboratories are in need of guidelines for operating the platforms. Technologies really need to come up with some technological guidelines. But one of the big issues is, what are the quality control metrics? And what are the ways to take a look at this? So what type of genome should we look at? Or what genomes should we look at to know whether or not this new technology is better or worse or stays the same? These are essential pieces that go into this. A need for communication between the groups developing standards. There's a lot of people moving towards this. There's different disciplines, there's different organizations, there's different government agencies that are all helping to drive this. And I think in some of the conversations that we had, the committee was struggling with, well, exactly how do we fit into this? Where does this come into play? And so for example, just a simple example, but some groups think developing spiked in, controls is the way to do it, others don't. So what are the strategies behind those different pieces? Laboratories are in need of standard samples, that's what I talked about. Analytical best practices. We've started to hear some more about this. This is really the elephant in the room often is what do we mean by analytical best practices? So there needs to be a defined set of standards in tools for analyzing these genomic data. Again, the standards need to be established. So what do we mean by quality? Is it the duplication rates, the minimum coverage, the quality coverage, what's missing, what's consistently missing, what's randomly missed, are different ways that we have to come in and be thinking about these. Then the standards for looking at what are the false positives and false negatives is a simple issue. For those of you that are involved in this, we talk about coverage a lot. We talk about 30X coverage or 90X coverage or 120X coverage. But 120X coverage that misses the same gene that's an essential piece is still always a problem. And as Heidi pointed out, do we then go into this where we need to have Sanger sequencing or other technologies that are in place in order to make those declarations as we go down the road and look at these? So should the standards should be platform independent? But the different techniques. So for example, how do you align is gonna be very different if you have a 10,000 base pair read versus 100 base pair read. So there's some major fundamental issues that we're talking about is what do we mean by a de novo assembly? What do we mean by an alignment? What do we mean by a successful call? The other issue around this is kind of the next tier from that is really looking at the need for software and standards and tools that feed into the diagnostic market. The data analysis tools are developing so quickly that it's difficult to define appropriate parameters. Again, it comes down to which platform are we going to do this on? It makes a big difference what are we gonna do this on? So one thing you could think about is there are a set of standard, I'll say, assembled and aligned genomes that are used for one application and there are others that are raw data or DNA that is used in another alignment. Software and databases that lack, rather than dynamically change to support the fact that software and processes must be validated. A big, big problem for almost everything that we're doing is Heidi mentioned that there's the clinical side as well as the research side. Well, the research side for data is changing as we sit here. People are dumping data into these things. The quality control around those data are variable. They're highly variable even in some of our favorite databases. There's big gaps in the curation and the quantification and validation of what do those variants mean and how do we assign information on that? Not to mention text mining tools and algorithms that are all being deployed to suck that information out as quickly as possible to process those. So there need to be ways to take a look at that. The other thing we looked at is standards for reporting data. Laboratories are in need of a defined set of standard reporting genomic data. Again, we've heard about it from some of the other speakers today. What does that mean for the medical record? What goes into the record? What's the standards that we put in there? What's the standards that the payees are going to be looking for? What's the standards that the lab directors need to take a look at? What information do we need to go? So the expectations as I was mentioning before of covering the relevant regions based upon the indication for testing. So if you're going to go in and look for a specific disease and you know specific loci, it's a little bit easier, right? You have that basis you can go on. But what do you do with the variants of uncertain significance that also fall in that chain? Or what do you do then with the secondary data that comes out of that? And that's also a major topic of issues. What do we report around that in terms of these secondary findings? What's the quality thresholds for these variants that are returned to the patients and when confirmation is required? So when do you have to do the second test? What's the standards behind that? Do we always need to validate with a second methodology? Do we sequence the genome twice? Do we sequence the genome with two different platforms? Really fundamental decisions that make a big difference in the ways that we think about those. Other issues in is reporting the clinical data is in need of standardization. Most clinical data is not structured. Clearly Heidi talked about this. We need to have structured data if we're going to be able to share this. We need to have ways of being able to share this. How do we share that? How do we do this in a clinical environment that allows that to happen? And even amongst itself, the terminologies are not standardized. Across the different fields, there's different ways that we're explaining things. Again, very simple basic things that we need to do in order to move this up to the next level. The other section we discussed was central repository for clinical comparisons. So the key issue then is determining the clinical relevance of a genetic variant. We'll require large cohorts. We'll just discuss from LabCorp. What does that mean for clinical utility and validation? If we sequence everybody, there's gonna be four to six million variants. There's seven billion people. We can do all that, but the clinical utility becomes a whole another issue around looking at what does that mean. ClinVar is one example, but reporting standards are still not clear. We're having a discussion at the break with Mary Relling is, are we going to do this? Is this going to be the repository? Or is it gonna be something else? Where are we gonna do this? So there needs to be some basic fundamental decisions. Well, the BIC is another example. And note that Maria just stopped reporting. Different types of submissions observe variants such as in a phenotype to be diagnosed or in a healthy population. What does that mean? What's the context dependency around these? In some senses on the research side, we've had a different way of looking at this. We've been looking for variation on the clinical side. You could have a variant that shows up that's supposed to be pathological and that patient is absolutely fine, at least from the definition of phenotyping and what's been looked at in the way we can diagnosis. So how do we reconcile those issues in that context? Looking at large databases is needed to interpretation and all the HIPAA and requirements around that obviously is essential. What do we mean by interpreting actionable variants? What is that? What does that mean? Is it sufficient to say that we can't treat but it's a value to the family to know that there's nothing that can be done but there is a diagnosis? Or does it need to be that we have absolute certainty that we can treat that? What are these actionable items that we mean around them? Managing the variants of unknown consequences, what does that mean? How often do we go back and relook at that? I was also mentioned, how often do you reanalyze? Do you do that every day? Do you have a spider that goes out and searches at and comes back in and tells you that now this variant does have significance? What do you do with that information? Guidance for lab directors, well how should they do this? In some senses, the lab directors, they always have this ability and make some choices around this but there needs to be some standards upon which we are able to look at the data between groups and because we didn't have enough to talk about, we added another dimensionality of looking at the regulatory oversight and consenting. The laboratories need a set of guidelines on how to operate in the genomic space including how to consent individuals for genomic studies and offering clinical genomic testing. It's clear that everybody's trying to figure out how to do this at the time that the speed of this is happening is faster and faster and what I mean by speed is it's cost drops. The pressure to do this gets higher. It's just a simple economics. When it's really, really expensive we don't have to worry about it but we're broaching that point where you can just say, well why not do it? And that why not is now creating some added tension around how do we put these different pieces in place. So who's got the regulatory controls? What are the guidelines? What are the regulations? What's the flexibility in moving around this regulatory environment? And the availability of trained personnel as well as the financial resources to consent patients for return of complex genomic results and interpret those results is a formidable challenge. Many places don't have genetic counselors that they have access to. In our case in some instances it's hours of consent time and spending time and data return with these genome sequencing individuals. None of that is reimbursed at a level that it's an economic model that would work. Not to mention the amount of time that goes back and forth once the results are in place between the physician, the families, the patients and what do we do with that? That's a whole nother realm that is not reimbursed for and it creates a tension around the financial factors and what's the relative value unit that we're getting out of these individuals. These are big issues that we don't always like to talk about but when we're talking about the financial model, people's time and the money around this is making a big deal. So these are other issues that come into it. We did drop one. We did drop the phenotyping one. We decided we had enough on our plate. So I think what we've accomplished successfully in six calls is we've outlined a tremendous number of key issues. I think one of the areas that we need to assemble then and maybe we do this tonight and maybe we can get some guidance from some of the NHGRI folk here is I think we have to obviously focus on a couple of these and it would be useful to spend our time in some places where we're not duplicating efforts. And so I think that's one of the challenges that we have is how do we do this and what targets could we go after that could then integrate. And I'll tell you the problem around that. Many of these issues, if not all of them, overlap at some level. It's very difficult to have a conversation about doing clinical delivery without talking about standards that go from the wet lab through putting the information back out. So in some senses we can break it up artificially which is what we did, but what we found was that everybody was having the same conversation. So I think that's one of the challenges in this is how do we dissect the complex discussion down to a point where we can get some bullets down and put some meat on the bone to be able to have a chance to take a look at it. So that's the brief synopsis of it and again, kudos to David Craig who pulled all this together and to the rest of the committee for attending as many conference calls as they could. I was pretty impressed that there are six calls that got in from December until May. So thank you for everybody's help. Questions? And there's other committee members in the room so if I can't answer the questions I hopefully you will jump in. If I've missed something, please let me, before we take questions, have I missed something important that needs to be addressed? I might just emphasize just to build on something you just said. So it was a very interesting process. The first thing that the group did was broke up into five working groups. And then amongst those five, three of them decided they needed to be meeting together and the other two decided they needed to be meeting together. And then it became very clear that they were still having the same conversation. So all five working groups then merged back together to have a committee of the whole. So it was a very interesting process to go through. Tons? So Howard, did you guys get a chance to set any priorities on these? So like, you've laid out obviously a research and policy agenda for some time now. And it would be very helpful, perhaps this evening as you said, you wanna focus on a couple that aren't being done. But it would also be helpful to know what are the really big ones? Can you comment? Well, you know, just a straw poll with some people here is that clearly the data and being able to do the analysis and being able to have data to compare against is a major issue. The wet lab, so if I had to pick and the rest of the committee, please chime in if you disagree. If I had to pick where we need to spend some energy, it's on that one. I think the wet lab piece is essential, but that is moving so fast. And there are other groups that are probably more amenable to the big genome centers and so forth that are looking at these technologies. Maybe that's a better place to have that being done. But I think the data analysis side that is looking at the, how do you, not just call the variant, but how do you assign a clinical relevancy to that? Is really some big discussion points because that then brings in your data repositories, your context and your clinical information. So if I had to guess, that would be a big piece. I know there's other groups working on that as well. So Debbie and Rex are gonna adjust my thought on that. No, no, I think that also we talked about actually putting together some data for people to access and to do analysis from all sorts of technologies and to provide what we know as validated variants among the sets. So I think that's a really useful, targeted thing the group can do. And I know David was telling me the CDC is getting together groups of standard samples. And I think that's one thing that the group can do to be very concrete, Terry, because there's a lot around analysis. And as Heidi said. Rex? Yeah, yeah, and there was a lot of discussion. So Les Beesicker offered some sequence data that he's got that could actually be utilized as sort of this dataset for these analysis projects. And I think that's the committee, I think thought that seemed like a pretty good way to go. Other questions from the broader group? Yes, in the back. Yes, so also being on bunch of these working groups on sequencing, one question that keeps coming up. And I think it's relevant to what you were saying that if you want to focus on the data analysis and look how you can exchange data maybe between different labs, different platforms, these data formats. And so did you give any thought to that? Because is it BCF, is it BAM, what works with different platforms? I don't think we've got to that level of granularity. The groups that are I know involved can manage it. We just have to pick which one would we do. My guess is we're probably leaning on the BAM side of the equation would be my guess, but we haven't settled on any format. Scott. Yeah, the other thing we talked about quite a bit was this idea that if someone's genome was sequenced in California and then they moved to say Boston, would that go with them and would it mean the same thing if it was done in two different places? And so we came up with this idea of having some internal standards that anybody getting in this game could sequence and make sure that certain milestones were reached like certain levels of coverage, maybe two or 3,000 positions that we thought were important would have to be score positive or negative. You have to call them accurately. So there's the analysis part, but there's sort of that wet lab standards and what would be those gold standards? Somebody mentioned at several human genome sequences that maybe we can get the consensus sequence, more or less good consensus sequence so that everybody can sequence those and then look how they compare across the platform and across different analysis methods. So that's an effort that you probably know about that's been going on out of CDC and also out of NIST that are trying to make these into reference standards. David, I wanted to follow up on that. I'm sorry, just along the same lines if we had the gold standards, less brought up this idea of maybe having 10 different cases that had planted in there some variants that anybody should be able to find and diagnose. So again, those kind of challenges might be very useful. David, did you have a lot better? Did you have something you want to add? Yeah, I'm trying to decide how to phrase my comment because I still get confused by the conversation talking about a variant and validating a variant when there's two kinds of validation, two kinds of evidence. One is a particular sequence change within a gene and we know what that gene, what disease or trait that gene is tied to is it a known deleterious change? That's one level. And then there are genes that we don't know whether or not they're tied to a specific Mendelian disease or trait and there's evidence about any deleterious variation in that so far unknown gene. And mixing those two gets confusing and part of it is in the GWAS world, you think about variants being associated by an association study that's almost independent of what you know about that gene. Whereas in the Mendelian genetics world, you first determine whether the gene is truly associated with the disease and then you worry about how many of the variants observed within that gene, you can interpret as functional deleterious changes and which one's not and it confuses me when those two different things get mixed together and I'm not sure in our slide summaries we've kept those clear, I'd have to look at them more carefully or slowly but in the conversation it confuses me when we go back and forth between talking about a GWAS type association versus evidence that a gene has been associated to a disease versus in a known gene, is that variant clearly deleterious or we're predicting whether or not it's deleterious? I wanted to amplify on that thought and that's the, suppose, so the scenario is you're doing whole exome whole genome sequencing for hypertrophic cardiomyopathy and each time you stumble across myosin binding protein C mutation you validate it with Sanger sequencing and then deliver that information to the physician. We can have a discussion about whether you deliver it directly to the patient because they've now moved from somewhere to somewhere else but that's another discussion. So suppose you're doing that and you stumble across a BRCA1 mutation. In the next generation data, do you go and Sanger sequence that one as well? I mean, the number of variants that you could start to have to, in the so-called incident alone, that you could start to have to Sanger sequence and then start to engage patients and physicians and the hypertrophic cardiomyopathy cardiologist says I didn't ask for this BRCA1 test, leave me alone. So what do you do with that? I'm looking at Howard or Heidi or Deborah or somebody. I can start to comment. So I think right now where there's very few of these secondary findings that meet the level of clinical action ability that people are returning, the standard has been to Sanger confirm even the secondary findings that are being put in a report and for instance, Baylor is doing that for their key secondary findings but it's at a very small level right now and it's doable. It's quickly going to get out of that realm and what we've drafted, it's not been approved but what we've drafted in the standards and guidelines is a statement that goes along with the secondary finding that basically says the accuracy of this data ranges from this to that and the physician must weigh the clinical, how that data will be used with the likelihood of an error when deciding whether to confirm it because there's certain information that you may decide to put there that may never be used if your warfarin dosing variants in the CIP genes were not completely accurate and you started a dose differently, the outcome may be a little bit different than if you did a prophylactic mastectomy in a woman with BRCA1 variant that was inaccurate. So I think it's going to be hard for us to concretely say all variants on a whole genome report must be sang or confirmed because there's just no capacity to do that but there's clearly circumstances when you would never want to act without, you know, whether it's a technical thing or just simply repeating for sample mix-up issues. There's another reason we come confirm did the sample get mixed up even if you really believe the technical accuracy of the result may have come from the wrong sample. So I think all those things are going to have to be weighed and they're a little bit more complex than just a simple answer. I know that you're working on the CLIN action proposal and I think one of the aspects of that is to think about genomic critical values. We have critical values in laboratory reporting all the time and so are there certain genomic findings such as a BRCA1 or 2 mutation that have clear action or, you know, MEN mutations? I mean, those kinds of things that have monitoring and potentially therapeutic implications that must be reported when they are incidentally found and you picked BRCA1, but that's patented. And so when you're thinking about doing an exome, I mean, I know, I mean. It may not be if it's whole genome sequence. Right, but it's not clear and actually having a ruling on that or, you know, some thought, someone thinking about whatever infringes the gene patents out there and what doesn't because the claims in every gene patent are different. But that's a huge issue as well. Do you not report that or do you say that you recommend that your patient have BRCA1 and 2 sequencing or testing done? But then you know something about a patient that's actionable potentially and it's complicated. Well, just a comment. So in RU01 led by Robert Green, we'll be looking at what the cardiologist does when they get that result back as well as the primary care and hopefully we'll be reporting it back in gathering data around what gets done. And I would just point out that in the ACMG committee on secondary findings that Heidi had mentioned in her talk, the critical value approach, although we're not characterizing it as such, is really one of the guiding principles. In other words, the thing that if you stumble across it, you know, particularly things that have a, that may have a long lead time or have a presentation where the presentation's like death. You know, very profound clinical implications where you could actually do something about it. So I think that that's a very useful construct. It won't map directly into how we characterize the genes and the variants, but that's sort of the list we need to begin to develop. I'll use this as an ad for our clinical research interface subgroup. But as I'm sitting here and listening to the very careful way you're talking about reporting back clinical in a clinical setting, I can tell you at the research level overseeing IRBs, what we are faced with are investigators who feel like they have a duty to hunt. And they'll use the critical value concept, but essentially if I'm looking for, you know, something to do with Alzheimer's and I find the BRCA, I am morally responsible for searching out that person. So I think that, I mean, there is a gap there and it just makes me very fearful in terms of, we don't know what platforms are doing these on, what the other, you know, just the sample issues. So I kind of have two comments, one of which is from a very practical point of view, depending on what state you practice in will determine what legal requirements there are for data return. So in many states there is a legal requirement or the DNA result is actually the property of the patient. So Massachusetts is one of those states, one of the Tacotas is just enacted a similar law. I think there is no substitute for knowing ahead of time what the patient and the physician wants to know. The reason why we implemented the policy we did of asking patients what they want to know is for this exact question, our first whole genome case, mother sister had died at 30 of breast cancer and the family was BRCA1, BRCA2 negative. The whole issue is if we found a power mutation, what were we going to do with that data? We decided we didn't want to be left with the dilemma of what to do. We wanted to find out what the family wants to know. So I think there is no substitute for actually ensuring the physician who's ordering the test knows what they're ordering and the family has a choice in finding out what they want to know. So I want to also put in for people to think about this. So you've captured as the audience and members of the committee the challenges of this subcommittee. For the subcommittee members, you have a few hours to think about what can we do that's implementable before we meet because these are all important but they intersect and we have to come up with some things that we can do. So I would just be interested for me but it has an opinion on it. Will we ever get to the point where we won't have to reconfirm? Yeah, I mean I do think we will be there. We're not there today. And what does it take to get there? Yeah. Improve technologies, absolutely. Better coverage, better depth, better algorithms to analyze the data. So I just wondered in this instance where tests aren't completely accurate whether Bayes' theorem still works. That is to say if you have a not completely accurate test then you go into it with a lot of pre-test likelihood that's a different outcome than if you stumble upon something different. I mean that's what Zach Kahane wrote in his paper on the incident alone was going through examples of applying Bayes to exactly those types of questions and the results are pretty scary depending on how it comes out. So I mean I think these are really critical issues and we shouldn't forget many of the issues of how we've been doing this for 50 years in terms of genetics and learning in the clinic that a lot of those lessons are still applicable even though we're talking about genomes rather than individual findings. If you apply for example, if you found an incidental hypertrophic cardiomyopathy mutation you might then look at the cardiogram and then Bayes' theorem and adjust your a priori possibilities or the family history, we'll have a family history debate later but I think we're in a funny world because the whole concept of the incident alone takes the a priori down to zero. David, last word. I just wanted to ask the many experts in the room what is the accepted level of accuracy for a test that doesn't have to be validated? In other words, what is the level of accuracy have to be in order to bring a test into clinical utilization? My rule of thumb is never launch a test that's worse than a prior test unless there's a real economic argument. Like you think about Down syndrome screening and the use of the triple screening. It's not fully accurate but it's meant to be used as a screening to then lead to a more accurate diagnostic test afterwards with amniocentesis. I think you have to think about the clinical context in which you're using them and whether you're acting on end points versus is it a broad screen that then leads you to a more specific and today I feel like either by using Sanger Confirmation or other orthogonal technologies in the context of the single test or you're recommending follow up confirmatory testing afterwards is where we are today but that may evolve as the clinical scenario has changed over time. The answer to your question is there's no objective systematic definition of what it takes in order to stop doing confirmation. It's a subjective process at the level of individual labs and the community of laboratories applying a technology. So if you're doing confirmation over a period of time if there's never a discrepancy you stop doing the confirmation and you campaign with your college to change the standards and guidelines that recommend a confirmation. So it's very subjective and should be quantified but it's not. I think that was my impression. I just want to hear it. There are lots of hands up. There are lots of hands up but we have a very packed afternoon and we need to break for lunch. So I'm gonna take Chairman's prerogative and just say we're gonna have lunch now. We're gonna start at two minutes to one.