 Comments for David. Yes, Heidi. So, David, I'm curious, you know, we all sign out cases all the time that are inconclusive, variants of unknown significance. So I'm curious to think about or ask you, as you think about whether to report out that last variant as, well, here's a candidate, but we don't have conclusive evidence to say it's causal versus reporting it out as negative because it didn't meet the bar. Like, you know, in thinking about the risk-benefit ratio to that patient and what may or may not be done subsequently to try and solve the story, and like, how do you think about when you decide to not report anything versus put something out there that's inconclusive in the context of a genome exome sort of study? Can you bring up my slides again on my computer? So we actually have pretty tight rules. I think one of the things that we've been very aware of is the amount of damage that's been done by variants of uncertain significance reporting and things like BRCA1, where people have had mastectomies based on the lab reporting of a variant of uncertain significance, and it being misinterpreted as causal. I think in a typical genome, you know, we get four million single nucleotide variants, about half a million interesting structural variants. Just by random chance, certain of those will fall in genes that are interesting. And typically, I mean, when we've done mapping this way of interesting variants that fall into interesting genes, we typically find between two and 20,000 variants that are quote, unquote, interesting. We are, I think, very shy of reporting out new genes, even with functional evidence. As a clinical lab, I think with my research head on those are the ones that are most interesting to go after and try and prove. But as a clinical lab, I'm not going to want to be the first person to call a new gene a pathogenic gene. So I think we fall very strictly into this box. This is the kind of the box. And we were talking earlier, you know, we're relatively strict about following the rule. So we won't report out category X variants. We will often throw them over the fence to the research lab and say, hey, you might want to look at this in this patient and a cohort of other patients that we have with a similar phenotype. But as a clinical lab, we won't report out. Other comments? I shouldn't have stopped you quite so soon. Let me just ask, as a dumb epidemiologist, I hear a lot about HGMD and the challenges and the problems with it. And I guess in my own mind, I'm thinking this is probably a group that wants to capture all the information that's out there and have other people do some judgments on it. But is there not some way to have it capture some of the judgments as well that we're, I mean, has anybody approached them about that? Yeah, we have. And they've said it has to be published in the literature for them to put that reference in there and they will not override what's been published. So it's been very frustrating. Or is it there isn't some way to work with that? Again, I, you know. Well, they just say they're very honest. They have no, they don't have the resources to do that. I mean, they would have to operate a 24-7 call center to be taking info from all of us who are doing this stuff about what we think about their variants. Yeah, I mean, as it is, it's $27,000 a year to use that database for clinical use, for clinical service. Just in building our pharmacogenomics resource, we think we've thought about this a lot. And I think the only thing that works is we curate as the author would have curated on the day they submitted the paper when the sky is blue, the sun is shining and everything seems to make sense. And then you have, so if you curate that way, then you can build on top of that an infrastructure to say, not so much, time has told that this is, but if you start making the initial curation filled with judgment, then you just have, it's untenable going forward. So, but HMGD and any other database can do that first level and then on top of that have mechanisms. So why not have a super HMGD or whatever that says, okay, here's where we started and here's the evidence that we've evaluated. Some of these filters that, in the examples today, a variance 20% frequency in Europeans, it should be pretty straightforward to pull out all 1,000 genome frequencies or all EVS frequencies and annotate them in HMGMD. So obviously there's the corner cases. A variant was reported, but there were no appropriate controls, so who knows exactly what's happening, but these extreme ones, it should be possible to flag all of them. Not have to go through them repeatedly. I mean, HMGD actually did work with 1,000 genomes to flag some of those 1,000 genomes variants, the show up again. But do they get flagged? I mean, I know, we talked about them in the project, a few times. So it's weird, they have some internal flag, but it's not distributed in the public release. It's really bizarre. To academics, it is distributed for some reason. Yeah, because I asked them about that. I said, you know, there's all these variants that are at like 90% in the ESP cohort, can't you fix that? And they were like, well, it's not in the download that you get. So maybe an action item that we take out of this is we don't necessarily have to make HMGD change the way they do business, but if we took it upon ourselves to make sure that a community action we undertake is to draw from 10,000 or 20,000 exomes that have been sequenced in the last few years collectively in a very accurate allele frequency and then publish that, some of our colleagues over here, as a single paper with a single supplement that actually articulates this, then maybe we'd have more traction in getting that uptaken by not just HMGD, but any database. And I would say just as a clinical lab, we functionally do this. We just have a way of filtering automatically. So we don't report anything that is over a 5% allele frequency. 5%. I'm sort of intrigued by the thought of the sheer number of in-house curated or fixed versions of HMGD that exist in the clinical population. I mean, I have no idea how many that is. Is there communication between clinical labs that could be used to help to merge those efforts into? Yeah, I mean, that's something that we're trying to work on. I've actually had contacted HMGD and said, because some of the U01 funded collaborative groups are all sharing these stories and we all have our own in-house databases where we've gone through all these efforts and we have all the HGMD variants, or I'm not all of them, but we each have our lists. And I actually went, because HGMD is planning to put together Scientific Advisory Board, so they'd asked me to be on it. And I said, I'd be happy to organize some efforts to bring everybody's data that has conflicted with the HGMD classifications and try to bring that to you so it could be put in there. And they haven't agreed to do that yet, but I think at least agreeing to form a Scientific Advisory Board, I think there's some hope that we can get some traction there. It seems like the interpretation of very rare stuff is very much a numbers game, right? And individual, I mean, one of the things I liked about one of the slides we had up there where it almost looks like a medical note, right? Where you're writing a note about a variant, right? And I just, it seems like, I mean, maybe HGMD isn't the appropriate sort of nexus for kind of organizing an antagonistic community about the problem with HGMD. It seems like what you want is something where everyone can contribute annotations from wherever they are about variants they observe in their patients in some addendum type way to get those numbers, to be able to add interpretation over time. Yeah, absolutely, and maybe Donna can speak from the NCBI ClinVar viewpoint in terms of creating a truly publicly accessible environment to be able to do that, and we've been trying to work with them, and hopefully make some inroads in that area. Other comments? My concern is that the utility of the HGMD, despite all of its problems, is because it's curated, and that sort of what distinguishes it from DBSNP, right? Where you can just put anything in. So I think that there does need to be some curation. Maybe the curators are identified as people who have a particular interest in one gene or another, but that seems to be an essential component of maintaining the integrity of such a database and engendering the trust that somebody would have if they were testing lab that you know does their homework when they report something to you. And I can add to that if you could put up number 14 on the screen. So in working with Donna and thinking about how to annotate the curation level, so that who's curated that data? This is sort of the scheme that we came up with where within ClinVar there may be data at the bottom that is uncurated like data in DBSNP and ESP cohorts, but then the laboratories, the clinical laboratories that we're working with to try and get their data in there would come with clinical assertions, but it would be marked as single source curation. So somebody said something about it and you see who said it, so you can perhaps pass some judgment on who that was. And then there's this interlaboratory multi-source where you're actually collecting multiple different curations that may be the same or may be different, but there's some ability to find consensus because there's multiple ones, but they're still done independently. And then you get to the expert curation where you truly bring a group of experts together who in a consensus evidence-based manner collectively decides a level of evidence on a variant and classifies it. And then there's some variants that now today exist within clinical practice guidelines that can be marked in reference to those guidelines, but that was our attempt to really set up a scheme so that you could have a place that has all data, everything from the only population frequency on up to the most clinical grade data and not have to go to eight different sources that are different grades, but yet still have some way to show a difference in the amount of curation that's happened on that data. So it's just what we've been thinking about. I think some of it is simply crowdsourcing. I mean, there's a number of places. We have several people doing just what Heidi described as that happens in her lab happening every day, what David describes. Some of it is just gonna be creating standards in a forum like here and then bringing together all the people who are doing that to speak to the same standard and just pull that data together, because it's happening. I think we just need to harness what's already happening out there and bring it together into one thing. David, Demak, you made the point that you're totally dependent, obviously, on what's in the research realm. So if you had your druthers, how might you re-engineer what we do in research aside from single papers being published on individual things that often conflict? Well, I am a researcher as well. So I think that's a hard question to answer. One of the things that really would make a huge difference is just putting the bar higher for publication. You know, I review papers where they say we found this variant in a kid with this disease and it's like, and the proof that this variant affects the function of this gene, just because a patient has PKU doesn't mean that you found a polymorphism in phenylalanine hydroxylase means that causes the enzyme not to work. And I think for too many years, we've accepted that kind of, should we say, sloppy assumption. And I think we need really as a community to tighten up the publication standards to really show people, you know, if they're gonna show causality, which I know is what we can talk a lot about tomorrow, that really they show causality, not just an association. And if it's an association, that's okay, but it needs to be described as an association, not as causality. And the trouble is association with GWAS has almost assumed a different meaning. So I guess we maybe almost need another term for it as something that we happen to find in an individual, but we haven't proven that this specific mutation leads to this phenotype. And so I think, you know, with the paper that everyone saw with the retraction for the missense mutation from the mitochondrial group, I think that was a very brave thing for them to publish. I think also it was very important lesson for us all, but just because a child has a variant in a gene and the gene doesn't function, it doesn't mean that that variant causes the gene not to function. There's not too many negatives and positives in a sentence. So I think really tightening up the literature is gonna be the biggest deal because then we spend much less time waiting through. And I think maybe also, once again, on the literature side of things, actually getting towards a more structured approach where curators can more easily pick out what an author is causing pathogenic versus associated, because I think, you know, one of the things we criticize the HGMD curators, but I think there are times when I look at papers and I try to work out what the author thought was pathogenic and what they thought was neutral. And I understand why the curators had trouble because, I mean, they're human and it's hard to read some of these papers. So I think also moving towards more clear-cut requirement that authors come out and say they think something is pathogenic or that they think it might be associated and why, and providing them to put in a structured way the proof for why they think that's the case, I think those things would make a huge difference. And then the other things that have already made a huge difference, the ex-embarian server has been a godsend to all of the clinical labs. It's, I said to Bogeet who works in Heidi's lab, this will change your life when we stumbled upon it. And it really, it's changed our life. It's changed all of the clinical labs that I know is live. It's just in a radical way. It's really, really useful. And that's just from a relatively small number of exomes. I mean, it's respectable, but imagine if we brought together all the exomes. And imagine if we actually had phenotypes to go with them as well. Yeah, I can really start dreaming now, right? Can you clarify for me? So this is from the, I mean, Gonzalo can probably speak to the exact number right now, but about 6,500 exomes from the NHLBI exome sequencing project from a, 6,800 from a diversity of heart, lung, and blood-relevant phenotypes. Gugli, EVS, Exome, GS, Washington. They have a really excellent variant server in Google. It comes up. But it has allele frequencies broken down by ethnicity, but not no individual level data and no phenotype data. The other nice thing about the ESP cohort is you can actually determine that a region was covered to be able to say that a variant was absent, which is incredibly useful also. What he's doing is changing his resolution. Mike has advised us that we should be at 1024 by 768, which is a kind of a wimpy resolution if you like a high definition screen, but that's what the display likes. So Mike also needs to test everybody's machine around the table. He's tested only 7 of 32 so far. So some of you could stay over a little bit later, and he can check them or check them tomorrow morning. But Les, go ahead. I just thought it was worth pointing out this paper. Oh, it was worth pointing out this paper, which came out late last spring, I think. David Rosenblatt is basically suggesting going kind of the other way with a lot against the arguments that we're hearing here today, which is to say publish everything. Just sequence them and throw it into a paper. It doesn't matter if it's causative or not. Just put it in a paper. So really the question he's posing is whether or not you should publish it, might change our question into whether, not whether we should be publishing things, but what we're allowed to conclude about our research. And so we're going in opposite directions here. I'm not arguing for this. I just thought people ought to be aware of it. So I just wanted to amplify in the last two comments. I do think that if we have a more structured literature where we can publish something as English text, but we also have structured ways of saying things, people are going to be able to mine that lots of these observations and get consensus kind of properly statistically and so forth. Oh, I just wanted to echo something that you said, David, about the fact that we shouldn't set a bar so high that things don't get published. Because for somebody who works with extremely rare diseases, I'm searching around for that second and third case. And I'm happy to look at something where there's not great evidence, but maybe those researchers and I can put that together to try to create better evidence. So I wouldn't want those to be excluded. I think it's just important to document what the level of evidence is. But we do have to be a little careful with that, right? Because that's creating yet another multiple testing problem, right? If everybody just throws the stuff up and you find, oh, here's two people with inflammatory bowel disease and rare variants in the same gene. You have a huge multiple testing problem there. Yeah, it's a similar point. So it's a really cumbersome way to, first of all, it's a cumbersome way to look for that second or third patient. And second of all, you have no way of assessing the ascertainment effects of what people put into the literature. So surely the way to actually think about doing this is to have some kind of a centralization system that's independent of publication. I mean, that's really what we want to do. You know, I might just note that six years ago or so we were in the same situation with GWAS. And that was one of the things that, you know, there were two things that I think came out of the discussion that Mark so ably started with the despondent gene was that we came up with some standards. And we also, I think, with many of the journals came to an agreement that, you know, if you're going to publish GWAS data, you need to submit those data to DBGAP. And several of the journals really got behind that. And now DBGAP isn't the be-all and end-all, but it was a start, at least, to having some of the data available. And I think we're, in many ways, we've kind of moved beyond that to get to a structured format. And so maybe something that we can be talking about a little bit tomorrow is what kinds of databases would be useful to develop and how might we go about doing that, as well as what the standards should be. Yeah, I totally would like to emphasize and highlight exactly that because we have this sort of now unique opportunity. Like the discussion has already sort of gotten back into trying how do we retrofit our interpretation and evidence based on a huge backlog of literature. And that's a notable and laudable goal to try and do, but we should also keep in mind exactly what the, if we sit in a room, like we were doing, and we try and imagine a future that we'd like to have, what is that future? What is, if we had talked to our future 20 year old, 20 year future colleagues that said, if you had done these things now, what would those things be? And I think Terry's exactly right. We have to imagine what kinds of databases we'd want. Maybe what kinds of things constitute evidence and what kind of category storms for particular variants we want to rate in terms of evidence. Because of course, statistically, that's gonna be the first thing we're gonna look to. And I think it's possible to do things like rate variants and be mindful about what kinds of rigor those particular variants have been, whether or not they've passed a multiple testing threshold or what kinds of evaluations given studies that have been done and discoveries that have been made on those things, what that actually means. We can put that into a database, into a way of structuring information and then start to gradually build and document evidence that's non-statistical on top of those things. I think whether or not, sort of, I think what Mark will probably tell us about tomorrow, sort of integrating evidence, I think clearly has to be a way of doing that. And so we have to imagine what the structures of information that we have to sort of hold that information and then what kinds of queries we might do to that. I don't know how else we're gonna, I think that's sort of what we should think about framing these questions. Because if we try and go backwards, I think we're gonna get stuck. That's an excellent point. And I think we were hearing while you can look backwards and find the history that makes you really uncomfortable. And I think the stories that David Valley and Heidi and David Dimmick have told us should make us all lose some sleep tonight. Because facing those patients is a really scary thing. And yet, we're really just at the beginning of this. I mean, how many of these have been published so far? Not very many. And there's gonna be a flood of them. And so this is our chance. So get energized for tomorrow. I would say that pseudogenes and paralogs don't just deck me from sleep, but they give me nightmares while I'm actually sleeping as well. I'm worrying about them and calls we're making for patients. I was gonna say one theme that we had been touched on briefly that was very relevant in the GRS here and is relevant here too. I mean, causality is often a multiplex thing. And so it's not just the significance of the causal relationship, but the magnitude of the influence. And sometimes that can be almost impossible to quantify with such small numbers, but it's not something we should forget when we're thinking about what we've traditionally known as single gene disorders, single variant disorders, almost certainly are oligogenic. I agree that we're just at the beginning of this, but the pace at which it is rolling is unbelievable. And I know that at a place like ours, which has a fairly prominent genetics group, there are clinics that are ordering whole exomes from a variety of commercial sources and using that information to deal with patients. And I have no earthly idea how they interpret the data, what they tell the families, anything. That's a sobering thought, goodness. That's called Hopkins Hopkins. That's a jake issue. Well, and the scary thing is obviously everybody wants to practice up-to-date state-of-the-art medicine. And if that becomes the standard of what that is, that's real scary. David Demick, you have your mic on. Is that to make us more nervous Okay. Well, it seems like we're at about a stopping point now. Daniel, did you wanna make any progress? No, I thought it was incredibly constructive and really sets the stage for tomorrow. I mean, we have a lot to get through. But I think if we can focus on the points that were raised in this particular session about the sheer importance of getting this done and getting it done quickly, then I think we can really make some serious progress tomorrow. So just again, to lay out the schedule, we'll have one-hour sessions from each of the different working groups. I would like to really try and focus on the discussion as much as the actual presentation. But of course, this is an opportunity for the working groups to show off all the work that they've done prior to the presentation in laying the groundwork for this discussion. And we would also ask if possible that the presenters in those sessions may be finished with one slide, laying out maybe five or six points that they see as absolutely crucial to help guard the discussion just so we can make sure that those critical points do get covered in the subsequent conversation. And on that, I think that's it for me. Yep, I think that's it. Any closing comments? We started at eight o'clock tomorrow, so, yes. This is the first time we've actually done live streaming and we're very excited about it. It's not that we're behind in technology, it's that we're trying to do it at no cost. And so, this is the no cost version of it. But we may not have that capability. Maggie, so they were asking, could it be posted two days? Yeah, so it won't be in time for the meeting. We can circulate the slides though from all the presentations, so that's what we're going on. Yeah, but you want them like tonight, right? Yeah, yeah, yeah, that's it. So, do you have the slides now? Okay, so we need, so we have Marks so far? Yeah, Heidi and David. All right, so if Heidi and David, you can send them to Chris and then you guys can send them to everyone. So you have the emails of everyone? Yes. We're done. Break.