 Okay, welcome back everyone. The clinical genome research or ClinGen knowledge base is an important collection of information that seeks to link information about genetic variants to the phenotypic effects and functional changes that are observed and brings in other important clinical data into that knowledge base. The current grant is about halfway through its current funding phase, and we figured this was an opportune time to give counsel an update about ClinGen, and here to do that is Erin Ramos, Program Director in Genomic, the Division of Genomic Medicine. Thank you, Rudy. I first have to thank counsel. He came to my rescue at about 2.45 this morning. My seven-year-old, well, I felt someone hovering over me. I opened my eyes. It was my seven-year-old. We walked down to his room, and he said, Mom, I checked out a book from the library from the nonfiction section he emphasized. It was on UFOs, so he was extrapolating to aliens, and I said, well, UFO is a generic term. It could just be something in the sky, we don't know what it is, and I said, that should be nonfiction, but if it's about aliens, the librarian should have put it in fiction. This went on and on. He was getting more upset, and then I said, I clearly was like a rookie parenting mistake, and then I said, look buddy, I have to give a presentation to the NHGRI National Advisory Council, and he said, oh, okay, Mom, and he rolled over and went to sleep. So I am calling on all of you, it'll be like every two, three weeks. Just pretend that I'm giving a presentation to you if you run into him. Thank you. Thank you. But in all seriousness, I'm grateful to give this presentation an overview and an update of the ClinGen program. Sharon and I originally planned to give this talk together, but we decided it might be more efficient for me to go through the slides, and then Sharon will give her reflections as both the ClinGen PI and the clinician before we jump into the discussion. Giving this talk on behalf of the NHGRI ClinGen team, which includes Lisa Brooks, Robert Fulham, Nicole Lockhart, Terry, Manolio, Natalie Pinot, and Ken Wiley, and especially want to acknowledge Natalie and Rob, who are the two ClinGen analysts that are very dedicated to the program. So I just wanted to set the stage for the rationale for why we launched the ClinGen program. Some council members may be more familiar with it than others, but if we go back a couple years, here I have 2010, 2011, we're at a point where obviously we are detecting hundreds of thousands of variants. It's greatly surpassed our ability to interpret their clinical impact. From a paper from the 1000 Genomes, there's an estimate of over 88 million known variable sites in the human genome that's a lot of variation to deal with. Of course, we know not all of it is going to be relevant to a clinical disease, but it's a big problem and it takes a community to curate this information and put it together. So this is sort of the state of where we were at at the time. There's a number of databases that were stood up to organize this data both on genetic variation and phenotypes impact on disease, research databases, locus specific databases, there's over a thousand of them that might be a low estimate, or they were focused on organizing information on a specific gene or disease. Population databases like exact, 1000 Genomes, OMIM, and of course the biomedical literature and others. These are all really important resources, but they often stood up without common data standards allowing for interoperability and they weren't necessarily applying the same approaches to interpret the impact of variation. And then on the other side, we've got clinical testing lab databases which are becoming increasingly important. There's a wealth of information in each of these databases from testing companies. And they were at the time largely absent from the public domain. So that's where we were at. We had a workshop called ClinAction, some of you attended that workshop and that led us down the path of issuing an RFA and standing up to ClinGen program. So the main goal of ClinGen is to increase data sharing and build an authoritative resource to define the clinical relevance of genes and variants for use in medicine and research. This is the website. The program was launched in the fall of 2013, so ClinGen phase one. This is a consortium of three multi-PI cooperative agreements. Jonathan Berg, Katrina Goddard, Mike Watson, and Mark Williams are the PIs on one grant. Carlos Bustamante and Sharon Plon on the second. Heidi Reem, Krista Martin, and David Ledbetter on the third. So in total for this first four years, we've invested $33 million. This includes about $5 million in co-funds from NCI, NICHD, and supplement funding from the Office of the Director, so directly from Francis Collins. And then in phase two, as Rudy alluded to, was launched four years later. We're roughly at about $39 million for this phase. It's the same constellation of PIs. And we have co-funding from the LC research program. The All of Us program provided supplemental funding last year. And then I'll talk about this later, but at the same time, NICHD stood up research opportunity announcement to provide funding to use the ClinGen frameworks to curate diseases of interest to NICHD. We'll touch on that in a minute. So these are the faces of the ClinGen leadership and coordinators. So there's 48 people on this slide. The PIs are on the top. This is all of the members of our steering committee, the co-chairs of many of our key working groups and the project coordinators. It's an incredibly fun bunch. They're very committed to ClinGen's mission. Some of you have seen this before. This is a graphic. It's meant to describe at the highest level. Very broad strokes, ClinGen's mission and aims. So first, across the top, you'll see, I'll use the pointer here. So this is our emphasis on data sharing. So the idea is to get researchers, laboratories, clinicians, and also patients to participate in the data sharing process. And then with that data and data from the literature and the other databases I mentioned, ClinGen is asking a few critical questions and setting up our curation ecosystem to help address these questions. So the first is looking at clinical validity. Is this gene associated with disease? The second is then within the gene, is this variant causative or pathogenicity? And then lastly, is this information actionable or clinical utility or actionable? All the curation, the evidence is assembled together into the ClinGen resource and distributed out to the broader community. So I'm going to start here looking at clinical validity. And this is just a quick overview of all the information, the protocols, training materials for each of these elements are available on the website if you want to do a deeper dive. So the ClinGen gene curation working group, which is co-led by Jonathan Berg and Krista Martin worked for at least the first two years of the program to develop this semi-quantitative framework to classify the strength of evidence for the role of genes and disease. This is at very high level, but essentially this framework includes matrices to score genetic evidence and experimental evidence. Putting that information together, it leads to this classification system of clinical validity. So we've got definitive and strong. This area of moderate evidence are limited down to disputed and refuted. So here there's evidence against a real relationship between genes and disease. The team at Stanford built our curation interfaces. So the idea is we want to make the curation process as efficient as possible. So this curation interface has all the elements of the curation frameworks using standard ontologies, data models, et cetera. When the curators are finished with their curations in the system, they can just publish and the results go directly to the website. So this is an example of a gene MSRB3, which was curated for non-syndromic genetic deafness and it ended up with a classification of moderate. If you click on the report, you can actually see all of the elements of the matrices and the publications and underlying evidence. So the first phase of ClinGen, the first four years, was standing up this framework, piloting, validating, and doing our first sort of first round of curation. In phase two, we've really ramped up curation. The network has done 670 gene disease pairs to date. And I just wanted to walk through a few examples to give you a sense of what we've covered. So this paper was published earlier in summer of 2018. It comes from our Brugata syndrome gene expert panel. And they essentially applied the ClinGen framework to the 21 genes that are typically on Brugata syndrome clinical testing panel, so multi-gene panel. When they applied the framework, only one of those 21 genes had evidence of clinical validity. The 20 of them actually were at the lowest level of refuted or disputed. SCN5A is the gene that had definitive evidence. This work was led by Michael Gollum, who's a cardiologist and researcher. So if you look at the data another way, this is just a pie chart, gives you a sense of, you know, that little slice of the one gene that was definitive. And there's actually been testing laboratories now that have removed those other genes from the clinical testing panels, so demonstrating the impact of this work. And then I'm just showing you a couple of other examples. These are all recently published from our different gene expert panels. So colorectal cancer, hypertrophic cardiomyopathy, and hearing loss. And you can see, if you kind of look at the blue slices, so that would be strong and definitive, in none of these examples are we looking at more than about 55 percent of the genes on the testing panels having strong or definitive evidence of clinical validity. And we do have a sense from the community that they're paying attention. So we know, like I mentioned with Brugata syndrome, that companies are paying attention to the ClinGen curation results and modifying their panels based on what the community produces. So then looking at, is this variant causative or pathogenicity? This is a multi-pronged effort. And I'm not going to go through all of it, but essentially public sharing of varying interpretations via ClinVar is the first step. It's really important when we get a sense from the different testing labs, what their interpretations are, assembled in ClinVar, they can use that information to work through conflicts between the testing laboratories and improve the curation in ClinVar. Engaging experts, just like we did with the gene curation, we do the same thing for the variant curation. We stand up expert panels and they work on the particular gene of interest. And I wanted to mention this covers both sequence variants and also we have a number of groups looking at copy number variants. Before I launch into ClinVar, we do get this question quite a bit. What's the difference between ClinGen and ClinVar? So we think of ClinGen as the consortium of people that are sharing data, they're developing the standards and curating the knowledge. ClinVar, as far as the variants go, is the repository where ClinGen and other labs around the country share their assertions about a variant in disease. So here's the ClinVar database. Excuse me. Here I just did a quick search just to give you a sense of what it looks like. So I searched CDH1. There's a number of variants that are returned. You can see this is a likely pathogenic variant for diffuse gastric cancer. If you do sort of a deeper dive, you can see which labs submitted evidence to ClinVar. So Ambri, Genetics, and VTAY. And here's the ClinGen CDH1 variant curation expert panel. And this work has been recognized by the FDA, which we'll talk about in a minute. So as I said, thinking back to our infographic, data sharing is one of our main goals of ClinGen. We have a really close working relationship with ClinVar to get the community to contribute data. So as of this past month, there's over 760 variants with interpretations submitted to ClinVar from over 1,000 submitters across 67 countries. Natalie Pino put this map together in 761,000. That's not what I said. That's 261. Oh, okay, thanks. Yeah, a lot more than that. In the past year or two, we've seen an increase in the number of submitters from South America, Australia, and Asia. And this is just showing you the growth. So if you think about sort of the second phase of ClinGen, Heidi Rehm's lab in particular works really closely with the external community to help them share their data via ClinVar. There's been a really nice uptick in data sharing. And if you think all the way back to my first slide, we had the clinical testing labs kind of siloed. Where we're at now is the top five ClinVar submitters are those genetic testing companies. So over 470,000 variants in ClinVar with interpretations representing 62% of the submissions come from these top five companies. So we think we've definitely sort of broken through there. ClinGen relies on the ACMG AMP criteria for sequence varying interpretation. This matrix you don't need to pay too much attention to if you haven't seen it, but essentially it's just the different components of varying interpretation that get you to a place of being able to say whether a variant is benign, likely benign of EUS up through pathogenic. So ClinGen has a sequence varying interpretation working group that is co-led by Stephen Harrison and Les B. Sicker. And their goal is to refine and improve these guidelines as they're continued to be deployed and tested by the community. So in the past year, in the second phase of ClinGen, they've come up with a couple of publications focused on specific elements of the ACMG criteria. So adding value, clarifying these different elements of the interpretation framework. They also are hoping to move this interpretation guideline towards a more quantitative framework. So they've published an analysis where they're modeling this framework as a Bayesian classification framework and have a sense that they ultimately would like to be able to move this interpretation to sort of using the a priori Bayesian approach. Just like with the VipGen curations, Stanford set up a variant curation interface. And there's all these different colored tabs across the top, and they reflect the different elements of the ACMG sequence varying interpretation guidelines. Again, this is just the way to try to pre-populate the curation ecosystem with publications, with data coming from exact population databases, functional data that's in the community, and getting into a place where the curators can jump in and get started. Once a curation is complete by one of our expert panels, it goes directly to ClinVar. And one tool that's been incredibly useful for this variant curation process that was developed actually by Sharon's lab at Baylor is the ClinGen allele registry. So if you're thinking about you want to curate a particular variant, well, it could, depending upon which build it's coming from, could have different HGVS identifiers, ClinVar ID, DVSNEP ID, EXACT ID. So this allele registry generates a canonical allele identifier or a CAID, which then rolls up all those other identifiers into one element. So the ClinGen allele registry has identifiers for over 900 million variants so far. And if you are seeing a new variant off your sequencing pipeline, you can quickly register a new CAID, either sort of a one-off or you can do it in bulk and register thousands of variants in a few seconds. So the CAIDs are being used by key resources that I've mentioned on the slide, Civic, ClinVar, and the EBI resources. This is just giving you a sense of where our ClinGen variant curation expert panels are. On this slide, there's 29 ClinGen panels, and then four panels on the bottom, actually three on the bottom corner here, where ClinGen has been shepherding them through this process, but they've done their work outside of our consortium. So this is basically, if you're thinking about groups that are just getting started, there's a group that's looking at coagulation factor. And then groups that are further along, like CDH1, those variants are already deposited in ClinVar. So I mentioned this NICHD curation effort. So that funding was used to stand up curation groups in monogenic diabetes, brain malformations, and mitochondrial diseases. They're into the various stages of progress. And then we've had a collaboration with the American Society of Hematology. They provided funding to stand up curation groups looking at platelet disorders and myeloid malignancies. And as we alluded to in Eric's director's slide, the variants that have gone all the way through the ClinGen process have achieved this recognition from the FDA as part of their human genome database program. I just want to say a few more words about that. So what does this recognition mean? It means that the data and the assertions from ClinGen that you can access via ClinVar are considered valid scientific evidence. So a genetic or genomic test developer, they can use these assertions to support clinical validity during the FDA's regulatory review. So instead of having to submit information on all these variants, they can just point to the ClinGen or ClinVar resource. The FDA hopes this program will increase public sharing, reduce regulatory burden on test developers, and advance the evaluation and implementation of precision medicine. It's not just the curation results, but this program also covers the people and the process. So there's an expectation of curator proficiency. So ClinGen has had developed training modules. Our SOPs have to be transparent and available to the public. And we have to, every time there's a new version, make sure all that information is documented and publicly available. So this was a ton of work for the ClinGen program. And they actually sort of thought we might spread this out over the next year or two, but decided they wanted to be the first to set a really good standard for the community and worked really hard to get their application in and approved. Just earlier this last winter, December 2018. So we showed you this in the director's report, but this is some of our variants from the MYH7 curation panel. And you can see they've got the FDA recognized tag. One thing that this program really helped us with led to improvements in transparency and access of our underlying evidence. This is something we were working on in ClinGen and setting up our curation interfaces to be completely accessible. But one of the tools that came out of this program was the ClinGen Evidence Repository. So all of the codes, the underlying evidence that goes into a classification of whether or not a variant is pathogenic or not is all available through this ClinGen Evidence Repo. There's open API access making it as available and transparent to the community as possible. I just wanted to say a quick word about ClinGen's involvement in GA4GH. We are a driver project, primarily participating through the genomic knowledge standards work stream. So there's a lot of effort focused on the variant, the data models that we're using. We're trying to leverage the SEPIO model. Just organize our data in a way that can make it transparent and interoperable. And then moving to is this information actionable or clinical utility? Similar to our other efforts, ClinGen, they work the first phase of the program to come up with a transparent and systematic evidence base for prioritizing genes based on their clinical action ability. This work is led by Katrina Goddard, Jim Evans, and Bradford Powell. We, clinical action ability, the definition that they're using is there needs to be well-established clinical interventions. The interventions are specific to the genetic disorder under consideration and leads to disease prevention or delayed onset, lowered clinical burden, or improved clinical outcomes. They haven't focused on personal utility, but they're sort of thinking about incorporating that into the model. Similarly to our other efforts, there is sort of a classification matrix. And these are the elements, severity of disease, likelihood or penetrance, the effectiveness of the intervention, and the nature of the intervention. So if you think about surveillance versus mastectomy, those have different levels of risk. This is just an example of some of the work that this action ability group has produced. So this is looking at genes, ATM and CHEC2 for breast cancer. And they apply this framework based on the intervention. So they look at it both for surveillance as the action and chemo prevention. And then you get a sense of what the scores look like. And you can view the detailed report. There's been a nice trend in, steady trend in growth in producing the action ability reports. And we started a pediatric working group in phase two of the program. And another area to emphasize impact is that the process and the results from the ClinGen Action Ability Working Group are integrated into the ACMG Secondary Findings Working Group. So all the evidence review and the results are passed on to the ACMG Secondary Findings Working Group. And they're using that as the base to add additional genes to the ACMG59. I wanted to say a quick word about how we're engaging patients. So ClinGen has a patient registry called Genome Connect where patients can go through a consent process. They often learn about Genome Connect through the physician or the testing laboratory. And they can enter health information through this portal and also upload their genetic test report. Those test reports are then curated. And if the patient consent shared with ClinVar, so to date, we've got about 1,800 participants enrolled, 805 ClinVar submissions. And it's interesting to note that almost 50% of the submissions to ClinVar, there was no previous record. So the patients are adding quite a bit of value to the ClinVar database. We do have an ancestry and diversity working group led by Alice Pope, Joy, and Carlos Bustamante. This also got off the ground in the second phase of the program. They're exploring how race, ethnicity, and ancestry are used in clinical genomics. And we have a survey jointly with the CSER program surveying the clinical lab directors, geneticists, genetic counselors, and researchers involved in varying interpretation and variation to get a sense of how they're collecting or if they're collecting information on race, ethnicity, and ancestry and then how they're using it for varying interpretation with the hope to put out some guidelines to the community. By the end of this phase of the program, we have an education and training working group that spends a lot of time getting this material out to the community. There's a maintenance of certification model that they built, which is available through the American Board of Medical Genetics and Genomics. And it's basically a module learning how to interpret variants in ClinVar and resolve discrepancies between assertions in ClinVar. And if you go through that sort of training exercise, you can receive MUC credits. One thing we've done to leverage the community that's a ton of interest in ClinGen is launched a community curation committee where if people are interested in participating in ClinGen, they can go through a survey. There's a lot of training that's available online. They kind of get assigned to a curation group of interest with oversight, and then they are starting to participate in the curation process. ClinGen has a YouTube channel. So there's a wealth of information here. Some of those videos have over 1,000 hits, so we know people are looking at the training material. And then we've got a number of other ways where people can get involved in ClinGen. We have a monthly consortium call, which usually gets over 150 people attending to learn about the latest updates in ClinGen. So when winding down, this is, I think, my second to last slide, if you think back to the faces of ClinGen, there was 48 people on that slide from the three grants that we fund. But through all these expert panels that we stood up, we are now at 850 researchers and clinicians from 27 countries just participating in various ClinGen working groups. So I feel like we've achieved our mission to make this an international community curation effort, and it continues to grow. There's new people being added every day. And I feel like we've elevated the community. They're thinking very carefully about what's the best approach to use for doing curation, for adding genes to testing panels, thinking carefully about data models, data annotation, et cetera. And then what's coming the remaining 2 and 1 half years of this second phase of ClinGen? I didn't say much about our CNV work, but ClinGen is partnering with ACMG to update the CNV interpretation guidelines where they will likely move towards a model very similar to what we use for sequence variant interpretation, so adopting the pathogenic VUS benign categorization. That work is still underway. I think it's going to go in front of the ACMG board soon, and then out for public comment. We have two variation in function disease, variation function and disease supplement. So the ClinGen investigators are thinking about how do we structure all this data that's coming from functional assays and get it ready to pull into our curation environment? We have a complex disease working group led by Carlos Bustamante, where we're evaluating approaches for measuring clinical validity and actionability frameworks for polygenic risk scores, sort of taking what we've learned from ClinGen already and applying it to PRS. The Bustamante group has some efforts in machine learning methods to improve our curation efficiency, and then Sharon and Jonathan and a few other, the PIs have been working hard to stand up new disease curation groups and looking at hemostasis and thrombosis, ophthalmology, neuromuscular and a few others. And my last slide is just to announce that ClinGen holds a meeting with decipher every year called curating the clinical genome, and this next meeting is coming up this May in Washington, D.C. So Sharon, I'm gonna turn it over to you for a few reflections, and then we can take questions if there's time. Great, and I'll just be a minute or two because I wanna allow questions. So first of all, I wanted to acknowledge Erin's incredible work as a leader of the consortium and our facilitator, whatever the right role is. She's been amazing with a really wide-ranging group. And I also just wanna acknowledge the work with NCBI, Melissa Landrum, and the staff at ClinVar really get on a lot of our working group calls and really try to make our tools interact appropriately. I do think one of the things that we evolved was trying to develop structured frameworks for how to do things. At the same time, we developed web-based tools to implement those frameworks. Has actually been a pretty successful model because people read papers, but then they wanna try it so then you can go online and get a login to the gene curation interface or the variant curation interface. And I think, especially moving forward, we're now having online training sessions for how to use those tools has been actually a pretty effective way to engage, not just our own working groups, but to get people to join us. And then I do, we just wanna emphasize something Erin said. She talked about the clinical laboratories, many of them being commercial entities submitting to ClinVar. Not only do they submit to ClinVar, they're extremely active members of our working groups. It's really been amazing to see the uptake by people that work full-time in diagnostic laboratories and academic laboratories and clinical centers all working together on these working groups and sharing data. So I always give this example from CDH-1 that when they were trying to analyze a troublesome variant, among the members of one working group, they had 850,000 tests on that one single gene that they were able to compare data across to come to a conclusion. And then I think Erin's already talked about some of the things we wanna do moving forward. I would say that our tools are now being used by their labs. I do wanna say that the allele registry comes from Alex Molosevic's lab at Baylor, not from my lab directly, but that's an example where now the Jackson Laboratories is looking at using that framework for mouse variants. So I think we are getting wider distribution of our tools. And with that, I'm happy to take any questions or comments. Yeah. Thanks, that was a great overview. So one question I had in relating to future directions was what's your thought on the status of functional testing of experiments in saturation genome editing? Do you think that that's ready for primetime to incorporate into this sort of database? Yeah, that's actually part of our functional supplement. We've been working closely with the group of the University of Washington, for example, as well as a group doing that for the LDLR gene. I think it's ready for primetime, but not in the way that I think people always think, which is we view it as one set of evidence. We don't feel like it alone has to be able to curate the variant. And I think that's where there's a difference. If you look at the ACMG framework, we're combining functional data with a variety of other data types. We don't require that the functional data alone be able to predict whether a variant's pathogenic or benign. And I think that may be where sometimes the errors occur when you're saying, I just want this one CRISPR-based assay to do it all. You're very likely to miss variants that are pathogenic for reasons totally unrelated to whatever function you happen to be testing, mislocalization of the cell or something like that. So we are working hard to be able to upload that data more easily into the variant curation interface. That's what part of our supplements designed to do so that we could pull bodies of data. It is actually something that variant curation expert panel spent a lot of time on. They're literally, so for example, for BRCA-1, they're 24 different published functional assays. And so the curation panel has to decide which of those assays do they really think there's good enough evidence that they actually are predictive of disease. They may be very good for that biochemical function, but that may or may not predict disease. You mentioned that ClinVar's trying to work towards being more quantitative. What do you think the timeline is for that? I mean, in an ideal world, it would be that this is a probability of pathogenicity with a confidence interval and it would be very transparent and really easy for clinicians and patients to understand. Yes, that is the ideal world. And the hereditary cancer world in some ways is closest to that because of the amount of data. And that paper, Sean Tavagene's done a lot of work on making predictions for BRCA-1 and 2. ACMG, so I should say that ClinVar is an aggregator of data. The ACMG guidelines right now are not, quantitative, are not calculated that way, although part of the point of the paper was to show if you actually look at the relative likelihood of pathogenicity of supporting versus moderate versus strong, it actually is pretty close. And so one of the things we're trying to do is they're members of ClinVar and ACMG does have a committee that is now looking to update those guidelines. How far they'll get towards a really statistical approach, I think it's hard to say because there's such variation in the amount of data. If you look at genes like CFTR or BRCA-1, there's just a huge amount of data and you can really do it. If you look at rare recessive syndromes, whether you could ever get enough data to say this is a 95% and this is a 56% is, I think, harder. So that's how we use these more qualitative assessments. I actually think you just answered my one question I had which now that I've heard Wendy's, I realized was way too simplistic to ask. My second question, though, is I'm very curious about how patients are getting their data in, well, I also wanted to say, fantastic presentation. I'd like your seven-year-old to come to my house before I have to give a job. I think that'll go well for both of us or all three of us, but I'm very curious about how does that even work? That was very interesting, but I'm curious. It's been a very interesting process and it's not one that I've worked directly on, so I'm talking about other people's work. So Genome Connect is an IRB-approved registry where patients literally can, if they have like a hard copy of their test report, can upload the PDF. There's a team that actually then interprets that variant. They look it up in ClinVar and the patients are essentially giving permission to upload not only their variant, but their phenotype. So it's an access to better phenotyping. It's led to some very interesting issues, which is that there are people submitting test reports where that variant's already been reclassified in ClinVar. So we had to then have a whole discussion about how do we handle that. So we now let the lab know that their patient has submitted and sometimes that own laboratory has updated it. Sometimes another lab has updated it. So we do let the testing lab know that we're gonna be contacting the patient back and letting them know there's an updated interpretation of that variant in ClinVar that they may wanna talk with their physician about. So it highlights the reclassification problem. I will say that we've had a large number of participants, but maybe not as many as we might have hoped, and we are now targeting also other disease support groups, particularly those that have registries that use the same platform, because we can provide the genetics piece that a lot of them don't even collect. They collect phenotype or medical information, but not genetic information. So we're trying to work with them. So in fact, I was talking with the CDH1 group actually as a registry, but uses this platform but doesn't collect genetic results and they might be interested in that kind of collaboration. I had a related, I mean, hearing your explanation of the last question, but then also hearing some of the statistics, which I wasn't familiar to the immense number of people that are now sort of within your network, country-wise, investigators, and so forth. I'm sort of trying to imagine how this has scaled as well as it has. And is it on the back? Was it out of control? Yeah, no, I mean, is it, is it, and it's, and there's- Scale is a question that comes up. Right, but it must partially relate to NHGRI staff and NCBI staff, I mean, sort of program staff. It must be the grantees. It must be also NCBI staff. Is this on everybody's back in some way? Because just even managing and adjudicating and coordinating, is it just everybody doing it? And is it scaling? Well, it just seems that it can't possibly be scaling linearly, so how are you doing this? Right, well, so one of the main goals of the second phase was scale. And so we're actually, we're really at the point now where we've got tools where you can go from a variant to the interface to the website, all working. Now what we're trying to do, oh, there you go, thank you. Now what we're trying to do is really build tools that will allow us to scale even better. So we're, we've been building a variant prioritization tool which allows you to pull like every variant of that gene and exact. It'll tell you what its population frequency is, what is it called in ClinVar. You can do all these filters. So then the committee can say, okay, here's 200 loss of function variants that are rare in Nomad that we now want to curate as a bulk because that's what it's really going to take. So right now it is more manual and we're really, to get approved as a panel, you only have to take about a hundred variants. What we have you do is take a hundred variants of range from benign to dispute, to uncertain to pathogenic, and show that you can apply the specifications and to classify those variants. And then it's really a question of now, how do you scale that to all the other variants we know about that gene? But I understood the concept of needing this phase, needing to scale in terms of dealing with the tsunami of variants. But I was actually asking about it as the tsunami of different people around the world you're interacting with. Because it's one thing to get a bazillion variants from two or three labs, another thing to get a bazillion variants from hundreds of different people. So that's a lot of emails to answer and a lot of that right now. Yeah, I think we have the social interaction. Yeah, I mean, we have an exceptional set of coordinators, a core group of coordinators. We have, these are funded, those are the grantees, yeah. And so, sort of that exceptional group of people, they've been very smart about the training materials that they have developed by setting up, for example, a bio curation working group. So as all the curators now from around the country, they wanna get up to speed, make sure they understand what the changes that have been made to our curation ecosystem, there's just a very streamlined way for them to get that information and it to be distributed across the network. That doesn't mean it's perfect, but I think it was just very smart choices as we were setting up the various elements of the program that is making at least the social part more scalable than you might have thought otherwise. It's a lot of conference calls. But I would say the coordinators are a key group. So what's the vocabulary used for phenotypes? Are you using an ontology for that? We generally try to use HPO and we will try to create HPO terms if they don't exist. We also have a working with Monarch. So we try pretty hard, it does get difficult as you get off to these highly specialized disease areas. But we've been doing a lot more work with the Monarch Initiative as well, trying to use consistent terms. And we also work with OMIM as well. Can I say something? Yeah, yeah, please. And then we have a group affectionately called the lumping and splitting working group. So for the expert panels and the curators to make choices about whether or not you're gonna roll up a couple of phenotypes and curate at the sort of macro level or you have to do a curation, like in epilepsy's example, right, where you have to really parse out the different phenotypes and curate that particular subset of epilepsy. So we're trying to develop sort of structured approaches for that as well. So it's not subjective choices, but people are following a set of roles to decide how to curate these complicated syndromes. Other questions? Okay, Erin and Sharon, thank you very much.