 Okay, so this is our afternoon panel. It's a chance to follow up on things that have arisen out of today's discussion. And I thought maybe we would start just with a follow up at the sort of end of the last session. There was a lot of interest in the question of how could the physician slash healthcare system be mobilized to help provide useful phenotypes that might be of value for both clinically but also for thinking about expanding the quality of the phenotypes that we have in the electronic health record. So there seemed like there was a lot of people that had thoughts about that. But I don't know, Peter, that's sort of squarely in your space, do you want to have a go at it? So I can, okay, yeah, there's a lot more to say than I could in two minutes. I'd like to maybe pick up something that Melissa presented with the phenopackets. And one thing that has been a terrible difficulty in our sphere is actually getting data from publications into databases. And one of our goals with the phenopackets project is basically to provide an environment such that journals can say, if you want to publish with us, we want you to represent the phenotype and the genotype of your patients as a phenopacket and send it to a database, which could be ClinVar, ClinSec could be the Monarch database. It could be Boutique databases if we're dealing with data that have more to do with the environment. Let's say another beauty about this format is that it can also be used to improve lab diagnostics. And I'm not entirely sure how things work here in the states, but in Germany, we generally get some handwritten one word indication that doesn't really help that much to narrow down the findings. And sometimes this even includes exome data where it's kind of ridiculous. And essentially, we know how software that can automatically extract text from an electronic healthcare record, put it into a phenopacket and send it to a lab. So we don't have a finished product, but we have all of the components that could be easily developed into that. And it seems like there's a lot of things like that that could massively increase the amount of data that's being collected both from the healthcare process and the publishing process with phenotypic data. So that maybe I'll stop there, but that I think is my most important thing I want to say. Are there any panel want to add to that, Carl? So I would second everything Peter said. I would add only a couple of things. One that I believe in order to really get to the next stage, we need to begin to make probabilistic inferences as has been said by many, including last year in the course of the meeting. And in order to do that, I believe we need much more quantitative data sets. And I think it's clear we're never going to be able to do that retrospectively. We need to begin to think about how we organize the data in a quantitative fashion, standardize it across all clinical collections. Because to be honest, it's useful not just for genomicists, but actually for all of medicine. The signal noise is increasingly something that people are noting in trying to develop the learning health record. We talked about that during the course of the morning. If we can improve that in a systematic way, it affects every project in biomedical science. And then the one other thing that I think came out during the course of the last two days, and I think Nancy sort of crystallized it quite nicely at the end, is thinking about some biological axes that could act as assay links between the clinical space and all the basic science space. Something that would systematically tie that together in addition to ontologies, in addition to existing phenotypes to allow us to just make more granular and rich the relationships between all of the data sets that we have access to. So there's really no reason why you cannot use ontologies to capture quantitative data. So you could easily add this to the HPO or to many other ontologies. And so I agree entirely. Another thing that I think is currently missing in the data landscape is common disease. So we and many other groups now have computational models of rare disease. In our case, these are basically lists of HPO terms with minimal metadata. This is not going to work at all for common disease. And I think as we move into projects such as the PMI, I mean, we'll just imagine you have a patient with sight problems and high blood sugar. Well, if both of these things occurred at the, you know, they were diagnosed at the age of two or three years, maybe it's part of Betel syndrome. But if the eye problems came 20 years after the sugar problem, well, maybe it's diabetes, our algorithms now wouldn't know how to deal with that. And one of the things we're now engaging on is developing algorithms that would essentially provide you something like let's say a Bayesian prior to interpret data. And I think that's something which is going to be hopefully valuable. And I think a corollary of that that I didn't mention is, we've heard the tension throughout the last two days between broad and deep. And I think we tend to let things go from deep to broad randomly. And what we need to do is to make active choices about what are the phenotypes that go from deep to broad and doing it in a way exactly as you outlined, Peter. Erin? Yeah, I just wanted to follow up. I love all these ideas. And some of them are a bit more sophisticated and will take a little bit more effort to implement. But one thing I was thinking, you know, when the clinician is submitting their form to the lab, I mean, what can we do just to capture a few additional phenotypes? So the basic, you know, the reason that for the testing, a few conditions that we know, we can either have standard HPO terms, et cetera. I mean, this would, I mean, paper or their electronic submission. Not paper. I know, but I'm just trying to think, you know, there's a lot of groups out there. So there's solutions there. But then once the lab has that, there's dozens, hundreds, Melissa might know, of groups submitting to ClinVar. And they're submitting the variant, their assertion, and their space for them to share some phenotypic information. So if they already have that, then when they submit to ClinVar, they could submit that data right back to ClinVar. Okay, so Wendy, Bruce, and then Liz. Well, to add to your point, I would take it another step farther, which is when an assertion is made by the lab given looking at everything, also in ClinVar and everything in the world they can find, they also tell the clinician based on my calling this gene with a pathogenic variant, you should expect to see, you know, these clinical findings in the phenotype. Please go back, check it. And that's, I think, what Liz has been calling, what is it, iterative or deeper phenotyping and so forth. And it's really just this feedback loop. I mean, I think that we, you know, if you look at what's in ClinVar, so I did a view, there's been a lot of activity because of by the ClinGen labs to try to harmonize what they say because it's, you know, a bit of an embarrassment to say completely different things if you're, and they can get so far when they're all looking at the same data and they share what criteria there are. But, you know, the conflicts in ClinVar, there are 17,000. And then if you start to think about them in terms of their importance and rank, there's a rank difference. And so, since I don't have a slide, there's my rank difference of five variants. I knew I'd have to do some hand waving without slides. And, you know, one pathogenic to very likely, not so important, that's a rank one. A rank two is sort of what Bruce was talking about when he undiagnosed a patient from, you know, pathogenic and VUS. And there's about 1% of those. There's about 2%, I'm sorry, 1% of the four rank, 2% of the three rank, and then all together about a quarter of those. And those are tractable if you're really talking about disorders that are very well-defined. So I'm not really talking about discovery and something that's in an exome and, you know, you're trying to decide about it. These are things that are attached to OMIM terms, HPO phenotypes, you know, you can go on MedGen and find a nice list of them. And all we really need are tools and incentivized clinicians. And not just, I believe, the geneticists, but what Cricket and others were talking about was really bringing everybody in, you know, to this, because it's, you know, it's all the patients out there. Bruce. So we're here to talk about the, sort of the clinical laboratory research gap, but it's, I think clear to me that there's also a clinical informatics gap. Very few clinicians I would predict could even define ontology or tell you what it even vaguely means. Nevermind would they know how to use one. And even in conferences I've been at where people have talked about these, they make it seem like, oh sure, it's easy. Just go on and you can use this and it's not easy. There are half a dozen different places you could go and it's not anywhere close to user friendly to a complete novice, which is what virtually all clinicians are. And I guess to follow on a point Mark made, I think the Holy Grail here is to create a system that is easy to use that does not require any informatic sophistication and is only done once and serves all purposes, meaning whether it's to feed information to the lab to which you're sending a test or to document information for the medical record or to share genomic information later when you either did or didn't find something. If this is made simple and natural it will be used and if we use the word ontology to physicians it won't be. I was just gonna echo what Bruce had said. We go one step further. I think it involves physicians at all. It will not happen at the scale that we require it to occur. It has to be ambient and automated. It has to be built into the platform we use for healthcare. It cannot be dependent on data entry from a provider. Before we go to Liz, so one of the things that we've done at our place is, I think many of you are familiar with the Promise Quality of Life Assessment Tool. That was actually developed at Northwestern. So we've now implemented that Promise Quality of Life Assessment Tool through the Epic Patient Portal. And in particular for things like, our first project was for people that are having joint replacements to be able to go in and provide regular feedback to the physician about quality of life, whether they're having problems in a very structured kind of way. One could imagine that you could roll out something very similar to that for somebody that is addressing a specific complaint that wants to come in to see their physician. And the beauty of that approach is that it's the participant providing the data. We can have probably a long debate about what the quality of the data that the participant provides or the patient provides might be, but actually the physician usually listens to them at some point anyway. So that seems like one interesting place to think about starting. So, Liz, I think you were next. So following on that topic, when the question is what sort of data would you want to have collected from that patient to get a deeper phenotype? Well, we can take some lessons from what a medical geneticist does. You could do a system by system review. That's been refined over decades, I think would be a good place to start. And then another point I was going to make was also to look into whether we can have patient provide some of that information before they get into the appointment, because that would help. It's not without problems, obviously, but it would be a savings and time for the practitioner when they see the patient. I think one interesting challenge for me thinking about that, since we've sort of got a questionnaire in processes, how could we even go the next step of actually thinking about tailoring that for the particular reason the patient wants to come in and see their physician. That seems to me like a little bit of a daunting informatics problem, but would be worth thinking about. So. Well, you asked to make a brief reflection of what we hear, and this is connected with the phenotypes. And one word that I didn't hear, and maybe because of the title of the conference, which is bent side to bedside, rather than bent side to, for example, dining table or healthy aging, is the word prevention. And that's something that didn't come out. And in terms of the phenotypes that are the, what we said before, the exposome that we should be collecting, this is what is my area of interest, is the nutritional data. So I think collecting some type of nutritional habits is important also as part of the record. And this is not trivial because whereas there is effort being placed in the pharmacogenomics, the nutritional component is also important. For example, one of our studies showed that the relevance of that, that was randomized, clinical, dietary intervention study. It was 7,000 people, it was five years. People were free of cardiovascular diseases at entry, but based on our results that we published, during those five years, if the people that have the specific variant of a gene were receiving the proper diet for that gene, you were saving 44 cases of stroke. And this is probably similar to what you find with some drugs. So if you extrapolate that to the strokes in the States, which is per year about 800,000, in theory you could reduce 10% of those strokes just by matching the nutritional habit of an individual with a specific genotype. So it's worth to consider that as part of the, of that exposome, that integration of omics and nutrition granted that we have a lot of problems generating objective and tight information. But after listening for a couple of days here, the problems with just collecting the medical information in general, I don't think we are too much behind in terms of the problem. So I think we should consider. I have tried to convince Queen Erin to incorporate that as part of the databases, but we'll see if we are able to do it. Barbara. I just had a question about that. In terms of self-reported nutritional information, how accurate in your experience is that? That has been probably the major Achilles heel you know that we have, the self-reported. For example, Phoenix, which is also part of what Erin is doing, came out with a toolkit in order to minimize the subjectivity of the data collection. But we are not happy with that. I mean, considering the amount of technology that we have available now in terms of smartphones and so on so far. And everybody taking pictures of everything and putting it in the social media, you have to harness that information, could really facilitate that, ameliorate the problem that we recognize we have. So Erin, I think you were next on the list, and then Mark, I guess, I'm sorry, Queen Erin. Microphone. I can't remember my question now. Distracted by my new role, yeah. Mark, I was next. Yeah, two comments. One is, I think we frequently, and this relates specifically to the accuracy of self-reported nutrition, is that we frequently, again, I think this is what Les was talking about was the illusion of nirvana. We say, well, it's bad information and therefore it's worse than no information. And I would argue that it probably is better than no information. And the same goes with family history where the pushback has been, well, we don't know whether this is accurate or not, or I have this anecdote about how inaccurate it is, but the reality is that for many disorders, it's pretty darn good. And so I would say that having the information, even in imperfect form, in a way that you could actually study, is much better than not having it. The second point is to get back to the role of the patient participant in healthcare systems and their willingness to contribute. So Rex had mentioned the integration of the promise. We've done work with our rheumatoid arthritis patients that shows that they're willing to come in on every visit and fill out standardized questionnaires related pain activity and that that can be combined with electronic health record and pharmacy data to give a heads up display to both the clinician and the patient about disease activity over time based on patient reported outcomes, which has high value to both the patient and the provider. And so they're incented to contribute those types of data. The next step that we're looking at is how can we scale this across all encounters? And one of the really interesting pilots that we're beginning to do is the idea of expanding the open note concept, which is one where patients can go in, review all their medical records, make comments to the providers about what they see there to letting the patient start the note. That they start the visit before the visit with this is why I wanna come to see the doctor. These are the specific things that I'm doing. And if we can in fact find engagement from patients to do this. And then the patient and the doctor co-write the note and co-sign the note. You could imagine on the front end of that based on chief complaint and historical information that you could push specific questionnaires to them around phenotypes and that related to the complaint which the patients are highly likely to fill out because it's directly related to why they wanna see the doctor in the first place. And so I think innovative approaches to how we time collecting the information is also worth exploration. Was it a follow up on Mark's comments or? Yes, just to comment that this is not without precedent to allow patients to report in certain instances. There's data in the cancer field that patients reporting their drug toxicity from cancer drugs is much more comprehensive and accurate than the physician reporting it because the physician seeing them once every three weeks or once every month and they're not knowing all of what's happening in between. We use the patient reported phenotype in one of our recent pharmacogenetic studies. So I think you're gonna like what I'm gonna say. We just finished translating the HPO into lay person ease. So now patients can actually use the same informatics infrastructure that we've been using for the clinical data. And the nice thing about that is it comes back to that contextual data interrogation idea as well. So as the patient's phenotyping, we can use those phenotype profile comparisons to help inform, do you also have this phenotype? Do you also have this phenotype? And actually do exactly what you just said and auto generate what those next questions and the questionnaires should look like. And we actually were going to apply this approach to the genome connect survey to improve the survey and have a PCORI proposal in right now to do exactly that so. So do you understand correctly then what that tool would allow you to do is present a couple of starting questions and then adaptively as you proceed based on the results of those answers produce the follow on questions so that they wouldn't have to know anything about the ontology that was underneath it. They could just answer those kinds of questions. Just along those lines, so we've also had collaborations with groups around the world in the GA4GH framework and we have complete translations of the HBO into Spanish, Italian, a lot of Portuguese, a lot of Japanese and the Chinese, the BGI and various other groups are now translating into Chinese. This is important because for instance the Japanese rare disease national effort is going to be using Japanese HBO but since it's all a code you can now really massively extend the amount of data that you have for matching. Aaron. Well this ties into what Melissa was saying about the collaboration with Genome Connect because they have developed a health system survey so when the patient registers in Genome Connect they are presented with sort of a body system survey that's relatively straightforward and it captures all the different systems and then from there and so it's great if we can automate and tie that back to the HBO and then they're developing sort of more surveys that would give a deeper dive into whatever particular organ system or issue that they're having so it is sort of that iterative process of getting deeper information and they seem very willing to do that deep dive and then my last comment is I think there's, you know, the directed consumer companies like 23andMe they've had quite a bit of success with pushing out those sort of short surveys to their consumers and they seem very willing to do that if they do it at the right interval without asking for too much information at any one time. I was gonna just add along the lines of Aaron's comment that it might also be worth thinking about propagating those questions based not just on existing rare diseases but also on biological processes that are known to be affected so you could actually have orthogonal axes of secondary questions some of which are based on traditional syndromic clustering and some of which are based on fundamental biology. And those might align nicely with the kind of axes that we heard about this morning from Nancy, yeah. All right, I wanna then shift to another topic we've discussed quite a bit and see if we can get to some conclusions. One of the things that seemed apparent to me at least through our discussions especially this morning is that there's still a disconnect between what fantastic tools and ontologies and databases have been developed on the basic side and the fact that we think most clinicians probably don't really know anything about them. So I think it'd be useful to have some discussion about what can we do to fix that? What, you know, is it as simple as an HGRI putting up a centralized webpage that points everybody to something or is there something more sophisticated? I know Marissa will have something to say about that but Wendy? So I'm going to maybe take a little umbridge and this is gonna be dangerous at what something Callum said which is the idea that clinicians can't enter in, well maybe I'm speaking, they just won't. They don't have time to do that. Okay, okay. I spent two years trying to get a minimal clinical data set in an enormous implementation at the Harvard hospitals and the final reason for not doing it was there is no time and energy. I don't disagree with that at all. Okay, but I don't think that's the end of it. Okay, I think that it means going to clinicians and saying what matters to you? Does it matter to you if this is part of your workflow for ordering a genetic test? And the payers say this is one of the points that you have to do and there'll be an insurance product that is a automatic yes for your patient and you don't have to go asking around and making sure that it's authorized and you can use it in real time. Okay, so that's one thing. I also believe that clinicians can enter this kind of information because they do for example for ICD-10 codes, okay? And so about three weeks ago I visited an organization called Intelligent Medical Objects, IMO, my new mark is aware of them. They work closely with Epic and lots of EHRs and they have figured out the coding systems but they haven't yet gotten interested in the granular genetic data that we care about. But they are able to do things like embed in what looks like Epic to the clinician just clicking for things and then they get to the code that they need. So it just, you know, don't call it an ontology, you know, just, you know, make it part of their flow. I could not agree more, I think it would be perfect where that feasible but I can give you one concrete example. So there is a seven point ICD classifier for diabetes that was introduced in Epic at Brigham and Women's in July. The incidence of diabetes has fallen by 15% because people basically will not fill it out. It's better not to put it in the record than it is to click eight things to get to the next step. So it's not gonna be easy. Okay, Melissa, and then go ahead. Okay, I have a couple of things. First is that even though I use the O word here, usually as an ontologist, we, if we're doing our jobs right, no one ever knows we were there. We're like the little blue men in the Twilight Zone, okay? So, you know, so I think it's really important. So one of the things we talked about at the linking animal models to diseases workshop that Rex was at and a few of you were at last fall was that, you know, an idea of embedding a data science informaticist person into the clinic as we learn to build these tools. There's a deep need to build innovative tools that are easy and fast. We've gotten some start on a few of them. Our group has one, but there are many others to, you know, do text mining, you know, many different types of algorithms. But what we don't have is we don't have the high quality innovative user interfaces that make it easy and also have that feedback that shows the clinician that they, you know, are doing something useful. Otherwise it feels like menial tasks, right? So the one thing that happened in the early part of the UDP that was so amazing was that we, you know, they asked us, well, how are we gonna know if we've done enough phenotyping? And we said, well, actually, we don't know the answer to that. And so we did a bunch of informatics and then developed a rubric that essentially queries our system and does that phenotype comparison. And then we implemented a five-star system that's just, it says scale of zero to one that says that, you know, this is the meaning of the phenotypes that you're giving us. And as soon as we implemented that, our phenotyping went way up. I mean, it was amazing what just a simple five-star system could do because all of a sudden people were knowing that they were doing something useful that was gonna maybe help with the analysis. Now, there are special population of clinicians that are very dedicated, but even before then we were getting not very good data. So, you know, think about if we could innovate on something that take that five-star system and actually have quality, get the gamers involved, you know, that kind of thing, that's what we need to do. Yeah, social media has been very effective of that. Just give everybody a badge and they will fill stuff out more, so go ahead. Hi, I'm Karen from Cell. And I was wondering if there was a way to sort of, this is pertaining to Rex's question he had just posed before, to systematize the kind of encounter that Monty spoke about at the poster at the meeting and, you know, what Doug said, he kind of learned here being at the meeting with, you know, almost with the social media idea, like a matchmaking system in some way, like I'm envisioning either a meeting or an online system where, you know, there's clinicians who have a set of problems and they enter to say, I wanna find a match with a basic scientist and there's some level of curation where it's like, okay, this is a widespread problem, you know, it's not just this one physician who wants this and then on the other side, it's these basic researchers who really want to use their systems that they've built to contribute to a translatable clinical problem and there's some level of evaluation curation. They have a robust pipeline built up and then they're sort of, I can do this phenotype, I want this phenotype and so forth and you kind of, I mean, ideally it would even be incentivized a little bit with a small amount of seed money for doing that and that would be a way to sort of on important problems, a curated set of important problems bring together these two communities that otherwise don't, it seems, have a really effective way to interact other than random posters and so forth. Well, what would be, what would drive the clinicians to that site? What would be the incentive, what would they get back from that? Excuse me, a physician would be better poised to answer that for me. I mean, it sounded at least from Monty's story that that physician was very eager to be able to see if their variant had a phenotype in the zebrafish and I can imagine that there are physicians who are sitting on a bunch of the U.S.'s who would like to, it's, yeah, I mean, it depends on the attitude. If it's, you know, everyone sitting in Howard's conference who said, I don't care what it looks like, then there probably is no incentive but if there is a different attitude that I could really get some functional information that would help me decide on treatment, it's actionable, treatable, whatever the, and common, then maybe there is the incentive. I think that's an excellent idea. We've done this in two settings, one in the undiagnosed diseases network, that's been something that has happened and certainly Monty and Hugo Ballin who run the Model Organism Corps have been very receptive to having variants sent to them by the clinicians and then we've crowd sourced internally. It's difficult because of HIPAA to do it more broadly because if you have more than a certain number of variants, you can almost immediately identify the individual but you can do that within the Harvard system to try and match patients and their variants with a basic scientist who has an interest in that space. Yeah, I mean, I think part of it is what is the motivation for your daily work and the German setting, we can do exomes on kids who have failed basically everything, everything, everything else and we did an initial study where we got about 30 and now 35% of the diagnoses in known genes and the clinicians first, they said this is like pulling teeth, I hate this but after about a couple of weeks, they kind of, they started talking about HBO terms and it was all fine and I think as we move out to precision medicine, right now there are relatively few areas where there's an obvious utility for your daily work and but I think we should start to emphasize those areas where today there is something, maybe pharmacogenomics, a few indications and just build out from that because once clinicians see this is useful, they'll do it and if it's still research like, you know, I'm sorry but if it's, let's look at a zebrafish, I don't really see this happening that often given the time pressures that clinicians have. Wendy? You know, just also building the case for what's in it for me as a clinician, these terms, HBO terms and so forth, there's software that's been developed by up to date where you put in a few terms and it makes a note for you and of course everybody's gonna love that. The only problem is that we want terms that match to IDs, okay, and concepts and are computable so but it tells you that's another place where it would be helpful for the clinician and I think it's very doable. I'd just extend that one step further again. I think what we want are concepts that are biological and that's the fundamental problem with a lot of the physician driven strategies for documentation is they're not actually based on biology, they're based often on therapy or reimbursement and so reorienting the entire ship in the direction of biology is actually a major undertaking and one of the reasons why I think it will be intrinsically difficult but I do agree with you. I think finding ways in which it becomes obvious that that actually has tremendous added value are the steps that we have to take. I think one of the issues is that every place probably has their own reasons why this might not work or different issues. We would like to get phenotypic terms or an ontology into Epic and we were told well that's a three to five year wait. I do agree that unless the clinician has some, I don't know, incentive but if you can't close your chart without filling this out they'll do it but other than that, you know, I think every place is different. At our place, if they have to do more work to get a genetic test then everyone gets referred to genetics, now it's only about 70% and I think that it's, they're pushing patients, most of the other sub-specialties and that unless they have to do it they're probably not going to. Now if you just want the geneticists then maybe CME and doing eventually getting the terms in there we might do it and I hope one of the reasons this whole education thing with Cricket is to get the other people excited. If you teach them why it helps then maybe they will be willing more but I think there's no one fix for all because everybody's got different issues I think. I'll just comment that I think it's not necessarily as bad as a three to five year wait. Actually we in the context of Emerge and the Emerge Pharmacogenomics Project did a collaboration with IMO where we actually were able to put in pharmacogenomic terms that IMO and be able to push out to anybody that uses Epic and has an IMO contract or service so I think there is hope for doing some of that. That was our institution comment so I don't know, you know, there's getting it into Epic and then your institution so hopefully there could be some leeway there but it wasn't going to be a quick fix at least when we talk to them. So I have Bruce next but if this is a response to this. Yeah I think that what you're pointing out Rex is that it helps to know people in high places and so I think one of the things as we discuss facilitation of this issue is to identify who are the people that hold control of certain standards that if we can get it involved in their thing it will just go into the electronic health records without us having to deal with a certain or an Epic. IMO is a great example of that. Loink for laboratory codes is the CPIC group has when we developed our standardized nomenclature we immediately went to Loink and had them create Loink codes that could then immediately be used and those are already compatible with HL7 and all the other sort of stuff. And so if we can identify the key points of contact in those standard organizations like the HL7 Genomics Working Group, et cetera as we come up with ideas we can in some ways engage them. And I think this is where Digitize has had some success is by having those folks around the table to really lower the barriers to trying to get these things moving forward because we can kind of end run what is otherwise a huge bureaucracy. I realize there's a tension between trying to make it possible for any clinician to order genetic tests but then especially if they're not well trained in this area there'll be a limited tolerance for putting in data. On the other side having medical geneticists do it they probably would be motivated if they were made reasonably easy. There is a middle ground actually and that is exemplified at our institution our hospital became distressed as they started seeing the vast volume of neurogenetic tests ordered by the neurologists which was costing a lot of money and there was question actually as to whether the testing was always indicated. They volunteered well after some discussion they agreed to pay the salary of a genetic counselor who is on call and intermediates on all neurogenetic testing that the neurologists send. Does all the kind of leg work to make sure the test is appropriately ordered all the requisite information is provided all the billing is appropriately arranged. And that kind of thing actually has been a win-win for everybody. The neurologists are happy because somebody takes the time to do it well the counselors enjoy it. The hospital actually has seen a reduction in the volume of send-outs and therefore they feel they're saving money. And I would argue that the quality of care has gone up measurably. So I think there are ways of doing this so that it's widely available but still there's somebody who knows what they're doing serving as an intermediary. Okay. I'd say I think that's an excellent intermediate step. I think the difficulty is if you think about how we do that for a genome and the scale of data that we would need to collect in order to be able to annotate genomes that you know at enormous scale that's never gonna work. We're doing that all the time though and I don't find it to be that cumbersome actually. But how many people are you seeing a year in that program? Not a, it's not thousands if that's what you're asking me because I think that's the only problem I have with a lot of what we think about that is physician intermediated is that it will not happen on a scale that will work for example with Daniel's projects. It won't work on a scale that looks at even one gene-gene interaction and then we start to think about gene environment and the scale of data that we need is several orders of magnitude larger than every counselor on the planet could generate in their lifetime. That may be true now. I guess I would first argue that we're not at a point where we're able to pay for thousands of genomes, never mind do them. I think with the volumes that we currently are likely to have this is actually a pretty workable solution and could pave the way towards much larger scaling over at least the next few years. Alyssa? Just a couple more follow on points to that. So I think we've seen some successes like this one but also where it's either a clinical geneticist or an informaticist or data scientist that's coming into specialized clinics and really helping in these small circumstances and the partnerships around those things are really fantastic. It really is a deep collaboration and I really myself have enjoyed the experience that I've had. I think that we need to help more clinical geneticists understand more about data science so we can turn them into emissaries. So not all clinical geneticists are created alike in terms of what they know about the whole landscape if we're gonna bring the utility of the model data to the clinic that they need some help there so I think there's an opportunity there. And then it comes back to if we have these smaller pilot kinds of examples where we have these settings that we're integrating clinical geneticists and data scientists together into these settings that's where we can really innovate these tools that can be evolved into much larger scale settings. We have to learn from those little collaborations in order to build those tools. We don't know really what they should look like yet. So we heard just to maybe expand a little bit off of that we heard a lot this morning about the flip side to this which were amazing and yesterday as well. Amazing tools that are sort of genome wide that are giving us collections of variants and changes in proteins and how they're likely to function that will I think make a big difference in terms of our ability to annotate. But maybe you or Daniel might wanna talk about this but what could we do sort of from the clinician side to push things more that would help you better be able to fill out your systems and Nancy that might apply to your work as well in terms of pointing you towards specific variants like for example what percent of the genes are you gonna be able to cover and what's that gap gonna look like what percentage of the proteins whose variants you're gonna be able to analyze what's left and is there a role for the clinical side to help you fill out your collections so that they're more usable for everybody. Yeah I mean I think I think I, oops, uh oh. Tried to make the plea right that in terms of helping us understand or I mean the whole experimental community I guess understand like what's really needed in terms of like on what genes can we take action and which ones should we stay away from. I mean we, you know that because they're just not gonna be so useful that's really that's the most critical thing that could be provided. I mean we can pretty easily say okay these are the ones we can do reasonably and these are how many we can do so then the question is given those resources like how to best invest them right for maximum impact. Well so for example a curse on me like a list and maybe this is a crazy idea but a list of things that are likely to be embryonic lethals might be things you'd not wanna worry so much about VUS is on right I don't know but maybe if it was something where there was more likely to be a gene we didn't have any idea what its function was but didn't think it was likely to be an embryonic lethal would that make it higher on the list for a target? I mean I think there are definitely some genes where we think loss of function mutations are likely to be embryo lethal but that said if you found a VUS within that same gene it could be a hypomorphic allele for instance that could easily be the cause of some other you know less non embryonic lethal but still very important disease so those are definitely still worth going after. That said I think that that list of very highly loss of function depleted genes that I talked about yesterday there's two and a half thousand to 3,000 genes where it's clear that although we don't know what the phenotype is or exactly how severe that phenotype is we're quite confident that truncating mutations in those genes do cause some selected against phenotype. That seems like a set of genes that would be extremely useful to go after and again as I said 75% of those we actually have no idea what the clinical phenotype is in some cases it would be embryonic lethality that's a good estimate for some of them but for many of those genes that there clearly is some clinical or even maybe subclinical phenotype that we just haven't spotted yet and it would be great to be able to fully understand those genes. Well continuing on what you say of the phenotypes something that it was mentioned yesterday and could fill up some of the gap is something for especially for metabolic diseases something similar to what is using cardiology the stress test and we need to capture the gene like an smoking gun therefore maybe challenges, perturbations on the patients on the subjects like for example the OGTT has been used for a long time but maybe oral fat loads those are things that could inform much more than just the steady state in which we are capturing the phenotypes nowadays. So certainly I hope that we can inform on some of those genes from the gene to medical phenome catalog but one of the things that I'd say is that having spent some time with John Phillips and the guys at Vanderbilt who run the undiagnosed diseases center at Vanderbilt in the network and the Mendelian sequencing centers. I mean so part of what's motivating the development of some of these catalogs the catalog around the Mendelian genes where we try to bring forward more of the highly significant phenome associations we're seeing with Mendelian disease genes and then the set that are like Mendelian genes in waiting are motivated by what it seemed like they needed and so I think bringing some of the basic science together with at an undiagnosed disease network meeting or the clinical sequencing centers which they do at the local centers but not so much broadly across other universities that don't happen to have one of these centers in house and it's probably no big surprise that genomes. So we've got people from the University of Washington where they've been doing this and it bleeds over into the basic science and Harvard and the Broad that's been deep into this for a while and the but there are many basic scientists at many other places that are just that don't even have those level of benefits. So I think there could be real value in seeing having basic scientists see more what it is that some of the clinicians wish they had sort of from the get go in doing these complex patients. I think one thing that we would be very interested in thinking about would be as we start thinking more and more about following up individuals from AXAC who have interesting genotypes is starting to think what is the right approach for doing that re-contact phenotyping? How do we even begin to approach that problem? Because of course it's incredibly daunting to say we have a gene, it's uncharacterized, this person is a homozygous knockout. How do we learn more about its function? And I think there's a number of people in this room that could really help to define a phenotyping toolkit that is a minimal relatively low, doesn't require a huge amount of effort but gains a high amount of content. You can explore a large amount of the phenotypic space with the right questions and the right assays, defining those assays up front so that we can start going after particular systems or asking questions in series so that you can say do a quick high throughput screen of multiple systems first, find something that looks interesting and then go in and do a deeper dive and that particular system would be extremely useful. Because I think, and this will come up again in the context of PMI, any situation where we're doing genotype first approaches, we're gonna be faced with that very difficult challenge of knowing how to do that phenotyping. And as far as I know, no one's really come up with a clear plan for that yet. Daniel, can I just clarification? Are you asking, you wanna understand what needs to be done or you're asking for an infrastructure to be able to do that? Well, both would be great, but I think the first is the most dire need, is just, that's right, this is true for us at the moment. So we have many samples and EXAC, most of them actually outside the US in Europe and elsewhere where there are enthusiastic collaborators on the other side. We have clearly very interesting genotypes that show up in those individuals, but it's just really not clear what the first steps should be in terms of phenotyping assays. And deciding on this early on will be really important because it means that as we, what would be ideal is that as we start doing these assays or asking these types of questions, if we're asking the same standardized phenotypic questions or doing the same assays as other groups that are doing this genotype first approach, and if that ends up lining up with whatever PMI approaches as well, then that means that we can just start building up a systematic database where it's unified from the beginning or at least there's some level of harmonization from the beginning in the questions we're asking and the results that we get. But is it your impression that your collaborators would have the resources to bring the individuals in and do the phenotypic work up? Yeah, we've done this in a couple of cases and it can be done. We've only done it in a very superficial way and a completely unstructured way currently. We've gone in with specific questions, we've asked them and that's worked. What we definitely have not done is gone in with a case where there's a gene that's really interesting and very conserved and is knocked out and we just want to figure out what is wrong with this person. I didn't even know where to start with that approach. Just one really quick comment about that that relates to what Melissa was saying is so there could be some standardized phenotyping, of course, but then if you did some questionnaires beforehand that could then lead you towards more specific kinds of assays you might want to do in person, then that could be really useful. I was just going to say, this has come up, this is the rate limiting step in the UDN the frightening thing is it happens with known genes. So you'll have somebody who has part of the phenotype, has a mutation in a gene that you know causes that phenotype, no other mutations and you're trying to work out if the neurological component of the phenotype or the cutaneous part of the phenotype is actually also caused by that same variant and so what it shows is that we really need to make that cellular bridge that most of clinical phenotyping as I said at the start is morphology from 1600s or superficial serendipity. It's not being systematically moved as a field from biology into medicine or from medicine into biology and I think that's what it would take in order to be able to build the type of phenotypic space interrogation that you describe. I know that in the PCGC they were interested in pursuing your developmental outcomes and conjunct disease patient and there really wasn't money or infrastructure that will do a lot of deep phenotyping so I think what they deployed was a questionnaire that the parents would answer and that actually proved extremely valuable. It was very short, it was like six or seven questions and I think they were able to really leverage that and generate a lot of very exciting data. So I'm wondering whether for the exact population you have a huge number of subjects and it's just not cost effective to find a way to do deep phenotyping but could there be a first level questionnaire? Maybe half a dozen questions on basic information on neurodevelopmental, other health history issues that then would provide a flag so that out of those 3000 genes of interest that you find those patients you look at that questionnaire first to see where to go from there. Mike in the back. With respect to the value of systematized phenotyping I think what we've seen with COMP and with IMPC is that they do standardized phenotyping on the knockout mice and in some cases they add phenotypes that were not previously known from the knockout because the lab was expert in one domain and not another where those phenotypes were captured in disparate parts in the literature by groups that were unaware that somebody else working on the same mouse got that and of course it's not gonna be sufficiently deep to capture everything yet I think has added a lot of value to our understanding of the genes because standardized domains are tested in each time and there's a chance to find unexpected overlap where usually it's difficult to capture that. I just wanted to add one other resource that I wasn't planning on bringing up. I think someone mentioned Phoenix, Jose did. So that's an existing NHGRI funded effort, it's not perfect but there are over 450 standard data collection protocols. So it's not just the CDE but the actual question that you ask across 20 different domains. So disease domains like cardiovascular, skin bone muscle joint, there's the sort of like questionnaires for parents for hearing assessment, things like that. And those are all mapped to Loink terms with the hope of being able to more readily integrate it into electronic health records. They're available for, formatted for RedCap and some other resources. Again, it's not perfect but it's a good place to start if you're looking for some basic questionnaires to get started with. So I think we're running the end of our time but it seems to me there are a couple of themes that keep reemerging here that might be worth thinking about for Carol and Terry as you think about what deliverables might wanna come. One is this idea of a matchmaker of some sort. That sounds like there was a lot of value both. Going both, and the nice thing about it is it's bi-directional. Clinicians would be interested in finding basic scientists and basic science would like to be informed by clinicians about what the priorities were. And so it seems to me that there's design on both sides so it would be a perfect place to create some kind of a tool. And the other thing I think we heard a lot of was the idea that there's a lot of value to the ability to be able to deploy some pretty simple kinds of questionnaires or surveys. There's a lot of tools out there for that. I'm not sure that we need to think about deploying the tools but what we might wanna deploy is the surveys themselves. The instruments that are used or the questions that are shared that might have some added value. So maybe in closing out of anybody on the panel wanna add anything? Anyone in the audience wanna add anything? Yes. Melissa wants the last word. Sorry. I love the idea of all the questionnaires but can we, I think one of the missing pieces there is making the answers computable whether it's HPO or other things. And that's the thing that Finnex is missing but we've been talking about with them a lot but all of these different questionnaires, genome connect, et cetera. That should be a stipulation, I agree. Mike. So one doable matchmaker opportunity would be the case conference format. ACMG runs a case conference. Others could think about it but I've found that when basic scientists sit in and listen to clinicians discussing a difficult case or an interesting case or a gene that's a problem, they often get into conversations that are useful. Howard, was that a... To be continued. All right, well, I think thank you very much for a really lively discussion and back to our chairs.