 There we go. Okay. Before we start, I just wanted to reiterate one thing, and essentially gave a great summary, but let me really emphasize that we're looking for both input and ideas around, you know, specifically around the genomic medicine and genomic medicine implementation along the lines of what we've spent the last day and a half talking about. But in addition to that, we're also very receptive to ideas about the process itself, other groups or other people we should be talking to, other things we might want to be considering. So, you know, don't feel you have to just confine your questions or comments just around genomic medicine implementation, although I'm sure that will be the emphasis with this particular group. Okay. So, Larry has this. Larry. You can't see him. You can see him. I can see him. I can see him. I'm standing here to moderate. I'll go first. I've never been at one of these. I feel like when you're a kid going to the mall and you get to sit on Santa Claus' lap and ask for what you want, and I don't know if it's bad to go first or not. How does that, how well does that work actually? So anyway, you know, I said this pretty much in my presentation there, but to me, all the things that everyone wants to do here are based on the ability to share varying information at a minimum. If you want to get to varying knowledge and applying genomic medicine in EHR practice and do implementations, you're going to need standards and services to do that. And I believe we have enough information and experience and samples out there that we can do that now. We just need more funding. So I'm going to go back to Steve Sherry's idea from NCBI when we collaborated with them on this notion of a legal registry. There was two challenges to that. One was, how do you model a variant? What is a variant? And that discussion still goes on today. And who gets to decide the policies on how all the challenges with all the different nuances and things with representing variation, whether it's pharmacogenomics, somatic, germline, different methodologies, cytogenetics, CNV, structural variants, you know, it goes on and on and on and on. And how do you model that? And what would the clinical community agree to as acceptable trade-offs from a technical standard of implementing all those various things? So his idea was, yeah, we can stand up services for you for the clinical groups and the research groups to use this to improve and lower the barrier for creating more adoption. But you need to set up a policy-setting committee that's community-driven that makes the decisions that will get instituted into those services and standards. So since then, we haven't been able to do it. But if there's some way we could set up and fund an organization where we bring, you know, members from across the community to start getting together to write this mission statement, and I would suggest we start with a very specific focus on just variation. And it could extend out the bigger things later on. And then that group, we can educate them on what we need as technical people, as why we're having so much trouble where the trade-offs are, and can you help us make decisions that we can put into standards? We can use the GA4GH to do a lot of the implementation. There's pilot projects there. We could help foster funding to increase those projects so that they're building tools that are immediately available to help broaden the amount of adoption that can happen across a whole bunch of different players that right now can't even pretend to invest and get involved. I'm going on too far. All right. So we have an hour, and if you can distill your suggestions into a series of bullet points that we can then have a response to, and then have a discussion around that would be helpful. Okay. So some type of a variant reference consortium, like a GRC kind of thing, or Genomic Medicine Consortium, is one thing I think should be funded so that they could set policies, and then the funding and driving the funding of developing the standards and the services to get some basic services to enable broader adoption. Okay. So how do you want to do this? Do you want to respond to that, or do you want to just accept that and have more discussion? Asking them to respond to that is a little bit awkward because they're in a receiving mode right now, because their mission is to hear what the community thinks and absorb all those ideas, and then at some point create a synthesizer, prioritize with the help of the community. But at the end of the day, somebody has to make those decisions, and it's going to be the people that you're looking at. So I don't think... You can ask clarifying questions. You can ask clarifying questions. You can ask the clarifying questions. So how's that different from ClinGen? It's very different. Maybe I'll ask Bob to help out a little bit, but ClinGen is really a much broader picture which relies on folks like myself that are in there to help set the standards so that they can get this expert level clinical quality curation done, and where we get stuck in ClinGen is the implementations of trying to standardize that. So we come up with our own model, and then we turn around to HL7, and they don't necessarily want to do it that way, and GFRGH wants to do it a little differently, and we work with them and we come up with yet another variation of the model and so on and so forth. If we had one group that just focused on this problem, it would not only work for ClinGen, GFRGH, HL7, but everybody. Okay. Heidi? To extend a little bit on that, if you look at ClinGen and how it's operating and ClinVar database, which is when a tremendously useful resource for the clinical genomics community, what happens is we all submit our interpretations of variants to ClinVar, but there's no access to the underlying data. So what happens is I discover that I submitted a variant that Bob's lab in Vite submitted, and we differ. And so I email Bob or somebody in his lab, and I say, can you go in and tell me the case level information on that variant? And Bob then digs and sends me an email back about clinical information through email, and this happens all of the time. All of the evidence base is being sent by emails between clinicians, laboratories, et cetera. And so I think to support a genomic medicine implementation, we need a structure to share case level data, patient data, that is standardized and sort of policies around who can get access to this information and how can we share it in federated ways. And that's based both on the standards that Larry's talking about, but it's also based on policies. If I want to, you know, ping Lisa's database or bio view at Vanderbilt, who governs the policies that says that partners can interact with Vanderbilt and what are those policies surrounding? There's a fair bit of legal and policy-based information that has to govern that to allow organizations to share the incredibly valuable evidence base in real time as a learning healthcare system. And I think that is something that we need to commit some resources to develop so that we know how to operate. I will say that the first two years of Emerge were consumed by creating memorandum of memorandums of understanding so we could do that among five medical centers. So yes, I feel the pain and I think that's a reasonable way forward. Peter, did you have your? From Topic, so I don't know if anybody else has anything related to this. Well, so we'll go ahead and we'll try to reorganize ourselves later. So I think, you know, when I look up at the bullet points, I say a lot of everyday community, et cetera. And I think one of the things that hope to come out of this is how do we better engage other community health systems that, I mean, you can argue whether North Shore is a typical community health system. But I can tell you, we've had multiple, you know, 10, 15 institutions that are in practice similar. You know, there are four hospital systems. They have a medical group. They have some nurses. They want to get involved in this process, but they don't have the research arm that's going to generate, you know, that NIH grant to be a part of this. So how do we really build towards that and engage it? Is it we really want to do community population-based genomics? So do you want to, I mean, again, I feel awkward asking you to respond because it sort of, like, commits you to a program that you may or may not want to, what I think the people at the front of the room will also want to hear, I mean, these are sort of, this is all in the late phase implementation phase. So as we talk for the next 45 minutes, I'd also like to hear some ideas in the discovery phase still. Maybe I'm just being primitive, and maybe genomic medicine is all about implementation only, but there have been discoveries made along the way in the big networks and in another environment. So let's not lose track of that piece as well. Mark, and if I'm going in the wrong order, just yell at me. I'll yield to Jeff after I've had my say. I've, for a long time, I've thought that... I didn't even see that. A patient, you know, facing approaches is really important. More engagement with the patients. And I think to your point, that's an area where there is some rich opportunity for discovery. And this was reinforced this morning in Gil's talk about some of the real opportunities that are arising with technology. I think if we don't do this, we'll probably seed ground to the, you know, direct to consumer folks that have already made a lot of progress in this area. But I think there's a real opportunity to think more about it from the model of, you know, the sequence as a durable asset that can be used by patients throughout their lifetime. And I think understanding all of the context around that and building some opportunities that say, how can we have people more effectively utilize this information to be advocates for themselves, for using this information? As has been alluded to, that's going to, you know, drive a lot of incorporation into practice too. We know already that people come in with their 23andMe result, which I think we'd all agree is not particularly useful in many cases, but physicians are responding to the fact that their patients are bringing this in. How can we really use, you know, highly vetted, high quality information associated with the knowledge that we're generating through ClinGen and other opportunities to really empower people to do it? I think that would be a fascinating research agenda. Jeff, or Jeff and then Bob and then Jeff and then Bob from Vermont and then Bob from Boston and then, oh, and Gil was in there somewhere before those Bobs. So Heidi mentioned this term learning health system and I would say in at least half the talks that we've seen over the last couple of days, there have been these nice circular diagrams showing how discovery is propelled into the clinic and then the clinical information drives discovery. And I guess I'd like to see that, you know, go from PowerPoint to reality in some meaningful way. And I'm thinking about, you know, to your point then about discovery, GM9, where we have a lot of phenotypic information and obviously a lot of sequence variation that is mostly in the uncertain significance arena. Can we actually build the limb of that, that learning health system paradigm to drive the clinical findings into more functional genomics and understanding of disease, mechanism, and biology? So I will take the moderator's prerogative to just answer that for a second and just say that I'm a big believer in that as you know. I suspect part of the strategic plan that we're probably not gonna talk about very much today is functional genomics in some way or other but I would also emphasize that the healthcare system itself can be a driver or an engine for assigning whether a variant is or is not functionally significant, Lisa has shown that, and many others have shown that, just, and I keep on saying the same thing, imagine you had nomad but with an extra column that said phenotype or an extra column that said inferred function, or they have inferred function but actual function in some way or other. So I think that would be something that I want Anastasia to write down. I can say that that definitely is something, what we have here are really some of the comments that have come out that we're specifically focused on genomics and medicine and health but we are making sure that we're having these discussions about the groups that are talking more about the functional genomic side of things and this more medical side because we do wanna make sure that we're thinking about that type of iterative cycle where we can build off of the discoveries that are happening on both sides. And part of that is the EHR, we think, or the large data sets. I do not know who is next but I'm gonna be arbitrary and say Dave, next. Yeah, so I'm encouraged that we're at a point where we are talking about phenotype being part of the knowledge basis and I think I'm just gonna take that that NHGRI is gonna own the phenotype. So I'm gonna segue into what I think is the most important thing that my CEO said earlier. Kids are the foundation of all healthcare throughout the rest of their life. We need to be looking at population-wide sequencing of individuals before they develop any health problems so that we can actually look long to tuning about what variants mean. If we're sequencing 50-year-olds then some of the interesting diseases in 20s and 30s that would have been preventable wouldn't be present. So I think the elephant, at least in the pediatric world is newborn sequence-based screening and I don't think the debate is whether or not it's gonna happen, it's how it's gonna happen, how we're gonna do it in an orderly fashion, how we're gonna do it safely, what sequence data we generate, what we return and how we use the rest of the data to inform our understanding of human biology. And so I think that is a huge project which has multiple dimensions to it and I think we have to think very carefully about what NHGRI is gonna own of that because I think it has huge implications for adult health as well as pediatric health. Clearly the immediately actionable things in the first month in life, child health may be more interested in but beyond that the longitudinal following is something that I think really should be owned by the Genome Institute. So it does raise, I mean, so I think those are very compelling points. It does raise the question of, and I'll ask this to you guys, the business of how do you see partnering with other institutes, that's an obvious partnership opportunity is a strong word, but how does that work? So I'll take the first part and then I can have you take the second part. So the first question about are we having these kinds of discussions with NICHD? We did just have a workshop recently that was co-hosted between NHGRI and NICHD to be able to talk about genomics in reproductive prenatal and neonatal health with the idea being that these are spaces that both institutes are interested in and that there are a lot of areas, that there are opportunities and challenges and how to be able to make this happen for health and healthcare in the future. And so we are continuing to have discussions coming out of that workshop that was held in April. There is information and minutes and videos from that meeting that can be found online on the genome.gov website. So you can find that information as well. And to speak to the second question of how do partnerships with other institutes develop? What's the, what's the, what comes out of them? How do they go? I mean it's, there's not a single answer. It's, there's lots of examples and they're sometimes not predictable but what I would say is there has been an incredible thirst to implement genomics in various ways across the NIH. There have been some very high profile examples of us partnering for example with the Cancer Institute and doing the Cancer Genome Atlas. As that came to an end, now the Cancer Institute of course is doing extensive amounts of cancer genomics. We're doing very little actually and probably will continue to do very little because this is clearly in their domain. Similar examples around microbiome work, similar examples in other particular institutes. I think the fact that there's new leadership of the Child Health Institute and Diana Bianchi happens to be a genomicist, a geneticist. And but she walked into a situation where we already were doing this collaboration around newborn sequencing and now is very interested to think what beyond that. And so there's an example of two institutes strategically getting together, really thinking about things. We have similar discussions at different levels of maturation if you will with other institutes and there's things that might come out of them eventually. I will certainly tell you, it's an Anastasia mentioned in her remarks is that this strategic planning process we will have a day or an afternoon where we will invite folks from all, program directors, a lot of Anastasia and Terry's peers from different institutes to a town hall meeting at NIH and really strategize with them to get their input about what we might put into this. And I'll be doing this similar thing sort of at an institute director level and already have had conversations with some. I could think of other examples, for example in Alzheimer's sequencing where we had a significant collaboration with the aging institute. So it's not a real formula here. They come together over problems or they come together over opportunities. No, I would agree with that. And I think we did have what we view as a very successful first step into looking at newborn sequencing, ran into a lot of challenges with that in both the regulatory and the technical realms that I think we still need to step back a little bit and learn from that before necessarily taking new steps forward. But I think one of the things that people need to keep in mind is that the research that NHGRI funds does not all need to be driven by NHGRI. That we really look to the community to submit applications and propose their independent ideas and actually we're being pushed more and more in that direction. So don't wait for the strategic plan to come out, but if you've got a good idea, please submit it. It's the only way it's going to get through study section and get funded. Okay. Gil, did you have your card up? If your card is up and you don't want to talk, put it down again. So I didn't have a little sign. Talk to the mic. I had a couple in the slides that I presented and I think Mark kind of raised one of them, but I think one of them is just to kind of really take into account the patients. So to enable patients and to providers to kind of collaborate on care rather than thinking of patients kind of receiving kind of care and not being kind of interactive about that. And especially sometimes we had a debate about are the primary care physicians as knowledgeable and so forth. Sometimes patients come in, they're very knowledgeable. So that was kind of one area. And so apps can be developed with that, but there are many other ways. And then the other thing was, I think this was really important is that there's a big issue about people don't, want to share data and they kind of, they name HIPAA in different regulations like that. It's been a really an issue and it is something that now some of the organizations are looking at and so it may be an opportune time to look at that issue because it would really help in open science to not have the blocking of the clinical or genomic data. And a couple of ways that it could be done is through patient, really pushing patient control of that information and sharing things. Once they share it, then it kind of takes it out of that HIPAA realm and the covered entity and all that kind of issues. And or I put in, and this was following a discussion outside about, I haven't seen this, but maybe something that's possible like partnering with CMS, FDA, OCR, or ONC to look at issues such as these HIPAA and also around payer reimbursement and things like that, bringing that community together to understand what the definitions are because many times we'll go to a conference and three people in different settings will say different things about whether or not a genome is identifying information. Just like it was simple things like that. And so if there was either guidance through a memorandum or there was a meeting that after that there was a memorandum so people weren't in agreement and understood what would happen. That would go a long way because right now the lawyers, we have different lawyers. Each lawyer has a different input about that and our lawyer will say something different than theirs and so we can't make an agreement on very basic things like that. Just that guidance I think would bring the open science a long way to make these large data sets more mind-blowing. So what should NHGRI put into their strategic plan for the next 10 years? Sorry? What are the bullet points that they should put into their strategic plan for the next 10 bullet points? Right, so I would say partnering with other HHS agencies to ensure that there's no information blocking of genomic sharing and sharing of genomic information. Okay. Can I just make an editorial comment about partnering in a strategic plan? Having done this a few times, the challenge in strategic plans is that you really want to have things that you own for the most part in your strategic plan if all of your strategic plan relies on the cooperation of others and all likely you'll be dead in the water. I don't think I'm telling Eric something he doesn't already know, but I think it's useful as we think about all the wonderful things that could happen that we really try and focus on what are the things that NHGRI can reasonably have most ownership of and should focus on from our perspective. So it's not to denigrate the importance of the point that you're making, it's just to put a reality check around what strategic plans do. And of course it's tricky because you don't want to not mention something that's an elephant in the room. And then when you deal with the elephant, you don't want to make it sound like you own the elephant when in fact we're not a regulatory body, we're not a payer. And so that's always the tricky thing. I didn't know if Laura Rodriguez wanted to make any comments about that. This is an area that her division at the end at any sure eye particular pays attention to. I didn't know if you wanted to. This is on the identifiability, I think, you go head raised. As an example, I think many of the things that have been mentioned come into that. I agree we have to think about what can we actually own. We've been trying to partner for a long time. And you have to find that sweet magic moment where they're ready to listen to Clinton. So, but I completely agree and it's useful to hear it in terms of thinking about the corollary themes we need to keep in mind to how we do things. And I think there are parts that NHGRI can own in terms of our responsibility to look at these things and to be a catalyst for progress in these areas. So it is helpful and I appreciate that but we also do have to walk the tightrope of not saying we have authority where we don't. And that really what we're looking at are partnerships and how do we deal with that in our strategic plan. But I do very much appreciate and agree with all of the things you issued that are challenges for the field moving forward and confusions especially when you bring in lawyers from every institution with their own individual opinions and then we have all of our many lawyers in our agencies with their own individual opinions. So, Bob Wilden and then Bob Green. So I wanted to talk a little bit about some discovery things since you would ask about that. Things that could benefit the clinician in practice and one of them is helping to further define in a clinically usable way what I call genomic dark matter the things that we don't see on next generation sequencing whether it be the gaps in that, the low coverage areas, the structural variants, those things that we know clinically are there but we can't see and we never know exactly how far down the testing road to go because of that and we let the insurance companies decide that question for us. So I think that would be something that I think is a discovery thing that would be helpful to fill in a gap. The other area is cost and I know I talked about this on the phone when we had a conversation a few months ago but in cost of the sequencing and not just buy and oversee because high throughput is the way to reduce cost but to think about the other uses, the uses for smaller throughput targeted testing and the cost of that still hasn't changed in particular when we think about cascade testing those are things which aren't going to be helped by NovaSeq I don't think somebody can explain it to me if I'm wrong. The other area in terms of cost is convincing either payers or in our organization the accountable care organization that they need to invest in some preemptive testing programs and what they're asking for is return on investment. So we need to understand what those costs are and as geneticists we're not really trained to do those economic studies. So to somehow facilitate those two professions or three or four professions coming together to be able to reasonably model those so that whoever's responsible for the dollars can understand what commitment they're making. So the order I have is Robert Green, Casey, Alana, Jeff and Bruce and I will just, I'll stick in my two cents worth after Bob Wildin and that is that there's a danger I guess that everything we hear is NHGRI should partner with and NHGRI should partner with and NHGRI should partner with and NHGRI also has to have a clear sense of identity of its own and that's genome science that they drive otherwise you sort of end up saying well why do we have NHGRI? I hate to say that out loud but there it is. So as we talk about partnerships and the field is such that obviously there are obvious partnerships that this institute has to drive but there has to be science that this institute drives primarily as well. Robert Green. Thank you. So I have three points. One is almost philosophical. I think that NHGRI has stood for and should continue to stand for equipoise. We've had an enormous period of advocates in some ways resisting genomic progress and we're now I think hearing about advocates who are encouraging genomic process and I do think that evidence based medicine demands that we continually even in a changing environment seek levels of evidence particularly for clinical utility. So I think that's philosophically a very strong position that you have held and should continue to help. Maybe that's not even in question but then secondarily as part of that I think that the field is moving, the society is moving toward the notion of prevention and genomics across multiple domains, multiple disease domains and multiple chronological domains moving from like David said moving from childhood to from infancy to childhood to teenagehood to a young adult to elderly. There aren't any real institutes that could claim I think that entire chronological sequence and it's methodologically very tricky to you can't do a grant for 50 years but if you're gonna give one I'll take one but I think it's important to try to find clever ways to look across the time continuum and I think that could be a unique place where you could look at all the multiple opportunities for genomics to add value and I do think that cost, what we've called econogenomics is a critical part of that. And finally I think that we're like I said earlier we're gonna have five wave after wave of things that are layered onto genomics like that great graphic that Eric Topol published in Cell where you have the proteomics and metabolomics layered onto genomics and at each one of those waves there's gonna be continuing questions of clinical validity and clinical utility and I don't know of anybody else who will be in a position to try to assess the evidence levels associated with that based on the great progress that you've made in genomics and I hope that you'll take that under your wing too. Great, Casey. So one of the things that came up during the discussions was around the need for oversight committees or stakeholders like the Cancer Genomics Board or PNT committees to really be able to review and translate evidence into decision support or other implementation approaches that are likely to be used and adopted. And so I think there's an opportunity for some novel technology solutions around gaining consensus with these groups in a way that makes it more streamlined and so this could be using kind of both asynchronous and synchronous ways of communicating and voting and those kinds of things and so I think that that might be a new area to kind of to focus on. So, I'm sorry, Casey, if I could just ask, so when you say novel technology solutions, so you would want NHGRI to sort of ask them to come up with novel technology solutions or use novel technology to bring them together and arrive at solutions? I think it could be a combination. I think of technology all the time but there could be different communication ways for groups to communicate that may be different than the traditional like boardroom approaches. But I was thinking technology solutions rather than just rather than having emails or if everybody's not in the room ways to get the input from the important stakeholders. A national virtual P&T committee. Am I close? Alana. So, I wanted to focus again on this, what we talked about, when is enough evidence enough evidence to start implementing and we've spent this day and a half talking about how do we implement and you've got folks that are putting their own, that have made strategic decisions to implement Geisinger, North Shore, Rady, Mayo, Quest. Implementation doesn't stop with if you build it, it's there and it works. We need to think about the sustainability that what happens afterwards and so something that NHGRI could really facilitate are the resources into these longer term, what happens after the Geisinger's, the North Shore's, the Rady's. What happens, how do people use this information and the deeper dives, so the longer term outcomes of actually doing it, the deeper dives into then how do we make sure that engage the stakeholders as Mark was talking about, the patients, the providers who are using this information, what do they need to use this information to integrate it into usual care to actually use genomic medicine to actually do genomic medicine, to use genomics as they grow from childhood to adulthood as they use it for prevention of health. We need the ability to do that but a lot of the focus is always on how do we get people to start it. Well, we've got people starting it. We need to have the ability to measure those longer term outcomes and those rigorous deeper outcomes that we need to measure that. A lot of times our clinical partners may not want to measure. Okay, comments? No? No, I agree. Jeff, Bruce, Todd. So David Dillock has been waiting for a while. David, did you? David already made the point a little bit but I just wanted to reiterate a bit. No worries. The phenome. So we don't have a national institute for the phenome. I agree with David. I think this is something that I would encourage you to have to be one of your tugboats that's fairly close to the center but I wouldn't underestimate the amount of effort that it will take to turn the phenome into useful information. I mean, it's 20 years on and we're still talking about how to classify variants and what their nomenclature should be. And so a whole new home but maybe unlike the microbiome and the metabolome, I think this one is becoming so integrally important in genomics and genomics. Not only deals with genomes but it breaks the mold in terms of in order to handle genomes, you need new computational approaches which are fresh to biology and pulling things in for the physical sciences. And so it does seem to be natural that this whole area of semantics and language processing and building vocabularies that have strength in terms of their connectivity. A huge need that we have is that we have multiple vocabularies and none of them are really designed for the purpose of establishing physiologic relationships. Fever and tachycardia are in different parts of different trees and not necessarily connected and yet as a physician, you realize the two sides of the same coin. So just that to me is a deep thought, exercise is, whoa, that's a big one? Is it a tugboat or is it a little bit of effort? I think all the things we talked about over the last two days, there's some other really big tugboats in there. I mean, implementation science, I think the take home message is nobody's taken responsibility for bringing that into molecular sciences. Lots has been done in other domains but nothing like genomics. There's no precedent here, there's no data platform that works for integrative analysis and to what extent is that going to be the future? It does seem to me that we're closing out the era of the genome institute focusing on sequencing, you know? Being there done that, we don't need a ton more species and a lot of the basic stuff and so there are some fundamental readjustments but I'm just so aware of the commitment that that makes and how that means saying no to a lot of other things. And I've said three. So maybe on your list of communities to ally with the community of computational linguistics of which there's at least one person in this room who knows something about that, is another place to go. The phenome, if you're gonna have genomes without phenomes, it's sort of hard to interpret them. Steven, can I press you up just curious just to sort of explore this a little? Obviously we're not gonna argue that phenome is not incredibly important. We obviously have great motivation to connect genomic variants to phenotypes. But you gave a very focused example of what we might be able to contribute by bringing in data scientists, linguists, so forth that help sort of mine, how I'm assuming you're meaning data and electronic health records as a means to sort of use that to define phenotype. Is that about as far as you could imagine we would go? I mean, because once you start down the phenotype road, you can get into the whole world of physiology and biomedical systems that might get huge really quickly and for a small institute that might be a little audacious. So I wasn't sure if you were trying to keep us focused on the phenome side or whether you meant we should bite off all of the phenome. No, I don't think it's, I don't think that's my call. I think that it's a disorganized world. I don't see that somebody's taken ownership of it. I see a great deal of need. I think we need to examine the topography of genomes as we've done with the topography of phenomes as we've done with genomes. We need to look at the dynamics of phenomes, temporal dynamics. And so I think emerge is a great start into that. But really this could become a very significant effort with a much deeper level of investment. And the question inevitably is, I thought it was a beautiful telling initial figure there of which of the boats are in the stream and which of them are floating in parallel. And I think that's a tough question to answer, but it's one I would bring up to the forefront because for me as an implementer, it's becoming increasingly my second most important home. And I'm just aware that boy, nearly everything we're doing, we're going back into a literature that's not medical, it's not even biological. No, I think you make good points. We've recognized for some time that having adequate phenotypic measures is really important. One of the first things we started even before we started Emerge was the Phoenix toolkit, which was designed really to come up with at least some standard phenotypic measures, not computable because it was way before that era that could be used in genetic studies so that people could at least be relating the genome to multiple phenotypes. And so if NHGRI needs to sort of prioritize a bit in that huge universe, at least perhaps starting with those things that are related to making a genetic diagnosis or understanding the function or the implications of a particular genetic variant, would you agree that maybe that would be maybe our highest priority and some of the other things may follow from that? I mean, we could probably set some standards or some examples for other institutes and other disciplines to pursue this. Yes, I hate to hog time, but I appreciate your follow-up questions. So to me, it costs us on average about $10,000 to enroll and sequence a patient clinically and return results. And I'm so aware of how many times you've done that across your panoply of studies. And frustratingly at the end of the day we wind up with a fairly complete genome and a highly simplistic phenotype. And it just strikes me that the depth and effort that we put into the phenome is going to be something which rewards us for the next two decades with much less investment than the initial data generation. And that seems an extraordinary opportunity. Another way to look at it, I think, Steven, is that if NHGRI doesn't do that across the range of human phenotypes, then who is? I mean, the Hart Institute will look at the Hart phenotypes and the I Institute will look at the I phenotypes, but there's not another institute that's really poised to look across all phenotypes. And that's sort of, I think, one of the lessons that we've learned along the way as well. So I keep on interrupting and I'm probably gonna keep on doing that. Dan, can I? Just last question on that follow-up. Is there something NCBI is taking on or NLM? So I can attempt to speak to that. I don't think at the moment they are. I will tell you that NLM under new leadership and now with a new strategic plan, which I don't know if it's out or it's about to be out. I mean, they are looking for their next set of major priorities. And I can certainly tell you that I think this is certainly something that many at NLM at the leadership level and beyond are quite interested in. So I do think there's some opportunities there. Another partner. Is that what you're talking about? Dan, can I? It's a very important partner. So I can't remember who, was it Jeff next or Bruce? Bruce, Jeff, Todd. So Bruce has been waiting for everybody. Bruce is a very patient man. Okay, Bruce, Jeff, Todd. David, do you have your thing up again? Yeah, I just wanted to add something pretty quick to Eric, so. Well, and Mark, okay. So you're gonna have to come after Mark. And we have 10 minutes and 45 seconds left. So talk faster or make it shorter. Okay, I'll be quick. So a lot of the discussion is how to understand the significance of the genome in the health context and how to decide what's clinically valid and clinically useful and then how to educate providers. And I guess the last mile problem in the provider to the patient, I think gets a lot less attention than it needs. I find it really hard to believe that we're gonna take all of these very rich data and at the end of the day, the traditional model of a health provider sitting in a room with a patient is going to be the way it's going to be implemented. And very briefly, I think two things that may be worth thinking about. One is what broadly I would describe as systems engineering down to the detail of what color do you make a single line in a report so that somebody doesn't screw it up. And I would argue that a lot of the mistakes that get made are not because the providers are ignorant about it, but that the reports are so complicated and so poorly designed that they invite mistakes, make it easy to make mistakes. And in fact, we should be seeking systems that make it really hard to do the wrong thing. So systems engineering I think is one area. And the other that I think would be of interest is looking at things like artificial intelligence. Is the genetic counseling, for example, or the consent for sequencing in the future going to be that I sit in a room with a genetic counselor or do I ask Alexa or somebody how this is done? And I'll bet you anything that it's something like the latter in that if we could be building artificial intelligence systems to deal with a lot of the routine, not so simple things that need to happen to better utilize genomic information, it will actually more or less obviate the discussions about education, which become a much simpler task and also about workforce development. Is there great points? Can I make one comment about the second of Bruce's two excellent points is that around artificial intelligence there's significant new interest at NIH at the corporate level around artificial intelligence recognizing some incredible opportunities in many nooks and crannies of biomedical research. There was recently an AI conference held at NIH, Francis Collins convened, and it's very clear this is an interest of his but also of many institutes. So I think we will, and in fact there's gonna be some major efforts that he's sort of strategic planning around that area on behalf of NIH we will certainly be a part of that. I think this will be an area we will be able to draft off of a more trans-NIH deep look at opportunities in AI, and I completely agree that the genomic aspects of it are a sign that we can certainly take advantage of. Jeff. Let me just ask a quick question first, is in the next phase of the NHGRI strategy is it still limited to DNA as the genomics piece? In other words, where does the transcriptome, proteome, and the other ohms fit into the next phase of NHGRI's plan? So I'm happy to attempt to answer it by saying a lot of this is to be determined. First of all, we would not limit it to DNA, so immediately we've always thought of ourselves as sort of a nucleic acid-oriented institute, so we think transcriptome is absolutely fair play for everything we do even now. Lots of discussion for at least 15, you know, since the genome project ended, does our ohm and omics stop at genomics or what happens in these other areas, proteomics, metabolomics, I can tell you that at a previous town hall we had in Seattle, this came up for multiple people and we looked, and in fact now we are in the process of putting a small group together that's gonna sort of, including external people to sort of help us think through. If we were gonna get a little more involved in proteomics in particular, what might that look like? There's a lot of interest in using advances in proteomics that could be anticipated to help us understand the function of the genome. Beyond that, we're receptive, we're cautious just because we're good at ohmics, doesn't mean we're good at all omics and we certainly don't have a budget for all omics, but we're receptive to hear what people think. So two bullet points then with that answer in mind. So one is investment in the statistical methodology to build better predictive models that harness multiple ohms. I mean, we still can do better than the Framingham risk score I believe for heart disease and the PRS data recently suggests that there are lots of methodologic opportunities that have not necessarily been unleashed. So that's the first point is the data analytics. And the second point is if you believe that medicine is gonna become more and more decentralized, the idea of developing point of care devices and a point of decision devices to assay proteins or nucleic acids will have a significant impact on the future of genomic medicine is that's my opinion. And certainly if they're affordable, they could be disseminated into the developing world that sorely needs them and doesn't have access to things like sequencing and so on. I have five minutes and 17 seconds on the clock. My watch says 229, so really brief questions. Todd? Okay, so I'll make the points in the form of a couple of questions. The first is how do we best return the genetic results to a patient so they don't do something really bad like stop a beta blocker because the genetic report says, well, maybe that's not the best one for you without any supervision. So that and related to that is a second point and I'm not sure if this is part of your preview but is there a better way we can or how can we better teach genetics or how to interpret genetics in high schools? So getting this going earlier, so maybe teaching them how to calculate relative risks or odd ratios rather than if a train leaves Los Angeles at five o'clock in New York at three o'clock, you know what times they get there. So maybe making examples that they can use to teach math in schools that are more relevant to this. And the third and last point is in a way echoing a couple of their comments about the economics of this is having started to delgenda some of the economics and trying to relate or figure out the cost and economic outcomes. I see a lot of people, especially in the geomics world, totally inappropriately using economic terms which the card-carrying economists get really confused because we're saying one thing, thinking cost effectiveness means 42 different things depending on who's doing it. And I think somehow making it clear or having some way to get everybody on the same board there with the economics discussions. David, did you have, and then Gail, and I think that then we'll be finished. Yeah. Unless somebody, oh, and Mark, sorry. So I was just gonna say if we restrict our phenotyping to EHR we will be phenotyping sick people, not well people and we're interested in health. And I think as we look towards patient facing methods of data collection, we need to make sure that they're inclusive of individuals who are not computer literate. Don't speak one of the three main languages in the US. And so I think if we're gonna make this for everyone, we have to go after healthy people in ways that are accessible to them. Good point. Gail. So I will say with respect to not using economics terms wisely we can functionally not use genetics terms wisely today. We'd like to all agree on the same genetics terms that would be a good start. With respect to what we really need in clinical genetics we need to understand what all this variation means and ClinVar is a really fantastic tool for that and I think investment in that area has really paid off. What I would like to see more investment in is the high throughput functional testing which people are starting to do but not just that testing but getting that connected to phenotypic data to validate that those predictions are correct and then getting it into ClinVar even for variants that have not been seen in other places and that would be extraordinarily helpful. Mark. This was triggered by Robert's comment about evidence-based medicine. I think we butt up against this wall all the time because in some ways what we do doesn't lend itself to evidence-based medicine which is really a population-based philosophy. Ralph Horowitz has published a couple of articles about a concept called medicine-based evidence which is actually an interesting idea. It's what can we take out of the electronic health record? So if I'm seeing a patient with a set of conditions and I wanna initiate a therapy and I'm choosing between A and B show me in my system the last 100 patients that presented with these types of conditions and who got A and B and how did they do? And Chris Longhurst here at UCSD has this live in their electronic health record. Now to David's point, it's a focus on sick people rather than prevention but I think it's a viable model and I think it's one that's actually much more tractable and relevant to what we're talking about here where we're dealing with these ends of small numbers and it takes advantage of the fact that we've already invested a lot in terms of how to extract phenotypes from electronic health record. So I did send Terry the articles so that she would have those for reference but I think that would be an interesting discovery opportunity to take advantage of strengths that NHGRI has already developed. I see a lot of signs that are vertical but I think most of them are old. So I think we're... Well I think Heidi said vertical for long. No that was... Oh is that a vertical? That was Gail. That's Gail. Gail's still vertical but yeah, okay. So I think we have 24 seconds left so if somebody wants to... Summarize. Casey has 24 seconds worth of... Guys just had something really quick about the AI point and so one of the points that came up was for the patient being able to navigate their care and maybe that could be an area for AI also. So I wasn't taking notes so my summary will be really sort of what I remember and what strikes me. I think the phenome resonates with a lot of people and the phenome beyond the EHR but the phenome in healthy people. Prevention as well as disease treatment. Going the last mile trying to figure out how to do that in a rational way. Coupling this to high throughput functional genomics which I think is sort of not part of the genomic medicine working group necessarily but certainly will be part of the strategic plan. AI efforts, what have I forgotten? There was a lot of discussion about bringing the patients and the providers together and making sure that the information was understandable to everyone in the system development. So yeah, a last mile thing. And then lots of partnerships but again maintaining a focus or a leadership position with respect to genome science I think is another piece of advice that we would offer. Don't forget the babies. Sorry? Don't forget the babies. No and the babies I think you make a very compelling point it's sort of so we're doing, in Emerge 3 we're doing, we're sequencing these 100 genes and every so often I think to myself well that's completely lunatic. All these people have no phenotype and what are we gonna accomplish? And I think then I think if we don't do it then somebody completely crazy is gonna do it and take it in all the wrong directions. So I think the babies offer those kinds of challenges in spades but somebody has to do it. Somebody has to do it in a thoughtful rational way so it gets rolled out properly, not in a crazy way. So yes, babies. We like to think that in baby seek we are helping to try to do that and we're looking at over 4,000 genes in those babies and finding actually quite a lot and then circling back and doing deep phenotyping which is one of the themes people have been talking about and finding that there's evidence, subclinical evidence for things that no one had previously found. So I do think there's a potent area for exploration there. There are lots, I mean we can talk about that forever but so I'll stop unless there, I'll actually I'll give Eric the last word. Here we go, okay. I'm not sure I have much to say, thank you very much. Very helpful, this doesn't end here. Other thoughts occurred to you? Again, you can readily email any of us or email the genomics2020 at nyhi.nih.gov or just stop one of us and chat and cause this is an evolving process for the next couple of years.