 Just as a reminder, we're focused in on the primary area of research recommendations for an HGRI. I want to mention that so we can maybe hone down the discussion a little bit towards that. We're getting a fair amount in that direction already, but even more as we try to flush out these and Mark, I know you're already ready for that. So, over to you. Great, thank you. He turns it over to me and turns me off at the same time. So, but I'm not the first person that Howard has turned off in his long career. So, thank you. We're all weak. Yeah, he'll be here all week. Don't forget to tip the waitress. Right, try the veal. So, panel four, metrics of progress and measures to impact, including cost effectiveness, outcomes to value to payers, influence and quality of care through learning healthcare systems. And I think that takes up our time. So, here's our panel and the panel four members are seated over there, Irwin, you can raise your hand. And Warwick, you can raise your hand. And Ruth has not returned from her deployment to lunch yet. So, what's our charge? That was an interesting question. I identified six bullets out of that panel title that we could potentially look at, metrics of progress, measures of impact, cost effectiveness, value payers, quality care, learning healthcare systems. And so, in terms of looking at the objectives of the major genomic medicine programs, I decided to start first with assessing outcomes of using genomic information clinical care. And what you're going to find is there's a lot of overlap between my presentation and Jeff's. And so, I'm not going to spend a lot of time reiterating. Here's Ruth, Wave Ruth, so everybody knows her. But I'm going to highlight a couple of things that seem to be coming out of the discussion. So, eMERGE, which is one of the projects where I'm a PI in one of the sites, we have been looking at outcomes, but our outcomes have mostly been focused on process outcomes, which is are the processes working, not necessarily health outcomes. Although we've had a couple of projects that have looked at suspected pathogenic variants in actionable genes on the PGRN-Seq chip, and then also looking at a clinical diagnosis of hemochromatosis in patients with certain variants in the HFE gene. But these are more sort of retrospective observational. They're not really related to any specific intervention for genomic medicine program. IGNITE, we've heard a fair amount about, but a couple of things that I wanted to highlight and that I'm going to come back to at the end are the idea of the use across the IGNITE sites of a consolidated framework for implementation research, which I think is really key. And if you'll notice in the second section there, develop and apply common measures across diverse projects to assess institutional characteristics, individual characteristics, intervention characteristics, and process characteristics. And so I think this was enlightening to me as I was reading through the different project summaries was the idea that there had been a real intentional idea about let's try and use similar methodologies across all of the projects so that we can really learn faster, learn what works, what doesn't work. And I think this is something that's really key. And I would just reflect on the fact that while we haven't talked a lot about Patient Centers Outcome Research Institute or PCORI today, one of the big investments that PCORI has had has been in methodology, which is how do we develop the new methodologies that can be applied and then actually fund people to test the methodologies within projects. And so I would at least highlight that as one thing that could potentially be an area of research for NHGRI. GAPH, which I don't know as much about, but we heard quite a bit this morning on this, and I'll just highlight number one under their objectives, which is to develop an evidence base on how to assess and where appropriate, integrate innovative diagnostics into health policy and practice. So again, there is a focus, a primary objective within GAPH about outcomes. Identify, address barriers to genomic medication, medicine implementation. Man, have we done a great job of identifying barriers. And I think you've heard, I think you've heard lots of people talking about that today, ad nauseam. So I'm not gonna spend a lot of time on this, but the bottom line is that here is the list of barriers identified. Now this is, of course, there were two pages in our packet of barriers, so I just cut and pasted and highlighted those things that I thought were directly related to our panel. Integration of data, this has been a significant issue for CSER, for eMERGE, for Ignite. One of the topics that has come up earlier is how do we work together, NHGRI, and it's fine, I'm sorry, Infinite Wisdom. Recommended that perhaps we have EHR groups from CSER and eMERGE that are looking at the same types of problems, actually develop a liaison relationship, work together. And we've actually done joint presentations and have had a couple of joint publications that have come out. And I think that's actually a very good model of how we can synthesize across a number of different projects. Clearly, the data issue is a huge issue, there's a significant body of literature there, and to some degree, there are so many people working in this space, it raises the question in my mind of whether or not NHGRI needs to take a lead role in this, but we'll talk about that a bit later. ClinGen has multiple working groups that are working at different aspects of this problem. There's an informatics group, there's a data modeling group, pharmacogenomics, as we heard about earlier. IGNITE has identified a number of different barriers, they've been focused much more on the implementation side and been looking at institutional factors. So how do we, and we've heard about some of these, you know, IRBs and clinical departments and divisions. How do we engage the clinical informatics team as opposed to the research informatics team? Turn around time issues. The fact that providers really aren't familiar with either ordering or interpreting the tests. The fear that we've done such a great job of creating about discrimination by insurers using this information. The fact that we, even within our own systems, we can't get our IT systems to play together. Much less between two different systems, privacy, protection, and then lack of understanding of all the regulatory boards that we have. CPIC, the Clinical Pharmacogenetics Implementation Consortium has an informatics work group that Bob Freimuth chairs and that several of us are on that are looking at informatics issues specific to pharmacogenomics, particularly trying to reconcile the current variant system which has been in use in pharmacogenomics with star nomenclature and that sort of thing. And the idea that a variant in an allele there is ultimately related usually to a diplotype and some type of a phenotype and so that's a different type of issue. The Institute of Medicine in their learning healthcare system, genomics in the learning healthcare system document which I didn't, it's 122 pages long, it's worth reading but I haven't quite gotten around to it yet. It's amazing how much of their output was related to this data issue. So we see interoperability of EHRs, clinical decision support and data sharing. That was the bulk of their recommendations as I'll show later. IOM has developed a group that was initially called the EHR Action Collaborative and has been renamed to be called Digitize, displaying and integrating genetic information through the EHR and they have a major ongoing effort to complete an end-to-end implementation for two pharmacogenomic use cases. So this is actually a recommendation that came out of GM7, the Genomic Clinical Decision Support meeting that was funded by NHGRI that I co-chaired with Blackford Middleton. One of the things was to do an end-to-end project and this was picked up by the IOM group and they're really working through this and what's great about this Digitize is that in addition to having the usual suspects, we also have vendors and others sitting around the table trying to solve these problems. So it's been so far a pretty fruitful collaboration. We'll see if we can actually bring that home. We've talked some about reimbursement policies and regulation, again just a reminder that we devoted GM3 to this topic and we had one follow-up workshop that was focused on partnerships to develop evidence relevant to payers. Unfortunately, we really weren't able to move that forward in a substantive way. There's still a lot of things hanging out from that particular meeting that we would like to follow up on. And as I mentioned in my comments earlier, the Genomic Medicine Working Group is beginning to engage with the HMO Research Network soon to become the Healthcare Services Research Network Genomic Special Interest Group to explore possible collaborative opportunities. And I think there's a real rich reason. You heard a number of reasons that I articulated earlier why this would be good. Another reason is that almost all of the HCSRN members also have a payer arm. So we have payers at the table that are used to working with healthcare systems. IGNITE has been talking about the evolving evidence base and changes in clinical practice, again, with limited amounts of evidence. And again, this comment that came up, preference to reimburse single tasks rather than panels. Evidence base has been identified as a barrier across most of the projects. ClinGen, in part, was developed to have a central repository of annotated variants in clinically actionable genes associated with an evidence synthesis. CPIC was developed to create evidence-based guidelines for the use of pharmacogenomic information in clinical care, although these guidelines are written for the scenario where the pharmacogenomic information is already available to the clinician and available for use. The reality is that evidence issues are inseparable from reimbursement issues, in my view, and all of them are dependent on outcomes. And I think we've heard that theme continuously. I wanted to also focus a bit on IGNITE because one of their goals stated in their program is to contribute to the evidence base regarding outcomes of incorporating genomic information into diverse clinical care environments. And so they've also identified some barriers, which is how do you ascertain, recruit, and retain patients? How do you engage providers and patients? The knowledge and education gap, the clinical validity and utility evidence gap, and lots of gaps. We should open a store called Gap. It'll never go, right? So... I reflected on one of my colleagues on the Secretary of the Advisory Committee for Genetics, Health, and Society. Actually, the chair of that group for a number of the years that I served was Reid Tuxin, who was at the GM3 meeting and has had a participatory role in some other meetings that we've generated. And he made a couple of points that I think are worth putting back in front of the group. The first is, he said, we can't afford incremental benefit with extraordinary costs or another add-on technology. That if genomics becomes one more in a series of add-on technologies that adds tremendous cost with very little benefit, it is just not going to go. And this is a paraphrase. He says, we are looking to you, meaning the genomics implementers, to transform the way we care for patients and, quote, solve the problems of the healthcare system. End of quote, to which I thought, oh, great. Thank you for adding that to our list of things to do. But I think it is worth our understanding of this very fundamentally that if we, and I hear colleagues talk about the fact, well, you know, radiology never had to deal with this or this didn't have to deal with this. No, we're in a different environment now. And in some ways radiology came about, well, were you still relatively flush with money? And we didn't have to pay as much attention to the value proposition, but now value is really key. And so we have to really do that. So here I'm proposing a new central dogma, which is depending on which analogy you prefer, the elephant of evidence in the living room or the 800 pound gorilla of outcomes, who's gonna win that? So again, I think, you know, one of the things, and we were the only panel that was specifically called on to address learning healthcare systems, although it's already been brought up in a couple of the other discussions. I think in some ways we throw around this term without a real good understanding of what it really means. And I'm not sure I understand necessarily what it means either, but at least the IOM has defined this with of course another term, knowledge generating healthcare system refers to an automated system that relies upon large databases of research and patient information. Information gleaned from patients and clinical research is used in learning networks to inform clinical decisions and create a more efficient way to improve healthcare for future patients. This concept is also referred to as a learning healthcare system or a rapid learning healthcare system. So I think that's what we're really talking about. And we've sort of dipped our toe in the water in this and emerged to, we've developed some tool kits that could support genomics in a learning healthcare system. We have open source tools for phenotyping and other things. We've looked at using strategies such as info buttons to provide just in time point of care education. And we're standing up a clinical decision support rules repository. Emerge three, as we've heard, is going to have much more focus on return to results and proposals actually needed to describe potential outcomes including cost effectiveness. And I'm again coming back to ignite here because they specifically talked about what are their activities to create a genomic enabled learning healthcare system. And again, to develop, share, evaluate and disseminate genomic medicine implementation processes and tools in academic and non-academic clinical practices. So processes to enhance buy-in of leadership providers and clinical informatics. Processes and tools to overcome common barriers. Processes to advance reimbursement. Processes to assess clinical validity and utility and processes and tools to overcome a knowledge gap. So again, if we think about the idea that perhaps the work could be in the development of processes and tools that we don't necessarily have to create research that solves these problems but provides the tools with which people can solve the problems that may be worthwhile. And then GAPH, the intra-program working groups learning network is in the process of being developed. And this is essentially the extent of in the documentation that was provided that we heard about Genome Canada but I'm sure it's moved forward from there. And then here, perhaps the most disappointing aspect of the IOM's report was this is all that was, so all of this is all the data stuff on the left-hand side and on the right-hand side of this little piece on implementation which is really about engaging groups with a particular interest who value genomics, measure and track health and healthcare disparities and support social and behavioral research. So there's really nothing very tangible there to sink your teeth in very high level. So in thinking about where our discussion could go, again, as I said earlier, there are lots of groups that are working in the data in informatics. And so perhaps as we think about it from the NHGRI's perspective, we could say, are all the gaps and barriers being addressed? And if there are gaps that aren't being addressed, how could we fill those? And is coordination across projects adequate? And if we say, you know what, we think the gaps are covered, we think the coordination is sufficient and our involvement is currently adequate, we may not need to invest a lot there. But we clearly need to focus on measuring outcomes of interest to patients, delivery systems and payers. We've heard that as a recurring theme. And I again want to reemphasize the role of standardized approaches such as Ignite. Could this be accomplished across all projects? And with that, I think that is the end of my formal presentation under time. And we'll open up for discussion. Before I do that though, I asked my panel, I said, if you have any very short, high importance things to add to what I've said or to disagree with what I've said, here's your chance, but we don't want speeches disguised as questions. So anybody from panel four that wants to add anything at this point? Yeah, I dare. I speak to truth. So. No speech, no speeches. No speech, I think, you know, since we had discussed this and I think what was eyeopening for us in Ignite and I think some of our colleagues here, Jeff and Julie, is the mandate to open up beyond the boundaries of the academic ivory tower into the community and healthcare systems that are different. And we have learned a lot, a great deal from addressing the issues that are, you know, facing genomic medicine implementation in nonacademic settings. And I want to throw this out that is sort of a differentiating feature, I think, of the Ignite at this stage in time. And we're learning a heck of a lot, in particular when it comes to pharmacogenetics and common disease variants in a sort of a primary care, medical subspecialty care environment. So that's what I have to say. Thank you, Irwin. Warwick or Ruth? At risk of being held out by the convener. I suppose for me it boils down, who wants the data and what am I to use it for? And then I think you start at the other end from research. You start at the policy makers and the clinicians. Mark put me off right at the beginning when he said, I think, and I'm paraphrasing, the American health system runs for clinical outcomes, whereas countries with universal healthcare, the money is always there. And so if we're having this conversation in Australia, the constraint would start really with the money. So for me, that means having the payers, the policy makers at the table from the beginning. And I'm sure there are important roles that NIH are probably already playing, but to make sure that all these little bits that are being put together, we're retrofitting an 18th century health system with a 21st century technology really. And it's changing everything, one way or another from patients through to the system. So somewhere there's probably a high level role for NIH that the rest of the world will look at, which is emerging, which I really applaud. Great, Ruth. What he said. So I will say that I have a little bit of a different perspective than the folks in the room. I don't have training in genetics. Mine is at my residence who is in preventive medicine. And so I come at this genomics world a little bit differently. I think it's important to know that furthering any type of program, whether it's genomics or otherwise, you need justification. And that comes through metrics, that comes through evidence. For payers, they want metrics on cost containment. But for providers in primary care, the interest is CPGs. We wanna see clinical practice guidelines. And that shows us that there's evidence before the primary care docs really buy in. And then I just wanna add that we should also think about the patients. What kind of evidence do we need to offer to them to gain their interest in participating in research or coming to their primary care provider and asking for a certain test or opening the door to that discussion?