 Hello, everyone. This is Pat Deverka, and I'd like to welcome all of you back to our last session, which is entitled Realize the Promise of Genomic Learning Healthcare Systems. You've heard my voice quite a bit throughout these past two days and Terry and I will be trading off roles in terms of moderating the session. And I would like to introduce our first speaker I'll introduce each of you before you give your presentation same format will take clarifying questions after each talk, and then we'll be looking for questions in the chat and in the Q&A to call on people for the panel discussions the same format, and then we'll culminate at the end with us going through all of the solutions that we've been able to capture over the past two days, and ask for input from everyone, but specifically moderators and panelists to make sure we accurately captured the solutions that we've heard and that we haven't missed anything, because as Terry has emphasized, this was really a way for us to acknowledge all the progress that's been made for genomic learning healthcare systems and say what do we need to do next to keep that going. Okay, so I'd like to introduce our first speaker, which is going to be Heidi Rehm. Heidi. I need the unmute button. Thank you for the introduction. Share my screen. So, thank you. It's a pleasure to join this conference it's actually our meeting. It's been outstanding talks and I've learned so much. I was asked to talk about for turning genomic learning healthcare system data into knowledge. And so I'm going to talk a little bit about some of the areas that are relevant to my work. There we go. So I thought I'd start out and just sort of reiterate what is the current genetic testing workflow the most common scenario of how genetic testing gets ordered. Typically a provider orders a gene panel test. No clinical data is usually provided maybe a very high level cardiomyopathy or something like that. The lab classifies all of the variants according to ACM gene guidelines using the existing evidence in the field. And then they report for diagnostic purposes a variant of uncertain significance likely path and path variants. Typically just a PDF goes into the EHR although it's a delight to see a number of places that have more than that. So it returns to the patient and there's often no further interaction with the laboratory in most cases. A lot of labs now submit classified variants to ClinVar but if you look at their evidence summary they're often not including the taste level evidence from the cases that they've tested in the lab. So that kind of model isn't really supporting a learning healthcare system from the lab provider back and forth dynamic perspective. But yet it's desperately needed and we've actually just been doing a project to collect a lot of data across all the major laboratories in North America. And you'll see on the right here, we've collected data from 1.5 million tests performed over two year period. 97, almost 97% of that data is from panel testing and then about 48,000 exomes worth of data 3% of it. So you see if you look at two questions diagnostic yield, which you can see on the right side whether you're looking at panels or genomic testing, both are fairly low in their yield. Yet they're fairly high in their yield of uncertainty sort of inconclusive results due to VUS is worse in panels compared to genomic testing and that both of those numbers are statistically significant comparing panels to genomic approaches. So either way you look at it panels or genomes, we generate a lot of uncertainty, and we need to have a better learning healthcare system to increase our yield and reduce our insert uncertainty. Interestingly, we did ask the labs, another question which is do you subdivide your view us is by sub tier. Tell us what you're reporting, you know, rate rates are for the sub tiers. So two of the laboratories did that in practice quest and mess general Brigham. And what you can see here is when you divide and I'm using the terms that were likely to put out through the new ACMG guideline of us high mid and low. And what you'll see is with the panel based testing it's roughly distributed high and low and most in the middle. But when you what you see is reported from genomic panel testing. It's much more skewed to the high side, or mid and nothing is reported on the VUS flow side. Why is that. Well, it turns out that in the practice of genomic or panel versus genomic testing. You can't supply detailed phenotype data to the lab in order to interpret genome you can't you can't do anything without phenotype. In addition, the standard practice for panels is it lads report all the view us is you can't do that in a genomic test there are thousands of them. So you restrict to based on evidence level and the gene phenotype match for the patient. And that leads to the reduced uncertainty that we're returning in these results as well as in a lot of cases, the better correlations to the result in the patient's phenotype. Also, because we use trios often in genomic testing, we actually introduce another element of learning into the system. So let's see if you compare the yield and uncertainty levels from panel versus or sorry from trio versus non trio testing is that added parental data improves the yield and decreases the uncertainty. So that's actually another layer of real time learning that's happening by virtue of including parental genotypes. So now some a small number of laboratories or a small number of cases, get a slightly better process. And I'll just outline some of the improvements, although this is a less common scenario. And I, it's the same as what I said, in terms of the text and black, but the added pieces here are in some cases that the lab has a v us that looks suspicious v us high category, the lab may seek out more data. That could be to reach out to ClinVar submitters who have submitted on that variant to get additional data from those submitters and we get emails like this regularly I'm just pasting on the right here. The last email I got a couple weeks ago. This is your colleague we've identified this variant in a neonate receptile hypertrophy we're contacting you submitted to ClinVar. Do you have any more data on this variant and any phenotype, and we replied with Matt Leibow reply from the Mass General Brigham lab. So this is the infant with DCM concentric left ventricular hypertrophy, this with the added variants. We didn't do segregation testing but here's the data and now this lab director replies and says, Oh, I thought we could dismiss this variant but looks like we can it's actually matching our phenotype, and so on. There's a learning that goes on in real time. When you do this extra layer. Also that happens in the case of follow up the provider with the provider. A lab may reach out to the provider and say, yeah, I found this variant, but it's correlated with this specific distinct phenotype. Have you looked for that in your patient, and the physician may come back and say, Oh, yes, I didn't include that on the right form but indeed my patient does have that, or I just want to test or another follow up and look for that. And those things enhance this sort of feedback loop. And when you report those variants you can then contextualize that with additional information. And in some cases that may change the classification of the variant. In other cases it may not but still help guide the physician and their use of that data. And then for some laboratories, they may include that additional evidence even if unreported in what they submit to ClinVar. This model does support a learning healthcare system, but keep in mind, this is fairly labor intensive. And we're all pushing the labs to reduce the cost of testing. This doesn't help it. So we have to think about these models and how we can make them more efficient. To just point out the more even more idealized genetic testing workflow that might increase some efficiency here, which is that during test ordering the full clinical data actually gets transmitted to the clinical lab and or they have access to medical records directly. The laboratory may classify candidate variants, but then follow up seeking out global databases for shared individual level data genotype and phenotype also be able to access functional databases with higher throughput analysis of hypothetical variations. You might know the variants impact even before it's been seen and a patient and there's increasing work in this space for saturating mutagenesis across important genes. Also access to familial genotype and phenotype data, not just running a trio when you have a genome analysis, but in fact through databases effect in the UK Biobank database 12% of individuals have relatives in that same database. You might be able to actually get the answer to certain segregation questions of de novo cards directly through globally shared data sets that may include familial relationships. And then we can really start to think about, should we not be returning all of these v us is even in panel based testing. The return of subset of us is like the view as high and those with a strong phenotype match and encourage follow up by the physician physicians cannot follow up on every view as your work. But if we were to distinguish those with high evidence that really can help, you know, tell them when to spend that extra effort and even guide them more specifically and what they should be doing. And then certainly improving the structure and genetic test results transmitted the EHR allowing this data to be combined genotype phenotype, and then shared more broadly for others to make use as well. And then certainly return that data to and submit it to clean bar with the evidence this time included. Idealized workflow does depend on several key components. And I will point out that if you look in clin bar, although you can't just look in clin bar and see this but if you analyze the data in clin bar 75% of variants in clin bar have been submitted by only one laboratory. So you often you get one shot at this one family. And that is where the this ability to not just have access to global data or better curated evidence from the literature things like that, but in fact, support this true exchange between the lab and the and the provider. In fact, we have this dynamic follow up loop to give feedback on candidate variants in cases of variants that might be more prominent in the, in the, in the population. That is where this globally shared evidence base becomes really critical for both individual level data and functional data. And I will point out a couple of systems databases that I've come across that we're intriguing. Because I'm part of the advisory panel for Caesar. You know, diver project that the New York City can kids can seek I think so project developed where the lab would do the primary analysis, but then the interface shifts to the provider. And based on the candidates and the phenotypes associated with the genes those variants ran as specific questions of the provider does the patient have this for that and they answer present or absent or unknown. And that information that's very targeted related to the findings in the genome, then get fed back to the lab to then further refine the report and I think those kinds of dynamic loops will be critical for us to most effectively interpret the genomes for many indications. On the right side of this database called decipher that's maintained in the UK. And that database is not only facilitating analysis of genomic data and giving summary information, but for every patient that's analyzed through any collaborating group and a lot of data from the UK and now I think the national health system is going into this system. The structured phenotype data is just deposited associated to the gene. And then if you put it in this case I put in the gene GNB one, which I found in one of my cases. Look up, and I can see every phenotype that's been identified in an individual with a variant pathogenic variant that gene, compared with a statistical comparison to that phenotype. And it's specificity for that gene, and it gives a p value. So in this way you can in real time be seen whether the phenotypes and the variants and candidates that you're looking at correlate or don't correlate to candidate diagnosis based on a real time growing patient calculation with phenotype data. And I think that again that's the type of, of higher scale process we need to engage in, as opposed to email exchanges between providers and laboratories. And I will say that the efforts are really well underway in a number of countries, further along than the US, partly because of national health care systems to be able to share both variants and individual level data that Cypher database I mentioned, gel genomics as their clinical variant arc that enables sharing of individual level data. Australia has built this sharing platform to network all of their laboratories across the country. Canada has the open genomics repository. Japan has developed database, some have more or less you know, where they are in their individual level data sharing but a lot of national efforts. What we want to do is then connect these national efforts into an international global network and we are working within an alliance to think about federated environments to share this data and be able to query on it, using systems that are built on the federated principles of matchmaker exchange that we built for gene level sharing, but actually now to do this more robustly we're a variant, you find a variant, you search the database you get the phenotypes back from any individual has that variant. And that works today in silos and systems like variant matcher and Hopkins or Gina to MP at UW or the Franklin, which is a commercial product, but by use of new API is coming out from global alliance will be able to connect global databases around the world to be able to share this data and query on it. So that so just to finish up with my last slide, the really the requirements for building a robust genomic learning healthcare system for genomic knowledge management in this context. I think is unambiguous genotype representation to share this data globally with a dot more standards and are being used today, things like the variant representation specification, other backend standards quality metrics for genotype calls that we know when a genotype is real or not. Also meaningful and standardized phenotype data collection and rare disease that has to rely on human phenotype ontology, but other phenotyping systems as we look across all disorders and longitudinal data interventions and outcomes, also willing it's an infrastructure to share individual level data globally. Sometimes it's not just I want to but how do I do that how do I connect and then reciprocal collaboration as I was really able to connect between the lab and the provider in a much more rigorous back and forth process that can be asynchronous because it's very difficult to get a provider and a lab on this on a phone call at the same time, for example, how do we do that through perhaps asynchronous methods. And then that allows us to capture evolving phenotype build very level evidence, as well as look at what variants we need to analytically validate through the system, and I'll stop there. I said, thank you ID that was an excellent presentation, and I think we do have a few minutes for some clarifying questions. Carol bolt, who was one of our moderators asked a question Carol do you want to be unmuted and asked directly because you had some specific details in there that you might be in the best position to ask. Yeah, thanks Pat. Yeah, so hi that was that was fantastic and I one of the things that keeps running through my head. Through some of these talks is how we can leverage genome biology for some of these interpretations and especially the highly structured phenotype genotype data we have from model organism. You mentioned this RBM 20 case specifically. And if you look at the phenotypes associated with variants in the mouse. They sound very much like some of the phenotypes observed in patient populations. And especially for view us is where you're, you're really looking for all the evidence that you can to functionalize these. How do you see incorporating or maybe better incorporating model organism data into these genomic learning healthcare systems. One question Carol and we actually have incorporated in the matchmaker exchange that we do for matching gene to gene. We've incorporated both monarch and a model, model matcher system. And monarch brings in, as you, I'm sure you know, model organism data, and that allows us when we submit a gene, having found a candidate and a human patient, we sometimes will match on a mouse model or, or other organism. And that can be very helpful to implicate that gene and disease and I think the same could be true to think about it, a slightly different way in terms of how we search for variants, where the homology across the genes are not necessarily sufficient to make it worthwhile to be searching for variants except for very highly conserved genes. So I think it's a slightly different model as we look at the gene but I will say matchmaker exchange system is now being used routinely for clinical labs in genomic analysis. So we have a special issue of human mutation, and I asked three of the clinical labs to write papers, and they, they alone have implicated a thousand novel genes in from their routine clinical testing. And that they're they're making use of these model organisms in some cases most of the matching, I will say is based on, you know, human case matching but it's there, if there are no human cases. Great, thanks Heidi. Okay, great, we have some additional questions but I'm going to take the prerogative of the moderator and defer those until the panel discussion because I were just a little behind time. And I want to now turn the floor over to Mark Williams, who's going to speak about the creation of a virtuous cycle to realize a genomic learning healthcare system. Thanks very much Pat, I'm sure most of the people there say well wait a second hasn't Williams presented about 10 times already during this meeting, but you give me one more time. So we're going to talk about a little bit about what is a virtuous cycle we've used the term a lot but we haven't really defined it. Why it's essential to include virtuous cycles and learning healthcare systems and how we might be able to achieve this in genomic medicine. The investor defines a virtuous cycle as a chain of events in which one desirable occurrence leads to another, which further promotes the first occurrence and so on resulting in a continuous process of improvement so this. I think really encapsulates a lot of the themes we've heard in the presentations about that this is iterative, it's progressive. And it's something that really is continuous it's it's never done it's always in motion. Well, I sometimes get really interested in what where did this come from and so I did a little bit of research and it's interesting at least to me. Mark, Mark, I'm very sorry to interrupt you but we're not seeing your slides advanced. What I screened. Let me try popping. Okay. That. Yes. Okay, good. Fortunately I told you everything's on the slide so we don't go back. So if we talk about the opposite of a virtuous cycle which is a vicious circle or vicious cycle we actually have etymologic origins of that back to the 1700s. And it maybe says something about the human condition that we focus more in the negative than the positive because it really wasn't until the 1950s that the virtuous cycle. And so we're going to put into usage, which does contribute the invention by Jeff Bezos on the back of an African in 2001 in which he supposedly introduced this concept. However, I do want to emphasize one thing that Bezos emphasized when he was defining this is that and that is the role of customer and I'm going to come back to that in just a second. So we've all seen this from the strategic plan. It's really exciting to see these virtuous cycles being included in the NHGRI strategic plan, but it also points out the idea that this is really an enormous subject. And so in 15 minutes to somehow address everything that's necessary to realize a virtuous cycle in a genomic learning healthcare system is essentially a sycophant, sycophant undertaking. Just as saying that is obviously a problem. So I began to say, how can I really get a handle on this and as I look back through the agenda. And all the focus that we've had on so many different areas over the course of the last two days I came back to bullet for in Peter Helix slide on the elements of an essential learning healthcare system for genomics and that's need the patient voice. I recognize that over the course of the time that we've been together. Well, we've talked about this some and Carol Horowitz had a beautiful presentation on the importance of inclusion. But we really haven't spent quite as much time on this as we have in other areas and so I made the decision to kind of focus the realization of the genomic learning healthcare system on the need for patient engagement within this and this is also based on a personal experience which you've heard me relate earlier in the meeting about how I think the success of the geisinger program has really been informed to a large degree by our early engagement with our patients. Two years prior to the launch of the program which is really informed how we developed it. Now I'm not alone in thinking this. This is from 2011. And this is from the series of learning healthcare systems that came from the Institute of Medicine now the National Academy of Medicine patients charting the course and here's a couple of full quotes from their summary. The patient engagement is central to taking advantage of banners advances in the personalization of care based on genetics preferences and circumstances. And at present there's a failure to fully engage patients in the public as active partners in advancing the delivery of care that works best for the circumstances, and ensuring that the care delivered is of value. I'm going to go back farther. This is something that any of you that have seen me present before have probably seen this slide because I use it a lot and this is from 1987. From Steve Pauker and Jerome Kassir personalized medicines the practice of clinical decision making such that the decisions made maximize the outcomes that the patient most cares about and minimizes those that the patient fears the most on the basis of as much knowledge about the individual experiences available. And I think they really nailed this definition. It's very patient centered. We have to understand what the patient wants to accomplish from the episode of care, and ensure that that happens. And while we're all talking about genomics, which is appropriate given the context of our conference. Let's not forget that it's not all about genomic that there's lots of other information that specific to the patient. It's just the themes that we've already touched on like socioeconomic status and insurance and other things that can really impair the ability of the patient to achieve the outcomes that they want. So what I'm going to do for the remainder of the talk is to take on the icon from this. And go through the different aspects and just talk a little bit about scholarship about patient engagement in a genomic virtuous cycle and talk about gaps and opportunities. The first area is the new genetic new genomic and clinical knowledge. And so one of the things that a learning healthcare system does is to prioritize translation of knowledge into practice, but we have to use this to identify and address problems that are really defined as important by the patients that can't just be things we think are interesting. And what that really means is, and again, heartening back to Carol presentation, we have to engage patients at proposal initiation to really define research or clinical questions that are of most importance to the patient. And we have to partner with patients in the analysis and interpretation of research results. So this is a modification of patient engagement framework which initially appeared in 2013 that looked at levels of engagement in care and care improvement. Looking at consultation disclosure involvement and partnership and shared leadership. So one of the things that Dan Davis and myself and others at Geisinger did was to say, well, what if we were to apply this continuum of engagement to research to discovery. So we've heard about consultative models where we inform patients about the discovery activities that utilize patient data. This is really treating patients as more passive receivers of information that we think is going to be good for them. Next step in the continuum is involvement. And this would be consultation. So we use patients as advisors. So perhaps we have an advisory group that we also then enable ways that patients can share their data, perhaps have some control over that data sharing. But ultimately where we want to move to is to have patients partnering with us in our research activities, meaning inclusion of them at the outset of project development as investigators with funding with protected time as is important. And to be involved in all aspects of the projects and so our default now at Geisinger is really that when you put forward a proposal for research, we have to see evidence that there's substantive involvement of patients at the investigator level at project conception, or we're going to ask hard questions about why that's not appropriate. The next area is the quality improvement strategies and as we've heard a lot of what we're drawing in learning healthcare system has come from the world of quality improvement. And this was the subject of a PCORI topic brief in 2014 where they asked the question, what is the role of patients in quality improvement activities that healthcare systems are engaging in. And they noted that that time was that there was increased interest in engaging with patients, both to identify quality issues and to use the patients as advisors, but at that time there really was no comparative effectiveness of whether involvement to patients really improved outcomes, or what impact involvement of patients really had. And this was highlighted as a priority research area, but there still hasn't been a tremendous amount of scholarship, although there are some areas. For example, there is an initiative to reduce falls in the inpatient setting. And one of the requirements of that has been to involve patients who have actually fallen in healthcare institutions as part of the improvement team. We're seeing some progress there but we still need more information about whether this engagement is leading to better quality improvement. The next area is clinical practice innovations. How do we do things differently. There's an interesting systematic review about patient involvement again in quality improvement from 2018, where they characterize engagement as low level unidirectional consultation that would result in discrete products. So this is again more than engagement model where we come up with something and we sort of impose it on you. In contrast to high level engagement where patients are involved in the co-design or partnership strategies that really led to substantive changes in care process and structural outcomes. Now again, there wasn't a lot of data to crunch to say, are we really demonstrating improvement in efforts that use high level engagement with patients in quality improvement. But it was clear that patients were resentful of being treated as tokens or just in consultation after decisions that already been made. So if we look at it from a satisfaction perspective, I think there's a very clear signal that this is not an appropriate level in the engagement from the patient perspective. But again, there are more opportunities for research. If we move on to the next outcomes data collection. It's important to recognize the patient centeredness is one of the six domains of quality as defined by the now the National Academy of Medicine. And we are starting to see more inclusion of patient centered outcomes into quality measures that are being used by CMS and the National Center for Quality Assurance. We recognize that there needs to be standardized outcomes for genomic medicine. And that was identified through several NHGRI funded projects. But we also need to have patients informed outcomes. And for the most part in genomic medicine, these are undefined. We're starting to see how other fields have addressed this. There's at least one major orthopedics journal that says if you are going to submit an article to us with patient outcomes. The outcomes have to be defined by patients and you have to use measurement tools that are validated patient reported outcome tools. Or if you can't as a physician or a, you know, other healthcare professionals say, well, it looked like the patient improved to us, and that's what we're measuring. And so I think we need to develop those in genomic medicine. So again, you're seeing a recurring theme here. There are lots of opportunities for research. This is the Proctor Implementation Research Methods. And I just again want to show that there are several patient related outcomes that are reflected here. Patient centeredness, satisfaction, function symptomatology, which should be defined from the perspective of the patient. So, and then implementation strategies involving interventional providers, consumers. So it's not that I'm telling you anything new here. But moving on to data analysis and interpretation, I think it's a logical assumption that patient involvement would improve interpretation relevance of findings. And it was interesting to look at the mental health area because we're beginning to see some best practices in mental health that are beginning to incorporate patient involvement in analysis of data. I've not been systematically studied in genomic medicine. I wanted to show you a table from this Jennings paper that I've cited that I thought was quite interesting. Here it shows in the preparation phase, ensure there are people with lived experience in the research team, recruit a heterogeneous patient public involvement co researcher group in the co production of the coding framework. Everybody aims explain the co researcher role and why their input is value. This is a training aspect. We can't expect people just to come ready to go. We have to involve training and education to help them to be able to fulfill this role. And there should be ongoing exercises relating to this, including feedback exercises, and then ultimately, you know, the, I might disagree a bit with this they say the research teams that amend preliminary framework on the basis of co researcher institutions, that to me seems a little bit less engaged than I might have framed it but I think the concept is essentially sound, and then to make sure that you use a broader group of co researchers for additional comment and do this. So I think this is a model that is quite interesting. So we've zoomed through a lot in a very short time here so I think my work here is done. Oh, wait a second. There is one more thing. This is something that I purposely hid from the initial slide on this patients charting the course. This is another quote that I think is extremely important in health systems, brighter preferences and supply, often shaped the care delivery. So this is the reality that we all live in is that as much as we like to talk a good game about engagement. At the end of the day, our new changes, much more frequently involve clinician preferences and system preferences. And so we need to be very cognizant that all of these roles and the role of the patient is essential, and we can't subsume that to what we may think is clinic and sort of that. What that ultimately means is that this is an exercise in cultural change. And so if you want to develop a learning healthcare system in genomics or anything else. You really have to do a cultural assessment and be ready for a cultural change because it's not tweaking business as usual. It's really a complete reimagination of how we do things and so I think if we go into it with the idea that this is a cultural transformation. We're much more likely to be successful. And with that all in. Thanks Mark. I just wanted to maybe comment on a few things that I'm seeing in the chat. Karim Watson says that there's a lot of opportunity to learn from best practices in community based participatory research which has been a big emphasis, for example on the CTSS and Sharon Terry from the Alliance said that they, she worked on a project where they actually created a tool to measure whether true engagement is being done and she provided a link there. So I guess my question is, you know, I know you've done work funded by PCORI. People have been talking about patient engagement in research and engagement of the public. Do you think there's anything specific about genomic medicine that requires a unique approach for how we engage patients, or can we pretty much take advantage of the lessons learned from, let's say the work done by PCORI. I think there's a lot that can be used from prior work. I don't think we need to reinvent the wheel. I think that there's also a possibility that there are going to be unique aspects to genomics that we do need to address. Again, I think some of the things that I've heard in previous. Thank you Sharon. We've seen in previous talks is the idea that, you know, there is, there are some differences about genomic data. It persists, it needs to travel with the patient. And so I think there's some really interesting engagement opportunities that go over and above what we might expect to see in other programs where perhaps you can delimit the encounter to an episode of care or one specific issue like a fall or a medication, inappropriate medication use something of that nature. Here I think there's some really interesting questions that address the whole idea about interoperability that would be fascinating to get a patient perspective on. Okay, great. Thank you. All right, so because moving along now, I'm going to introduce our next speaker and Jeff Ginsburg is going to bring us home by talking about scaling the genomics enabled learning health system to optimize research and clinical care. Great. So, again, thanks Pat and Terry for for having this great meeting. Also for inviting me to share my thoughts about really scaling genomic enabled learning health systems to the to a national platform. And it's fair to say what I've seen here in the past two days and when I think about scaling is that we really have. This is the problem statement. If you've seen one genomics learning health system you've seen one genomics learning health system. And the reason why I say that is obvious health systems are different. They serve different communities and demographics and the architecture and the domain expertise. It certainly has been different as we've seen many different types of genomic learning genomic enabled learning health systems over the last couple of days and what they've implemented. So, Mark has shown you this vision I'm not going to belabor this on the skip this slide but I'm interested in sharing with you. At least one version of this that came from Charles Friedman who's been a guru in this area. And this is his notion of the same idea on the left hand side is a is a system, which like most academic systems teaches but really does not learn. On the right hand side is a system that learns as it teaches as it learns by assembling data by analyzing data by many elements of the cycle of the virtuous cycle that that mark just went into quite elegant elegantly and it also takes the further step of delivering results and monitoring changes as they occur in the context of that delivery. Now, when I was in 2015. I was a co chair of the then Institute of Medicine Roundtable. I was in Precision Health with Sharon Terry, and I was also the co organizer with Sam Shaykar the chief then the chief information officer at Northrop Grumman. On a workshop that you've seen highlighted here many times and not to be repetitive but I really just want to emphasize three trusts that came out of that workshop because I believe they're really foundational and form the basis for the scalability of systems. And the first, as we talked about yesterday is the interoperability of electronic health records if we don't have systems that can talk to one another they cannot scale and as we heard yesterday as well there's been significant progress in this area with the integration of clinical genomic data in workflows. The workshop in 2015 supported regulatory incentives to drive interoperability this was really interesting to see how meaningful use drove the uptake of electronic health records, electronic health records but it hasn't really come the full cycle of interoperability. And we've seen through efforts of GA4GH and HL7 the ability to create genomic standards for data so that they can flow. And we've also seen demonstration projects, such as through emerge and ignite NHGRI funded projects that can actually be used to demonstrate the interoperability of tools from one health system to another. The second area which we also discussed today and was the subject of a part of the workshop that Ken and Mark ran last year is the clinical decision notion of clinical decision support. And again, I think tremendous progress has been made in this area, creating stores of clinical decision support particularly in the emerge program that can be shared across different groups and the ignite network is actually doing it clinical trials that are testing the hypothesis that clinical decision support information can actually drive changes in clinical outcomes. And the last area, which as Darryl indicated we might be falling a little short, but are making progress in our building platforms that are that use fair principles, findable, accessible, interoperable and reusable. But the areas that and then Mark even alluded to this is that we're falling short on our really engaging the public and our patients to create a donor culture and this is something that Terry I think mentioned in some of her summative comments this morning. We've seen a lot of use of patient provided information into healthcare and therein lie some important gaps in phenotypic information for sure electronic health records being clearly sporadic and the information they provide, and the integration of the learning health systems with other data stores that can actually provide useful information about genomic variants as highly as Heidi emphasized a little earlier in this session. The question is, can we create a national scalable genomic learning health care system. And I was pleased to learn recently of a national learning healthcare system that could be a model for one for this for us in the genomics field. The vision is run out of the National Institute for Mental Health. And it's, it's, it's on early psychosis and intervention, it's an early psychosis intervention network. The acronym is epi net and you can see the vision here is to accelerate advances in early psychosis care recovery outcomes and scientific discovery to a national early psychosis learning healthcare partnership, which seems if we change the words might actually emulate what we would like to achieve in genomics. And this was initiated in 2009 19 Susan as run who's mentioned on this slide is in the audience I've asked her to attend today if there are potentially questions around epi net. Epi net actually is main strategies, I believe aligned with many of ours the coordination for the facilitation of data sharing in this case. Epi net took 101 existing early psychosis clinics and and galvanize them in the context of a national data coordinating center. Driven the curation and harmonization of various measures and data elements. There is a uniform or or attempting to have a uniform healthcare informatics strategy. There's emphasis on practice based research and of course the dissemination of their findings to a broader clinical and participant community. The standardized the the coordinating center in particular has enabled the standardization of measures. The unification of informatics approaches and the mechanisms for sharing rapid tools and they've created a culture or at least aspire to create a culture of collaborative research with participation between academic researchers as well as community practices. I think their goals also align again with ours and delivering more personalized and precise treatments to do quality improvement projects in the health systems that they're interacting with to do rapid piloting and innovative approaches and also to enhance their statistical about the power to detect rare. So, I've asked Susan for some information about what has happened since 2000 2019 since you know the pandemic, not withstanding but they've been successful in implementing a core assessment battery across the network which again has the standardized measures being used by the clinics and and coordinated by the by the data coordinating center, they've initiated a variety of quality improvement programs and shared learnings. As I, as we've been discussing the patient self reported measures are now being implemented into routine care, and the research community has identified gaps that that they're using that gaps and research that are the basis for further proposals for funding. And even some of the outcomes measures to date have influenced the decision making of local state mental health commissioners so this is quite at least, I think, positive, and I think it we should be optimistic that that a national network could have an impact. What would we do in the genomics community we've already identified a number of amazing stakeholders that have taken this on, many of which have shared their systems with us in this meeting, but the other stakeholders that we need to engage in so we talked about earlier today and the patient advocacy groups foundations, even industry industry groups that are providing genetic tests, and the and the CDC, it could be that the nature I could see the coordinating center, whose goals would be to share data with and gather data from genomics medical clinic clinical genomics programs nationwide, and to support genomic medicine in practice quality improvement and benchmarking. So, what would a research platform look like this is the left hand of the virtuous cycle for a genomics learning health system. Of course I'm a little biased that perhaps the all of us research program is one possibility to be that platform as you know this is longitudinal population study aiming to recruit 1 million people from across the United States. We've consented our 500 thousands participant in June, and the core of the program is in a is a public an accessible data platform that we call the all of us research workbench or all of us research these are data from from June of this year, we have there over 370,000 survey responses physical measurements on over 300,000 individuals electronic health record data from over 250,000. So this March we released the first tranche of genomic data with 165,000 arrays and nearly 100,000 whole genomes and there's also wearable technology data that's available to research on the workbench. As we discussed yesterday about diversity in the gap in genomic data, data sets I want to just emphasize the commitment that all of us has had to reducing this gap. In fact, we're intentionally over representing minority populations in our cohort. Data we're recruiting from all 50 states in the United States with a larger footprint of enrollment and in the southwest and some of the northeastern states would help to make that more homogeneous for the middle of the country, less than 50% of our participants identify as white 80% are from underrepresented populations and biomedical research and you can see some of the categories on the right hand side that that that we use to classify this in terms of age races and ethnicity sexual orientation income geography brawl versus urban etc so so really this could be an amazing potential resource for for discovery findings as it pertains to the underrepresented populations that will be served by some of the health systems that we've been doing for the last two days. And of course, the vision for the all of us research program on the left hand side is to accelerate accelerate research and health. Sorry, accelerate health research and medical breakthroughs, and on the right hand side, enabling individualized prevention treatment and care. So the, the all of us program effectively embraces this virtual virtual cycle as its vision. And from my perspective it makes it a learning cohort, driving mechanistic insights from clinical data that we're gathering and driving those back to impact for its participants. And just this month we began, we initiated our health related return of results using the ACMG panel as well as seven pharmacogenes and really excited about the forward movement of our data moving into the clinic. All of us could be a research engine for a national genomic enabled learning healthcare system by virtue of its scale and its diversity and accessibility could jumpstart research findings from a from a network of learning health systems and also gather new data from such a network. So this is potentially a view of the big picture today across the United States. We have a genomic enabled learning healthcare systems across the nation, they're operating independently. Arguably that many of them are siloed although we heard some great examples early today from the VA as well as from Intermountain Health. But there's an opportunity perhaps to create a future state where there's national coordination and that's the thing that I have sensed over the last couple of days that we're really lacking is a coordinated effort to bring the learning healthcare system that we've heard about and really galvanize a community that could actually do things together than rather as independent operators, it would be bidirectional flow of standards and information, and a genomic learning health system. But you know will address several of the challenges we've we've talked about over the last couple of days, it will enable rapid learning and implementation and outcomes research. It will be enabled by standardization of quality and evaluation metrics, it will be enabled by the standardization of implementation frameworks, it should catalyze research on rare and common diseases, environmental and drug response mechanisms. And it could fill critical gaps in training that have been highlighted over the last couple of days so this could be a broad platform for trainees and more homogenized training if that's desirable. And it's a source of dissemination of knowledge of various types, it will engage providers who are super interested in delivering cutting edge genomic assessments and also delivering great equality of care, and it will engage participants, particularly with return results and other values. And then we talked about sustainability early today so the partnership opportunities with genomic and diagnostic companies, electronic health record vendors and as we discussed to an hour or so ago with insurers could be quite powerful. So to end, you know, looking at success through the lens of a scalable national genomic enabled learning healthcare system. If we did this, we would have hopefully see a vibrant community of national health system learning health systems implementing genomic information and integrating tools into their workflows. Health and data flow will allow us to provide outcome measures that will be critically important for more universal adoption guideline uptake and coverage decisions as we talked about earlier. This would also enable the flow of data to create an accessible research platform that enables novel findings validation of prior findings nationally or internationally and the development of a genomic knowledge repository that can be used by not only learning health systems but other health systems across the country. And importantly, the diversity of this is should be a top priority so it should be available to and serve all people. And in terms of sustainability, I think we all agree that the ultimate goal for genomic medicine is that it is medicine and not a separate, a separate area for for both insurers providers patients and participants. So with that, I will thank you and happy to participate in the panel discussion. Thanks again. Great. Thank you so much, Jeff. I don't see any clarifying questions in the chat right now so I'm going to take this time since you mentioned Susan as we're in by name, and if she's here, I would be happy to give her the opportunity to weigh in and provide some additional perspective on epi net. Susan here. Can you hear me. Perfect. Well, thank you so much. First, Jeff for inviting me and reaching out to me. It's, it's really an honor to be here I'm, I am a clinical psychologist by training my area is mental health services research. And I am age. So this is an entirely new world for me in genomics, and I'm really learning a lot. And even though we have launched our, our own learning healthcare system around early psychosis. You folks have some fabulous ideas and have gone into some, some pretty sophisticated stuff in some areas that that we haven't. We haven't got to yet so I'm learning so much about the genomics and it's informing my ideas on learning health system. So let me say that a couple areas that especially resonated near Mark, Mark Williams talk and Carol Horowitz talking about the stakeholders and the importance of their engagement and how to engage them and I, I agreed to, you know, with I think everything they said and couldn't emphasize enough the importance of engagement at, you know, multiple levels, both the local level, engaging patients and providers in the actual settings, where you're deploying your learning health system, as well as the national level, where you want to engage advocacy organizations, payers, and, and, and so on. And our epionet grew out of a very successful National Institute of Mental Health Initiative, involving a team based multi element treatment for young people with early psychosis. And the effectiveness of that strategy and the stakeholder engagement among payers and advocacy groups and many others created this, you know, national buzz, essentially, that led to huge congressional funding for these early psychosis treatment programs across the country. Like before our, our initiative, there were a handful of these programs and then with the effectiveness demonstrated through a clinical trial of this coordinated specialty care for early psychosis went from a handful of programs to them hundreds of programs across the US and that is what created the opportunity for epionet. There are all these excited motivated programs if we could bring them together into this network and you know leverage this tremendous expansion. We could create a learning health system. So it's sort of like it both started at the grassroots with these small local programs, but it never could have happened with all without that engagement at a national and regional level of the advocacy organizations of payers of our federal partners. So, this, this idea, I think if I'm articulating it correctly Carol. You know, we want to be with them, you know, learning with them, making decisions with them, not just approaching them. And you tell me that that might be might be helpful. And the, I think the strategy that that we tried to use that that was effective is the first start out with what are their major problems, which hopefully you have a little insight into at the start when you talk to them, and then so rather than selling them on the idea of your fabulous learning health system from the start, it just, how is this going to help them, right, with their, with their problems and, and it was very different when we're talking to the Social Security Administration, or the substance abuse Mental Health Service Administration, or the National Alliance Mental Illness they all have very very different needs. So we can start with that as the common ground. And then, so of course that what they shared about their problems helped us inform what needed to go into the learning health care system. And, you know, we're, we're the NIH, and, and so we're like a knowledge generating entity, you know, we, we're all about scientific discovery at every level. What is the knowledge that our learning health care system can can produce that there's going to be actionable for decision makers. Right, right. As we've seen from other presentations and the beautiful slides of learning health care system, learning, learning healthcare systems are so many moving parts, because we are goals are to generate the information that improves patient care at the individual patient level. So what, you know, what's actionable at that point for the patient and the clinician, and then what's the information that's actionable for healthcare systems who make, you know, decisions, sometimes about what to pay for. So what was services job or what information, you know, is is actionable for payers for insurers, I'm part of a whole another work group that is focused on financing of this intervention coordinate specialty care for early psychosis because it's the, the coverage of it by, by insurance in the uneven and public health systems and, you know, for the most part not covered at all by commercial insurance. So I, I, I think I'll stop there. That's great. Yeah, no, that was very good. And with many other thoughts that have been elicited from these wonderful presentations but I'll just stop. Thank you for that opportunity. Absolutely, absolutely. And thank you Jeff for inviting Susan and for sharing this with us. I mean it's clear that we don't need to start with blank sheets of paper for any of these things that we should be looking for where there are examples of systems like every net and so that's really helpful. So I'm going to, we only have a few minutes now we went a little longer for the panel discussion but I do want to make sure that I get a chance to have the panelists respond to some of the questions that have been asked by our attendees. Okay, so the first one I think is directed at Heidi, and it's from Mary Relling and Heidi, it says great. Well, besides, she's complimenting you she says I think some folks think that we can use such individual reports to generate new pharmacogenomics data for common feed and types and this would require a lot of caution and is not the best way to go. So I was just asking I think about sort of the applicability of some of your recommendations for germline diseases around pharmacogenomics. Yes, it's a great question I did answer in the chat. Okay, do anything, but but I think it's a valuable point in that when we're dealing with more common phenotypes, or common variants, where you're finding these variants and both, you know, the general and in your test population, then you really need, and the phenotypes are not, you know, it's like breast cancer, you know, lots of people have breast cancer. So the prior probability of the association doing a variant in that gene and what's known about the gene to phenotype relationship is not tight, it's not specific. In those cases, you really have to use a well run case control study. That's certainly true of pharmacogenomic associations. And that would also be true of a lot of the cancer genes other genes associated with more common rare phenotypes. However, in a lot of rare disease, you know, about over half of the diagnoses we make the variant is actually unique to the family, and is absent from the entire global population or at least all those sequence which is not the entire population. So we can, we can more easily use that data when we're talking about rare or absent variants from the general population, and use that gene to phenotype relationship to inform the evidence but but it's a very different game than in pharmacogenomics. Okay, great. Thanks. Thanks for that answer. I see another question here for Mark, and Megan Haley asked me Mark that, although she's supportive of the idea of the need for patient engagement and patient informed outcome she's concerned that sometimes the outcomes patients care about aren't necessarily the care the taxpayers are willing to pay for. For example, in a rare disease, a diagnosis doesn't necessarily change outcomes but can be hugely beneficial to the family. How do we address this disconnect. Yeah, I think that's a really good point. And in full disclosure I did spend five years as an associate medical director of a provider owned health plan so this is a question that is actually quite practical and and the range is very large. We see things like people that have some type of a rheumatic disease that says we would like health insurance to pay for a hot tub. There's no question that the hot tub would improve the quality of life but it's not something that would traditionally be considered a health insurance. Now that's kind of an absurd example but it gives the idea of what's the balance between personal utility and utility that is of enough value to a larger population that it could be considered to be covered under insurance. And I think the one that the example that was used by the person asking the question is spot on. I really am mystified by the reluctance of payers to pay for diagnosis because to those of us in medicine diagnosis is the foundation for everything that we do. And I would argue that we're probably the only specialty where we're being scrutinized because we're trying to make a diagnosis and we're not paying for that and some of it has to do with the fact that we're Johnny come lately and that our tests up until relatively recently are very expensive. But the reality is that, you know, and I do this when I present to insurers as they say look, most insurers, or many insurers aren't prior authorizing MRIs, and yet if you use my MRIs for developmental disability nerve developmental insurers, the diagnostic yield on them is less than 5% and it almost never changes the treatment. Yet that goes through whereas an equivalent test of genome sequencing that's about the same cost as a diagnostic yield of 60%. If not more it changes care. So we've gotten ourselves into a dilemma regarding this diagnostic realm and I think that's an argument we have to continue to say is that this is foundational for medicine. You can't do a treatment without a diagnosis, but you can't also operate or you say that well, because we don't think there's going to be a treatment we shouldn't allow an attempt to diagnose so yeah much better, we need to do a much better job of communicating on that and we need a much better range of treatment so that we can actually have more things that we can offer our patients. Does anybody else on the panel want to weigh in on that. I'll just emphasize what what Mark's points are and say that the cost that is continually incurred by these families for non genetic evaluations over time for things like MRIs and many metabolic panels and different specialists neurologist cardiologist you know they're trying to get any sort of answer and those services are being in many cases, even though the genetic testing may not, but the ability to end that diagnostic odyssey that is incredibly costly is should be viewed as an economic game. Yeah, and one of our speakers and maybe it was you Mark, or another person I mean all the work that was done to talk about the cost effectiveness of genomic sequencing in the NICU. We're really essentially trumpeting the diagnostic odyssey and avoiding that long period to diagnosis so that there could be a decision to treat or withdrawal treatment, etc. So I think that is an example of work and pairs are caring about, you know, avoiding the type of diagnostic odyssey at least with these acutely ill children, babies. And I think Nancy probably has some interesting percept insightful I think was the word that she used for me so I will return the compliment. Well, if I may, I mean, I agree with everything you guys have said, and I think it's also their implementation barriers. You know, United Healthcare pays for exome sequencing for anybody really that doesn't have a diagnosis. But often it's not done when a patient is in the hospital because the bills are part of a DRG and the hospitals won't allow people to do the testing. So it's a yes and is what I would say. I also think we need some health economic studies that show that ending the diagnostic odyssey, you know, produces morbidity mortality and saves costs. And then finally the other thing I would add is, there is value above and beyond an answer right and saving costs there's the value to the other family members there's a value to connecting people with community there's a, there's, it goes on. So, and it's also an equity, you know, people who are on a diagnostic odyssey are often people with developmental disabilities or have physical disabilities, and it's an equity issue. Right. Okay. There's one question here from crystal associate and I'll ask Jeff to take it initially and then if others want to comment. She's asking how much information related to indigenous data sovereignty is imparted to providers of potential indigenous research participants interested in all of us. Yeah, thanks for that question. I'm, I'm, I just want to make sure I understand it as, as you're asking it could you just, would you mind asking the question live. Sure. So, I encounter a lot of indigenous tribal members who may want to enroll in all of us program or other research studies, and they are requesting their providers to sign off on HIPAA releases to enable the sharing of their HR data with with the program and other groups. And for members that are clearly from a single tribal nation that request obviously goes filters through their community, but in particular for care related to IHS but our Indian health services. The question is related more for urban indigenous patients and research participants, especially considering that that's where the all of us research program is largely recruits as urban based Native Americans, and I just want to have an understanding of the pathway by which the sharing education and indigenous data sovereignty is imparted at all these different points in which a provider who wants to recommend just any types of genomic care or participation these studies is actually imparting those data sharing risks on to urban Native individuals. Great, great question and I am a call on Karim who's here from the program to help me with the answer but first, we do have a significant number I don't know the exact number of American Indian Alaska Native participants in the program today. We've been very careful and thoughtful about the privacy of their data so they may share their data with the program right now but it has not been released to the research community, what, who are indigenous people and who are not. That is a process we're going through right now to make decisions about releasing that data and how we protect the privacy concerns of the individuals involved. And Karim, do you want to comment on Crystal's question about the provider, if you know the answer. Sure, thank you so much Jeff and thank you so much for that question crystal as, as, as Jeff said right now we are being very cautious first I want to note that before we did any AI and engaging we actually completed tribal consultations with all the leaders of all the tribes so all the engagement that we're doing. I'm sorry Chris you shaking. Oh it wasn't with all the tribes you did engagement at six different sites with with that wasn't all 574 travel nations. Well thank you with the tribal leaders excuse me with the with many of the tribal leaders and we can I can follow up with you on that from that tribal report. And that from that tribal report we have been intentional about not doing first of all any recruitment on tribal tribal land and any engaging we do with AI and participants. who are free to enroll in the program and if they self identify as AI and we've been very careful to not release that information until further guidance is come out in terms of that data release. Does anyone else a crystal do you have your hand up did you want to comment on that. Yeah, no I appreciate the pointing to the report I read the report the question is not about release of data. The question is more about collection of data. And yes I understand that there's no recruitment efforts specifically on tribal lands. But I do know a large recruitment site is in for instance with you obey and Tucson Arizona, which does that medical system does care for a lot of urban native individuals. So, in when that they are seeking care at this site and also other banner health sites for instance in in the region. And are informed about the program from a participant viewpoint. If they're interested they're also asked to provide a HIPAA release from their providers. Now my question is how much provider education is given in terms of imparting those data sharing risks and. And also risks related to re identification, due to being a member of a small minority reidentifiable population. Great, I better understand your question now thank you for that. We're doing some we could definitely do more in that space and but one of the things that's why the that's why I tied my answer to data release, because it's that data release that could allow that re identification. And so that's why I tied it to that please. And so I can address it and I think there was a another question in the panel about that, but the most important pieces that the release of that data is the holding that up because we want to make sure that there's enough providers sufficient provider training that goes along with that. So we're doing something we could do more. Okay, great. This is a really important conversation and I hate to be the person that has to be the timekeeper as well. But we did want to make sure that we left time for discussion of the solutions as I mentioned. Although all of us are getting sort of tired I'm just going to encourage if you could just stay to the end because this is where hopefully we're going to take all of your great input and make sure that we can actually come up with some really good recommendations for for solutions to move forward. So the format that we're going to use is Terry is going to go through in chronological order, the takeaways from each of our panel discussions with really just you saw some of this from the first three sessions this morning you haven't seen it for the last for the today's sessions, but we're really just going to ask you to weigh in and say, did we miss represent anything that you think is an important potential solution or did we miss anything that's a potential solution so I'm going to ask Renee to help me monitor the chat and hand raising, because Terry is going to be going through the slides. Great. Thank you so much Pat and thank you everyone for for your comments and I know it's it's late in the day I only have 40 slides so it shouldn't. Now you can see how many slides I have. I'm going to leave it in this view. Can you see the gentleman LHS day one brief recap. Yes. Great. And of course I didn't fix that so brief recap of both. I've seen these points before one of the things that that we thought would be helpful would be to try to highlight some of the places among the solutions, where we might be able to develop some collaborations and so that's what this teal like cyan I guess it is board of bold print is about. And yet, you know, before going into that just to just let you know that solutions that that we kind of pulled out of this and many thanks to Renee and Pat and to the moderators who sent me their their notes. Again, some potential solutions in the lane the ground work, you know, setting up gentlemen learning healthcare systems, the data donor culture is something that we continually work on what could do better at the dashboard of clinical management steps is a is a real step forward, particularly compared to where we were in 2015 and seems like something that we should try to expand and share with other groups, because it really is critical to effective implementation and uptake within a healthcare system, improving integration of measures of structural discrimination or structural racism that goes beyond racism to other forms of discrimination and social determinants of health. This is something I think that everyone agreed on and of course sharing educational content across organizations and we do have some efforts at that I had mentioned the ISCC for professional education and genomics but there may be others. So with that, let me ask, do these capture at least some of the solutions that we heard are there other critical ones that we'd like to add or you want to modify any of these at Renee please moderate that. I'm not seeing any hands raised. So it looks like people are pretty much in agreement and obviously they got to see them earlier today as well and maybe think about them. Okay, great. All right, I'll go on then again on the IT infrastructure it seemed as though genomic health information exchanges to add on to health information exchanges might be a place where collaboration would be possible. We're estimating data standards which GA4GH is doing quite effectively as well as other groups and then another place for collaboration expand or extend the interoperability studies that have been shown that Peter described. So any data sharing and expanding, expanding training, but do these two areas genomic health information exchanges and expand interoperability studies do those seem like logical places for collaboration. We'll all speak at once. So of course I can't see who's who's raising their hands. No, yeah, I will definitely let you know if somebody raises their hand. I was just going to say well since I can't see if anybody's raising their hand and there's silence. Mark just raised his hand. Oh okay great Mark. Yeah thanks. Just relating to the genomic health information exchange I would just include that there may be some policy implications about genomic data. It's a lot of it of course is informatic standards, but there may be some specific issues relating to policy and regulation that would also need to be incorporated in that. And still Mark. Okay great. So the interoperability issue, I might ask those who are are heavily involved in medical record electronic health records and that is this something that can be done in other settings we've tried to do it in the emergency network for example in other places. Nobody's nobody's going to stick out their neck on that one. So, this is hard to do in a format but we'll do our best. Okay, you know it's one thing to it's a lot of standards are disseminated I think the thing is to get them adopted. Right. This, because you have 15 different groups you'll have 15 different standards. So, I, yeah I think they're at least on a handful of standards for how to organize and annotate and structure these data is going to be important. So, yeah, no thanks girl that's an excellent point I remember that the former deputy director of the NLM one when Betsy Humphries when we were talking about standards and she said you know the problem exactly what you said the problems we have too many when we say, you know we need to adopt standards, everyone says, Sure, they can use ours, you know, exactly right. Yes, yes, and you're right back where you started. Exactly. Yeah, but there are some opportunities for collaboration here because as we've heard. We have the HL7 genomics working group and GA4GH that are developing normative standards that could be incorporated so we don't want to be the one that's developing our own standards I think we should take advantage of those. That's right and I, but I think it's important to, to recognize those ones that are mature enough and robust enough to be adopted widely versus what's what are the gaps for standards that we still have where there might not be something like HL7 in place. And then we also have a comment from Nephi. Can you hear me okay. Okay, so I was just going to say, I think one of the things we tried did a lot of when I was at Geisinger previously did a lot of work in this area and I think we need to identify all of the stakeholders in terms of who where this data is going to be used. I mean ultimately I'd love to have people's genomic data on their smartphones so we can go to the other system we're having here in our mountain health care we're right next door to University of Utah. We share a lot of patients we don't have a way to share genetic data, and so identifying the different stakeholders which is you know your laboratory systems your EMRs, and you know even ways to share that with a patient and then engaging them in some way that encourages them to adopt standards. I also know that one of the things that has held up the adoption is worries about maturity. And the problem with even the standards groups is, is genomic data is sort of a it's not necessarily static and that I think if we try to model too much in the standards, then we'll never get there. We need to set it at a granular level and then say this is where we're going to go and move forward but identifying the different stakeholders and then having some kind of incentives to to make them adopt the standards. And those groups to make sure that they get them in a state that those stakeholders feel comfortable I don't know the best way to put that, to put that on into the, to the slide but those are the problems that I ran into. If that makes sense. Very much so thank you ways to incentivize adopting standards. That's about what you were saying. They may have cut you off. You may be back into the silent. Okay, hopefully that's what you were saying. And, and we will find ways to make these available so that you all can can suggest. We will as we're doing this on the clock. So, I also think that Rob has Rob rally has his hand raised. Yeah, I have a question about a wording. I know I don't want to work Smith as much as that integrating and replacing that with exchange genomic data among health care systems I kind of hear that from defy and mark is less integrating just seems like an end point of information. Well so it was integrating into care. Yeah, I guess I'm saying it's genomic. Yeah, because I mean that at the same time is that information might go to the health care system but might also go out. I think care into care seems like you're just trying to get it to the point of care and when there's a lot more than going on to deliver it. Will that work. Or you just want to get rid of integrating complete because it seems as though that's still an issue. I think that wording is good I think you have both integrating and exchanging so I think that's that's good. Thank you Rob for that point. I'm also noticing in the chat that because these are all complicated issues Jeff Ginsburg is saying it's that it seems like these areas need driver projects Jeff to say a little bit more about your definition of a driver project. And I'm taking the term from GA for GH, which is heavily involved in the, in some of these standards but I do think to galvanize the community to, to change the way that they're currently doing things or to do new things requires incentives and things like NHGRI may be in a position to deliver that incentive with a with a, you know, a project that could bring a number of systems together that currently aren't working together. Well I see that Bruce Korf has his hand raised Bruce were you going to comment on on the second one on it infrastructure. Yeah, I was actually just going to ask the question, although I know our focus is on healthcare systems. Should we be recognizing that even now a fair amount of genomic data is obtained outside the context of formal healthcare systems and I would bet in the future there will be. And it does there need to be some way of integrating data obtained from various sources into the healthcare system and it ultimately could empower patients much more than they currently are, in terms of having control of their genomic data. Okay. Good point. So, and I am conscious of the time and wanting to give people out by five but that may not. Okay. So let's go back to being the time keeper police healthcare. We have four more session for more to go through. Let's go to health equity and access. All right, so. So just again focusing on some solutions and areas that we could collaborate on try to obtain clinical data more efficiently in, in or through HIV ease was one suggestion for a place we could look to collaborate. To find ways to share and develop effective engagement plans, probably something that lots of groups could use to help on the people agree or disagree with those are important potential areas for collaboration. No one has their hands up. You know, people are probably thinking and reading and which I do understand. Although I just want to comment that Sharon Terry and Sharon if you want to weigh in yourself, because you're, please, you know, raise your hand and we'll unmute but I think you're just making some very important points about using people as a central unit, not the data. And don't calling people data donors is Sharon are you there. She said no need to speak. Oh, okay. All right, all right. I think it's very this is Devon McGraw I think it's very consistent with Mark's presentation. Right earlier in this session. You know it's people as people in communities as genuine participants and not just opinion givers or consultants or people that things are done to. Right, right. Okay, great. All right, Terry so I think anyone else on the equity and access. Okay. Great. So these may need a little more work because you haven't seen them before. So just included a couple of key points. There was a lot of discussion or a lot of interest in the Mount Sinai internal medicine curriculum we've captured those questions and we'll share them with Nora and her colleagues and hopefully can get responses back that we will post on the meeting webpage which I'll show you at the end. But again, if you just Google NHGRI general medicine meetings should come right up in terms of solutions. We heard some interesting approaches to a consult service at Vanderbilt, although it's very time intensive potential collaborations could be developing implementing and assessing the impact of both the training tracks that we've heard of and consultation approaches. So, so that might be an area where some implementation research could be tried. We heard a lot about focusing genetic counseling efforts on post test rather than pre test counseling, but we didn't hear a lot about addressing low resource settings in that situation and that may be a place where some collaboration could be made. Unloading automatical and basic functions from providers seem to be a good solution although difficult to implement establishing templates. There are some out there and it would be nice if those could be shared so that they could be possible implemented in other systems although the systems are really quite different. The idea of monthly case conferences or boards was an interesting one is that's something that we could do some collaboration on. We have seen that the Dyson or MyCode group does have a conference that they share. Baylor for a long time was sharing theirs as was Stanford. There seemed to have dropped off or at least maybe they dropped me off of their listservs, but at any rate, is that something that we could consider as a, you know, a place where practice could be moved forward in a more general way, recognizing there are issues when it comes to sharing patient data. Sharing quality advances and I did love the point about let's not compete on quality, we don't crash as much as the other guys. And then there was also mention of the access to genetic counselor services act as federal employee I can't suggest advocacy and just mentioning it here and other potential solutions for genetic counselor access. Comments from the group on news. There is Cynthia James has her hand up. I would say and while I didn't cover this in my talk there was a lot of conversation in the chat from our outlet health professional colleagues so in addition in the key points related to medical education I think certainly education and nursing and PAs, and so on, maybe worthwhile. As a key point. Yeah, there's been a dirt sense a niche peg went away in terms of some of the curricular developments there that I think have languished and I think also there's a potential role for the inner society coordinating committee to look at this I think that was also mentioned in a chat comment. I think there is some, some active work going on in that in for positions assistants if not for other, in fact, they're, you know, long ago, and HGRI did did sort of spearhead through Rocky Rock River and other, other folks, a series of tracks for for different kinds of providers. And if I has his hand up again so I'm going to unmute him. And I'm going to direct this a little bit towards Mark but at Intermountain for heredity and we've formed variant review committees for the different domains and I know there was a lot of that that what happened at Guy Singer as well. And then I know there's also all the stuff that's done through Clinton and I'm just wondering when you talked about ways to sort of distribute or collaborate on these types of things. So that's an area, I might just throw her to you Mark to think about, you know, how can we, we've all got these these things going on how do we make them work together a little better I guess. Yeah, and I think that there is some, you know, clearly clinch and has some funding from NHGRI and there are a lot of the variant expert panels and that that would reduce the potential to do some public facing activities related to that. And I had also put in the chat that I think this is something the American College of medical genetics and genomics would be potential interest in exploring. So I might just just jump in and point out the reason we started ClinGen Nephi was was exactly what you're saying is that there were there were 20 at the time we knew of individual groups doing exactly this, and you know much the same way coming to the same conclusions and so that's why we have ClinGen and if there's a way to get more folks under that tent, recognizing that it may not be fast enough for an individual, you know, clinical provider or institution maybe those are that's something that ClinGen can try and tackle how to address that. You know that sounds good I mean that yeah the challenge for us is that we we have kind of a timeline to get the results out to the patient so we can't. But we're putting a lot of effort into these interpretations and so it would be nice to make sure that everybody gets the benefit somehow. So I'm just going to ask Renee. Is there a way that we can save the chat because I think given the time we're not going to be able to address everyone's very excellent input and I don't want to miss any of it so if there's a way Renee that we can save the chat that would be automatically saved. Okay perfect. Unless you clear it. Yeah okay so I'm Terry I just, if we have 11 minutes and I know we have two more panels to go to and then I did want you to make sure that we have that opportunity just to say sort of what the next steps are going to be. Absolutely that's those are on those slides as well. So session five. So you can see the key points here in terms of collaborative solutions. If we can work together to demonstrate economic gains as well as improved outcomes from implementation and we do have some programs going on in this area. And HDI is a small institute and we can't fund it all but if you can, and sort of leverage the work that's currently ongoing and try to bring these groups together that's something we can do pretty well. And so that's a lot past, sorry, Nancy's idea of adding a third cycle on to those two virtual cycle so really intriguing one and we ought to think about what that would look like for payers policy makers regulators would be important in that third cycle as well. There was a broad set of discussions around Daryl's presentation of the novel maturity model which which really was deep defined outcomes and several people commented and presenters as well on to define and measure outcomes. And while that was a very useful and very interesting analysis that was done across 150 settings, it would be interesting to kind of look at what I referred to as the sort of clinical validity to others who might be using that information. So, so do these measures, you know, bring true with clinicians with system leaders with payers, etc. And then also the suggestion that these be applied long to to the way to assess progress that patient satisfaction measures like in that promoters for clinical utility studies here advisory group needs to be engaged we have tried in the past to engage with payers it's been five or 10 years since we've done it actually we did it 10 years ago and we did it five years ago but probably time to do it again so so that would be an area that collaboration would be helpful. Following patients in their outcomes across insurance plans and health systems, which is difficult to do health information exchanges may help and real world evidence is critical to demonstrate clinical utility and the value of clinical medicine so generating that real world world evidence on this on this. That's great Karen. Thank you. Okay. Yeah, I think so. I think as long as we I think most of these comments are refinements nuance and as long as we can incorporate it afterwards. I think that it's best to move on. No, that's great and yeah pulling them in the chat is probably the most effective way to get in there. I'm going to skip over the key points and really because there were a number of solutions discussed in the last session. And I think we heard from from Heidi how crucial it is to have dynamic iterative interactions between clinicians and labs for interpretation and diagnosis we heard this Carol bolt may remember it are at our ninth. You know with medicine meeting about how important that interaction is. And so, finding better ways of doing that would be very important. An ambiguous genotype representation and standardized data storage we've heard this before as well as meaningful, standardized data collection and there are some standards that are, you know, started starting to cook their heads about so they're not 20 anymore than maybe, you know, for willingness and infrastructure to share individual level data globally we don't have great infrastructure for doing that although as you heard there are four or five different sessions I was intrigued that Heidi didn't mention which is another way of doing that but in the UK and Canada and Australia. And multiple node, no, noting that maintaining a high level of patient engagement in design in partnership and carrying out the research, etc. and analysis and interpretation leads to really substance of changes in care processes and structural outcomes and so. So that is something that really has to, has to be part and parcel of this work and it's not something that we've done terribly well. In terms of research opportunities. One thing to that could be done collaboratively collaboratively is to look at the impact of patient engagement on effectiveness or outcomes in genomic medicine that has not been done in this field but it's been done in other fields. I think there's defined patient informed outcomes for genomic medicine. I'm going to go back and look at girls. Eight measures I'm not sure they were necessarily patient informed outcomes and as Mark said, right now outcomes tend to be defined by the care providers and health systems. Jeff had suggested creating a national learning health system network so a network of all of you in some kind of a coordinating function which is something that that the NIH can do. Include stakeholders in that process and have them have a meaningful role and consider studies of the value of ending the diagnostic odyssey we have considered these and are actually nothing. So the undiagnosed diseases network is looking at this as well as decipher and is likely to address comments on this. So Cynthia James did put in a comment. But do you have something you'd like to say Cynthia. No, but particularly urgent the chat is fine. Okay, Erin has her hand up. I'm happy to summarize it if you'd like me to but know what, sorry Cynthia but if you put it in the chat and you're good. Why don't we. Sorry. Can you hear me my my comment isn't. It was more of a word missing but where we say defined patient informed outcomes might be defined and and catalog. I think we talked about this a little bit previously but it would be nice to have a collection of standard outcome measures that we can use as a community. On the third piece of that is to require utilization so like the promise measures. You know, there's a lot of funding announcements that say you have to use the promise measures or develop new ones that meet the promise standards and that would be deposited so one and Phoenix Aaron being you know being one of the key founders of Phoenix that might be a place as well. Anything else on this one. No, we have three minutes left. Okay, and I want to thank that a seven of you who are still on. I am impressed so alright so our next steps from this meeting we we at NHG or I particularly and many many thanks in advance and and later to Johnny be a new ruler and Ellie same or who will work with. Renee and me to produce a draft meeting summary and an executive summary in the next few weeks will share this on the GM 14 website and provide a way for for everyone to give us comments but we're not going to be in a position to have sort of open editing or, you know, forward smithing or that sort of thing so we'll work out if you see something that is like just really egregious will will find that change that change that and then if warranted will draft a white paper for publication. These typically include the moderators and the presenting presenters, but you have to meet ICMJ ICMJ criteria, so at a minimum you have to review and respond you've already contributed but so questions about the next steps if not I can wind up and then Pat you can close this out. Okay, I just wanted to thank again. All of the presenters and moderators and also to remind us what I mentioned in the beginning, the Duke University group has been fabulous that's taking record for us and Pamela Williams. And John, I mean, and Ellie, who is an Ellie who are our record tours. And Gerald and his group who are making all of the it happened but I really wanted to be sure to thank Renee rider who was the driving force behind this entire meeting you all got many, many emails from her. I'm not even any more from you and many thanks to to Renee and my partner in crime, Pat, Pat, I will turn this back over to you. Okay, well thanks Terry so just I'm sure everybody I impress 80 some people are still on the line I just want to add my thanks to all of the contributions from the speakers panelists and participants. I'm leaving with a clear conclusion that we've made great progress since 2015 but we still have more challenges ahead, but with all the input and the intellectual power of the people on this call. I really feel confident that we are going to come up with some solutions that will be truly actionable and meaningful. So thanks everybody. Thanks. Bye.