 Okay, so welcome back everyone. We're now made it to the final session for the day. I'd like to thank everyone for hanging in there. So for our third session, which is titled Research in the Stakeholders Perspective, Enablers and Barriers that affect the integration of genome-based clinical informatics resources in the healthcare system, our two co-moderators for this session are Dr. Patricia Deverka and Dr. Siddharth Praytap. Dr. Deverka is an executive director at Deverka Consulting where she assists biotech companies and startups in evidence development and clinical adoption of innovative technologies. She has extensive experience with drug and diagnostic product development, reimbursement planning, cost-effectiveness, analysis and bioethical issues surrounding the use of new technologies. Dr. Praytap is the director of bioinformatics and also associate professor in the School of Graduate Studies and Research at Mary Medical College. He's also the adjunct associate professor in the Department of Biomedical Informatics at Vanderbilt University, where he directs the... Mary Bioinformatics program is core labs and his research focuses on implementing, connecting bioinformatics and biomedical informatics approaches for health disparities research. So I turn it over to the two co-moderators and take it away. Okay, thank you, Ken, and thank you, Mark. Is my audio okay? Just making sure. Yes, I can hear you fine. Great, thanks. Yeah, so really enjoying this discussion so far and thank you NHGRI for organizing this. We're getting a lot of just really great perspectives on this. I'm just going to introduce the three speakers we have for this session. Our first speaker will be... And please correct me if I mispronounce your names. Our first speaker will be Dr. Guillerme Del Fior, MD PhD. He is an associate professor and vice chair for research at the Department of Biomedical Informatics at the University of Utah. He leads the research arm of the re-imagined EHR initiative, which aims to transform clinicians' use experiences with EHR through standards-based approaches like using FHIR, SMART, and CDS hooks. He is the national leader and development of health IT decision support CDS standards. He's been doing this for over 20 years, including being elected the co-chair of the HL7 CDS network, CDS work group. And he's also an expert on the HL7 FHIR standard whose adoption is a high priority from the NIH. Guillerme is also a lead developer on several CDS standards, notably the HL7 info button standard, which is required for EHR certification in the U.S. Our second speaker will be Dr. Chun-Wa Wang, a PhD. Chun-Wa is a tenured full professor of biomedical informatics at Columbia and an elected fellow of both the American College of Medical Informatics and the International Academy of Health Science and Informatics. She is a leading biomedical informatics. She's been leading, co-leading the biomedical informatics resources for the Columbia CTSA since 2011 and is an associate editor for the Journal of Biomedical Informatics. Dr. Wang has published on data-driven optimization on clinical data trial eligibility criteria on scalable and portable electronic phenotyping and on EHR data quality assessments for data analytics and technology engineering from a variety of tech sources, including not just the EHR, but PubMed clinical trial summaries and so forth. And our third speaker today will be Dr. Mark Hoffman, PhD. He is a professor in the Department of Pediatrics and Biomedical and Health Informatics at the University of Missouri, Kansas City School of Medicine and a research professor of pediatrics at the University of Kansas School of Medicine as well. He is also the chief research information officer for Children's Mercy Hospital and the Children's Mercy Research Institute of Kansas City as well as being an inventor of 22 patents and also a member of the American Academy of Inventors. Prior to joining Children's Mercy Hospital, Mark was the AVP of research for genomics at CERNR heading a lot of their initiatives on the public health and big data research. And Mark's current research focus is on the use of massive de-identified EHR data sets as a resource for evaluating real-world clinical processes, digital phenotyping, patient trajectories and outcomes. And he's also the PI on a project examining trajectories for pediatric leukemia patients. His team works closely with the Genomic Medicine Center to support HPC platforms needed to accelerate genomic discovery. So with that, we have a really interesting docket coming up. I will hand it over to Guillermo Del Fio. Okay, so we will talk about a really important aspect of clinical decision support, which is a clinical workflows. We often talk about the importance of having access to high quality structure data in computable format and as well as access to logic, CDS logic in computable format. But oftentimes we overlook the importance of clinical workflows. And I would argue that you can have the best data, the best CDS logic without matching the clinical workflow, CDS tools are very likely to fail. We're also gonna briefly talk about interoperability standards in this area. There's a set of emerging standards that are bringing interesting opportunities for CDS, not only in genomics, but CDS in general. So we can probably categorize clinical workflows for CDS into two large buckets. There might be more, but these are the ones most commonly found in the literature, point of care CDS and population based. And I'm gonna talk a little bit about each. This scoping review on pharmacogenomics, clinical decision support, I use this as an example. There have been other studies, more recent studies, but this gives a good snapshot of what's going on. So they found 31 studies, all of them focused on point of care CDS, nothing on population based CDS. There's a strong emphasis on point of care alerts. Some work on provider inbox or delivery information to a provider's inbox in EHR. And also a lot of work on genomic test reports, this different visualization approaches for that. And some studies looking at entering automatically findings, genetic findings on the patient's problem list. Let's look at a few examples. Here, just to illustrate the different kinds of triggers that can be leveraged to present decision support to clinicians. Most common in pharmacogenomics are the pre-test and post-test interactions. And both are triggered by an event where a clinician is trying to prescribe a new medication. Other options beyond pharmacogenomics are found in family history based testing and cancer screening reminders. A really important emerging standard in this space that can enable all sorts of EHR triggered or event triggered CDS hooks. It's a standard that allows an EHR upon a certain trigger to call out an external web service that has some kind of logic and the service brings back conclusions into the EHR. So for clinical genomics, you can imagine a centralized cloud-based service that has clinical genomics knowledge encoded and updated as needed in serving multiple EHRs and multiple healthcare organizations. So you don't have to update CDS logic at each EHR at each healthcare organization in that kind of framework. These are just typical examples of alerts. Again, the most common kind of CDS that been found in clinical genomics. The one at the top is a pre-test alert. So when a provider is prescribing a medication, the alert is recommending the ordering of a genetic test. And the one below, a genetic test is already available and the alert recommends a dose adjustment for that medication. On-demand CDS, unlike active CDS, is used as the name implies on-demand by clinicians when they feel it's appropriate. Examples are test reports and info buttons. And relevant standards again are the HL7 info button which is required for meaningful use EHR certification. And smart on fire is also an emerging standard that's enabling the idea of an ecosystem of third-party apps that healthcare organizations can purchase or download and enable within their EHR. So that smart on fire opens up tremendous opportunities for innovations in terms of apps for genomics within the EHR and also patient apps. Most of the focus on this space has been on helping clinicians and patients interpret genetic tests and also provide guidance. Here's an example of a genetic testing report associated with guidance for clinicians. You see at the top a number of medications for which this patient might have problems based on a positive test for CP2D6. In this particular report, if it was implemented as a smart on fire app, you could imagine the report querying DHR for the patient's active medications and you could highlight those medications in the report making the report more personalized. In terms of info buttons, this is work that we have done in the context of the ClinGen project and Mark Williams was part of it and one of our students led the work. And as a demonstration, we showed how from the patient's medications list or a prescription for clopidogrel, you can, providers can click on an info button and that would link directly to FarmGKB which would provide testing recommendations and those adjustment recommendations for that medications. Population-based workflows, unlike point of care, happen in the back end. And the overall pattern is you imagine an algorithm that goes out, scans medical records for a number of patients, identify patients who meet certain criteria and then you can use patient outreach approaches to find, recommend some kind of personalized care and you can imagine things like people who need some kind of, would benefit from some kind of generic testing or generic counseling, the ability to update the test interpretation for people who already have a sequence and also people who need some kind of change to clinical management based on new knowledge. Probably the most important standard in this space is fire. So you need to be able to identify patient cohorts. They need to get access to data and that would be in a standard format using fire. As an example, we have this is NCI funded research that's supporting a randomized trial at University of Utah and NYU. And the idea is that we created a population-based CDS platform where we scan records of hundreds of thousands of patients looking at their family history in the EHR and trying to see who are the patients who meet NCCN criteria for breast and colorectal cancer. And this is a snapshot of the logic for breast cancer. So the workflow works like this. We run the algorithm, find people who meet criteria and then we use EHR capabilities to automatically send outreach messages to the patients who meet criteria. The message have a link to a chatbot, interactive chatbot that helps the patient understand the benefits, logistics and other issues related to genetic testing. And at the end, they're invited to do a test. If the patient's desire to do testing, they'll get a test kit on the mail. Once the results come back, a genetic counselor or a genetic counseling assistant will review and for patients who test negative, they'll get another chatbot link with a post-test chatbot that will explain the implications of the result. If the test is positive, the patient merges into a usual care workflow with a genetic counselor appointment and followed by looping back into primary care with a clinical recommendations moving forward in an update to the patient's problem list. So in summary, prior research in clinical genomic CDS have strongly focused on CDS alerts. That's the most predominant kind of CDS. And I would recommend that we need to go beyond that. There's many more different kinds of workflows that we should study and figure out different ways to deliver clinical genomic CDS that goes beyond CDS alerts. We also need to think beyond pharmacogenomics, really think about how to integrate primary care more effectively, ideally without overburdening our busy primary care providers. Also think about patient outreach and engagement approaches, again, that do not rely on primary care providers having to face all those alerts. And we have tremendous opportunities, I think, with CDS hooks and Smart On Fire to innovate in this space. And as almost every single talk, a previous talk touched on the point of health disparities, every time we've come with these new cool technologies, it's almost guaranteed we will introduce health disparities and widen the digital gap. So I believe the idea is when we design these new innovative tools, we think about health disparities from the ground up to try to avoid widening those gaps as opposed to addressing them after the fact. Thank you. Great, thanks, Jeremy. We have a few, we have time for a few questions. And let me just make sure I get the one here that just came up in the chat here. Besides complimenting you on a great talk, they said that now it's a point for a long time, I think was the only EHR vendor to provide support for clinical decision support system hooks. Now it's still two EHR vendors. Do you want to comment? Because it sounded like the CDS hooks were going to be an important innovation but it seems like maybe their penetration is limited. Do you want to comment on that? CDS hooks is an emerging, very new standard. It's still the maturity level. HL7 has all these scale of maturity for the standards and CDS hooks has the very early stage of maturity with a draft standard approved. As CDS hooks becomes more mature and evolves, it's quite possible that EHRs will, more EHRs will start adopting. I think we're going to see the same as we've seen with Smart on Fire, which now has been adopted by a larger number of EHR systems. Okay, great, thanks. And there's another question. As you're building this at University of Utah, who is responsible for ensuring that the CDS is current and consistent with guidelines? As it's likely that these are going to change over time and I guess sort of thinking inter-institutionally, isn't there the possibility that if each health system is building its own CDS system in the manner that you did, you're going to, what the result will be, would be care heterogeneity over time? Yeah, that's a great question. So at the University of Utah, maybe Ken Kawamoto is giving a talk tomorrow. My touch on this, but we have a CDS governance group that they meet regularly and they're in charge of reviewing the CDS logic and not monitor the CDS response over time to make sure we're not having any problems and also to optimize CDS over time. Many other institutions are following similar approaches. In terms of kind of reinventing the wheel at every single organization, I believe the key, especially for genomics, areas where the logic is complicated, it's expensive to develop and maintain, we're gonna have to be able to share logic somehow, either through cloud-based approaches where you have web services on the cloud that are kept up to date and EHR system access those cloud-based systems or you would still have a web service approach external to the EHR, but installed locally at your healthcare organization. That's probably the most common approach right now. There's significant concern from health organizations to call out external web services because you have a PHI data going over the wire. So, but regardless in both approaches, what you have is the logic sitting outside the EHR and being updated regularly by a central source. Okay, there's some questions that I think will be good for the panel. I just have one question from me, is as part of the workflow, are you also measuring effectiveness? How are you actually accounting for people acting on these alerts as this will be sort of a theme, I think that'll weave through and sort of making sure that we realize the value of these systems? Yeah, so that's a great question. Yeah, every CDS needs to be monitored over time because you have performance may deteriorate because of unintended consequences, any kind of change, changing logic, changing the underlying data, changes in the clinical workflow downstream. So, every CDS may have a different kind of measure of effectiveness. CDS alerts, you need to measure the response to alerts. Alert overrides are an example, but you also have to measure downstream in the workflow if clinicians are following a recommendation of the alert. They may cancel out an alert, but still follow the recommendation at the end of the patient encounter. Other kinds of like population-based approaches have a completely different kinds of measures. For example, you need to measure patient engagement. Are patients following, if we're sending any kind of outreach through chatbots, for example, we measure the rate which they actually engage with the chatbots and also measure their decisions. Okay, great, thanks. Well, thank you for staying perfectly on time. And I think we're gonna transition now to Dr. Wen. Okay, good afternoon, everyone. My name is Chun Hua-Wen. I'm from Columbia University. Thank you so much for having me. So in my talk, I will share our institutional experience of realizing the value of genomic decision support at the point of care and talk about the opportunities, challenges, and strategies for success. Before I start, I'd like to acknowledge the contribution of my collaborator and mentor, Menti, Dr. Jordan Nester. She's a system professor of medicine at Columbia University and also a fellow in genomic medicine research. And then she's really the driver behind the results and the studies I'm going to present at today's talk. So as we encourage clinicians to embrace and practice genomic medicine at the point of care, it's very important to recognize and support their on-med information needs. So here we identify some example on met information needs. Let's often ask among clinicians when we talk about genomic medicine. So for example, what genetic syndrome should I suspect in this patient sitting in front of us? And what genetic tests should we order? And how do we interpret genetic findings and what are the next steps in the patient disease management? When should we refer the patient? Who else in the family should get tested? And where can we find management guidelines? So in order to address these information needs, the most promising tool approach is to develop the clinical decision support embedded in the EHR environment. They are the most promising portable and scalable solution to address these needs and also fill the knowledge gaps for our frontline clinicians. So in order to explore the feasibility of developing a genomic decision support in our EHR environment, we conducted two studies as part of our Emerge 3 project. So next I'm going to briefly describe the first study. This study is also in collaboration with Dr. Mark Williams. So we focus on one disease, which is familial hypercholesterolemia. And then this is one of the top three genetic disorders syndrome identified by CDC. Its population problems is one in 500% patients. And then we took the 2018 guideline for early management of patients of acute ischemic stroke. And then we turned this 120 page guideline into a decision tree and embedded into our EHR process. So this is the first step is we select this guideline and convert the text information into a computer interpretable modules. We include in this decision tree module, we include a corresponding lab test that's available in EHR environment. And then the decision notes if the patient currently have a high intensity statin, then we will look at the most recent LDL level. And then if the LDL level meet the threshold and then we will continue with statin or maintain the aspirin. So this guideline was reviewed by one of our senior lipid experts, Dr. Henry Ginsberg at Columbia University. And then was agreed by our expert panel that the content is accurate. And then after we build this decision model, we conducted a small group scenario-based qualitative study to evaluate the usefulness and the usability of this decision support tool among our clinician target users. And then, and here we include some comments of our and feedback from our clinician testers. So basically the key finding is although they acknowledge the general usefulness of this tool, they alluded to a lot of unmet information needs. So for example, they may say, oh, I'd like to see a link to the full guideline in a tool which will provide the better context behind the decision recommendations of the tool. And then they also talk about the complexity like this doesn't work when the LDL is already less than 70 the users, it is not prompt to start high intensity study. And so these are kind of prompt us to think more about the deeper about the user, our met user needs. So in order to make this work in an EHR environment, we further implemented this as a dashboard in our IMYP EHR environment through a participatory design process by going through multiple iterative design evaluation and refinement of the user interface design. And David and Yiping are the technical programmers helping implementing this. And then so in this dashboard, we actually display the relevant clinical data of the patient with the diagnostic genetic finding of FH. And within the dashboard, we also show the physician support tool to guide the clinician, we are hoping that this tool can guide the clinician regarding the next steps for managing the patient. So in this process, we run into a lot of social technical challenges. So here I included three most significant ones. The first one is they are competing guidelines actually for managing patient with genetic syndrome. So we end up have to merge multiple guidelines with contradicting information. And then we also encountered barriers to integrate raw genomic data into EHRs. There were still a lot of controversies regarding what data goes into EHR, goes into a production system. So some information ended as the PDF files as a certified genetic tester results. And also the heterogenearity of IT infrastructure is the persistent challenge. So we, as some speakers mentioned in previous talk, we adopted a model of the peripheral CDS design building an interactive web-based decision support that can talk to our EHR system. And by the majority of data and knowledge base was stored separately from our production EHR environment. So that's not ideal, but it's our work around solution. So next I'm going to transition to the second study and then more details can be found in upcoming Jamie Open article, we just get this accepted. So I talk about the process of understanding user needs and also build it. Then the next questions, if we build it, will people come and use it? So we, in order to address this question, we conducted a log-based EHR log-based analysis. So to just using a non-intrusive approach to assess if the embedded genomic data will be used and then to what degree will be useful for clinicians. And then the finding is very striking. Basically, we find only 1% of all the e-merge three genetic test results in a EHR were viewed by clinicians who did not initiate genetic tests. So this is the breakdown of the users who will find to review or access the genetic data in the EHR environment. We look at, if there's any role difference, if there's any specialty difference, if there's a time difference, people working at certain hours were more frequently, more likely to look at the results. And also we look at the setting, like outpatients and ambulatory setting. And then again, so more of the results from this study can be find in the upcoming Jamie Open article. So from these two studies, we want to summarize the key barriers and challenges and then recommendations about how to go going forward. So we think there's a big disconnect between the genetic, genomic discovery which is generated by genomic scientists, researchers, and also healthcare delivery who are primary driven by frontline clinicians. I think our clinicians, they are eager to adopt, embrace genomic medicine, but they still have a lot of unmet information needs as mentioned earlier. The guideline resolving the contradicting knowledge among competing different guidelines and then the IT infrastructure, all these issues need to be resolved. And also the second barrier is the lack of coordination, the lack of workflow support. This was also mentioned by the previous speaker and then the policy for liability at hospitals. Regarding what data can go into EHR, there are a lot of controversies still at the hospital, at an organizational level. And then the third barrier is the lack of current knowledge to practice evidence-based genomic medicine among clinicians. A lot of the genomic knowledge were rabidly evolving. So the variant reclassification is a big threat for building reliable genomic decision support because the knowledge is constantly changing. We currently don't have the infrastructure to rapidly incorporate this changing evolving knowledge into the decision support infrastructure. And so that's actually can hurt the trust in the recommendations made by the decision support tools by among clinicians. And then also the poor EHR user interface, alert fatigue, information fragmentation. This is also a huge barrier here, not only for like the case study here for other diseases we also run into this issue. And then the no genomic knowledge base, as I mentioned, a lot of our evidence about genomic medicine is locked in free text PubMed articles. And then there are so many, how can we make this genomic knowledge computable? So we probably need more advanced tools to be able to mine the literature and then converting this knowledge into a computable form and make it interoperable with our current decision support tool. We think that's critical. And see the integration of this knowledge base with the PubMed literature. So we also know there are a lot of public database such as clean water, clean drain and then PubMed literature. So very often we need to link this knowledge base and then PubMed to better interpret the conclusions, put the conclusions in the context. So that's why we also need to better mine the PubMed literature. And then finally to integrate knowledge into a year trial environment for automatic triggering of genomic practice guidelines and how can we harmonize different guidelines? So all these are potential informatics and data science research topics, I think are highly relevant to enabling precision medicine. So we also think there are some enablers such as build up or leverage the synergy between NHGGI and NCATS, especially we have the network of the CTSA, almost every academic institution, their CTSA hub. How can we develop a learning health system to incorporate this evolving genomic knowledge into the CTS development, build best practice among the community? I think there are a lot of, can be done. And also incorporate the events in biomedical informatics, clinical informatics, a lot of year trial based decision support strategies and methods have been developed. How can we integrate that with the implementation science for genomic medicine? And as I mentioned, text mining and knowledge engineering about genomic knowledge, knowledge using the free text, PubMed text and also fire as mentioned by previous speakers. So basically we feel the strategies for success should really be stakeholder centered, user centered, social technical approach and informatics research for genomic evidence computing. How can we make this evidence computable, interoperable, make the knowledge base interoperable with the EHR and then really have a large shared centralized the genomic knowledge base let everybody can do knowledge data provenance, knowledge version control, things like that make the knowledge more accessible and then trustworthy for our frontline clinicians and harmonize different interests of multiple stakeholders for the collaboration and really provide clinicians centered workflow support. So I think this is my last slide. Thank you very much. Dr. Wang, thank you for an excellent presentation. I think we have time for a couple of clarifying questions. One is from one of our other panelists who says that at your institution at Columbia that you have a homegrown electronic health record and can you just comment on how scalable are some of the concerns that you've mentioned about the EHR interface? How scalable are they to other commercial systems? Yeah, that's a great question. I think we have a complex EHR ecosystem. So we have our homegrown system, INYP but we also have APEC and in the past we had a crown. So our general approach is INYP provides a very flexible test a bad for us to design and test the novel decision support in INYP. And then once it's working there, then we can migrate or move it to the most standardized EHR platform such as APEC. So that's the approach that we have been taken. So right now, we already have APEC in many of our practice. So in IT team, they are actively working with our APEC team by leveraging fire to use these latest standards to develop this CDS. So hopefully going forward, we will have more fire-based solution. OK, great. And some of the strategies to overcome the barriers that you mentioned was you described what you called a socio-technical approach of stakeholder engagement or youth or center design. Yeah, excellent. And you had a good list of stakeholders here. Just curious, we've talked a lot about patient empowerment or patient centricity. Do you also think about engaging the patients as part of this user-centered design? And would you recommend that we broaden the stakeholder engagement to include that perspective? That's a great question. So we definitely have a separate team, which is highly focused on patient engagement, patient facing, or to say, we have a separate team to do patient engagement. But here, I think we also need to look at the unmet information needs of clinicians. Because after all, the lay-out of people or users of these decision support tools, they also have the knowledge and then authority. So I think these two parts should go in parallel and then eventually, all these will be harmonized together. OK, great. All right, well, thank you so much. And now we're going to turn to the presentation of, we built it, but will they come by Dr. Hoffman? So I should have used a similar expression. So my comments come from, if you go to the next slide, please, 20 years of advocating for integration of genomic information into the electronic health record. I spent 16 years in industry and then the past eight years affiliated with academic medical organization. So I've been able to see both sides of it. And so my comments really come from the perspective of, what are some of the challenges and barriers? What are some potential solutions to those? So almost 20 years ago, we filed a patent. The US patent was actually not. This one was not accepted in the US. But the concept is the alert user if a clinical order is contraindicated by genetic information. So these ideas have been out there for quite a while. I think, in some ways, maybe we were too early. So if you go to the next slide, please. So both in my industry experience, but also things I've observed in academic medical settings, many important and useful things have been built. So certainly a lot of useful development in the laboratory information arena have really enabled the processing and generation of genomic findings. We took a stab at the terminology standardization by developing the clinical bioinformatics ontology. And this would have been in the mid-2000s. And it was a semantically structured ontology that allowed the mapping of variant findings to a hierarchy that could be crawled semantically. Worked great when we were still in the genetic medicine era. But we couldn't sure. Once we started to see really the growth of sequencing, it became very difficult to create standardized concepts for every possible variant that would surface through diagnostic sequencing. And then we also had a clinical decision support strategy. Some successes were in the laboratory arena, definitely an appetite for capabilities to support lab workflow to generate those discrete standardized results. There's definitely strong support and interest among specialists in genetics and genomics. Some clinical specialties, especially oncology, have a strong appetite for this integration. But when you look at the numbers of where providers are generalists and most specialties, still haven't really jumped onto just, of all the things they want, genomic integration typically isn't very high on their list. And so that still remains, I think, a gap that we need to really continue to pursue. So go ahead to the next slide. So in my experience, I grouped the periods of EHR genomics into three phases. The first thing I noticed was, beginning to talk with physicians about genomics and they still weren't really ready to start thinking about how to integrate genomics into their practice. And then between 2000 and 2008, frequent barrier was what is at that time what we would call an EMR. And many organizations either didn't have an EMR or they had a homegrown EMR. We did make progress in the laboratory systems and lab automation. The next phase was, meaningful use provided funding for the purchase and installation of EMRs. So the barrier of the absence of an EHR was resolved in a dramatic way in the mid, late 2000s, early 2000 and teens. But the process of implementing those EHRs were completely consuming to the organizational leadership responsible for fulfilling the meaningful use checklists which for the most part really had very little to relate to genomics. What we did see in the field and we've seen examples of this throughout today's presentations are creative organizations that worked within the existing capabilities of their EHR to do ground up build for either discrete data capture or genetic based decision support. I think now we're in an era where EHRs are implemented, they're operational. And I think as now you're looking at how to utilize genetic and genomic information. There's so many startup companies offering variant analysis. There's still definitely a view that this is an academic exercise. On the positive side, I think patient portals are definitely fully implemented now. And so there are means to really engage with patients. And then we can't also ignore right now we're heads down focused in a pandemic. And so most institutional decisions need to be framed and how is this gonna help us during the pandemic? So when we look at institutional priorities, we've seen this evolution. And I think that's an important way to frame some of the stakeholder perspectives. With that background, go ahead to the next slide. So at Children's Mercy, we see a lot of this playing out. We have a fantastic genomic medicine center that eight years ago, they were the first to do a complete sequencing, both the sequencing and the analysis in about 24 hours. So they've been involved in diagnostic and research sequencing for a long time. We have an active initiative now called Genomic Answers for Kids. The goal is to sequence 30,000 children and their family members. We're already having massive amounts of data generated consistent with that follow up question for the previous presentation. We really see the most important stakeholders are the patients, in our setting, the children who will be diagnosed and potentially have some treatment options that wouldn't have been available had they not gone through the process. Still a big believer in pharmacogenomic decision support, I think for safe healthcare, I think there's so much missed opportunity there as well. So the architecture of our genome center, if you go to the next slide, I think it's very consistent with things we've seen and heard today. We have the electronic health record. The EHR can generate a test order that's received by the sequencing lab. They perform bioinformatics work to generate their variant interpretation. Initially, that was with a homegrown set of systems. We're in the process of migrating a lot of that to the cloud for greater efficiencies and we're evaluating commercial options. The painful part to me is that the output is still a PDF report and that that PDF report is how the provider sees that. And to me, a PDF is a dead end for decision support and reanalysis. So this is the general architecture that we see at Children's Mercy. And I think, again, it's very consistent with what many of the organizations that we've heard from today also have. If you go ahead to the next slide. So one of the tasks for this panel is to think about resource allocation. And so I've kind of articulated on the EHR side of funding. What I've observed is most funding goes to employees and to people. There's very little willingness to invest in activities that will increase utilization among providers. Human factors, I'll talk a little bit more about that. And we've heard some other comments consistent with that. In our setting and really all three of the settings I'm affiliated with, Fire is interesting to the organization, but we're not using it in widespread practice. What we do see a lot of the direction of funding is on the laboratory side. So we're making major investments in cloud storage and analytics, next generation sequencing is generating long reads. And so the whole informatics framework to support that process is very much an area that we're investing in. And in the laboratory side, they seem very willing and able to invest in third-party niche applications. They also have gaps and so standardized clinical interpretations has come up as a gap or need. And then I anticipate policy clarity around automated reinterpretation of results. And so there's definitely a need on the policy side to offer some clarity around how that will work as we don't need to do a new sample but new findings change how we interpret those samples. Go ahead to the next slide. So some of my just very direct observations from the field I think Fire does have a lot of promise and potential but it's still early. I think in many organizations we know that we need to address it but there's not yet the experts in-house who are ready to tackle that. I'm a big fan in the pharmacogenomics world of CPIC and I think there's a need for some more consensus-based process for interpretation and how to incorporate those findings into clinical practice. The really key thing is, and I think this came out in some of the survey findings too, in many health systems there's a tremendous reluctance to use open source applications in our production environments. And now in this era of ransom where I think the security teams have pretty strong footing to push back on things that are either developed in academic mode or don't have the security features that they see with many commercial offerings. I think as I said, labs are willing to pay for third-party niche applications but in the provider side of the equation I see less willingness to pay for what I think of as EHR adjacent genomics clinical decision support. So in terms of commercial decision support or even things that are developed academically, the appetite to incorporate those among providers when you get out into either the generalists or the broader group of specialists, I think there is that reluctance to do that. So if you go ahead to the next slide. So what do we do about it? Again, my thoughts are consistent with many of the comments earlier today but I do think we need to really think about some rebalancing. I think at Children's Mercy, we have a really impressive genomic medicine short course and providers of any specialty and in some cases employees of Children's Mercy who aren't providers spend a week, their personal genome is sequenced and then they learn the basics of how to process and interpret that that gives them the exposure to these concepts that currently is not offered in medical school or other settings. So I think providing that training and exposure to these concepts that are daunting to people that don't live in this world as most of us on this workshop do. Also a big believer in human factors. I think the experience of interacting with complex information is a deterrent to many providers. And so I think the human factors element is really important. How do we present these complicated findings in a manner that can be digested and is actionable? I've already commented about content. I think economics is also critical. I think being able to demonstrate the financial benefits of using genomic information to improve care is an area that needs additional investment and support. I think the more we can do to increase the demand for genomic analysis and testing, all of the other things that we're aspiring to will start to fall into place but we need to put more focus on the financial benefits. And then again, having lived in this world, EHR vendors respond to their clients. And so if they see their client base pushing for and asking for genomic features, they're more likely to make the development investments in that. We did that for a long time at Cerner. Epic has more recently taken a lot of steps to incorporate those concepts into their systems. But they think everybody would agree that they have a long way to go. And so keep pushing those vendors to provide those capabilities so you don't have to reinvent the wheel over and over in your local implementations. So these are some things that I think if there was additional attention to them would really start to help shift the balance and increase the demand. So go ahead to the last slide. So despite all of this, I am still cautiously optimistic that there is a place for genomic integration with electronic health records. And I hope to see progress more on the consumer side rather than the generation of the results side. So with that, I'll stop and take any questions. Great, thanks Dr. Hoffman. That was really helpful. I have one specific question that's come through for you. And then I think what I'd like to do is also invite Sid to participate. But I have a number of sort of broad questions that I'd like all the panelists to respond to. So since you brought it up, the question about data security, if you could just comment on some sort of solutions to addressing what we need to do to optimize data security. Given all the hacks that we've heard about China, for example, acquiring sequencing data, how do we put security into the workflows between vendors and providers and participants? Sure, so one of the things I've found is just engaging early with our security team rather than at the end of a process is helpful. Getting their comfort level and treating them as a partner rather than an adversary is an important way. They'll understand what they expect in terms of how those interfaces are secured. And so I think that's a key part of working through some of these issues is trying to treat this as a partnership. As we're going through our cloud migration, our security group has a lot of concerns, but I've articulated the cloud is in many ways more secure than what you would see in a lot of local installations. And so I think there's a lot of advantages even from the security perspective of moving workflows to a cloud environment. Okay, great. So in keeping with the chat, Marilyn has a question. Marilyn, can you go ahead and unmute yourself and go ahead and ask? This is for Mark and Chenwa. Hi, can you hear me all right? Yes. Okay, great. So while both of you talked, I kept coming back to this point that came up really early this morning in the strategic plan and this is training. And specifically here though, it's the training of the typical clinicians. I'm not thinking the academic medicine clinician scientists who are embedded in this space. I'm thinking of primary care. I'm thinking of your standard nephrologist and your standard cardiologist who isn't so focused in cardiovascular genetics. How do we get genomic medicines or these disease risk genes and also pharmacogenomics more embedded in their medical education? Cause I do think a huge part of the resistance is that they're busy, they don't have time. They don't really know much about this. And so they just kind of aren't interested. And so the only way that I can think to get them interested is for them to realize that on the pharmacogenomics and they could actually prescribe medications quicker to the right patient if they had that information and use it. They wouldn't keep coming back saying my medicine's not working or I'm having side effects if they got the right medication. Or they might be able to diagnose something faster if they did an exome panel and saw that this is actually a genetic disorder, not just a kind of consequence of lots of symptoms. There's actually a genetic disorder here but they're not trained to even think about genomic medicine. So I'm just wondering if you have thoughts on how do we kind of infiltrate medical schools? I feel like that's a huge piece of it. Yeah, I don't have the data on my fingertips but I've seen just in terms of the medical school curriculum just a few hours at most dedicated to genetics. And I think it's not just not having time but I think it's intimidating to many providers. And I think kind of finding ways to integrate it both into the education and then practice. I continue to believe pharmacogenomics is the most accessible aspect of this as well as in some cases where disease diagnosis. But we have to get it down to the provider training to start to crack that. I think there's also what I sometimes call point of care education and that's not necessarily an alert but something that's expressed in accessible language. A lot of pushing a provider to a ClinVar website where they see RS IDs. If you don't know what an RS ID is that's not meaningful to you. So we need things that are expressed in more accessible content. Yeah, I'm also thinking like the diagnosis and management of genetic disorders requires it's still a team sport. So this is why we need a better workflow support and coordination between the primary care providers and the genetics counselor and clinical geneticists. So right now I feel like we don't have a lot of infrastructure to coordinate the referral and then the collaboration between these two different roles. Another thing is in terms of training because we also emphasize evidence-based medicine. So what's the computable easy accessible evidence for genomic medicine at a point of care? So that's why I was thinking about we need a better infrastructure to make this evidence accessible for EHR-based decision support. A lot of knowledge right now is really locked in pre-text guideline and PubMed literature. How can we make those available, interpretable with CDS design infrastructure? So these are the things I think these are the areas needs a lot of development. So a number of you have talked about the concerns around the cost of doing some of this and to demonstrate to payers and others that it is a useful thing to do. This is also sort of a bridge to some of the comments from the last panel. From each of you as you think about it from your perspective, what would you say is the rationale to health systems outside of academic medical centers that are funded to do sort of this very technical development to invest in the information technology capabilities required to adopt genomic medicine? Is there some way to essentially demonstrate the value of the investment in this infrastructure in a way that will make it more useful and sustainable over time? And maybe Jeremy we could start with you since you've been, we haven't had a chance to hear from you recently. Yeah, so that's a really complicated problem and not just with genomics. I mean, it's been a decade long challenge in cherry clinical decision support capabilities logic whatever across health systems beyond the academic medical centers. We do a lot of work with community health centers in the Rocky Mountain States and low hanging fruit for them are basic things like colorectal cancer screening where we have rates from 20 to 40%. So yeah, genetics doesn't even get close to their priority list. So it's a very challenging problem. I don't know what's the solution for that. That's why I'm thinking we have CTSA which is very comprehensive. We have learning health system, implementation science, community engagement, patient engagement within CTSA. That's why I'm thinking for genomic medicine we also need a learning health system. It's potential that we can leverage a lot of development in a CTSA space and then potentially leverage the community engagement, patient engagement development there to help address the needs here. Mark, did you wanna say anything? I think another important aspect is across all specialties providers will have patients that they suspect are being bounced around and those will be people who are often on a diagnostic odyssey and that can be a five to seven year process. It's enormously expensive to the health system but because they get punted from specialist to specialist there's not always that high level visibility. And so I think recognizing a patient who's on a diagnostic odyssey trying to refer them sooner to genomic analysis is also an important part of this. Okay, great. I'm seeing that Mark Williams has a specific comment that he wants to make or question. Did you wanna chime in Mark? Yes, thanks Pat. I wanted Mark just made a very interesting comment about the diagnostic odyssey and I think there's a real interesting opportunity to think about some, this may be a little bit more towards session two but I know that in the sustainability discussion for the end diagnosed diseases network there's been a lot of interest in predictive modeling about how we can actually identify these people early on in the process rather than having an undiagnosed diseases program that is a program of last resort where you've already spent tens or hundreds of thousands of dollars and now you're adding sequencing on at the very end of that. And I think the work that Lisa presented is another way that that a phenotype approach could be taken. But that wasn't the point I was actually going to make it's just I had to get that in since Mark teed it up for me. The question I had is that we're actually generating a lot of sequence clinically but we're treating it just like any other laboratory test which is we do the test for an indication we look at it and then we move on. And that to me is a real waste. We could reuse that information for many different purposes down the road if we thought about the durability of the data as opposed to thinking about it like every other laboratory test. And so I was interested to be here particularly from Mark given the maturity of your program whether you're how you're dealing with this sort of reuse of data. And certainly if the other speakers have thoughts or experience with that I'd be interested to hear from them as well. Yeah, I think currently and we still have a lot of work to do on figuring out how to regularly reinterpret the data and then return those results that is a part of the genomic answers for kids research protocol which is intended then to inform eventual wider scale clinical practice. And that's where I think also there's just a lack of policy clarity. So so much of this diagnostic work is done under a pathology mindset and model where the test is ordered, paid for, billed and interpreted and that's the end of the process. But if the lab is going to reinterpret it is that also a billable event? Or those are things that I think need as I commented need some further clarification. Okay, great, thanks. Well, we're winding down to our last five or six minutes here and just to sort of facilitate the summary. Each of you did an excellent job of identifying the barriers to clinical implementation of genomic CDS. And I'm wondering if I can press you to sort of to prioritize from a research agenda perspective which of those barriers youth that you've identified in your talks you think could be framed as questions for NHGRI to potentially support. So Guillermo, I'll just start with you and then we'll proceed through the panelists. Yeah, I think I urge us to think about workflows that do not rely on the primary care providers as the main source for doing extra work. We keep coming up with innovations that at the end of the line it's always adding burden to primary care providers and they have way too much on their plate. I mean, in a 10 minutes visit where they have to go through four or five chronic diseases and think about a number of issues, educate patients, look at their compliance with medications you still wanna add genetics to their plate this is just not gonna happen. So we need to think about innovative workflows that tapping to other healthcare workers in the workflow pipeline and also very, very important to try to engage the patients. Okay, great. Dr. Wang. Sorry, I muted. Yeah, so for me, I would prioritize the development of the standards-based computable evidence base for practicing genomic medicine high because right now a lot of the information are really locked in free tax and not easily accessible for clinicians. They also don't have time to read tons of papers at a point of care and the searching for relevant evidence is also very hard. So if we can build a shareable centralized standards-based interpretable knowledge base that make this genomic evidence easy to access and then integrate at a point of care, I think that will help with a lot of the needs for developing genomic decision support. Just for a follow-up question, you did a lot of nice stakeholder engagement work in the study that you presented, but you only had like a very low uptake of some of the alerts that your system provided. Are there particular, if we would hear directly from the clinicians, let's say, what might they say? Is it really development of standards for practicing genomic medicine that they would say would help them be more likely to use this information as part of the CDS? What would they say from their perspective? Yeah, I think at least low uptake, it can be attributed to multiple reasons. First of all, it can be they are already pressed by time. They don't have time to look at additional data in addition to the frequent data that they use for the patient in front of them. So I think it can also be due to user interface design and then also can be due to the lack of coordination, like who is supposed to look at what data, right? Who is responsible for integrating the genomic information into a patient care? Lack of familiarity. So all these can be causes. So I think it's complex. I definitely more studies will be needed. Okay, no, that was great. Thanks. And then maybe we'll just close with you, Mark, with a comment. And then I think Sid is gonna summarize. I think some research into the economic benefits of doing genomic analysis for both the health system and the patient. You know, there's some compelling evidence that it can be cost effective to incorporate genomic information and especially pharmacogenomic information. But I think further investment in that will really help as well as clarification of the payment structure for providers and reimbursement. All of those still, I think feel like areas of confusion. So I think research and communication around those topics. To look at cost effectiveness, you need an agreement about credible effectiveness measures. What would you recommend? I mean, some of the things I've seen in the literature about adherence to recommended practice or medication decision quality or even patient outcome measures. Is there an effectiveness measure that you would recommend to do the cost effectiveness analysis? I think that's a lot closer to your area than mine, but I think patient outcomes is the, ultimately that again, that's the stakeholder we need to focus on. Okay, great, thanks. All right, Sid, and thanks to all the panelists. Sure, I think given that this is a stakeholder perspective, I just wanted to bring up a point that when we had a pre-meet with some of the presenters, we thought we had a pretty overlapping mutual view of what a stakeholder would be, even though we knew we were coming from different places. So I just wanted to put that into a discussion for tomorrow or ongoing, is that even within this group, which is fairly specific, that discussion goes on about like, who really is the stakeholder and how can we better define that? So that's, I just wanted to mention that. That was a very good intro to a discussion. I'm looking forward to tomorrow. I think with that, we are at time. So I think we'll hand it back to Mark with a C and Ken. Thank you very much. Thank you to our co-moderators and our presenters for a wonderful session. So we are at the hour and I thought I'd just share with you some of the things I took home from this, from today's session. I let you know that this is not comprehensive, but just to kind of try to summarize what I took from this. So let me share my screen. Can you see my screen? And if so, which one are you seeing? Tell me, Ray, of day one. Yes. So some of the things I took from today's meeting was there's a need to understand that there's different barriers regarding the implementation of Duma-based clinical informatics tools between rural and urban hospitals. And that what type of research that we're trying to take needs to be able to address those barriers so we can get broad implementation. We also need to understand more about the... There are also additional work that needs to be done both on the genotype and the phenotype representation to make them both useful for by systems. We've made progress in the genotype, but it's not complete and we definitely need to improve on the efforts that we're doing as far as phenotype representation. We also need to make sure that whatever algorithms or tools that we're developing that we make sure we address the biases that are negatively affecting people of color when they're being implemented in care. One thing that is common that was crossed, another thing that was common across was with better workflows are needed, but these workflows need to be developed and implemented and maintained with input by the clinical Jonas research community that can go beyond just the clinician's use. And we also need to make sure that progress, even though we acknowledge that there's progress that's made in trying to achieve the elements from MSR and then the DisRata, there still remains a lot to be accomplished. So there's areas of research that needs to be focused on those remaining elements that we need to start on and also those that we've made effort in but also to improve on. There's a need to understand more about provider parents and those to implement genomic-based clinical informatics resources and tools. If we don't get the incentives from the provider payers and we're kind of stuck in the water in some regards. We also want to understand how open source tools can be developed, but also in the manner that they can address security concerns. Fire and CD hooks were mentioned throughout the today's sessions and they are used and this group considers them useful resources, but it still wasn't clear to me how they can be used for clinical genomics research. And I think that each of us are trying, the community is also trying to understand more about. And we also need to address provider training. And we need to have, there's a need to have a shareable interoperable genomic knowledge base. So Mark, I don't know if you want to add anything to that, cause I know I missed several things. This is just what I could take and while I was also trying to listen to everybody else's presentation. Yeah, no, I think those are good. I do have a few other takeaways. I was particularly struck by the way Chenwa talked about the socio-technical challenges and strategies, which I thought was a really nice overarching thing. So I've already asked to steal part of that for the summary tomorrow for our summary discussion. I think there's some interesting opportunities relating to research agendas around policy and education. I'm of course always on the lookout for the patient centeredness. I think that that's something that there's a real opportunity to explore in more detail. I was really interested in some of the bias. And I know you had specifically referenced bias of the data related to concepts of race and ethnicity, but I think my takeaway from Dr. Jeff was that the bias is probably much broader than that. And so thinking about bias more broadly will be important. And I think also the need for some research into evaluation and methods is still needed. So those are some things that I would say in addition to that. There's also an interesting comment made about the business model for research. How can we engage a broad audience of stakeholders including the patients, the providers, systems, vendors, et cetera? And is there a sustainability piece to that? So what will happen overnight is that I'm gonna ask Ken to send me his slides that he is actually working on. I'll start to add in some of the pencil and paper stuff that I've been working on because I'm old and we'll merge those together. And that'll be sort of the first half of the summary presentation that we'll do at the end of the day tomorrow, but we'll be doing the same exercise during the next two sessions, four and five, to grab important content there. So that at the end of the discussion period, we should hopefully have a pretty clear idea of where we want to go from a research perspective that will be incorporating your feedback and hopefully to some degree some endorsement. So with that, that's all I have. So thank you, Mark. And thank you to the participants, the co-moderators, the speakers, and the attendees, we will meet again at noon tomorrow for sessions four. And thank you very much. And we'll close this meeting up for the day. Have a nice evening, our afternoon, where you are. Bye everyone. Bye. Bye.