 Okay, so we thought we'd put together some basic questions to seed our discussion going forward, and so we'll put these out there as Mary did for the last panel and then see where the discussion takes us, because I'm also interested to hear what kinds of questions you all have from your experience or what kinds of questions this information has raised as you sit and think about it just this morning. So first, of course, is a question about if and how the development of patient-facing tools fits within NHGRI's mission. As we think here throughout this meeting about what should NHGRI be doing in this space to ease implementation of genomic medicine? What sorts of evidence do we need to validate the tools that are out there or as we are developing them? Importantly has been mentioned here. Bill mentioned it as it came up even as we were talking about the clinician tools, but incorporating the patient's voice into this. What are their needs? How do they want to hear about it? Where will they look for it? At what points in time, et cetera? And a question also raised earlier in the presentation of are we ready to standardize these tools? Do we validate them? Do we have standards that we could begin to apply to them? Could the IPDAs perhaps be something as a starting point to be able to think about this? And then, again, looking towards dissemination. Would a clearinghouse of patient-facing tools be useful? What are the other mechanisms that we might be able to use or think about developing to make sure that people can find this information when they're looking for it? And those people, the audiences, I think, are different. Of course it's going to be patients, but it's also going to be those who interact with patients who can make sure that they can get the information to the patients at the right time. So with that, I'll just open it up for comments. Mark. So I recognize that the charge to the working group is around tools. But it seems to me that that is a step down from where we need to be talking about. I'm reflecting in particular on the first stakeholder meeting of the Precision Medicine Initiative, which was really more focused on engagement, and in particular the need for a substantive patient engagement. And I think that engagement can lead to tools, but I don't think that I would see the tools as being that that should be the focus. I think really the focus for NHGRI, and I would strongly endorse this, is how can patients be engaged beyond the issue of consent, which we focused a fair amount on, more in the idea of in the implementation space, what is the role of the patient in fostering implementation. I think we've heard repeatedly in the discussions that one of the things that motivates clinicians to get interested in this is when patients show up with information in their office. And so that suggests to me that that could be a pretty powerful tool for implementation if we looked at this with a research agenda looking at engagement, which tools could be a part of. Jeff? Just to add to that, I guess I know that NHGRI in the past has done work in trying to understand public literacy in genomics and public literacy, numeracy, I guess, is the best word to begin to digest some of the quantitative information that's being thrown at them. So I wonder if you feel that that agenda, from your perspective, is already complete. I doubt so. And particularly in the light of the PMI and where the field is now, those studies which may have been done several years ago could be outdated. So I wonder if before you deliver the tools to Mark's point, understanding the landscape is a component that should be invested in. So I'll respond to that just quickly. And as you were talking and even mentioned the notion of the fact that the genomic literacy question might be complete, I was just wondering what Vince was doing behind you. And then I saw his head pop over to think, because I think there is a lot of work that we still see out there to be done. And Vince, I don't know if you want to directly comment on that or not, but I do think it is something that still needs to be addressed greatly. I would agree with that that there's still a lot of work to do and ways to assess it in different ways for different populations with regards to literacy issues. And Bill, did you have something to respond to Mark's point earlier? Oh, that was mainly, I would agree wholeheartedly with the patient engagement throughout basically the entire research process. And to the extent that as you're setting up your plans to think about research programs and such, it's important then too. Terry? Just to push a little bit further specifically on the patient engagement aspect. So this kind of getting, I think, more traction at the NIH, particularly with the Precision Medicine Initiative, but even before that. And we need to be careful that it doesn't sort of become a passing fad that we, okay, we've done that now and it's over with. But I think in looking at NHGRI's genomic medicine programs, we don't have a lot of patient engagement, at least at the level that we see it. Now maybe, and I think it is happening within individual sites and individual programs, but is there a need for something a little bit more systematic in the approach to that, that we could do more at a steering committee level? Yeah, I would say yes. And I think that, you know, one of the, I think to some degree, unexpected outcomes of our participation in the PCORI project that Janet mentioned was that it really transformed the way we thought about patient engagement and research across our entire organization. And it really, in the context of revisiting our research strategic plan, was articulated in an aim to move from patients as subjects to patients as partners in research. And we now have laid out a pathway by which we can achieve that goal. And so I think incorporating that into more of a strategy, particularly in the right side of the five aims of the implementation side and the science of medicine side, would be a very desirable outcome. Just to push on that, and then I promise I'll yield the floor. So you had said previously that some of the, you know, clinical workflow kinds of things were really local problems and we shouldn't be dealing with them. This sounds like there may be some cultural aspects to it, but it really isn't the local problem. It's something we can address systematically. Yeah, I think so. And I think we've heard nice examples of, you know, the patients self-organizing around whether it's diseases or genes or whatever and really trying to push a research agenda, whether it's from patients like me taking on ALS or whether it's an individual with a child with a rare disorder that finds other individuals that have other children with rare disorders and engaging. I mean, it's clear that there is a hunger on the patient's side to have some leadership in the direction that the research is going. And I do see that as being much more global than local. Could I just ask a related question, Mark? Do you have patients or patient advocacy groups on your peer review? That's something that we're specifically moving to. And again, the way our system works is that we don't have a formal scientific peer review for all of the things that go through our IRB. We obviously do have community members on our IRB so that we do have that input. But for example, for our large sequencing project, we have an external ethics advisory board and four of the members of that board are geisinger patients. And so we are being very intentional about increasing that voice. And for the PCORI project, we actually have a patient as a co-investigator and we're replicating that model for not only other PCORI proposals, but also we're trying to create a culture of research whereby we are asking the clinician investigators to say, why don't you have a patient on your investigative team? As opposed to, oh, isn't it cool that you have a patient on your investigative team? So I'm gonna respond myself real quick and then I've got two people on deck. And one, I just wanna point out that we are falling into the common issue of where we are and where the science is right now. If we're talking about both research and clinical implementation right here, and we're going back and forth between people who are participants in research versus the patients and from an implementation standpoint, and I think that's okay. I think that's where we are. But I just think it is, there's some different needs that there might be, particularly as we go through time. So I just wanted to note that. And I think too, Mark, to your point about needing engagement and not just tools, I think we do need tools and we need valid tools to be able to engage participants and eventually patients in what we're doing. And so I think they go hand in hand in what we're talking about. And certainly within PMI we're very focused on how do we engage the public or participants in joining this research project and how it may or may not interact with their own healthcare while they're participating in the research project. But I think that we do need to keep them together. And Steve, do you have a point to respond directly? Yeah, just talking about the sort of clinical versus research engagement. I think here we're talking about a very particular place in the research continuum, which is implementation research and particularly implementation research around communication and education for patients. And so in my mind it's that particular sort of narrow place on the spectrum where patients as partners in the research is critical all along. But in that particular place I almost think we can't do it without really good patient engagement. And I agree. Okay, Janet. Yeah, I wanted to agree that we actually at Geisinger, because we are a healthcare system, are looking at patient involvement in all aspects of research, but our research is all clinically directed and focused. But the comment that I wanted to make was back to the comment made by Irwin regarding consent among populations who might be seen as minority populations or populations other than Northern European whites. And the idea that those populations that were where they had representation in developing the consent and in communicating the research were much more successful. And that that's a real lesson that we need to have in terms of looking at population investment and involvement is that if we start at the patient level we may be much more successful and begin to be able to address some of the misconceptions and concerns that individual populations may have experienced or hold on to from past experiences. Okay, yes. So I want to second or third the notion of patient engagement, because I think that's really what we should be looking at. And I think it actually has implications for the existing grants that you have and anything that you go forward in the research space because I think that patient engagement is not just, let's be on boards and let's be on councils and so on, but also very specifically the results of the research and the very particular results. So if a patient wants the sequence data, that should be available to that patient. And we had a little bit of a tiff yesterday, a little bit of a disagreement about whether patients should have access to their data or whether they should be managing their personal, their health records and so on. You know, I couldn't agree or disagree more strongly that we keep people from having the information that they want and I don't believe that it's a disparity issue for those people who can't deal with the information, but I just want to raise another example of a person, Stephen Keating, some of you may have read about his case in the New York Times. Oh, I don't know, some number of months ago. There's a young man who is a PhD at MIT and I met him not too long ago, a PhD student in the media lab and he had some kind of an abnormality that was found on an MRI and the doctor just said, oh, you know, it doesn't look like anything. And he started doing some very deep research. Now, I understand that he's at one end of the spectrum and not every individual is going to be able to do that. Nonetheless, when you read that article, you find that one of his biggest complaints is the reluctance of the healthcare system to give him access to his own data. And I found the same thing when I was doing clinicaltrials.gov and people would write and say, well, it's great to participate, you're pushing me to participate in a clinical trial, but I never see the results and that was a number of years ago and so I think when we talk about patient engagement, we can't be, I'm sorry, I'm getting a little bit passionate about it, but we can't say we only want you at this part of our research plan or that part of our research. We need, if we're going to engage patients, we have to treat them as full partners in the activity, so sorry for the little bit of it. Bill? So just from a concrete comment on engagement, under, on the PCORI website, which is PCORI.org under the funding opportunities thing, you'll see what do we mean by engagement. And we have an engagement rubric document, which I don't particularly like that name, but that specifies some examples of how people have engaged patients in research and maybe worth looking at. Yes. I'd just like to follow up a little bit on this idea of using participants within a health system as a test ground for clinical things at Geisinger. Do you have examples of how you've created some educational tools? Understanding that these are a slightly select population, but it's certainly a patient test that you've moved from letting people know about either in general or down to the specifics about their involvement or findings from the study, et cetera, into a clinically useful tool that is used more broadly now by the patient community at Geisinger. Yeah, and the only change I would make in the question that I think you're asking me is that we look at all of these as communication tools, so we don't have things that are solely patient focused or solely provider focused. We actually are looking at leveraging the electronic health record in the portal to create dynamic interactive systems. And this began in our rheumatology clinic with all of our rheumatoid arthritis patients at the time that they came in for their appointment, entering patient enter data using standardized question of pain scales and activity scales and that sort of thing. And then having an app that would then pull data out of Epic relating to medication use and that, so the physician would then get a heads up display that they could show to the patient that showed disease activity as captured by the patient enter data, along with the medications that were being used. And it was extremely effective. And it was really that was sort of the epiphany for the system that, hey, involving patients in seeing their data is a good thing. And so working with folks at Harvard, we went to the open notes project where now patients can view their entire medical record. They can actually download their images directly. We don't have a blue button implementation, like some people have talked about with a one click and you can get it all. But that's the philosophy of the system has been, let's open it up because we're seeing value and we're not seeing unintentional bad things happening from opening up the data. And so we're now moving into other implementations for things like heart failure and diabetes, co-management using these dynamic apps that combine patient entered and provider entered data so that there are patient views, provider views, educational materials that are linked to that. And that was what Janet alluded to in terms of leveraging that activity in the genomic space to try and see if can we do this in fact in genomic medicine. Yes, Cynthia? Still one of the challenges in research in newborns and children involves the use of dried blood spots. And I know that NIH has already convened a meeting to think about this, but as of March, no one can use stored dried blood spots for research unless there has been informed consent obtained by the parents. And so some states have thrown out millions of their stored dried blood spots because of this. And how to go about getting informed consent, utilizing informed decision making in the public health setting, but in order to use those very valuable dried blood spots for research is something that I think we can't forget about. I don't know what I did. Maybe that means it's time to move to the summary discussion. Are there any last comments before? Yes, I just wanted to mention the fact that there's a magazine called Genome that is a public facing magazine specifically meant to try to perform more of the education and patient engagement, public engagement in this field. Vince is on the advisory board of myself and Howard and maybe there may be some others in the room that are working with them. And I think it is the potential to be a very useful platform to accomplish some of the goals of what we're talking about here. It's obviously not a total solution, but it certainly can help bring a lot of these issues into the public sphere as well as I think the editors are now thinking about making it a much more interactive platform so that it's not just unidirectional flow but bidirectional flow of information. So I urge you to look at genomemagazine.org. I think that's the URL. Are we at the summary stage? People are hungry? Yeah, I can do it from here. So I captured in three or four slides some of our discussion here over the last few minutes as well as some points that have come up in the conversation amongst our panel. I think in terms of knowledge gaps, I think the, oh, that's actually. Yeah, it's not the right one. I thought I did. Let's, let me try that again. How about that? Excellent. Except for the little audio connection thing which doesn't show on my, I think that's on the, make that go away. There you go. So I think one of the major points that we took away is how best engage patients and families in this whole issue, which is a fundamental issue, a cross-cutting issue that's important to the development of tools but also way upstream from that. Second issue sort of implicit in some of our discussion is if we're gonna do research to develop tools, what are the key endpoints that we're looking for to be able to say these tools are effective in a grant that some colleagues and I submitted recently. We, to educate patients around hereditary cancer issues when they're doing tumor sequencing. It was not at all obvious what the right endpoints were that we should be looking for as we developed educational methods and decision aids to support patient family decision making. The issues alluded to in our presentation of how to do pretest counseling for exome and genome scale tests. There's some discussion in the genetic counseling world of the possibility of almost like a subspecialization in genetic counseling, of genomic counseling, how to educate patients and families outside of specialty settings that are used to doing this. If we are going to be soliciting preferences for secondary incidental findings, how best to do that in advance when almost anything could come back at you. And then, not mentioned, but ways to capture family history. It's better than most of us do it now outside of specialized genetics settings. Some of the other key barriers, we talked about workforce limitations. I think there is now a pretty good evidence base that distance education and counseling does work for genetic counseling and genetic education, but how to implement the infrastructures to support that and also some of the regulatory and billing issues remain a challenge. All throughout today, we've been talking about how to integrate patient education and decision-making support into clinical workflow, and then the important background issues of public literacy and also public trust. In terms of recommended approaches to addressing this, again, I think the key message is engage patients as partners not subjects in all research, but particularly in implementation and communication research. And then I think a couple of particular messages for the Institute, integrating tool development and the necessary infrastructure for tool development into the funded implementation projects that the Institute support. And then also to think about other just clinical settings outside of specific implementation projects where genetic and genomic medicine is being practiced, and think about ways to develop and study and validate tools in those clinical settings. And then finally, we actually haven't talked much about trainees or training needs in this context, but I think the question is out there, is there a role for supporting trainees in the communication and decision sciences and the implementation sciences that are needed to have a workforce that can develop, validate and implement tools? So, and if there are any last comments or any additions to this, but this is the sort of summary that I think our panel took away and also from the discussion. Yeah, Mark. It's not a separate topic, but I think one of the key things that I heard was that these things need to be dynamic, that they can't be static. And so as we conceptualize what that might look like from a research agenda, that would be a concept I would include. Yeah, dynamic and interactive for sure. Other additions or reactions to what we took out as these proposed conclusions or recommendations? Terry? So I guess I might ask maybe ask Heather or others here who are involved in education efforts. The communication education, I mean there are people who spend whole four years studying communication. Maybe we're talking more or maybe you're talking more about training in education as part of a subset of the training that people would get in this area or are you really thinking, let's have journalists or communication specialists in that involved? So I mean I think perhaps all of the above, but my focus perhaps our focus has been on the training of the scientists who can develop and validate evidence-based tools at the highest level, which is a particular scientific discipline. My sense of the NIH as a whole is that this isn't a, this isn't widespread across the institute or across the institutes, although some particular institutes have a particular focus in this. I think NCI is the one where it's the most developed, but there is a field of communication sciences and does it have any role to play in helping us and do we need to develop a scientific workforce to help us navigate these challenges? Again, this is relevant to the genome institute, but I think a relevant question across the NIH. So now I just add to that, I see it as similar to the education question and education research question, it's how do we do education? It's not the actual education, I think the same for communications, it's how do we communicate to our different audiences that we need to understand and we need some research in that area? Yeah, and broadly the field would include shared decision-making, which tends to have the experts come from a bunch of different fields, including communication decision science, clinical sciences, but generally, the way that's looked at is it's education, values, clarification, and helping the patient and caregivers communicate with the clinicians their needs and preferences. Yes. Some number of years ago, there was a cross-cutting, NIH cross-cutting funding opportunity in health literacy. Is that still going on and is NHGRI part of that? But don't know, don't think so. Larry, do you know, or Benz? Health literacy was a little bit narrowly defined there, but so in this new era, I mean, if something like that was going to happen again, one could broaden the scope to include the kinds of things that we're talking about here. I think I'll turn it back to the meeting co-chairs. Thank you very much.