 Howard? At most of the centers represented here, there are radiologists or pathologists or others that are hyper-specialized that may only look at the brain or might only look at the lungs. But then out at the community sites, there may be just a radiologist who reads whatever film comes in front of her or him. And I think from a genomic standpoint, there isn't that equivalent model. We might have a bunch of medical geneticists at our centers, but in the state of Florida, for example, there are very few medical geneticists in the entire state. And if you look at the few academic sites, including the two that are represented in this room, we suck up like two thirds of all the medical geneticists. And there's no one else at the place that kind of can serve that role. And I guess as in terms of education, one of the ideals would be finding a way for someone to be the local champion without them having to go out and get board certified. And that sort of thing is, I don't know how to solve it, but that's one aspect I think that's kind of missing right now. Yeah, I think one of the, coming from the Northwest and having practiced for 10 years in Idaho, it is a problem. And there aren't that many people out there. And there are a few people who are, you know, a few practitioners, family docs or whatever who really get into this. And they're very valuable. The other really, really valuable resource is genetic counselors. Because the systems that are out there can have genetic counselors or the central system may have a geneticist, but genetic counselors are deployed throughout the system. And they can be really great resources both for the patients and the providers. And they generally know where their limits are and where they say, okay, that's out of my realm and you need to go talk to the doctor. But genetic counselors can be really useful in that regard. The diabetic model, I mean, there are diabetic educators that are above a bunch of different backgrounds. I'm almost wondering whether a certification approach might also have some role here where a family medicine person who wants to go and get certification, not board certification, but whatever the diabetic, yeah, something that is more than just a piece of paper. Right. I think that's a great idea. I was going to make one more comment about a lot of places get their genetic services either by sending the patient to a tertiary center or by having outreach people from the tertiary center come to see their patients. And those contracts are structured basically for patient care. So there really is almost no opportunity during those interactions for providing education to the local providers. And that's something that could be addressed. Yeah. Go ahead. Well, we've baked this into the concept of our MedSeq project pretty deeply and are starting to publish on this already. And of course, our thesis is that primary, even primary care docs are going to be able to adopt to these technologies and become at some level genomics practitioners and will need to be. And that this is not so different than sophisticated technologies and complex specializations from other branches of medicine. Of course, we have early adopters in this, but we're actually performing an experiment where we're collecting these kind of data when we confront primary care docs with specifically designed reports that we have oriented them to understanding with a lot of safety nets, audiotaping the interactions, following up on what they order, following up on the electronic medical record. So I think this is, of course, remarkably well linked to the prior conversation, which is all the flaws of the electronic medical record and all the needs for decision support in sort of obvious ways. But in the meantime, I think that we have this challenge of being able to somehow integrate this with the practice of medicine, absent the nice decision support that we'd all like, absent certification programs for large numbers of primary care, and frankly, absent sufficient number of genetic counselors to go around either. I think it's doable. I think there will be early adopters and confusion, but also ways of doing it. And finally, I would say there's a huge gap in comparing what we're doing between all these different consortia. I know that we're deeply interested in this area, and maybe I haven't been paying attention, but I also haven't sort of been aware of what the other groups are doing, so I would love to be. Mark, and then. Yeah, I just want to pick up on what Robert said, and also then reflect on something that Bob had presented, which was that nice info button graphic in the middle of his presentation. One of the things that ClinGen is doing is enabling genomic resources to be accessed using info buttons, info buttons for those of you that aren't aware of what they do, do context-specific searching. They're a standard that will be in all certified electronic health records as part of Meaningful Use 3, meaning that if you have a certified EHR, it will support the info button standard. And if we can have the genomic resources, info button accessible, and ideally even info button certified, then what that allows is that transition that Bob was referring to, so that if an individual gets a genetic report or a genomic report and has a gene or has a variant, that they could conceivably be linked to an info button that would go directly to information that would be relevant to what do I do with this information. And so I think there is an opportunity to explore how best to do that, how to engage the end user to say, what would you like to see in that resource that we're going to link you to, as opposed to what we usually do, which is get a bunch of geneticists in the room and say, well, this is what we think that they need. And then we show it to people and they say, no, this is what we need at all. I think there's some real opportunities there. But I think that that's a transitional step along the path to the you know, certification or certificate or something. And that's been discussed with ACMG and some of the other professional societies about how we might be able to support that. So just to clarify, does the info button go to a centralized external resource from all around the world? Everybody goes to the same place or where does the info button could actually access any number of resources that where links were established as long as that resource is able to respond to an info button query. So an info button in a genetic test report could go to the genetic test reference or it could go to gene reviews or it could go to genetic home reference or it could go to an individual laboratory site, depending on what the individual institution configured as long as each of those sites content was able to be retrieved through an info button query. But the site that's being tapped into by the info button query is not manned by a live person? No, and that's the whole point is that you don't have to as long as you have your content collection organized. If it's variant A in gene B, if you click info button, you'll go directly to the content on variant A in gene B. So you don't go to a home page where you have to enter search terms or anything like that. It directs you, it does all that searching. And so it dramatically reduces the time and effort that a clinician needs to access information. Jonathan? So I mean, I think something that came up was the lack of providers or appropriately trained providers. And I think that's an existential problem that medical genetics has had for a long time because we lose money every time we see patients because what we do is we talk to people and we think about their problems and we don't do any procedures and that's not a problem that the NHGRI can solve. But there are programs like CSER that's evaluating in certain contexts, whether it's a primary care context or a Mendelian disorder or cancer genomics type of context that really does deal with that relationship between the laboratory and the clinician, which I think traditionally has been, you know, a provider saying I want to know if there's a mutation in this gene and sending it to the lab and getting a result to now much more of a potentially a need for an interactive thing that's a bit more like the way that radiologists operate where if you're the physician and you order a chest x-ray and you can actually call them up and talk to them about their findings before you get a report that, you know, those are the sorts of things that I think could be explored in a research context about, you know, where is the need for the physician and the laboratory to interact on the interpretation, how much of that information gets imported into a place where the physician is actively perusing it. And I think it's all very context-based and there's going to be certain contexts where in genomic medicine, things like pharmacogenomics or risk profiling can be put back in as a decision support type of tool, but other ones, particularly with rare variants and things where the training needs to include when do you refer to the appropriate specialist. And so I think there's a lot of different ways of approaching it there. I think it's Howard and then up front here. So I want to follow up on what Robert was talking about in terms of report generating. There's now multiple groups that are doing reports and we've just been through a very interesting, I won't say easy, exercise with Baylor in comparing sequencing pipelines and looking at reporting. And we're now in the midst of looking at interpretation. And I think a deliverable could be getting the reportings together, groups, and looking at how are decisions made and how are those reports coming out and what do people see as being critical, not critical, superfluous information inside that because there's now a bunch of groups that are doing this. And I think that would be incredibly helpful to get a working group around that and to make some comparisons. And I would even dare say it would be great if we could figure out how to do this is to do a bake off where there would be some other groups doing the same analysis on the same patient and look to see how people do on this. Because I'm telling you the problem is, and we see this with our clinical partners, they're nervous about the fact, depending on who you get the data from, what's your result going to be. And here's a great chance to do that. And actually, I think clarity on diagnosed, which Boston Children's just put out, is essentially going to do that. Yeah, so... I'm sorry? Right, but the one that was just announced is specifically on the report, we'll focus specifically on the reporting to clinicians and to families. I wanted to just indicate that we, I spent the last few months meeting with our clinicians around whole genome sequencing sample report, asking their opinions and what they wanted, how it might fit into the workflow. And there were a couple things that they indicated that I thought was really helpful. One of those is that they really appreciate the idea that there may be a report for clinicians as well as a report that goes and is designed for the patient, and that they would be able to see both of those because it helps them to know how to explain the result that comes back. The second recommendation was in terms of linking out to resources like the I button, it would be helpful if they could go to a site that was similar to resources that they already used, such as Up-to-Date or Med Info, or there's a few different things that are regular clinician tools that are very short, very brief, but offer you the depth that you need. And so thinking strategically about those sorts of, that sort of resource would be helpful for clinicians. The other interesting finding that came out of my work was the idea that clinicians are very concerned about understanding the results that come back. They welcome any of the support that could come with the interpretation at the time they receive the report. Many of them admit they will not have a chance to look at this report before they're in the office at their computer with their patient and they bring it up. And so any types of ways that you can embed supportive information, whether it be the actual recommendations or practice guidelines that might offer checklists for them to know that they're appropriately addressing issues with that particular genetic finding would be helpful because most of them recognize that they just will not have that familiarity with the gene or the condition that's being diagnosed. I wanna build off of something that Jonathan said a few minutes ago. And I think it's important to distinguish between two different scenarios. One is when a result comes back, there's an opportunity to trigger, to make the result, reporting document and educational document or to provide info buttons or in other ways to trigger the education in real time in just in time that people need, that clinicians need. I think it's much harder when you're talking about upfront, how do I think about this patient from a genomic point of view? What's the role of pharmacogenetics? What's the role of considering genetic or genomic conditions on my differential diagnosis? When do I go down that pathway? And that's a situation where it's much harder, I think, to do just in time kind of education, although certainly not impossible and you need a lot more sort of anticipatory and preparatory training and probably practice guidelines for at least the most common situations. You're kind of saying it's much harder to educate people about when to use genetics than how to use the result once it's back, even though that's also very difficult, but yeah. Julie? Yeah, so just make a couple comments based on our experience, which is in pharmacogenetics and could well be different from disease genetics. So first is I think that we make certain assumptions that in our experience haven't held up. The first assumption is that the young clinicians are into this, understand it, get it, and the older clinicians are less comfortable and that was absolutely not what we saw when we rolled out Clopetagral and CYP-2C-19 testing. In fact, all of our resistance was from cardiology fellows and very, very young practitioners. All of our support was from older, more senior, and I mean years, older as in years, physician. So again, I think sometimes we think it's the older generation that's gonna be hard and the first, I think the first barrier to educating about when to do the testing is that they have even an openness to do the testing and they're willing to sort of undertake the educational effort that would be needed to do that. The second that I think, the second assumption we make is that those in academic medical centers are gonna be more engaged and enthusiastic and we've just done an implementation in primary care setting and that assumption also has been violated because our practitioners in our family medicine group at the University of Florida have in some ways been more challenging even though they came to us and wanted it as we tried to extend that to the broader group have been more challenging than a private practice group in Orlando that we're working with. So I think we have to be open-minded about the fact that many of our assumptions about this might be wrong in terms of who's gonna embrace and engage these things because they have to engage the concept before they'll be willing to, I think, undertake the educational efforts. And then the other thing is about the info button because on our CDS alerts, we have, it's not an quote, an info button per se, but we have a link to a variety of types of information and the last time we looked, those links had almost never been accessed and so again, maybe that's different in pharmacogenetics than it would be in disease testing but again, it sort of goes back to the busy clinician and are they really gonna take the time, if they're comfortable with the information that you present to them on the front end, are they gonna take the time to go read that in-depth information? So I think we can't rely on them clicking the info button as a way to educate. Oh, okay. I think there's still a need for good, accurate, easily accessible, free online educational tools in medical and human genetics. In this day of Khan Academy, people wanna be able to go and get a course online. Coursera, I think, has tried to address this somewhat although that's not always available having recently looked for a good reference that I could refer somebody to. So I think that's an area where NHGRI could help with on the educational side of things. Along the part of addressing the issue of certificates, the American Board of Medical Genetics and Genomics has been approached over the years in providing certificates and I think a lot of the ABMS boards have steered away from certificates because they're difficult to provide. There's so much variability. I mean, do you have one certificate for somebody with a cardiology background and another for one with a nephrology and so on? But it really often comes down to a lack of money because it's expensive to make certificates. You've gotta have certification examinations. You have to come up with appropriate questions. You have to do MOC and so maybe that is something that could be addressed if there was some funding for it. Regarding the workforce in medical genetics, 50% of available positions in medical genetic residency training programs go unfilled and how to address that is a big challenge that many people in the country and I don't know if NHGRI can address it but I mean, that's a big problem and trying to attract people to the training programs and for the genetic counselors, I think as Robert was saying, there are too few genetic counseling training programs. There are too few genetic counselors coming out of training to fill all the available positions, especially now that the clinical commercial laboratories are snatching them up and often paying twice what academic medical centers can pay. I actually have a question for Bob. You noted that the UK is spending 20 million pounds on educating providers on genomic medicine. What can we learn from that or what do you know about that? I don't know anymore. I asked the question when the presentation was given at the ClinGen meeting, how much money is being earmarked for that and that's the number I got. It might be interesting to take a look and see what they're doing. It's a lot of money. I was just gonna say that one question since we're being asked also to directly ask questions of NHGRI or ask them to comment on things, is NHGRI interested in supporting research in education and I've heard mixed messages and if that could be clarified, that would be I think a really interesting question. I've heard that they are not interested in supporting education per se but possibly could be interested in supporting good quality research in education. So I actually would ask our education group to comment on that. I think it's a little bit unfair to say we're not interested in supporting education because NHGRI has had ongoing education efforts from kindergarten on for census inception. I don't mean that way. I mean in the form of grant proposal kind of things. So there are other mechanisms potentially to fund that kind of work. I think the more important question and one that I was gonna ask is, is there research to be done around educational models and what is most effective and what has the most take? I mean we have to decide what take is but when Howard was talking about reports and shouldn't we do some kind of program to unify reports? Well it would be awfully nice also to figure out what reporting formats or what content in that is most useful. I think in CSER we have several different models for reporting this back. We need to compare those. It's a golden opportunity to do that. So I just came out of residency two years ago and I actually had one lecture on genomics and it was one that I gave. So I'm very passionate about education for providers. I think making it relevant to providers is the key. So making it relevant to residents is either having their residency director be an early adopter and provide that education through mandatory training. The other way is through GME. So if the board exam included questions on this topic, then residency directors and residents would be interested in learning. If it's not on the boards then it's not as relevant as the things that are. And so I think that's a challenge in residency. The problem for the general practitioners who are already out there is when you look at their template of appointments for the day, they'll have a few follow-up appointments for medication refills, for blood pressure, for fatigue, for back pain. And you don't see anyone coming in for genomics, for genetic testing, right? And so it's just not relevant. I think the moment it becomes relevant is when someone comes in to ask a provider what they should do about their 23 and me results. And that's when the provider has to go figure out what to do. And that, in a 20 minute appointment, is a huge challenge. I'm not sure how to address it. We're certainly grappling it with it ourselves. I think info buttons, once you have lab results is great, but the providers don't even necessarily know what to order and when to order it. And how do I identify a patient who needs to see a geneticist or a counselor? It's, we have to sort of tackle step one before we get to two and three. The Air Force is, we don't have a lot of geneticists. We have a total of four, none of whom are tasked with seeing patients full time for the needs of the Air Force. They have other responsibilities and other jobs. So we're developing telehealth opportunities, a reach back capability, not just for the patients, but for providers to reach back to the geneticist and say, what do I order? How do I counsel this patient? Counseling, in my opinion, ought to occur before testing is done, as well as with the delivery of results. And it's a problem we're tackling within the Air Force. And I'm glad to see we're not alone. I mean, is there research already funded or planned that addresses that issue of when should genetic tests be ordered and how frequently are they not ordered when they should have been or could have been? I mean, obviously that's basically been done in pharmacogenetics, but are there other areas of genetics or other funded NHGRI programs where that's been true? No, researching that specific question, probably not. I mean, we get at that a little bit in Ignite and Emerge where we're really trying to implement this in practice. I'm not sure we do necessarily needs assessments before we go ahead and start doing the project, which is sort of what you would describe. You'd assess how often are they doing the right thing and then you implement a program and then you'd figure out afterward how often are they doing the right thing. So I don't know if the Ignite programs are doing that sort of thing, how critical, where we would put that in the priority, but I would ask. I would add something sort of tracks this conversation that we are funding an SBIR to Mickey Siegel who's developing a program called Simul Consult. And that's basically offers a variety of things, including allowing the physician to enter phenotypic information and do a sort of differential diagnosis. And then it helps choose which genetic test to order. And there's some EHR integration to that program. So that's sort of one small, it's a decent sized SBIR that we're funding to help us with that. That's an NHGRI grantee. When Ruth was mentioning the 23 Me example, and that happens to be one of the companies, but obviously it could be any of the companies. I mean, that's something that happens, is happening all over the country right now. And not every day, but when it does happen, it's shocking. And I get calls from my cousins who are general family medicine people saying, what do I do? And in that voice. And it seems like there might be an opportunity to partner with places like 23 and Me to help co-fund some sort of, so basically an educational grant. What's it called? Wouldn't they have, don't have restrictions? Unrestricted educational grant would be. To, yeah, for example, to do some of this education in order to provide a resource. Because right now they have a lot to gain from being better educated out there. We don't want to help them with their business plan. On the other hand, we would have a lot to gain in terms of helping the community grow, in terms of the knowledge. But so I think Jonathan may or may not like that idea. I would just say that the day that 23 and Me style risk prediction is clinically useful, you will see all primary care physicians adopting it. It's a question of where is the evidence of clinical utility, right? And that's what's driving why a primary care physician is going to use that type of technology. I don't think they want to use it. I don't know that they even want to use it at all. The problem is their patient has brought it to them asking for help. So they can say, I ignore that. Or they can try to help them. But I have to disagree with the idea that once something is shown to have clinical utility it will immediately be adopted. I mean, there's decades of evidence in clinical practice that demonstrates that's just not true. And so yes, we can demonstrate clinical utility, but we need more than that in order to make this happen. And I guess I would sort of ask, is this a suitable use of federal research dollars to try to work with a company to do something that I think many of us have some uneasiness with in terms of the direct to consumer testing? Right, I guess what I'm getting at is, you point to the full docket of things that that primary care position has to deal with, right? And when it just, we haven't gotten into the list of things that are important to do. On the other hand, one, I'm sorry, Mary. One could imagine, we've heard in some models of kind of a patient information pamphlet that's actually to give to their doctor. And it says, what to tell your doctor about whatever you just had done? And that might be something that would be reasonably easy to develop and something that these companies or others would have a stake in. Well, and there's certainly a lot, in our area, I don't know about the rest of the country, there's a lot of companies that are going to primary care physicians and saying, here's some testing that you should order preemptively on your patients and we'll help you interpret it and we won't build your patients and we won't build you and we'll do it. And some of them have gone over the line and had massive class action lawsuits against them and had to pay 24 million or whatever a few months ago. But a lot of general practitioners are now ordering these things because they have no reason not to until they get the results and can't interpret it. And so even just having some education on when this is even a good idea, which is right now probably very rare, would be valuable. Bob, you've been right on it. Yeah, I was just gonna circle back to this question that they're not motivated until they have it and in front of them was something that they made. Bob, you really need to speak close to Mike. We have a lot of competition in back here. That the practitioners are not motivated until they have something in front of them that they actually have to deal with and then they don't know how to do it. So it's kind of how do you get people to pay attention to the tsunami warnings? I mean, when they've never had a seen a tsunami before. And I think that the answer is that you have to keep trying, but also you have to build systems and that mitigate the impact of the tsunami when it comes. And I think that's the kind of thing that Mark is talking about and the information resources that will be there when people come or about. You wanna summarize? No. All righty then. I guess I heard a little bit about the distinction between educating clinicians about when to order tests or when to even think about ordering tests, which is probably going to be a more difficult problem to solve than what to do with genetic test results when they come back, which it sounds like there were at least some ideas about ways to address that issue, part of which could come through improved CDS and improved EHRs, but probably not all of it as it was pointed out. There's some test results that are very difficult and not amenable to just an automated CDS interpretation and recommendation for actionability. That there might be some funding opportunities to try to address these issues like what are the goals of education? When is the best time to educate? What's the best way to educate? How often do things like info buttons get used and if so, are they useful? Which are the kinds of practices that are most amenable to educating clinicians about using genetics and how to best do that? I'm sure I know that Wendy was taking notes and some of you are taking notes too. There are other major issues we should make sure we get summarized and I think getting some clarification from NHGRI about whether they're wanting to support research grants to do education or to test educational methods would be maybe helpful. Wendy's got some comments. Well, since I was taking notes so much, I didn't have the chance to speak up but I did wanna make a couple of comments in listening to all of this and my own experience with managing websites that have to do with genetic tests and genetic conditions and variations and so forth. And it's that in so many places, our assumptions are challenged. And Julie talked about how you might, and I'm sitting here thinking we have to wait for the young people to take over and then this will be halfway solved but apparently not. And at NCBI, I think what we're good maybe too good at is putting a lot of information on one page and expecting people will explore all over the place. And so that link is right there but when you look at it, people don't really click it just like you were saying. Are they gonna click the info button and we have to do even better kind of spoon feeding if you will, unfortunately. So in terms of education, it just seems to me we're really at the beginning of this. Let's not just say we're geneticists, we know how best to do it, we have to go out and ask. And that's really the kind of pivot, if you will, that we're doing for genetic testing registry and our other resources is not necessarily to assume that clinicians are gonna come to the genetic testing registry and go through all our filters which can find you any detail in any test. They really probably don't have the patience even if they come to our site. So I actually think we're right at the beginning of all this. So I just wanna clarify something and extend a bit of what Wendy's saying. The fact that info buttons aren't clicked doesn't represent a failure of info buttons. It reflects the fact that the clinician doesn't identify a need for additional information. And so if you build your best practice alert so that you have sufficient content, they won't need to click the info button. And in fact, the research in info button shows that only about 10 to 15% of clinicians will click through to get additional information on a particular topic. So I think we have to understand the metrics of success in terms of instantiating something like an info button. But I think that Wendy is absolutely right on is that what we really need to do is to have much more engagement with the clinician end user and I think ultimately as we'll hear in the next group the patient end user about what is it they think that they need as opposed to us saying, this is what you need. That's where we've failed over and over and over again in the past. Hand in the back, I'm sorry. So if you wanna cross off at least one item that you said, the last one unambiguously NHGRI through the LC program will support research into education, effectiveness and methods. What we don't support, which I think we used to was just the raw production of educational materials. And we can tell Robert that when he comes back. I think maybe one thing we also heard around the table is that if there is another country that actually happens to speak almost the same language we do that is spending some of it, some of it, yeah, that's right, some notable exceptions, but is spending 20 million pounds, which is not an inconsiderable sum. We need to learn from that and we need to borrow their information and their materials to the degree that we can. And so how can we translate it? Yeah, exactly, change all the SSTZs and that sort of thing. But at any rate, how can we sort of have a dialogue there? I think that's something we really need to look into. Okay, thank you.