 Great. All right. Why don't we get started Renee, take it away. Thank you. So I just want to thank everyone first for for coming today. We know you all have really busy schedule. So thank you so much for taking time out. To give you a little bit of context as to why we're having this working group. A lot of you did attend our genomic medicine 14 meeting that was held a couple months ago. And two of the items that came out of that meeting was the need to create a consult service or expert panel to help educate clinicians about genetic test orders interpreting and determining next steps. And by clinicians were really talking about those non-genetic trained providers. And then second to that was to develop a learning community of practice, a listserv or something to provide information and updates, you know, potentially supplemented by a panel of experts. And really what today's workshop is about is how we could actually develop those. So today what we really do want to do is examine the infrastructure and the logistical needs to identify barriers and solutions to putting that together. And then to also talk a little bit about outcome measures, how could we measure what it is we're putting together. So that's our big overarching goal for today. Did you want to add anything to that. Thanks Renee that covers it except I might just mention one of the things we were urged to consider at the 14th meeting that Renee alluded to was to ensure that we promote equity of implementation in low resource and underserved settings so that's always a challenge one that we want to be sure to address. Absolutely. So, the way we wanted to start was just if we could introduce each other and then we'll move really into our discussion, but first, let's start with introductions and we just want a brief introduction your name and which institution you're at. So if we could start with Adam. Hi, I'm Buchanan chair of the Department of Genomic Health at Geisinger. Thanks for one. There's on how me, Professor of Pediatrics clinical and chemical geneticists at Vanderbilt University Medical Center. Leland. I am Leland column, general internist at Mass General Hospital. Muin, and I know. I'm Muin Corey from the CDC Office of Genomics and Precision Public Health. Melinda. Hi, I'm Melinda Massard. I'm a family medicine physician and I run a primary care precision medicine clinic here at UPMC and Pitt. Howard. Hi, Howard McLeod. I'm medical director for the Geographic College of Consortium and executive clinical director for precision health for the Intermountain Healthcare System. Carolyn. I'm Carolyn Menendez. I'm the director of the very new program at the VA through the National Oncology Program called clinical cancer genetic service. Dan. Hi, I'm Dan Rader, University of Pennsylvania where I'm chair of the Department of genetics and chief of clinical genetics divisions in medicine and pediatrics. Nathaniel. Sorry, Nat Robin. I'm the director of medical genetics at University of Alabama Birmingham. Todd. Everybody had scored me at Indiana University professor medicine and the director of the Indiana Institute for personalized medicine. Jason. Hi, I'm a general internist at practice at the Boston VA, but I'm also involved in genomic medicine activities at Brigham and women's hospital. And then Terry. Great. I'm also a general internist Terry Monoglio. I lead the division of genomic medicine and you Renee. And I'm Renee Ryder. I'm a new program director at NHGRI and I am a genetic counselor. Okay, so the first session that we want to start on was infrastructure and Terry's going to moderate that. Great. Thanks. And what I'd suggest we do, we have six items that we want to pick your brains about. And what I thought we would do is, you know, we've got about 50 minutes so maybe spend six or seven minutes on, on each of these and what I'd like to do is kind of go around the virtual room alphabetically and just ask, you know, three or four people for their input on a given one and then we can have some discussion. And that's, you know, for warning year that you're you that you're up first for types of expertise. So, keeping in mind what we're, what we're looking at here is setting up and studying because we are a research institute, a nationwide genomic medicine service so so not traditional genetics but really the more the broader genomics results and input. So setting up consultation service to deal with those kinds of things. So, Adam, what thoughts do you have on the types of expertise that that might need. Well, in addition to the genetics expertise itself, I think a couple other types. So, we've gotten a lot of input from primary care clinician colleagues on implementation in the routine care so the implementation of that information that's critical here because that touches the breadth of patients over the longitudinal care that you need to provide. The other is some implementation science expertise as well. And so many of our attempts and genetics to try to educate our non genetics colleagues have failed because they haven't fit and so they know we haven't brought the right sort of expertise to understanding what we're looking for care and implementation provides some of those tools. Super. No, thank you. Riz one do you have thoughts on additional expertise. So, when you say Terry explain what you when you say additional expertise, what does that mean. So I think Anna has identified that we would need genetics expertise in a in in a console if we were to set up a nationwide console service primary care physicians are very interested in implementation so we probably need some implementation science expertise. In addition to a geneticist. I would say informat test station or informatics person would be. I think important because somebody with knowledge with somebody who can deal with databases and has some degree of programming knowledge because I guess the way we think about the way I think about this is it will require some some aspects of it will require some information. And I think it will be important because we as clinicians and as, you know, research scientists we can come up with a lot of ideas, which may sound great on paper but may not be implementable within a cost structure or time frame but those are. In addition to obviously that you need you need clinical geneticist and you need folks with genomic knowledge, whether they're PhD folks and genetic counselors, etc, etc. But I think those, those three for types of people. I think there's a lot of people with data who can who have clues about me not clues but who expertise of database management, people who are informatic specialists. Okay, great. Thank you. So first one alluded to perhaps genetic counselors as well that hasn't come up yet what do you what do you think. I mean I think from the perspective of translating both to generalist clinicians as well as to patients, you would, that's where kind of general genetic counselors would be invaluable. I think that, you know, at least at our hospital in terms of kind of questions that we get the primary care sends in for a service like this, it's overwhelmingly cancer genetics, OBG genetics. And then I would say that beyond just thinking about what specialties you need. You might need to think about just more of the triage within even the specialty fields to the specific person who handles that specific concern because that is a huge navigation issue within like cardiology. Who does arrhythmias who does, etc. Even when you have the specialty, who is the specialist. Okay, great point, Leland. So when it comes to the triage is that I know in some of your services as they were described in the in the meeting materials, some have a genetic counselor do that some have other people do you have thoughts as to who would be a good triage person. Yeah, so we, we started a pooled E consult service where we've actually had a genetic counseling assistant who works under a genetic counselor who kind of is like many of you know Heidi Reem, of course, who and her like genomics groups so they have kind of a footprint across a lot of different specialties that have offer specialized genetics care. And so the genetic counseling assistant many times because they're scheduling in those clinics can will know kind of this is the person but if not could go to kind of a supervising genetic counselor in order to kind of direct to like within that facility. This is the subspecialist for that concern. Super. Okay, great, thanks. Moine, do you have additional types of expertise we should consider. I think I may have a lot but you know I'm lacking the context, because I wasn't at the genomic medicine meeting and may not have read that recommendation so if you know just looking at the materials for today. But a little bit of a loss because you say you know we want to do this but it's not traditional genetic services is genomic medicine. I need a little bit more before I can offer my opinion, maybe I mean not the whole lecture but just a two minute. I think Renee described it well and I should have should have reiterated that we're really talking about non geneticist, probably primary care or non geneticist physicians as well as nurse practitioners physicians assistance etc. So, so when they get a result back or when they see somebody that they think might need genetic testing, you know, what kind of expertise do we want to send them to and I think we heard about some some sub specialties, cardiology and oncology would be fairly obvious I think that we want to have people who have genetics expertise within those sub specialties are there other specialties you can think of way. Actually, can I jump in real fast and just also say that, you know, I think what we're also trying to do is differentiate this from a traditional referral service that a lot of places are having a hard time getting. They don't have a geneticist in their, in their facility so they don't have someone to refer to. So what we're really what we've been asked to do is to examine the creation of a consult service so instead of sending your patient to see the geneticist, the consult service would let you figure out as a primary care physician, how to take care of your patient. We'll let you, we'll let you think on that. We'll let you take on this. Okay. All right, please. Dan I noticed you nodding did you have anything you wanted to add. Yeah, I have a few few things to add I think it goes without saying I think we definitely need LC expertise here to be part of this. Suggest that you some element of, you know, expertise in health disparities social determinants of health I think you comment on that Terry I think we really need some real input into that. Another would be some of these are going to be research results that come back to, you know, the patient and then the physician and the physicians like I don't know what to do with this so someone who has some real actual expertise in return of research results might be going to think about. That's a whole discipline unto itself of course. Behavioral economics is a field that is very big at Penn and maybe I've drunk the Kool-Aid but I think, you know, it's a big field that has to do with incentivization not just monetary but how to get people to do the right thing you know especially caregivers. So expertise in behavioral economics I think might be something to think about. And then Adam didn't mention it but pharmacogenetics it seems like this pharmacogenetics could be a component of this service in fact I think increasingly it might be. So that would be another area to think about. Great thank you. We need to move on to the next issue but was there anything else that anyone wanted to add. Yes, Carolyn. I just I just wanted to reinforce the statement about is someone with expertise in cancer genetics. So I think I think having an oncologist I'm a surgical oncologist a breast surgeon in my program and I think it does add an important layer to a service like this. Great, no good point. And you raised your hand as well. Yeah, I want to echo the behavioral change need right because primary care is so busy and really getting them to take on any education around how to integrate this is a huge struggle and challenge and very overwhelming so really thinking about how to drive that behavioral change. And then to I really think we need to think about the payers as well, and how we can actually simplify and streamline testing as testing is so complex and nuanced. We know that it's actually overwhelming right now in its current form, and if we can advance payer support of streetlining testing and that will really go far in terms of creating orderables that are much more clear to non geneticists. Thank you. And I see we have two hands up when I'm going to ask you to hold your question because we do need to move on but Jason we're in already so Todd go ahead briefly. Yeah, like Melinda said the biggest thing that drives testing for us is reimbursement that basically so somebody who understands that how to appeal how to do that. I'm concerned to use it comes up is legal it's like, if I do this and my lot what am I liable for if I'm the ordering doctor and I put this in the records. But I think legal is a, you know some sort of legal advice on that is critical and then a pharmacist, because it's not only for the pharmacogenetic but also for the other stuff because what drugs they're on what drugs they're considering plays into the, you know the genetic interpretation to that. Thanks, Jason last comment. Yeah and I would just play off Melinda Todd's comments. So yeah we should work with payers to improve the situation but with the expertise we would need if we were going to stand this up now is someone who can navigate the current insurance system as it is. So when someone calls up and they're uninsured, they'll get different advice and about what kind of services they can get then if they have Medicare or if they've got VA insurance so someone who knows how to navigate the current system, which is going to be tough on the national level. Yeah, absolutely. Great. Thank you. So, so I mean, since I sort of surprised you with the last one would you like to comment on the target audience for these kinds of. I mean, this is such an important work. I mean, I think the target audience would be the primary care the physicians at large who I mean we're not geneticists who see patients either for prevention or, you know, regular care but if I can get to that first one also because I'm thinking about that package. It looks like sort of in multi level implementation science, putting all the pieces together all the disciplines, and is this going to be at the end some kind of a massive national research study to evaluate the components and put that in the parking lot because of course I mean we need the ethical the legal navigation of all of these but you know in a in low resource settings I mean or whatever I mean people want to consult. I mean, it's looking like you're describing a massive effort. And maybe I'll ask the other panelists whether all the components are needed and for what purpose so just just put it out there. Yeah, I think, I mean, if you don't mind, we'll, yeah, we'll, we'll kind of maybe focus on on the things that we've addressed I think that will come up in the other topics that are raised but let's as you say put it in the parking lot. We're not thinking of something massive here, we are thinking of something practical and useful but we also need to do some research around that. So, maybe if, if I could then go to Melinda would you like to comment on the target audience. Yeah, I mean, I'm the target audience. I am. I am a family medicine genomics practitioner and and would love to see all of, you know, my colleagues become the same. Many of you have heard me say before but I truly believe that primary care practice is about 70% of every subspecialty. And there's no reason that genetics and genomics could not be one of those. If we could start integrating it into the education system from the get go all the way through residency training and then ongoing CME for for for actual real life implementation. I think it's everywhere from the physician to the, you know, to the mid levels that the APPs who are working heavily in the primary care space to pharmacists who are working heavily to support primary care providers. And so I think that really is the target audience, although I will also say I think there are many specialist providers out there who are adopting genomics into their diagnostics. And they too need to become non genomics practitioners or non genetics practitioners of genomics and precision medicine so the audience is quite large. Great, thank you and what are APPs. Sorry. Oh, sorry, advanced practice practitioner so PA and practice. I've never heard that. Okay, great. Thank you. Howard, do you have thoughts on audience in addition to what we've heard. Yeah, I think the biggest need is, is at the clinical edge. And I would, I would argue that they don't want to learn about genetics just want to know what to do. I think the audience that needs this the most are the folks that have 30 seconds to take on something and then apply it so it's, it's a rapid fire audience of all the people. We've all that we've already talked about, including the maternal field of medicine folks which we haven't really been brought up as as clearly now because they are also a kind of a real time environment, as opposed to setting up appointment for three months from now. Great point. Yeah, and I think that also the need in medical so specialties probably pediatrics specialties as well because not every cardiologist knows what to do with sudden death or other things. So, maybe last person on this on this issue Carolyn, do you want to make any more additions. I think it's very important to bring in the APPs as our target audience, because that that really when we think about cancer genetics we think about the survivorship clinics. So really I think with a large initial focus being PA and nurse practitioner programs, getting this education, and, and going backwards just a little bit that kind of made me chuckle that I left out, marketing people, because I'm, I'm, I'm in the midst of rolling something out on this scale doing this national VA project, and you come up with these beautiful plans and then you say how we're going to tell everybody what we just built. Great point. Yeah, yeah, because that is, you know, a real challenge I think how we're going to do that. So are there other comments now on target audience or do we have it pretty much covered. See Dan you asked a question about assuming patients would not access this directly but rather through clinicians I think that's what was recommended. Yeah. Okay. I think that's right. Okay, yeah, now I, I shudder to think the kind of service we'd be able to get there if patients could access it so. All right, why don't we move back, move on then to the type of user feedback me what we might want to get, keep in mind that the, the genomic medicine 14 meeting was on genomic learning healthcare systems that have that virtuous cycle of implementation and modification and then re implementation so so assuming we won't get it exactly right the first time. Dan could you comment on on what kind of. Sorry, it's not Dan it's not sorry Dan. Could you know I guess it is down my apologies. Could you comment on what kind of feedback we might like to get from the users of this. So that brings up another form of expertise which is expertise and qualitative research so clearly we're going to want, you know, reported surveys from the from the users of this in terms of what their experience was in a formalized qualitative research kind of way. Obviously there are a lot of metrics that that would be generated. That would be useful. It's nice to have some sort of quote outcomes meaning what what actually happened. That's going to be harder to collect and a lot more expensive. I'm just talking off the top of my head those are a few of my thoughts. Thank you. And we do have actually we have sometimes specifically set aside to talk about outcomes. So, agreed. In terms of the making the system better I think that's what we're looking for on the feedback so so thanks. Nat, did you have some thoughts on the feedback. Yeah, I agree. We need to have some feedback and but there's, if you want to guarantee people not using this, give them a big survey to fill out as soon as they use it. So, to me, in my experience with this the biggest thing people want is timeliness. So they want to make sure their questions are handled in a timely manner and completely. So you could kind of do biopsies where you survey maybe every 10 interactions, and I think surveying the patients is equally important to make sure their questions are answered. I could circle back one second. I read what everyone's doing I read what we're doing, you know the plan I sent in is only like a preliminary plan we've really gone back and forth on this over the last couple of years are trying to figure it out. And what's clear to me is the danger of doing something at a national level is that one size doesn't fit all because the needs of Boston are very different than the needs of Birmingham and the state of Alabama. So I just want to point that out that while we have some really good ideas here and smart people thinking about that. There are different things so you know we're going to be servicing if we do something like this will be literally servicing chunks of four states, you know, and we're going to be servicing people who may not have internet access. So I think that's something that needs to be remembered especially when you talk about equity. You know the folks were it we're going to deal with the cancer surgeon Selma Alabama that's very different than dealing with somebody in Brooklyn for you know if you're downtown Boston. I mean financially necessarily just mean for access once right and any thoughts on how we can make this accessible to Selma as well as to Brooklyn. Yeah, a lot of things but I don't think you know that that's the I don't you know what again what we're doing here doesn't necessarily make sense and one of these are hospitals doing is we're buying up. Most of the smaller hospitals in the region we're creating networks so there has to be some kind of infrastructure, either with a shared EMR, or an accessible EMR. And there has to be, I forgot who mentioned but a very wildly publicized service and that's even within you a be we're trying to do that is get people to understand what is the economic benefit Dan mentioned that the holding my behavioral economics we've tried that amazingly but we've tried to show people this will save money the only time we've ever gotten any headways when we show in the hospital, you're losing X amount of dollars because people don't know what they're doing around genetics. People are not happy they're not getting the proper pre and post as counseling. One other thing to point out real quick is I don't know how many states have lost track of counseling licensure for genetic counselors. Those states have pretty strict restrictions on who can provide pre and post as counseling it's within the state. You can't have a primary care provider do that violates state genetic counseling laws and many states it doesn't Alabama for example. So there are all these other issues that I think are state based and will be difficult. I don't want to say impossible to overcome but maybe one of those things where there is, you know, I don't think, let me say different. I would bet there is not one solution that will fit 50 states, not even, you know, eight regions of the country this may be something that has to be done. Initially, if not a state by state level just a couple state level. That's excellent point. When you had had a comment. I want to want to continue that train of thought about state versus national. You know one of the. I mean there are 50 states obviously each has their own flavor of health and health care and delivery of services whether they're genetics or otherwise. And I think it behooves NHGRI from the beginning to do that state partnership through the organizations that speak for states like the Association for state and territorial health officials. The health officer of each state is kind of in charge of the health of the state if you can get the buy in of the state health official then they can work with the state and local. They try to do the, you know, the equity I mean rural areas and Alabama are different than rural areas in Nevada. So I think developing the one size approach at the national level will probably be succeed if you do that state by state sort of public health approach in the same lines. I mean, you know, HRSA is, you know, that's generic services as you know, and they're reinventing their enterprise I mean they're, they're traditional, you know, the regional networks are only focused on maternal child health issues. I visited CDC not too long ago, about two weeks ago actually I had a chat with the new division director and they're thinking, they're thinking outside the box a little bit so genomic medicine is not generic services as you point out but that kind of partnership with CDC HRSA and through the states and the regions could be beneficial for such an enterprise and could complicated a little bit, but I think longer term success will be more assured. Great. No, that's very helpful. Thank you. Yeah, I think Todd we may shift over to the equitable access which has has come up already but if you had additional thoughts on on access. I think it's, we need to remember this is a different equitable access than trying to contact patients out in rural areas and do that right so really talking about the primary care docs or pharmacists or nurses out in there so I think the issues some of the issues we face with, you know, patients not having smartphones or having paid, you know, time things I don't think are issues I think as long as it's, you know, either, you know, even phone or zoom or, you know, online chats or, you know, some sort of thing that I don't think I mean it seems like that would, and there's not a big cost to it, and it's fast. Right, so you don't have to wait three weeks that you know it probably isn't be instantaneously but you know the amount of time it takes them to do it in the turnaround time, as long as those things are met. I don't know that. I mean I think those be key things to make sure that you know somebody out in the world, Indiana would be able to use it. Absolutely. Excellent points about the difference between reaching out to patients versus reaching out to two providers, and I think we're focusing really on the providers here. So, Jason. If I could jump in real quick, please, because I think that there's an important point here. Again, my experience different than other people's experience but the majority of providers outside of the academic medical center have. They just want somebody to say here, help me order this genetic test, tell me and then help me counsel them, because they're not equipped to do pre post test counseling. Right. So while I agree we're talking about reaching out to providers. And I think we can't forget that many instances, part of this will be reaching out to patients as well. Good point. Good. Jason, did you have additional thoughts on equitable access. Well, I'll talk anyway. So, I guess by the fact, you know, by the premise that we're already talking about patients who have some amount of access right they've made it to a clinician, the clinician now has a question about whether there's a role for genomic medicine in their healthcare so already, the patient has already passed that first hurdle, which is a big one to access medical care. And now there's the point where a clinician can actually think formulate the question. I still think that the question then, when this when this provider is then calling up the service or accessing the service. A lot of the, a lot of the impact on whether the patient has equitable access will be how knowledgeable the this expert consultant on the phone is this hotline is about how to get that particular patient with in their particular social circumstances connected to the right test. And if they're uninsured, what is the answer to that question. And so, I still come back to kind of being able to navigate the, like the real world where the rubber meets the road, what lab do we send this patient to kind of kind of questions and who's going to pay for that. And, and someone at the national level who can know the answers to that is going to require someone who's very resourceful. Genetic counselors actually as a profession are very resourceful so they actually might be very well positioned to answer those kind of questions. But it's going to take that kind of expertise. Great. No, excellent point. Thank you, Leland. I mean, I think that, you know, I was originally envisioning that this is a centralized service right but I think really more and more. What I'd be thinking about is genetic counselor navigators who can, you know, go to a clinic or have a region that they rotate through to provide kind of on the ground care. Who are your patients who you're concerned about maybe those patients have been on the back burner for a while because you don't know how to navigate for them, or they have. How do you avoid maybe some of the, you know, on an equity standpoint if somebody writes in the chart, well get six months breast MRIs and those aren't covered for that patient from an e consult. And you're like, Well, I can't follow through on that recommendation, then who's on the hook for that, you know, from the provider standpoint so maybe having that conversation kind of upfront, rather than all electronically. Excellent point. Adam. And thinking about where to target both development and evaluation we can also think about some underserved communities and specifically federally qualified health centers and looking to get down into those places and make sure that we're hearing what the key issues are there and that gets us a little bit closer to Jason's point and actually getting to the patients who may not have those regular interactions with with the health system so I think we want to be really deliberate about where to focus the development efforts and then where to think about the evaluation piece as well. Super. Thank you. So maybe we can move on then to to something that has come up a little bit in terms of licensing and liability. I don't know Adam if you want to pick up on that topic. So, there are certainly cross that line licensure issues in, well, all of medicine, really, and it's when you talk about tonight counseling it is complicated by the variability in which states have licensure which don't and what the rules are and under each states medical that said, there are groups that have solved that pretty well, including some of the telehealth companies. And so, you know they solve it by having individuals who have. They've got kind of a combination of licensure that ends up covering the country so taking a similar approach with the consult service could help solve some of that. Great point. Thank you. And I think after Adam we have respond go ahead. So, so I think I will go back to what net for saying earlier, I think, and that actually brings in the licensure issue. I think the solution to this. Well, energy or I sets a overall guidance, guiding principles and provide support, the solution to this would be at a state level or regional level because it is just impossible for a central organization to know what the insurance costs are at a state level, for example, at Vanderbilt, I can provide counseling to folks in Kentucky, you know, Carolina's gorgeous because we get to see patients from there. So, so I think the solution to this is has to be has to be driven by state or regional levels I think one thing I would just point out is that, you know, talking about diversity and equity. So part of what we encounter encounter is that that some populations in states are reluctant to interact with big medical centers right so many times policies are driven by big, you know, medical centers like Vanderbilt and stuff so one way to kind of bypass that issue is look at how the state new one screening program was set up. Right. We don't, for example, we at Vanderbilt provide new one counseling or advice about hey this test is what do I need to do right so there's a framework set up and we actually have people reaching out to us from community hospitals like the computer patient family practitioners nurse practitioner, because they feel comfortable with the way the system is set up at a state level. It's not like a Vanderbilt it's like a state of Tennessee new one screening program which is set by the genetic advisory committee. I've always wondered if you're going to do that, whether that's the way to do that maybe this, there's a state genomics advisory committee with this of the system and in powers. The, let's say the in Tennessee there are three major medical centers who provide the new one screening coverage, something similar to that so then you are, for example, at Vanderbilt I am in response for middle Tennessee I know all the insurance I can actually tell the physician, what to do what will be covered. I know my own state Medicaid, Medicare rules. So I think that is, yeah. Yeah, that's very helpful. We're sorry to interrupt you I just I want to be sure to give others a chance as well and when you've had your hand up for a bit. Yeah, I just want to continue that discussion because it looks like we're, I mean I want to continue down that state state rabbit hole. There is a way. If we take newborn screening as an example, where state and national have interacted successfully I mean newborn screening panels where, you know, sort of the Wild West for many many years until the advisory committee came in set some standards and recommended the universal universal screening panel, and it got things organized the implementation of genomics and practice may require something like this it's not a state run program like newborn screening with all the blood spots go to the state labs for testing but because of the intricacies and the differentiation among states with vis-a-vis practice vis-a-vis, you know billing and, you know what's covered and not something like this could actually benefit not just the consult service per se, which is kind of a narrow, narrow perspective here which we're talking about but could facilitate the whole implementation pathways for genomic medicine in general. Now how you go about doing it can become highly politicized you know advisory committee is take it's not ever they're not easy to set up, but we can work to try to help you do that. It's a way of thinking at least thank you. No that's very helpful thank you. Maybe any last comments on the the issues related to liability and licensing. Very quickly that if we're not practicing medicine and we're curbside consulting as it were, then that may have relieved some of the licensure concerns point. Yeah, I would just say we did we you know we have this sort of system going in our institution, we're trying that at random institution level and I think we were told by our legal department that yes, if you give advice, if you get an opinion to another provider. You don't have that you don't have that higher bar about liability. I think it's the liability becomes the issue if you're directly giving advice to the patient. Great. Okay, no that's that's good to know. So just in our in our last little while sustainability is always an issue with genetic and genomic services. And I saw in some of the materials that some of you bill your practice basically for providing these do you have some some thoughts and I guess we're up to Leland, some thoughts on on how one would sustain something like this. Well actually when we were talking about measurement before we didn't have too much I one thing I was thinking about was measuring the downstream burden on the referring clinicians to any kind of central service, you know, with any kind of E consult there's downstream work orders, etc. And so, considering that as part of the evaluation, I think would be important. Okay, great. Thank you. Now can can I pin you down a little bit on sustainability. I think that that directly impacts sustainability, because if you lead to a bunch of work that is, and I think that that's also where, as Dan mentioned, the qualitative data and really getting probably rich qualitative data in terms of kind of what is happening, not only for the patients, but also the referring clinicians and or genetic counselors and like a navigator program and how would be invaluable. Great, thank you. Howard. We've, once we quantitated how many surgeon slots were opened up and how many drugs were now able to be prescribed in oncology. The things changed a lot because you now are basically it's it's genetics becomes a physician extender or surgeon extender, which doesn't sound as glamorous as a geneticist but is worth as far as the dollars go. And, you know, a surgeon in the OR or a surgeon seeing a new patient has higher value than a surgeon seeing return patient or something that is less well reimbursed so that's been a lot of the dirty secret part is, it's, it's putting it's going top of license to different folks and there if you quantitate that you can pay for the program. Great. Thank you. Melinda and Carolyn have their hands up so Melinda. Yeah, I think that it's important because the historically a lot of these services have been uncompensated and and in particularly genetic counseling. And I think it's really important that it actually is a billable service and that these dollars are captured for sustainability. And I think it's absolutely critical that the genetic counselors become billable providers to be able to support any kind of programs such as this growing across the country in any region. And, and so that's the interesting thing, multiple models have been suggested today right from a national level to a state or region level consult service and. And, but regardless, I think that there needs to be some way to financially sustain that and expecting like any GRI or, you know, that even the government to do that individually or independently is probably not very reasonable for a long period of time. So clinic, this is essentially what we do we essentially have a console clinic and, and it's been highly successful and completely pace for itself and all of the staff, because we are actually billing and they are mostly new like Howard said they are mostly new patient providers. So we're billing. They're very complicated and time wise we can build at a very high level. So I think this is a really critical aspect that has to be integrated it does then tie back into the liability discussion again because then it's actually patient care and not just necessarily providing information to two other providers but absolutely has to be integrated for sustainability. And let me just ask when when you do this, you don't have them all come to you presumably you can do some of these virtually or how does that work. Actually are almost 90% virtual. Wow. Super. Caroline you had your hand up and you took it down. And no I just took it down so it wasn't annoying because you had already recognized it. My, my issue about sustainability is kind of a funny one is you have to make sure that it's not too big of a hassle for the people actually providing the service. And so when we talk about our target audience, part of what we have to include is the actual consultants in that target audience. It has to be something that is rewarding, you know, challenging but rewarding and it can't just be painful bread tape every time. So it has to be vetted enough to where it's a experience that your actual providers want to continue to provide as a sustainable service. There's a lot of things that you need to do in multidisciplinary conferences and patient care, where you're just constantly losing the providers because it's too long a block of their time, and too big a pain in the butt. Those are excellent points thanks and before I call on that, I might ask, Todd, you had made put a comment in the chat about hopefully eventually a private providers get to learn that they know enough about this so they don't need the console to help you in your experience in that. Yeah, I think, you know, we've, like in some of the far good jack stuff here, eventually once the providers start using that, and they don't they no longer need to consult us anymore and so then it's just self sustaining they just do it as part of the practice now. I think that would be part of the goal that said then I think the console service, there's always going to be new things coming down the road right so then it could move from stuff that maybe today needs a little, you know, a couple years from now maybe that now everybody knows how to do that and they move on to some new other things so that's sort of the idea so the sustainability part becomes essentially an education. Super now it's an excellent point it's great to hear that that's happening to. And Dan you put in a comment about a model where this is so value to the genetic companies that they contribute to sustainability. That's a great question. It's this. Yeah, it's a question I mean obviously this could lead to a lot more utilization of genetic testing over time, which of course would benefit these companies they they employ a lot of genetic counselors. I just, I don't know. I'm just putting it posing it as a question. Do you have thoughts on that specific question. Yeah, it's just overall, I think sustainability is such an important question, and we struggled with it and I think the licensing part where you're giving to make this service sustainable, you. I think getting getting reimbursement is important that then will cut down on equity. And I think that will bring up the issue of licensing I think somebody mentioned that so I think sustainability probably requires a much bigger discussion. Once, then we can think it's settled. Okay, now thank you. So last comments in three minutes, Nat and then we. Thanks. So one thing to remember when we talk about billing for these consoles, we're getting $15 maybe the key to me is institutional support and one of the big hurdles we've had an institutional support is we cannot genetic counselors are not independent providers there. They're not recognized as such because they can bill independently. So I think a lot of this comes down to the national the national effort to get genetic counselors able to bill independently they have licensure. And I think once that happens I think that'll set all the cascade of things where counselors can build they'll get back, you know a few dollars, a few cents on the dollar for their time but really that's all the hospitals want to see. And then, then having the data which exists to show that these kind of services help at the institutional level. And I think that's a cascade that will prove very successful but again, I'm kind of focusing more on this effort at the local level, not the national level. Okay. No, that's fair, fair enough. And then last comment in a minute and a half. Okay, so sustainability really depends on the ability of the initial, the initial setup to show impact in terms of whether or not the right diagnosis or the right prevention or the right treatment or the better health outcomes can be measured. So this could be highly influenced the way you set it up which services you would focus on, and I would, of course, favor the idea of focusing on those services that what I call tier one meaning that there are accepted evidences of clinical validity and utility, so that, you know, at least in that domain you can begin to make an impact if you focus the initial offering that way. That's a great point. Thank you. So Renee, I think we've just barely made it. So, at this point I think I'll turn this over to you for the next topic. Okay, perfect. Now I know that Adam is going to be leaving soon so I did want to introduce a new panelist who is going to take over for him when he leaves. Can you promote Cassidy to a panelist. So while he's doing that. Did someone know. Okay, so anyway, so we're going to be having a new panelist her name is Cassidy. Before we move on though it looks like we have one hand raised when did you have a question before we move on, or is that just an old hand. That was an old hand. That's fine. Cassidy, could you just take one second to introduce yourself to the group. Or hi everyone, I'm Cassidy Colada. I'm a genetic counselor at Geisinger. I work directly with Adam Buchanan and one of my roles at Geisinger is as co-chair of the genetic counseling professional council. This is a charter group internal to Geisinger that is directly responsible for the recruitment retention education and professional development of our genetic counselors and genetic counseling assistance. Okay, thank you so much. So our next section is, you know, assuming that we have a working consult service that other topic that we were asked to address through the meeting was coming up with a way to curate advice for recurring questions. So like, you know, we get the same question about BRCA one a lot, you know, is there a way to document that so that people can go and look for those answers, you know, examples of models are the ClinGen or the NCCN guidelines or CPAC. So let's start with, you know, are there any of those types of models that you guys think would be useful and I will start with Jason. I hope so. I can't name any off the top of my head but the analogy came to mind was, you know, nurse triage lines. So this is outside genetics but, you know, things that are very common and can be protocolized in essentially a phone script. There's someone on the phone who and maybe this is genetic counselor or even a genetic counselor assistant, or someone someone, you know, who can be trained in a very in a phone script to really help help get maybe 90% of the work done. I would defer to genetic counselor colleagues for what what whether such kind of rubrics exist and how to how to get things, you know, is there a BRCA one, you know, phone script. There seems like there should be. Next, let's go to Adam. So the maintenance of content is a big issue here, you know, you can think about a learning activity chat GPT that keeps up with those BRCA associated answers or something like that but that's, that's always the big challenge here for anybody that's trying to build clinical decision support like this somebody has to keep the information up to date and also has to put it in a place where clinicians are accustomed to looking for it. We have built things in the past that have that nobody's paid attention to you because we didn't put them in a way that they can be easily accessed and what didn't fit into the flow of care. So, you know, up to date, for example, is accessible through the electronic health record physicians go there when I have a question so we should think about looking to tie into existing places that are normally part of a physician's workflow. And that don't require them to go lots of different places. I mean, did you have anything to add. I'm not sure really. I know there are standard resources out there but I mean, no, I'm going to just pass on this. That's fine. Okay, let's go down to Leland. Well, I think that this comes down to, you know, also, maybe it's a knowledge gap but also, we're talking about personalized medicine right and so people want to feel like, even if they may think they know the answer to a question, is that answer appropriate then for that patient, right. And so sometimes this may not be a battle gap in knowledge but more about reassurance for that person. And so I think thinking about addressing, even if that's just a pat on the back saying you're right. I think that, you know, simplified methods of, you know, saying you're right. Good for you. Whether that's a phone triage or a message like good. Or I think also just we've talked about training and I think there's been a talk about training like the next generation and genomic medicine but I think really in general for training we know that across medicine we need to shift towards the consumer of up to date information as opposed and using that information effectively as opposed to potentially memorizing information that's going to change and genomics is no different from the rest of medicine so I'd encourage thinking about it that way. Melinda. I think that is a really great point and I think that's the biggest challenge with any curated data database at this point right is this that knowledge is constantly evolving. You know, literally daily in this space. I also agree that it really needs to we really need to think about leveraging resources that exist already like to date right and utilizing the current workflow of practitioners. And even clinical decision support embedded right directly in the electronic health record itself so really at point of care clinicians can access what they need in that second, and then be able to access larger information or trigger a consult. So I think this is this is a really big challenge. I like the resources that folks have already suggested. But I also think we don't want to keep building new resources we should leverage things that are in existence. That final point is exactly what I was going to say is, I've been on the nccn panel for the last five years, and I know how hard we work to keep that up to date and I know that the minute it's published it's already out of date. And I know that last year we added a recommendation for an annual follow up clinic. So I'm on the HBO see genetics and genomics panel. Because examples like palbie to the everything changed yesterday we gave you good advice, but it was wrong today we need to tell you to consider removing your tubes and ovaries. So I would strongly encourage leveraging resources that are there being a great source that has links to places that are very carefully maintained and strongly discourage trying to recreate that it's a it's a big effort that's out of date the minute you're done. Cassidy, do you have anything to add. Yes, so beyond just thinking about these models for for providers. I also think it's important to build in that education and communication piece as well. So, you know, thinking about success of providing presentations back to a group of providers and taking those nccn guidelines as an example and applying them to some case presentations to help make it all of the information tangible would be an additional resource that we could build in. Okay, Howard. One of my favorite things that's been said so far was by Leland, and that was around the idea that someone has your back. What we found is that we create a consult service. It's rarely used, but it's one of the most popular things we do. And that doesn't make sense, except the idea that someone is available to have your back allows you to make decisions that you're wondering about and you think you know but now you can go forward and you know if nothing else if we could do that nationally. If anybody who wants to do something and thinks they know the right thing knows that somebody has their back that would push things forward faster than a fancy database. And so, I just, I love that she said that so I want to say it in a different way, another time. Todd. So some of the scenarios where it's, you know, their repeat questions that that you're mentioning, it might be related to the somebody mentioned to having a case report if, if you could have some ways okay we've got this test on whatever like a Lynch syndrome test or BRC one or another test, whatever, and then have a variety of things that might influence the path that that goes so the provider could then say, Okay, yeah, they're over 65. They're on these, you know, they're on a medication that's x, y or z. And then that would sort of give them the workflow and answer the question so they could, you know, put in some of that sort of stuff might be a way I mean we do similar things like that for some schools case studies that without we do, I don't know if it would work but that sort of blue sky and that idea. Thank you. Yeah, so, okay, as I've listened to other people I've kind of had two thoughts one is something that Howard mentioned I don't mean this in any way. And I think Lee one mentioned to disrespectfully but one of the things that terrifies me and my experience are the people who think they know what they're doing and they're wrong. So, you know, having the resource they're not using it, because like okay I got this resource and now I'm going to move forward I've seen way more times people will then make the wrong decision to make the right one. But I'm just making that point that just because they feel people feel confident. I mean I've been perhaps the worst decisions I've made the ones I was the most confident. But then the thought is to have, you know, like our counselors and physicians we all have hundreds of save notes for different indications, you know, I have a William syndrome note, you know, we have a BRCA one BRCA to know whatever. And one idea might be to have 1025 I don't know how many places around the country where you assign them like like Jean reviews does your you guys are in charge of keeping up to date the information on palbie to or HHT testing or or whatever. And, you know, they're in charge with Jean views you do it every couple years but maybe they do it every couple months. And so that when it all goes into a shared database then people can access it just a thought. Thank you. And then new in. So, there is obviously no right or wrong answer here just plethora of approaches so that there is no substitute to humans. Reviewing the literature reviewing data and putting it together in the form of, you know, evidence questions answered, although you can try to teach machines to do some of that. And recently we've had some experience with chat GPT I tell you they are, I'm, I may be out of business soon, because most of what we did a little bit of an experiment within our office we asked questions, and as somebody to write the answer and then we asked GPT to do the same and there was a lot of discordance I know I heard yesterday that the Bing artificial intelligence is not doing that well, but at least chat chat GPT. It's not a joke, I think, given how quickly the field is moving, trying to have that organization of the and updating it on a routine basis, so that it can become available and a personalized way in different parts of the country going back to the state by state disparities rural urban is so important because the context for influencing and success of this initiative will depend on the local context, much more than on national body like NCCN, or the US preventive services task force which has done recommendations around the RCA. So I think, you know, part of the research experiment here is to try different ways and see which one. What is the best approach moving forward but it could be a combination of things. Thank you. Rizwan. I'm not sure if I'm. Yeah. So, so I think our second, I think all, most of us, most, most people have already raised all the important points as a part of the thing to remember there in my mind is to decide ahead of time what the scope here would be you know what what is going to be the scope of this sort of what is going to be out of scope so that is, that is going to be very important. The other thing is I think for many of the first, for some common stuff. Yes, educational materials can be developed information. And for what the, what are providers fear the most is a genomic report. It's important to have visual areas, but I think I get why I find more clarity whether that's more additional clinical work testing, etc, etc. So I think it's doable, but I think important point to remember here would be we need to eventually figure out what the scope of the service would be. Thank you. Dan. Well I put a comment in the chat I'm no AI expert but it's already been raised that it does seem like recurrent questions, including continuous pooling of updated information from various sources as it has been discussed would be it would be an optimal application of AI but I think that's nothing to original. And I think last is Todd. Any additional stuff to add to it. Okay. Before we move on to the next question any other additional comments on this one. Okay, so then our next question in the section was that we were really trying to look for limitations in the existing sources and possible solutions we've kind of gone over some of them, but are there any specific limitations that you would like to point out and give possible solutions to. Let's start with Jason. I think the biggest limitation is that last mile like okay great I've read the guideline I need to order this test. Where is that in my EMR. How do I order that test who do I call. You know, and that's not a national question there are medicine question that's a logistics in my in my practice setting. So that's a limitation. So what's the solution I don't know that could be a research question. Other limitations are, I don't know that I would have specific examples but where a patient doesn't exactly fit the specific cookie cutter template. So having a system set up that can identify that outlier, and that that that being an outlier is important and actually might change the recommendation. Also, of course, kind of relates that contextualization of the patient's context is really important. You know the presence or absence of living first degree family members might make a difference I guess that could be incorporated. So just stuff that's going on in that patient's life, could make a difference on whether this is actually a question that you really need to address in the big scheme of things. And that kind of those aren't in guidelines. So it's, you know, it's, the limitation is it's not a clinician practicing medicine. Perfect. Leland. Oh, looks like you have something in the chat but I was making a comment to the last point so I'll just leave it I have in the chat. Okay, perfect. So I think I also put a comment in the chat earlier that might be responding to this but I think there's a real gap in algorithms for how to practice genomics in in that in the primary care space and really thinking about integrated algorithms into the electronic health record with clinical decision support and branch team logic because these we have the capacity to build these now. And, and I think that ultimately is going to provide a lot of confidence in the decision making by each individual clinician and allow them to not only make the correct decision but to learn from from this process so that, you know, over time they do become experts at each small section of this practice. I was still mulling over Leland's comment it's such a beautiful comment. I'm sorry. Can you repeat the question. We're trying to look for limitations in the current existing curation systems. What are the limitations do you see that might apply to what we're doing and how could we fix those. Yeah, and so that's where I got stuck in her comment about the making this something that the people providing the care are going to continue to do are going to find, not lonely, and sustainable. Okay, thank you. Cassidy any comments. Yeah, so I agree that this is is too carved in some respects it's not only that technological aspect of it. But to me it's also thinking about pulling that tech space into the who the ordering provider is so does this sit on do these next steps for example, sit on the PCP, but about bringing specialty care into it, and then sure let's say a PCP orders those next steps but is PCP owning follow up from that as well so that seems to be a larger limitation we have guidelines that's great. Someone can follow them but who should be the follower. Perfect. Howard. Yeah, I'm sticking through models that might attack some of this. I think we need to do something different that the model we have is not scalable and not answering the problems that that are really the real problems they're answering the ones we have at the academic centers now that the community. I'm almost wondering whether an uber black like model is going to be needed to, to try to deliver the white glove service in places that have mainly pickup trucks. So that's, I'm not sure I have an answer for that but that that's kind of as I'm distilling down with people are saying we got to, we have to have something that is more agile than our usual approach and right now that usual approaches are great for us so great for them over them is perfect. I'm trying to mull over this question very seriously and I think Leland pull it very elegantly in the chat in the in the chat. One of the main limitations of existing resources like guidelines is that a they're slow to come around and be they're not updated on a frequent basis. I'm always chasing the last guideline and specifically in the RCA I mean this is a specific example here. I mean the task force that something in 2007 and then 17 so anyway the bottom line is that you know the system doesn't work very well for a rapidly moving specially like genomic medicine, and it needs to be supplemented with sort of interim pieces of information and yes artificial intelligence to put it all together. Again, we can experiment with the best way to do this but resources can be put into it. Think NHGRI has done a great job with the efforts like ClinGen and and things like that where similar targeted effort can lead to immediate dividends, if it's evaluated with good research protocols. I'm on mute again. Can you can you rephrase the question or because I'm having trouble with really really kind of wrap my head around it. Right so what we're talking about now is you know if we have a consult service rather than having people consult continually on the same questions. Is there a way that we could create an expert panel or somehow curate that advice so that you know if you answer it for patient X, it'll be available so that patient, you know the provider for Y and Z can just see it, and maybe you have to go through the consult service again. So, you know, it's like when you go to an online, you know shopping site, and you know they're frequently asked questions and I'm sure, you know, especially the younger generation will is more likely to be able to get information from it then you know somebody my age who you know just wants to talk to somebody. So you know creating that kind of database where if it's something's been asked and answered properly, they can just click on it makes a lot of sense. But the thing I keep getting back to is is buying from other people buying from the community. That's not really what you're asking about but I'm just struggling again imagining this, the difficulty we've had at our own, you know, academic institution, maybe is a pretty reasonable academic institution, not to mention the community and I'm just trying to imagine this thing on scale. And I think somebody said this comment really earlier I think the most important thing we should be looking at is PR firm. You know if we're going to try something like this there has to be some kind of way to publicize it and show the benefit to people, but that's kind of an aside. Thank you, Dan. I don't know if this is directly answering the question you just posed the way you phrased it but I've been thinking about this service in terms of providing the upfront, you know, advice around genetic testing so the primary care physician. Should this patient be tested what panel should I use what company might might run that panel. At the end of the day it seems like one of the big blocks is, what's my patient, you know, what's my patient going to have to pay out of pocket are they going to be hit with a bill, the kinds of things that those of us have counselors we the counselors help us figure that out, or the GCAs, but how do we how does this service help the physician in the community, you know get to the point where they, they feel comfortable ordering a panel without placing a big bill on the patient. And I think it seems to me from my perspective that's a big block that I'm not sure I know how to overcome. So are just are you saying that something, you know, we put together genetic advice all the time, is it would it be helpful to put together advice surrounding the non tangibles like payment and what stuff, you know, here's a list of all the providers in the labs and you know if you're with this insurance and these are the ones to pick that kind of thing. Definitely that's exactly what I'm saying something very, very specific around this patient is this insurance or is on Medicaid and here's the company that you should use for this panel and here's what they're going to get charged out of pocket and exactly I don't know if others agree but that's my perspective. So clinical support tools for ordering in ways. Okay. I, I think that's an interesting comment. I think, you know, given the diversity of our healthcare system though, like, I would almost envision the opposite, which would be an office hours with a genetic counselor and genetic counseling assistant who comes in and over is like, Okay, I had like these patients over the past, you know, that are kind of bothering me on my panel or runs a report of their patients and says, Hey, all of these people qualify for BRCA testing, let's do it together and let's place it together. And, you know, this is how you're going to get it back. And this is the support you're going to have and we'll come back in six weeks and help you with that. Because I, the, the logistical barriers without these labs being integrated and the portals and the, I think are just too high for anything, unless they're integrated in that healthcare system. I have two comments. I think the reason I'm struggling with the answer to this is I think my answer is no. So, I think that if the goal was to have the service be some written up to date thing, it could not not be personalized, could not be personalized. There you go. So it would be, if that's the intent, then I would say it's more of an E consult service. So that the person providing the response to the specific case could be templated, because we can keep up our templates. But it couldn't be something that a provider would just say, tell me today about this or about that. So I think that's why I've struggled a little bit with how to answer that question because I think the answer is no, if you're talking about, could you just have an online resource that would be no longer individualized and and have the big lift of keeping it up to date. But the second comment I wanted to make a couple of people have touched on reimbursement and insurance and in my new job, the first and most shocking thing to me was that I have none of that. But once you meet the criteria, the patients covered. And so if you knew that was the answer, would you give the same answer to what our testing criteria. And so I wonder in setting this up as a national program if that shouldn't be exactly how we think of it, because if you give answers based on will it depends on where you live, and will it depends on your insurance maybe we're not giving the big national answer. And maybe the point to this is the pretend that insurance didn't matter which is kind of my life at the VA now. What would you say to this situation to this scenario but it would remain personalized and not generic. Thank you, Todd. Yeah, so I think to, you know, address the reimbursement which I think is a, you know, to address the limitations bullet point one as was mentioned, one of the things that I recently discovered that IU health has is a phone number that any patient can call that phone number is an estimate of what the test is going to cost they will, based on your insurance your model deductible you've already met, and everything so it's going to be the out of pocket cost. My understanding from talking to a revenue cycle is that was required as part of the affordable care act, but I might be not quite I mean that have that exactly right. Maybe some of these resources that a lot of health care institutions may have that a lot of people don't realize is there I mean having struggled with reimbursement for quite a while I only found out about this about two three years or two three months ago when we found the right person revenue cycle and talk to so that might be something somehow that we tap into those resources to actually address that. So I'll just quickly that's not in the guidelines is the liability risk and who's responsible if your name is on the test so it's another limitation. Is there anyone else who would like to make a comment on this type of curation service that we haven't covered is one. Yeah, so I was going to say so I think if you think about it, I think the you can say yes to answer is yes you could do this is that this question, a problem, and, but that does then add some barriers meaning the informatically one way to think about doing this would be that this the question is not asked by a phone but by a web portal the physician says okay, I have this patient. He writes whatever question, whatever test results that they want explanation for, and then that particular website keeps track of that specific question and the answer that was given so that the next time the question is asked, even before the physician presses send. He will he or she will get the answer like okay somebody anonymously you know ask this question about the same gene variant and this was the answer given and see if that so I think that can be done but the barrier there is it's a lot easier to pick up the phone and if it is a phone based service to ask that question as opposed to using Qualtrics or Red Cap or some other sort of database I mean there's some attractiveness to using a database based model. Or a web portal to for this sort of service because then you can actually collect information and you know, you know I'm saying it's it's it's you can do a lot with that data and further fine tune it figure out what questions people are asking further fine tuned the answers. Anyway, I think those are the comments. Thank you. Anyone else want to chime in on this topic. Okay, is that an old hand for that. I think it's an old hand. Okay. Yeah sorry about that I don't know what it's doing there. Okay. In that case, I think it's a good time to move on to the next session that Terry will moderate. All right, thanks everyone. We've been going a good hour and 22 minutes. So thank you all for your attention we just have a little bit longer to go. And now we're going to talk about outcomes. So we had had kind of identified sort of three areas that outcomes might or people groups that outcomes might be important to starting with experts and the clinicians receiving them and and outcomes important to patients. Back to the alphabetical approach because that's the only way I can keep track of everyone. And Cassidy, could I ask you the people who are providing consults what what kinds of things do what kinds of outcomes do you think they would like to have measured and would be important to them. It's a good question. So for those who are providing these I don't think that important metrics would be. I mean I think satisfaction is an easy answer here satisfaction of those. So you're talking about the experts who are providing the consults not the clinicians. What's what's going to keep them doing this. I would think satisfaction feedback from the clinicians who are coming for this, this consult service. But I think that's an easy gimme. It's an obvious one I don't know but we thought it was important to try to address a variety of outcomes. So, and I realize I may have thrown you a bit because you're not really alphabetical but you're Adam Buchanan so you are. Anyway, Rizwan do you have some thoughts on what outcomes might be important to and let's not just say the experts but the people providing the, the, the consoles whoever they may be. You're on mute. Rizwan you're on mute. So the first section would be one the other one would be whether this is actually making an impact. I think finding out that okay the advice you're giving actually so I think that actually feeds into what is important for the, for the, for the person who's getting the advice, are they making an impact in, in either cost reduction, patient outcome, patient satisfaction. I think those are probably the, you know, are the main things that are going to drive it because as a, as a clinical provider, you'd like to know that you are, or as a, or the genomicist you would like to know that what advice you're giving is making an impact. Great, and maybe I'll ask both of you challenge you and Cassidy when you say satisfaction. We, you know, there are songs about that but, but what do you mean by by satisfaction. I think about it. What's with this particular question the satisfaction from clinicians who are approaching this consult service that they continue to come back for further input and so they're satisfied with the information they've received previously, so that they're coming back for additional input as, as they need. Yeah, I think satisfaction means that the physician on the other end found that useful that information. And they've thought like okay I'm getting some thoughtful feedback back, as opposed to oh this is just a sort of templated thing that I got, and I don't find it very useful so I think, you know, we have, we have been doing this for now a year and a half here at Vanderbilt and we found out that is a very important thing and we have to go back and forth get the feedback okay what do you find useful what do you don't find useful. This advice helpful with this advice and we have continued to tweak we call it the genetic, you know, we have two names for this but we have to tweak we had, we've been tweaking it all along every month, figuring out what our customer wants from us. And that is very important because if you're not providing what they want from us, they stop using this. Okay, no that makes that makes perfect sense. But do we want to think about outcomes for the people who are doing this in terms of how easy is it to you know how much time does it take them to do it. Are they answering the same boring question over and over again I don't know but but what do you think Dan you look like you might have something you want to comment. That's interesting Terry I was thinking about that like the job essentially the job satisfaction for the people who are doing this and what determines that. And I do think in addition to feeling like they're actually helping clinicians and ultimately patients, just what what is the nature of the day to day job what's the variety of the, you know, questions they get asked, how much of the routine stuff can be automated so that they can focus on the more you know interesting individual questions. I'm not sure how to how to quantitate all that but I think those are very important to try to capture in some way. Great. No thank you very much. Leland you want to take one one last crack it's it's the people who are providing the consults other things for outcomes. Sorry, I thought it unmuted. So I think focusing on just ways to measure kind of top of license care I think the other thing that you could think about is some kind of, since we've been talking about kind of the community, knowledge and comfort with genetic testing as well. And then I think you could think about maybe kind of some kind of social network both at the patient and provider level in terms of kind of how does, is there any spread of information or misinformation. As a result of accessing these services amongst providers and amongst patients and communities. And I'm going to try to pin you down a little bit on top of care do you do you mean providing care that is, you know, at the, at the highest level of their expertise can you explain what that is. Right, so I think of top of license care is kind of minimizing documentation and or, you know, administrative burden, while also. Yes, it's kind of doing what for whatever clinician role that is and I think that that would vary amongst clinicians, including the genetic counselors and advanced practice providers, what they. I'm not quite sure what the metrics are there but would have to look into it. Okay, no fair enough, and we have two comments now go ahead. People have given some really good comments I'm kind of going to go in a different, slightly different direction is when I talk to people get these calls and myself included. They're, when they're not reimbursed or recognized or included in your time. They are a huge pain in the butt. So, one of the things in terms of satisfaction of the consultants will be making sure it's accounted for in their effort, especially true genetic counselors who, you know, when it comes out of your hide you're in your quote unquote free time it's it's viewed very negatively, where I think if it was included as a set aside amount of time, it would be viewed much more positively. Great, no excellent point thank you now. Caroline. It's extremely basic, it's that I think the biggest position to satisfy or I've ever seen is when there's a mismatch between what we consider successful. So, a good example is if we set something like timeliness as one of the things that we're tracking and following the physician or the provider may know that the patient delayed this appointment, the patient was offered an earlier appointment. At the end of the day what was reported out was it was a month to see that patient, you know, so just having, having it clear to the providers that we're all agreeing on what success looks like. Kind of basic but but important. Yeah, no that's that's super. Okay, when do you want to take one last last crack at experts even though I gave we won the last crack but I'll give you. I think the way the way you've kind of parsed out the the four different buckets for outcomes important to experts clinicians patients and research questions is kind of you get immediately into the weeds before you you're trying to evaluate the impact of the whole thing because you know you're embarking on something new, and therefore the you're going to experiment with ways to evaluate its impact from all of these perspectives so to me the, the experts providing the consoles. There the outcomes could be whether or not there is any different outcomes if they're paid for their service versus not paid. I mean that's a research question, or they're doing it in their spare time, you know Sunday morning Sunday afternoon versus, you know as part of the regular workflow but obviously just designing that the series of outcomes that sort of allow implementation science perspective where you, you look at the different levels, and you approach it from a comparing different approaches and see which which works the most obviously for the patient, the service, he or she would love to see their services taken up and making an impact to the patient, but from the, the whole service perspective, the ultimate outcome is a societal one, do we need that service, I mean has, has this service. Saved lives or made better diagnosis or led to better treatment than without the service. So to me a holistic approach to the, the impact is, you know much better than focusing on only the components. So maybe you were going to get there but just couldn't help but Well, and I wonder, I mean, if you might want to comment the CDC a different part of it I believe does provide consultation on infectious diseases, for example. So, so, you know with with not Ebola I think they did it for Ebola as well but with coven. You know there was a hotline that you could call or contact and that and so so what were the metrics that assess that or was it just something that hey this is the CDC we have to do this and I don't know the answer to that I mean CDC during the pandemic kind of all hands were on deck and obviously in a fast moving pandemic situation the evaluation metrics tend to be a little bit different than something you do more carefully. So in this case, I think the infectious disease consults may not be the right comparison to something like this but you have the luxury of designing something at the outset, and evaluating it but I can check to see if the whatever CDC that during the pandemic made a difference but for my perspective it doesn't matter CDC gets beaten up, no matter what, either way I mean whether or not we did the right thing. But I think you do have experience or the part of the CDC has experience with with what measures, you know they've tried and have been you know whatever and so, so maybe if you could put us in contact with the folks. That would be because I'm unaware and maybe others of you are aware of anything quite like what we're thinking of doing other other than you know this the CDC model. So does anyone know if you know as a practicing clinician I can't think of a place that I can just sort of call up and say hey, tell me what to do with you know a GI bleed or you know whatever it might be. So Leland just put in the genomics telemedicine consult. I don't think they ever published their like findings about the regional versus. Did they ever publish the results about like what the impact of the centralized telemedicine at the VA. So, so I might I'll call on Jason but I might also put Renee on notice since Renee was part of that whole system before she joined us that will call on you as well but Jason please go ahead. She and her has looked at some of this and compare and comparing kind of more traditional genetics consultation from from nearby services. And it's a mixed bag you know she found you know of course you, you can get access out to rural veterans through through a hub and spoke model or through a centralized model. She also found concerningly some racial netness and ethnic disparities that also resulted, and that actually having local services seem to ameliorate that so I think it's been looked at it. I think I think like anything, it's going to take a mix of different kinds of interventions and programs to really reach everybody. And the result of those is the program that just opened in September, the clinical cancer genetic service so the genomic medical services in general genomics service. Clinical cancer genetics is in response to some of those disparities that were identified in cancer patients specifically. Renee, do you want to share some of your experience. Well I think it's important to note that that was a strict referral service, like there was very little E consult we did try to do an expert E consult for a little while, and our expert ghosted us, because he just didn't have the time, you know so I think that the expert satisfaction is really important making sure that this is a reimbursed part of their job that they're getting credit for otherwise it's it's not high on their priority list. I also think it's really important that when we're comparing these types of services, we're comparing equally resource services that you know I think a lot of the comparisons of the telehealth services to the traditional genetic services is a really great idea. But when you have a national telehealth service with seven counselors going across the nation, can you really compare that to a local service with three genetic counselors going to three hospitals. So I just, I think we really need to keep a perspective when we're doing evaluations. Super. Maybe we can. We've touched a little bit on this but there may be more thoughts on outcomes that are important to to the clinicians receiving the consults so I've, I think I stopped it moving so Melinda do you want to comment on on what might be important to the receivers of this information. Yeah, I think the timeliness of the consult so the turnaround time for the consult and how quickly they get access to the information would be critical. I think that clinicians requesting the consult will also care about the patient outcomes right in the impact on the patient satisfaction, which directly then reflects on on the satisfaction with their own provider. So I think those two things are critical and I think the confidence as has been mentioned many times I think the confidence in the information they're receiving and the ability to take that and implement it will be important to them. Right, thank you. Howard you want to add to that. I do think as much as I hate to say this out loud. I do think that the clinician and patient satisfaction piece is really important, not as a metric that we tout about but if it reflects that they feel their perception is they were helped. And it's been surprising to me sometimes I have said I don't know, and that's been incredibly helpful to people. And there's sometimes when I've had an answer that was too fancy for them and, and they didn't think I was helpful at all so, and everything in between. So, I think that is a really important part part if we're going, if our ultimate goal is for genomics to be normal, and just part of medicine. We have to be going in using those metrics to help guidance. Good point. And Linda did put in the chat the volume for the service will be NASA. I mean we would hope. There's something like this together and nobody comes but, but it can't be be smaller part time and I would agree, as it as it grows, but probably do me need to make it larger and larger depending on the demand. So thanks for that point. Let's see Howard so Caroline did you have other things that you wanted to comment on the clinicians receiving this information. I guess I just have more to say about the VA so I'll wait until that's more relevant, but the big picture is, we are doing more and more e consults because that's how you scale. And the other thing that we're doing more and more of because of scalability is point of care. So job one was getting getting education to the providers and getting the nitty gritty nuts and bolts of how do I order this test that that's the end of the day is what button do I click. Getting that information out so that this can be scalable because of the volume. And the VA now has a single medical record system. No. How do you tell them what button to push. Yeah, it's there. There are a lot of talks going on right now about this there are two ways to write the lab order in your local system and there is one way to write the order for the portal that'll be covered by the national precision oncology program. So, so we've got it all narrowed down to if it's going to be paid by N pop. Here's how you can order it directly on the portal. Here's how you document it and then I came in a little heavy handed and said but you're not clicking the order unless you can click several of the informed consent points. So you you get to the order through a note. And the note has key informed consent parts to it. And then based on the result a large percent of them will be seen by e consults or the V us is, and the high risk negatives will come in as a consult. Otherwise we can't possibly scale it nationwide. Dan, you have a comment. Yeah, I think with regard to importance to the clinicians. One big issue is time. So is engaging with this service actually taking more time away from them or is it saving them time. And again, hard to quantitate but I think a really important issue at the end of the day, they're going to use it. If they perceive that it's going to save them not in addition to giving them important information that they don't have. If it's going to save them time and they're not going to use it if it's just going to add to their burden of time. Excellent point. Great. Right. Anything else about outcomes or feedback from clinicians who are using the service that we want to be aware of. Yeah, just just to pick up on Dan I agree. I agree time is important but if they're also persuaded that patient care was really improved as a result they'll take the time. So if so something somehow being able to demonstrate to them that actually you've delivered more guideline concordant care as a result of the information you got the your change in medical decision making or the action ability of the information you got actually improved things patient patient care or patient health outcomes, then they'll it'll be it won't, they won't invest days to make that happen but they'll invest some extra time if they're persuaded that that really was a result. So a method you could get at that would be, you know, asking in their opinion did they think patient care improved as a result. It's pretty pretty subjective, but short of short of obtaining medical records from the patient there might not be a lot many ways to assess that outcome. And Leland put in the, in the chat repeat visits to the services a proxy for satisfaction that's an excellent measure, assuming that they're continuing to see patients who have these things, which I think we can, we can fairly assume for some things maybe not for for others. Another thing that might think it just back one question about what other services might be a similar thing. I think I can think it was maybe the poisoning and toxicology hotline, you know, and I'm not a clinician I never called us I don't know exactly how it works or whatever but that might be something that might be the close. That's a great idea. I've called the poison hotline many many times, and they are incredibly helpful. So and I do wonder how they are, you know, staffed and paid for so that's that's an excellent idea thank you so much we'll look into that. I would add to just to wrap together a few things that were said about the clinicians, not only looking at their confidence, but the impact of whether or not they feel more empowered, but then the market ability market ability aspect of this as well are clinicians who utilizing who utilize the service, talking to other clinicians who then pick up on this service as well. And do we have frequent flyers who continue to come back to it because they find a lot of value in it. And they're not learning. I mean, another reason to come back right, but hopefully not. So, cool. All right, and then in our last few minutes maybe, you know, we're encouraged to think about outcomes that are important to patients, recognizing that this may not a service may not be a service that deals directly with patients are there things that patients would want to know that a service is providing obviously better care would be would be a pretty clear one. But I'm wondering, where did I leave off. I think with, with that. So, Todd, do you have have thoughts in terms of what kinds of things we'd want for patients. So, only just from some of the interviews and stuff we've done with them return and formula checks results. They just want to know what to do. As much as, you know, as we may think they may be interested in genetics, they, the majority of them aren't. And they just want to know, you know, what do I do and what are my options. Other than that, you know, I think the obvious things like you mentioned. And this is the patients not the clinicians right. Yeah, it's the, it's the definitely the patient so we've done, you know, probably now 50 or 75 interviews of patients that we've returned farm good genetics results to, you know, and it's the thing. It's just typically just, I just wanted to know what to do. Right. I don't care about all the other things. I don't want to learn Mendelian genetics just tell me what to do. Great. All right, Melinda, you had a comment. I think as part of the equity discussion. There are a lot of patients who want to know who is learning this information about them. Genetics is a very private detail and very personal and and we see that with referrals right when we call the referral to schedule them they're very startled about someone else knowing about this about them. How do we, how do we know it? Why do we have this information? And so I think that's actually going to be important to patients to know something about this service. Great. That's a good point. Carolyn put in the final product with the family sharing letter many patients really want that above even personal steps. Do you want to comment on that Carolyn. What do patients consider a successful visit virtually every genetics visit and again I do cancer but virtually every genetics visit they they won't let you even try to stop until you've talked about implications to their family. Really. Okay. Oh yeah. Good point. Yeah, good, good, good. Great. Other things other thoughts of what might be important to patients. I think downstream healthcare utilization and costs that they face. If they experienced those as a result of any referrals made. And then I think you could also think about the impact beyond just family members on the general community, you could consider doing some kind of social network analysis or asking about whether you know made any changes based or shared these results with other community members. Great idea. Super. All right Howard. A lot of genomics currently set up as being a kind of a one off or just a few visits and many patients now, especially on the cancer side are are now at risk for multiple cancers and they have to see a breast surgeon and a gastroenterologist. The neurologist the male or gynecologist at the female. You know, we've set up a clinic where we can go one place and be guided by all those expertise, and they can be managed and almost more of a genomics medical home type of mindset, and I don't see that very many places and I that's really missing the people that fall through the cracks it's like visit three or visit, visit four, visit one and visit two or nail this the rest of it that falls apart so some I'm not sure how to fix that but if we are setting something up, maybe having a mechanism or referral a triage whatever, where they can be managed perfectly virtually over the next years, as opposed to just one. Excellent point thanks. And any, any other ways to be made here. I think I would say, I was just going to second what Leland was saying and I was thinking about it and this is based on our personal experience to, you know, in clinic is that did did the information that patient get lead them down to a rabbit hole, meaning lead lead led them to additional pain, meaning more frustration I think that will have to be captured. Most patients are not like that but I think a segment of patients. We would want to capture, you know, what they thought about. It's now suddenly finding out that they have to see 10 more specialist. All right, Dan with regard to patient outcomes, I think another would be impact on family members so so for you know dominant conditions you know what you know what what was communicated to family how many other family members were diagnosed as a result of you haven't been diagnosed would be really great to try to collect that information. Yeah, good point. All right. I think we're pretty much at a close to to these kinds of things so we're at the next steps. Do Renee you want me to start with that you want to start with it. You can start. Great, well we will consider, consider all of this we may come back to a few of you with some some thoughts, and probably will draft something that is more, you know, basically a summary or not a transcript but a summary of what we heard. As you're probably aware when, when we, we start to think about what initiatives or concepts we might move forward with. We tend to kind of go into a bit of a quiet mode on that so you may not hear much about this after, after this, this meeting but we will put this recording we did record this and you all said that was okay. So we'll put this recording on the website for the genomic medicine 14 meeting primarily so that those who weren't able to participate, get the same information that, that you all have and that we have so Jason you raised your hand. Yeah, sorry I was waiting for the bullet point about kind of other outcomes. But, but, you know, going back to our sustainability question those kind of research questions can be built in from the beginning. So stakeholder engagement from the ground floor will help identify who's going to. So if we're talking about what NHG or I can do you know fund a research project. So NHG or I is going to want to know who's going to take this over when the five years is done. And those kind of questions and the data to inform the answers those questions should be collected. So that's how much does this cost how many FTE do we need to hire who will take this over and what will the payment model for that be moving forward all that has to be collected from the beginning. Thank you for reminding me of that I'm going to take since we are only at 1252 Eastern other other research questions that we should be addressing in this model. You know, one of the things that that that can help keep people engaged, if they come up with additional research questions as an example so I sit in on our molecular tumor board and, you know, look at drug drug interaction stuff I don't get effort covered for but the reason that keeps me coming back is because you're looking for extreme toxicities extreme responders. There's additional questions that we've now brought into the research realm so I think that's, you know, the opportunity for that is something that can actually keep the people who are, you know, I guess providing the consults engaged. Great, great point. How are you have your your hand up it's probably an old hand but I'm going to ask you anyway, what what thoughts you might have on research questions around this. The question is, how do I get the hand to drop. I think the, the thing that's merging in genetics and in this area is just the pleotropic nature of the results and you know I hinted at it with the need for a bracket to patient to be followed up by a bunch of different specialists. I think that's going to be true. The same is already to a pharmacogenomics. Same is going to be true with some of the project risk scores because you're, if you order one you're really going to get 20 or however many that company does it because it's the same platform so research on how does one handle lots of information when you just wanted a little is kind of going in because we're going to we do now we get stuff we didn't ask for. You might be looking for something and you find a leap from me well you're glad you found it but that's not what you're looking for you know. So, I'm not sure what the right question is but that that's an area that needs to be tackled more systematically right now it's just kind of everybody's winging it. Great point. Yeah. Super and I see Jason suggesting impact on health equity will also be a critical outcome I would agree. I believe we had from Linda how does this fit into a larger phased approach to fully integrating genomics into primary care. Yep. Thank you that's a good one. Other research questions. And this may be one you want to give a little thought to maybe maybe once you see the the summary and that we're, you know where to find us. We're always available and interested in what kinds of research gaps there are and how we can contribute to filling them. And I think with that, we might be able to give you a few minutes back Renee did you want to make some closing comments. No, I mean I just wanted to thank all of you for spending the past two hours with us this has been incredibly helpful and we thank you for your time and your expertise. Great. So, thanks a lot. Really appreciate it. Thank you all. Bye everybody.