 1250 and I believe we have a we should have plenty of time for questions But I want to start by putting Mark on the spot because I know he can handle the pressure and Mark would you restate the question you asked so that? Kristen and Howard can have a chance to to respond to it because I know you wanted the Insights of all three of the speakers sure, so the question is how do we engage with the Intended audience of end users to better improve the products that we're creating to serve them and So specifically the examples would be things like the clean gen resource the ignite page those sorts of things We're just not engaging with the end users in a substantive way to be able to make sure that what we're delivering is what they want Okay, well one one of the things so we've done that a little bit locally and pointed out sources and One of the things we found is that if if the if the resource meets an unmet need Then they're they're all over it and they'll use it and go for it If it doesn't meet an unmet need they might use it once because it's kind of neat and then we'll never go back again So I think part of it is making sure that as many of the unmet needs as possible are included there and and you'll try I mean, you know logical stuff, but That's barely been the because when it comes down to a lot of the practitioners Both an oncology and then we do some work in salt or transplant in the psychiatry area All those different areas they only have a fixed amount of time for any kind of education Some less some more and and just don't want to know about anything unless they can use it You know, so I think part of it's just making things more practical and more more obviously practical Because that's then they'll find it But in some ways that's total logic Because if you can't get them engaged on the front end Then it's difficult to get that information back, you know, because we have good evidence that you know Clean gen and I think it was a subject of an entire NHGRI meeting was a clean gen was a really Necessary resource and that everybody endorsed the fact that this is meeting an unmet need But I think what we're seeing is is that while we're trying to make it practical and be able to deliver information At the point of care when a clinician has a question so they can answer it quickly Actually getting to the point of having them even use it that one time and provide feedback is is the difficult That's the difficulty we're having. How do we reach them? Turned it off now, isn't this more like Apple in the iPod? I mean it's convincing somebody they need to buy something They don't know they need yet So if we could figure out how to do that approach that yeah that model would work But the problem has been we we haven't we haven't had tools that answered the broad burning questions to the point Where everyone's got to have one Even though they didn't know what it was five minutes ago They just got to have one and that and that's been been part of it We also don't have the marketing budget that Apple did in order to drive drive people to want to have that I mean, I do wonder I don't know if Bob still here, but I don't have we ever tried the like a white label approach where a Society can stick their name on it powered by ISCC Type of thing because that that might be one way of getting getting past the mass mailing issue We have not tried that And I would add I guess two things to Howard One I think is really important is that I think we have to keep in mind that not the information doesn't just have to be there It has to be easily accessible and usable and so I think looking at I know I don't see Donna in here But I think Donna did usability testing for g2c2 and did it in conjunction with another conference that you guys had last Last summer and so I think next year with the precision medicine conference We'd like to do usability testing with the ignite toolbox and really say does this is doing what we think that is doing So we I think we have to ask that question. I think the other piece is incorporating into training so Bob mentioned the trig program and rich Haspel's really led It's really an amazing program from the content and the approach to learning But you know it's really a part of what they do as part of the training and so it's got to be incorporated into kind of that Applications based this is how you use it. They really have to it's almost like it has to be part of the training to train They use the tools so I'm gonna Push back because we know all the yeah really So the issue is not how to do it. I think we know how to do it If I'm trying to sell my vacuum cleaner, I can't even get my foot in the door To the people that are using it so so I could we've done them the meeting thing with usability But we've done it at genetics meetings and and how can we get to all of the other meetings? You know to try and get in front of people So I guess what I'm trying to say is is that it and the reason I was picking on Bob Was because we have a convened group of People across multiple specialties that are engaged in this area and so It's it seems to me that that's a group that could serve this function But we've not been able to really get that across the the finish line You mentioned a convene group that are engaged in this area And maybe that's the key is you really need a convene group of people who aren't engaged in this area to kind of see What is it? What's the barrier and what's keeping? You know if it's if it's not being used what what and why that is So my comment is related to that so the traditional academic approach to is to build a solution And then find a problem to solve with it. And so if you're more in the commercial space you would Say I think I know what problem I need to solve But then an entertain a marketing group to help you know that does focus groups and really tries to get under the hood of what's going to really get Traction if you actually develop the product what solution is it trying what a problem is it trying to solve and how would it be? Adopted so I don't know that I think you're you're sort of going in that direction where you would say Let's really get the user community Which is what you were saying earlier to to help us understand what their needs are and then modify or build those solutions Yeah, I mean I was gonna sort of say something similar I so maybe mark you can clarify who end-user is I mean is that the clinician physician whatever at the point of Care because I'm not sure that they have they feel that they have an unmet need Right, and I don't know that it's realistic that they're gonna pop on to Clinjan or ClinVar or fancy KB. I mean we you know in our CDS alerts We'd link them out to evidence and the last time I looked no one had ever clicked on it Right, so I think I don't I mean I completely agree with Jeff I'm not sure that I'm not sure they feel they have a need and they're not gonna go do something extra in their Been a busy clinical time To do something that fills a need that they don't even recognize that they have yeah, and that may be the issue Now again, you know, I'll just you know push back in terms of Jeff is that you know We did we had pretty good signal that you know that there was a need for ClinGen from a pretty broad Group of stakeholders, and so it was it was compelling enough that you know There was a funding announcement that was issued and an act that Anna has now been refunded so but And I think the point that that Julia's raising was a very good one is that even in the implemented things using Resources within the electronic health record We know that if you present a best practice alert that has a link to additional information that at most 10% Will actually say well, why did I get this alert the rest of them will either follow or not follow the alert? So, you know, it's it's it's that 10% where you might be able to get some substantive feedback about how to make it fit within workflow and that that we're really trying to To get at and it's it's a challenging it's a challenging group and and again I I don't want to have this dominate the whole hour of discussion because I think we'll retread a lot of ground that we've spent the last Four years talking about and my question was really a very simple one to the ISCC as opposed to but if somebody has a brilliant solution we can see the CMIOs and the patient safety folks have been some of our biggest friends at not just at Moffat, but at the other health systems we've worked with because For two reasons one they recognize the systemic nature of the problem and so it's a it's a bigger one to them Secondly, they have the tactics to achieve What we just talked about And and so it's you know, it's been a great group who'll go in there on on behalf of the physician community implement something and you know with some you know champions and stuff like that, but so I think they're you know That might be the group we need to engage with more And I think to add to that to you know My experience is that they might not necessarily click on whatever that resources within the electronic health record But they're they're very eager to use kind of the the one-page summary that they hang on their wall Or that they can put in their pocket to just tell them what is it that I do in this particular scenario And so it may be the question. I know recently it is I Thought it was a really good opportunity. So when the last ISCC meeting Bob had People go through and just present what their educational programs were and it stuck out to me I thought so gecko it which is the genetics group in Canada the way that they approach this is they have kind of three levels of Education for each Topic and I'm probably saying it's completely wrong But this is what stuck out in my head one is a one page one's like a three page and one's a lots of pages And so maybe the the the missing or the key is is looking at what is it on Clinton? And then how do we put that in a format that's usable in a one page or half a page that really just answers the one question that that clinician has as an end user And that's the approach. I'll take it my organization because I can do that but There's not somebody like me fortunately for the rest of the institutions in the United States and elsewhere That's that's doing that and so in some ways We're trying to walk that Balance of saying We anticipate that there's going to be a broader need We won't have the people on the ground to synthesize that information in most places So how can we compromise so that somebody can get an answer like because people use things like Up-to-date and and we have indeed to answer clinical questions And we know that we have a 30-second window to get them to an answer And so we're we're building off of Techniques that we know can work and the physicians do utilize when they're confronted with clinical questions But We just haven't been able to get you know the confirmation that the way we're presenting it is really The best way to do it or if we could do it better based on their and yeah And I think I think that's where the sort of partnering role for ISEC and bringing these different Expertises together so there are dissemination platforms out there medscape up-to-date things like that that we have not a systematically Looked looked at that are already out there. We don't have to build anything We have the resources we need to understand the needs of the end users and match it up with the resources and figure out How to disseminate them so you know in term and you sometimes you need a hook and so for did for this conference the hook for me What would have been a boy? You know and given a choice between two agents with equal efficacy you can have one that has a 15% Major adverse event or you can have another with a 5% major adverse event Which one do you want to choose or which path do you want to take? That's a pretty good hook So so that's that's that second thing. I want to say is is about ISEC ISEC is is participant driven I mean we provide Some administrative support limited we can't do all of the work And it really we're trying to shift it away from it being an NHG or I heavy in terms of focus So that we can encourage people to participate more and to build these things outside of you know my heavy hand on the calls and on the meetings so That's really really important is to stimulate that awareness that This is an organization that will be driven by the participants and if the participants aren't doing anything It's not going to do anything. So that's really important while I have the microphone I want to follow up on something that Howard said about Including the other professions and nurses for example, and this is another plug So Donna Messersmith and Kathy Calzone Donna Messersmith from my office and Kathy Calzone from NCI have a Summer short course. It's this year. It's four days long for nurses nurse practitioners and physician assistants and their educators And that's on the NIH campus campus the applications for participation are is still open I think through the rest of this week you can contact me or Donna Messersmith and or look on the genome gov for Short course nurses and you'll find the course. There's a link there to the application Website and I encourage you if you have people who are interested in doing that by all means do that We are also aware that one of the barriers that hasn't been talked about much to Engaging providers in terms of participating or learning how to do this in their own Things is not the knowledge or or the the the willingness to do it But not knowing actually how to put it into their workflow So if they work with an NP or a PA or something who what's the best way to sort of work This into the workflow of their particular office that works for them And there may need to be some discussion and education around that to help them become comfortable with Implementing the information that we're trying to teach them And just a just a quick follow-up to the web's web MD kind of up-to-date having worked as a writer and editor at one of those Publications previously, you know, we were kind of limited to maximum of 28 lines So it's kind of like if you we had to get like how to treat hypertension in 28 lines And so maybe that's part of it is saying we have you guys have 28 lines You know and making it kind of makes you pull out the meat of what people need to know You know following up on both what mark was saying what Bob was saying in in clinical IT when we have an Intervention that we want to do and we want to Release IT that alters a workflow We find that the best way to ensure that that gets adopted is to make sure that it will save the clinician time and I think that so one of the things that may be necessary here is to think about how Could expand the scope a little bit beyond just pharmacogenomics, but but including pharmacogenomics to providing support around a Particular type of decision that a clinician needs to make and provide enough support so that for that Transaction for that encounter you can actually save them time and that may help with adoption Yes Think it goes along with this same theme We've studied adoption a lot at our in our 1200 patient project And we went in with a hypothesis that it was going to be around this idea that people who got prior education in genomics We're going to be the earliest adopters younger Providers and it turned out not to be the case at all and the biggest factor the biggest driver was how much how many Patients the doc was seeing that day about whether they looked at the alerts or looked at the clinical decision support So I think it really echoes what a lot of people are saying is if it's in the workflow And if it can be done in a short amount of time we use the 30-second idea then usually they all adopt so I have a comment or a Question from Lynn and who had to leave earlier and that's that in use case development and She was pleading for These sites that have use cases posted that there be you know real-life cases where pharmacogenomics Made a big difference in a patient's like based on real-life Information and evidence and she was looking at some of the sites and and was finding kind of these theoretically based use cases but not you know real-life examples and so she was Compelling the groups that are developing these educational materials I think to keep that in mind because she felt that that these would be particularly compelling types of cases to include and I actually Howard to your when you said about the molecular tumor board about you know the jack one amplification isn't that cool so This is Jack to sorry Jack to write that's even cooler It goes to the kind of the opposite end of the spectrum of what Lynn was talking about which is You know I've been on molecular tumor boards is the basic science Representative where we have a patient who has nothing in their tumor genome. That's actionable But they have a really cool amplification. You know, so there's nothing actionable about it Right, but it is it is an observation that comes out of the testing that we did So, you know how to communicate that whether to communicate that at all So the first thing we say is there's nothing actionable in this patient's tumor based on these genes that were sequenced But there are some interesting molecular pathways that look like they might be dysregulated But so how do you present those kinds of cases where there isn't anything actionable? But there's interesting biology there if there's interesting biology often Often the group may come to the conclusion that it is actionable because there are The option is to Put people put someone on a therapy where there's not even a hint that it might work And so that's what we've done in the past put people on a trial because the main criteria was the trial was opened at our place at a place and the patient fit the fit the you know the organ function criteria or whatever Now at least there's some hint that that pathway is just disrupted in some way You know can they go on to that preferably on a clinical trial type format? And so it's not but we've had a number of situations several that have resulted in in our one grants where we have this accumulation of abnormalities that were not druggable That we put to the the more basic folks. They took it into the lab generated some data and They they got funded at the NIH for research purposes Partly because they were using real data and trying to solve a real problem as opposed to doing just disruption of every base and you know Try to find a problem that things so it's not but we we've had a number of situations There's there your research in clinical should not have a divide There should be a continuum, but often academic centers are not structured in that way They're structured your research or your clinical and you know don't even meet in the cafeteria Jeff So I just want to echo Lynn's request because I think that idea of telling genome stories is a is a really Important one when you know whenever I see presentations from the undiagnosed disease network It's incredibly impressive in it and it hits the patient provider It hits the family. It just is a very compelling reason to keep going and doing this kind of work so if we can tell the same stories like Angela's story and You know there's so there's both successes and failures I would say that we could tell those stories that would make it real for the average clinician and patient that doesn't understand What pharmacogenetics really means so and I think maybe we put that as part of the ignite tool Box or or someplace, but we have to put it somewhere co coincidentally Anastasia wise and John Mulvihill are presenting UDN to ISCC as we speak Well, I think there's an opportunity to with the case reports that you know while we've tended to make them sort of Static and durable materials. There would be no reason that those case reports couldn't be presented in a in a format That like the Grand Rounds or something and then have the actual case report Available as well. I think that could potentially increase the utility of invisibility of them and you not to put more work on the Institute, but You're having a webcast You know cases in genomic medicine or something like that Might be a way to get that out the you often unless you know a journal that will take case reports You don't tend to write them up because it's too much hassle this would be a venue where you'd get credit for it and From a from an oncology standpoint the American Society of Clinical Oncology has just opened up a new journal the precision oncology JCO precision oncology and Part of that is that they and a welcome case reports and the reason why is that the more case reports that are published The easier it is for an oncologist to get the drug paid for by an insurance company So there's you know added value to case reports beyond just a way forward But actually a way forward that might actually get the drug paid for So Bruce Corf is the incoming editor of the Journal of the American American No, the Americans. Oh Yeah, yeah, so anyway, one of those journals, but but I think he's But but I I think you want to start over Jeff No, but I think he he's a he's obviously a very good friend of this community and I imagine he would be Interested in least entertaining that this is a venue Is this are the what kind of what we're thinking of mark is this similar to the genome case conference series that you host out of geyser I think it's the genome first conference series as far as what we're talking about just from an educational perspective Even though it's not looking at pharmacogenomics specifically Well, I think it's just a I have a little bit of an affinity for the case reports in ISCC just because I Was involved early on with that sort of the developing of the format and the templating in that and was hoped that Again, this would be something would go out where the societies would take ownership and create cases that were relevant to them So the model could even be and Bob can tell me if you know how many of the societies have really taken this forward But out even outside the realm of PGX you could imagine that if societies have venues by which they can present case reports that if we could assist them with some of the You know relevant information for their society whether it be around pharmacogenomics or other type of genomic interventions that using a multi Media type approach with a formal case presentation the durable case report, etc. Might be of some value Yeah, the case reports have have The case studies working group created the template and those first two or three Cases and it has not been taken up widely by the others We haven't had the bandwidth to Lobby And really I think it takes sort of going to each one and saying hey, did you know this is there? Pick out the things that are most relevant to you and we'll help you write the case study where we'll get other people Geneticists to help you write that yeah yeah, I think Figuring out how to incentivize people to write those case reports and to write The the lessons learned type papers so to the the point you're making earlier I think that Lynn made about the the real use cases that people have experienced I mean, I'm often involved in conversations about should we write a paper to tell everybody like we we tried these four things and like These three failed miserably, but this one kind of worked but but in the the list of all of the things that people have to do Your data papers and your discovery papers and like the things that will go to a higher-impact journal tend to get the attention first but for the community some of those Mid-tier paper lessons learned here are the things we tried that failed and here's what kind of worked is Really meaningful for moving the field forward so thinking about how we can kind of push ourselves and our peers to Put more of those out and to realize that even though they're not going to get in to New England Journal and JAMA and Nature genetics and science They're still really important for the others in the field to know what to try what not to try or even if it failed at one institution It might work at another so another issue The question that was raised in my mind Kristen during your presentation was the certificate program So you have a curriculum developed around this certificate program. So How common are these certificate programs? How do you develop a curriculum and keep it up to date and was that that the certificate itself? Was that a demand by the end users who wanted that as something they could take away or was how was that decided? so so there are two things that we have that and I have the word certificate in them one is a Certificate certificate program. That's a 25-hour program for practicing pharmacist And it's part of the continued education model that you can award a certificate based on certain requirements And there has been a need within express me within the pharmacy profession for more education and certification Within pharmacogenomics because it is difficult for many of those frontline pharmacists So that was in response to a need and it is associated with Something that they can show, you know evidence of that. We also have the graduate certificate Which is an academic program that's nine credit hours and we're in the process right now of developing that curriculum and we really have been doing that by you think it's kind of starting with with a template and then Just sitting down and talking through with our oncologist with our different groups to kind of get input on what that looks like and It's actually we're in the middle of doing that right now It's been incredibly valuable process, but it is kind of saying what do you think that that people need to know in This area from a lot of different disciplines. I don't think that there's another program like that out there So but we're excited about that one Yeah, so Also in Florida Jeff Vance at University of Miami and something in the water He has a master's program which runs concurrently with medical school and Is a number of people who have actually come to that medical school enrolled in that medical school because of that program So it's actually a draw to the medical school In addition the the UK National Health Service the education program They had what did I say 20 million pounds or something for three years to do this provider education around genomics And one of their implementations was to create programs sort of a step-wise a program where you can step off at any point You can get a certificate or you can go on and get a master's degree that are taught at academic centers And the providers actually get time off to to actually attend those and get those certificates or degrees So there are other examples around the world for this But as far as I know everything else is in Florida Yeah, so just to clarify so the graduate certificate is you know essentially a working professional program. It's online So at UF we have a robust Working professional educational program we graduated on Friday 75 master's students in working professional master's degree programs The vast majority of those are in forensics. So So the idea is that this may evolve into a master's degree program if you know if we see the demand But that it's really tailored around something that people can do sort of from home, you know Do graduate level coursework, but from home, you know in the context of their position to advance their knowledge and skills Well, that was another another question about not having to travel So how many of these things are actually some of them are offered passively is after the fact videos and stuff are posted But how many are actually offered is sort of interactive online? Courses that people could take without having to travel to Florida unless it was a winter As far as I'm aware there aren't any online interactive programs kind of in synchronous But we are I had talked to Lenk said as she had that question We did record the conference and we are making all that content available online As much as we're able and that's kind of probably them the most that's out there right now Once we we get that as far at least as far as pharmacy is concerned Yeah, but right exactly that the graduate certificate itself like Julie described would be interactive and online And so it would be a fully online program and and actually, you know It'd be possible for someone to just take one course out of that So it wouldn't necessarily you'd have to take the whole you know the whole thing because that would meet that need and One of that one of the enhancements to to these courses even if they're sort of in person at a particular site That we think will be helpful and this has been piloted at City of Hope is to build a community of practice around The group of participants and and have that grow every time a new group comes comes through and that's primarily an online Group and different people do a different way just through a simple lifts lists serve or other methods But that has proven to to be a way to keep people engaged Over time and in continuing to keep up with what's happening Yes, I just want to mention the University of Colorado Denver does have an interactive Pharmacogenomics certificate program that they recently launched a couple years ago So in that is that a multi? Is that a multi day? Is that a sink? How does that? I don't have a ton of details on it I think it's it's it's an online For about a month. I think it runs a couple hours a week sort of a thing But it's it's synchronized you there's modules that you go through along the way and and our company is actually providing participatory Genotyping for that program so One of the things we found with our pharmacist after we had the belt doing the pharmacogenetic tests and after the training program Is they came back to us and they actually said they wanted more interaction. They wanted discussion boards They wanted to share some of their experience They want to have an opportunity to discuss some of their cases that they had so we Implemented a monthly webinar as well as part of this Yeah, and that that was another question I had about sort of follow-up. So there's you know Participants take these courses and there's usually some sort of immediate follow-up follow-up about the impact of the course But what's the longer term follow-up and engagement opportunity for these people? Because I think very much the same same thing is that people people want to continue the discussion In an interactive way and that's sometimes very Challenging to support in these kinds of approaches, but I think very important for long-term traction They want the feedback as well as to whether or not they handle a certain case Appropriately and you know what worked for them or what did not work for them and they want the discussion Yeah, and we had our participants requested that at the conference to create a group that they could continue to interact with each other So we we would have done it anyway, but they they really asked for that Mark I think one of the examples we could look to is the city of hope cancer genetics course They've done this. I don't know how many years, but it's well over ten years originally funded by grant but then has now become self-sustaining and Not only does it provide intensive? Hands-on activities both in-person and remote for the course attendees Specific to cancer genomics and cancer genomics practice, but it also provides a sustained Forum by which people can bring cases consult with other course attendees and graduates and they have an annual symposium That many of the attendees come in for refreshers and updates and so it's it's ended up being a very Robust model and if we could replicate something like that in the pharmacogenomics space. I think it'd be very interesting So in in the last five minutes here I'm wondering if each of the three speakers would Even if it's restating stuff you said in your presentation sort of sum up in a minute or so what you think are the major Educational gaps as they relate to promoting Implementation what where do you think the effort needs to go in the near-term future? for educational activities to really impact Implementation and let's just go in order. So Kristin you want to start and then Bob and then Howard First I just want to echo what mark said the city of hope That's an excellent model and one that I've really looked at in developing our programs to you But the one thing that I would take kind of my biggest soapbox passion is implementation And so we base so much of what we do within our educational programs on cases in patient cases that we you know Have that we've seen within our clinical programs And so I think that is essential is that it has it's kind of the chicken before the egg But I think that having your implementers involved your front-line clinicians involved in the educational process is so important Because that's where you're going to get that the things that people need to know that really practical piece So to me that has to be a part of it Thank you Bob. Yeah, so I think that starting at that at that target learner end and really understanding You know what is driving them to learn and what kind of information that is actionable in their clinic now and how? understanding how to do that on Across multiple specialties and on scale over time because this is you know pharmacogenomics It's just one piece of what's coming down the pipeline and we really need to understand how to do this At a large scale in the future, and so I think this is a really good good test case for that so getting that participation as well as Focusing on the Quality of the implementation of the education and that involves Partnering with education adult education specialist people who who know how to do this who have data behind their methods So forth and if we find gaps where there is no data, then we need to get that data I think those are the two biggest things Netta at a local level Trying to include the education in the things that are already happening So you know stealing a month of the he-monk rotation type of thing where they're already going to do a rotation Why not have it focused on that area? I think it's an opportunity that we haven't really exploited very much and then on a larger level I think we're focusing too much on the genome the if this meeting today was efficient therapeutic medicine Using genome as the tactic I think it would have broader appeal than genome medicine and Those of us that are here know what genome medicine is but a lot of our target audience are just trying to Practice medicine in the in the pharmacy sense of it or the medical sense of it the nursing sense of it And so I think keeping that focus I think will be valuable because that way people because that's what they're wanting to do And then we'll get stuck on the double helix part Okay, any last comments from the audience? If not, I want to thank everybody for the discussion and thank again our three speakers Terry I turn it back to you So what we're going to try to do now is kind of run through things that we've heard in the past two days and also come to some consensus at least on on those things as well as as sort of opportunities for research and Clinical informatics and a couple of other things so Mary and Simone and I have prepared a Summary and I'll just walk up there and do