 Thanks for your unstenting efforts in this regard, Josh. Why don't you go ahead and sit down and we'll open up our discussion periods. Lynn, you mentioned one of the barriers to patients getting access to pharmacogenetics at your community hospital setting was price and cost. But we haven't really discussed what you're seeing, everyone's seeing out there in the marketplace. What are you seeing typically for a pharmacogenetic test? So a few years ago, they were $3,000 to $5,000. Yeah, I don't think there's anything called typical. When I first started, they were $750 for a panel and then went down to $499 and then to $330 and now we can get them for $249 if it's a cash price. So they've come down quite a bit in price. $249 out-of-pocket expense for an underserved individual? Yes, definitely it's a barrier, especially if they're Medicare or Medicaid. But we also select laboratories based on not just price and quality, but also based on patient assistance programs. And so the ones that we've chosen to work with have really good patient assistance programs so that if the income is X, all you have to pay is $10. So that works really well in our clinic where we get patients from all over. Some will say, oh yeah, I don't want to go through insurance. Here's $249 pay-out-of-pocket versus others who say if I can't get this covered, there's no sense in me getting this testing done. There's one other component to that which is in many, if not most, institutions when it's a send-out test, the laboratory will generally add a multiplier to the cost that the patient actually sees. And that can range anywhere from three to ten times the cost of the test. So a $250 rack price test might be $2,500 with that additional thing. So it's, again, the cost is very difficult to get a handle on. So there's lots of issues that we really have to pay attention. I see knowing nods from people that have been down that route. And I'll just say just so that there's no confusion, we as an institution don't do the billing. It's the laboratory that does the billing. So whatever the laboratory charge is, that's what it is. And we luckily don't have to, it's not institution billed. So it goes to the insurance company. And then Julie. First, just one clarification. I was asked by Dr. Johnson what our number was in 1,200 patients. Patients are actually getting a prescription and then having an actionable result. Ours was about 15% and that's not far off of what Todd's reporting. My question is around what can we learn in the germline pharmacogenomics field from what's happened in cancer? Three of the presentations we just had talked about foundation medicine. Their test is now in front of the FDA and CMS for clearance. And it's a panel-based test, multiple genes on it. And we've been talking all day about, you know, can we take a multiple gene approach in germline pharmacogenomics. So how is it that the foundation test is being considered like that, where I would argue, you know, I'm a medical oncologist, most of the genes on that panel have much less evidence than what we're been talking about today. And those tests are getting paid for. I've sent them hundreds of times on my patients. All of our institutions have homegrown oncopanels and nobody ever asked how those are paid for. And then why are we facing something different with germline? So let me give you a direct answer to your question of, are we asking if it's been paid for? Yes, we are asking that question. I will tell you we're asking a lot more of those questions in terms of what's being paid for and how are we paying for it in genomics. Then we have ever have done. And part of the problem in this space is, well, there's three fundamental problems. And we'll talk about it a little bit more later, but just to give you a synopsis of not only the answer to your question, but also the fundamental issues. And this we'll talk to some of the earlier speakers' issues about reimbursement. There is a fundamental issue in this domain around coding. I don't mean coding the drug. I mean coding for reimbursement. We see, our belief is there's about 65,000 tests on the market. We see about 10 new tests per day. And for those folks that are in the weeds here, there's only about 200 CPT codes. For folks that know what a CPT code is, that's what we will largely pay off. So there is a fundamental mismatch between the number of tests that everyone is working with and the number of codes that a payer is making a decision on what gets reimbursed. That is, in my opinion, out of all the discussion I've seen, I've been part of most of it for today, the most fundamental issue that this community needs to resolve is that. Because then we will start paying for things in a more precise way with great accuracy and we're going to reduce a lot of friction in the system. The second thing that we worry about to push on your coming around pharmacogenesis quality, we worry about the variability in the wet lab side and the dry lab side of quality. When we think about a lot of these lab tests, it is very difficult to get a measure on what is good, right? We really worry about it on the interpretation side. Most of the academic medical centers we've seen in the last eight weeks tell us that they're the best. I'm sure you're all wonderful, but by definition not everyone can be the best, but that's okay. So we need to have better measures on quality. So if you think about it, what are the key themes around the limitation of pharmacogenomics and why aren't more payers paying for it? Number one, you've got to get some lubrication in the system. You've got to get the coding right. We need to figure out a way of solving the coding problem. And when that gets solved for, a lot of these issues will be much easier to address because we can figure out, I've seen this claim come in with this code. This should be reimbursed. Right now, there's only 200 codes and given how many tests there are, it's very difficult to know. This is the pharmacogenomics test. It maps to the CPT code. We should reimburse for it if we have a policy for it. There's a practical issue here. On the foundation medicine piece, those CPT codes are clearer, so it's easier for us to run through our own internal systems. That makes sense to everybody. I cannot stress that enough if there's one thing this group of people, and you're all super smart folks in the room, there's one thing that you try and figure out is how do you help solve the coding problem from 65,000 tests to how a payer can actually pay for it? Right? Secondly, solve the quality problem. Thirdly, and we'll talk about this a little bit more later on, is how do you solve the health economics? There is an absence of health economics in much of this space. I think it's some work going on. I'd love to learn a little bit more about it. But we need to get better communication to the pairs around what some of the health economics are. For other tests in other areas, that evidence is clearer to us. In this area, I think it's still fairly nascent. So to summarize, coding, quality, health economics. That makes sense? A long answer to your question. But that drives how we make those kind of decisions. So let me move along, and thanks for that. The coding and the quality are things that are being attended to by ACMG and CAP and AMP, so not particularly... Well, they are. So I mean, you can disagree, but that doesn't change the fact that they are actually addressing those issues in as much as they can within the system, and we can talk offline about what's actually happening. But specific to the research questions that are really... and the implementation questions that are relevant here, I think the economic piece, which we'll hear some more about tomorrow, is relevant. But I think Scott, to answer your question a little bit more directly, which was about the differences, there are also some issues about tests that are brought to market as laboratory-developed tests, which don't undergo an FDA approval process, but can still go out into the market versus those that are submitted for FDA approval that can either go through a full approval process or if they look vaguely like another test can go through an expedited 510K process. So there's different ways to get to the market, and there's inequity on the test developer side in terms of, do we really go the full FDA route versus if we can get into the market and decrease our cost? So that's another big issue that's being talked a lot about in the laboratory community. So, Julie, I think you were next. Yeah, and so, perfectly situated. It was a cost question. So, Josh, on one of your slides, you had a list of... What is Josh? It's like right between me and this camera, right? You had a list of, I think, reasons that physicians did order the tests. So, at the top was something about evidence that the bottom was about an alert. You know what slide I'm talking about? So number two on the list was absence of out-of-patient... absence of out-of-pocket costs for the patient. So, again, just sort of thinking about this, this is a real barrier if the only way it's attractive to physicians, if it's free, because it can't possibly be sustainable if it's free, right? So, again, there's this artificial thing that's been created in some ways by the grant funding, but it's not sustainable and it's not scalable. So, I mean, I'm not sure if it's a question for you or if it's just a question for all of us. How do we get past that where physicians only value it if it's free, but otherwise they worry about it? Yeah, no, I think... I mean, it's obviously a bigger question, but it's a huge point. I mean, and so internally, that was at a time in which it was all paid for in our institution before we started switching to billing insurers for the test. And as with you, I think we found that there's reasonable reimbursement for that, but then there's just a variety, just as Lynn has talked about, sort of a variety of potential costs and even a couple hundred dollars, 200 dollars could be a lot if the patient doesn't have the wherewithal to pay for it. I think it's the... But in our perspective, it's almost more the uncertainty about knowing what will happen. I think that drives some of that as opposed to you sort of know exactly what's gonna happen with the CBC or any other sort of standard tests and you know that if they don't pay for it, it'll be a pretty cheap cost and things like that. You know, the provider generally doesn't know what a test costs and in this environment, this could be an expensive test. So there becomes more uncertainties about things like that. Yeah, I think that's a good point and I think a lot of times whenever we say genetic, everybody automatically assumes us thousands of dollars. But Josh Peterson in his work is beginning to incorporate some willingness to pay analyses and I think we also need to look at other approaches like value of information analysis to really start to get at some of these questions about where do we really need to be positioned to be able to have something where we could check off some of those things and eliminate some of those cost concerns as being the primary driver. The inequity of healthcare services is manifest across all the issues. It's not specific to genetics, but at least if we can get some specific information on that, that should be something that should be included in further economic analyses. I had Heidi next. So I did want to go back to some of John's points because I think they are really critical for us to actually proceed with implementation and that is around the coding question and the quality of interpretation. On the coding side, I think mainly organized by AMP, there has been efforts to develop new codes, but it's been incredibly slow and I think some of the barrier is just getting the volunteers within the community to come up with appropriate codes. But I've seen it work well where I was part of a few different groups that came up with hearing loss codes and cardiomyopathy codes. The idea was not to come up with a code for every test because there's just too many tests that will continually be iteratively developed with small incremental changes, but to come up with codes that can be generically applied to say, all right, if this is the broad indication, let's say hearing loss, you need to test at least these 10 genes, but it could be 10 or it could be 100 or it could be anything above 10. This is the minimum to make it a clinically useful test. If a lab wants to add as many as they want, that's up to them and this code is okay to use for all of these different. So I think incumbent upon this group in pharmacogenomics would be bringing together a group that can say, okay, here's some, not completely generic, but slightly more generic code sets to use in pharmacogenomics that doesn't require a brand new code every time somebody makes a minor snip change to a test that can be applied broadly and I think that that would help the transparency in the coding process without an enormous burden of a new code every time somebody launches a test. So that's one thing that I think this group could help with implementation is actually to help with that code definition working with existing efforts that AMP and others are leading. Another thing is around the accuracy of the interpretation, which is extraordinarily hard to objectively deal with and I would argue that no one is really tackling it with one exception and I obviously am a bit biased as a PI of ClinGen, but I have watched extraordinarily or extraordinary improvement in the quality of interpretation from laboratories submitting to ClinVar. A couple of labs that I won't name but I don't think did a very good job and when they started submitting to ClinVar there was a bit of an out of the problems, but it made a huge difference in them changing their practices and in one case going to their CEO and saying we need to actually hire people who know what they're doing because we just really messed up here and I've also seen, and this is somewhat anecdotal but it's been validated by many of my colleagues, a correlation between quality and whether you submit to ClinVar at all and so I would argue that simple requirement of submission to ClinVar to create a transparent, peer review of your own interpretations has a huge improvement in the quality of interpretation but I can tell you that having gone to CAP multiple times to say please put this in as a quality assurance requirement just like all of our labs have to check temperatures on every PCR machine and every freezer and 8 million things I do for quality assurance reasons that probably have little to impact the quality of my test. That is one that I think has a huge impact yet CAP has not agreed to do this so I think it's going to have to come from the healthcare providers and frankly the insurers and the biggest movement I've seen beneficial is when Aetna required laboratories to submit to ClinVar to reimburse BRCA too all the laboratories all of a sudden submitted on that gene and I think the insurers are the best to push this issue because I haven't succeeded in getting CAP to do it. Great, I think there was yes please. Hi, I'm Melissa Clark from Howard University so in listening to some of the presentations there were some cost data and cost effectiveness data in some testing arenas and so I'm wondering if anyone especially those who might be from accountable care organizations have been looking at the idea of using pharmacogenomic testing as sort of a loss leader absorbing that cost and seeing what kind of effect that can have prospectively on your cost as an ACO either on your own as an ACO or in conjunction with a payer as partnered with you. Not everybody at once. I think it's an interesting question I think the the challenge as we've thought about this is you know the the confounders you know it's particular for people that are on a lot of medications where they're more likely to have a pharmacogenomic benefit it's there's a lot of different things that are happening a lot heterogeneity it's it would be difficult to capture the data the only group that I know of that is doing um I don't think they're capturing the outcomes data so I'm not sure I know I probably Josh do you want to comment in terms of how whether or not this is something and you know all of us that are capturing the outcomes I think are dealing with this this issue but in terms of being able to sell that as this is you know using a loss leader concept I think that's a little bit different I don't know that have a lot to add to that I mean we're certainly we're modeling it but you know it still has a I should say we Josh Peterson but you know it still has a cost and there's some cost per outcome and the whether or not it actually leads to an actual difference we haven't actually talked about it with insurers across a group um it is interesting though that we have these visions of how things like this could work across a network of hospitals and we're certainly thinking about it but we haven't had those discussions and I'm aware of well I think one of the examples that I heard earlier um I believe there was a quote of a four percent um potential avoidance and readmissions for those individuals on plavix I mean that would be for example a cost model that could be expanded to look at okay if the hospital prospectively paid for this how much would be saved in the readmissions rate for adverse drug events or failure therapy in that particular example for instance is built into the simulator and you can you know try different things and actually I think that might have been on one of my slides that um I actually did do an observational study a cost effectiveness study to say okay if we had to test um in in patients all 1400 patients and compared that to the amount of money we were spending because these were 30 day readmissions you know would we at least break even and you know there's a lot of assumptions you have to make in that and with those assumptions you know I had evaluated that we would even be ahead by 50k uh now what they're asking me to do and rightly so is to have their cardiology decision support people within their own cardiology group identify the patients go through the medical records and look to see if those um cost savings and expectations and estimates you know are still on target um but I still think that until they see the data on what the testing helped them do they because there's it is multifactorial and there's other reasons why a patient may not have you know may have gotten another MI um a lot of the work in that area is um how you define a failure a plavix failure and so a stent that's been from boast as a plavix failure that's that's one component and asking Lori about that in some of her studies it is a small component but what about the MIs and what about the strokes and so people are also defining failures differently and so the economic analysis um is different based on what they're considering a failure and I can't stress enough that this behavioral economics um idea of why we make the decisions that we make as physicians as patients etc in terms of you know what we're looking at what is our cost benefit so even when you give numbers it doesn't always motivate someone to make a change I think the other thing to remember is that avoided cost sometimes is also avoided income uh from the hospital perspective I mean that's the perversity of our system the patient that Josh presented they got paid for all those stents and uh and so you know we have real issues uh depending on your perspective and I I'll talk a little bit about that in my talk tomorrow about how different perspectives gives you different answers so I'm going to go to Todd and then to Howard and then to Julie I think the discussion that Lynn brought up and what we're just having is is really relevant what do we really you know when you say you want to cost uh estimates or analysis being a pharmacogenomics guy when we first started talking about uh you know we're going to do a cost effective analysis well you then realize what that means to a geneticist is completely different than what it means to an economist because we engage in economists and within the ignite network when we were we got the data from the copitagral study it's like okay let's do a cost defectiveness analysis on it well we had like 12 different opinions as to what that cost effective analysis really was um and it it's different with you know between economists and and geneticists and it's even different within so I think any sort of guidance that we can get from you know from whether it's insurance companies or CMS or whoever as far as how we really should run the trials you know for some of these things do they mean cost of illness do they mean cost to prevent a death does it go greater than one year are you only interested in you know up to one year because of their so I think there's a lot of things like that that you know and I think a collaboration with us with any of the payers would be really helpful in trying to hurt so we can understand what it is that we really want and it goes back to designing the clinical trials we're talking about what clinical trial do we really want well depends on what the question is and what the really end points are so we've had a Moffat said at accountable care organization that's oncology specific for the last four and a half years and so in that context we use pharmacogenomics a lot because we're in shared risk models and we in that model could almost care less about CPT codes because you're holding the risk jointly with united or wherever else the partner is but it's a it's a very different dynamic and so it's very much is the testing going to avoid an event that is both bad for patients and bad for the bottom line and having both of those things is a real stick and carrot to make change happen and so a milo leukemia patient that gets a fungal infection costs us $29,000 extra in the first year to manage we lose $29,000 a year giving someone an event they don't want that can often be fatal so lots of incentive to not do that as opposed to the Josh Jenny model where you want to give nine stints to everybody but the idea that we're in a situation for that and so it's been quite valuable when we go into price our bundles we go in and say well a breast cancer patient who gets neuropathy from taxings costs $8,000 a year more to manage than a breast cancer patient without neuropathy let's model that and do preemptive work to try to avoid that so it changes that dynamic a lot and becomes institutional value becomes more important than insurance or these other things so I think we probably won't get to a time when it's all accountable care stuff nor will we get to a time where it's zero but it definitely has introduced a greater appreciation in our leadership of why pharmacogenomics can be valuable to them now and later Julie well and I think the thing that always gets lost in the conversation and in the Clopidoregal example the events that you didn't capture are death and we know that from the larger Ignite project and of course in an economic analysis I mean that's a cheap outcome and so going back to Mr. Anderson's presentation we cannot lose the patient at the other end of this and death is the worst outcome and our economic models don't necessarily put value on that so how we start to pay better attention to the outcomes that these patients are experiencing and not just the health economics part I mean I think again part of this culture shift and it's not just in pharmacogenetics it's in all of healthcare but a fair portion of the events that we saw in the Clopidoregal study which Laurie will present were deaths so we just can't forget those So I had a question for Nick and this is relevant to some of the work that we've been doing and emerge in others where we actually have some longitudinal information on patients and I was thinking about your exceptional responder study and thinking about ways to identify those patients out of transactional electronic health record data and it seems to me that if you have a longitudinal record the good news is that we have pretty specific codes for things like metastatic breast cancer that type of thing and so in going you could presumably identify a cohort of patients with that code and then look to see who's still around after a set period of time and I'm wondering if you've been using that type of a method or whether that would be something that we could take advantage of some of our collaboratives that have electronic health record data where we could potentially identify those for some type of research related question but I think that's a great idea one proxy that we've used for exceptional responses so you mentioned how long they've been alive or how long they've had that code so that's one but when you're looking at a specific exceptional response to a specific drug we've had definitions per subtype of metastatic breast cancer for how long you're on a particular drug that might be an outlier so in certain subtypes if you're on a particular drug for more than two years that would be pretty unusual and so you could do a pretty quick search that way in longitudinal emerged data with codes so we've talked to some folks that have had that type of electronic data but I don't think we've ever talked to emerge about that that would be a really good discussion to have Howard, you end up finding some very unusual practice patterns as well especially in a tertiary center we've had patients who have been on hepatitis and kinase inhibitors and found some patients who have been on the drug for several years four or five months into their treatment the drug stopped working but the patient felt more comfortable staying on the drug and so it was continued to be prescribed and other things were added to it you know these steps of things which you don't want to find thankfully it wasn't a patient manager at our institution but there's still a lot of cleaning that has to be done because in the real world you get these unusual things where even though disease might be progressing there might be another reason to stay on the drug right or wrong yeah that was $9,000 a month Mary I have an unrelated question for Todd so you're 30% of patients who were already on a high risk drug that had a high risk genotype is that the right way to interpret that so the 25% of the patients that got one of the trigger meds and then were genotype about 25% of them had some sort of an actionable note that was sent back to the provider and what accounted for the majority of those was that PPI's and 2C19 I mean was it something that is going to make a big difference or a little difference in clinical outcomes I don't remember exactly the distribution of those to be that common it seems like it must have been 2C19 there was some of those that I would have to actually go back and look I don't remember I mean there's a wide variety of not even 25% of the total prescriptions were PPI's so that wasn't the majority of them because there was a really broad distribution there's a fair number of pain meds to tram it all was the most there's some statins and stuff so it wasn't dominated by any one specific thing and we're actually as we were enrolling we were a little concerned that we're going to have the first out of the first 500 patients you know 450 would be PPI's but that's actually not the case it's a pretty wide distribution Jeff so I'm wondering what we could do with all the wonderful implementation sites and projects that we've heard about today you know I'm impressed as you'll hear about tomorrow what Julia and Lara did with the Cip 2C19 work and to aggregate a number of groups that are doing similar things so I guess the question is how do we harness the sites that we've heard from today to band together to build evidence bases to build economic models to actually drive this over the goal line because I think the effort seems to be somewhat distributed and not as coordinated as it possibly could be to have kind of impact that we want to see so I'm just curious as to how all of you who presented today could think about you know another model that would allow these types of measures to have impact. The one thing I think that could help that is what I mentioned earlier about knowing exactly where the goal line is that we're trying to get to so what is it from a cost standpoint or from at least a reimbursement standpoint what is it that we really need to get done to actually get it to a point where it's reimbursement which will drive a fair amount of it if that's the case. I'm not sure that at least I don't know exactly what it is from the cost standpoint or how many deaths need to be prevented per $20,000 of genotyping or those sort of things so that would help I think. So what you're saying is that even with everybody in the room here it's an incomplete type of model because we don't have the some of the payers or other groups that are going to make the decisions but we saw in JASA's presentation that right after the American Heart Association meetings it was a significant uptick in ordering of CIP 2019 so even though that wasn't necessarily a payer driven model the provider community really saw this as valuable. Well I'm sure there'll be more information tomorrow when we go over the cost effectiveness but some of the things that Todd and I have interacted about is that we have a self-insured employee population at mission and we have other groups in the area that are self-insured and in trying to pilot out a study to say well when you say cost effectiveness what parameters are you collecting and so for our employee health program they're really wanting to know should they be covering a panel of pharmacogenetic tests and so working with one ohm we're trying to identify what are the different pieces of information where do they come from, how do we get it and working with you and others to say well these are the parameters that we looked at total cost of care is kind of huge for me to think about but having specific parameters to say how are we going to get this information in and collected and survey it and then monitor it and evaluate it I don't want to have to come up with my own end points and yet that's what I've been doing and why haven't I reached out again to talk with you so I think there should there can be much more of that we're all busy and we don't always think about it and that's why these meetings are so important Terry well I might also address this issue of what evidence is convincing it's not only convincing the payers and probably John we'll talk about this a little bit more in the panel presentation but it's also convincing the physicians as we've heard around the table physicians tend to be convinced by they say randomized clinical trials although more than half of what we do in clinical medicine probably vastly more than half has no clinical trial evidence behind it I think one thing that we might want to consider is whether as Jeff suggested there's a way to bring these QI projects that are already going on in hospitals and medical centers around the country to bring them together and this was actually raised in our eighth genomic medicine meeting back two years ago and yet we haven't figured out a good way of doing that so if the collected heads around the room can help us with that that would be I think a tremendous step forward to try to harness what you guys are already doing I think one thing that should be mentioned is that there is an ongoing effort between Ignite Emerge and Caesar to try and harmonize outcomes that we're all looking at in our different studies and I think that the intent is to have an outcomes repository as part of the Ignite program but with ownership distributed across those groups but there's no reason that it should be restricted to that because all of the things that we've talked about all of you are going to be measuring certain outcomes of care and you're going to be defining those outcomes in certain ways and by getting those out into the public then people that want to do a similar study can at least begin to choose similar outcomes so much as we've looked at promise for example for patient reported outcomes we're going to land on that set to use across these studies we could presumably use these types of outcomes across multiple studies and then at least we don't have to then have the argument about well you measured your outcomes this way and I measured them this way and how can we actually then reconcile what we think we found. I've heard a lot of comments about you kind of want to know where the goal post is particularly for payers you know I think that's a question that people ask a lot and obviously it's often payers specific but we did a study taking a look at many coverage policies across large private payers looking specifically at those for sequencing based test and looking at multi gene panels and payers are still at least for publicly available coverage policies they're still making their assessments based on the framework of analytic validity clinical validity and clinical utility and still looking at large randomized trials or well conducted observation large observational studies as the evidence base going to professional guidelines technology assessments the same sort of evidence base for their assessments and it actually complicates things in to make to get a positive coverage decision for multi gene panels because their view is they're still looking at the clinical utility of individual genetic variants and actually the pharmacogenetics panels were some of the ones that had the few affirmative coverage decisions unlike some other gene panels so it's still they're lagging behind in sort of the assessment of these things but again it's still the evidence base is the evidence base of what people are making the American College of Cardiology American Heart Association deciding about pedigree it's the same evidence base but if you stick it in a cardiology panel with 50 genes it's going to make we might think that that sounds efficient but they're still worried about what are the downstream possibilities of having to act on the 49 other things that you might find in that panel that there's far less evidence about what to do rather than to see 19 so that's the story and inviting them to the table and having them part of the study design process is certainly an important model and I would I think that we should consider that Laura not to spoil my presentation for tomorrow too much but one thing I've thought a lot about for PGX where we're trying to look across many different institutions is we tend to focus a lot on outcomes we can measure at the individual level and then that gets really hard because our implementation is so different and we have all these caveats I think that we have a lot to learn from implementation science and looking some more at outcomes that we can look at at the institutional level and I think bringing in the payers who can talk about what institutional level outcomes would be most meaningful to them would be a really great way to sort of engage them in the conversation what gives you hope about that what's your example of what well I'll talk about this tomorrow but we tried to do a project looking at response to CDS and trying to compare it across institutions and we were pulling our hair out so we can't make this comparison and this isn't valid because we look at this and we sort of rolled back and made some conclusions and then I started reading the implementation science literature which sort of made all the conclu- if I'd read the implementation science literature more carefully before trying to do that project I would have come to all these conclusions before we tear our hair out for seven or eight months so you know but it's tough but I think there are some institutional level outcomes that we tend to overlook that could be meaningful and can help synthesize these really different projects that we're doing across lots of different sites I mean in some ways we have a model out there and that's PCORI because PCORI as part of their investment invested in a methodology service where they have an aggregation of methodologies that if you apply for PCORI funding you're expected to identify methodologies that are already represented there that you're using and that you're mapping to or to provide justification why the methodology you're using because it's not there and they've also invested in developing new methodologies as part of their they've funded researchers to research new methods and so it's an interesting model it's not one that I'm aware of that has really been used in other funding sources but it is a way that they've tried to get some consistency across the tremendously heterogeneous projects that they're funding so I think it's a valid point to at least think about Mary So in terms of thinking about a big project that we as a community could take on I guess I go back to the Vanderbilt story where you looked at your 53,000 patients whatever it was and you had real data on what percent of them received a high risk drug and real data on what percent of them had a high risk genotype we might quibble with exactly what those genes or drugs were but let's say we could do that again at least really carefully catalog the drug use in big groups of patients what were the criticisms that you got of that I mean it seems to me your conclusion was that you saved 383 serious adverse events you could have saved had you saved a high risk genotype prospectively and acted on it we don't need to genotype 53,000 people to know what their genotypes are going to be we can use all the population data that's available and estimate the frequency of all of the serious actionable variants in actionable genes can we merge those estimates with real data on drug use in these large populations additional estimates of how many bad events are prevented and thereby avoid doing another negative poorly planned clinical study I mean I'm just curious what kind of pushback do you get on when you state that that finding that you saved so many serious adverse events because to me that seems really impressive you genotyped people it didn't cost you that much money and you saved hundreds of patients from having something bad happen I mean what more do we need to see interesting question I mean that was published a couple years ago so I can tell you with Inning Knight we are doing another ecological study that we're looking at I think we've changed the name of that actually again now to think about it but the the ACPIC prescribing study we're looking at the same sorts of things across a broader group of hospitals and we certainly would others can join too and with that we could do the same calculations for to calculate adverse events and what the potential outcomes would be hopefully you tell the story of our larger community maybe it becomes more convincing and perhaps even with that can be intersected with some of these cost effectiveness models where you could even for a subset of those maybe we could actually put some data behind behind some of those questions around other corollary questions like cost effectiveness we haven't thought about those next steps but I think it's important to bring up the question now and start to think about what they are one of the comments I want to bring out Heidi mentioned sort of the multiplex pharmacogenetic testing question we actually did submit a CPT code early in the night maybe our second year in the night or something like that or something for that it was it was up to me to argue and I didn't do a very good job arguing its existence apparently but the idea was for a panel based test it had minimum criteria in the very same way we said genes had to be tested that would have pharmacogenetic indications with evidence and the panel just didn't feel like there was enough evidence to recommend this sort of this combined panel perspective test and there's a lot of reasons why that's hard but part of the reason why is there's not a lot of studies out there to support it the study that I talked about I think is one of them and that's one of the things that led to the CPT prescribing study that we're undertaking now is to try to say across the community there really is the advantage in fact in that study 14% of the people had four drugs that they were exposed to so basically one fourth one out of four people that get one get four so there is an opportunity to test because the hard data to get are the drug prescribing data but now a lot of you in these large centers healthcare systems have data on thousands and thousands and thousands of patients and the pharmacy benefit managers have millions can you get access to it though? a lot of that is publicly available so we're working with an economist that's actually used that publicly available data to do some work we're now using it on our own but it is out there can be used and I think you could probably answer some of those questions I mean Josh's point about arguing to the CPT committee having done that myself it's a very idiosyncratic process and it's generated by most of the CPT panel are surgeons so it's the way it's set up and it's under the control of the AMA and so the decisions that are made are not always particularly rational so I have in the back I have Todd and then I have Heidi and then I have okay so you asked the question about what made this researcher hopeful and I would say about getting payers at the table I would say payers are people too and we had a positive experience here in DC we looked at a cohort of about 180 patients on buprenorphine who were not adhering as judged by urine talks and we did CIP3A4 testing on them and were able to show that with pharmacogenomic testing guiding dosing recommendations that there was increased compliance with their buprenorphine regimen and we were able to successfully lobby the Medicaid agency in the city to change their prior authorization around that medication and our next step is really to see whether Medicaid will actually now cover the cost for testing for all buprenorphine patients so there's hope even with small cohorts my question was why was she hopeful that knowledge of implementation science would have resulted in better outcomes because it does no it's axiomatic no Todd another thing we don't want to underestimate is the fear of the providers for the liability of genotype data being available right now there's a lawsuit against Quest for somebody who was genotyped and they weren't acted on appropriately based on there was apparently two publications, I don't know all the details but it sounds like way less information than there is available on a lot of these other things and very often there is a lot of fear among physicians like maybe I now know what to do with 2D6 encoding if that's the drug I'm giving them but if I later give them some of these other ones what is their real liability what is their chances of being sued for that and some sort of clarification other than me saying well it's usually the guidelines from the expert societies and stuff that's not very reassuring to them and some sort of hard evidence or hard document or something that clarifies that I think would be very helpful yeah I think liability is an interesting beast because clinicians are three times more comfortable with an error of omission versus an error of commission and some of it has to do with the attribution of that but you know the adverse event is is real for again as Julie said for that patient whether we omit something that we should be doing or commit something it would be my argument that it's not going to be too long that we're going to see a stent patient like Josh is saying why didn't you do this test because the evidence is out there so you know it's a comfortable barrier to hide behind but I think it's one that's intellectually not defensible particularly if we take a quality and safety approach to this because again they would never make that argument for a drug-drug interaction they would never make that argument for an allergy so it's back pretty strongly when we hear that I think I had Heidi and then I have Julie so I just wanted to go back to Josh's example so although I understand your argument for the utility of a multiplex panel no no no I'm saying I understand it although I can see why insurers are still you know not convinced but my argument wasn't actually around arguing the utility of a multiplex panel it was actually to say if we can define the drug indications and where there is actual you know clear utility you know from CPIC guidelines etc even if it's just for one drug but then the idea is that the code says at least this marker and then it's up to you if you want to for the same price throw in 100 more because today for most of the platforms we work on it's not much incremental additional cost to throw in the rest of the entire panel for the cost of the few snips you're doing for whatever drug and we at Brigham and Mass General worked with we being the pathology departments worked with Ithabrelucross Blue Shield to define 20 cancer genes that there was felt to be sufficient utility that that would be paid for and a price determined around those I think it was 20 and I don't know if you were involved in this next genes but then they agreed that we you know we could add as many other genes as we wanted as long as the price didn't change and that that code could you know could be applied for the set that was had utility and documentation but it was you know up to us to continue to add whatever we wanted and that would be irrelevant so that's the kind of argument I'm making is you define your code based on what is clinically defined as useful but the labs can keep adding content to their hearts to light as long as the price and reimbursement doesn't change which is what should be controlled by the insurance and you know utility questions does that make sense we need some data around are there any real downstream costs the pushback is always going to pay for all the stuff you're going to now find with the extra content but I've yet to see data nor have we created any of our own to show that it does or doesn't create extra cost and you know there will be the unusual patient there but just like with imaging you find that unusual blip but you know there's always that pushback you know you're going to do a panel you're going to find a leaf from any syndrome now we have to scan them for well do all these extra things and you know right yeah the whole thing I mean but there's that acute argument our approach to insurance companies and you can correct me if I'm wrong but our approach has been only look in the first 12 months because after that the patient may have switched to a different insurance company so in that 12 month window are there a bunch of extra things that we've added that would harm the insurance company and that's as a field we haven't answered that yet and I should clarify that it wasn't necessarily agreement that we would return the results of those additional genes so I you know they wanted a very clear protocol around the implementation of this what was the criteria that patients would get this test that they would then pay for and making sure that we had a workflow in process to evaluate those patients to make sure they were appropriate for the test and I but I think addressing the issue on the downstream end is also important so what of these genes will you actually give back and what's the criteria to determine that and creating that sort of workflow so that it's clear you won't have just return willy nilly and then all of these downstream things happen which is unclear because it's a very frustrating discussion to have with the treatment we're doing a lot of partnered pilots now never say the word research with the insurance company they'll pay for pilots but not research so do pilots but that's always the thing that comes up is well what costs are we adding what problems are we creating yeah Julie so just back to the liability thing I wanted to completely agree with Mark when we were beginning our clopidogrel program the conversation came up with our interventional cardiologist about what was the liability and we brought the at the system our institution in and we came to the exact conclusion and the cardiologist actually rapidly agreed which was the fact that you don't know their genotype doesn't change their genotype their genotype is there and the absence of knowledge doesn't protect you and so I agree I mean I think if you're getting that you should push back hard because it's you know the fact that they don't know it doesn't mean it's not true I think the concern where I'm getting is not that initial event it's the later if six months later they get you know if there was originally a 2D6 encoding thing and then six months later they get you know and they did not actually look back because we don't have the alert system in the EHR to notify them and then they have an event it's like and then they go back it's like well doctor you should have known because this genotype was in the EMR EHR and you didn't do anything about it so it's not so much liability of the actual the initial event it's the getting the alerts and stuff getting to the patient or to the doctor in time. Yeah so you can fix that then by just building alerts for all of the drugs in that alerts when they have the genotype plus the drug? That links up to in real time and they can get at the data but it's got to be something that the genotype data's got to be structured in the EMR so they can do it right and today we can't do that or we don't at least we don't have all of our genetic data in a structured format in the EHR so we can actually do that eventually yeah I agree but tomorrow here today and tomorrow it doesn't work maybe in a couple months yeah I mean my view would be is that you know you if you can't do that you don't return because I think the obligation then is a systematic one because with the difference with the the one exceptional aspect of genetic and genomic data compared to most other data is the persistence of the relevance of the information it doesn't matter if the sodium you know is you know you're not going to go back to it you're going to do another sodium but for this we're going to go back to the different on the implementers to build in those systems and if you really aren't at a point where you can do it then you can say well we're going to return to C19 but we're even though we've done this chip we're going to suppress the rest of those results until we have that infrastructure into place and that would be the covenant that I would make you know with our clinicians I'm not going to put you in the position of you having to try and remember to it's great for Julie and Todd and Josh to Dick and Ollie to have systems in place to do this but you know when people move the question of what who how does the obligation follow the person does the does the person take on the obligation of knowing their own genotypes or does do we think of a way of creating a national infrastructure that allows people to access their data from wherever they are and access current clinical decision support so it's easy to say well you know you have 50 genes just develop 50 pieces of clinical decision support that's easier said than done too but I think that if one of the things that comes out of this meeting is thinking about things that we ought to be doing to move the field forward that is something that seems to me to be a priority. And on that note we'll draw our conclusion to a close for this particular session and I would think that ending on a patient centered note and the fact that the information has to follow the patient wherever they go is a critically important take away so with that I'd like to thank all of our speakers and all the discussants.