 At any rate, what I'm going to try and do in this talk, probably unsuccessfully, is to combine some of the things that I thought were interesting from that inquiry, and combine that with some conceptual frameworks that I hope will pull together some of the things that we've been hearing about how we might approach this. I want to try and do three things. I want to try and go beyond clinical utility and talk about something that I've been trying to think more about, which is value, and present a very rough theoretical construct around that. I want to actually take that construct and then apply it to some of the variants and the data that we obtained from the inquiry, and then I'm going to finish with some general comments of things that I found interesting in the results of that inquiry, and then we'll move into our general discussion. I might disagree with Bob Nussbaum on one point, which is the idea that we actually understand what we're talking about with clinical utility. I agree that we don't understand what we're talking about with actionability, and so this is my lame attempt to define it, that knowledge of a variant implies a specific action for a given clinical scenario. In this paper, what is the clinical utility of genetic testing? They refer to that as referring to the ability of a screening or diagnostic test to provide or ameliorate adverse health outcomes such as mortality, morbidity, or disability through the adoption of efficacious treatments conditioned on test results. That's their first definition, they say, but you could use a broader definition, which is any use of test results to inform clinical decision making, and then they use the broadest definition, which I didn't include, which is essentially we can think of just about anything that we can use this for and call it clinical utility, which I think was represented well in Robert's talk. When we think about clinical utility, the reason it's hard to define it is because there's a lot of components that go into it. What are the different components of usefulness? What are the benefits? What are the drawbacks or what we might characterize as harms? How can we define and measure these factors, and how should they be weighed against one another? When we talk about usefulness and relative benefit, for whom are we talking about this? This information is taken from a really nice article that I came across by Andrew Smart, a multi-dimensional model of clinical utility that was published in 2006. I actually stole table two, which is a summary of dimensions of clinical utility. Here I think I'm not going to go through this, but this is the real crux of the problem, is that there's all these different pieces that have to be lumped under clinical utility if we really want to understand it at its most nuanced. There's all sorts of different perspectives under the acceptability. Is it acceptable to clinician, to the clients, or patients, to society? It's very difficult. We're almost getting to the point of having a Potter-Stewart-ish definition. I can't define clinical utility, but I know it when I see it. Now, Steve Toich, who has been on the eGap working group and was chair of the Secretary's Advisory Committee, represented how we might actually try and do a decision factor matrix for clinical utility, and this is what he came up with. Now, I present it not because I think this is the solution, I think it actually represents part of the problem, is that we can think about all of these different things and if we try and determine what is the evidence around each of these different cells, you can see why it can lead to paralysis by analysis in some cases. I'm kind of a simple person, and so this is my attempt to make things simple with the question mark, I think being highly relevant here, and that's trying to think about this from the perspective of value. When I think about value, I think about it in a very simplistic way, which is what are the outcomes and how much does it cost? So this is not meant to be an actual algebraically representative. I'm sorry, I just went into paralysis by analysis myself. It's not meant to be algebraically accurate, but it just represents one way that we can kind of look at it and the things that we need to consider. Now, it's important to recognize that there's a bunch of things that we can look at under outcomes, there's a bunch of things that we can look at under cost, particularly if we look at it from whose perspective. But if we accept that this is a reasonable way to think about things, then we can create a little bit of a simpler matrix where we can have outcomes of care on one side, cost of care on the other. Outcomes can improve, they can be unchanged, or they can worsen. Outcomes can be decreased, they can remain the same, or they can be increased. And assuming that you could create something like this, then based on evidence, you could assign a given intervention treatment test into one of these nine cells, which would also in some ways inform whether or not you should implement it. So if you had a intervention that clearly improved outcomes and clearly decreased cost, that's no brainer, that's something we should be doing. If it improves outcomes and leaves cost neutral, that's something we should be doing. If it improves outcomes but it costs more, that's something we maybe should do. And the example of that is imatinid for a chronic myelogenous leukemia. It's a very expensive drug, but it clearly improves outcomes and it's universally prescribed and covered for appropriate indications because as a general rule, if we have a dramatic improvement, particularly a treatment for what has previously been an untreatable condition, we will tolerate cost increase to the system, although it's also recognizable that in the global perspective that's unsustainable, which is why our health care costs are probably 17% of our GDP. On the other hand, if we have something that worsens outcomes and increases cost, we're going to say get that out of here, we don't want to do it. Unfortunately, there's a lot of what we do with day-to-day in medicine that does exactly this, but we haven't really analyzed it. And then there's a couple where you might say, well, most likely we wouldn't do it if it didn't change outcomes and it was cost neutral, but there may be certain circumstances. So this would be one way that you might be able to do a very rapid first pass to say can we prioritize things so that we can exert our energies on these things that look likely to be in the upper left-hand corner of this matrix and eliminate those things that are more in the lower right-hand of this matrix. And so I thought it would be interesting, I may be a minority of one here, but I thought it would be interesting to take some of the results from the inquiry that we sent out to you and just kind of see if we could use this conceptual framework to look at it. Now, the variants that we chose were meant to span the range from what we would consider to be ones where, for instance, this one, the Avakavir situation where there's clearly an FDA black box warning that says you should test this for everybody that you're going to prescribe Avakavir for. To others, we were deliberately selecting things that we said there is almost no evidence that we have that this in any way should perform, even has clinical validity, much less utility. So it was very interesting to see the results. And I should also note, as we look through these things, that a number of the folks that responded said, I'm responding to this because you asked me to, so I'm assuming your mother raised you right, but I'm not a clinician. So I, in some ways, don't really know what it is that I'm actually responding to. So we'll go into that in a bit more detail. So this is the Avakavir scenario, a 28-year-old man with HIV-AIDS looked to be started on Avakavir. Should we look at the HLA-B5701? Should we routinely use this in this clinical scenario in just a bit over 50 percent? So definitely or probably yes. The other large minority said, I don't know, have no opinion or I'm not sure. There was only 2 percent that said probably not. Now if we look at the reasons for that, you can see that we pretty much hit it right on the head, that this is for prevention of an adverse event. This is probably the cleanest pharmacogenomic variant that we have because there's no efficacy issues associated with this. It's simply to prevent severe cutaneous reaction. So we pretty much identified the right reason that we were doing this. There also was the importance of the availability of evidence relating to this. And also noted that there was some recognition that there was a black box warning. So I think, and there is actually some cost-effectiveness data on application of HLA-B5701 testing in a vacavir that shows that it in fact improves cost-effectiveness. So if we were to categorize this, I would put this in this. We're improving outcomes and we're actually decreasing costs based on these studies. And that's probably where we should classify this. So then the question is if we really believe that, how can we ensure that all patients who are going to be prescribed a vacavir are tested? So that's really an implementation question as opposed to an evidence question. The second variant, and you've kind of heard my views on this already, and I intentionally included two scenarios for the TPMT-STAR-2 heterozygote. The first was a four-year-old girl with acute lymphoblastic leukemia being considered for treatment with six more captive hearing. Should this variant be routinely used in this clinical scenario? And for some reason, this is not projecting the way it's supposed to. Oh, well, I hope that doesn't continue. But I'm sorry? Could be, yeah, if we all got it at once, that would be an interesting thing. The bottom line is this looks pretty similar. Note that they're in the book. Yeah, they are in the book. So if you want to look in the book, that would be the new delasty. But the bottom line on this one is that these results looked almost exactly the same as the HLA. That a slight majority indicated that it probably or definitely should be used. A little bit more, 5% said probably not. But still the majority were, or the large minority were not sure, no opinion. Again, by a weak majority, prevention of an adverse event was the most important factor given there clearly was recognition that this impacts the dose that you're going to use. But there's also some recognition that this has an impact in efficacy and changes patient care. So I want to contrast. So if I were to characterize this, based on some, again, cost-effectiveness data that has been alluded to before, you could probably put it in this category. But the asterisk is that if you only are looking at prevention of adverse events, I think you could probably characterize it as here. But as I indicated in my earlier comments, we have to have evidence for both efficacy and for adverse event prevention. And there's at least some literature. It's very weak methodologically. But there's some literature to the suggest that the individuals that are treated with standard doses of 6MP who have the star 2 variant actually have better outcomes related to minimal residual disease. And so that raises a question in my mind is whether the standard dose is actually too low for those individuals of wild type. Now, Howard mentioned earlier that if the treatment practice is actually to treat to neutropenia, which is a phenotype measurement, then it probably doesn't matter because you're treating to the phenotype of interest, which is you're treating to the edge of bone marrow suppression, which is maximizing toxicity. And you're using a phenotypic outcome rather than a genotypic outcome. But if we were to use this data to lower the dose for all of those individuals, that may not be the right decision. And in fact, when I saw some of the studies and I thought if my daughter had acute lymphoblastic leukemia and they were wild type, I might ask for a higher dose. Now, here's a different scenario, same variant, but this is a 37 year old woman with inflammatory bowel disease being considered for treatment with 6MP. Should this variant be routinely used? Now, we definitely saw a change in the number of people. Slightly over a quarter said that you should definitely or probably use this information. The vast majority didn't know or were unsure. And then about 6% said probably or definitely not use this information. Again, the most prominent reason for this is prevention of the adverse event. But there was also recognition of the importance of the availability of published evidence or in some cases lack thereof. And I included a couple of comments that came along with this. The level of evidence is fair to poor. No professional society guideline that I know of need more evidence. Implications of testing is still uncertain. And I think that that's reasonable. So again, if we were to put this up here, you might say, okay, it would be in this category if we're only concerned with prevention of the adverse event. But I think this scenario is instructive, at least for me, because the endpoint of inflammatory bowel disease is not mortality in the short term. Where the outcome of inadequately treated ELL is significant morbidity and mortality related to relapse, leading to bone marrow transplantation and death from the disease. So I think we could be more tolerant of toxicity in adverse event in an ALL scenario than we would probably want to be in an inflammatory bowel disease scenario, where the risk of a commission error is much more likely to result in the death or significant morbidity of a patient as opposed to the disease itself. And as Howard mentioned, the problem that we have in the scenario, and there's literature to support this, is that the monitoring for these patients is much less robust. And so in some ways, the clinical phenotyping is not being done adequately to prevent harm in this scenario. So there may be more importance to rely on the genotype. So how do we account for contextual differences for use of the same variant? SIP 450 profiling, 43-year-old Caucasian woman with depression being considered for SSRI, should this variant be routinely used in this clinical scenario? Only 3% said definitely yes, 13% probably yes. So under a quarter, a lot, probably just over a majority, probably or definitely not. And what you can see here is the most important factor actually was availability of published evidence, and Muin should be happy to see that EGAP was prominently mentioned related to this. So some impact, at least of the audience that received this inquiry, that they were paying attention. Evidence is insufficient to guide SSRI. One could make use of PGX data without testing by simply choosing a drug metabolized by a different enzyme. Unproven EGAP was the single answer, like that's the answer to everything. So that's good. So if we were to approach this matrix, we'd say, well, we can't put an X in the box because we have insufficient evidence, at least per EGAP. However, and spoiler alert here, Dr. Murazek is going to present some data tomorrow, which I think is very intriguing. I've seen it in a different context. And if you believe the Mayo Clinic experience, then we may be in fact up in this area. So the question that that raises in my mind is how do we rapidly translate clinical data into evidence for dissemination and implementation beyond the initial system? So in other words, if it works at the Mayo Clinic, how can we make sure it works everywhere? Or how can we make sure that that evidence is generalizable? And that was an issue that came up in Gervanites and others' talks about the idea that we really need these real world methodologies and we need to be able to understand how they work. The last scenario is the scolioscore, which is a multi-gene panel. The clinical scenario is a prediction of risk for curve progression in a 12-year-old girl with idiopathic scoliosis. Should this variant be routinely used in this clinical scenario? There was no one that said definitely or probably yes in this case. There was a lot of uncertainty about what this test actually is. And then a number of people that says probably or definitely not. And there was a fair amount of ambiguity about what would be the most important impactor. The change of patient care is relatively high on the list. And here's a, I really thought this comment was excellent here. This is a good example of the detrimental effects of the vacuum that exists with respect to LDTs and regulatory guidance. I also think this test is available direct to consumer, another gaping regulatory hole. Consumers get the test push for expensive interventions, limited evidence of clinical validity and utility. But the consumer and doctor feel good about delivering the best medicine because it's personalized. I thought that was a really salient comment and something that really captures the idea of the downstream costs that we can inject into the system if we get this wrong. So if we base this on the currently available evidence, I would say we would put this in the definitely no category because the outcomes are essentially unchanged and the cost of care is increased because we're doing an additional test. However, the claims indicate that this will improve care because much like the oncotype test, what this test is designed to do is to identify the 95% of 12-year-old girls with idiopathic scoliosis who will not progress to malignant curve progression. And so you avoid the X-ray monitoring and the incumbent harms associated with that. So it will actually improve outcomes and decrease costs. So the question here is how can the resource that we're talking about help clinicians to balance claims from a company against evidence? A couple of general comments. It's interesting, again, considering that this is a group that's probably farther along the implementation road than many. There were only a handful of variants that four respondents to the survey, which is about 10%, are actually using in a routine basis. And then the handful of others. And then this gets at the resource question. Here's all the different resources that people are using to try and get information. So my takeaways that I thought were interesting from the inquiry is that there was a tremendously large percentage of don't know and not sure responses. Some of these may be due to the fact that we had a number of non-clinicians that responded. But I think that that probably reflects the current state of knowledge of most clinicians and certainly the general public. So I think it's something that we have to account for. There's even amongst those variants that we might consider to be really ready for prime time, there's still variability of opinion about whether or not they should be used. And in fact, there's quite a variety of opinions about whether or not things should be used. So the issue that we've come back to about how can we choose consensus is relevant. I think the thing that this inquiry demonstrated the best is we really need this proposed resource. I think that came through loud and clear. So with that, I'm going to end my somewhat prepared remarks and we'll move into the general discussion.