 Well, I'm really, really pleased to be here to talk to this group. It's now the second meeting I've gone to hosted by NHGRI, and I'm starting to recognize the similar faces in the room, so that's been rewarding. I wanted to just tell you how much I enjoy putting Colorado Trust on my name tag because no one knows what the heck that is. So I left the state health department a little over a year ago to become the head of a private health foundation, the Colorado Trust. And so I moved from bureaucracy to philanthropy, which was a really good move. It's much better to be on the philanthropic side. And we give grants to improve the health and well-being of the people of Colorado. So that's what a nice mission. I actually wanted to tie that back to this meeting, though, because there are over 200 foundations in the United States that give money to improve health in their state or across the nation. There's a group called Grantmakers in Health, and it's relevant because I will tell you there are no projects in genetics. This has not been seen as a mainstream activity to try to advance health through philanthropic giving. So I'll just kind of put that placeholder for those of you trying to advance things moving forward. I think I'm here, though, because I'm the chair of EGAP. Steve Toich and I wrote the EGAP methods. Russ Harris and I wrote the methods for the U.S. Preventive Services Task Force, and then I wrote the methods for the Secretary's Advisory Committee on Newborn Screening. And I only say that to tell you there is a standard for translating evidence into recommendations. It may not fit as well for genetic testing, although I'm going to make the case that it does, but I'm going to present it as a standard approach to thinking about how to move from evidence into recommendations. What you do with the recommendations after that are the policy implications, paying implications, et cetera. So that's what I'm going to do. And I was really pleased to hear Rex's talk because he basically gave my talk. So hopefully I'll get over those really quickly. I want to talk about how EGAP works, then talk about barriers and challenges, and then a lot of things you've all been talking about, solutions and opportunities for the future. So these are the questions I think you've heard that come up today about genetic testing, valid and reliable. How well do they predict outcomes? So that's both analytic validity and clinical validity in one sentence. What are the benefits and harms associated with the clinical use of these tests? We'll talk a little bit more. And then once we have them, what should we take, action should we take based on the results and how do we figure out what to do with the results once we get them? How should we respond to this exploding area? So EGAP was formed in this milieu to look at these questions as a CDC initiative that is non-regulatory, independent, non-federal, multidisciplinary working group, which is important because we're volunteers and we can't be held accountable to anybody. That's another nice thing to have. It was supposed to be a joke. All right, so we are accountable to the science and the methods and no conflicts of interest and really being transparent and publicly accountable. So the idea was to really work on the basis of what people already knew. So we took the ACE framework. We took the standards of how to look at the quality of individual studies from the USPSTF and the evidence-based practice centers. And then systematic evidence review, there it is, from the EPCs. So we really built our methods on the shoulders of people who've been doing it for a long time. And then I think we've added to that to some new modeling methods to address evidence gaps, which I'll mention again a little bit later, and then ultimately develop recommendations with linkage to the evidence. So that was how the working group did its work. Here is this kind of standard process. You select the topic which is around a genomic application to be evaluated. Step two sounds really mundane, but it's probably the most important part of the early process. In what setting would you actually use the genetic test? The interesting thing about genetics is that a test that is shown insufficient or has no clinical utility today in one clinical setting could be found to have really relevant clinical utility sometime in the future. So this issue about revisiting tests that we heard earlier I think is right on. We have to keep our minds open that you're not always insufficient. You could change over time. If you create an analytic framework with key questions, then you go out and find the existing data and determine this thing called net benefit, which just says that almost everything we do in medicine has both the potential to do good and to do bad, and we want to evaluate those equally and then compare them and make a decision, which relates to a recommendation. I was glad Sean put the word certainty up there. We have Russ Harris to blame or thank for that, and certainty is the flip side of certainty is the risk of being wrong, right? So if I have high certainty, I have a low risk of being wrong. So that's what we're trying to achieve. This is an analytic framework, and I used the example for SIP 450 and SSRI therapy, and I also understand that there's new evidence, which is always good. The overarching question would be that holy grail randomized controlled trial that someone actually showed that using SIP 450 testing improved health outcomes. Those are the studies we rarely have, and so rather than just say we can't say anything, we can build a chain of evidence through other questions, like does the test have analytic validity, does it have clinical validity, and does it have clinical utility? So that's how these questions are arranged, and we'll tell you that this is an interesting and key question area. I'm so thrilled to have the laboratorians, because the preventive services task force never asks that question. The whole issue of analytic validity has never come up, because we assume, Deborah, that it's valid, and that you've done your work and we can go on and be happy. So then the other question, so here for SIP 450, you know, we wanted to talk about how well do you predict the phenotype once you have the genotype, how does that translate to kind of clinical intermediate outcomes, and then hanging off here are these potential harms like, oh, I got it wrong, or I've made the wrong decision based on the test. So these are the key questions around analytic validity, clinical validity, and clinical utility, and really kind of lead us down the way to saying if we can answer all these questions and say that the harms are outweighed by the benefits, it becomes a positive recommendation. Those of you who actually read it, SIP 450 ended up being an insufficient, and we called it an insufficient negative, because we actually saw a disutility in use before we actually proved that it would be helpful, and that's the way EGAP put its work together. I will also tell you, though, that EGAP was put together not as the U.S. Preventive Services Task Force as an ongoing service to build a portfolio of recommendations. It was built as a proof of concept that could this methodology be used and applied in the genetic setting to create evidence-based recommendations? So I would say the answer to that is yes, and the answer is should there still be an EGAP is a well maybe, so we'll go into the rest of that. So here are the barriers and challenges. So having done this for about six years now, just as you heard, there are significant evidence gaps, and there are gaps at every point in the process. So analytic validity suffers from the laboratory-developed tests, proprietary interests, and I notice lab course here. You'd probably disagree, but some way that we get towards standards or that we get to validity is regulatory standards. And so without FDA saying all tests, Deborah, I was really happy to hear you say clinical utilities got to be in a CLIA lab, but we have to get to that standard for genetic testing, I think, to make sure we have a standard for analytic validity. Clinical validity is kind of interesting because we've talked about GWAS studies, and I think that that establishes an association, but recognize that the association doesn't translate to clinical utility. And in the December meeting and the CLIN Act project, I think these are really pretty exciting because at least we're setting standards for getting down to clinical validity to do tests that might be useful. That's really important. And then finally, in clinical utility, I'll just tell you the RCTs are rare, few and far in between, and it makes it difficult to get through the process. And then finally, this is a problem for almost all medicine that we tend to not think about harms, right? So when we develop new technologies, we're not thinking about the downside. I will tell you that the people who decided to irradiate thymus glands were not trying to do something wrong. They had a new technology, sooner or later they're going to stick a baby in the X-ray machine, they saw something they didn't understand that they could actually make go away with radiation, another better living through radiation story, and they did what they thought was right, okay? I'm not comparing whole genome sequencing to irradiating thymuses, but I want to make sure that we're always thinking about those examples when we, you know, employ new technologies without thinking about potential harms. So that's the genetic evidence gap, which a lot of people have talked about, including as recently as 2011. And I like Jim Evans, we need to try to push genetics into medicine, although I would like to see medicine accept genetics as something we can use. Other barriers and challenges, the sheer volume of the tests alone, right? So I just put up the single gene disorder tests over 2000 that we found through horizon scanning, more than 200 new omic tests since 2009, and then I think about 6 billion genes in the human genome, and it's like how are we ever going to deal with all that information? You take that volume and you point out the fact that there isn't, we have not figured out a quick and inexpensive way to do evidence review, synthesis, and translation, and it's not like for lack of trying. Because we've tried. We tried in the preventive services task force, we've tried in the community guide task force, we tried in EGAP and we tried in SAC-DINC, and it ends up that as soon as you try to cut a corner, you missed something, or you don't provide a compelling enough case for the recommendation. So I don't have an answer for this, so huge volume, time and resource intensity for getting from evidence to an outcome, and that's a problem. And then I'll change my slides, Deborah, to get away from the bad word, but genomic sequencing, you know, now we're looking at incidental mutations, nonsense mutations, and then just the sheer volume of the information. So I'll just tell you, these are challenges that we need to face as we move forward in genetics medicine. Then there are other sides, research and researcher interests, and this used to be just about getting promoted, but now it's about ties to additional sources of income, right, for researchers. And I will tell you that those interests may be malaligned with, not intentionally, but may be malaligned with the issue about the successful implementation of genetics into medicine. It's the same, and we want to be supportive of innovation, because if we don't innovate, and we don't do research, and we don't dream, and we don't look forward, we're not going to get better, and we're not going to figure out how to use what could be a critical tool in medicine. And then I have to pick on somebody else, so industry interests and direct-to-consumer advertising does not, in my opinion, work in our interests in trying to do this right purposefully, and in a way that's going to help the health of the people over time, that gap between the genetics medicine and the health of the public, which I hope that Toby's going to talk about, and we'll get to it sometime. We have some experience now with GWAS and the problem of small associations, and there have been some nice work about a lot of little associations don't translate to any real utility, so I think those are important things to keep in mind, and then this idea of the improvements at the margins of usual care. So I will point out that we were doing okay. I really want to make that point. We were doing okay with the old way of doing things. Mendelian genetics family history, that took us a long way, and it's not that we can't improve, but remember we want to improve on what we already have, not throw it out. So the GWAS studies and small associations and cardiology, for example, cardiogenetic profiling, said, well, you know, you don't want to just think about it as a test. You want to think about it. How does it add, right, how does it add to Framingham? Because Framingham works pretty good. So I think these are just kind of important concepts to keep in terms of chair challenges. We already heard about the ethical, privacy, and informed consent issues. What are we going to do with carrier status testing? If you test me as an infant, right? How am I going to have that information going forward? Who's going to store it? How is it going to follow me? Especially if it's in the state health system, which doesn't tell anybody anything because of privacy issues. This concept of selective return of results to individuals. So now I know all this information, and in Massachusetts, right? It's your information, but I don't know what it means. And so should I just give it all to you? And what about the ethics of giving you information that you don't really want? And how do we assess that and then do real informed consent around those issues? And then finally, can we ever get out from under the inhibiting and actually counterproductive protections in HIPAA to kind of purposefully move forward to do the population and longitudinal studies that we just have to do to translate the human genome into real public health? I don't have the answer to that, but HIPAA, you know, we're 10 years into this law that has done way more harm and cost way more money than its provided benefit. So what are the solutions? So, well, at least we should be able to decrease the noise. Deborah is saying, you guys spend too much time in here, but we should be able to say what not to look at, at least for now. Let's not spend a lot of time on implementing things for which we know there's insufficient evidence. But let's provide a clear research path to fill in the gaps. So, Sean, it was getting to your issue about letting that information out of the cell and having the receptors of the research community accept those and recognize that there is a path forward to fill in the gaps. Let's have recommendations for those actionable results where there's good evidence on clinical validity and maybe some, but we still need to do more work on clinical utility. I want to make a pitch for innovative study design approaches, such as retrospective collection of cohorts based on tissue samples, which is what, you know, Oncotype DX has fairly better studies on, but also to bring up the issue of modeling to fill in the gaps, right? I know that's a tough phrase for purists to hear, but, you know, even just, I think, recently, three days ago, SysNet projects on mammography, who to better screen within 40 to 49 came out, and it came out for modeling information, and it's information that really, I hope, will translate to better decision-making for women thinking about screening in that age interval. What other innovative design approaches are out there so that we can move the science forward while still minimizing uncertainty? And we've already heard about collaborative networks, laboratory, but also in clinical study, so all of these are things to kind of move forward as solutions. Tears and bends. So this is a classification systems that have links to needed research. Muin Khoury first proposed the three-tiered approach that they're tests that are ready for implementation. We should be moving on those, figuring out how to get them covered and coded and into clinical practice if they're going to result in improved health. And perhaps for tier two, where we're not quite sure about clinical utility, we could at least use informed decision-making or coverage with evidence development. And, you know, I don't always worry about that. Can you ever withdraw something that you started covering? That's what hopefully Blue Cross Blue Shield will tell us. And then discourage use if you just don't know. I mean, so the tiers were translated by Jim Evans and Dr. Berg into the binning project, which I think a lot of people know, and you know, how can we classify these in useful ways to say, boy, we ought to figure out what are medically actionable that we can actually do things with? Where do we need to do more work in clinical validity, and how can we not do this? And then it starts becoming more manageable, where we're looking at dozens of tests where we have clinical utility. And you know, quite a few in the area of clinical validity and then tons and tons and though we don't know yet. And then how do we have methods to move from bin one, bin three to bin one when necessary? I think EGAP could provide some information and utility in doing things like binning. So we're actually actively engaged in this activity. If there's not analytic validity, we need to turn to the new genome sequencing technology and get that tightened up. And there's good discussion at this table. If there's poor evidence for clinical validity, let's figure out how we get the path to more research. If clinical utility is poor, again, needs more research. And then let's get implemented the things that we know work so that we start realizing the benefits of the human genome. I was thinking that we actually have a natural example of the implementation of a new technology, like genome sequencing, to look at and that I hope we learn from. And are there any newborn screeners in the room? There's this thing called tandem mass spectroscopy. And so with one new technology, we suddenly gain the ability to screen for over 80 known metabolic mis-patterns in newborns based on blood spots taken from heel sticks. And it was like overnight. It was like, now you have the machine. So now you have the machine. How do you know which signals to turn on and act on and which ones not to? And the Secretary's Advisory Committee is doing exactly that work. Now, admittedly, they picked the Core 29 based on very little evidence and expert opinion. So don't do it that way. But now they've developed the methods and are looking at the new testing and the old testing through the evidence-based lens to kind of tighten up that question of benefit versus harm. And I'll leave you with my last drumbeat. So Sean, this is comparative effectiveness research recast in a different paradigm, which is really talking about marginal costs and marginal benefits. And marginal doesn't mean small. It means at the margin. So we should be looking at our genetic testing ability and the things that we want to bring to clinical medicine and always asking these questions. Does it improve compared to usual care? That's the marginal benefit. And is that improvement worth the costs and harms? That's the marginal harms. And if we apply this paradigm to every new testing strategy, not just for genetics, but for everything, we will create a system that has value in the health care system and hopefully gets control of some of the costs while maximizing as possible our impact on health. With that, I'd love to answer any questions. Oh, I did have to show Muin Query's new blog. Can we have our genome and eat it, too? Really make and then deploying one base pair at a time. Well, we may have to do it a little quicker than that. But I'll let you take down the reference and see Muin's latest musings. OK. Mark? Mark? Mark Retain, Chicago. So how does EGAP judge the quality of a publication? Great question. And definitely were weeds I never got down into, right? But the way that you look at the quality of a study is basic epidemiology. It's all the stuff that we taught you during study design. There's nothing magic in it. There's nothing new in it. I would say that the new spin on it, the grade, the grade system is put on it is probably, in my opinion, a little better than the EPC approach. But it's really talking about the threats to validity. So if you can get through the six criteria for threats to internal validity and generalizability of a study, cleanly, that's a good study. It's high quality. If you have some threats to internal validity, you become fair. And if you have fatal flaws to internal validity, you become poor. And it's pretty rote. Now, inherent in there is judgment. And this may be at the heart of your question. Because nothing is inherently good, right? And those criteria are applied with the lens of the person reading the study. So when you look at the Cochran assessment of mammography, they took the same seven studies and graded all of them as poor except two. And when the EPC, supporting the US Department of Services Task Force, looked at the same seven studies, they graded four of them as fair to good and only throughout three of them as poor. It's the same metrics. It was the lens of the people doing the judgments around that quality. And there's no really good way to pull that inherent variability in judgment out of the process. So the best you can do is be transparent about it. Well, I guess my concern is as someone who works in pharmacogenomics and who has read some of your assessments in the pharmacogenomics space is, you don't have anybody on your group that has ever done any pharmacogenomics studies. And you've made some very significant mistakes in your analysis, for example, lumping studies together that use a drug alone versus a drug in combination with other drugs or not adjusting for dose or things that people in pharmacogenomics know that you shouldn't do. And I'm concerned that this is something endorsed by the CDC without adequate representation in pharmacogenomics. And so there are some real issues out there. I'm not being at all critical of your non-pharmacogenomics studies, because that's outside of my expertise. So I answer that, because I think it's a really legitimate criticism. So one of the lessons learned, I think, from EGAP is that most of the methodologies that we pulled from were methods pulled from the prevention world. So all the other organizations I've talked to you about don't play in the clinical application world beyond screening. So big lessons that we learned halfway through was that we need to think about this. So we actually did add a pharmacogenetics expert, because we recognized exactly the inherent issues that you brought up. And we didn't have an oncologist on the group. People who literally were using these tests every day with a patient in the exam room, and so we added a clinical oncology specialist. So EGAP has tried to evolve to answer those issues because of the things that are pointed out. And remember, it was a proof-of-concept approach that that learning was important. I still, I will admit to you having questions about whether the EGAP system, the way we currently look at it, really is the right way to move beyond the prevention and screening world to the more clinical research world. Because the value system and the questions asked are dissimilar.