 Good afternoon. I couldn't help but notice that CDC is the last federal agency, so last and least, and not least. Okay, so I'm going to take you on a journey of three things this afternoon. One is making the case that public health has a lot to do with genomic medicine. I have made that case for the last 15 years within my own agency, and Lisa Lee here is smiling. That's exactly what I'm talking about. I'll tell you briefly about some of the experiences we've had over the last 15 years. I don't want to revisit the past, but I think the past can be a very good medium for food for thought because, A, I want to thank NHGRI for having this discussion. I think the time is now more ripe than ever before. I tried to initiate some of these collaborative processes when the time wasn't right. So at least you'll get that experience and tell you a little bit about our priorities for the next five years. So for those of you who don't know, public health is about efforts organized by society to make people be healthy. This is a couple of IOM reports. We have really three essential functions, and CDC is one of the main public health agencies, and primarily we deal with health departments around 50 states, but we deal a lot with healthcare. So assessment, policy, development, and assurance. What are these things? And let me tell you a little bit about CDC first. CDC is about improving health and preventing disease. Some of you know about CDC because of outbreaks and global security and so on and so forth, but there are two other priorities that you may or may not be aware of, and there are programs for everything from cancer to heart disease to diabetes to infectious, non-infectious maternal child health, essentially better preventing the leading causes of illness and disability. And if you think about those leading causes of illness, most of them, if not all, involve genetics. So at least it should be a foregone conclusion that the public health agency should be thinking about genetics. And the third priority, which is of intimate interest of the current CDC director, is strengthening the public health care collaboration space. So for the last four years, CDC has focused on so-called winnable battles. I don't want to go through them, but these are things that we believe progress can be made in implementation in programs, and you can decide on your own whether genetics or genomics has anything to do with these. So I want to tell you a little bit about two functions at CDC. One is assessment, which is surveillance, and we believe very strongly that what gets measured gets done. In other words, if we don't know what we have, we don't know who gets sick, we don't know who has access to care, we don't know what we need, then we cannot construct valuable programs. And I already mentioned the Healthy People 2020, I'll get to that in just a minute, but surveillance is a lot of what we do. And then a lot of what we do is implementing programs. This is from a recent forum that our director was in a global health forum. And all the research that you want cannot lead to improvement in health until you implement it into programs. So what have we done over the last 15 years? This is a sort of a historical 15-year trajectory, and I want to focus on three or four of these things because some of them endure, some of them have fallen by the wayside. So our office was formally formed in 1997, 1998, at a time where genetics and genomics was scattered all around CDC. Just like NIH, genetics is being done out of birth defects, it's being done a little bit out of cancer, but it wasn't done in a coordinated fashion. We held a number of national conferences bringing stakeholders together, including the last one in 2010. We developed a set of competencies for public health practitioners. That's about half a million people. We developed models, state programs, two rounds of funding. I'll mention a little bit of that. We, together with NHGRI, we launched the Family History Public Health Initiative. We put our own tool into a clinical trial. And then we launched EGAP. I'll tell you a little bit about EGAP because I think the need for EGAP hasn't diminished. And then sometime towards the end of the decade, I foolishly launched into a collaborative initiative just like this, bringing the partners together. We called it GAPnet. And I'll show you a couple of historical images that could be useful. And then in 2011, the bottom fell out and we lost most of our funding. So what happened over the last 15 years is that we kind of constructed a rational progression from discovery to practice. And I think Jeff already referred to the T1 to T4 pathway. And things don't stop at the bedside. We have the best tool right now, which is next generation sequence. And we need to do the evaluation, which may or may not involve clinical trials or efficacy. And then when somebody says, go do it, then we go do it and implement it. And implementation research and outcome research are very much part of what genomic medicine should be all about. The problem as we identified throughout the whole decade and the mid-2000s is that most generic research is upstream. There is very little T2 to T4 type research. This paper is a bit outdated. 2007, I'd love to repeat that analysis. Maybe the numbers have shifted a little bit in that direction. The reason why is because the funding is all upstream. I spend half of my time at NCI. And our cancer genetics research portfolio, this was in 2007. Analysis was repeated in 2010 of 1,000 plus cancer genetic and genomic research. The vast, vast majority is upstream research. Very little is spent. Less than 2% is the kind of research that would allow things to move into practice. So no matter how much we say that, we need to implement, implement, implement. There is an evaluation step that needs to occur before we can start implementing. Now, when things are ready for implementation, we don't know how to track them. Recently, I collaborated with VA in Boston. This is Julie Lynch and her crowd to try to look at administrative databases. And we're trying to work on that from the NCI perspective because most of the cancer applications and most of the genomic applications are in cancer, as other speakers put out. So you look at these and we were able to put together sort of these administrative databases. We don't have information on individual people, but more on health systems and hospitals. And you can see the differential implementation in the EGFR testing for lung cancer, which is sort of a genomic test that helps focus this therapy for lung cancer. So one of the things we put together in the mid 2000 is put together sort of the equivalent of the US preventive services task force that ARC has talked about this morning, which is figuring out a systematic approach to figure out what's ready, what's not, so that we can go about implementing it. That experiment is now in its eighth year and a number of both methodological work as well as substantive recommendations have come out. So I think what has happened because of the EGAP initiative is that we've uncovered a lot of gaps in the research and that research could be funded to, I mean, the gaps could be funded with additional research so that we can move things into practice. Now towards the end of the 2000, I put together this working group, very much some of the people around the room actually were part of this inaugural discussions where we had on GapNet and some of you may remember, I saw Mark Williams in that picture and so I saw Jeff Ginsburg. Ron, I don't know if you were there, you were there. I think you were there too, you were there too. So the idea is to put together the stakeholders to figure out how to move things into practice and the notion is to put the stakeholders together. It's a stakeholder driven initiative to synthesize the knowledge, disseminate it, use it as a basis for evidence based guidelines and recommendation, move it into programs and then do the research that fills the gap. It's a very simple idea, the time wasn't right for it, but the idea behind it still stands that as the upstream research keeps pushing down more and more cool tools for use in practice, we need to synthesize that knowledge, move it into guidelines implemented in programs and then do the research that fill in the gaps. So that's all history and now to the present. So what has happened since 2011 when our office lost most of the funding? Is that when you have this kind of a near death experience, you kind of wake up and say, okay, there is life after all. So we did a number of consultations that led to two reports. I'm happy to send them to you if you want them. One is a priority report which was published in 2011. And then the second one was published last year as an action report for what can we do now to save lives. And before I get too far, I just want to mention this because this is a placeholder. This is something that CDC is very much interested in, which is the use of pathogen genomics for acute outbreak investigations. This is an initiative that our director has been very enamored about. It's called AMD and that doesn't stand for age-related macular degeneration, but advanced molecular detection. And it's very contemporary and it's going on as we speak. So what has happened over the last couple of years, and I'd like invite everyone to look at this website, is that we've taken sort of the bull by the horn and said, okay, EGAP cannot do it all alone. US Preventive Services cannot do it all alone. Let's see what's out there in terms of what we call that tier one, tier two, and tier three. And the tier one is sort of when somebody makes an evidence-based recommendation on the basis of a systematic review of the evidence. And tier three, when we don't know what's going on, tier two is where sort of that wiggle room is when you have enough information on clinical validity but promising information on clinical utility. This is a busy slide, but it captures essentially where we are with this. And we've identified three or four applications. Of course, newborn screening subject we don't talk about. But it's the ultimate tier one application. It's in all four million births are screened every year, four million babies to find 31 or 32 mostly generic conditions. So you can see where you are on the continuum. And I think you have to start with what you know today in order to drive both research and practice. That's why knowledge integration and knowledge synthesis is something that we really subscribe to. I want to describe briefly these three applications. We've already heard about Angelina Jolie, but there are two others that are equally tier one in nature because they've been recommended by evidence-based panels. And the sum of these results, basically, these are not genomic things. They are single gene disorders from the old generics. There's not much from the new genomics that has made it all the way to recommendations with the exception of cancer. And what's really unique about these three conditions is there are about two million people already affected in the US. And from the little bit of evidence that we know, the vast, vast majority of them don't know they have it. So there are lives to be saved right now. And that's sort of the action items that we've been focusing on over the last few years. This is one of many, many papers that recently have appeared that is underutilization of BRCA. So US Preventive Service Task Force made the recommendation that about 5% or 6% of women with a certain type of family history or ethnicity be referred for generic counseling. What's happening in the real world is much less than that. And if you look at this is the same with sort of the analysis in Michigan where in the last three years they used education to educate health care plans in Michigan to adopt the US Preventive Services Task Force recommendation. And this was before the Affordable Care Act. So just from public health educating health care plans, more and more adoption and coverage of BRCA generic services were done in Michigan as a sort of a public health issue. And this is the EGAP recommendation. It's one of the few positive recommendations that came out of the EGAP Working Group. We heard and now I mentioned this earlier. But you look at the real world, what's going on in practice. Of course, implementation always lags behind. Academic centers always implement first. So the disparities, the equity that you heard about from Carolyn Clancy earlier this morning is very much on our mind in public health. Because we want tier one applications to be implemented to save lives for everyone. And tier three and tier two to be studied a little bit more. So with a little bit of funding and mostly networking, we've established a sort of a Lynch syndrome screening network that's very much alive and well and been trying to integrate Lynch screening in I think more than 90 academic centers as well as community centers. Some of you are even involved in that network. So I won't dwell too much on the Healthy People 2020 objectives, but a few years ago a few of us, including HRQ and IH and NCI, put together, we tried to look for these Healthy People 2020 objectives. And I think I was on record saying I got depressed because of all the genomic stuff that's going out there, we only could find two things that we can track over the next five, 10 years. And there are very specific and I can tell us more about what goes into the Healthy People 2020. There are many, many indicators and we need to track them like physical activity and diet and screening. And you have to have data sets that can measure progress. And for much of genomics, we don't have those data sets. The CPT codes may change this. You have to have evidence-based recommendations to track that. So we were lucky that we're able to add BRCA and Lynch as a developmental objective. And then we were working with states and working with states to try to begin to measure, implement, and do what states do, which is working with their health care sector. And we worked with Oregon and Michigan for a while until basically the Division of Cancer at CDC. And you heard Carolyn mention this morning about how the cancer registries are now being used to track progress and genomics. And each state has a cancer registry. So all 50 states, and by definition, are able to register and do surveillance on all new cases of cancer, whether it's colon cancer or BRCA and others. And just to supplement this, I mean, these registries are primarily not made for research. There are other registries, the C registries at NCI, which cover less than the total, I mean, all the states. But at the same time, they are more ripe and more user-friendly for research. And we're trying to work very hard within NCI to use the C registries to track genomic progress. So I'd like to leave you with one thing. And then I want to talk about whole genome sequencing. I can't help it. I have to talk about whole genome sequencing. So what our office has been doing over the last few months and is to try to develop toolkits for implementation, very sort of down-to-earth for state health departments to work with health care providers for implementation of Tier 1. We will have a map of such activities that people can click on the website. We are developing a set of population-level surveillance indicators, not only for these two, but for other things. And we're trying to refine our Tier 1 to Tier 3 evidence classification. I really invite all of you to be part of that process, because this is an ongoing movement. And the other two will be... So I want to mention two minutes, if I have time, Terry, two minutes, whole genome sequencing. I think it's both our course, but at the same time, it maybe provides the best opportunity to move forward, really, and I mean it. And I think a systematic approach to whole genome sequencing is being adopted, and NHGRI is taking the lead in trying to move it into practice. And my only bias is that whole genome sequencing for the diagnosis of rare, mysterious diseases is not going to cut it from a public health perspective. We have to reach a point where whole genome sequencing could be beneficial to larger and larger segments of the population. If we can make that evidentiary case, then a lot of these issues around return of results can be dealt with. And that's why a systematic approach, like the one proposed by Jonathan Berg a couple of years ago, is an order, and now it's been implemented as we speak by NHGRI recent initiatives. I think the ACMG recent recommendations for reporting incidental findings are, although they've generated a lot of angst, and this is just three or four of the many, many blogs that have been including our own, the CDC one is on top, about that anxiety that's coming along with the fact that ACMG has recommended that 57 genes, was it 57 less, have to be reported back to participants of whole genome sequencing or next generation sequencing regardless of age, because these were, they could provide benefits to people. But at the same time, ACMG said, well, we don't have evidence of clinical utility for most of them. So I think there was a problem. It's sort of one negating the other. That's why I think the ethical community has been up in arms about this. Now EGAP and Katrina Goddard and some of the members of the EGAP working group have recently proposed the method of binning, which I sincerely hope it could be taken up to industrial scale over the next few years. I have every reason to suspect that this might be the case because of NHGRI leadership in this. If we are able to develop the methods for classifying and returning incidental results or incidental findings from whole genome sequencing, that would really make the field move forward. I want to end up with this last slide because I know Jim Evans always thinks outside the box and maybe that's why I like him and we work very well together and he recently made the case along with a few other people that, look, I mean, the whole genome sequencing, I mean, we screen newborns, don't we? I mean, we screen 4 million babies every year to find 10,000 that require urgent care to reduce morbidity and mortality. Now, couldn't we study the idea that if we provide a whole genome sequence to everyone that care will benefit? And that's a research question. It's an, you know, and maybe the time is not right because, A, it's still expensive, it's still not analytically valid, but sooner or later we're going to confront that question. If we're going to have our whole genome sequence once and be used throughout our lifetime for, you know, drug, et cetera, we need an evidentiary foundation for that. And the time is rapidly approaching where this public health question has to be asked. If we all have to have our genome, we have to have an evidentiary foundation for it. We can't just rush to implementation. We have to study it. And the reason why they're suggesting it is not because of SNPs or anything, is because they think there is about half a percent or 1% of the population that have rare genetic diseases that are not being picked up by the healthcare system for which there are interventions at work. So half a percent of 300 million people, that's a lot of people, and that trumps newborn screening any day of the week if we can get to that evidentiary foundation to support that. So with that, I'm going to close and I'm really welcome the dialogue around the federal table here and would love to participate in whatever next steps are planned. Thank you. Great, great, thank you, Moine. Maybe before we take questions, I might ask Les B. Sucker if he wanted to clarify what the ACMG recommendations actually said about those 50, or what his view is of what they said. So Les was one of the co-authors of these recommendations that came out from the College of Genetics for those who are not familiar with it. Actually, I think we could do that. I have 30 minutes tomorrow. So why don't we just hold on Moine's characterization of what the recommendations say, just hold your thoughts about that and we'll talk about it tomorrow. If you want to correct anything that I said factually wrong, just say it in five seconds because many people won't be here tomorrow. Okay. Can I just follow up on, we screen newborns, don't we? So what's the current cost for screening for a typical newborn? See, I don't have those numbers and I don't know if there is anyone from the newborn screening community here. The problem with newborn screening, not the problem, the newborn screening grew organically over time and the first time it was done in the 1960s, it wasn't done with an evidentiary foundation, it was like after PKU. And then the incremental cost of adding more and more to the panel became less and less. So it started with PKU, hypothyroidism and then over time more things were added and then it became a statewide occurrence and now there is an advisory committee that advises HHS on sort of what should go into the panel and I think they just added Pompeis disease last week or the week before. So I think we need something probably, I mean, see, you know, newborn screening is a public health program whereas genomic medicine is implemented by the blues, by Aetna, by United Healthcare, by the VA, but we need something a little bit more rational for how we go about implementing it. I'm not necessarily calling for an advisory committee, but something like an EGAP-like process, I would call it EGAP 2.0 or 3.0, but, you know, short of that, everybody is going to implement it any way they want without the required evidentiary foundation to do it. And as long as I wanna say something here, I was surprised with Joe Selby saying that he didn't get too many genomic applications. I mean, I've been advertising PCORI, PCORI, PCORI and it's patient-centered, patient-centered. So I don't know why people are not applying and so thank you, Mark, for announcing that you are applying. I would encourage as many people to apply to get PCORI funds as possible. So if I could just respond, because I think again, we do a lousy job in our healthcare system of actually understanding costs. And so, you know, much as we talk about the $1,000 genome and what that really reflects is the cost of the reagents and doing the test. In the same sense, newborn screening is the same sort of thing. And we have 50-plus different ways of actually paying for that, where some states really have it as a public health program where up through confirmation, it's fully funded by a state program whereas others are a mixed model. And so for analyte screening, generally a few hundred dollars is the cost that is bandied about for the purposes of budgeting. But when you add things in, like hearing screening or congenital heart disease screening, which now involves audiologic evaluations and echocardiograms and pulse oximeters and these sorts of things, that's a very different modality. But I think where there is similarity is that we think about the tandem mass spec, which is, you know, was again implemented because of economies of scale. For the most part, and the fact that we could detect things that previously were not detected by newborn screening. But what it's done is it allows us to look at a lot more disorders at essentially a zero marginal cost, at least for the analysis. Now the follow-up is a different issue. And that's, again, I think what a lot of us are worried about is that if we blow everything out through that it's not gonna be the cost of the testing, but it's gonna be the cost of the follow-up that's gonna be the problem. I think that the difference though is the persistence of the information. And I would push back a bit with something that you said, although it may have just been the way you said it, not what you truly believe. But I would argue that if you can make the case to do a whole genome for, say, a child that has a rare undiagnosed disorder, that that's a good investment if we keep all of the information and reuse it over time. So we're essentially amortizing the cost of that genome over time. So whatever the indication that first gets you the genome, if we only use it for that indication, it'll never be cost effective. But if we capture that information and reuse it over time, that's when we really begin to see some of the economies of scale. And I agree completely with you. I don't know what I said that made you think this way. What I meant to say is that if we're going to introduce the whole genome sequence, introducing it only for the diagnosis of rare mysterious diseases is not going to reach a substantial fraction of the population. And when you do, which I agree that the indication, I mean, the yield may be increasing as time goes by. Yes, I agree with you. You can still use that information, whether it's 57 genes or no matter what that number is. But from a population perspective, I think introducing the genome on a larger scale is what I'm after, but it has to have a strong evidentiary foundation. And I think the case that Jim Evans is making is that the strong evidentiary foundation initially to do the initial testing is not what people think it is, which is polygenic models or SNPs and GWAS, but more of a newborn screening-like model is let's find the Lynch syndromes and the familial hypercholesterolemia because that's about 1% to 1% of the population that currently are not being served by healthcare. And then once you have that information, then yes, all that, quote, incidental findings will have to be rolled out throughout a lifetime with a lot of studies that can be done with it. Now, I don't know how soon will this happen, but I think we need to start thinking about it. We need to start discussing it. We need to start doing research to see its added value. But I'm assuming that's what the whole purpose of the whole genome project was to begin with is to have that genome sequence available for healthcare and disease prevention. Thank you, Jeff. Yeah, I'd like to take advantage, Muin, of your 15 years of experience in building networks and tools for advancing genomics into public health and ask you for some advice about what we're gonna do in the next, after the break, in terms of trying to potentially organize the federal agencies into a cohesive and effective effort. You mentioned, for example, around genomics. Sort of around genomics. You mentioned, for example, the failure of GapNet, and I'd really be curious as to what advice you have and the things that we should be paying attention to and what are some of the lessons learned in your experience. See, I think the partners were very enthusiastic. At least you can see from the picture how happy they were. I think two things happened. We lost the funding shortly after these big events. So I had no more fuel to add to the fire. So that's case closed. But more importantly, and I think some of you were involved in these discussions, there was no overarching theme that people were pushing forward. And I think what happened in the 15 years period from 1998 to 2013, remember, 1998, we only had candidate genes. The Human Genome Project was only almost done. And we went technologically from very little to an explosion of tools and applications over just the last couple of years. And that will continue. So I think the purpose of a network or a network of networks and stakeholders is to try to coalesce around very specific goals that could be studied, could be informed research, could inform practice. And I mean, my advice is very simple. The goals should be as quickly as possible to implement tier one applications. These are the applications we know that can save lives now. And while we study whole genome sequencing or next generation sequencing in a sort of aggressive way, and third, never forget about family history because family history will always drive number one and number two anyway. And I think if the partners can coalesce around a few things that can make sense from a healthcare impact or disease prevention impact, I think it's more likely to succeed. Thank you. Let me just see if there are others before Mark speaks up. Is there a hand in the back? Do I see one now? Okay, go ahead. Yeah, I just wanted to follow up on that. I would add one more thing to what you had said, Muin. And I think we saw some of this in the EGAP stakeholders group, which wasn't on the list for probably pretty good reasons. But I think that one of the issues, and this is an issue that affects NHGRI activities and that is that we, for pragmatic reasons, are heavily reliant on volunteerism. And the reality is that there's just so much capacity that volunteers can bring to an effort unless you really have a strong value proposition on the back end, which is if I participate in this on the back end, I'm going to really get a lot of value out of it. And I think that's where EGAP stakeholders group was trying to go was to define that value proposition for a variety of reasons, weren't successfully able to do that. But if we can actually define the value proposition as Caroline was talking about earlier where we can really say, you know what, if we can pull this off, then this is how I as a payer or I as a healthcare system or I as a doctor or I as a patient can really benefit from this. Now you've added impetus to say, I'm willing to pitch in and actually give freely of resources to really make sure that we accomplish this. And so in some sense I think that's the end goal and the public health perspective is a very strong perspective that I think is a very high value proposition, but only for a subset of the stakeholders that are engaged in that, even though when we all really sit down in the cold hard light of day, the reality is we're all practicing public health or at least we should be. We should be thinking about it from that perspective, not just the person that's sitting across the table from us. Can I just add one more thing to Mark here? The world of public health is still very skeptical about genomics and for good reason. I mean, that's one reason why I lost most of the funding is because there are so many things we know what to do and they're not being done. So show me how genome sequence is going to improve health and reduce healthcare disparities right now. So I mean in the scheme of comparative effectiveness research, whether it's clinical or public health, there is a lot that's being done. So I think that that resistance that I've seen personally in my own backyard is maybe also reflected in the healthcare settings, primary care, et cetera, because there's many people are not convinced of the added value. So as long as NIH and some of the stakeholders around the table are willing to invest the value proposition to show that in fact, we can improve health outcomes. We can actually diagnose more. We can lead to better screening, recommendation, personalization of drugs, et cetera. But that has to be supported by research. We cannot skip T2 to go to T3. We have to do that T2 and we have to get evidence-based panels, whether they're professional societies or USPSTF or EGAP to actually make pronouncements after they look at the data that actually it does make a difference. And I think there will be more adoption on a larger scale. And that's my hypothesis. Part of my takeaway from participating in EGAP was that the process of evaluating the evidence took so very long, predominantly because the evidence was so bad in general around genetic testing. And so I wonder if there wouldn't be a better approach if we could actually design the studies that are needed to answer the utility questions and then do that research on a population basis, not necessarily through randomized control clinical trials because that's not the reality of clinical care and practice out there in Vermont or wherever. And so it was a bit frustrating participating in that process. And I don't know how we get to the core questions that really would have a great impact on patient care and do those because no matter who you talk to, they have their particular disease of interest or they're not really looking at the most critical disease processes or the most costly, sorry, but cost is a big factor, even if PCORI won't fund it, or the most costly disease processes in the United States and design the research programs on a national scale using real clinical care to distinguish what we should be doing. So just to answer your question, EGAP went through an evolution. I think Mark has witnessed it because he was on the stakeholder group and now sits on the EGAP working group. I think the last couple of years, the first two years where they're trying to figure out what to do, they're developing new methods. Each systematic review took a year, although we kind of knew what the answer was before you did it, so why do it? So they were trying to refine and define the processes. The last couple of years, however, they published new methodologies. David Vinstra's paper last year, their output has quadrupled. So they have four or five new things and give Mark some credit here, but it has gone through an evolution over the last two years. It's too bad because the stakeholders kind of disappeared at the crucial moment where we needed them. And the reason why they disappeared is because I couldn't fund them anymore. So I think what the process will work and it is an iterative. And I think if it's adapted and adopted by all of the agencies together. And by the way, the EGAP working group has a steering committee and the steering committee has many of the people around this room. I mean FDA, VA, CMS is on the steering committee. So you don't have to call it EGAP, call it something else. I don't care, I mean I don't necessarily have that sort of sense of ownership, but you need something coordinated where all the stakeholders buy into it and including strong federal support from both NIH and the rest of the agencies. But how do you drive it so that the evidence is being generated that is needed by the evidence review groups? Well that has gone through an evolution and I think having PCORI exist right now is the perfect time because PCORI takes into account what the stakeholders want and what the patients want and the committees that review sort of the evidence have to at the outset. See HRQ has also evolved at US Preventive Services Task Force and I think your notion of EGAP is still probably frozen in time from 2007 or 2008. Right but it's still PI driven questions that are being asked as opposed to some kind of national assessment that these are the research questions that need to be answered and let's get the answers to them. So there was a robust horizon scanning which we still do which is try to define what are the important topics from healthcare and public health and then those are brought to the table and then one has to weigh, I mean should we take topic A versus B and that's where the stakeholders come in. Okay Mark. I wanna come back to the idea of outcomes just as an aside I hate to tell you this Joe but your website has underwent a malware attack and so people that have been trying to access the PCORI website have not been able to but I know what I, Right at your own risk, at your own peril so there's an outcome. But we've been throwing around the term outcomes but the sense I get from the discussion is that we have a very narrow view of what the outcomes are. So when we talk about warfarin we're talking about bleeding or clotting and the reality is that that's the 2% that Dan's talking about but the study that we did at Intermountain showed that in 100% of the patients who had genomically informed dosing they had on average too fewer INR determinations. Now from a patient centered perspective I think that makes a difference because it's two times they don't have to come back into the doctor's office to get their INR drawn and disrupt their life and take time off of work or arrange for transportation or whatever. And so I think when we talk about outcomes we have to really think very broadly and one of the things I would put on the table as a takeaway that we could use this broad group to evaluate is what is really the true range of outcomes that we could consider reasonably to look at. So it's not just a definition of this particular medical outcome is what's going to make or break warfarin pharmacogenomics as a bad example. Oh and by the way, two fewer INRs, don't tell Joe but it costs less. Would you like to respond? I'd love to respond to this. Several things that have been said. No, first of all, one of our criteria for funding something in the first place is that it costs a lot. So we are very specifically interested in high cost propositions. So that's one point. The second point is you very well may and are invited to measure resources. Just don't translate them into dollars. So that's the second thing. But the third point in response to what Dr. Leonard said was that it sounds to me as though you are calling for a framework that could be an evaluation framework that could be used from question to question. And that would be something that PCORI would welcome. And from research project. From research project, from research question to research question, right, from proposal to proposal as we're reviewing them. So we would be, I think, interested in and could even conceivably be interested in contributing to a process that developed a framework like that which we could then invite applicants to look to and shape their proposals in response to. It seems. Can I comment that I'm thrown smack into the middle with my recent move into a CMS shared savings program project that's going on in the state of Vermont. And it would also potentially direct where the work should be done to reduce costs, have better outcomes in the CMS shared savings program that is lots of money being spent across the country. So it not only would, and I would think that those places that are doing the shared savings programs would be ripe for PCORI funding. But that connection doesn't seem to have been made. At least in my large state, every time I mentioned PCORI funding, they go, oh, that would be a good idea. But it's like it's not connected. And I don't know if that's everywhere, but it's frustrating. Additional comments on that. One thing I would like to raise is I think we've heard repeatedly throughout the day about the evidentiary gaps, and Deborah again alluded to that very strongly, has everyone. And I think one of the opportunities we have here is that we have representatives of very large integrated medical healthcare systems, the VA being probably the largest, but the DOD healthcare systems as well. And recognizing that there are barriers to getting access to patient data to us, there aren't barriers to you. And so it might be possible for us to sort of define a research project that could be done in each of these kinds of systems, and y'all go and do it, bring the data back in some way that we can then compare, at least use standard methods, and do it in a systematic way. And that would be something that I think we would be very interested in. PCORI would probably be quite interested in, and it would benefit everybody. So perhaps one of the things we could talk about it is is there a way to engage the military medical systems, the VA. CMS has tons of data that again, can be challenging to access, but there are people who can do that. And what could we do to do some evidence generation just within the federal systems? So with that, we probably ought to go ahead and take our break. We will try to figure out what we're going to say after the break, but we're looking forward to an engaged discussion of how we can move forward. Thank you. We'll be back at 325.