 It's a pleasure to be here and participate in this really fascinating discussion. Our panel title, as you see, is Making Biomedical Science Nimble. It's The Patient's Stupid. Preparing for this panel put me in mind of some of the experiences that I had as a graduate student in the biomedical sciences in something called the Medical Scientist Training Program, or MSTP. Just curious how many people know what that is. It's basically a federally funded program to support the training of physician scientists, so people with both a medical degree and a PhD. And I envisioned a career in medical investigations and research that involved people, which is why this program attracted me. But the orthodoxy at the time was expressed by the person who was then the director of the MSTP program at my university. He insisted that all the students in the program cut their teeth on basic biomedical sciences, basically going into the lab where we could control most of the variables and conduct clean and elegant basic investigations. Well, that, he said, was the foundation for any type of research that we could hope to do in our careers, even on much more complex systems like whole people or heaven forbid, health care institutions or systems. This latter type of research was looked down upon and considered soft and error-ridden. So off I marched into the lab prepared to exercise the practice of clean and skeptical scientific inquiry. But what I saw in some, and certainly not all labs, was that the pressure to attract competitive research grants, establish a name, advance an academia, led to practices that weren't very scientific. It was like repeating experiments until the results the researcher knew were right, were found. Not publishing negative or null hypothesis results, the things that basically we tried and didn't work as we had expected. And as Dan pointed out in his earlier introduction to the conference, there has been more and more in recent years a recognition that these types of biases or sloppy or inadequately robust results from basic bioscience have not been appreciated enough by the folks who are trying to translate these promising discoveries or what seem like promising discoveries in the lab into treatments or methods to fight disease in actual people or in populations. And because that hasn't been appreciated enough, there have been recent articles such as one in the prestigious journal Nature by C. Glenn Begley and Lee Alice who raised this issue in the context of cancer research. And basically they painted a picture of dozens of clinical studies and patients who were needlessly volunteering to participate in research in studies that really had no chance of working because the basic science on which it was based was not replicable, could never be repeated, and in fact was spurious, wasn't of value. And so they've called for raising the quality of preclinical research. Well, our panelists today have recognized that the problems and the opportunities to fix them go even deeper than that. They challenged the orthodoxy that was expressed by my old MSTP director and the value of our medical knowledge enterprise, since we're using that phrase today, places the value that that enterprise has traditionally placed on different types of investigations. And their work it seems follows from the belief that some of the most intractable problems in medicine can't be addressed solely by research on lab rats, cells or molecules on a petri dish. And in fact, often we do understand the basis of the disease or we may even have the drugs or the methods that should be effective at treating it. But we have as yet failed to translate that into effectiveness for individuals in a population. So these are complex problems where the biochemical mechanism of the problem is but one variable. And what we will discuss today are the possibilities that it's not just different tools that we need, but different skills and approaches that don't necessarily flow from fluency in how to do the basic science. So the work of people like our panelists is really part of a movement. And I'm happy to say I recently went back to speak at my MSTP program, annual retreat, and in fact now a much wider range of student investigation is being encouraged, including PhD research on health economics and a lot of translational and collaborative research. So it may be that these types of research are in fact gaining currency and respect even in the top institutions. So I'd like to begin with Ron DePres, who's the director of the Center for Community and Public Health at the University of New England in Maine. And also the director of UNE's Center for Health Policy Planning and Research. His work focuses in particular on how to deliver healthcare to people with chronic conditions, the ones people live with for a long time like diabetes and hypertension, heart disease, asthma, COPD, and prevention strategies. And Professor DePres, in January, a group of smart people from a variety of clinical and non-clinical backgrounds, including New America's Shannon Brownlee, released a report under the auspices of the Entrepreneurship Foundation, the Kauffman Foundation, on improving productivity and quality in healthcare. And I want to just quote from them. They said, a recurring theme among scientists in the discussion was that medical biology is the hardest of sciences. Quote, if only it were as easy as rocket science. One task force member bemoaned. Biology is hard because life is complex and adaptive. Laboratory biomedicine is harder still because of the incalculable, incalculable variety of pathologies and treatments and the even larger numbers of ways they can interact with the body. Clinical medicine is hardest of all because one must deal with human beings, the most complex and unpredictable of creatures, end quote. So do you agree? And if so, how do we begin to deal with this complexity in research? And not only in clinical medicine, but also in what you focus on, which adds even additional layers of complexity, implementation research. That's a good question, Sherry. And I would say that you're dealing with applying applied research when you're dealing with clinical medicine, you're not going to invent some. But I want to go back and talk about one comment on clinical research. This is the placebo effect. You've all been exposed, I think, recently to a lot of studies that are looking at the placebo effect. And basically what it tells us is that that has a real effect on people. People get better, even though they haven't been taking the medication. I've been taking this, so it's really kind of a backdrop to understanding science that we have this thing called regression towards the mean, which has governed my life in the sense that every time we see something, we study it more, it comes back to a normal differentiation. Not to say that we haven't had great strides in clinical research, because we have. But the problem, as you mentioned before, is how do we take these great strides and apply them to populations? And populations that aren't anywhere near alike in many instances, whether they live in rural communities, whether they live in urban communities, whether they live in suburbia, they're very different populations. And it's hard to get these breakthroughs in science, whether it be taking care of diabetes, taking care of hypertension, how to take care of COPD, lung disease, how to get them to be effective in rural and in populations. Because now you're dealing not only with the application, but you're dealing with the economics, whether they have access to care, whether they have transportation to services, whether they have the insurance to pay for medications, whether they will take the medications because it gives them a side effect, which they don't like. 60% of the population of not only the US, but other industrial societies, they don't adhere. Use the term of adherence to the medications or to the prescriptions of their providers for a host of reasons. So we have failed in that regard, it seems to me. And the second way to look at it, we're talking now about clinical transformation. That's the buzzword in Washington at CMS. It's a really good word. How do we take things and transform this health care system to be more popular, people-oriented, to be more effective? And we have a lot of good research on comparative research going on, etc., etc., about how we can do better treatments. But how do we take that and transform the system? And that is what we don't know how to do, I would say. We don't know how to do it very well. Health delivery systems think about how do we make more money out of the system? Or how do we make this profitable so that we can continue going? But that's not the answer. The answer is how do we get populations to change their behavior? To change their behavior and adhere to whether or not they're taking medications or changing their lifestyles or changing whatever they did to become healthier because the facts of the matter is we aren't growing at all more healthier at the pace at which we're developing new treatment method, treatment remedies, at the pace at which we're developing new surgical procedures. We aren't growing the improvement. We've improved in this country and other countries, mostly because of very simple things. Nutrition, lifespan is expanded because of better care to moms. It hasn't grown because we've done a better job, necessarily, of taking care of people with hypertension, diabetes, COPD, other kinds of current conditions. And how do you translate that? So that's basically, I love that comment about the complexity. It's not about rocket science because I think what this calls for is a sense of rocket science. And how do we transform what we know now into actual practice among people as well as providers? Now I want to go back to one thing. Back in the 80s, I was part of an NHLBI 10-year study, I'm sorry, five-year study on hypertension. And we knew then how to take care of hypertension. And the question was, how do we get it to people to do it? And the study came out of a set aside by an Ohio congressman who was head of one of the subcommittees. And people poo-pooed it. But actually, the state of Maine, among other states, Connecticut, there were 10 states, Michigan, California, really took this money and was able to change the way for at least a period of time people practiced. And people actually lowered their diastolic and systolic blood pressure as a result of interventions in health care delivery system. I'll stop there. Just real quick, actually, there's a great quote by an online activist, patient-e-patient Dave. He always likes to talk about, you're 50% at here. And patients are 50% at here. And actually, doctors follow standards of care, known standards of care, only about 50% of the time, too. So in reality, we're actually practicing medicine about 25% right now. So we've got a long way to go in terms of actually getting to even what we know is right. And we can talk about the unknown, which is probably what we're going to get into a little bit today. Thank you, yes. So we're going to talk next about a disease I know you know a lot about, amyotropic lateral sclerosis, or ALS, or as it's commonly known, Lou Gehrig's disease. It's a devastating illness. I'm sure most of you are familiar with it. Patients lose the ability to move their hands, arms, legs, eventually the muscles that govern breathing, while their minds remained perfectly intact. And for a couple of decades, mice with multiple copies of a faulty gene, superoxide dismutase 1, served as a model for the disease. And scientists relied on them to test promising treatments. But since then, close to a dozen drugs that were reported to increase the lifespan of these mice have failed to benefit ALS patients, leading researchers to question the very premise that these mice were an appropriate model of the disease. Similar problems have arisen with models of other so-called neurodegenerative diseases like Alzheimer's. Our other panelist who you just heard from, Ben Haywood, is the president and director of Patients Like Me, a company that provides a platform for people to share their health experiences with other patients, with researchers, or other organizations and companies that focus on these conditions. And Haywood co-founded this company in 2004, inspired by his brother Steven's battle with the disease, ALS. So Ben, your company contributed to ALS research a few years ago in the study of the drug lithium. Could you please describe that study for us and the larger concept of how our current information and communications landscape could contribute to the biomedical knowledge enterprise? Yeah, no, I mean, I'm going to go back even a little further to tie in this sort of pre-clinical area as well, just in terms of my family's journey. So my brother was diagnosed back in 1999, and actually our family did two things. One is start patients like me, but my other brother started a nonprofit biotech doing hardcore in vivo drug discovery. And all of us are actually mechanical engineers, but manufacturing engineers by training. And I think Jamie actually spent $70 million doing hardcore in vivo drug discovery, running about 30,000 SOD1 mice through his lab, and repeated every known study ever published in the 20 years of that disease. And there's a lot of questioning of the mouse model. And what he found is that almost every known study was in the noise floor of the mouse model, so that scientists were not taking into consideration the noise of the individual, of the mice population. And I think what that really challenged for him is that we really don't have a baseline understanding of disease and illness, whether it's in the mouse model or in the human model to use the same language. And so we started patients like me really to begin to actually understand and measure and value, and measure health in particular chronic illness in a much more patient-focused way so that the end user, the patient of what we're trying to solve here in healthcare is actually determining the value of both the products and services. So what we built is a platform for patients to come and actually share their experiences, but it's not just an anecdotal site. It's taking patient-reported outcomes, validated instruments, and having patients share them in an open environment. And that has allowed us to do many things. And you mentioned the lithium discussion. So that was basically a real-time comparative effect in this observational study that occurred on our platform. It wasn't designed a priori. It was a result of being on the platform. Basically what happened was there was a published study in PNAS, a good publication in Europe that showed that lithium dramatically slowed ALS in a small phase one clinical trial. So about 16 patients on drugs. Obviously in a universally fatal disease, that was a big deal. So 10% of the patients on our platform went on that drug, obviously with their doctors because it's a prescribed medication. We now actually have about 20% of the ALS patients just to give you a sense of scale of the size of the site for that disease and about 5% of MS patients. So about 10% went on drug. And about nine months later, we were able to show that in the real world using our 3,000 controls of ALS patients plus the 300 or 400 plus patients who went on drug, that there was no effect of lithium. And so while when we first initially published our findings via a poster, three studies subsequently started NIH funded studies in the $20 plus million range. And they all stopped within six months of futility basically validating the results we showed basically in near real time. That our work was actually eventually published in Nature Biotech going back to this in the medical industrial complex. One of the things about a new methodology like ours is that we have to actually really go back and utilize publishing in other ways to validate it's a new methodology. But it just demonstrates the power of what you can do when you bring the aggregate patients and you give them the tools and the instruments to measure their own health and measure value both on an individual level and then on a population level as well. And we're across 1,000 plus diseases now but we have significant depth and a handful of neurology for diseases. Going back to Professor DePraes. Most of our $30 billion in health research is biomedical research. And so the questions that you raised as fundamental questions that we need to be focusing more on is this also a question of how our American funding priorities are allocated? Do we need, does that need to change? Or is it an attitude question? Or are we lacking in tools and imagination? Do we need a national research program on the science of health delivery as the Kaufman Task Force called for? What are your thoughts on those questions? My thoughts are I think we've, I'm not saying we should detract from the basic biomedical research and which means applied research and basic research in some kind of term that we use the term transformational research which usually means getting the experiment to the drug that works. But not, I think we need another form of research that helps people understand how they can bring that to apply it to the health delivery system and apply it so it's effective and so we can measure it. And it is a science. It's not a science of a lab science. It's science of how do you take information about what we know works and get it to work in a population? One of the things that we're doing in Maine and my background is I had an opportunity to work all of the country from Alaska to Texas to study health delivery systems. And that gives you, every system is different. But what we're trying to do now is trying to look at how can we bring diabetes care, for example, into the patient's home so they can actually learn to take care of themselves and keep their A1Cs down and keep their diet down and keep their exercise levels up and do their foot exams on their own. They don't need a doctor to do that. They can do it themselves. It's that kind of, but we don't really know how to do that very well and there's not much funding for that. AHRQ, which has been around for many years, it was first started as a national health services research group in Washington, part of the federal government. That's been traditionally underfunded because most of the funds go to NIH. But AHRQ is the health services research arm of the federal government. Now most basic scientists cast that up but without that kind of research, we don't know how to take these experiments and take these knowledge and apply it. So I'd say we need more emphasis on how we apply it through that. And in the methodology, we don't really know how to transform medical care system. How do you know? What has to do with organizational research, has to do with medical research, has to do with people and what incentivized people has to do with healthcare delivery systems, how they're paid for. So it's very difficult stuff. It needs a new type of paradigm. And what I think of when I deal with my pharmacology friends is that how do you study a mixture? Because most of the pharmacology studies around forever were about studying the effect of formaldehyde on cancer, or studying the effect of any chemical on this or that. But we don't count. We're not chemicals, single chemicals. We're people. We're populations. We have issues. So you do need some kind of a way of looking at that which takes what we know from other disciplines and applies them to the delivery system. Thanks. Well, you got to the question of incentives. And it seems to me that a lot of the reason why we may not have more of the type of research that would be transformative has to do with financial incentives. And I'm curious, Mr. Haywood has a business background. Your company is for profit and in journalism we learn follow the money. So what can change if economic incentives don't change? And how can we get the economic incentives in place to do the types of post-clinical research that you believe are needed? Yeah, no, actually. So we are a for-profit company and that was a very active decision we made early on. Actually, as I mentioned, my brother had run a nonprofit and we really thought about starting this as a nonprofit as well. But we felt fundamentally if you couldn't figure out a way of aligning the patient's individual interest with financial incentives in the healthcare system that we couldn't scale and actually make the impact that we wanted to. And so we felt that we had to figure out a way of aligning a patient's interest with the healthcare system. So a couple of the challenges we had though is as we began to help, particularly the chronic life-changing illness space that we're in, those patients begin to measure and understand their own health is who in healthcare actually pays for measuring outcomes? And the truth is in all of our $2.4 trillion that we spend on healthcare, the only people who really spend money to measure fundamental health are pharmaceutical companies because they're regulated and need to measure health. So in terms of the most effective and sort of mature and measuring health effectively, it's the pharma industry. And so that's where we started. I don't think that's where we want to end up. We want to have an ecosystem where patient value drives market value of products and services. But there's a long way to go for that realm. So again, we started in pharma for a very specific reason because that's where the money was for us. But we don't feel like that ultimately where it needs to end up. And as ACOs spring up and payers are beginning to look at patient value and you look at what's happening in the UK with NICE and even the EMA, which is the European FDA, they're beginning to look at patient end value in terms of approval of drug indications on a much more narrow scale and then opening it up as they prove indications in the real world, that the system is gonna have to get much more sophisticated measuring health. And today, there are some illnesses actually particularly ones you focus on. We're pretty good at actually, I mean, if you look at diabetes and chronic illness but actually some of the illnesses that we focus on epilepsy, MS, fibromyalgia, there's really no true standards in the measurement of those diseases. And you talk about, and we ultimately, I mean, our philosophy is that if you engage patients in that problem, the system will be better for it. And just a small example that's in sort of the space you talked about the AAN last year or maybe it was 2010 now, published guidelines for epilepsy patients, for epileptologists and neurologists about the best standards of care. It had nine simple things, like asking about pregnancy, asking about driving, asking about side effect of drugs, very standard, simple things. And what we did in concert with the author of the guidelines is we actually surveyed and engaged patients in whether or not they were receiving that care. So instead of waiting down the road in parallel, we knew with a decent sample site, 300 plus epilepsy patients, how they were meeting the standards of care. And we were able to already measure that the epileptologists were doing it better than neurologists that were doing it better than GP, which you might expect, but actually people were wondering, why would you publish such simple guidelines as those AAN guidelines? And the reality is because the standard of care wasn't being met broadly in the wild. And just following up, one of the ideas, I think it was in the Kaufman report that most intrigued me was this idea that you would have like a sort of a preliminary drug approval process and then a couple of years after that where the drug would still be studied actually out in the public. And do you see, is there a financial incentive for that or would that regulatory change that would need? No, I mean, I think that we're moving that way. I think the both Pharma and the FDA are doing more and more what's called phase four post-market surveillance. I think the reality or the challenge actually for the system is that the rigor of phase one, two and three trials are being applied to that phase four level. And that is ratcheting up the costs of having a drug on market significantly and squeezing sort of phase one and pre-phase one clinical dollars because it's coming out of the same pot. So I think what the system needs to do is begin to look at more and more models for sort of broad monitoring. I mean, our platform is one of them but there are other ways of doing that. And I think it's very important. And the FDA has a long-term initiative called the Sentinel Initiative in that space. But I think it's very important that I think we, random glass control trials are very nice for tidy, clean research. You even mentioned cocktails. They're almost financially impossible in doing cocktails. And so we need every day in doctor's office with patients there are millions and millions of experiments going on and we capture very few of them. There's a hope that EMRs will fix that problem but the reality is for a lot of illnesses we don't actually capture health information in EMRs. So we need to again build out new systems of truth and that's gonna take a long time. Electronic medical records. And I guess along those lines, the last thing I'll ask you is, this type of research and the research that could be built on your platform, the critique of it would be that it lacks blinding. So the idea that the patient doesn't know, get it getting back again to the placebo effect you don't know which drug you're taking. And that's an important part of testing drugs in the earlier phases. So how do you respond to that? So the reality is all research has biases. RCTs are very narrow focused populations with very scrubbed down criteria that don't represent what's gonna actually happen when you put them in patients and they're 50% adherent or they're dealing with working nights and they're- RCTs are randomized controlled trials, sorry. And observational studies have been done, I imagine you've been involved in a number of them. I mean, that's been done for a long time and there are great ways of doing the Framingham Heart Study which is one of the most famous ones where most of our knowledge about heart disease have come, came from. And so what we're looking at is a version of that where you're going direct to the patient and it has its own biases and it has biases that we're still figuring out. But as evidence, as a methodology of medical evidence, randomized controlled trials are 70 years old and they're really good at what they measure. And as our methodology or even observational studies are have less maturity, less dollars put into them. And I think they will become as important evidence as RCTs going forward. But it's going to take a while and it's certainly going to take a while before the regulators value them as greatly as evidence. So the point being we don't need to throw away our old methods, we need to add on new methods. And there are new funding levels to do some of the things that I'm talking about. I mean, you take the Health Services Resource Administration, HRSA. Well, it used to be they would just dole out money for due services. Now they're under this current administration, they're requiring evaluation and evaluation research can tell you whether or not things work or not. CMS with this whole Center for Health Innovations is they have to spend $10 billion a year to try to transform this healthcare system. That's part of the legislation. And they are funding large-scale efforts now to try to see how we can transform the system. That the only criticism I have of that is that it's often so predicated on changing the healthcare provider system and not dealing enough with patients and patient self-management and those types of questions because they're harder. People don't like to deal with hard questions like how do you change people's behavior? Thank you. I'd like to open it up to the audience. Please wait for the microphone on its way. So while I was recuperating from a stroke last summer, it was blogging it of course. And I learned a lot about telemedicine and applications to stroke specifically which is a no-brainer, literally, sorry to use the expression. In other words, getting the right set of eyes, right neurologist, he's sitting around at home, he can Skype on steroids, he gets to a patient in some emergency room and gets the right medication. So the technology is there. It's a no-brainer, it's in like 600 hospitals but it's not in 6,000 hospitals because the innovation that has to happen is administrative. And I would like to get some input from you on how often you see where the big bang for the buck thing isn't actually the new molecule but looking at the administrative bottlenecks and finding ways to use R&D to look at systems and see, in this case, the bottom line is Medicare doesn't reimburse unless the doctor's at your bedside physically. And that's a problem with telemedicine except in like three areas of the country that are officially rural. So what are the missing opportunities there? Missing opportunity, what you're asking about is the opportunities to get what we know out there into the general provider population. Is that the question? At the administrative level because you have to have financial incentive to do these things. And you find that most systems cannot or say they will not do it unless they have the financial backing to do the right thing or do what's required. I'll just give you an example of CLPD. We know how to diagnose, take care of CLPD really well now. It's the use of spirometry. But the problem is we haven't trained nurses how to do spirometry. We haven't got spirometry out there in the general communities coming but we've known it for a long time. And so we don't, spirometry is used both to diagnose CLPD and to formulate the treatment protocols. It is essential. It's like A1C was back with diabetes back in the early, in the mid-80s. A1C we were experimenting with. We were part of a study with Dartmouth and we were basically knowing how to do it knowing what it was. But we found how to do it and now we do it. But to get it out there being used is a tremendous problem because of not, it's not about a hospital administrator talking to you but it's about the whole apparatus of change in a complex organization that requires financial backing. It requires people who want to make the change. It requires people who want to educate the health provider workforce to do it. Because it's not just about docs. We don't deliver care anymore just with docs. We deliver care with lots of different health professions. I mean, I just, briefly, I mean, I think, I agree that there's probably a huge opportunity there but I will say that I think the system is going to be challenged in ways that we don't even comprehend at this point in time. I mean, I think, if you look at biology and new technologies and biology and I know you were saying other areas if you look at new technology, proteomics, microbiomics, all of the omic technologies, there's basically Moore's law is happening four times as fast in biology as it is in computers. And you think, okay, the first PC was 32 years ago now. So in eight years we're gonna have the revolution of evolution, whatever you want to call it, of computers did to society in potential biology. And yes, the system is not ready for it and cannot handle it. And so I don't know whether it's going to break it and what's going to force change. We believe that enabling patients in that dialogue and that journey will help the system, well, will force the system to adapt more quickly. But as often the case, engaging patients is challenging in the context of they are not populations, they are individuals and they have problems that we don't typically call health that are really related to their health. Healing is not about just A1C, it's about all of the factors that go into managing it, which often are not medical. And we need to capture all of that in a system today that is designed not to. And there are some good successes around this. I mean, look at Vermont. Vermont's doing what's called, by the way, there's no mystery why Vermont is the so-called healthiest state in the nation. For me, I mean, maybe folks think it's because up here in the northern part of the country and it has all these rural people who eat all organic food, no. It's partly that maybe, but partly the issue is that they're doing things like the Vermont Blueprint for Health, which works with people and they embed providers and social work and other kinds of people in the primary care office to work with people and their problems of taking what you've prescribed for them to actually turn it into reality for their everyday lives. That's a system change, that's a sea change. It's happening in North Carolina and it's happening at Geisinger in Pennsylvania. And those kind of how we do it, we need to follow up with doing that elsewhere in this country, because it can work. It just takes time. Well, I'm sorry, back there and then up here. I wanted to carry on with the amount of information that's now being produced. Mark Segoff, I'm sorry, at George Mason University, the cost of sequencing genes and so forth for producing all this information, storing it, relating it, just following. However, our laws and rules and regulations regarding privacy, consent, the confidentiality of information is working out of the way so that I wonder with the, there's an advanced notice of proposed rulemaking now that HHS has put out, I told in the water, but it would impose all of the very toughest scrutiny on patient data because of what they call into informational risk on anything a patient is asked. Especially by a non-medical professional, including other patients. I wonder if you can comment on how patients like me negotiates around these very difficult issues of confidentiality, privacy, consent, and so on. Well, there's a straightforward answer of how we do it today. I mean, we went directly to the patient and we're not a regulated body in the context of HIPAA or other things. And even in the spirit of HIPAA, we actually do go directly to the patient and ask for permission to do what we do with the data. But actually that goes to, I think, what you're talking about. I think for us to be successful thinking about integrating three things, which is taking the technology, so clinical research, integrating it into the clinic, and then actually having patient value be the primary driver of that interaction. The system's gonna really have to change and we have these paired buzzwords that we talk about, but I mean, I think as you mentioned, it's gotta go from much more close to open. So on our site, if you're a patient, you're sharing with every other member of the system. So it's an open environment. So private to shared, which is, we gotta get more comfortable with sharing this information. Patients, we gotta start talking about them as subjects and talk about them as partners. And when you partner with patients, then they're much more willing and able to engage in research and actually free up that data liquidity that you talked about. We talk about Secure and think about Health Vault, which is Microsoft's product for integrating personal health records. It's a great concept, but they branded it around a vault, which is starting from the wrong way of getting data liquidity, right? They branded it as a place that you put money and it never goes away. So that's a terrible, that's even just a branding failure, you let alone creating the environment they wanna do. We often talk about validated in the context of healthcare. So validated patient outcome scales are other things and in my mind, the term validation in healthcare is an excuse to stop learning because once it's validated, we never go back and see, hey, can it get better and can we integrate it? So we need to go into an integrated continuous learning environment and that actually can get down to the clinical decision level. And then lastly, I think, aggregated data and population health is important, but the only way I think you can affect that is to begin to personalize it and make it meaningful to the individual patient. And you can talk about that for us on the micro level around an outcome measure. So measuring how a patient is doing, what, yeah, population outcome, most population, I'm sorry, most patient reported outcome scales are designed for population management. And to be honest, to get a patient to engage in that problem, you gotta measure it for an individual and have it relevant to the population. And by the way, what's a population gonna be when we break down disease into the molecular level in the next five to 10 years? There's one area just we have to just mention that we haven't covered and haven't got time to cover it, but that's to do with population prevention and how do we do prevention research? Because it's really a very challenging, difficult area to get people not to start smoking, to get people not to eat right. Those are areas that we don't know how to do very well where we have very little money put to that, to prevention. We talk about it all the time, but it was very little effort, very little money and that has to do with really then population. Health policy, how we change health policy, not just being health policy, I mean transportation policies, food policies, those are areas that we don't do a good job of understanding how to affect, because it's political. But a challenge we have, and this is important, I think for this group, I think it's important is that, is exactly to the point you're making it, which is we actually don't talk, when we say healthcare, everybody means different things. And so prevention and population management on a large scale is different than chronic serious illness and is different than acute care. And we need to really start to dissect and optimize to the ones that make sense. And I don't think we do. I think people, both policy makers and then individuals, and when little Jimmy gets sick and gets leukemia, those are all different problems and we all have different emotional attachment to them in different ways. And we really have to dissect that in a good way so we can actually solve the right problem at the right time. So we have one minute left, 30 seconds for Deb Blum's question, 1515 for your answers and your wrap up please. So in my other life, I'm a chemistry, toxic substances blogger at Wired. So I spent a lot of time looking at, thinking about what I'm gonna call the educated patient, which is what I hear running as a theme through this that we're developing a medical system that relies to some extent on the educated patient, the patient that knows about telemedicine, the patient that can tap into patients like me. And again, the patient who is informed say about herd immunity enough to make an intelligent decision about whether or not to buy into a vaccine or not for their children. But are we investing in that? Are we in a system that's increasingly relying on patients to do their own homework and be educated patients? Where is the policy that is giving us those educated patients? I think that's what we're talking about. We need to understand how to do that better. And we need to experiment on how to do that with additional types of research that will, it's the educated patient, but it's the patient, I think, as partner. Because they have their own, mostly patients have their own best interests at heart. They wanna get better, or they don't wanna be overweight. I'll stop. Yeah, I mean, just briefly, look, I'm not a policy guy, but I will say that when you look at companies like 23andMe, which is the direct consumer genetics platform, what I think is most powerful about them is that they're learning how to do that well. And they're starting with the people who are techno-radi and probably a little bit more literate and able to engage in that, sort of scientifically literate, engage in that problem, but they're learning how to communicate and simplify the science in a way that we can actually move that down the spectrum and actually begin to educate a larger population. And similarly, what we do is the same thing. And you gotta start somewhere. And if we throw darts at innovation that's sort of talking at the top, then you're never gonna figure out how to move down the pipe. But they're different problems, I think. And, you know. So I think we've had a fascinating discussion today. We've painted a vision of the future where we ask different questions where we don't accept that the drug is the answer, but it's implementation in the population where we change policies and incentives, dare to ask more complex questions, and use today's information and communications knowledge infrastructure to promote engagement on a patient level. Thank you, everybody. This was great. Thank you.