 So, it's my pleasure to welcome Pat Cronea to our stage and he's going to be moderating our next panel. Pat is the Executive Vice President and Chief Medical Officer for National Health Plan and Hospitals Quality for Kaiser Permanente in Oakland and in this role he oversees Kaiser Permanente's national quality agenda, helps ensure that our members and communities receive the best quality and service Kaiser Permanente can offer and advocates for the advancement of evidence-based medicine and proven innovation for the industry. And I'll let you introduce our evidence panel. So, Pat, thank you. Thank you, Mary, and thanks for the chance to moderate what I think is going to be a really very interesting and exciting panel. So if you could all come on up, I'll set the stage a little bit and introduce you and give you a chance to present your materials. A lot of really interesting things that I think we're going to talk about here. Just by way of background, I feel as if I have been delivering telehealth services since the start of my practice. And we talked about that a little bit earlier today, the idea of basically connecting a patient's problem to whatever expertise or information they need or their curiosity or their questions to the answers that they need, whether that is by telephone, which is what I'm talking about in terms of delivering telehealth services all the way through to newly enabled and much more smooth and effective approaches to connecting those patients' questions and problems to the expertise and information necessary to resolve those problems. So if we think of it in those terms, while there are new tools and new ways of designing the experience that didn't exist years ago, those are fundamentally the things that we are looking to achieve, I've actually been seeing demonstrations of things like video visits since the early 90s. It never has gotten the kind of traction though that it needed, partly because the technology wasn't advanced enough to put the tools easily in the hands of the patients and the providers and others who are members of the care team that we needed to be able to realize the promise of that. One of the other challenges that we face though when we begin to bring those problems together in fundamentally new ways that overcome the barriers of geography is that we create a whole new set of different venues that we need to pay attention to when it comes to understanding the quality of the care, the quality of the experience, the experience not just from the patient's perspective but from the care team's perspective and doing so in a way that helps us to understand whether or not we're delivering and designing those interactions in ways that are satisfying to all. And then there's also the question of whether or not it does indeed make care more affordable. Does it do so in a way that doesn't duplicate care but actually substitutes for more expensive care, resolves questions or problems that would have ordinarily been brought forward in a face-to-face visit in a completely new way? And then also in expanding the capacity in the different venues, does it give us the opportunity to solve problems that we weren't able to solve before? Being able to understand that is critically important and in order to do so, we have to have a framework to understand how are we performing when it comes to quality, how are we performing when it comes to experience and how are we performing when it comes to the affordability dimension of this which is critically important. Unfortunately, the available information has been relatively scarce. I have been involved with organizations with an organization such as Virtual that has done some of that work to demonstrate the ability to intentionally design a system that substitutes in-person visits for virtual visits and does so in a way that actually saves money, delights patients and drives better quality. So it is possible. And those experiences for me have given me some pretty strong opinions about what it is that we need to do in order to assure quality and ensure a clear understanding of the performance of telehealth services for our members, for our patients, for our communities. And so that really is, I think, a very important conversation, understanding what we have been able to achieve so far and understanding where we need to go. And that's the intention of this panel. The panel includes Jason Goldwater, who works as a senior director at the National Quality Forum, and he oversees activities related to the evaluation of electronic clinical quality measures, as well as projects focused on the use of electronic health to improve the quality, safety, and efficiency of health and care, basically what I've just been talking about. He holds a bachelor's degree and a master's degree from Emerson College and a master's degree in public administration from Suffolk University. Mary Reid is a research scientist for Kaiser Permanente. Hi, Mary. And she is in our Northern California Division, and her research interests include the impacts of patient and clinician-facing health information technology on clinical care, quality, and outcomes. She holds a doctorate in public health from the School of Public Health at the University of California in Berkeley, and completed a postdoctoral fellowship at the Institute of Health Policy Studies at the University of California in San Francisco. And Neal Sika, a physician who is associate professor in the Department of Emergency Medicine at George Washington University. Neal is an associate professor at George Washington, as I said, in health sciences, and the chief of innovation practice and telehealth section at the George Washington Medical Faculty Associates. He earned his medical degree at Washington University in St. Louis and is board-certified in emergency medicine. And with that introduction, I'd actually like to welcome you, Jason, to the stage. All right, great. So I'm Jason Goldwater. I realize that this is just after lunch. Everybody's blood sugar is presumably low. I will try to be as entertaining as humanly possible. I think, Gary, for that introduction, I have no disclosures other than I really, really like PowerPoint, which you're all about to see in just a few minutes. All right. Okay. So, to start off, I have been fortunate that I was the project director along with Helen on the development of a measure framework or a guide, really, on how to develop quality metrics around telehealth. And we finished this project with the Health Resources and Services Administration as the primary sponsor in early September of this year. And the project really has taken off and gotten an awful lot of attention, which is grateful to all of us that worked on it, and to everyone, I think, that is really working in the telehealth space. And so I've been very fortunate to be able to deliver a talk similar to this in a number of areas. And I always start off with this, one, because I really like the graphic, and two, I really like the quote, which is medicine. And by the way, a disclaimer, I am not a doctor. I don't play one on TV. I have no clinical training whatsoever. So if I say anything clinical, I am making it all up. So medicine is a science of understanding and an art of probability. And I think that really plays well into sort of telehealth in a way, because telehealth really is medicine. It is not anything different. It is a form of clinical care delivery. Many people will think it is something brand new or something magical or something mystical. So sometimes I start off with this presentation with a big picture of a unicorn. And then I have to break people's hearts by telling them that unicorns are not real, which really disappoints a lot of people for some strange reason. And I tell them this because I say telehealth is not a unicorn. Right? Dennis, we said this a lot on our panel. Telehealth is not some magical, mystical thing. It is a form of care delivery, and it is really understanding care through the modalities and also understanding the data that is provided in order to make the best decisions possible. So why did we then sort of create, I told you, a really, really like PowerPoint? I'll be happy to give tutorials after this is over with. All right. So there are some great benefits to telehealth, which really explain why a framework like this is needed to understand sort of the regulatory challenges and how to measure telehealth effectively to be able to not necessarily overcome those challenges, but to understand what telehealth does well and conversely what it does not, to be able to understand what regulatory changes such as reimbursement need to be made, to understand sort of the financial impact of telehealth, which is information that is extremely difficult to get. A measure framework and the development of measures as Helen knows well and said earlier this morning drives accountability. It really makes sure that telehealth is accountable and gives an idea of where it is effective. Does it actually increase access to care? Does it prove timeliness to care? Is it cost effective? Are you able to do care coordination through various telehealth modalities? And my belief really is that eventually if you get to a point where you're able to develop a core set of metrics to effectively evaluate where telehealth may be successful and may be effective, then that is how you get to acceptance, which I think was Mark's point, which is telehealth has been around for a long, long time. We actually had somebody on our panel that said the first telehealth encounter was in the 1800s. Right. He goes, right. That's not true. I'm kidding. Which was a phone call between a patient and a provider. That was the first example of telehealth. And you can take that as you wish, but it does lead to a greater form of acceptance when you're able to actually measure it effectively. So why did we then sort of create the framework? What was sort of the genesis of the project? It was one to sort of understand its principles to inform its future use because if you have a core set of measures that you can use to effectively and objectively assess telehealth regardless of modality, then it sort of informs that future youth. It provides an objective and independent way for assessment, and it's a method for putting it all together. The elements that make telehealth, it's not, and Helen said this earlier, and I'm not just repeating everything that Helen said, but it's not, really, I'm not. It's not just video conferencing. I mean, that's long been sort of the thought, which is telehealth is real-time video interaction. We are way, way beyond that now. It is not just real-time video. It is also secure messaging. It is stored forward, both synchronous and asynchronous. It's remote monitoring. And the real explosion has come in the digital health, mobile health area, in which there are roughly now half a million applications that are dedicated to healthcare. And in fact, most of the wearable technologies, including the Apple Watch, which I have on, serves a lot of functions. I mean, yes, I can get the latest CNN updates, which is great. But people primarily, and even Apple acknowledges this, use the Apple Watch as a health device, and that's really where they've been putting a lot of emphasis on. So people look at this slide and they ask, why is there a paper airplane and a coffee cup in this? There's no significant meaning. I just did it to sort of mess with people's heads because I thought, oh, that'll get people talking. All right. So what were the steps and processes that we did when we went ahead to create the framework? That's cool. When I did this slide, I actually rehearsed it in front of my wife, Yael Hera. Some of you may know who she is. And she looked at this slide and said, in our 18 years of marriage, this is the greatest thing you've ever done in your life. And I said, you realize that we do have two children, and she sort of ignored the fact because I can make a rocket ship move on PowerPoint. Helen, this is what I do in my office when my door is closed. Right. Okay. So we did an environmental scan of all of the literature, not all of it. We really focused on a 10-year period from 2006 until 2016, which is the time we were doing the scan. And we weighed the articles, evaluated them, and out of the roughly 600 articles that we came up with, we came up with a cohort of studies that we used. We formed a committee, which NQF is known for. Dennis was, we had the great pleasure of having Dennis serve on our committee, along with a lot of people that actually ran telehealth programs in a variety of states, including Alaska, New Mexico, Arizona, as well as those that have substantial expertise in telehealth, but also substantial expertise with measure development. We identified appropriate outcome and process measures that did not necessarily have to be modified, that could be affiliated and used to telehealth without necessarily saying that telehealth was the mode of delivery. We reviewed MACRA to see what measures would be used that could identify with telehealth. We created measure concepts where we found that there were gaps, significant gaps that were existing, particularly in the areas of access, timeliness, and so forth. We then developed a framework that took all of that information that we had and actually then created a structure by which people could then look and see, I want to create measures around access of care, great. I want to create measures around access of care for patients. Here are the concepts that the committee said are the measures that are really needed. And then after that, we got feedback through a public comment process, and we fine-tuned it. So developing the measure framework is really understanding what needs to be measured. So what is a measurement framework? I mean, we get asked that question a lot. I get asked that question a lot. There's a lot of different elements. Each one is incredibly important. A framework is a conceptual model, a guide, really. There are measures that don't exist. Here's a guide on how those measures can be created. We take all that information and we group them into high-level domains, those areas which the committee considered the most important to evaluate in telehealth, and Helen mentioned this this morning, which were access to care, financial cost and impact, experience, and effectiveness. Then we take those domains and we break them out into even smaller subdomains. So we have access to care. What do we mean by that? Access to care for patients, access to care for providers, access for the care team, access to information. And then from that, we then develop very specific measure concepts. No, a measure concept is not a measure. NQF cannot create measures that would take our status away as the Geneva of the measure development world, where we have to be a neutral entity to review measures and endorse them. We subsequently can't create them, but we can create, through our committee, concepts, which is an idea for a measure that has a description of what the concept is and a plan target. So the domains and subdomains that we came up with, again, were access to care for patient, family, care team, and information, experience. And notice that we talked about experience, not satisfaction, because satisfaction is sort of a nebulous word that could have various meanings. And I always equate that with airlines. So I could take a flight that leaves on time, arrives on time as the worst experience in my life. But by standardized metrics, it should be a great experience. Then I could take a plane ride that doesn't leave on time, arrives a little late, but all of the flight attendants are singing and dancing and giving out free drinks since the greatest experience ever. By a pair of standardized metrics, that would not necessarily work. So we focused on experience. We did financial impact and cost for patient, for family, the health care team, the health system and the payer, as well as society. And then effectiveness, we looked at it from a system, a clinical, an operational, and a technical standpoint. The priorities, areas of measurement. There are a lot of areas to measure, but the committee really decided that there were six areas that we really needed to focus on. Care coordination, patient empowerment, what the added value of telehealth was, actionable information, which they considered to be the most important. What do we mean by that? Actual information is, does the telehealth modality provide information to the physician so that they know what to do next? So they understand what has to be done immediately thereafter. Timeliness and timeliness is providing timely care as opposed to no care whatsoever, and then travel. What's the reduction in travel cost as well? So what measures, when we did the review of measures, which ones did we come up with? By clinical area, we found a few that we really thought would apply to telehealth, regardless of modality. You didn't need to use the GT modifier to say it was a telehealth mode. You could just take the outcome of how the service was provided, independent of modality, and evaluate it against the measure. We found measures in chronic disease, in care coordination, in rehabilitation, and in mental and behavioral health. And what's not on here, because I ran out of room, and the head would only cover so much space, is we did find two measures in dermatology as well. So accepting the framework, right, this is the hard part. So we built this great framework, we're not being biased, and we have a guide on the developing measures. We understand where the gaps are, we understand the pathway to develop the measures. So how do we use the framework, not only to develop measures to assess telehealth, but also in a way to make telehealth acceptable and to make it widely used? So here we go, right, there's the unicorn. It's not real, sorry. Just because Starbucks made a drink out of it doesn't give it any more justification. It's not a real entity. Telehealth is not a unicorn, despite its many benefits and advantages. It's not a magical solution for healthcare. It is a form of care delivery, straight and forward. And so you have to measure and evaluate it as that, as a means of care delivery. There are different ways of using telehealth, whether it's in the ER, whether it's ICU, whether it's neurology, whether it's mental and behavioral health, or whether it's ICU. It's covered in far more domains with far different modalities than ever before, which really justifies and necessitates the need for measurement. Because the only way you're going to get to accountability and eventually to acceptance is be able to show objective and independent metrics that say, this is what it can do and this is where it's deficient. I know Helen and I do a lot of speaking on measurement and the one thing that we always say is, measurement is not always to tell you what you do well. It's to say what you don't do well either. And there is never a technology solution that will solve everything. There's always technology that will facilitate the solution to a problem, but how well and how effective you need to be able to objectively measure. So how do we move the pathway working from increased access to standardized care? Technology evolves. You don't want to limit innovation whatsoever. You want innovation to continue to thrive. The more innovation, the better, because eventually you come to better solutions. There does have to be, we hope, through the framework a core set of standards to evaluate those modalities. And there really should be a way of certifying, to some extent, these digital health applications to think that they meet at least a minimum threshold of functionality and usability and literacy to the patient. We have to say that it's utility, measure the utility, the benefit to the provider and the patient. It's usability, the ease of the modalities of care for both the patient and the provider and the way that that information is understood. Is that information accurate? Diagnostic accuracy is a major issue. Does telehealth provide the necessary information and data that is captured through the care team that provides accurate information so the provider knows what to do next? And then eventually you get to a pathway of accountability and when you get to the be able to make telehealth accountable and you are able to see that it can do what it's supposed to do and where it may be deficient, that is how you move to overall acceptance. Leveraging the framework for acceptance is a very valuable part of the triple aim. You really have to look at things like quality and the modalities. You have to develop measures that are delineated in the framework. The telehealth, if you do that, if we're able to develop measures using this as a guide, it'll focus on what telehealth can provide and its impact on quality and its impact on how it can improve health for patients and for populations. How does it intersect with measurement? It becomes valuable and useful when you align policy initiatives, when you really align things like MACRA or MIPS with telehealth because then it becomes an integrated part of value-based care. When you align measures, so again, you don't want, as Mark was putting it, micro-measuring. You want measures that are core to what telehealth can do and also on outcomes that don't have to be changed because of the modality of the way the care is being delivered and then you have to be able to drive outcomes. Can the use of telehealth in areas that are not necessarily just located in rural areas? Telehealth, for the longest time, was focused in rural areas and it still is, it still needs to be, but it is just as focused now, if not more so in certain areas, in urban areas. So in Dallas, for example, where I'm from originally, yes, I'm a Cowboys fan, sorry, that Dallas Children's Hospital of Dallas just worked with 200 schools in the Dallas Independent School District to implement mental health and behavioral health, videoconferencing facilities with American Well to provide that information, to provide those facilities for students. So how do we put it together? The impact of telehealth on quality measurement, make sure that it's comprehensive, make sure, again, that it really focuses on the patient, make sure the infrastructure is sustainable. Again, looking at things that measure that matter, getting to a core set of measures, not necessarily just every measure, but focusing on those areas the committee deemed were important and building measures that really evaluate that. Being innovative, streamlining so that, again, you're focused on telehealth as a whole, not just focused on modality, but telehealth as a means of delivery and then making sure that it's both feasible and that it's flexible because as innovation continues to evolve, the framework has to evolve with it. Thank you very much. Hi, thank you for the invitation to participate today. I've really enjoyed being part of the day here so far. I'm a researcher embedded within Kaiser Permanente Northern California's Division of Research. So I will be able to share sort of, the organizers asked me to share a little bit about the way I see the evidence for sort of two modalities of telehealth, and I agree wholeheartedly that the modalities of telehealth are very broad, but specifically patient-physician secure messaging and then patient-physician video visits with the patient at home, essentially, rather than from a clinic. I'll share a little bit about sort of how I see the evidence and then a little bit about things that we've found in our setting. So I guess I'll skip through this really quickly. I agree that the primary benefits as we see them for patients right now, for both patient provider email and video are essentially access and engagement. The statistic that was quoted earlier, I think, is the sort of key operational one to think about this two hour burden for the patient to see their provider can, in some cases, not all at all, is not the approach we're going at, but can in some cases be sort of offset through telehealth. And we see impacts on improving engagement also. So another sort of reflection I have before we dig into the literature is a little bit, as I was contrasting these two modalities, I was realizing that patient-physician email through a secure messaging service like in a portal really has come an enormous distance in implementation just through meaningful use incentives. And there was an interesting contrast to really the, and there are lots of flaws to probably that model, but it's amazing when you think about sort of adoption that has been pushed there, sort of from it being a rare thing, probably less than 10 years ago. Whereas video visits are really increasing due to basically consumer demand, absent a clear sort of policy incentive, it's consumer and employer demand that sort of seem to be pushing that forward. And therefore, many of the experiences outside of these integrated systems or these third-party services really trying to meet that consumer demand without necessarily having integration with an in-person healthcare system. So quickly on patient provider email, and I think the research literature there is, there's still plenty of gaps to fill, but it's much more developed because adoption is much farther along. We see typically the patients who email their provider get more or better preventive care and chronic conditions care, things like that. Some of that might be because some of the patients that are more engaged are using secure messaging with their provider, but a lot of it is probably because the secure messaging is really engaging patients and helping them to further their self-care and care-seeking. There's really still a fairly sort of complex interaction between patient, physician secure messaging and in-person visits. It's kind of the question that's always on the table. And I think the evidence is fairly mixed there. Probably some of that, having done sort of a PCORI funded study where we talk with patients, patients often are able to tell stories about ways that they intentionally, because it was more accessible to them, avoided an in-person visit, say. And then other patients say they might have gone in, even hoping to avoid an in-person visit, but find themselves, you know, the provider says, hey, we should probably have, this is concerning enough, we should probably have you in for some follow-up care. And therefore there are these kind of countervailing trends, I think, that create a lot of sort of complexity in the research evidence there. But I think the gaps are still in the digital divide, which it remains. Language and other kind of communication barriers are still difficult to overcome in patient provider email systems as they stand now. Caregivers and family care partners really use these things fairly often, but there isn't a whole lot of research evidence out there about it. And then, as has been mentioned a couple times today, sort of the storm forward model of patient physician secure messaging, I think, while implementation, I think, has progressed fairly well, there's really pretty limited research evidence as I see it there. So this is just a slide that we found in a survey of patients using the portal in our system. Just to give an idea, this is patients who use the portal to email their doctor with their primary care provider. And what we see here is just that patients are using it as their first method of contact across a whole broad set of concerns that might come up in primary care, most often here for medical test results, but also for new medications and new health conditions and ongoing conditions and referrals to specialists. It's really like a broad set of clinical interactions that are happening by secure message. We also ask them, because health outcomes are challenging to get at, and those are things that we're trying to study directly. One of the easiest ways to get it is by self-report. So we ask patients, did it improve your health? And a third of people say it did. Two thirds said they couldn't tell a large measurable impact on their overall health. And thankfully, hardly anyone said it made their health worse. In terms of that question of sort of impact on in-person and telephone care, we also ask patients to sort of tell us about their experience there. And about a third of people said it decreased their office visits, being able to secure message with their provider. And 40% said it might have offset phone contact, so we might even see sort of greater benefit there. This is a paper that we published just last month. It's a research letter in JAM and internal medicine where we looked through that PCORI funded study at caregiver experiences using the portal, which we found to be fairly common. At the top there, there's sort of this intergenerational effect where the primary use case was for a spouse or partner, but people are also using it for children and grandchildren, and then up towards maybe parents or grandparents who are aging. And overwhelmingly, we find that people report that it's a convenient way of participating in their family member's care, that it's faster than other ways of participating in their care. And that's sort of, again, a use case that I think is not very well described in the literature. On the issue of digital divide, which we're always looking at and trying to do our part, essentially, to overcome, we see, and there's lots of papers on this, a broad range of rates of use across associated demographic characteristics. And one cool thing that we've done is to think about the barriers to that. A lot of them are device and internet access. And when we account statistically for having internet access and device access, essentially, we see a lot of those kind of dampen. So we have some reason to be hopeful that as sort of technology proliferates, maybe that will help us to handle some of the digital divide issues. I'm gonna switch here now to video visits, which are kind of a little bit sort of the hotter part of telehealth that people often think of. I'd say here, the evidence, I agree very much with what others have said, that essentially there's a ton of evidence on very targeted specialty use cases, especially rural and specialty access. The evidence is actually really, really strong. There are a whole bunch of RCTs where people have implemented a specific program, whether to bring the patient into clinic and telehealth out to a specialist or to do it at home, where they've been shown effective. There are a lot of these smaller efficacy studies. I think where the gaps might be is in sort of broader use cases. And there are a couple of studies on this sort of more third-party consumer-initiated telemedicine visit where the patient contacts a third-party service that is not linked to their existing provider. I think that those are widely in demand, like we heard the employer group perspective represented earlier, widely in demand, and people are widely satisfied with them. I think that evidence is a little bit uncertain there, also, about whether or not it might offset in-person care. And the quality impacts are really uncertain. There's sort of mixed evidence there, too. So basically, the evidence gaps in telehealth are really wide. It's basically everything else, besides ruralty and specialty care. There's little out there on primary care, new, ongoing, urgent follow-up issues in primary care. Little out there about integration with ongoing healthcare and providers. I think it's definitely happened, as we've heard today, but we haven't really been able to get that stuff out in the evidence yet. More broadly, the quality impacts and the health outcomes, the signals are positive, but the real ability to demonstrate those scientifically is still catching up, I think. And along those lines, like I've mentioned, sort of the impact on in-person care, I think as a researcher, we think a lot about sort of the causal relationship between those things and how to capture them. It's just a very tricky thing scientifically to demonstrate because when a patient wakes up, like the patient example today, wakes up in the morning with a health concern and wants to reach out through telehealth, we don't have any prior measure of sort of wrist pain before you contact the health system. And there are a lot of ways, I don't want to get bogged down statistically to accidentally attribute that wrist pain to the telehealth visit, because our measure of it is the telehealth visit rather than knowing that it was addressing a prior health care need. So hopefully that's clear. I like to talk about these things. Happy to more. Oh, this one. Okay, so I'm gonna share also some preliminary findings. We don't have these out there yet, but they're close. Where we looked at patient physician video visits in Kaiser Permanente, Northern California, where essentially they have rolled out video visit capability, patient physician video visit capability to all clinicians. Everybody can do it across specialties. And we looked at sort of early adoption, both very descriptively, sort of general adoption statistics. And then what we did was actually pursue patient surveys like I've showed you before on primary care video visit patients in the last quarter of 2015. So I'll show you sort of both the large scale automated data and then a little bit about what the patients tell us about their experience. You'll see the blue chart just shows that essentially adoption was increasing rapidly in that first year. Uptake happened broadly across specialties. This one has a lot of stuff on it. I think the take home is basically that the video visits appear to be feasible across a variety of patient characteristics, ages and gender and race, ethnicity and even SES. That doesn't mean that the adoption is equal, but just that it looks like these things can be done by a lot of different kinds of people. What we looked at in the bottom couple rows there is we were curious if these are primarily patients who had not been engaging in regular healthcare, whether these are people who, you know, access barriers had kept them sort of away from the health system. What we find is actually the opposite, that 96% of them had had an in-person visit in the prior year, three quarters of them had had some kind of in-person communication in the prior 30 days. So these are really, in our setting with sort of this integration, these are really sort of ongoing care for ongoing health care, ongoing health conditions. And then what we found, interestingly, is nearly six out of 10 of them were visiting, having a video visit with a provider that they already had seen in person. So these are sort of known and ongoing patient-physician relationships, which I think is unique given the integrated model. We've lost our titles. We just looked descriptively at the length of these visits, what was happening and whether patients were doing them from a mobile device or, you know, you could do it through a webcam also attached to your computer. We found the majority of them were actually done by mobile phone, a few by tablet, and then a third from a desktop computer. Interestingly, sort of the length of the visit sort of tracks with the mobleness of the device they used. So the mobile phone ones were about 7.7 minutes and the desktop ones were about 10 and a half minutes. Kind of interesting to see how the patients sort of what devices they used. We asked patients about the reasons for having the visit. 82% said my provider can see me and better understand and treat my health condition. I think that reflects the patient experience that was shared earlier. Similarly, over 80% said it helps me miss less work, school, or other activities. That seems to be one of the most compelling sort of selling points to the patients is that they can avoid transport issues, inviting some, people need others to take them to a visit or don't have transportation available. They're sort of caregiving responsibilities and childcare that need to be arranged and some of those are basically non-starters for patients. So this offers an opportunity to skip over some of those hurdles. The bottom question there is kind of an interesting one. Almost half of patients said I'm more comfortable asking my provider sensitive questions, which was kind of an interesting and provocative finding on our part, but might speak to sort of some of the complexity of in some situations, this sort of technology offering enough connection and also maybe some distance to raise things that you might not have been interested in or willing to raise during an in-person visit. We also asked them about their experience. Which was overwhelmingly positive actually. 92% said the provider was familiar with my medical history and care from other providers. Again, this is in this integrated setting where the EHR is fully shared and it's documented in the same way as an in-person visit. 90% said they're confident in the quality of care delivered through their video visit. Down at the bottom there, we asked that same question of whether having a video visit decreased their in-person visits and by self-report, interestingly, it's in about the same range, about a third of people said it offset potentially an in-person visit for the same condition. So it gives us a little bit of a signal of what we might find in larger studies. Of course, video visits, like I've mentioned, are not designed in our minds at all to replace all in-person care. And it was actually reassuring to see that even among patients who completed a primary care video visit, 40% of them still said, you know, in general, I prefer to get care in person that this was not sort of in their mind a complete substitution of any sort with in-person care, but sort of as needed probably service when it's convenient. Some minor portions of the study population were still concerned about sort of preparing themselves for the video visit and how to set up their space. And some 11% still said that they're concerned about privacy of medical information. I think those issues might always persist. So overall, we find that a diverse set of patients can pull these things off that nearly all of them had had recent in-person care, which was actually kind of remarkable in our minds. And thereby, the video visit was often with a provider they already knew. This was sort of an extension of in-person, say, care that had been already ongoing. We found the video visits were successful and convenient. These are like 10 minutes or less kind of convenient things that we can picture people fitting into their day, that they were high quality, they addressed their health needs, the providers were familiar with their history and the patient did perceive that. And then there is that sort of about a third of self-report that they had fewer visits for the same condition due to being able to handle whatever the health concern is through a video visit. So these are sort of early findings. We're still continuing to look at these, especially using even larger data sets, even though I think this is one of the biggest out there in the literature, to look especially of an NIDDK funded study right now to look at the impacts of patient-facing technology in patients with diabetes. So we're looking both at video visits and other types of portal and mobile app use. And then I'm very interested in also transitioning to look at patient-initiated telemedicine choice, like how patients make a decision, say with a given health concern between the different modalities that are available to them. Describing that more I think is important. Great, thank you. Green goes for it, all right. Great, so I get to be the very last talk of the day, which is I guess that's better than being the four o'clock talk. And a lot of great things have been said today and I think what I've noticed is that everyone's been I think very straightforward about I think some of the challenges in telehealth today that often I don't hear. So that's been kind of pleasing to say to be here. I wanna give you a little bit of my background. I really got involved in telehealth back in about 2004 after my residency and the program that I've extended since 2004 has been a maritime telehealth program. So really serving patients who are in an extremely austere environment. When you talked about someone mentioned this morning, access or convenience, these are patients who have no access because they could be days away from care or be near care that's very inadequate. And so we've been doing that since actually 1989. And at that time we were highlighting the fact that people could reach us through a technology called Telex, which actually I don't know what that is. It's so old. But now we focus our austere environment access through telephone and email most of the time. And in a small percentage of cases, real time video. And a lot of that is due to the fact that our clients are in those austere environments and rely on satellite connections which are really costly for them to use video. And so the solutions we choose have different requirements than that you might choose in a domestic environment. The other thing that we do in austere environment medicine is we have a different reimbursement model that we're dependent on here. So we've heard about integrated delivery models and how telehealth fits there. In my day to day practice here at GW, I live in a fee for service model. And in my maritime telehealth, I live in a subscription model. You could almost call it a concierge model. Where we don't want our clients to be limited in how they reach us. We want them to call anytime and as much as possible so that we can make sure they get adequate care given the scenarios that they're in. So it's just interesting. What's been interesting for me is now to take all of those lessons we've learned over 30 years or my last 15 years and try to apply those to what I call kind of local, regional and domestic telehealth that fits within the fee for service paradigm and as we evolve to these bundled and kind of other new models of care. So given this day so far, it's really hard to say new things. I don't wanna be redundant. I wanna add a couple other comments. One is we mentioned video visits just now and I think the data was really, really interesting. And then I alluded to the fact that the majority of the work we do is in phone and email. So probably less than 10% of our consult or video. And I was just walking, I think to put my coat away and I noticed a little graphic over there that said that Kaiser did 80 million telehealth visits, but 65,000 video visits. And so we're talking a lot about video visits which are driven by probably not in the integrated delivery network perspective but in the world I live with, which is fee for service driven by reimbursement where audio video is a requirement for reimbursement. And so most patients may not need video and people are extending the concept of well, what is telehealth? Is it text messaging? Is it remote patient monitoring? Is it something else? And my guess is that next year we'll be talking about how within this whole digital health spectrum, how are we gonna measure the efficacy of things like virtual reality and augmented reality and other emerging technologies as well. So hopefully that was not the redundant part. I'll move on to what I had prepared before. And so as I was tasked with talking a little bit more about the evidence base, I came across this 2017 bill and honestly I don't know if this has passed or where it's in its cycle of the policy world but the evidence based telehealth expansion act of 2017 and I saw bullet number one and bullet number two, reduce spending under this title without reducing the quality of care. And then the second one was exactly the reverse, improve quality of care without increasing spending. So obviously the emphasis is do things better without spending any more money. And I guess we all get that but that just means we have to find evidence to figure this out, but it's a big challenge. So, you know, Neil's already, I think spoken about the programs at the VA and one of the things I was tasked was just talk a little bit about cost savings and I think, you know, we don't need to dive into the details of this component. I think on broad brushstrokes, the VA has shown that telehealth and home monitoring have large impacts on hospital admissions and presumably even length of stay of those admissions and management of care. And so those numbers are really compelling on cost savings, at least within, again, within an integrated delivery network and with this type of specific intervention and even cost savings with some storm forward interventions. I think it's clear that rural community members save time, save costs of travel and save lost wages. Like that is extremely clear and I think everyone understands that. What I think the literature has been a little mixed on but we've come to the conclusion is that in CHF patients and congestive heart failure patients that we can reduce cost of care by doing remote telemonitoring, providing access to a nurse who can help patients manage care at home for patients to actually monitor themselves at home. I think that is also fairly clear if implemented correctly and in the right selected patients. And I think this was alluded to earlier today and I feel privileged to be able to share a little bit of data this from data regarding the next bullet and that is that you can reduce ED transfers from skilled nursing facilities, long-term care facilities or acute rehab by using telehealth. And this was shared with me from a company called Call 9. They're a startup in this space. Really interesting story regarding how they have come to understand the limited resources available, especially after hours in skilled nursing facilities and how telehealth with an acute care provider can help to triage those cases using also point of care technologies can help to potentially intervene on those cases and avoid those transfers. And the numbers here I think are pretty staggering. 70% of some of those visits can be deferred or transfers can be deferred. So I think this makes a compelling argument for us to look into this area further. So then the next area to talk about is quality. And I wanted to talk about a couple of the projects I've had the privilege to work on. One was in collaboration with our nephrology group at GW. We had a CMMI Innovation Grant to 2012 to 2015. And we deployed remote patient monitoring and video visits for home peritoneal dialysis patients. So we were trying to be extremely strategic in our application of telehealth. We said, well, these patients meet a couple of interesting criteria. One is they live with a cycler machine at home to do their peritoneal dialysis. So they live with technology. Two, their disease process really requires them to be intensive in their home monitoring. So they need to check their weights because that impacts their cycler time. They need to check their blood pressure. They need to check their glucose. Like these are things they do and are in general more compelled to do maybe than the average patient. And so what we did is we deployed a connected health scale, a blood pressure cuff and a glucose monitor. We gave them extra training. We gave them access to online education. And we have a telehealth call center that supports our maritime business. And we use that to monitor their incoming biometric data and feed that to the dialysis centers and see what kind of impact that would have. And so I think the surprise was the level of adoption but we were hoping that this selected patient population would have high adoption in which they did. And then I think what goes along with most of the telehealth literature is that patients like doing this. They were very happy and satisfied with the intervention. What was unclear was what was the true impact on their care. And while I say that the adoption was very high that was on the remote patient monitoring side and on the video visit side where we thought that patients would have the opportunity to share with us. Oh, this is an emerging infection at my dialysis port site or I have a cellular, I have a rash here or some other maybe visual type of complaint that we might be able to adjudicate through video. We really had very low uptake on that component of the intervention. And when we asked our participants, what was the driver for that? I think a lot of this, we're in a new era, right? 2012 is a different world than 2017. But it was a lot of privacy concerns. They were concerned about the provider seeing their home, seeing the condition of their home. They were worried about once they were connected on video where they did we have kind of the peephole into their house. And a lot of that came up over and over again. And then there were also some barriers in the way this was designed, I think, that may have limited some of that use. And then, so what we did find was that patients who monitored their biometric statistics had better control, but it did not necessarily impact across the board ED visits or hospitalizations. You can see some subsets did have improvement. The next study I'll briefly talk about is a ongoing study we're doing based on a Care First Foundation grant on telehealth. And we basically, we're looking at a problem we have in our clinic. We're a multi-special group practice. And our overall no show rate, especially among Medicaid patients, is quite high. It was disproportionately high. And some of our community partners refer us to these specialty cases all the time, but there's numerous challenges to those patients reaching the academic center, whether it's transportation or navigating our great phone tree or trying to make an online appointment. Those things are complicated and difficult, even for me. And so, we decided that we could try to implement these services in the community and we partner with the Unity Health Center, where we, which is a federally qualified health center in DC, and we built a telehealth room and we trained their providers to be our clinical presenters. And now the patients have an option to be referred, instead of, in the traditional manner, referred to the GW clinics and make an appointment and come over, they can be referred to GW clinics, but it'd be seen in the clinic. And so far, we've seen about 75 patients and again, patient satisfaction has been spectacular. We've seen some amazing outcomes already with our endocrinologists reducing A1Cs and being very aggressive with their care. But we really, it's way too early to tell, based on sample size, but we haven't seen massive impacts on the no show rate. We still have patients who don't show up for those visits. So we'll have to wait to see, but we'll be measuring no show rates, claims, and clinical outcomes for each of those three disease processes we're looking at are hypertension, diabetes, and kidney disease. The third area I wanted to mention on the evidence-based side is something very particular to, I think, mostly emergency medicine. And that is, we were interested in clinical decision support and there was a study that was published, I think now a couple of years ago, looking at the secret shopper study. I don't know if you all are familiar with that one where they called some of the online companies, like Doctors on Demand and a couple of the others, and they kind of had a canned scenario. They walked through and they wanted to see did the providers online meet what the clinical guidelines would be. I think there were some problems with that study, but it was interesting that there was a lot of discrepancy, especially in certain scenarios. And so what we decided to look at was there's a clinical decision rule that ER doctors use quite often. It's called the Ottawa Ankle Rules, a well-validated study that if you use these rules, you can reduce the number of patients with an ankle injury who need an x-ray. And in the literature, there's a couple of studies that say, well, let's teach patients how to apply this rule for themselves because maybe then they don't have to come in. And if you look at that, every patient thinks they need an x-ray. It's very hard for a patient to apply this rule themselves. So we decided we would design a study where a clinical provider through telehealth would apply the rule with the patient, help guide them through that, and then they would go through our normal process in our ED, and the ED doc would commit to the same set of Ottawa Ankle Rules, and then we would look at how these compared. And so far, we haven't met yet our power goal, but this is how it looks. The doctor on one side, they're helping us examine the ankle. We walked through the five steps. But so far, it's been great. We've had about 60 patients enrolled, and the CAPAs, where we're looking at inter-rater reliability between the telehealth provider and the in-person provider have been extremely high. And so I think this is gonna show, you know, presumably this will eventually show that this rule can be applied through telehealth as it is applied in person. But I think there's a whole series, this is a gap in the literature, there's a whole series of types of things that we do every day or rules that we, as physicians or providers, apply every day that yet have not been validated through telehealth and probably should be, and so we're looking at that. But it also depends on what your comparator is, and I don't wanna belittle the value in saying, hey, you know, if you're comparing this to no care at all, maybe it doesn't matter. But as we get more sophisticated in our application of telehealth, then it does matter and quality is a big driver, and this is one of those kind of questions we can address. So when I looked at gaps, I think, you know, we've talked a lot about the Integrated Delivery Network and we've alluded to the fee for service world and, you know, how does a small practice or an individual provider apply to telehealth? I think it's a hard use case unless they're in a rural area or they have some other driver, they know their patients have this need and they can somehow put it into the workflow. And I think studying that is important. A couple of comments I wanted to make before I run out of time is the efficiency of the traditional workflow and what we've learned through some of these projects are that when we do telehealth and especially when we partner with facilities that are in the community, we're often just putting in our normal workflow into that facility, right? And so are we taking away the services they were providing in person? And so we need to really look at that carefully and try to understand, we don't wanna be replacing one for one, we need to figure out how we can scale a telehealth into the community so it adds value, especially when we were looking at underserved populations. So things where brainstorming are how do we train new clinical presenters so we're not taking a nurse out of the fold to do telehealth when they should be seeing patients and we can bring maybe a community member or a community health worker or someone else who can serve in that role. And same when you think about locations, taking space from an existing in-person clinic and repurposing it for telehealth may not be the right choice. But again, it depends on what your comparators are. So clinical decision rule is obviously a passion of mine that I just discussed and looking at standalone telehealth versus system-based telehealth, of course, of course, is a big question. So training is another thing we brought up and I think the question of how do we train our providers and then how do we measure how good they are at providing telehealth, again, as we become more sophisticated are gonna be other important gaps that we need to address. Thank you. Let's first maybe a quick round of applause for Jason, Mary, Neil for really, I think, teeing this conversation up very nicely. And I think we have a microphone again out in the audience and would be more than happy to take questions from the audience. But I do have some that are prompted by the materials that you've presented today. I was, it was nice to see, Mary, the data that you had about overcoming some of the disparities that can arise as a consequence of making available new venues of care and not knowing about how we overcome those disparities and seeing that the deployment of the technology actually helped with that. But that's one type of disparity. There's the geographic disparity. There's the access to the technology disparities that can arise. Talk a little bit about what are the barriers to understanding and measuring and studying disparities that come up in our current environment and how they might be amplified by the telehealth space. Sure. I mean, I guess I'll just sort of reflect on the questions, but I think that the issue of health disparities broadly and also sort of the digital divide and access to technology are sort of very large things that we're all struggling to do our best to address. I think that what we saw is at least what we would hope that while there are persistent differences in access out there to technology, we hope that the technology itself is fairly agnostic to patient characteristics. And so what we see is that once patients use it, and I didn't get to have a chance to present these today, but we have some newer findings that we'll be publishing that show that once patients essentially are able to gain access to the technology, that means through a device and having the right internet signal and having a camera or whatever is necessary in order to engage with the portal or any given telehealth technology that they report, and these are against self-reported data, they report similar benefits that it sort of engaged them that it gave greater access to their own information that it was convenient. And so we're a little bit hopeful that as the needle continues to move in terms of technology access, that that will sort of help level the playing field somewhat, I know that that's a very sort of complicated issue and there are many barriers out there, but we're sort of hopeful in this direction. Okay, great, great. Neil, by the way, I've been out on the ocean days from access to care and I do know what TELX is because I'm a bit older than you, but it raised an interesting question in my mind. A good deal of the research to understand the benefits have been done in the context of those sort of, this is a substitute with what we presume to be the first choice, so rural environments or other access barriers that can be overcome as sort of a secondary alternative to what we would believe is better, which is face to face. Now that shapes the nature of the questions that you ask and I'm curious whether you see opportunities to, for future research, understand not just how it compares to the standard but how it might actually be a better alternative than the standard. Could you speak to that a little bit? Sure, I think you're right. Finding that comparator is really important and figuring out what you're trying to measure in the end game sometimes helps you walk back to the right question and we know that, I think it was mentioned earlier, that we learn things on telehealth encounters that you would never have the opportunity to learn through an in-person visit, things about the environment, things about what's going on at home, maybe engaging a family member. So I think that's really important is to think about not always looking at, we have to disrupt the whole way we're thinking about a visit and rethink about the visit itself and therefore when we look at the comparator, it could be something completely different than we think about today. Jason, you spoke about the framework of measurement and understanding and I think that's a particularly interesting thing because if we take any lessons from today's conversation, we will actually be seeing broad deployment of these capabilities before we know all of what they will do. So I'm curious, your perspective on how thoughtfully creating a measurement framework will help us actually to better understand the impact that it's gonna be having on the quality of care and how we can draw from that lessons about how we adjust and change and expand or reframe the way we use these venues of care so that they're actually driving the kind of results that we want. So that's a great question and I think we took a lot of lessons that were learned from past framework projects, particularly around health IT in which sort of the premise was there's a new technology solution or there is some newer technologies that need to be adopted. Let's push forward a framework that will accelerate the adoption of those technologies because in the end they will provide better, more reasonable care. Telehealth's been around for a long time and we can debate whether it started in the 1800s or not but it's been around for a while. So it's not anything that's new. What's happening that's new is there's greater innovation. There's more increase in digital health. There's more increase in wearable technologies. Patient portals are being used more frequently. Video conferencing has greatly improved from what it was. I'm old enough to remember when we were first using the internet for education in the mid 90s and the quality was awful. You can laugh at that, it's fine, I'm that old but now it's greatly improved to the point where the video is extensive and so given that telehealth's been deployed I think it's okay to allow the innovation to continue to proliferate but I think what we have to take a step back in is even though telehealth's been around even though there's so many technologies around it and different modalities of care there does have to be a way of measuring what telehealth can provide and how telehealth impacts quality without changing or inventing new quality measures because there are 600 plus NQF endorsed quality measures. There are 2,500 quality measures overall. There's a lot of measures and the last thing we wanna do is be creating more measures. We wanna take the measures already out there and really think about how do those apply to telehealth where could telehealth actually have the same where could we measure it in the same way we would measure an in-person encounter so that way the outcome is independent of care delivery system, it's just being evaluated and I think when we start with a framework and a guide I would say the wisdom of our committee and Dennis, you're welcome is that they narrow down out of all of the possibilities of things that telehealth could do they said there are six things that are really, really important care coordination and empowerment and access and actual information and timeliness and travel those are the things we should start with first and if we develop sort of a core set of measures around those and then out of all of those measures that are existing in terms of clinical quality pick out the ones that we think would be applicable to telehealth and we come up with a core set you're evaluating telehealth even as innovation is continuing even as it's proliferating even as CMS changes it's reimbursement rules which it just did two weeks ago when they added 14 new reimbursement codes for various clinical services and telehealth and Adam Darkins, your former colleague who is also on our panel made a big point of this which is telehealth is going to continue to go we're not stopping it and we need to let it continue to flourish but we at some point in time have to put a framework together so that we can actually show as Helen put it again referring to Helen, real world evidence of how it's working not randomized control trials no offense to the researchers not systematic reviews of literature not that I'm knocking those those are all wonderful we use a lot of them as our foundation but real time evidence of here is how telehealth decreases travel costs here's how it provides actual information greater communities to care and so forth yeah that brings up another interesting dilemma that we face I think we have this very interesting sort of standoff when it comes to telehealth we have patients with regards to video visits who are apprehensive about being seen in their own environment and being self-conscious about that so there's a barrier there we have providers who are concerned with the flood of new things that they feel that they might need to have to do in order to respond to emails and secure texting and video visits and the very real challenge is there and we have folks responsible for the financing of these things feeling anxious about the idea that they'll unleash a flood of new services being used and yet we've got a demand environment where those things are beginning to happen that complicates research because you can't even create a controlled trial because the environment is so uncontrolled and changing so rapidly that it is difficult to get insight so from your perspective and I would say from any of you who have thoughts on this how do we come to know the impact of the introduction of these new technologies broadly deployed when the cycle times are that fast the standoff between all those stakeholders is so significant and the momentum is gaining strength. Any thoughts? I mean I would say this that's not the first time that question's been posed to me and I think that what I've said is you can't really look at telehealth in a vacuum and what I mean by that is you can't assume deployments are equal across all populations or all age groups so for example my parents who are in their 70s will never use telehealth. I mean granted they don't really understand what I do for a living but regardless of that they would still never use telehealth they feel more comfortable seeing somebody in person. My daughter on the other hand who is 14 would much prefer to use text messaging and video conferencing for any hospital visit she could ever have. If it doesn't mean she has to leave her room and she could continue to listen to whatever emo music is popular on that day she would be happy to be doing that and I think you have to evaluate it that way like deployments across populations are very, very different in the elderly population there seems to be more of a focus on perhaps using telehealth as an adjunct in person care whereas for millennials I think there's a preference of using telehealth as the primary source of care and in person care as the adjunct and so I think you have to frame studies and evaluations that way not just broadly in a vacuum but really the demographics and population characteristics. Yeah I agree completely. I think there are a whole lot of small RCTs out there of really specialized tools but we similarly I think agree that there's a lot of room to look at the real world implementation of telehealth and you're right there isn't a perfect comparison group but there are a lot of statistical tricks that we can use to do our best to account for patient characteristics and we do need to look at sort of how patients are making their choice that it's not a replacement tool that's sort of offered as a direct replacement really it's sort of a patient option where patients can select where it fits their own preferences and their own possibly sort of travel barriers they can engage in it and so we're trying to think about it in a real world setting you're right there are challenges but I don't think they're insurmountable at all. Brief comment I think just the fact that we're talking about it beforehand is really important that we're trying to measure something as we start new things right when we started treating whatever we started treating hypertension we didn't think maybe necessarily forward about the fact that we're treating hypertension through this vehicle that came to my office at this site and this site there was a difference so it's the same idea I think this is a new vehicle and we're all being thoughtful about how we implement at least the beginning small projects everything starts with a pilot that seems you know and so we're thinking to measure the effect of this in some way and just the fact that we're doing that is important. So I'm gonna keep some of my own questions in reserve in case I need them at the end because we're starting to get some questions from the audience. If you could just introduce yourself real quickly before. I'm Deena Puskin I am the founding director of the Office for the Advancement of Telehealth in the federal government and involved for many years starting in 1988 with the development of telehealth programs. So I have a somewhat historic perspective. I certainly appreciate the panel and everything that's been done and you've done so far. To me and all working in early days of measurement it was very important to ask the right question and I sometimes think we're asking the wrong question in the context of telehealth implementation. So for example in mental health early on we found out that video interactions with certain kinds of patients were actually much more effective than in person because the patient can control their sense of how far the practitioner looked to them and their comfort level was much greater. We learn this in the 80s. Another example is emergency care. The idea that we're going to in rural areas save an emergency transfer. Well actually at Fletcher Allen in Vermont what we found and I think we found it over and over is that sometimes we did but more importantly the information that was transferred with the patient before they arrived at the higher level hospital was critical to the outcome of that patient. So sometimes avoidance, asking the right question in the context is very important and finally as another example in chronic disease monitoring. The issue may not be whether patients are as happy or whatever that's a very critical question but another question is how long do they maintain monitoring? Do they do better with using telehealth in terms of how long they maintain a high level of their monitoring and then what the outcome is so ultimately. But the length of time you follow so many, many studies are for three months, six months. What you want to know is what happens over the longer term and what your interventions are. So I just want to raise that because in the context of measurement and the broad measurement as you apply the measurement to indicators ultimately the right question is very critical. Otherwise you just say, well did it save money? Well is that really the question? It's one question. I think that brings up a good point because it sort of dovetails Dino with a question I've got quite a bit as I've spoken which is well what are you gonna do now? You created the framework so what's next? Like that's great, what's gonna happen? And so I think the next thing that has to happen is that measures need to be developed from the framework. So how do you go about doing that? Do you do that sort of abstractly which is you use the framework as a guide and develop measures and then put them forward and say okay everybody here's what you use. Or do you take another approach which is my colleague Jud Hollander said to go pressure test the framework which is to go to telehealth resource centers or to go to telehealth programs and look at the measure concepts and see how they apply to these programs. Would they actually provide a metric that would be valuable in assessing its effectiveness? And the only way you are able to determine that is if you're asking them the right questions. If you're saying how do you evaluate timeliness? How do you evaluate access? And seeing if their answers align with those concepts and if the concepts have to be altered. And then once you have that done and you have those concepts created and you build use cases from those concepts because of the information you collected then you can develop measures that hopefully will cut across all the telehealth programs because you've done exactly what you said to do which is you've asked the right questions and you've applied the framework in ways to know where the adjustments have to be made by looking at it in real world situations. Yes another angle on that issue of question comes to mind and that is if the objective is to improve the quality of the decision in the moment. In other words, a patient comes to one of these venues and is trying to find out is there something I can do at home and stay there? Do I need to go to some other venue of care? Do I need to supply some additional information? Has there been any research or are you aware of any research that looks into which the impact on decision making and whether it's a higher quality decision and that higher quality decision may actually be to use more care because someone would have decided to stay home when it was critical for them to go in. Anything that you're aware of in that vein? I'll make one comment on that. So I mentioned the secret shopper study and I think if you look at that, one of the scenarios was back pain. And so the guideline is for patients without traumatic back pain not to get an x-ray. So there's an inherent bias I think in this question because if there's someone on telehealth and you're looking at a non-integrated type of online delivery model and you're measuring how often the provider recommends getting a lumbar spine x-ray and non-traumatic back pain and it's better or lower in the virtual visit than it is in the in-person visit, the question is was that visit really better or not because it's inherently biased in the mechanism of ordering. So when you look at test ordering, when you don't have access to test, they probably will be lower. So that's the insight I have but I don't know the answer. Yeah, the venue creates a set of options that might not be available in another venue or vice versa, okay. Yes. Hi, Patricia Lynch from Kaiser Permanente and maybe Dina who was in front of me wants to contribute to this as well. One, there's been a lot of exciting conversation today and about the use of telemedicine as just a clinical tool and an extension of regular clinical care and not something separate of which you showed some pretty compelling evidence. But on a policy level, we've been having trouble, particularly with the federal government in trying to address requirements around network adequacy as we provide coverage to people. Looking at, they still look at time and distance which is what we call dock on a block where you have to show, you have to just contract with lots of physicians and don't take into consideration the elements of quality and access and really the offering of telemedicine as part of the product is not generally taken into consideration and states have less of this issue, it's really at the federal level and you may be the wrong people to ask but I'm wondering if you have any ideas as to what sort of data points we could develop that would help to move our policy makers away from a fee for service centric measure of network adequacy to something that looks at really what serves the needs of the patient and provides a quality experience. So my immediate recommendation is to look at the framework that was created because the effectiveness domain talks about a lot of that, particularly the technical effectiveness of telehealth that does speak to technical, to network adequacy and that is a significant problem because in these presentations I've had conversations with an awful lot of rural health providers and nurses and pharmacists and people that are in incredibly remote area so I was talking to a group that was running a clinic in the northern most part of Arizona right up against Utah, I mean there's no hospital anywhere near them and they just got fiber optic. I mean they were running T1 lines for those of you that don't know the history of the internet T1 was fast two and a half decades ago. I mean that would be the equivalent of dial up right now without the annoying noise that we used to get when you did dial up. And so the fact that they just got broadband shows that pockets of network adequacy are existent so you have to be able to evaluate that. You have to be able to assess is the network reliable enough to deliver the services that they intend to serve and I think the way to sort of drive policy in the government on that space which agreed that they really look at it very much in the fee for service model is to show where there is inadequacy and what the effect of that's going to be on variety of clinical components. You know particularly in mental and behavioral health care which is mostly almost all video conferencing if you have a T1 line you're not doing video conferencing not without significant buffering or numerous interruptions and in those areas where they've said children and adolescents in particular desperately need effective tele-mental health particularly because of the disorders that they consistently see which are depression, anxiety, eating disorders, substance abuse. If they don't have the network adequacy to deliver that services then they're going to fail to meet those quality metrics. That to me is how you start to drive policy and that was sort of again the wisdom Dennis you're welcome again of the framework by creating those effectiveness domains and the technical area to really assess that adequately and objectively. Neil you may have a unique perspective on this given what you were saying about the maritime. I mean that's the network adequacy challenge isn't it? Yeah I mean we, yeah I guess there's a parallel there. I mean we're really focused on kind of inertial and early stabilization so that's the ER as the hub but then our network is international global unknown. Cause we're looking at different ports and we're trying to compare that to what is US standard of care. So there's a rather broad question that I want to ask and maybe have each of you respond and this goes to the public policy and maybe how we as a nation organize our availability of funds to do research. What do you think would need to happen to the way we fund research whether both privately and publicly that would give us the availability of funds to really answer some of these difficult questions on the timelines and cycle times that we need to when the pace of change is so rapid. Any thoughts on that? Giving you a chance to ask for money for research. My exposure is I'm a researcher. And so my answer feels fairly obvious. I think as a health services researcher there are and especially one embedded in a really rich healthcare delivery setting where there are a lot of questions that could be asked and answers that could be offered to the broader US research community and health healthcare delivery community that where funding is a real obstacle I think and I think we can't answer all questions and there are a lot of things that still will be challenging and no one can predict the future but I think there are a lot of things that could be answered through high quality sort of rigorous research today if we were to invest in that kind of funding. I mean obviously that's true and I think it's important for the funding agencies to kind of recognize that this study of telehealth is kind of as important a question as some of the basic science questions that we are pursuing and so it's really kind of elevating kind of the critical need for this. I mean the more and more every day everyone's saying telehealth's happening, right? Five years ago people are still kind of like well what is telehealth? And now I was interviewing residency candidates yesterday and they're like well tell us gonna happen so you guys are gonna train me how to do telehealth in my residency, right? And I'm like yeah we are but a lot of places aren't yet and so as that shift is occurring hopefully that escalation will occur as well. I would I think just sort of echo what I've said before which is it really needs to be real-time based. I mean I think you need to have real-time evidence, real-time metrics that really show where it is effective, where it is not. And I think also where's the greatest need. So again conversations on this topic have said that well I think the framework that was created has been enthusiastically received by the telehealth community which we're grateful for. They've also asked can we do smaller frameworks on mental health, on stroke, and on chronic disease management because those are the areas they see the most frequently and having a core set of metrics that evaluate telehealth services in those areas would be valuable and providing real-time evidence and real-time metrics on those services as well as tell us the whole I think drives research because it drives improvement and it also meets the market need at the moment. Great. Well we are coming down to the end of our time really, really appreciate Neil, Mary, Jason, your time, your insights, your experience, and your advice about work coming to understand what it is that's actually happening in the telehealth space. So thank you very much.