 home stretch. So we're now going to go into our third session, and this is going to be focusing on implementation issues surrounding genomic CDS, and our moderators for this session are Ken Kawamoto and Casey Overby. So we'll let them go ahead and set the stage. So we'll just introduce ourselves real quickly. My name's Casey Overby, and I'm at the University of Maryland School of Medicine in the program for personalized genomic medicine and Center for Health-related Informatics and Bio-imaging. My background's in biomedical informatics, to my PhD and postdoc in biomedical informatics, and also some bioinformatics, previous to that as well, with a focus on clinical decision support. And I've been involved with several of the NHGRI-funded projects like eMERGE and Ignite, so the implementation, I'll be able to contribute to the implementation discussion based on some of the experiences that we've had in those projects. So I'll introduce myself too. So I'm Ken Kawamoto. I did a lot of my training at Duke and had the mentorship of Jeff Ginsburg there. That was great. Right now, I'm at the University of Utah. I've been there for about three years. I'm heavily operational. I'm one of the associate chief medical information officers, focused a lot in this kind of area. I also am heavily involved in standards. So I co-chair the HL7 clinical distance support work group, and I've been working on behalf of ONC and CMS for a variety of their standards initiatives in the clinical distance support space. I coordinated something called Healthy Decisions, which was ONC's efforts to develop standards and clinical distance support. And now I'm co-coordinating an initiative called Clinical Quality Framework, which is taking the CMS-related quality measurement standards and combining them with the clinical distance support standards. So very much in standards. And I also, through NHGRI funding, developed an open-source clinical distance support framework called OpenCDS that we're actually now using operationally number of areas and some companies have built it into commercial EHR systems and is being deployed in the VA. So have a lot of interest there, and Brandon actually used that framework for prototyping doing genomic clinical distance support. And so just wanted to remind everybody of some of the survey results that are related to this topic of implementation issues around genomic CDS. One of them is maintaining the linkage of molecular observations to laboratory methods used to generate them. This may also include the analysis methods and keeping track of that providence that we were talking about earlier. Another survey response related to this is support for both individual clinical care and discovery science. There's also been some discussion about being able to keep the genetic information separate from the knowledge so that you can support those as well. Another topic is clinical decision support knowledge must have the capacity to support multiple EHR platforms with various data representations with minimal modifications. And then the fourth one was leveraging current and developing CDS genomic standards. And so these are things that we can think about during this discussion. We have the three questions that we can talk about as well. First, related to some of my experiences, some of the influencers of implementation are characteristics of the existing systems. That might be a topic that we can discuss through experiences in both e-merged in collaboration with CSER and Ignite. There's a diversity in the types of clinical decision support available at the different sites and diversity and also the projects that are being pursued. So thinking about the characteristics of these different sites is something that is a consideration in implementing. Also there are existing priorities that need to be considered such as meaningful use requirements and I know many of the universities and institutions are maybe switching their vendor systems and so keeping that in mind and implementing projects. Also we can in information science we a lot of times draw from theories, behavioral theories, technology theories and those can be considered a way to understand the best ways to have uptake of decision support projects and so that's one of the topics that's talked about a lot in the Ignite network and then also considering the context has come up several times as well already. So who is our audience and when is it the healthcare provider, is it the patient and there's been a little bit about the patient but I've been exposed more to our Biobank initiative that we're getting started and there could be some venues for decision support for patients as if their data is being collected how do you have a venue for educating them over time on understanding their risk in the context of their conditions that they currently have and there are several questions that we can think about. So I don't have to wait in line to raise my hand to provide some comments so I'll go ahead and start with some comments. So just some thoughts on some of these questions. So the first question about how should we provide this kind of distance board and work flow, I think that's obviously a really key question. We've talked around it but like what is it that's going to be the end user experience and I think some key questions to think about it is from an end user perspective what's that going to look like, what's that experience going to be, specifically what content is going to be provided and what are the technical options that we may want to use so there's a variety of different mechanisms to deliver clinical distance for what should we be targeting. And I'm just going to provide a little bit more thoughts on just the second question. How can this be done in a scalable manner and this is something that I know a lot of folks like Blackford and others have been really really focused on and it really comes down to it's great if we can do it at one site but then we've done it at one site and then especially if what we did at the first site required say external support and funding to do how you obviously can't replicate that at 2,000 clinical sites so how are you going to do that and some of the thoughts in particular here would be around standards, you can't interoperate unless you have standards like we've been talking about architecture, we talked about different approaches. I think a key part is ROI which we haven't really talked about, the return on investment, it sort of comes down to we only do things when it makes sense from a, you know, at least from a value and financial perspective, from an institutional perspective and this is again getting away from the grant and research perspective but I think the intent is to come up with approaches that regardless of whether NIH is funding it, institutions will do because NIH can't fund clinical implementation throughout the country and so I think it's really important to think about how we can demonstrate clear return on investment for implementing these kind of systems and then the last point I thought we can really focus on is alignment because there's so much going on that's large and really on everyone's mind and if we aren't part of those thought processes and decision processes then it's not going to work out so one is what are EHR vendors and systems doing? I think we really should start by saying what is easy to do in current EHR systems, how are they architected, what is their philosophy on doing things and we should start by saying how do we align there rather than thinking of how they should change. I think we should start with the assumption that let's do what works in current EHR systems and then if there's deficiencies then point it out and say can we work on this but my personal belief is there's so much it can do using current EHR technologies. Another key players, ONC and CMS, they are heavily involved in this area for obvious reasons and actually have regulations and mandates etc that it would I think be silly for us not to be aligned with. In particular there are standards efforts in this area that are highly highly aligned and I think can be easily adapted to these areas. There are standard development organizations or SDOs in particular HL7, there are other groups like IHE, we really should align there I think. There's a number of open source efforts so I mentioned my colleague Gilar Medelfield's open info button effort, I work on this open CDS thing. Basically my thought is if there are freely available open source tools that anybody can contribute to and there's no intellectual property associated with that, that's a pretty nice platform to develop something that in particular can be taken up by other groups and then in terms of alignment I just wanted to add one other group which was the institution. So oftentimes decisions at least in operational clinical perspectives has to align with the priorities and incentives of the institution we're in. So if we for example bring up a genomic clinical distance support use case, the questions are going to be the same, questions that are going to come up for anything else you propose in an institution, why do we want to do this? Why is this a higher priority than implementing this meaningful use requirement or why is this a higher priority than dealing with this particular clinical issue we're dealing with or implementing patient self-pay in the personal health record? And I think we need to be very thoughtful about how we align with what's important to health care systems and what it would be that would make these systems want to implement these technologies and approaches we're talking about when we present to them and say hey look at what we've done and this is how easy it is for you to do it and here's the return on investment and here's technology to help. So that was just some of my initial comments but maybe what Casey and I can do is just open this up for discussion and I do think it might be useful because we've been talking pretty technically to start with the work full issue and especially if they're practicing physicians and clinicians here they could comment on what they would think would be the right ways that genomic clinical distance support should be provided. So maybe I should scare Kurt and put him on the hot seat here to get it started since you did in fact open your mouth about this issue earlier so if you're willing to maybe take the opening salvo on that I'll give you the opportunity. I'm going to say things that are you're not supposed to say if that's okay. Among the things I would say is that you know the average physician when they're in their office they're focusing on accomplishing the tasks they had to complete their day and they are unlikely to want to do things. Among the things we would like to be having happen is to collect you know for example structured data about phenotypes of individuals so we could have data to figure out how to do that Bayesian calculation to figure out what's significant about variants in the genome but there is no way that you know you're going to get the physicians to collect that data unless there's an incentive for doing it or it's very, very easy. I mean I can imagine if you could make it easy enough so they could accomplish it while doing their tasks they might do it but if you go to a physician and say I'm thinking about this idea can you fill out this survey form every time you do it it just never happens and the other thing that I think it's sort of the 800 pound elephant in the room I think Heidi sort of spoke to a little bit and that the 800 pound elephant is that the billings you know that the epic and things like it are billing systems. They're not designed for collecting structured data and getting data out of it in a research point of view and we're having a huge amount of time to put additional things in there that we'd like to do if you wanted to do a structured thing and it's not just epic it's everyone you know the homegrown medical record system we used at UNC before that had the same property and you can understand why. I think the you know the although you know I like to look at genomic data and things like that you know coming to solutions with visits and supports is going to be absolutely essential and of course it's going to have to be very context specific in general and it's going to have to be dynamically telling you what you need to know when you need to know it if it's going to get used. You know I think Dan's comment about you know you can only keep so much in your head before it explodes it's sort of apropos. You know I think there's a couple of interesting dimensions to the pure clinical perspective on where and how to provide the genomic CDS and one question that you know we've kicked around for a long time is to what degree can the patient supply you know the family history with sufficient detail and rigor or a proxy for the patient perhaps to supply a family history and if we were to ask them to do that what are the key things we would ask of them certainly there are some models for the surgeon generals tool and whatnot but I'd love to know if we could wrestle down that issue because it might really help from the clinician side you know a lot of data gathering which is kind of wrote or you know mundane. I'd like to follow up in a former life I worked on a non Alzheimer's dimension it turned out it was be pretty turned out to be really important and it turns out it's one of the most common causes of non Alzheimer's hereditary non Alzheimer's dimension and when I sort of stumbled into this space because of a family it wasn't known by any physicians and so we started figuring out how we'd identify patients when physicians didn't know how to see the patients. We learned that in fact we could ask we could ask questions of caregivers using simple instruments that experts helped develop that would get at that where we could get as much information from a caregiver with an unskilled person as you could possibly get from a physician that didn't know what they were looking for and so I think you know you can imagine doing this in spades. I mean the physician is sort of a rate limiting step in a lot of this and I think it's the historical perspective there's the physician being the center of the universe where he communicates to peers as a courtesy and you know that you know that's from ancient times practically is how the system is still evolving and working and I think you know it's going to it's going to have to undergo drastic changes in the near future because we have these terrible problems that we've alluded to today and I think that it's likely you know that you know for genetic medicine and stuff it's it's all the problems that you know that are present in the rest of medicine in spades you know we've all had the experience of hearing something bad happen from genetics I've known people that misinterpreted results and got you know bilateral mastectomies because they didn't really understand what was going on and so you know you hear about stuff like this and it's happening and we have the potential to do a lot of harm and we have to take care of a lot of issues you know excellent points all you know the second part of it I guess is beyond who's gathering those data which might be relevant to genomic cds how do we actually fit them into the workflow and what I want to explore a little bit further perhaps is you know we think about classical clinical decision support to be oftentimes just the reminder on something which is you know kind of important to do easy to forget doesn't fit into my normal cognitive pattern or what or what not but I wonder if we know whether or not genomic cds is actually of the same character or is it actually something which is more reflective requires a little more thought to interpret and use the information and it's really not akin to a pop-up reminder of any kind whatsoever Josh you want to weigh in on that? Yeah I think one of our observations has been that there's many pathways that you can have genomic information effect care so because of its persistent value simply having it displayed in the EMR in a prominent place or in a structured way can influence care and also you can have people like a pharmacist use the information to prompt changing care and that can actually happen before all the conditions come together for cds to fire so I think before we actually implemented our program we were thinking cds would be the first thing that would happen but in many cases it's more like the failsafe because it relies on a user to go in and essentially start making an error by going against the genomic risk and then they it corrects that error but a lot of times there's preemptive actions so you never reach those conditions Yeah I do think it's useful to as I think people are coalescing around is to think about some fairly different use cases so you know one is the real-time alert you're ordering a drug you didn't realize there's genetic information there and it tees up for you the fact that the patient has got a risk allele for adverse event related to that drug and that's sort of real time but then there's scenarios where you the knowledge updates you already ordered a test your patient has a variant but then there's evolving knowledge surrounding that variant that may be either put into a system that is accessible to the physician or maybe actually proactively delivered and then also support for family members which is something that's somewhat unique and we've dealt with we've had this really challenging situation where you know pro band comes in has a variant 10 family members get testing half of them are negative half of them are positive later we realize the variant is benign and now we need to update well the only people they get updates are the people who had the variant whereas the other half of the family that didn't have the variant and were told they were risk-free they need to be updated too and they don't have the variant so there needs to be a notion that these people have been tested for this variant and we're told something different and don't you know like and the physicians want us to tell them all that who's related to who but that information didn't come through the physician it came from other sources we're not allowed to say who's related to who in the EHR so that's an entirely challenging situation we have to deal with and then we move over to our somatic cancer oncologists that we've been supporting and they don't want us to even tell them what the variant means they just want a system they can query find all their patients with the variant and look at their outcomes so they want the clinical decision support tool that is a query based system so you know I think it would be really useful to sort of bucket these very different use cases before we get to the point of sort of you know talking deeply about workflow I get to have a completely different clinical perspective on challenges that we're going to be huge so we recently pulled the Cesar and Emerge working group EHR working group members to see how genetic information currently enters the EMR and some people were surprised others not so surprised that it comes to the EHR through various different pathways and that almost all of the decision support that's going on right now is on a very limited set of genetic information that comes in through the laboratory that's associated with the hospital that's providing the decision support whereas quite a large amount if not the majority of genetic information most institutions come from external hospital external laboratories reference laboratories boutique testing laboratories and they came up in the last session that you know testing might even come from a patient the directed order that direct to consumer tests and that as a laboratory as we are challenged to enter this information into the medical record we're limited by how the information is provided to us and so if the external laboratory provides a PDF that's all we can put in the electronic health record and occasionally we try to extract and in very specific cases we try to extract some coded information from those to be able to be used in downstream systems but that's a very rare case and it's extraordinarily labor intensive and so that in order to so the standards for data representation for data transfer don't just need to be reflected in the electronic health records but also need to be reflected in the laboratory information systems that through which the information almost uniformly goes before it gets entered into the electronic health record sure. I was just thinking the context at which it matters and what type how CDS would fire depends on almost where we are in time so today almost all these results are ones where we would order and we have to wait to get a result back and then and we've already moved forward with this current care plan and then have to modify our care plan which necessitates things like ways to surveil a population and when the result comes back to be able to retrospectively act upon it which is not necessarily done most efficiently through the physician but if you push forward to CDS becomes the first line of defense if the data is just embedded there. So in the future where you have all the relevant variants where you need and knowledge comes available then a lot of times you do the right thing the first time probably only because of the CDS instead of being kind of a last line of sense. One of the questions was at least what forms of CDS one might be aiming at I think adjusted dosing would be one great example of where it's seamless and it's actionable and it may have a one line header that explains why it's got adjusted dosing but it wouldn't be a separate pop-up alert. At least at the hospital I work at I don't have the strong sense it's time for this yet. As I think it was said 90 percent of genetic tests are sort of targeted and they're looking for something in particular I think we got a number of years before it's rampant but I think we need in the re my guess is it needs to be in the research context we need to figure out how to do it right and along the lines of some previous comments I think establishing standards for what are reasonable triggers and what are reasonable actions and what forms of actions work best in a user centered fashion. So I'll qualify this start first by saying I'm not a clinician so but I've worked with some clinicians on our pharmacogenomics implementation at Northwestern so but I do ask for clinicians to correct me if what I'm saying is totally off base. So one part that we found at least as far as workflow where it in retrospect seems obvious is when the result actually shows up to the physician and in some cases this maybe isn't as relevant that the physician is the one who explicitly ordered a specific genetic test for something but if they're doing pre-screening for something doing a panel and just giving them back some type of results isn't enough. We anecdotally had a lot of physicians go well what does this mean and justifiably so one thing that we learned was carefully crafting the wording in the message and the results that are coming back but it's just I just wanted to highlight that it's a very important area of decision support to provide the information that's needed at that point in time when the result first comes back. It's in addition to when actions are taking place during the clinical encounter and then more active alerts are being executed on and I think that also that another area I'm not sure how necessary this is and again this is where I'd ask the clinicians to to provide input but a lot of the stuff that I've heard from clinicians that I've worked with is what do I do now? I want to know what I do now and if you need to send out for a lab test is there decision support that not only says you should order this lab test but in the interim while you're waiting for the lab test here's the best course of action that you should reasonably take and then expectations downstream. Yeah, I think those are really good points and I did want to comment that we are monitoring some of the Twitter feed and that particular idea tell me what to do now in a very short actionable message is a recurring theme that's appearing on our Twitter comments and I think it's something that we have experienced and particularly in organizations that have a much longer experience with clinical decision support where there's an inherent trust in the knowledge management that there's much more willingness to act on information that's presented in that way. I wanted to, let's see, I think we have Alex, Ken, and Jeff in the queue. So I'm shifting gears a little addressing item number seven which is dual purpose. So if these genomic databases are to support research they would have tremendous impact because of many say rare variants being pointers to new basic science research in basic biology of relevance for human health. So if we are to accomplish that then the genomic module if you wish that stores this data will have to evolve actually at a different pace and expose the data in ways which are not necessarily serving clinical decision making but serve that other purpose. Now if we assume that that's that module is there then the question is how does it interface with the clinical decision support system, EHR, you know, LIMS. Certainly there'll be two branches from LIMS one into EHR for more established tasks another into this genomics module but the question is will CDS also take only a portion of data or not? Well the answer is probably only a very small portion of it. But the question is how do these components interface with each other and this is where the standards may come in and need to be defined. What are the interfaces? What are the data exchange formats between the genomics module EHR, CDS, and LIMS? I was struck by a comment that basically a lot of stuff is simply not ready for primetime yet and I think that in fact probably is true so when we think about what do we need to scale we can I think we need to start from the ground up and say well first we need the clinical evidence that treating patients in a certain way makes is actually good and I think we need that and then we need next even if it's not scalable just when we implemented this kind of system at our institution these are the positive impacts we found and I think we're starting to see some of that but I think that's important that's a kind of a separate research track saying don't worry about scalability figure out a good way to make it work in a workflow and see if it makes a difference that's clearly needed if you think back to general distance support I mean there was a randomized control trial by Glenn McDonald showing that back in 76 in the New England Journal of Medicine right so like we need that before we can say how do we scale it and then of course we need to work on the scaling issue but I think sometimes we tend to leapfrog a little bit and I think we need to first start by creating very definitive evidence that caring for patients using these approaches makes sense because what you're competing against in an operational setting is for interventions where it's been shown for years that that improves best care and then showing how you can do this in a scalable manner so I thought Ken and Casey's opening question to the providers was really important to try to understand what the customer needs are the customer being the providers and we heard some great also and heterogeneous responses from some clinicians in the room but I would argue we may not have sufficient data on what the customers really are looking for maybe EMR vendors do and maybe JD can comment on that but through Ignite, Emerge and Caesar I know Ignite has several hundred providers enrolled as subjects in the clinical trials particularly for the implementation components of genomic medicine and I imagine Caesar and Emerge also have providers enrolled and I would at least suggest that we use that as a resource of providers that are in diverse geographic, demographic and also specialty, non-specialty environments and underserved as well as academic medical centers and really see if we can craft that landscape of how providers are really thinking about the utility of clinical decision support and how it fits into their workflow. I mean sort of a related thought one of the things I think we don't have yet really in this space is kind of a fundamental understanding of the epidemiology of the baseline if you will what are the characteristics of genetic test ordering if you will across the board because it's still very early in the CPOE world years ago we had good evidence about ADEs and drug interactions and we were able to thereby clearly demonstrate the impact of CPOE on adverse drug events so one of the things we might want to do is to get some epidemiology on how test ordering is occurring and again not in my field maybe it's great but you know to understand what's happening in the wild if you will so that we can then look at interventions and see their impact. Yeah and that's a point that we had a bit of a sidebar conversation about David and I that this is an issue and we have somebody from Blue Cross was field tech but in the context of one of the other genomic medicine working group meetings which was a payer focused meeting there was a subgroup that went on to do some work looking at you know the role of physician education and the genetic test ordering because there was there's empiric evidence that shows that there's a lot of problems there and there's been a recent paper by Chris Miller out of ARUP that looked at if you put a genetic counselor at the front end for molecular test orders you know how much redirection of testing would take place and they found you know and in a relatively larger referral laboratory setting that there are a lot of orders that were redone and that the savings to someone was in the range of $30,000 to $60,000 a month. Now actually while they used the term savings the reality that was coming out of ARUP's bottom line because had they just simply run the test they would have gotten paid for it even though it was a wrong test for the wrong indication. So in some ways as you look at that you say it's very nice of them to be altruistic but I don't think that that's a generalizable principle across all laboratories that they would behave that way but then it raises the question the tool brought up which is to say is there a role for decision support as a guide to ordering more traditional genetic testing and if we think about it from the point of return on investment we actually have data that we could say here's the impact. Now perhaps a solution to the problem other than having a genetic counselor review every laboratory molecular laboratory test that goes over is that we could set this as a task for clinical decision support to somehow address and potentially solve at least at some level. And so I think we can decide whether or not that would fit within the what we're talking about but I think there's at least some there's a clear return on investment with some evidence relating to that epidemiology that you speak of and so if you want to add to that based on the the tax experience. Sure so I'd say looking at what's happening with ordering genetic tests is incredibly difficult to do even for a system like the blues which in part is due to how genetic tests are coded right. So claims data is only as good as what gets put in there and so our inability to really differentiate between tests at a claim level is quite problematic when we want to see how things are being ordered but I think even when you speak to laboratories now who are running tests they speak of sort of those same massive amounts of savings particularly in the oncology space where you can put together very clear decision tools because the rules are more clear in that environment than they may be in some others and I think certainly large academic institutions that have instituted some of that have seen some large savings. I think for us too we're you know we do a lot of evidentiary assessment in my group and we put together a lot of decision support tools and actually have a meeting next week where we're showing some of these decision support tools particularly in the genetic space. Now the way we use them is different than the way they're being discussed here but I think that there are some synergies and I think understanding how you know plans and payers use decision support tools in their businesses to you know look at claims both in a pre-approval space and in an adjudication space is important because they are being used on a daily basis obviously not only in genetics but in other areas as well. So I just want to make one point which is you know in the workflow we've kind of focused heavily on a clinician I think there's a piece of it that we discuss which is the patient aspect of it especially when we discover new variants and there's a time lag of years right and so how do we contact the patient back and that becomes a challenge when we think about not understanding who the attending physician is at a point in time the patient might not be scheduled for something it's an encounter with inpatient side. So I think you know one of the things that we were discussing or looking at is how do we actually inform the patient since that's the constant factor and then have the patient actually contact a provider for follow-up. Our current EHR system and even our current processes aren't really very much set up to do those kinds of things so that is you know perhaps one of the barriers for CDS. So I analyzed that data for Chris Miller when I was at ARUP and so I had a little more in-depth look at how they did that decision support of letting clinicians know that their tests weren't being ordered appropriately and it was highly labor intensive and involved calling clinicians back and finding out their exact pedigree so I'm not sure if that's I'm not sure if that's automatable and I think that it illustrates that one of the opportunities and also one of the challenges of much of the genomic clinical decision support is that oftentimes these are rare tests that need in-depth understanding and so one of the opportunities is to develop decision support infrastructure where there can be multiple inputs from multiple people at different institutions and it can also be shared across institutions and it's also one of the challenges. At our institution when we tried to roll out clinical decision work for pharmacogenetics on a very small level we found that our larger concern was not about alert fatigue but about alert shock that you know this is the first decision Luke Rasmussen commented on this as well as if it's the first time that a clinician sees something and maybe the only time they're going to see it in two or three years and they need more information about exactly what to do and exactly how to handle that and we were discussing it from a research perspective is are n going to be large enough to provide meaningful results but it could also be seen from a cost issue as well is the yield providing some type of clinical decision support that's only going to influence one or two people every year large enough that it justifies the resources necessary to create that clinical decision support flag and on an institution level the answer is going to be absolutely no if that is shared for rare rare situations across multiple institutions then perhaps the answer will be yes it is justified and cost effective yeah I think the points with ARUP are well taken although I think there are some again there are variations I mean there are ones that are just dead simple like we've done this test before it's a genetic test we don't need to do it again and Bob and I have both been involved looking at our own institutions and duplicate genetic testing and it's particularly interesting if you have a genome and then somebody orders a specific gene test whether you know the approach would be different so with that being said I think there are some things that could still be approachable but you're right for many of them it did require a lot of intensity and it gets to the point that was being brought up by Kurt earlier which is you know part of the problem is how much information is the clinician willing to supply along with the laboratory request which I know is a daily problem for Heidi and the reality is is that when we really look at the workflow it's the clerk or the receptionist that's filling out the requisition it's not the physician that's doing it so there are a lot of those issues so I have Robert, Paul and Ken so I think it's interesting one of the themes that we've been on here for a few minutes is this concept of when genetic tests are being ordered but you know the other side of that coin is a preemptive model where the genetic test is ordered once preemptively and over time it is not the what we would consider the traditional genetic test the sequencing that's being done that's being ordered and done again it's the interpretation that's ordered and so one of the things that we may want to consider here is how that model changes you order the test the physical part is done once and what the clinician actually does is order the interpretation Regarding the question of trying to prevent the clinician to erroneously order the thousand dollar test instead of the twenty dollar we've done that with HIV tests for quite a long time so that the expensive ones were being ordered frequently and what we did at our institution was change the orderable name to HIV routine test and suddenly it sort of went up yeah and our our institution if if we were confronted with this we would put it into the work ordering workflow so that you wouldn't be able to order these expensive tests without answering a few coded questions that we could go by and I think I forget if it's LDS or Intermountain or somebody had had some good antibiotics decision sport tools that really showed quite a bit of difference yeah and I think that you know this gets to a piece that seems to me emerging from the discussion is that you know CDS is not equivalent to alerts and that you know one of the areas to explore within the genomic CDS realm is what are the different ways of doing it and in the AR UP experience I know that there were several others that you did address by the very simple thing of changing the name so it looked different from the test it was being misordered well that's something that it's not interuptive it's just you know it's just helping to build the guidelines so that people do the right thing and that's then it's really not even a workflow issue it's just you know you know providing it's like your ATM not accepting pulling your card and so you'll leave it it's you have to take it right out again so that's seems to be an area where we may want to to focus so I had Ken next and do you want to interrupt Terry you're paying for this so I'll let you no no no I'm not I'm not is the people are paying so the people can can interrupt no I just wanted just specifically on that point it hearkens back to what one of the points Dan made earlier this morning that you know you can put in a simple mechanical change like having somebody put their hand on the on the pilots hand and and this seems almost the same way that you know you just just change the name and say here order this one and people will do it there's been a lot of discussion today about closing the loop and evaluating and I I think that's really important I mean operationally it's so common that you implement a clinical distance border some intervention and you never end up evaluating whether it actually made a difference it just the way it is because people perceive a problem you implement something you never actually find out whether it had an impact I think a big part of what we should be doing is measuring from the beginning setting up these epidemiology or problem assessment and know what the problem it is that we're actually trying to solve because that will by itself solve the ROI issue because we'll be able to tell whether it actually is worth implementing these systems I wanted to return to the patient for pre-attentive since we have that listed out there and we talked about a couple of different roles for the patient one is you know how much patient entered information could we rely on in terms of developing structure data that could be used who were not solely dependent on providers to put that data in another one that I heard was using since the patient is the constant how might we be able to use the patient to return new knowledge as opposed to dealing with the challenge of well who's the physician and who's responsible and how do we track them down but another aspect that I think is interesting and it reflects a definition that many of you have heard me use in talks about what personalized medicine really is and the definition that I use is the one that Steve Pauker and Jerome Kassir put forward in a paper in 1987 where they say personalized medicine relies on understanding what the patient desires the most from their therapy what they fear the most on the basis of as much information as is available and the thing that I like about that definition is that it's first of all it's patient centered it puts the impetus on us to really understand what it is our patients hope to accomplish with their interaction with the healthcare system and with the treatment but also it does not promote any given type of knowledge above any other and so you know again somewhat heretical for me as a geneticist to not promulgate a definition of personalized medicine that doesn't have genetics or genomics in it but nonetheless I think that they really nailed it and so the point I'm making here is and the area that we haven't really explored yet is if we talk about clinical decision support we think about you know doing you know a certain thing based on a certain set of information but if the patient has a different perspective on what they want to accomplish then that could influence a message and Jeff and I have talked about this in the context of family history and this is something that we built into the tool at Intermountain that we built with Nathan is that we collect all the information we can run risk algorithms based on the information that the patient enters to identify where do we think you know the money would be in terms of doing that but as we thought about it more we said well what we really need to do is add one more question which is now that you've entered your family history what are you worried about what what is most interesting to you because that then you know sort of pre-negotiates what should be talked about with the clinicians so as opposed to saying you know launching and saying well based on your family history that you entered you know you've got this risk for cardiovascular disease when they're really worried about the fact that their father is just diagnosed with colorectal cancer you know we can have you know really meet them where they're at and we might be able to sell some of the same sort of prevention messages but in the context of something that they're currently interested in as opposed to something that we may be interested in and so I just put that out there is an idea to say you know how might we encompass that within the discussion of genomic CDS yeah I think that's a great point you know I think this is where patient portals can really help augment some of this of when we talk about incidental findings I mean this this really is pointed right to the patient because when we do genetic testing of course we will find other indications other than the test for what the test was ordered for you know I think we really have to have in opt in opt out methodologies for the patient where if you know we do genetic testing we can actually determine through a pick list what a patient wants to know and what a patient doesn't want to know you know even with some very horrific diseases like Huntington's disease you have certain family members even though it's known to be within the family that just do not want to know until they start you know having symptoms of the disease and so having this type of I don't want to go as far as a consent because I don't think we really need to consent people for this but having them have the options through a patient portal that interlinks with the EMR I think would be very valuable for the patient so I think it's something they would like and also related to that so in thinking about what the patient might want thinking of broader ways to of implementing decision support might be relevant also so you mentioned the patient portal but also if there are a way to text message information and so thinking more broadly in different ways to involve the patient and getting information and educational materials or have you I think that's spot on I think you know we we've talked a lot about traditional computing systems we really haven't talked about mobile computing and how that's going to affect the whole clinical decision support so I think that's extremely important you know I just think that you know the patient seem to be want to be more involved with their healthcare decisions today than they were say 20 or even 30 years ago and so I think it's really important but we have to be very cognizant of the type of information that we flow in a clinical decision support to a physician or a geneticist or molecular biologist and helping make decisions about what patient care would be has to be very different if we're going to present the same type of information to the patient because they are not they tend not to be as sophisticated not saying all patients aren't but certainly there is a different level when you're talking about third or fifth greek fifth grade reading levels for the majority of our patient or our you know patient population we really have to think differently about how we present this type of data yeah although to present a bit of a countervailing argument there we have some preliminary data that we're further exploring that indicates that our physicians prefer the patient level material related to this than something that that we build for them because it's at at a much more understandable level so I think we may we may be meeting in the somewhere but it may be closer to the patient side than on the on the provider side so we'll more to come on that within the next year or so so yeah again totally agree and when I talk to physicians I hear one thing pretty much all the time and that's kiss keep it simple stupid hi Jeff so I second the notion of using the patient portal and this is not necessarily specific to genomic medicine you know we ask patients through the portal about readiness to change when it comes to certain lifestyle and behavioral sort of paradigms you might might be recommending for them but I just wanted to highlight at least in our family history program when a clinical decision support rule fires on the family history data it sends a report to the provider which is written in a certain certain language for the provider but it simultaneously sends a report to the patient that at least indicates that they may be at risk for something and to have them ask their physician so it's meant to engage and motivate so whether that can be adopted more ubiquitously to other cds paradigms I mean I can certainly imagine some patients might want to know that they've been recommended a drug and why that drug was recommended to them perhaps there's also language that could be used for a patient centered or a patient facing report about why their doctor has selected this drug and if they have any questions to talk to their doctor so it really kind of engages them and may even get to your question about what what patients preferences might be yeah and we have certainly in some of the projects like open notes and that sort of thing when we've looked at alerting and opening the electronic health record more fully to to patients they're the simultaneity of it they want the result when the clinician gets the result and to be actively engaged has been a recurring theme so I think that's a you know a very good approach to to go I think I've got kind of coded to that yes you may because actually one one study we did up in in Boston looked at delivering a co-management module for diabetes to the patient via the PHR and it was something that was shared by the patient and shared in the EMR with the provider and interestingly 75 percent of patients actually opened up the diabetes thing and did their work to create a journal and 75 percent of physicians looked at what they were being submitted what was being submitted so the the lesson actually was not only did that thing activate the patient but it also activated the provider great ahead Paul first the example was given of caregivers of patients with dementia engaging with online sort of resources at least those tend to be very distraught and also very motivated persons sort of looking for answers it's far different for healthy particularly young patients who haven't had too much intersection with hospitals and that sort of thing I may just be oblivious but I'm not aware of any proven widespread engagement for these healthy patients for portals and if that's true that would be a problem for a wide family history collection to just small points one is if if there is a proven method or if there's engagement I'd strongly recommend combining it with review of system smoking depression alcohol family history would be just one more component and the only thing that I'm that I personally motivated by is insurance premiums so what I if they want me to fill something out they tell me I'll get $20 off a month if I do it so to address the question in the family history tool that we built at Intermountain we did in fact collect some of that the specific review of systems information Nathan I know we've done a lot of analytics on the patients that have actually completed that I think we have several thousand that have entered data do we have any information related to the characteristics of those patients related to disease burden and that sort of thing yeah overall I think you probably characterize them as predominantly female average age in their 30s it would be interesting to understand the specific clinical reasons that drive them there but in terms of of engagement you know I think there's there's various ways you can go some is to let the risk assessment or the capabilities themselves act as the reason for which you take the time to do it and I think that it's likely that the review of systems and the other data points they've already brought up would naturally come in addition to the family history at that point but but I do think that in many cases the decision support on top of that actually acts as the reason to take the time to do it to begin with Adam I had you next so I just wanted to go back to what Jeff and several others were talking about and I don't think anybody in the room is from the VA if they are so what I was going to suggest is it might be worth engaging with them they they have the blue button initiative which allows VA veterans to get their health information directly out of the VA connects to the entirety of wherever their health data might be whether it's from laboratories or hospitals or other outpatient services so it's a it's a neat system to be able to try and look at how patients directly engage with their data and their 2013 updates of the system actually allow for self reported data to be put into the system so they're thinking about new ways to collect information from other sources and also how to actually engage patients so I would suggest maybe that might be a group to try and engage with in this conversation as well you know if they've been looking at that in the context of the their million veteran program is that something that they're looking for any synergies there so the and so I I should qualify I'm not from the VA so so I've not I'm not talking on their behalf my understanding of the MVP program is that it's completely research based right now so they're not connecting into the the clinical side I think their long term strategic plans around it though are to try and move this into clinical practice so but as of right now my guess would be that there isn't that connection yet probably like I said long-term strategy one thing I was just going to make a comment I'm not aware of anything that's addressing healthy populations but another thing we might want to think about connecting into or things like the Indiana health study where they are looking at healthy individuals and how they're actually interacting with the environment and it's a fully integrated with their electronic health records I don't know if they have a patient portal I was trying to do a quick look before you called on me but I didn't get quite that far but it might be interesting to see if if places like those those longitudinal studies that are actually trying to look at healthy populations were supposedly healthy populations see if they might be thinking about portals and how they're engaging I mean that might be a way to get some of the data that you're referring to so just having the perspective of doing the year in review for the most of a decade the most robust literature about this proactive engagement electronically with large groups has come out of the large west coast HMOs so Kaiser and group health and it is the case that they very proactively once they know you have an electronic presence they they communicate with you a lot and give you lots of opportunity and they have cell smartphone apps and all that sort of thing promoting healthy behaviors and they provide financial incentives for doing it and you can actually get a payment or a Starbucks credit or you know you get to choose so I think that those partners for doing testing of personal genomic apps would be a very appealing opportunity for somebody and just to follow up on that thought I had the chance week before last to go out to health 2.0 anyone else go out to that it's sort of the startup conference of you know all new health apps very patient oriented and half of the things were all about patient engagement patient activation and one of the things that was just really prevalent was the idea of gamification of you know the patient engagement process so if we can get you know sort of the family history tree gathering data application to play Donkey Kong or something you know it's going to work Pong in my case probably but so I'm curious there are a couple of examples I know that Illumina and I don't know if anybody in the room may have answers to this but I'll just put it out there so Illumina has returned sequence results to individuals and I think that's also been done at least in a limited fashion with oh shoot I'm blocking on the other it'll come to me have there been has there been any published results about anything related to how people are interacting with their genomes I've seen some stuff related to 23 and me but but not much else in that space is anything anybody aware of that nothing dispassionate and analytical it all seems to be promotional of the 23 and me strike or inflammatory is a reactionary I've been talking a lot about what this in sports should look like and how we should engage patients I was wondering if we can transition a little bit and talk about assuming that we will come up with the evidence and the knowledge base because unless we believed that we wouldn't be here that you know genetic and genomic information is going to really be tremendous help to caring for patients and caring for ourselves assuming that's that's going to happen I think it's it behooves us to think what are the mechanisms to scale this how how practically can we set up infrastructure that once we have that really clear clinical evidence that this is how you should care for patients and this is how you should be using genetic testing that that can be scaled out widely because we all know that the likely path is that'll happen and maybe 20 years later 30 percent of clinics will be doing it that kind of thing so maybe I just wanted to see if we can transition a little bit to talking some more about that topic I guess I just react a little bit to the sequence of events that you the way you described it and that I would argue that having cds is going to be a critical part of the strategy for building the evidence so I are I'm sure this is everybody's experience but many clinicians don't even know how to access a lot of the genetic tests we're trying to evaluate so the cds regime would allow for individuals to know the tests are available that might be useful for a patient with their with certain predefined characteristics and that might be very important for streamlining and making more efficacious our ability to do pragmatic clinical trials which I think are going to be at least one important component of building the evidence base for genetics and genomics and just a little bit along those lines I know one discussion that comes up a lot is do we do this in a research setting versus as a quality improvement type of project and just curious about the experiences that folks have and opinions about the problem with research grade data is the fact that you can't act on it clinically and so then you are put in a position where if you identify say a somatic mutation and the b-raff gene for melanoma patient you literally have to go back to the paraffin block in order a CLIA certified test the test for the same thing it may be next generation sequencing it may be a PCR test so I think you know research data has a lot of utility especially when trying to make decisions about what are going to be the clinical decision support mechanisms that we can put in place but when you look at this and look at the duplication of data that you have to generate when you have just research grade data I think it really puts us in a place where we have to look at if we're going to generate a genome for someone let's spend the extra bucks and do it in a CLIA certified laboratory so that we actually can be actionable with the data that we have available to us we ran into this numerous times when we were trying to do patient trial matching for clinical trials or actually trying to put people on therapeutics when we saw that they had eGFR mutation to BRAF mutation and such we actually delayed the treatment because we had to go through the CLIA certification process so I think it probably the answer is to do both because with QI type studies you can achieve a certain scale that is more expensive or more difficult to achieve with consentive research but by the same token sometimes you don't get particularly downstream all the outcome information that you might want and unless you're in a the kind of environment where that all that information flows electronically into the right place then you're you're limited in what you can do if you don't consent the patient up front there's there's certainly room to do consentive research and run a CLIA certified test and then look for outcomes that's sort that's one of the directions that something consortium are going to Similar to our dependency on the timing of when rampant genotyping will occur the only proven model that I'm aware of for scaling in health IT related to research environments academic environment showing models of CDS and EHRs and all that followed by meaningful use and that's when it really exponentially started picking up so the question comes up we may never get stimulus again but there may be penalties that when MU4 and MU5 and MU6 come along and that's probably it seems likely that's when it would really pick up well I'll note that I mean a lot of these efforts O&C and CMS are funding it's obviously because they want to have something they can point to in regulation moving forward I would caution though that it's I have working on these things it's very dangerous to be building standards to say oh once the EHR vendors are required to do this everything will be fine because that you're basically saying nobody would use these standards unless they were required so it's a bit of a chicken versus the egg thing but I think granted those can be incentives but we really need to make sure we build standards that people would implement regardless of whether they were required just to follow on that maybe JD you'll jump into the I think if it's functionality that's exists in the current platforms then that's viable but I think in terms of what what drives the development prioritization in the commercial sector is the regulatory compliance and so that will bump things up to the top of the list if there's modifications needed to reach those ideal workflows so so I think I think I think it works both ways the other thing that I think can help drive a lot of this is gains and efficiencies to make clients faster and more optimized because right now the big discussion is around value how does the health care system get faster how does the health care system save money end up with without having never events etc and so that's really another one of the drivers like Mark said you put a regulation out there it's going to happen next thing is we've got changes in economics that are going to drive this for a variety of reasons independent of the growth of molecular testing that will push things in this direction clients are already asking for it because they they see the waste and I think because they see the waste because the money's gone down and so now what wasn't a problem because you had excesses in cash flow now is a problem because you don't so that's the other big driver out there that will help push this forward yeah and I mean frankly we used to get reimbursed for waste so right and that's changing as well so yeah but I think that does come back to the concept that you introduced at the beginning and in teeing up this section is where is the return on investment what is the you know what what sort of a story can we tell and I think we've heard several different ideas about that you know one of the questions that I would put forward is that that's clearly a very important thing to do and I know that genome has been becoming increasingly interested in some of the economic aspects relating to what it is that we're doing and so is that something that we could potentially you know create an agenda a research agenda around in terms of understanding what really is whether it's on you know through policy or something of that nature you know one following question could be more to Suzanne from where she sits with blue cross do we start to look at the driver of getting a patient's genome being sequenced not initially done out of some disease state but as an overall assessment of risk as a patient moves into an ACO environment or comes into new healthcare policies I need to know who and what you are and then I'll figure out what your and that brings a whole host of issues with Gina and all the kind of stuff but you look at in terms of that is truly a benchmark to figure out what kind of risk does this patient have both for disease and for health I think you're right that it raises all sorts of issues and so I won't go there but but I will say that as I've been listening to the conversation these are sort of some of the notes that that I've been taking you know so as as I hear people say well of course we're going to do this for the whole genome my notes are well who initiates the test right so I've been those are sort of the notes that I'm that I'm walking away from because I think it is a paradigm shift obviously and for us in the way that we review evidence we have to sort of think in another realm so I appreciate you bringing that up yeah and just to harken back to something I said earlier you know one of the presumptions I think you know for what we're talking about here is that we're relatively that will that we assume the existence of a genome you know that's within or touches the clinical activities in some way shape or form we're agnostic as to how that genome came to us but I think we're all in agreement that if a genome is available then it behooves us to you know beat the hell out of it to get as much value as we can while minimizing the potential harm from misuse of information that we don't understand completely you know of what happened so you know I think we could also argue the point that perhaps clinical decision support could be used in certain scenarios to say when you should do a genome as opposed to I think we heard some examples where you know rather than in the laboratory setting rather than doing you know a bunch of single gene tests that makes sense to do a genome and I think economically we understand that in certain scenarios that occur not infrequently like a newborn screening when you have a kid that fails newborn hearing or fails immunodeficiency where you're talking about hundreds of genes that are potentially involved you know to me that seems to be a natural place where you might invoke a whole genome sequence to try and answer the question instead of going gene by gene by gene but again that we can decide whether we want to think about do we build decision support about ordering a genome as opposed to another test as part of the discussion or whether we just want to focus more on the back end which is we have a genome how are we going to use it and how will decision support help Kim That's a question for that does anybody know what the cost of newborn screen is versus the projected cost of a whole genome sequence? You say newborn screening mean typical or what people can say is we're just going to have be doing this anyway I guess what I'm getting at is when the cost becomes comparable enough you could easily justify we're doing this anyway why not do the more comprehensive test Yeah, that raises an interesting set of questions and I've been thinking about this quite a bit from my perspective I think that we can't look at we won't look at sequencing as supplanting traditional analyte based newborn screening because I can't imagine people sitting around and parsing the sequence of the phenylalanine hydroxylase gene and trying to infer what the phenylalanine level is from what the gene tells us you'll measure the phenotype and I think again that gets into the dangerous waters of assuming that genetic and genomic information is always preferable to other information I think if you can directly measure the phenotype you can do it reliably and cheaply then that's something that you want to do now there may be genomic implications in the sense that if you have a certain mutation then that could make you eligible for certain medications and a different a treatment approach to that but I wouldn't see it as necessarily replacing but to answer your question I think that newborn screening costs at the present time range depending on the numbers that are being done between about 60 and $200 per patient And maybe I could just comment that the question of what sequencing adds to the current enzyme based screening panels are is the goal of one of the goals of N-Site so that's the the newborn sequencing program that we're working on with N-I-C-H-D so hopefully we'll have some kind of an answer to that at least I just make the point that the company that runs newborn screening for the U.S. Park and Elmer they actually have 100% of the business so they've been looking into the dynamics of genetic testing you know on a state-by-state basis and their conclusion was it has to be actionable so for them to to proceed they've been looking at it for a couple of years but it's not something they're kind of comfortable really pursuing right now except for at-risk individuals Also that there are multiple levels at which the current billing system does not fit and you say that how much does the genome cost you're assuming there's an analyzed genome that appears first year the cost is oftentimes quoted as a raging cost but the analysis cost we all know is way more than the the and then that analysis cost you could say well can we have analysis as we go if we have a physician order analysis but there's no billing system to order analysis right now and none I don't think on the horizon if somebody be able to say can we order re-analysis or specific targeted analysis maybe someone can correct me if there is something in the pipeline and then also the transfer of information from laboratory to from laboratory to laboratory to whatever clinical to whatever EHR or from one EHR to another EHR is non-trivial and there's also no mechanism for moving that information from one place to another place in a validated way so that we know that the information is you know lossless and transfer from one to the other I'll just do a hereby dragons we definitely do not want to go into the swamp of the problems with the reimbursement and try and solve the issue so while we need to be cognizant of it I don't think we want to necessarily spend a lot of time discussing it so I'll get Liz and then David just a comment on the analysis costs they're not they're not a factor higher than the sequencing costs these days they're actually about equivalent analysis costs have come down rapidly as well so I would argue against that that that's a quick and dirty analysis that gives you an overarching overarching representation of a few things about the genome can be done very rapidly but clinical level analysis for specific clinical questions is not as trivial so I would say in our clinical setting where we do clinical interpretation of genomes the cost for the sequencing component is probably about $5,000 the cost for the analysis component is less than that and the cost for the interpretation phase is about $4,000 and again I don't want to get into argument about this because my contention would be is that the costs we're talking about are not relevant because in no cases do we have an agreement about what is a clinical grade sequence or a clinical grade interpretation we're basing our costs on what we're doing and that does not reflect the standard of care and until we have a standard of care we really can't do cost comparisons so I'll just end that discussion with that definitive statement and I know you disagree but that's okay but I'm going to bring another aspect in real quick and I know you all know this but you know because storage is on the list I mean when we're talking about one genome per person with everyone in the United States we're literally talking about three times 10 to the 20 third bytes of data that we have to store there's not enough storage in this world to even handle that and then when you talk about the transportability of it especially if we're talking whole genome sequencing now all of a sudden our pipes have to be much larger and you know we're talking commodity internet or internet too they basically are not going to be able to handle this data so a lot of the pushback that I'm hearing from health systems and where I work is the fact that there is no reimbursement model that exists to pay a health system for the data storage the internet connectivity and all of that that they would have to adopt basically to bring in genomics as a technology for patient care all you really need is a variant file and that's about five megabytes so I mean size is not an issue I disagree I disagree I think size is a huge issue because one of the things we were talking about earlier was reanalyzing the genome not for CDS I mean for decision support you're not going to be sending a full raw variant or full raw genome files on these variants but if you're going to reanalyze the genome because we heard earlier about I don't want to order a new genomic test I want to order a new analysis of that genome right and you just look at the variants really I disagree because it's not just variants it's let me interject as referee here I think again we're dealing with we're giving definitive answers for something for which a definitive answer doesn't so we're having a faith based argument here as opposed to an evidence based argument because the reality is is that you know the variant files that we're currently looking at for the most part are being generated off of references that we treat as somehow a reference genome but we don't understand what a reference genome really is and so there's a lot of this that's still evolving but I think the so the point is we don't know what the right answer is although I think Dan and you're I think it's desiderata number one it's the loss of a state of compression and I don't know if that's relevant to bring up in this contact and that paper actually recommended a kind of digital subtraction lossless and recommended if NIH would support a reference sequence whatever it was that in fact was just the Euclidean minimal distance between the variants you might observe you could get something between one and two orders of magnitude of compression just because the genome's so repetitive and all that sort of stuff but I think the countervailing view would be well even if you represented as ASCII bytes it's still about one DVD of information per person right DVDs all about two gigabytes and so it isn't in the scale of clinical data imaging studies especially multiple slice CD and high resolution MR are already PAC systems are in that range so I think the physical storage although it's not in place is kind of not the showstopper I think in many people's view but it would require some work for sure to make efficient ways of doing lossless compression so to pull this back to the discussion that we're having around implementation I think there are some relevant issues and so I guess the question that I would ask related to the issues of the amount of data and storage and portability which have come up before you know what would be the things that you know would be potentially studyable within that space that would be within scope for this particular group to consider Ken one thing I'd propose is to just say there are approaches that have been proposed beyond genomic CDS really for potential inclusion in EHR certification criteria meaningful use CMS regulations et cetera the build on a lot of the work done by this community but BlackWords work the CDS consortium et cetera but they're being proposed there are pilots ongoing right now with vendors and with health care organizations et cetera to validate that these actually do work and that it is usable and appropriate potentially for later inclusion in federal regulations I would say one very reasonable approach is to say well why don't we see and evaluate whether genomic CDS use cases can be covered by those identify issues and evaluate them return back those conclusions and at that point there should be very strong alignment with the overall industry direction and instead of for example potentially meaningful use being the reason why genomic CDS does not happen because EHR vendors and implementers and health care organizations are too focused on those maybe it's a part of those sort of trajectory so I would propose that as a very I think pragmatic approach to say let's see the direction that the industry is going and see if it could meet the needs of genomic CDS and identify areas where it does not and try to get those fixed so essentially what might be constituted as a or represented as a gap analysis to say let's take this let's throw it up against what can do see what it can and can't do is that fair? paper based analysis and actually just doing it because that so in the context of the pilots we consider for our federal activities we don't consider paper based exercises to be sufficient we consider it only sufficient when you actually do it and see where the issues are because we will find issues and so I think we do have to go beyond tabletop exercises and actually do it at least if I was trying to focus on the problems at hand you know and I was putting together an RFA I would focus on a consortium potentially you know aiming for five groups or something like that so that we could fully test the standards that might underlie the CDS and they where the data resides et cetera it'd be terribly user centered and workflow centered because that's where the real problems with CDS are these days it might focus on the exact triggers the exact actions the user satisfaction with those triggers and actions whether it fits into the workflow might have formal user testing might work with color schemes dosing adjustments pushing of reports and again trying to figure out what is the model that doesn't seem to disrupt clinical workflow too much so kind of related to what Ken was saying the approach that that I've been using locally is kind of a phased approach to implementation where you first see okay we have we have this pilot study what do we do based off of what we have currently in our systems and then you have like a lessons learned or or needs at the end of that phase which you use to kind of to inform what what additional development you need to do and it seems like that's a little bit of a pragmatic approach and I don't know if others have approaches that they're that they're using for implementation be curious to hear those I'll just add I mean like Casey says it's not rocket science right so we have proposed approaches of doing things we try it and see where where the issues are and particularly the issues tend to come can you really do it where the knowledge creator side does has no contact with the knowledge consumer side because anyone who's implemented to send sport knows that oftentimes you will take advantage of the fact that you're playing both sides that you are the healthcare system who's implementing you know in our system this is how the labs are formatted and here's what the genetic test results are oh and we know that we actually have a good problem list and it's coded using snowmen versus ic9 and we make take advantage of that the real challenge when you try to scale it is how do you do it when you don't have intimate contact possible between the knowledge creator and the knowledge consumer because to scale you cannot have those end to end contacts and so for example when we took order sets from the zinc's approach and put it into a standard format and translate it into a EHR vendor format a lot of issues came up like we don't have a national orderable catalog so we need to have communication or we don't know if we mapped it correctly so these are the kind of issues that I think if we just have the question to begin with and this is what we're doing in our pilots to say if you really didn't have communication could you really do this is is the big question because if you have in a pilot setting in a research setting etc if you have the ability to contact both sites of course you can generally make things work the challenge is how do you do it when you're supposed to let this out in 2000 sites implement it with minimal tech support so let me ask a question related to that because it seems to me that you know these types of pragmatic experiments a lot of times take place in our institutions and some of us may have a developmental environment that we can play around with this before we put it into production and we all know how our production people get really testy about implementing something that crashes production what would be the likelihood that we could create a general a developmental environment that would be accessible by multiple users that could be standards-based and where you could throw a bunch of experiments at this and test out some of these things but not have to rely on institutional goodwill and trust that you're not going to break things comment on that so part of my hat is I'm on the VA's national knowledge-based systems team which is thinking about these issues and that team has developed a sandbox for this and there's ongoing work to build this into a more robust system with some synthetic data etc so there's already people have thought about this issue because it is very hard to collaborate really in a environment where I can't give you access to my EHR system because you are not employed by my system kind of issue and I think that would be a wonderful thing for NHRI to work on which is a sandbox where this kind innovation could happen I was glad to see Jim Siminoe raise his hand without me having to ask him to weigh in so well I wish I had then I could pretend I was paying attention so um now I I uh if you if you find a link between your your create a link between your EHR and info buttons and you can convey the the laboratory information in that context then you can work outside the EHR environment to experiment with different kinds of information resources that you might bring so open info button for instance once you have a link to that then you can put in links to all sorts of things and and see how they're impacting both uh practitioners and patients well let me ask a more pointed question which would be I mean I think that that's certainly one solution but I think what I'm hearing from a number of folks is that you know there's lots of different things that we could potentially use to solve the problem that different solutions may be more would may fit better with with different use cases so would there be and obviously I don't expect you to commit to this one way or the other but I mean is there a role for NCBI you know with or without conjunction with genome to say could we create a certified EHR developmental environment where we could actually do research on these types of questions is that would that be within the purview of what you would see the mission of NCBI doing? Well I don't represent NCBI so I'm I guess I can say whatever I want um I you know I I I I don't see them doing that and I you know I think Don Lindbergh has said for years that the National Library of Medicine doesn't have a health record and so I don't know if they would be amenable to that it's interesting though that just downstairs from NCBI is Clint McDonald and he's got a personal health record which is very it's definitely a place that could be something you could experiment with unfortunately he's not here right now but hopefully I think he'll be back tomorrow yeah we could I could promise on his behalf we could volunteer him for this right so so I think you know but I don't see the NCBI getting in that realm they haven't moved in that direction yet I I don't think Don Lindbergh would he hasn't shown any interest in doing that so I don't know if it's under what organization it's under I think the need is there and has been for decades and we've had it for decades it'd be similar to the practice-based research network where there's a infrastructure in place and it's leveraged many times over many years for many things the electronic health record needs the ability to in a test environment test small types of things whether it's color schemes or whether it's placed on the right or left or whatnot and I don't think I don't think the major vendors at this moment have that in place so that in a randomized fashion they can test any aspect and and they're terribly busy with all their installs and stuff but there has to be some sandbox and whether that's vendor-driven or whether it's academic or both doesn't matter but you gotta be able to test nuances to the EHR to make progress yeah I just wanted to quickly say that possibly a partner in this venture could be NCI so they're looking at the TCGA data set and putting that up in a cloud simply because of the massive size of it and so partnering with them and actually using the data out of the TCGA may help with the genomics use cases that we would come up with JD can I can I put you on the spot and just to get your perspective on this of course you can put me on the spot in terms of providing a sandbox I think that's a wonderful idea I think one of the things that we as an HIT community have to figure out is how to support this initiative because there's got to be some kind of common infrastructure around it to where it's there will always be some type of even if you take smart on fire's example not saying that's that's the that could be the vehicle of this it could not be the vehicle of this but you look at the way we work to get smart on fire applications integrated into our own EHR there's still going to be some type of little bit of legwork like that that every EHR has got to do but if you put it in a spot to where it's it's as plug and play as you can make it it's a place where we can go and test I think we've already got some help in this area if you look at what IHE does with their connectathons where various vendors come they're trying to support a standard they plug their stuff in run a couple of demo scripts prove okay yeah I can push data from A to B and the lights go off that's great we can probably corral ourselves around something like that in terms of okay here's a sandbox and it could even be a virtual sandbox to where you've got some environments that are hosted by someplace that just live on a code repo you download the latest version great everyone's happy as opposed to something that's a long-term durable yes we got a server turning in the cloud on some HIPAA place that kind of thing but no I think definitely we've got to move in that direction because floating around this is a need for not just a sandbox but also a toolkit an SDK if you will to use programmer parlance so I just want to echo this you know I think it is a great idea to create a separate environment and and our experience a number years ago trying to launch this within the partners EHR was it was just really really difficult to work within that environment and extraordinarily slow and so we stepped outside and created a system that could interact with many different clinics and be a standalone system URL based web based so any clinician all they need is internet access to and we were concerned initially that if it wasn't you know a single sign on through a physician's EHR that they weren't going to use it but as it turned out if it's useful to the clinicians they'll happily go to you know a website and use it there and so I think it speaks to the idea that this is a perfectly fine place to start at least in my mind is to create this separate you know environment and I don't you know I think anyone who has the capability to support it could be the one who creates this environment to support and it could be you know not just a sandbox it could actually be the clinical decision support environment that may function for some time until there's really a robust way to integrate it into the EHR systems that stand today before I get you Jim Blackford with CDSC was there any attempts in this in this area as you were working on interoperability of CDS? Not in terms of a sandbox we actually found willing victims slash partners and working with those vendors in Regen Street you know we're able to define the the knowledge transaction the data transaction in a pseudo standards based way I was just going to say that it seems like I2B2 might be an environment where you could do a lot of experimentation so I'll just note that the VA sandbox does actually have a VISTA implementation so it is with an open source EHR obviously with it has to be open source I mean it would be difficult for a vendor to just say here's our code for anybody in an open sandbox but I think there's a lot of potential there and there's efforts to expand it what I've learned just working in the open source community is there's a lot of demand for these kind of things that just very hard to fund so for we get a lot of requests along these lines of can you make it easier for us to use test out can you have a sandbox environment and what it ends up being is it's very hard to fund unless there's a group willing to do it so I I love the idea of a sandbox I'm just trying to think from a practical you know point of view about this is the end goal with such a thing just to be able to test it out in an environment or to test it out in such a way that we can make these decision support rules more broadly implemented and distributed to Dr. Simino's point about I2B2 as a potential platform that that's a good idea in that it also plugs into you know it's smart enabled and would plug into what JD described with with Cernar's capability so I'm not trying to put down the idea of it I'm I I guess I'm just kind of curious if if the end goal is you know do we really have are we really trying to see every how this decision support rule is going to work in every EHR vendor because given that certain vendors don't even let us show static screenshots you know without permission and get it you know getting access to and not talking about JD and you know it just it just it just from a practical standpoint of view it seems like it might be more wish than practicality although there are other open source alternatives which which are worth exploring Ken may have the same thought but you know in some ways there's lots of different ways to cut this piece of cake you don't necessarily have to have a common you know demonstration EMR platform you might have a service harness which different EMRs could plug into and if the harness is robust well described then the vendor gets to experiment with what's the presentation later look like and I think that actually is going to be much more doable than a smart architecture in the near term so I'll note I mean I think the objective for this kind of a sandbox is to have service standard interfaces APIs services etc so for example this VA environment and currently it's being done by a group called Cognitive Medical Systems with Emery Fry formerly with the Department of Defense the VA commission not only the the inclusion of for example the OpenCDS as a Disinsport Service that's HL7 compliant Vista etc but we also commission the development of new service interfaces for things like an ordering service an event publication and subscription service for triggering events communication service etc and there are open source implementations that match the now HL7 draft standards for those services so I think the intent for something like this is to make it so that any EHRs as long as they were compliant with these APIs and services interfaces could work and welcome anybody to join but to have something working because it's very hard to to work on things collaboratively when you're not actually in the same development environment I agree fully with the concept that this is not genomic specific I don't believe if we take the model of the PBRNs it's an investment in infrastructure on which you can you can test something that hundreds of thousands of providers are currently using I mean somehow this got pushed out to the entire country and there are a lot of frustrated clinicians because this is pretty tough to do it's there's not much easier than just handwriting a few things that get done we have to sort of nail down details of of order writing I mean just the fundamental unit of how to order write and do what is slick as possible but it's not genomic specific so I think we're at a point now where we'll sort of begin our synthesis of this session so I'll turn it back to Ken and Casey to kind of see what what points were extracted from the discussion and go from there so I guess just so we started out with a lot of discussion about workflows and the roles of different care providers or patients or caregivers just the roles of several different potential stakeholders and then we also we talked a bit about use cases for decision support including involving the family as well as incorporating different pathways for for genomics let's see there is some outline of scenarios when to order when to order a genome versus a single gene how to interpret the data what actions to take currently versus in the future so you want to know what to do now and then we also talked about the a little bit about dual purpose of genomic modules evolving and the interface between EHR, CDS, and limb systems also what the one thing that came up is that what the end user needs isn't always known and so whether we need to look at these in a in a research environment versus versus a quality improvement environment there are also several issues with how do we actually think of decision support from the point of view of the patient do we can do we think of the support more broadly in terms of delivering that using the pHR also adaptability of the system so once you initially get information does that change do you want to do you want to change what's displayed over time and how do we avoid duplicate genomic testing so keep so how do we support these kinds of things within the decision support environment there is some discussion about closing the loop so seeing what's the impact of what's the impact of decision support over time and thinking about that ahead of time before we actually implement CDS again there is a long discussion about involving the patients and thinking about shared understanding so maybe we might provide a family history report to both the provider and the patient and that's something that's going on and thinking about metrics like readiness to change for change also engaging with the VA which is is now providing data directly to patients we also talked about models for storage and transportability and and how to involve how to be reimbursed for these kinds of things and how vendors might be involved and well we had a long discussion that we just finished up with talking about a sandbox for learning about or for collaborating on decisions decision support processes and getting input from end users so I'll just try to add a few notes so with workflow I think consensus was just there's a lot of different potential ways of doing this and we need to investigate one thing I hadn't anticipated but seemed to be a very common theme is that we really need to quantitatively identify the problem we're trying to solve and in terms that would be of interest outside the genomic medicine community like what are the like the adverse drug events for the CPOE context kind of notion so that we can understand you know when we what we're trying to solve and when we've been successful at it I think that's really important with the art of scaling we had a lot of sort of back and forth on this but the take home I took with the whole genome sequencing is there's a lot of issues but that could be a game changer in terms of scaling and and so if we could have that data and get over the the rat the reasons why it's hard and actually have that data and not have to wait a few days for genetic test results to come back etc I think the consensus was a whole new world of opportunities would come available so I think that's a very important place for this group to pursue we also talked about the need to demonstrate return on investment and I think the notion of defining the problems in non genomic medicine but more sort of general clinical general financial terms make sense Jeff made the point that this is what is perhaps needed to develop the evidence of genomic medicine and I think that's a great point because of the complexity of actually implementing some of these algorithms it's perhaps not even possible to have an intervention for genomic medicine in many cases without distance support in the first place and so that really raises distance support from implementing what's known as best practice actually developing best practice which sort of breaks the model for a lot of the ways we typically think about evidence generation and then adoption but probably needs to be done we talked about federal regulations having a potential important role to play and also economics being a very important driver and that being in probably an active agenda for for work to really understand how does the current and moving healthcare economic landscape change what needs to be done in this field and then we closed really with talking about the notion of a sandbox and the notion of consortium based testing of some proposed standards and this really resonates with me because it's I think it's needed it's it's potentially a big win because if these efforts can align with what's going being pushed by lots of groups and I can tell you from the operational healthcare environment things that ONC and CMS are doing and requiring get inordinate amounts of attention because you have to do it if you want to get paid and that's that's a that's a big carrot so I think figuring out a way that this community can engage with that and perhaps have a sandbox have demonstration implementations et cetera seems like a very practical thing that could be done to really advance this field great thank you that I certainly seem to capture the discussion very well any additions or corrections that anybody around the two use would I could say double use I guess would care to make for that great okay so the last agenda item today is going to be extremely short the summary of day one I think the the summary is it's over we survived everybody for the most part is still is still conscious so these are all good things it's been incredibly productive discussion what's going to happen next is that while all of you can go off and and you know eat or walk or exercise or go to bed or whatever it is you want to do Blackford and I will have the task of taking the information from the different sessions trying to organize that and so that tomorrow morning the first order of business will be to come back with you about what what we think we heard where there are areas of agreement from the different sessions that we've had today that might be able to be combined so that we don't have a laundry list of 18 things not add anything to the deserrata if we can possibly avoid it and then try and do a little bit perhaps a prioritization using prioritization metrics that are yet to be developed we will lead off with that we were scheduled for an hour on that I would be very surprised if we actually use all of that hour but essentially what we'll want to do after we do that presentation is to then have a discussion perhaps focusing on certain things that we've identified that we think we need to flush out a little bit more but hopefully to then be able to synthesize a list of here are some next steps that this group would recommend moving forward and potentially even some ideas about how we might be able to move that forward so any questions or comments before we adjourn again I want to thank Casey and Ken for leading us through the last session and thanks to all of you for your very active participation and for those of you who have been listening on the web thank you very much and continue to send in your feedback via Twitter we are taking a look at that so with that I'll let you all go and enjoy your evening