 And what we're planning to do will be to kind of go through these two slides from key question 5, drill down a little bit and try and come up with some perhaps plans related to each of these items and to some degree some prioritization. So as we march through these, we'll do a little bit of straw polling here with the few, the proud, the remainder, to give you a sense of how much. Now, of course, the temptation is to vote for everything, so I'll trust you to kind of think about what's most important. Before we get involved in that exercise, Dan asked that we again kind of talk about output from this particular meeting. I mentioned earlier that Blackford and I will be taking the lead in producing a paper, not so much a proceedings of the meeting, but what we've typically done with these meetings before is to set the problem. And then we actually have some, I think, very interesting data related to the survey around the desiderata that we can kind of present, which I think would be of some interest to the informatics field in the sense of where does a group of folks that are involved in this on a day-to-day basis really believe the synergies are around those. But then talk about the process of the meeting and focus mostly on the output and the lessons learned and the next steps. So that'll be something that we'll be producing hopefully over the next couple of months. I think one of the things that we might ask, we haven't worked out all the details yet, but we might ask our co-moderators to perhaps take ownership of specific sections related to their key question and having that as part of a co-authorship. So we'll see how that goes. Sometimes the group writing exercises aren't as good when they, you know, as Yogi Berra said in theory, theory is better than practice and practice it ain't, so we'll see how that works. I think a lot of these meetings we've targeted genetics and medicine as the target journal, and in my view I think that we may want to actually either target the informatics, an informatics journal like Gemia. Alternatively, although it would be more work, is that if we can't, if we think that there's enough information to maybe deconstruct this into two papers, one that would be more informatics focused, one that would be more genomics focused that we might look at, you know, companion papers, but those are things that we'll kind of talk about at the end and decide how we want to do that. Blackford has had some initial reach-outs to the editor of Gemia to sort of assess their interest in this. Genetics and medicine has been generally interested in the output of these meetings, so I think would certainly be receptive to that. And then I think the other thing that we're looking to do as part of this is I learned something from John Steiner, who's a health services researcher at Kaiser Colorado. He's the current chair of the HMORN governing board, and whenever we have exercises like this, he likes to proceed them by saying, we doesn't live here. And so what he means by that is that when we say, well, we're going to do this, he'll say, well, we doesn't live here. Who is actually going to do this? And so I think what we'd also like, and for those things that we think are prioritized and we want to move forward, we'll probably also look to say, are there people that would be willing to potentially lead a group in doing that and be a little bit more intentional now? We may have some of them where we're all busy, this is all volunteerism, and we may not have anything, in which case we'll kind of hang back and reflect on those more within our genomic medicine working group about how we might be able to move those forward. But for some of them, I think they would lend themselves to actually having someone take some ownership. And so if you feel so inclined, if the spirit moves you, then please let us know and we will permanently attach your name to that item with high expectations of deliverables. Dan? Probably not a good time to raise your hand, but first of all, to endorse. So some of these actually look like the natural, it would be a kind of a white paper advisory to NIH staff. This one really does, I think, sit in a space where there'd been a natural broad readership, especially in informatics, if a manuscript was to focus on, well, what's the difference between genomic decision support and decision support, as we've known it for the last three decades, then what's the current state of the art and then where are the research frontiers? That could, depending upon how it's written, could go to a number of journals, but it's clearly going to find a home in an academic publication because of its natural, the broad natural interest in those kinds of questions right now. I think the other opportunity, and unfortunately I think the timing may not have worked out exactly well. I don't know exactly this, but it seems to me that this would be a really cool topic to propose for a panel at the AMIA Translational Bioinformatics Meeting. And I think the proposal date for that is past. Now, whether or not we might go the route of, I think, with our EHR integration special issue in genetics and medicine, I think we actually went a presidential pick route or a late-breaking session, so if there are some opportunities, I think we could probably put together a pretty cool panel on this, and that would be something else that would be worth at least exploring to see if there would be interest, perhaps sponsored by the AMIA Genomics Group. So the joint summit deadline for 2015 is September 25th. Sorry. That's what I thought I remembered, and I said, you know, just it didn't quite work out, but we still may have some opportunities to potentially do that. I think, well, Jesse Tanabom, who wanted to be here but was actually in the U.K., is the chair of the Genomics Special Interest Group, and is she also not incoming chair? That's right. Bob is the current chair. I always forget the timing on that. So we have the current chair, and then Jesse is incoming chair. And isn't Jesse also up for board? Yeah. Okay. So we are slowly infiltrating AMIA as well. I can say with a high degree of confidence that the AMIA Genomics Working Group would be happy to sponsor that sort of submission. Okay. So I think we will, unless there's strong objections, we'll officially add that to our, I'm sorry? I might be able to help as chair of the board. As chair of the board. Yes. Yes. That is true. I didn't mean about that. I didn't vote for you. That's because I wasn't an AMIA member at the time you were elected. So that's the only reason for that. So I think we will add that as an action item that we will propose a late-breaking session related to the output of this. Josh? We're also doing a year in review on translational bioinformatics. It's kind of a late-breaking session that Neil Sarkar is organizing and I'm going to talk at it. And so, you know, I can just highlight this as part of that as well. Clearly, I think going for a separate thing is advisable too because I won't be able to talk for long on that, but certainly can highlight it. Great. And we'll do that as well. And I think, yeah, so we'll look for other opportunities that we can kind of hold this up, not only to the informatics crew, but I think that we probably are at a, there's certainly lower knowledge about the role of clinical decision support in the genetics community, and so I think we should also look for opportunities potentially to present at some of those venues to begin to put these types of concepts in front of the geneticists that are practicing to begin to get them used to the idea that this is something that we'll be doing. There's very little content in the informatics realm at the present time at our professional society meeting, but I think that that will ultimately change. And I think one of the real opportunities, David Flannery and I had a chance to chat about the break is around the ACT sheets because in addition to the newborn screening ACT sheets that I referenced, in fact, they're developing ACT sheets around the ACMG incidental findings list. So I think there's some really great opportunities to take those ACT sheets, which are things that would be very amenable to representation through standard, but also because they're being built in with clinical workflows that lend themselves to creations as L2 artifacts and then hence for coding. I think those are, again, low hanging fruit that we could look for partnering. So we're sort of jumping into some of these aspects already, but I think these are all good outputs from the meeting that would be very easy to move forward. So let's go ahead and start to dig into these. I may go a little bit out of order here because I think there are some of them that may be a little bit easier than others. And in fact, I think I'm going to start on the second slide here because in particular that first bullet, I think that there are several things that we can move forward relatively quickly. So we've mentioned several times the opportunity that Ankh offered related to moving some genomic CDS around some of perhaps our pharmacogenomic use cases to see if we can actually go to representing these in a national CDS repository using that immunization model. And Ken, thanks for requesting some information that we can ultimately distribute to the group. In my view, that would be something that is highly desirable, is something that I think we could potentially charge the eMERGE PGX group to take ownership of to say, okay, amongst us where we're all implementing this, let's pick three to five that we can all agree on that would be good and then try and take the leadership of engagement. So we would have a who and we would have a what. So that would be at least a proposal that I would make to the group for purposes of discussion or violent objection. I think I might have mentioned this a little bit, but the VA actually has funded and is funding this work and there is already an initial sandbox available. And so I think it could dovetail very nicely. And if you know Steve Brown from Vanderbilt, he leads that group. And I think even for future work, it's very possible you could just influence Steve to get genomics into the contracting vehicles for the VA knowledge-based systems team so you can even actually have this paid by a different agency and get this work done. We've discussed in personalized medicine before too. I'm sorry, what is the VA funding? The VA is funding some contracts around developing models, standards-based is in sport, and included in that is the development of open source sandbox environments that work with Vista. So in general, but we have discussed funding things in genomics before and I think there's potential interest where I think it could be very amenable to getting that in there. And Larry Meyer has certainly been a friend of this group and has participated in a number of prior meetings. To my knowledge, I don't think, and Dan, maybe you can, you may know the answer to this, but I don't know that the VA is moving forward at the present time with any pharmacogenomic implementations. We're actually in conversations with them about implementing a project in that space and they don't see it as being clinically valid or ready. So we're talking to their research people, Dan, you may have had other conversations. But in the HIV AIDS Abacavir, I mean that, to me, there's no doubt about that one. There does not seem to be does there, but as far as we can tell, that is not one that they are implementing. So I can go back and ask again and maybe others know better than I. Okay, Josh. I was just going to mention that we do have VA as a site for Ignite and so we're moving forward quickly that we just got funded, Josh is the PI, and we have a really great investigator, Michael Matheny, engaged in burning pharmacogenomics at least to the Nashville VA. So it sounds to me, oh, I'm sorry, Betsy. Terry, I just wanted to say that the VA actually has provided some funding to NLM for expansion of terminology standards in areas that are directly related to them. And Steve Brown, who was mentioned here, is also the person around that. So, you know, I could easily set up something for us to talk about this. I know it's hard to imagine, but it's possible two pieces of the VA are not talking to each other. This never happens at NIA. Heresy, heresy, yes, amazing. If I could just add, just to note, the people building this sandbox, Cognitive, is actually now doing the bulk of the work around exactly this clinical distance board infrastructure for the Enterprise Health Management Platform, which is the Next Gen VA system. I think it's like a $180 million project. So there's extreme synergy where obviously the people building the sandbox, actually building the same infrastructure for production, use VAY for the Next Gen system, could have real overlap where what you do in the sandbox obviously has a really good chance of making it into production if it's the same people working on it. Can you make introductions, Ken, or send info? It's Emory as the main person, so you can know. Josh, Danny. With regard to the VA, Betsy, with regard to the VA and NLM connection there, is that around like RXNORM, would that potentially include pharmacogenetics in RXNORM? It could potentially. VA funding has been used to move ahead the prescribable names project where they're, you know, get something that's a little more user-friendly in CPOE for the clinicians, and so the issue would be where they would see it on their priority list, but it's not out of the realm of possibility, certainly. Casey. I just wanted to quickly add that University of Maryland through the PGRN project also does some pharmacogenomics work with the VA, so that could be another site for the sandbox. Okay, so I haven't heard anybody that thinks this is a terrible idea. Yeah, because one nuance that I think we should maybe talk about a little bit and ask for your guidance, particularly from institute types and the like, is, you know, in these kinds of papers, for example, the AMIA Board position statement on usability last summer, we actually made recommendations to particular agencies for particular parts of the problem, and that's presumptuous at a level but, you know, might help to focus the conversation for the several different pieces that need to be correlated. Is that a good idea? Yeah, I think in general it's a very good idea. It helps to have a strong scientific foundation for those and to, you know, clearly show what that foundation is. Yeah, but, you know, briefly and succinctly but in a compelling way, but yeah, it's very helpful to have those. So the group that I hear that, and I'll just kind of start to volunteer some people and if anybody falls over, you know, collapses or shows, you know, elevated blood pressure will maybe reconsider, but it sounds like the group that might be, you know, willing to move this forward and have the appropriate connections to the various pieces would be Josh Peterson, Ken Casey, and then Betsy. And I don't know, Terry, if you would want to be engaged in that. Or one of your minions, yes. And Ken Wiley, I see raising his hand back there, so thank you, Ken. Does that seem like a reasonable group that could move this forward? Just could you define this very clearly? So this is, first of all, a notification of three to five pharmacogenomic clinical decision support rules, or use cases, you know, or use cases that could be then moved through the process to be represented in the ONC-ARC nascent clinical decision support repository, whatever they're calling it. And Blackford has captured the different groups that would be there include EmergePGX, VA, NLM, University of Maryland, Vanderbilt. Bob. So one other group that could be leveraged here is the PGRN TPP project, which has eight different sites that are all implementing their own PGX rules. And we've already done initial surveys about who's doing what there. So if in the course of putting together and identifying these three to five use cases, if you wanted to contact TPP, let me know, I can set that up. That would be great. So we'll add Bob to that. I need a translation. You didn't like all the acronyms? So, yeah, PGRN Pharmacogenomics Research Network, and the TPP is the Translational Pharmacogenomics Project led by Alan Shuldiner at University of Maryland. Right. So we have the Maryland connection and a couple pieces there. And in his remaining 20 percent time as a faculty member at Maryland, we can maybe task him to do that. But, yeah, Bob, if you're willing, let's go ahead and engage that, because I think it would be nice to be able to not solely rely on one project. We can have a couple projects come together and say, yes, we all agree that this is something where the evidence is sufficient. That would be a good thing. Okay, great. Let's see. What did I do? Can we get the slides back up? I don't know. I may have hit something by mistake. Well, the next one that came up a couple of times and Ken, I wanted to toss this to you to start with. So you mentioned, one of the things that I think came out of the discussion is a very high value opportunity relates to the, you know, this developmental environment, the sandbox. And, you know, you obviously are involved in some efforts that are going forward. And so I guess the question I would start off with is if we think that this is something that would be important, what would be the opportunity to take advantage of what VA is doing given that it is open source? What, in your view, would be required to move this forward or make it available for people in this space to be able to use? So the intent always was to make it open and to make it publicly available. This is something like I've been talking to them about contributing our software, for example, for at least several months now. I think it's going to happen and it's really just a matter of just identifying who wants to engage and if other people have any other open source software they want to contribute. And I know Brandon, when he did his work, we did it with Tolden and other open source EHR, but if we can identify anybody else who wants to integrate other than VISTA or whatnot, like open MRS, et cetera, I think there's a lot of opportunities and this is exactly what this effort was meant to do. And of note, Cognitive has done genomics related stuff for the Department of Defense, the prime contractor for this too, so they have a decent amount of experience in this area. I was just going to say, we've chatted about this before, but never got to do it. Maybe taking the open knowledge repository stuff from CDSC and changing its foundation to be the open CDS infrastructure, we should consider some kind of open library as part of the architecture. At least so far, almost every group contributing to this is using the most liberal software license possible, which is Apache 2, which basically means free, and you can do whatever you want with it, commercialize it, no restrictions. So I think as long as we can keep to licensing models that basically have no strings attached, I think it can be a pretty easy collaborative area. So besides you, are there other folks sitting around the table that are currently involved in this effort? I don't think so. I mean, Brandon, Guillermo with this open info button, we're planning to plug in there too. We were going to talk with Davide Starr, try to get his work in there around building these knowledge artifacts. We've been talking to Jyoti over at Mayo to see if we can collaborate there. But I mean, bottom line, most of the folks we're talking to have licensing policies which basically mean you can do whatever you want, which makes it very easy to collaborate. So I guess then the question to the group would be if we in fact endorse the idea that this would be a great opportunity to experiment with some of the things that we want to do. So if we think back to the prior slide where there's a lot of issues related to the standards around knowledge management, we really need this type of developmental environment that can be contributory. Where would the strongest opportunities be within existing NHGRI funded projects and who might be some of the key individuals that are either here or potentially not in the room, but we might be able to volunteer in the role of we are their boss to say, hey, guess what? You're going to be doing this. So do we have some ideas around that? It is... To be specific. So you're asking what are the strongest opportunities within existing NHGRI projects for national development certified EHR environment? For what? Right. So in other words, I think what we're saying here is that we recognize that there are... Well, first of all, we recognize two things. One is that there are certain existing standards that we may or may not be leveraging as well as we could leverage in our projects, which means we could be developing inefficiencies. But the second thing is that there may be gaps in what's currently available in the existing standards that we could then contribute to and fill those gaps based on our knowledge in this specific area. And so the idea would be as actively participating in this type of developmental environment would make it easier for us to do the work that we need to do by leveraging existing standards, but also would move the ability for everybody else to do it forward in a faster way. Did you want to say something in follow-up to that? Yeah, I was going to say probably of our portfolio in the six or so that people have been mentioning repeatedly, that the two that are most relevant to sort of system-wide implementation are Emerge and Ignite. There are many other projects that kind of touch on this, but those would be the two main ones I would think. Right. And that was my thought as well. Lee? Yeah. We actually recently founded by NOAM developing pharmacogenetics and also drug interaction ontologies trying to synthesize the annotation of clinical evidence. And one of my collaborator friends in Pittsburgh also been founded by NOAM and doing similar things as well. So we're thinking about that domain of research activity can be really useful for some kind of knowledge presentation as well. So that's something we can basically contribute to this group as well. Maybe one point to try to clarify is what is the sandbox notion really going to be used for versus what is a sort of demonstration at scale project going to entail. In some ways I think the sandbox is a place to sort of test concepts, test representations, test in a laboratory kind of environment. But then we'll need vendors who will say, you know, we understand what you're doing, we understand the interface or the APIs developing test harnesses and stuff like that. So I want to make sure we get those two notions going forward. Yeah. I think that's definitely right. I think, I mean, we're not doing a sandbox for the sake of doing a sandbox. We're doing a sandbox obviously so that we can take the tools and apply it in settings other than the VA. Although the VA, it would be nice because it's the same infrastructure to port over nicely in their big site. I think the primary challenges for vended systems are use licenses don't allow it to, for example, just put it in a sandbox. So we'll need to do it in our own environments after that. I think the work that I did with Ken was largely a sandbox. We tested out and I think what we're going for is largely what we did, but that was me as a PhD student without any, you know, NHGRI funding or anything like that. So I mean, I think that's probably, from what I've heard, that's probably the closest thing to what we're looking for, but that wasn't funded by anything, any of these agencies. We all know that the work of masters and PhD students is a great force multiplier since we can ask you to do any number of things without anything other than a salary. So that is highly useful. Not anymore. Right, Jim. I think the vendor point is really important because, and maybe this is horribly narrow of me, but there's a community that thinks, they don't think in terms of open CDS, you know, unfortunately or all these other things they've been talking about. They think in terms of Cerner or Epic or whatever, and they say to somebody like us, oh, it's cool what you've done. We're Cerner too. Show me what you've done. And that's their total context and there's people that want to do this and they need it in some form that goes in terms of their vendor. I think the competitive issues with the vendors are definitely going to be a challenge, but one model we did at Cerner while I was there is we had Vic Deshmick from the University of Utah come to Cerner, spent a summer and did an internship project. We simulated allele representation in the decision support engine, gathered data, published a paper on it. So that's another model that might work rather than creating a wide open sandbox that I doubt that Cerner, Epic, there's just too many things that that could be a... From a day job perspective, we're going to be building it into our own system. So with Cerner we built in the Cerner Advisor capability, which is basically building in these apps that are triggered by the rules engine to Cerner Expert. With Epic we're looking at how we can use ClinKB, which is an approach where you can build in apps into Epic and their rules engine, BPAs can actually call out to services like OpenCDS with their latest release that we're planning on using etc. So I think because so many of us now live in vended system environments for our day jobs, we need to and we will be using operational dollars if research dollars is nice, but we will be using operational dollars to make some of this happen. I was just going to point out here in the vendor relationship stuff, we might want to just build on some of the existing relationships that we've done some of these trials with before because they were willing to do wild stuff and maybe new partners would be interested in doing that as well. The other approach is some of the vendors, for example, Auscripts, offers an application development environment, SDK for you to build stuff that they will vet and use. So that might be a possibility as well. Right. So I think the point of the vet, you can't build, sorry, let me figure out where I want to start. So at the University of Washington, we were trying to build on some very nice work that Casey Overby did with decision support rules for farm code genetics and her work was, she struggled to get things into the, figure out what a realistic use scenario would be and we struggled with the same thing and tried several times to see how we could get some realistic scenarios for physicians actually to play with that were anything like what we had in either our EPIC or our CERN systems and we found that it was just incredibly difficult. And so the Sandbox sounds incredibly appealing, but is it going to have the same limitation that what works in the, not just what functions, but what physicians interact with and the key limitation that we saw was that we're once, we were doing alerts, but the clinical decision support alerts as they fire on a patient that has all they have is genomic information or they only have three pieces of information is very, very different than clinical decision support that's on a complex patient that has multiple different notes and laboratory pieces going on inside it and that was the big challenge for us in getting kind of our limited representation to be anything close to realistic and I don't know if what you envision for a Sandbox will have anything close to realistic patients and will even be kind of representative of what physicians will actually be seeing on the ground. I'll just note the Sandbox will have simulated big data generated through other contracts that capability has been developed by Cognitive and it'll be part of the Sandbox, large amounts of simulated data. So this is interesting because it's beginning to move into another area that we have identified and so one of the things that I'm sensing in the discussion here is that to some degree I think we're struggling with the idea of Ken who knows, we're not struggling with Ken but Ken obviously intimately knows this environment and understands what it can do and what it can't do and the rest of us are saying well it seems like a cool idea but we don't really know so maybe the next step here and I don't know if this would even be possible would be some type of a demonstration or WebEx or something where interested people could in fact see what the opportunities would be, understand some of the workings behind the system and that might then lend itself to better opportunities. So shameless self-promotion at AMIA this next month. I'll be presenting the prototype of the work that we did. I submitted a paper on it and I'll be presenting on it at AMIA. Yeah and I think that that's great but I'm really thinking about a much deeper dive where we could really begin to, you know, you could walk through some of the things that you've already built in there and really give a full description of the functionality. Is that a possibility? Sure and I think just the challenge is in terms of how we would integrate with vended systems I'm just not sure we're allowed to disclose all the things we know about the mechanics of how vended systems actually work because I think that's against our use policy. So acknowledging that we can talk about things but there are some things we just cannot talk about unless it's other customers of that same system. Yeah and I think that that at this point would be okay because I mean while we all know that the end goal is to get these things into our systems the reality is right now is I think we're much more interested in saying, you know, we have to be able to test different approaches to how this can work and if we leave that, you know, the end game aside for the period of time having an environment that we can actually work in would be a good thing. So maybe the action item for this would be to facilitate doing that type of, I'll just use the term WebEx or demonstration of the sandbox environment. And I think for, we can include whoever wants to demonstrate anything around those areas and so Brandon's sandbox is ready this VA sandbox is not yet up so once it's up we can have that but it may not be like next week or two weeks. Okay so this will be a, so the tickler on this will be to check in with you to say when is this up and ready to go and then what we can do is I'm assuming that Terry we could use our distribution list through NHGRI to be able to let interested parties know. Sorry you mean the distribution list for this meeting? Yes. Yeah so as long as nobody here objects to that that's fine. Does that seem reasonable any other comments about that? Okay so we started to touch on this and I think this is another one that was of high value and this is the, I just have to figure out which one it is. Somehow it's not on here. So we've talked a number of times about understanding more about the user, the end user and I think we've probably spent more time focusing on the clinician end user but the idea of the workflow piece and the, I know it's up here somewhere. I just can't find it now. Okay fine so workflow user interactions that type of thing and we mentioned that the potential for the sandbox is maybe being a place to begin to explore some of those things but also the idea of developing a research agenda around some of the sociocultural social technologic aspects that could influence and in particular as Brandon pointed out, trying to build different approaches to clinical decision support that would fit with different use cases and so I think there are two things that are inherent in this that would be of use. One is to build a variety of use cases and that was I think on one of the prior slides. Here we go. On KQ3, the use cases defeat information to CDS systems across a variety of different projects so that would be one piece and then the second piece would be then to take those use cases and whether it be in a developmental environment or in some way shape or form perhaps even using a paper wireframe approach like Andrea Hartzler uses quite a bit at University of Washington to go to clinicians and say okay this is what we're thinking of doing. Here are different approaches to the best different methods of delivering clinical decision support so that seemed to be something that was of strong interest to the group. Did you have a comment? Yeah, I mean sort of in the classical development paradigm you'd like to do a whole bunch of formative usability assessment and then some of it at the end and I guess this is where the lab and the VISTA implementation could be really interesting may not be generally reflective of what other systems look like but we might entice some of the vendors to allow us to do some of that perhaps some of the level stuff too. Dan? Well so all of the last 25 minutes here so it has a strongly organizational development focus and I guess the question would be if this all turns on a public library infrastructure then that's also a big socio-technical R&D thing and where do we envision a prototype which is the entire enchilada including the public library a prototype of it it doesn't have to be its final supporting organization but some notion that you have to have this pooling of the accumulated knowledge and then you redistributed and all that sort of thing could be left out of all this at least as we've discussed it and I don't know if it's a separate thing but I'm sure we want to make it a holistic picture. So fortuitously the VA actually commissioned some prototype work in building that so the same group has built the initial knowledge repository with this rating system where people can rate what they want and identify resources et cetera obviously it's not something that would scale for nationwide use but I think there are other efforts like at AHRQ with USHIC et cetera because this discussion is coming up where do you have a national knowledge repository? I think that can be definitely solved by just talking to the right people where they recognize the need and are planning to build something like that. Well you could solve it that way but I'm much more intrigued by the idea that you sort of closed an RFA for the best creative ideas of how you create and it might map exactly or closely to something that already exists or maybe it's all together how you might do this using extant state-of-the-art tools and stuff which would be much more than the kind of spirit of the output of a workshop like this rather than kind of in a contracting way just extending something already or basically blessing something that already exists. And it's not clear actually if this is going to be a academic venture or if things work look good work and look good work well and look good you know could the commercial enterprise is taking interest could be that way as well. Yeah and I guess getting back to the point that you're making Dan again this relates to this sort of the exceptionalism piece which is what you were proposing specifically related to some of the genomic CDS or is this a broader perspective of which genomic would be one piece of it and that's relevant from the idea of you know if there would be such an RFA that would be developed you know where would it be likely to come from since you know to some degree we're providing some level of guidance maybe that's the wrong word you know to genome the convener to say okay is this something you should be involved in or not involved in. And so they have the option to either it's important enough to genome that they want to do it themselves so they can find partners and especially for things that have broad applicability and I think there are multiple right solutions here but my guess would be it's a socio-technical thing the technical thing does have a genome genomic medicine specific piece the socio piece of it it looks like it would be entirely reusable if you had a public library right could be all decision support. Although there's potentially interesting policy issues related to some of the regulations relating to genomic information as opposed to other information where it is treated somewhat differently that might present some unique or at least challenging issues to stand up a library of that type that would be within the policy realm to potentially explore in terms of the implications of or whatever related to this type of activity. You'd hope that doesn't lead to kind of separatist insulated development because it clearly speaks to the notion of escalating complexity in clinicians having to navigate in a knowledge space that's just way way too deep in all dimensions and so I think taking advantage of genetic exceptionalism would be good in the short run but it generally senses that it would be bad in the long run. Just to note we're talking some production level issues of how would we stand up something that is the knowledge of prostate for genomic medicine in the country, that kind of thing. I think we just need to keep our expectations for what we do in a sandbox and etc. To not get to that level I think we should build initial prototype show how it can be done and then let other later mechanisms approaches to build the production great things or to have commercial entities step in to build I think the key is to define the standards and interfaces so that whatever we build can be replaced gracefully by better things later. And Eric who just left has repeatedly said that NHGRI and Terry as well they view their proper position is to be trailblazer or pathfinder support those kinds of activities and then if things demonstrate their value someone else kind of does the hardening and scaling it up and all that sort of thing. And this seems like it's in that space naturally. So to again maybe pull back a bit and make sure I'm clear based on the conversations that I'm hearing that the idea of this the end game the library the repository clearly has to be on the radar screen but perhaps has the potential to if we start with that or at least say that we have to have that as piece of where we begin that it could potentially slow down some of the early work and so that what we might want to think about is staging this such that we would begin to do the experiments with the use cases and that and with the idea that as we learn more and get them to the closer to what might be considered a production version that we then begin to think about that repository which maybe you know we may have more options at that point in time. Although I'm reminded that in the National Library of Medicine's long range plan Ed Feigenbaum said I recommend the library repeatedly be extravagant in your demands on the technology because if you set the bar low and you think that all you could achieve is this that's exactly where you'll end up whereas if you set the expression very high you may still end up at the lower level but you have a pathway to something bigger so it's I think in the nature of these workshops to view a broader better world and not be constrained by well we couldn't possibly do that as extension to what we're currently doing. And I would agree and underscore the notion of describing the ideal state inclusive of all its components holistically but then parsing you know into incremental research paths to get to that vision then they will actually can coordinate. And we have another long range planning effort coming up very soon at the library of medicine so this workshop can be partial input to that as well. I do think that there could be easily a distinction between what would be the most robust you know longer term instantiation of a public library of CDS in this area and other areas and to get back to Ken's point what we might do to illustrate the utility and the immediate benefit of this by integrating it with some you know at a different level with some things that people are already using you know or the place where the next set of things might do so I think probably we want to do both of those things. Alright so you may guess what they'll be talking about the research aspect of the valuable information that will be there and the clinical decision I mean I understand 80 or 90 percent of the research should be medical informatics but I would challenge you to expand you know your perspective on the opportunities that are unique with genomic data that are not necessarily of the same quantity and perhaps quality as other types of you know medical tests that are being administered and this is the volume of the data and actually the opportunity to take this clinical information and even information about decisions made based on that data and then do research that's more focused on mechanisms biology and so on using that so to speak phenotyping data and clinical decision making with outcomes and so on as a kind of phenotype very complex phenotype and then look at genomic data and patterns biomarkers and so on that maybe then be mined from it. Now the kind of genomic data that's then used is actually much broader much larger than the one used in the decision making and it may not be validated but keeping this data integrated and making this translational research opportunity available to genomic scientists and others actually would be of high value and it may translate in the long run into the knowledge that will then feed back into your knowledge systems you know validated tests and so on. I'll just push you on this point. Could you envision developing a set of use cases to describe what you just articulated that could be then potentially tested in this type of an environment because we've listed in our examples here things like BD2K as a potential source so from my view the broadest representation of use cases that we could have with owners that want to test just what they can do so is that something that you would see as feasible? I definitely do. One example and I'm not a medical doctor I would need input from you but one example that comes to mind is searching for predictors of adverse outcomes or of predictors of lack of consistency of certain relative new drugs preventing harm based on genomic biomarkers that may lead then to a better understanding of biology or pharmacogenetics and so on. So in some ways some of the things you've mentioned there are almost and this could actually relate to some of the public health issues that we talked about. Some of this would almost be a post-market surveillance type that would definitely tie into something that FDA is clearly interested in and others are clearly interested in so that would I think would be a potential utility. Sorry can I keep interrupting you? So just a few notes I mean obviously this kind of sandbox would not have actual patient data so we could create the mechanisms to do the predictive analytics etc and then port it in your own environment to work on it and I think to the zenith and I think it makes sense I think it just makes sense to do the typical agile thing where before we sort of try to do perfection let's have something that works and then incrementally make it better so we always have something that works. So what I'm the way I'm thinking about potentially again I'm not hearing any descent around the group about that this isn't a strong thing to do and I think the prior discussions reinforce that this is interesting. So I have a number of groups listed up here and it seems to me that you know maybe the first step in this would be to reach out to these different groups and again this is not necessarily an all-inclusive list and some of them of course aren't even in existence at the present time although we anticipate they will be shortly to say we would like to solicit from you cases we would obviously have to provide some parameters about what we would like to see and to some degree some of the formatting would need to have some consistency across it but then we would begin to have a collection of things that would span across the diversity that we want to look at that would be the where we could then use that as the jumping point to begin to choose those that we can test and learn from so that seems to be a relatively low energy sort of recommendation from the group that would be a request I think that would presumably since I think most not all but most of the ones that are listed up there are NHGRI funded is that a reasonable request to make of these groups to ask them to submit something like this do you think Terry so again you need to make this the request fairly specific and make it also clear what the what the objective is because not everybody would have been part of this conversation but yeah I think we can certainly ask and do we have I know that Caesar and Emerge do the other groups also have sort of informatics interest or working groups or something associated with them that would be the logical point of contact UDN does I'm not sure about ignite maybe maybe Josh Denny would know so but you know sending it to the chair which is Jeff Ginsburg unfortunately I had to go you know that would be the way to explore with them there's not an informatics group in ignite yet is there or did you start one we keep talking about starting one but it hasn't happened yet it will probably spontaneously form at some point it will spontaneously ignite is that what you're saying Josh Adam let's hope we don't have any spontaneous ignition here I sorry I had scheduled a conference call over the actual break so I missed most of the conversation have you discussed any more international kind of aspect to this I know groups like global alliance and G2MC both are focusing on a international slant for IT and bioinformatics might be worth sort of thinking about bringing that context into this discussion well I know that the GM 5 being was it was that our global meeting 6 the active working groups that came out of that meeting actually is the informatics group and yes Steve Blau from inner mountains so and that's that G2MC that's in that's that's G2MC yeah again we're we're in acronym help yeah but the global genomic alliance for global health GA4GH yeah has also a very active bioinformatics group but remember they are mainly focused on research and pooling data making it available for research so how relevant they might be to this I'm not I'm not sure right but certainly I think the the the the informatics group that Steve is leading would would certainly be able to do that so we should add that to the to our list up there as well Steve Blau B-L-E-Y-L and it's actually chaired by Steve and Eric Heligo so from Estonia there's a lot going on in Europe it would be cool but rather daunting to engage all of that but something to think about at least yeah but one of the things that we did articulate in our key objectives was national and international so I think if we can demonstrate that we're actually trying to move it from that perspective that would be a really good thing to to do so I think that that's probably the the initial action item now that the question then would be is you know who would make the request and sort of who would then collect the use cases and then move that forward and I guess that's sort of a thinly veiled request for volunteers related to that one group that could not volunteer is any government agency because this in a way is kind of soliciting for things that we're not supposed to solicit for so so it couldn't be us even though in some ways it might be logical for a request to come through us but some of our collaborative groups potentially might be interested and I know we have joint calls with Emerald Caesar so maybe maybe Brian you and I could potentially get started with those two groups and just kind of see how that how that goes and then we could potentially extend it from there is that okay sure let's talk later alright I think that that probably is sufficient for that right now so let's take a look at some of the other ones that we have listed here I think the other one that we've spent a lot of time and there's been a lot of interest in talking about is the end-to-end project the how that might look and in my view this is probably something that's much more in a visioning or developmental stage in other words what would this actually look like we've talked over the course of the two days about the different pieces that would be potentially needed but I don't think that they've necessarily coalesced into an obvious way to move forward so it would seem that if there was a group of folks that was interested in sort of laying out what an end-to-end project might actually look like almost like a fish bone or something of that nature to say where are all the different components that are needed what do we have what don't we have where the potential things that need to be solved or that could be studied that that could be of some use so I just put that out there as a potential for discussion I'm not aware but I thought I've heard notions of Vanderbilt work, Josh and Josh and Dan you know is this something that could leverage predict work and think about an end-to-end project with predict absolutely in fact we've put together some proposals and that sort of thing to do exactly that so are you volunteering Blackford no predict is volunteering I heard Josh do you want it I know you're involved in a lot of different things but would you like to lead out on this Josh I'll turn it a little more about the details and then I'll get back to you but it sounds very promising for predict I think it's probably a good use case I'm sorry what does it pay Terry just said twice what you're currently getting so but it's all psychic income Canadian dollars Dan so let me ask are there other folks that are particularly interested in this or might be willing to talk with Josh around this area going forward so Sandy and Lee so I'm speaking for my friend Paul he just left so he is actually in charge of a couple projects implementation in Indiana University so he's very I was very happy to work with whoever in this particular component okay I'm sorry just to hop on he's also part of Ignite and we have very similar overlapping projects so there are a lot of collaborations there okay that sounds good and it's always good to volunteer people that aren't here and can't defend themselves when I was raising my hand I'm happy to work on this but the IOM group has also been beginning to look at what are the challenges associated with each of the different actors that need to come together in order for end-to-end clinical decision support so we could share information as that develops too okay so to have you as part of this then I think would allow for that type of coordination I think that that's a really good idea so that seems like a reason, go ahead yeah I'll just refine my answer with all the volunteers I will give it a big thumbs up from my point of view I think having members of this group join would be terrific okay great one of the beauties actually of having contributing to the weaving of this vision is to compare and contrast and take the best of whatever is being done anybody else interested Mark Hoffman will also yeah great one of the things we haven't talked about and this is this is something that has been successful in as an outcome of some of the meetings as successful for others has been the role of the larger group in terms of continuing to get together to kind of hear report outs from the different working groups so one potential outcome is to have this group with some working groups that could then be reconvened over time to potentially do that now one of the challenges of course with this is that for convening a group like this there are costs involved even if one tries to do this telephonically and there's been some experience with the inter-society coordinating committee in terms of convening this I think had some success and so but I also know that the capacity of the genome to manage all of these is also finite and so this may be leaping ahead a bit but now that we have three or four different things that are coming out of this that I think are engaging the larger group in some ways I could see value to bringing this back to the larger group for comment and coordination so I don't know if that would be potentially on the table to explore he says looking at Terry we can always consider it I'm just reluctant to commit to it mainly because we're struggling to keep up with the ones we have and I don't want to then sink those as well as not do a good job on this one so I guess we'd want to see how much of a need there would be and I think most of the work it's done on working groups anyway and to the degree that things could sort of be sent out for broad comment if people want to comment that's great we could put up together a website where they could look at things and comment on them even passively arranging calls and having minutes and that sort of thing gets into a space where we just may not be able to handle it so that raises two questions one is I guess the sense of the group in terms of the value of having input and doing it through electronic means the second is would there be another entity that could potentially be the convener going forward of the group whether that would be Amyia genomic centrist group or something of that nature so I think the IOM's initiative initiatives which are roughly clustered around the learning healthcare system and its various components is I mean they're also resource constrained and bandwidth constrained but it would be a natural alliance if not a home if they really get fired up about it so maybe Adam I don't know if you want to specifically comment or deny or now that Dan has put you on the spot yeah I wasn't quite ready to be put on the spot for this one so there's a lot of activities going on at the IOM around the learning healthcare system some within my group some within other groups so I certainly couldn't speak for the entirety of all the activities that are going on we're currently hosting our action collaborative on genomic information integration to the HR and I think aligning having for instance the two of you that is going to help bring these groups together at the moment I couldn't commit to actually housing another group like this though this is a bit beyond our current constraints so let me just maybe take a straw pole and just get how many people around I'm going to give you three options here one would be how many people think it would be essential for the large group to get together to review the work how many people think it would be great if it could happen but not essential how many people think that would be a big waste of my time so how many people think it would be absolutely essential that a group like this would reconvene periodically so CERNER is willing to take this on and fund JD this is I would never have thought but given you were the only person to raise your hand it's natural to me Kansas City okay I'm sold and this may be the first public-private partnership I mean I had to add that to your resume for the year-end report right? that's really phenomenal so we'll I think what we'll do in the spirit of that since Blackford I committed him before he actually truly said yes it's like dog shaming on YouTube but at any rate since we're going to be participating there then we'll just as part of that plan to explore how we move that forward and then think about when would be the right time to reconvene the group but that would be fantastic great thank you JD thank you and I think it's extraordinary leadership and enlightenment for a vendor to take that kind of position but one might ask would you be willing to have some of your compatriots in attendance in observational status or participating? absolutely because the place where we would host it it's in our vision center where as Mark remembers it's where we host client activities there's a very large amphitheater auditorium in there that doesn't really say serner I don't even think there's a logo inside and so we can set this up as a very benign vendor neutral way I mean for example on a much smaller scale we host the CLMA chapter meetings in the Casey metro area where you've got various EMR various LAS types clients if you will coming to that with no issues whatsoever thank you that's perfect great so let's see here we've ticked off a number of these those are the ones in the list of the two slides here that I think there was the most excitement and discussion about let's just kind of go through the the remainder and get a sense for where people are thinking it might be good because we have we all have a limitation of bandwidth and while it would be great to do everything I think we also know that that's not realistic so let's just kind of get a sense here so the ones that are kind of leftovers here are the business case ROI clinical epidemiology I think the ideal presentation layer that is now sort of fitting into the sandbox environment so I think we have included that one in our broader discussion about the use cases we have this large area of standards which is always can represent a bit of a morass lots of lives have been lost in the that's related to the sandbox though yes so okay so do we want to declare that as part of the sandbox yes okay great that's good that works for me we talked about the demonstration project we haven't talked yet too much about whether or not engagement with public health outside of the potential use case around newborn screening which is one of the largest public health projects if not the largest public health project that is focused around a lot of genetic disease and then the other thing that we haven't talked much about at all is the role of patient caregiver and how they may play a role in this we talked about the exploring different types and then the funded CDS center which came up in the discussion so let's go back here and just kind of by show of hands how many think that developing some work around this case return on investment would be a really important thing for this group to do just raise them high and I'm going to hold up Jeff Ginsburg's hand because he's not here and he would certainly be voting for that okay so I probably about half the folks okay clinical epidemiology health services research what is baseline for genomic clinical decision support okay perhaps a few less there public health versus the public's health public health role in some of the issues relating to maintenance of genomic information portability those sorts of things how many people would be interested in exploring that space well we hadn't we hadn't put those yes I think that's correct although I think the point that was being made earlier was that you know our current EHR systems do in fact have a role in some of that public health reporting and that you know again this may be something that's a little bit farther down the road Alex would post market surveillance fall into this category well that's true we did bring that up and we didn't specifically attach that to anything but that would be something that would clearly I think fall into that that's an overlap between research care yeah public health yeah so we should I think it would be fair to add that under that public health role I would be interested in that so if we so if we I'm sorry Jesse just volunteered by twitter oh to do to lead the public health eleven minutes ago anything about this then anything that has not attached we can for okay great I like that okay we are pulling up Jesse's twitter feed so Blackford will parse this in just a second so so the things that I might add to public health to kind of again compress our list a bit would be the post market surveillance and then I think also the perhaps the newborn screening piece could be a part of that as well I think I would probably not include the patient facing activities as part of this at the present time that well I think that is a separate bullet okay so any any disagreements with that sort of reorganization so given how we've reorganized that then what's the level of interest in terms of what we're constituting is this public health public health view okay didn't help as much David not holding his hand up even though he's running the newborn screening network so oh okay and then on this one the exploration of the role of the patient caregiver for genomic CDS so this would be patient facing decision support patient ownership of data transportability those sorts of things and again the idea that this could be potentially a role we're looking for co-funding opportunities with PCOR and I didn't mention before that you know with the different PCORnet networks both the patient research network and the healthcare delivery system research network that are being pushed together there's an infrastructure there that you know addresses or overlaps a lot with some of the projects that we're doing that are looking at EHR data and patient enter data now they're not nearly at the level of maturity that a lot of our efforts are at but they're certainly making an effort to say how can we aggregate data across you know groups of systems that have tens of millions of patients to do this so something that PCORI is called out specifically in their enabling legislation is a focus on rare disease which they frankly have not had a tremendous amount of success in terms of finding projects that are fundable in that space and so it is an area that I think gives a leg up if that was a potential area of focus for what we wanted to do in the CDS realm so with those additional areas of expansion what's the level of interest around the role of the patient caregiver how many people would be interested in exploring that okay we have a few that are certainly interested in that area um so I think whoops I think there was probably most interest related to the business case return on investment it seemed to see more hands raised around that so let's have some discussion about what a project might potentially look like what would be the components what would be the sorts of things that we would want to study I just want to mention there's already a health economics U01 that is funded and includes a number of genetic use cases I will say that um that is not well um defined within that group would be the technology side and so to the extent that this might be focused on technology I think it would be a nice compliment to that work and remind me I know we responded to that one and and didn't make the cut but is that through NHGRI it's the common fund oh that's right it was a common fund initiative right okay um so again thinking about the opportunity for collaboration and maybe your perspective and maybe Terry's perspective as well in terms of how we might be able to engage with that group to perhaps explore some of these opportunities so they are right now trying to put together a large stakeholder meeting in February and so one approach would be to have a representative that is focused on the investment technology side at that meeting that's going to happen in February so I can certainly be a conduit to see if we can get a space of the table well that would be a relatively straightforward thing to do Ken I think when we think alright there are several things we need to keep in mind one is RI for whom and so who the stakeholder is we're trying to show return on investment on and assumption is it's going to be for for example policy makers but just being very clear what the underlying payment model is that's going to influence it because depending on the payment model the RI will differ and the other part is that most healthcare systems have a very very rudimentary or nonexistent cost accounting system so a big part of return on investment is identifying cost and most places simply just look at charges and sort of estimate so I think that's a really important aspect and I'll just put a plug in we undertook at our university a really extensive intensive effort to rebuild our whole costing system we were just like sequestered in an office for like five months straight you know not allowed to leave kind of thing so I and that a publication on is just about to come out in Jamie it's just in production right now so just one thing I think I learned is most healthcare systems actually have no idea what it costs to provide care and so if you're starting with data where it's extremely vague and inaccurate you could get very wrong ROI results I think just taking some best practice principles of how you do cost accounting should be taken into account when doing the ROI yeah that's a very I can't agree more with that so Jamie maybe I might put you on the spot a bit here since I know that you know O&C has certainly been thinking some about this you know what your ideas might be in this space about how we might look at a business case or ROI I don't really have specific ideas to share at the moment but I would say that if there's any follow-up discussions I'd like to be part of it that's probably all I want to say because I haven't been part of some of the CDS discussions at O&C yet and so I don't want to misrepresent what they've already been talking about okay fair enough when you think about this we have to keep in mind our previous discussion where we were discussing how do you separate the genomics from the genomic CDS and how it seems like that could get really messy in trying to create an ROI so you have to be really clear on what it is let me just add maybe one more thought out of the box which may have to do with business cases I mean that are alike investment kind of reasoning that's actually specific for genomic type of information there's certainly more cost involved in it right so the question is what would the potentially pay off be so imagine a healthcare organization that may maintain good electronic health records and genomic layer of information for you guessed it research purposes right I've been talking about it all the time but what would be a business case for them it would be engaging pharma companies who could then do either clinical trials or post market surveillance and so on and that may be a huge investment actually for maintaining electronic health records signing physicians to participate and then profiling to generate research quality data genome white and that data may have a dual purpose you know being used for diagnosis choice of therapies but as well as using as a kind of genome white surveillance screen in say post market and so on for pharma companies who interested how their drugs are doing when applied across a large population I think on this thread one I'd just like to point out you know there are of course the detailed return on investment analyses that are done post hoc you know after technologies implemented what does it do in addition though lots of institutions do ROI estimations and simulations and CITL did some of that previously and that can be very helpful just from a modeling sense to get a sense of what is how big is the problem how big is the opportunity and you know that doesn't really require a whole prospective data gathering but requires an intensive modeling effort I think I guess I've just learned so much about healthcare financing of late so I can share a few perspectives of what people in the healthcare you know sort of operational area would be looking for in terms of ROI so one is in traditional inpatient payment structures you are paid a set amount of money depending on the diagnosis related group and the severity adjusted diagnosis related groups the MSDRGs so in inpatient setting there's typical US hospitals there's already an incentive to reduce costs for a given admission because you're going to be paid the same regardless in most structures so an obvious ROI in inpatient setting is do you can you reduce costs can you shorten length of stay etc a similar part is when there are certain conditions where you're not paid for insurance reducing readmissions another part a lot of groups are starting to get into managed contracts and bundled payments and prepayments for certain populations and there you have all sorts of incentives to manage your patients and prevent them from going into higher cost care etc so I think there are if you just talk to the you know folks are running hospitals health systems it's very clear where the incentives are to show ROI and if you show it in those circumstances you're very likely to be able to find areas where you find that you know managing a population of patients with diabetes or hyperlipidemia and you can reduce their overall costs you will get investment in it and I think that kind of ROI could be very helpful okay I'm sorry Brian it's an interest the concept of ROI for this is interesting because the investment is really in the labor of the people in this room and other people will be working with us hopefully for this type of project and that right now is completely unknown and I think that that has to be defined at least in some way to be able to what we're looking right now is a cost savings you're describing cost savings which is only one side of the equation of return on investment and I think that how well this type of clinical decision support is distributed and how interoperable it really is will have a huge impact on what the actual investment institution has to make in order to do these things functionally if we get something that's minimally interoperable that requires a huge setup and huge customization at every institution then no matter how big the cost savings is then the equation is not going to help I doubt that any implementation done in the context of a research grant is going to show ROI if you actually took into account how much effort went in to identify cost savings and identify the theoretical potential for the ROI when you can do this efficiently across systems in an interoperable manner so there's a couple things that occur to me from this discussion one is is that as we collect use cases one of the criteria that we could use would be to say what do we anticipate would be the impact if things work the way we thought that they work what would be the impact of implementation on avoided costs whether it would be avoided at adverse events or something like that and not that we have to do formal calculations but if we can get a rough order of magnitude you could prioritize use cases that there's consensus around the table that says if we could really pull this off this could potentially move the needle as opposed to this one which is fun we've been talking about the other case but the reality is is that we're talking about very small numbers of patients that are being impacted by that so even though it's highly important for those individuals the cost reduction potential for that is probably fairly minimal so that would be one criteria but the second thing that I heard is that perhaps it's just a bit early to be launching into this that some of the work that needs to be done around that usability interoperability acceptance needs to be understood because to really calculate an ROI you need to say well how much work do I need to do on the customization and other things before I can really stand this up and see if it works and if that's the case then this may be something that we would look at 6 to 12 months down the road Ken? I mean you can always start with these costly efforts I mean it doesn't take that much effort to say who worked on it and what's their salary and how much did we spend on software and hardware and just in a one hour meeting you can get an estimate of well if we were doing this using standards etc what do we think it might have cost so I think regardless it should be done the degree of sophistication is different but we at least should have some understanding of what the ROI is so I think what I'm hearing is that for this effort in terms of gathering the use cases which I think Brian and I volunteered for actually I volunteered Brian to be completely accurate is that one of the criteria as we build them needs to be some sort of an assessment of impact and cost of care an assessment of cost of implementation or something of that nature and that maybe that's where we start let me just add that you know one way is to looking at fixed revenue and just you know cutting costs another is at looking at additional you know source of revenue say from research organizations such as pharma companies that may actually fund you know genome white profiling and so on maybe they get some value out of the exercise so this is where the research aspect comes in I don't know if you're open to thinking about it maybe it's also the scope of this you know group but something just throwing out one more time you want to join us yeah I can help you know whoever takes the lead on this yeah okay great so that way we won't we'll make sure we're not losing that that research and we'll have a positive and potential value proposition for an organization that could be realizing you know developing additional income streams just curious and geisinger are you thinking about that how are we thinking about that well actually we're beyond the thinking stage and I think as most people know here we you know are partnering with regeneron and you know the idea is that with the ability to mine clinical hypothesis at least that regeneron is investing in is the idea that they can do more rapid discovery for their potential agents in the pipeline based on mining clinical phenotypes and then use that to identify those that are most likely to respond and so hopefully cut some of the development costs so that's that's their value proposition that they're testing with us the value proposition for us of course is that we are being funded by them to improve our ability to rapidly phenotype so that we can use those for either clinical or research purposes on our side but also the idea that the sequence data that's being generated by regeneron comes back to us with no limitations in terms of this clinical application so we'll have a lot of data that will be able to you know put into preemptive genotyping data that we could potentially use for clinical purposes. The structure of clinical decision support may be important because if you have standardized practice managing of electronic health records and so on then that helps adds the utility to this cohort right? Yeah I think that you know from our perspective the developmental costs of this are going to be much lower because we have this sort of other income stream that's funding some of the structure that's needed to to make it up and get it up and running. Yeah so that's the case in point that's exactly what I was thinking about. Great. Kurt I think you had a comment that you want to make? You know most of when I look at the return on investment for this I'm kind of struck about how we're talking about doing this you know I'm one of the naysayers about the utility of pharmacogenomics in general and that I don't think that's proven particularly effective because we don't have very many examples that it's a huge deal to make a huge amount of infrastructure and similarly you know where genetics has been the most useful right now you know based on the CSER project things like that as explaining you know diseases by genomic sequencing where you end up reporting a few variants the part of genetics that hasn't been done very much that it seems like it's in our future given that most of the traits the complex traits that were interested are quite polygenic is to use another approach and the other approach you know there's some skepticism about whether it's will work or not but I think it's I think that's been misguided and that I think it's more useful and that is in predictive genomics there are some examples that are quite interesting and you know people can argue that we're very different but you know substantial fraction of the cows are being genotyped every year in this country and the reason why is they've been using to figure out how to improve the herd quality and it's been going on for 15 years it's changed the improvement of quality of herds from the historical average of half a percent a year to over two percent a year and it's been worth billions and billions of dollars to people in that business of you know getting better cows choosing the best bull for the next year you know in the United States we think about healthcare and what we could do with this but you know in predictive genomics there is the potential to identify is drug A better than drug B for person X and I think that's actually something that we don't put in our model right now because the best return on investment we have is reporting in the medical record rather simple means this person has a BRCA1 mutation and I think you know thinking forward in the future where return on investment is it's going to be in a model that requires good access to phenotype data ability to share across large populations information from you know good quality genomic data I don't actually think we need you know whole genome sequencing to do that you could do it with a good GWAS chip and you know that can be done relatively inexpensively in this era I think it's implicit but just want to make explicit that I mean we're talking about cost and revenue etc but it's been shown that improving quality of care and outcomes generally results in better financial bottom line so I don't want I just want to make sure that we're on the same page that I mean ultimately we're about improving patients it's just also showing the downstream financial benefits of it as well I mean obviously all we were doing this was for you know reducing cost and improving revenue then we're probably in the wrong line of business but I think that's you know that's a really good point and shame on me for forgetting that but this now also I think would be implicit in the end to end project that we're talking about because as we look at the patient outcome then that does in fact have an implication related to cost of care and potentially could track back to return on investment so I think what we're doing is we're kind of taking this and saying you know we can put this into a couple of other places that we're interested in in exploring and not necessarily have it as a standalone and just pragmatically you'll need to show the financial ROI to convince an operation to invest the system resources to building these kind of capabilities but in order to convince clinicians to actually follow these guidelines and to use them they will not care about the costs they will want to make sure that it's improving care so you need both for a very pragmatic reason because you need to convince two stakeholders as Brent James the dinner mountain says cost and outcomes are two sides of the same coin but which side of the coin we pay attention to depends on your perspective unfortunately that's the reality okay any other discussion about the ROI business case okay so there wasn't I think what I would propose is you know to not take off the list but to perhaps you know put as potential considerations for the future the role of the public versus the public's health and the role of the patient caregiver for genomic clinical decision support although perhaps with the latter one if there's no objection we could add that to the range of use cases that could potentially be explored and that would be a relatively probably a fairly straightforward thing to create and just kind of test that out so if no one you know thinks that's a bad idea we could just add that to the list and so we're at least doing something in that space really none of our projects work in that area is there a group that we could link well it just so happens Terry that I'm very close to a group that is looking at it precisely this no this would be us guys in your since we're internally funding a study looking at developing patient-facing genomic test reports so I could put my name rather than we next to the use case there in a sandbox we even have a personal health record that we could test as well so I think that fits into that sandbox if we have some use cases that we want to prototype and do that in that sandbox I think it's doable and the interesting part of our research is we're actually doing ongoing engagement with our families of these children which have complex disorders with semi-structured interviews to get that information back and we're learning some very interesting things about that we tend to again focus on things like patient portals and this sort of thing but in the wilds of central Pennsylvania issues of connectivity and that sort of thing are problematic so people are saying could you just give us a flash drive that has this on there we have the links and everything but then we don't have to spend hours trying to connect up through the patient portal to actually get at something so those are the sorts of things that we're talking about that we're talking about from the physician standpoint that are also important from the patient family caregiver standpoint so Brennan just referred to the sandbox as something that was up and running and available but obviously we don't have one so do we need to talk about where that would be or who would host it or when is an appropriate time to bring it up because it seems as though it would be related to a number of the things we're talking about so the key folks just really get this discussion going with Steve Brown because he's the sponsor for this or to start and I mean the idea would be sandboxes usually are created using virtual images of virtual server images etc and then if anybody else wants to host it somewhere else you just pay for an Amazon cloud service and just host it elsewhere but generally we create sandboxes so that it's easy to transport between servers it's in the cloud that's the answer to everything now right so it's at the Vanderbilt cloud okay so I want to come back to you Terry because one of the objectives of this was to look at the potential for developing information that might be of interest to NHGRI in terms of developing you know funding proposals going forward not that the be all of this was to say okay NHGRI is going to commit to funding this much in this space but are there have you heard what you need to hear to take back to your colleagues and say well within the space here are some things that are of potential interest that we you know might want to explore further facilitate etc yeah well yes and no so yes in terms of lots of cool ideas in lots of areas I think the challenge we have is that we're a small institute we have a pretty specific mandate and it's not entirely clear to me how we can you know bring forward this mission on our own obviously we don't have to do it on our own we have further institute partners in the room but I think what we would need to do would be you know this isn't something genome can do by itself that's become pretty clear to me and it's wonderful that Jim and Jim and others from NLM are here because they do seem to work in this space in terms of medical records and standards and that sort of thing to the degree that we can bring along other institutes that too would be great it's not I haven't heard necessarily you know a couple of use cases and they may come up that would identify you know an obvious other institute partner and if anybody has those in mind that would be great to hear or if those of you who are working in the clinical decision support realm know of of institutes that are supporting work in this area and we can we have databases we can search for that and we can obviously talk to our colleagues but if you know that oh institute X is like really committed to this area that would be helpful to know as well so the charge to the group then that we could you know put out in our summary would be to you know if there are potentially willing partners around a specific area that could work together with genome to at least propose those so that there could be some exploration right and I guess you know maybe I could have phrased that better we at NIH should know what each other at NIH are doing just like other parts of the government show know what they're doing but if you're aware of particular areas that maybe not are not even on the radar screen of a given institute but should be you know so that the disease X is something that's you know really hot in terms of decision support it presents all you know the huge spectrum of problems that genomics does and here's a real opportunity you should go talk with them that would be helpful to know and so let me just ask are there areas in decision support that y'all are aware of that are like this well so that's the current decision support portfolio question I think my I I extended slightly to receptive institutes that have the problem that may not know they have it so I think there the new leadership of CBIT for the National Cancer Informatics Program would be a natural group of folks to host the conversation and I'd be happy to since we're the advisory committee nudge there an elbow in the ribs to at least start that conversation to see if there is common ground and Warren and David Patton and Tony Colard are interested in that I have spoken to them in depth about that that is possibly one area so it sounds like we have at least one potential idea let me come back and again I think you can kind of tell given you know that this is a personal interest of mine in terms of the role of the patient caregiver in this process and I know that you've had some outreach to PCORI and to Joe and the Joe was actually at one of our previous meetings given that there have been other ICs that have actually partnered at least a couple of them that have partnered with PCORI to develop an RFA is this something that you see as being of any interest to genome is this something that we could potentially move forward on and at least explore possibilities genome is always interested in partnering with other groups we can't do our work without doing that basically so absolutely we would be interested I think the question is how much our work fits into what PCORI is mandated to do and while I think they see that their patient centered focus it could relate because patients think genomics is really cool and really want to learn more about it to what degree they can actually commit in this area has not been clear to me at all and it's all their bandwidth is pretty much at this point if I'm accurate observer of it fully consumed with just trying to get PCORI net up to do even very very simple demonstrations that they can connect with patients and get data back and so I think it would be pretty much out of their available bandwidth for at least a while for PCORI net probably so but of course that's only one portion of their portfolio they have ongoing funding in a number of different areas particularly things like communication dissemination would be an area but they've also then funded the NIA project on falls prevention is sort of was outside of any of their standard funding funding cycles it was not tied into PCORI net I think they do have some additional capacity to get funds out the door if they have a good partner a compelling project and it fits within their orientation of comparative effectiveness and patient centeredness and in some ways the argument that you can make is what's more patient centered than the genome I mean using a genetic centric focus of the world now obviously that's a faith based and evangelical argument but the idea of treating people from the perspective of not only their individual beliefs but also individual level data as opposed to applying population level data should at least in theory be of some attraction well sure but you know there's theory and then there's other things and I think you know if we can come up with a compelling you know project as you said something that is patient centered and is focused on genomics I haven't that hasn't bubbled to the top as we've been talking but there may be some and we you know perhaps identifying a couple of those to take to PCORI would be a useful thing so maybe that's something that we could take back I think this has actually been on the agenda for the GMWG steering committee in terms of some of the discussions we've had I seem to recall my name at some point in the past being attached to PCORI yes so maybe that's something we could just you know resurrect in that forum and see if we might be able to within our group develop a couple of ideas that might you know potentially resonate and then see if we can establish connections with some of the PCORI folks related to how that would work Ken? Track record wise AHRQ has funded CDS Consortium and very related work so maybe just talking with John White might be useful and what's your sense of where they're currently related to that either Ken or Blackford? I mean they had a very significant portfolio starting approximately 6 or 7 years ago I think it was 40 million or so and there is interest in actually restocking if you will the funding for next generation HIT research but I don't know how big it is yet alright I think we've pretty well gone through the list any other ideas comments, questions contributions at this point from the group as a whole Jamie? Yeah I know I missed this morning and I apologize and I'm not sure if I followed enough to understand if you did talk about this already but I was really attracted to the idea that Dan brought up yesterday about the whole library and I don't know if that's something that could be covered in any of the work that you've outlined here if it's an additional topic Well we actually Dan did bring that up and I think it may have been before you came in this morning one of the questions was where would that reside who would have ownership of that and so in some sense we would be interested in and would be willing to explore with the group about what that might look like and whether or not that would be something that ONC could have at least some ownership with or partner with someone to own I think that would be highly valuable we discussed that in the context of developing the range of use cases thinking from the perspective that as we begin to work out we would be very aware where they could be placed as they become more mature and implementable so we'd love to have your engagement on that particular process Happy to be engaged I don't know if ONC is the right owner but we definitely our role as coordinator we'd love to help start the conversation So Dan would you be willing to kind of take the lead on maybe beginning those types of explorations? I think the architecture of how you try to make progress socio-technical progress on more than one level is an interesting question how to partition and then get natural sponsors and owners for the public good that results in a new topic to talk about as again that's sort of more social than technical but it's got a clear embedded technical layer So I would think that would be something that NHGRI could actually participate in and would have a little bit of bandwidth Ken Wiley would be the logical person We were talking at the break with Betsy and Jim Sumino about the possibility so I don't want to volunteer you guys but we can certainly keep you Jim's in, all right so it sounds like we've got and of course he would be volunteering on behalf of NCBI and Jim Ostell sitting here being quiet I think we should also invite AHRQ to that conversation Yes, yes we should do that a little late The AHRQ of course did fund the CDS Consortium which created a prototype library so I'd love to be helpful We also will have someone from AHRQ coming to ONC to fill some of our acting positions as a detail so that'll be a great time to start that conversation Okay so it sounds like we've got a pretty good NIDIS of folks to move that forward so that's great we can attach some names to that as well and then that'll just be a matter of coordination so it's Dan, Ken Wiley, Amy, Jim Simino and then an AHRQ victim to be named later so give Jim Ostell as well What do you think Jim? It's such a kind, flattering Flattery will get through everywhere Okay great, so good we have a lot of names attached to that particular item now and then we'll just, that'll be something that we'll have a case collection group and the sandbox so we'll just kind of think about that as a throughput so we keep all those pieces hopefully talking to one another Great, any other comments, questions, concerns, Ken? General notion there's a lot of people who are engaged here and obviously people who can volunteer for different groups is a more limited set if there's a way to just keep other folks just if there's an e-mail or an update that can be sent out just something to consider like a typical HL7 or SDO approaches that acknowledging a lot of people can only just read the minutes to see what's going on but might want to pipe up Yeah, I think we'll try and do that in as much as we can, I know we certainly have the distribution list available to us, one of the challenges sometimes is actually having the ownership of the groups to actually create that sort of an output although one would hope that that would be part of the work but I think in as much as we can create that we would have a distribution pipeline that we could utilize Okay, before we officially end I want to have Rita and Teji come up here to get an official thank you and round of applause for their outstanding work related to logistics Yeah, and in particular I didn't mention this before but Teji actually created the survey and did a lot of the analytics and that sort of thing so that was a tremendous amount of work and kind of a last minute hair brained idea that we came up with and so she turned it around very quickly so thanks very much for that and then I also want to thank our representatives from NHGRI so Jackie and Annie and who have also been taking notes and for all of those that I've forgotten you are in my thoughts even if you're not in my not in my mouth and most I'm sorry? Maggie and Al oh yes, yes our webcast yeah right thank you so much for doing that because obviously we had a lot of people engaged at distance and I was telling Maggie earlier that this has been a really well behaved group because I've never been at a meeting that Maggie's been at before we didn't have her yelling use the microphone so you guys did a great job so take a atta boy out of petty cash for that one but mostly thanks to all of you for taking time out of your your busy lives to participate to take the time to fill out the survey to actively engage in discussion to volunteer for more work to be done and for the efforts that we will be asking you for in the future and Brandon wants the last word are these slides going to be available? right, so I don't know exactly how we're going to but this will all be publicly available I don't know if we'll create a web page for this so if you Google NHGRI genomic medicine activities one of the first thing that comes up is our web page of meetings and each meeting has a highlight that links you to the agenda the videos when they're available and these guys are very fast at doing that and the slides the actual PowerPoint slides so those will all be available relatively soon but not tomorrow thank you to my co-moderator who may want to also thank you Mark I just want to say thanks to the group as well really great to see these two communities come together and it's been a pleasure working with you and meeting a lot of new folks great, safe travels everyone