 Well, as we've done with the previous sessions, we'll open it for questions and comments. Our intent was to be sufficiently entertaining, provocative, non-standard in our view of the things that e-merge has done. And I hope you agree we've succeeded with our soiree of presentation. So we're interested in input from anyone who would like to contribute. Sharon? Well, to start off, I'll ask a question of Matt from what he talked about at the very end, which is, do you have a sense of how many communities are out there? So an example I've given before, so I apologize if people have heard it, I'm a type 1 diabetic. There's actually an entire group of parents that first figured out how to monitor their children's blood sugars, develop MongoDB databases, all done free and with just helping each other. And it's really only been in the last year that I know University of Michigan now has a grant to, we can give permission to share our data. But so like the academic piece came much later than this whole technology. And I just don't know, is Facebook willing to share that kind of information? Do we have some sense of how many such communities may have germinated on their own and are out there that might be willing to partner? Yes. Good question. So I don't know, I don't think anybody's ever done a census of all of these new communities that have popped up. I would estimate that there are thousands of them already, which are often, you know, the MyGeneName community, and the reason they pop up is because the first one usually creates a Facebook group or Google group or something like that, and then suddenly they're searchable. They can be found by other patients very quickly that way. Actually, what's interesting is there's often now multiple communities per gene. I suspect there's probably for every disease, multiple communities now, and even multiple nonprofit foundations no matter how few patients there are. So I suspect, having not talked to Facebook, that they would be willing in some way to partner with academic researchers and enable better support for these communities. And I think there's a lot that you could do actually if Facebook were willing to actually provide customized support for these medical communities. Certainly, they could provide them much more fine-grained control over the privacy of that group. You know, right now, I don't know that some people are sort of hesitant to join these groups just out of their privacy concerns. Some people might not want their membership to be visible to the outside world, for instance. So even simple things like that would make it much easier for these communities to form and sustain. Thank you. Clearly, a central question is the role of the physician and the physician-patient relationship as these new data models evolve. And I wonder what the opportunities are for this program in the future to have novel components that look for novel ways for those relationships to evolve whilst avoiding the bottleneck on research and discovery that the current models give rise to. So I'm stating the obvious once more, but I feel like we're acknowledging this implicitly as an issue, but not stepping up and saying, well, what could this group do to change that relationship in a productive way in the wake of new data models? So I invite the comment, please. Well, I think that that's a huge problem. As I said, physicians are in favor of genomic medicine, but their experiences so far may have not been optimal. And we have to create these tools. And I think the electronic health record is a tremendous potential means of implementing genomic medicine in a way that doesn't pose a burden to physicians. And tools such as genomic decision aids, appropriate clinical decision support, knowledge resources within the EHR will all be very important in that regard. And more so over as we continue to develop these apps or app-based modules for certain disorders, that'd be really helpful. And I think it'll also depend on where you're practicing. If you're a family medicine or a family practitioner, maybe there's a different level of tools or genomic educational literacy that you're expected to have versus a specialist such as a cardiologist who is dealing with certain heritable disorders. So I think it's an ongoing area of work. And we certainly need to engage and seek feedback from physicians in this regard, rather than designing tools and then saying, here it is for you. If I may jump in very quickly. So that's a physician-centric answer. And we have a unique opportunity. You're next to someone who might give a patient-centric answer. And I wonder if we can concentrate, contrast those two. So perhaps if it could be. Yeah, I could also jump in. OK, go ahead. Heidi. So I was going to also give a patient-centric answer. I think one of the challenges with a lot of these patient organizations is they are very strong advocates for their communities. And harnessing that kind of energy and motivation is incredibly valuable. But they certainly don't want nor need to reinvent the wheel in terms of how do they capture genetic data from their constituency? How do they gather phenotypic data in structured ways? And so I think if we can partner with them and provide them some of the tools, and we've been trying in ClinGen to do this on the structured genetic data collection side by providing our Genome Connect patient registry where they can just upload a scan copy of their PDF report and then we capture all the genetic data off of that for them, working with them to connect lay community terminologies for phenotype data, but us providing that linkage to ontology systems. So it can be standardized in terms of phenotype capture. So these are just two examples, but I think there's many ways that we can partner with those patient communities, provide them some consistency in tools and standards, and then allow them to be more advanced and more capturing of the rich amount of data and information from their group. And perhaps the general statement of that is to say that to date eMERGE has used very traditional channels and very traditional reporting relationships and very traditional kind of paternalistic healthcare, but isn't it obviously well positioned to do some alternative way of providing sources of information and then measure the impact of doing it because it's not a solved problem. It's an interesting research question in its own right. So wow, a whole bunch of hands went up, and I think yours went up first and we'll go around the table. I'd like to ask Matt might about the opportunity that the social media and the new ways of relating to each other give us an opportunity to find family history data in these social media communities. I think that's a really important problem that we haven't conquered is to be able to benefit the patients and the pedigrees with these genes percolating through these families in ways that we don't appreciate. Both from the phenotype to genotype perspective and from the opportunity to organize and to deal with the privacy issues and the other issues that inhibit our ability to be able to get at pedigrees. That's a great question, and I think there's already opportunities with the tools that exist right now to start getting better pedigrees. So for example, I do know examples of cases in the rare disease communities where in the event of a diagnosis, the families will sign up for, say, 23andMe and use that to search for other relatives that they were unaware of. I mean, there's cases where this can actually be quite useful. So there's also a case I'm working on at UAB where we want to find a variant but we don't have quite enough family members. So we're going to try to find more using 23andMe to see if we can see if there are others that are out there. This is the case where the family was immigrants not too far back, and so we think there are other relatives affected by this very unique condition so can we find them this way? So I think these sort of like hypergenomic social media tools are already being quite useful for fleshing out pedigrees. It certainly will become more so over time, I imagine. Marilyn, did you have any? Yeah, I was going to address the comment that you just made, Dan. As I was sitting here listening to Stephanie earlier and you made a couple of comments and when you just said what you said, it triggered that it's almost as though Emerge is the R&D engine behind something like the All of Us program. So because we're so much smaller in scope and complexity we could do things like collect some of the data that Iftacar talked about. So we could start to do surveys of Emerge participants to link with their longitudinal EHR data. We could do other omics, we could, and we are starting to bring in environmental data from geocoding. But maybe another opportunity is looking for the interactions of our genome with all of these other sources of data in the patterns of our kind of disease trajectories of the patient populations that we're working with. And we could try lots of different types of things that in some ways are kind of pilots that all of us could do some kind of small shoot-off pilots but the whole program couldn't possibly test all of those things because somebody earlier said it's like the large ship versus like the smaller ship. So I forget the exact analogy. But I think there is an opportunity here. We have really rich clinical data and we already have a lot of genomic data. We've done kind of all the standard GWAS, FIWAS things that are kind of the out of the box vanilla things to do. There's a lot of other data we could get from our patients that would enrich the data that we have. Maybe some of those things would explain the reduced penetrant. So maybe it's interactions with other genes, interactions with the environment, some other methylation change or transcriptome change that explains that reduced penetrant. And we might be able to actually assess that in the data that we have. And Rex, who had the nautical metaphor in the first place. Yeah. Yes. I won't revisit that. Although I've had conversations offline with several people about that analogy, which I still like. But just to build on that a little bit because I think it's interesting. Matt's comment that six of the public equal one PhD. I'm surprised the number is so large. But it does raise the question of things that are difficult to scale, like clinical annotation decisions and some of these others. And I know Clint Jen's thought about it. Eric talked about crowd, cloud sourcing. I think we are really well positioned maybe to think about doing some of those kinds of experiments in a merge. And I think the key is we've got a gold standard you can compare it to. Whether it's the right gold standard, we can debate a lot. But the gold standard is what we can get out of electronic health records. But if we were really able to throw a whole bunch of new patients thinking about it in a different way at the problem, what would that look like? And could we get a more efficient approach? And I don't know. I mean, Matt, maybe you've got the most experience with that. But it seems to me that's an experiment that would be very interesting and fun to try. Yeah, I always enjoy these discussions about disruptive opportunities. But it always raises the philosophical issue that one of the inherent properties of disruption is that it's unanticipated. And which raises then the second question, which is, are we likely to be the ones that are disrupted, which I think is more likely than be the disruptors? Because we're essentially trying to disrupt from within, which has not been a successful enterprise over time with a few notable exceptions perhaps. The point of this is I think the things that are beginning to resonate in terms of what a merge can do is to think about it more as the skunkworks, where we could try some interesting ideas that would potentially be orthogonal to the way we traditionally do things. And within small pilots might be able to demonstrate value in a way that we haven't been able to do. Because all of the things that we've heard today about, well, we can't get the clinicians to follow the CDS rules. Or we can't get them to buy into the evidence. I mean, these are recurring themes that we will continue to come up against unless we can somehow end run it and make it happen. And I think the patient is a great opportunity there. Now, PCORI has tried to do this with the PCORnet, and particularly the Patient-Powered Research Networks. I don't know that there's a lot to show that when you put patients into what is a very rigid and in some ways ossified system that they can affect much change. And in fact, probably there's more risk of Stockholm syndrome than anything else. So how could eMERGE empower or partner with patients in a way to really do something that's truly innovative? And that's why I'm really glad that Matt is here and that Matt is also in the position that he's in. Because I think it gives us some opportunities to really explore that. And thinking about how that might be able to come across. I must admit to some skepticism that, you know, as much as we say in review that innovation is one of the scoring elements for proposals, in study section there's a lot of concern about innovation. And there's a lot of concern that if you don't have data that this is really not something that we can really risk. We're risk averse. And so all of these things, all these tensions make it very challenging to think about how within an extent system, how we could actually do something that would be really innovative. Yeah, I think one of the solutions to some of the problems that Mark's been bringing up and others have is that every time a clinician successfully deploys appropriately one of the eMERGE tactics, they get an automated health grade and Yelp post. So they can counteract all the other stuff. The other group that I was going to mention kind of got alluded to as well is that right now, places like Walmart, CVS, before the insurance purchase, Walgreens are trying to do more and more of primary care type activities, trying to be more than a doc in the box, urgent care place, trying to be less than a traditional primary care. Those are places where there's a growing segment. It's still a minority of people going there. And who knows if anyone will have health care in a few months or maybe they will all be going there. But the idea that we could be reaching out to those places now before it's too crowded would be also linked in with some of the other suggestions that Facebook, Google, Amazon, close flight from your home is pushing into health care dramatically. There are places that are going to be, like Mark said, we are probably going to be disrupted. So we should be thinking ahead. So I would say that eMERGE has been a bit disruptive. So I think that just in thinking about the various communities, 23andMe has told us that you could actually get valuable information from people. Know each other, are connected to it. You publish 80 papers. And some of that data is very valuable. It can be a starting point for research. eMERGE has, I think, been disruptive in saying, health data is useful. And we can figure out how, at first time, you get a bunch of people with disparate health care systems to act together. There's also been one other area, maybe Eric or the other data comments people can talk about, that I think is going to be necessary as we start bringing together. What eMERGE has done is brought together research community and actually tried to link it with an implementation, the clinical community. And I think all of us is going to be really the innovator and really adding to that the data that comes from the users. But all these communities have traditionally worked separately. And one of the things I think that we have, a disruptive area of innovation, where some work's been done in eMERGE and others, is finding a place or a space where all of these disruptive contributors, as well as the carrier class entities, Qaderi, UK Biobank, MVP, all of us, can come and play together in the same sandbox to create the knowledge base or the implementation. So I mean, Matt could comment on how to do this in a secure way that makes all the different users happy and puts the right boundaries around everything. I think eMERGE has solved some of those problems and that, I think, could be very, very useful in maybe pushing the federal government to come together and create a space that would make that easier to happen. I just was reflecting on what you're saying, Mike, and just the theme of all the discussions really all went through, looped through the patient in some way. And the novel kinds of information we can get there. I'm glad, Mike, that you thought that we've been disruptive. I think that I kind of feel like harking back to Dan's first point that we have and some initial hypotheses. But I think what is that next step? And I actually think that there is an opportunity to leverage the strength of the connection to our primary strength and connection to our health care systems. But then we brought in with eMERGE 1 the extra domain of genetic data. And eMERGE 3, especially a little bit in 2, we brought in the extra domains of sort of CLIA-level genetic information. I think in the next round, sort of robustly diving into participant-provided information and having them as sort of borrowing all of us language, partners in the process, which can involve in sort of all that sort of rare variant phenotyping efforts it could get into as we've promoted new methods. 23andMe said, we could do cool things with patient-provided phenotypes. Well, maybe we can do cool things with patient and EHR-provided phenotypes and start asking interesting questions. I mean, one of the things that I alluded to a little bit in our summary was this idea of a phenotype risk score, which is aggregating in sort of Mendelian patterns, but it could be any kind of pattern we're using it in the UDN now to test against rare variants and find sort of symptomatology at sensitivity levels beyond what you would in the typical EHR fashion. If we combine that with participant-provided information from surveys, from these, from other kinds of things leveraging that connection to the health network, maybe that would be sort of the next level, which foreshadows all of us kind of things too. A few years ahead of time. Despite what I hear is general enthusiasm, I guess I wanted to push back a little bit on the idea that Emerge would become the skunkworks for PMI. NHGRI is a relatively small institute. And the 21st Century Cures Act, I forgot the exact number with the budget, and that alone was four or five times the entire NHGRI budget. All of us, yeah. But I just think that there's more to offer first the scientific community, and I think there's more to offer to NHGRI, and I think there's more questions that are NHGRI focused than just becoming a pilot or an experimental sandbox for all of us. I think before I would go down that road, it seems the Emerge investigators should think long and hard whether that's a healthy long-term road. But we've heard a market basket full of interesting new ideas throughout the day to day, and it seems to me that there are natural alignments with other major research efforts, and maybe it just falls in that category. You don't sort of do it as a subcontractor to all of us, but it turns out they have an interest and you have a native interest in well as well. And instead of questions so practically, perhaps it's solved by natural alignments rather than unnatural alignments. But I would maybe, we're getting to the next session, which will be our summary and wrap up. So let me make sure I've got everybody whose hand was up for comments on this one. And those who are putting their hand up again. Sorry, I wasn't in response to what Eric said. So it's a really good point. And I think remembering that this is NHGRI, so we have to keep the genome part in mind. So I like the idea of these other data sources, but specifically with the context of trying to do more with the genome. So I don't think what we necessarily want is a bigger sample size, more GWAS. Everybody always says more data is better, but we already have millions of GWAS and we're getting to tens and hundreds of thousands of exomes and tens of thousands of genomes. And so it's not a matter of just more genomic data as much as can we capitalize on the genomics data and the richness of the EHR and add something else to it that others can't do. So a lot of the other cohorts don't have 20 years of electronic health record data that they can capitalize on to bring with the genomic data and maybe the wearable data or environmental data or family history data. It seems like low hanging fruit that theoretically should be easy, but we've yet to do well. And so it seems like one that we should target because it would be so ripe with the EHR data and probably yield a lot of successful findings. Another good question. Sharon. Well, I was just the follow up. I have been a bit surprised that there's, although we've talked a lot about all of us, we really haven't talked about interacting with what used to be referred to as large scale sequencing programs, but now have a different acronym. And I always forget what it is. Where are there, you know, doing exomes on genomes or 25 or 50,000 individuals more for disease discovery? And I do think that's a group and a data set that I've been kind of surprised has not come up in these discussions that could be used to mine information or not. The other part I was just gonna make with regard to participant based data submission. A lot of what people have been talking about is data submission. It's really a two-way street. And I'll just give the example that in ClinGen, and I don't know if Heidi's still on the call, but our Genome Connect data set registry, which a number of people around the room are involved with, patients can upload their DNA test reports and give permission for that data to go into ClinVar. And very early on, it was identified that patients often don't have the most up to date classification of the variant. And the laboratory may have classified the variant differently and maybe their physician never, there are many reasons they might not have it, but that's a good example where the committee then had to decide, well, how do we let our participants know that in fact the public database has an updated classification of that variant. So it's just an example where if you're getting data from participants, then you're interacting with participants and they need to know about your results as well as just being a one-way communication path. Casey and then. So I just have one comment about, since we've been talking about all of us and Emerge and the relationship between the two. For Emerge, my understanding is since we have biobanks, this is more, the biobanks are more reflective of the population. And so I'm wondering if there are distinctions in the kinds of questions that we might ask in a population, a resource that reflects a population if we're able to get phenotypes for all of those patients compared to a population where there's volunteers that are donating their data for research. So it's just kind of a general question. Maybe I can just respond to that. So I think there is an opportunity there because one of the things we all know about electronic health records data, even though we've said multiple times today how we've demonstrated that it's great for use for research purposes, it is still incomplete and wholly with an H data. And so I wonder if there's an opportunity for the participants to help fill in some of that missing phenotype data through participant provided information. And that seems to me where there would be a real opportunity if we had a collection of participants that had genetic variants that we knew about and our HR data, we know to be incomplete. If there's an opportunity for them to provide some additional context and maybe where we got started today to that data, that might go a long way to help us actually figure out why some of the issues related to penetrance are related to why things are seen in one patient and not another. So I think there is a real opportunity there for the participants to help complete a data set that we know to be incomplete. Is this on? Yeah, so can I just respond to that? So as Matt kind of alluded to, some of the modern initiative and the layers of phenotyping. So I have a PCORI grant to develop and refine two patient cell phenotyping methods. So one with Genome Connect and one using the layperson term. And those would be available, we're setting them up to be available as an app. So patients could go, I'm just saying that going forward, hopefully we will have some of that type of thing just for the reason you're saying is because the electronic medical record just doesn't give you everything. And patients really know, ask Matt. I mean, this came from him. They know better about their phenotype than we do. So I'm just saying that we are developing some of those tools. I'd like to respond to the, I guess the penetrance question. And with one of the ideas being to throw more data at this to find GBE and epigenetic interactions. And I think that's great and that's true. But I think another reason we're not seeing a lot of the penetrance is a lot of those interactions you're only gonna see because they're actually epistatic interactions, right? So in epistatic sweeps, one of the problems is they don't scale well with traditional methods. And we've looked at this with a single phenotype and a population of 28 million SNPs, it would take 240 million hours, CPU hours, to do a single 2D sweep of the genome, all right? And I'm sitting in a supercomputing environment. We can actually do that for one phenotype, but we wanna do it for 100,000 phenotypes. And so that doesn't work. But the good news is we are working on machine and deep learning methods that do all of a sudden make epistasis scalable by a five orders of magnitude increase in speed. So all of a sudden we can not just do 2D sweeps, we can do fourth and fifth and sixth order sweeps of entire genomes and discover signal that we're completely missing with single QTL mapping. And so I think GBE is certainly things to go after. Other data sets, certainly things to go after, but we also need to think systemically about many of these interesting complex phenotypes are due to heterogeneous collections of genome variants that are working collectively to lead to these complex phenotypes. And we can all list long collections of interesting complex phenotypes that are not gonna be super amenable to simple QTL mapping and understanding the entire story. But there is good algorithmic development that combined with supercomputing capacity that we have within DOE, we can actually approach these problems in a way that we've never been able to before. And that's literally proving out as we speak now. Yes. Since we've talked about patient reported information, I think we'd be remiss. And I think Matt did allude to this, but I wanted to come back to it and be explicit that as we think about outcomes, we should capture patient reported outcomes. There are standardized measures for that, the promise measures. My guess is that if we examine those, there may be gaps related to promise measures for genomic medicine, which would be something that I think Emerge would be well suited to deal with and then test the implementation of those. So I think that would close the loop for involving patients in a project of this nature. Yes. I just also wanted to respond about the patient engagement. I'm sure many of you remember in Emerge one, we did a lot of patient engagement. Those many of the participants we were engaging were from our biobanks and many of us still continue to have relationships with our biobank populations who are most of the people who are genotyped in Emerge. And so I think we have a lot of relationships that we can build on and bring them in as partners, maybe again into Emerge and to some of the things that we're thinking about and also ask them what they want, what would help them in understanding their genome and understanding the information because I think those are many of the things that all of us is trying to do. And I feel like we've sort of skipped over that a little bit and have taken the attitude of, we're a very medical sort of top-down approach. And I think a lot of these ideas are focusing on what patients want and how they can work with us as partners in research. And I think that we have a lot that we could go back to and build on from those early relationships. So you know with your hands up, oh, George. The main challenge in phenotyping is not just data quality, but the data often tells us something different from what we really want to have to make up this algorithm to convert it. Patient data is going to be the same way. We're going to find that the reported weights are strangely different from the true weights if you were to go in with a scale and things like that. And we're just going to have to learn that and do pretty much the same process that we're doing for EHR data. We're going to have to do the same thing for patient data. And it'll just take us a little while to get there. So I have to say, although my job number one was to be timekeeper, that's the one thing I actually didn't do. So Sharon, do you know where we are? I don't even have a copy of the agenda. You're ahead, but we're not going to stay way ahead because we're just going to be happy if we don't want to fail. I know some people from Seattle need to leave. I wondered if we should take our brief break right now, unless did you have some summary slides you wanted to do for your session? What I found myself is I started out with the summary of novel and disruptive elements. And it turns out it actually overlays on all the other aspects of data acquisition, analysis methods, reporting of results, and the overall process of how we do discoveries to translation. So rather than actually, if I start reading those, I'm actually just going to steal my thunder for all the other topics. Because it turns out we had something to say about nearly all of the dimensions of what eMERGE has done. So that's a long-winded way of saying, I think we'll let the session stand on its own merits as well as a rich discussion that followed. And then after the break, we'll actually begin that enumeration of what we think are kind of the most notable points. And I make no representation that I've actually got the most notable of the notable points. So we're going to do this as a group editing exercise where I show you my outline of the things that I happened to highlight in red as we went through the day. And then you can correct where I've done any violence to the truth, or even add additional things that come to mind as we go through the list. Great. So why don't we, I don't think we need 20 minutes now. I think people agree. Why don't we take a 10-minute break? Yes? I don't think there has actually ever been a 10-minute break in the history of 98. Well, but if we make it a 10-minute break, so we could call it 10. If we make it 10 minutes, then that means we'll get started in 15. That's right, exactly. So 10 minutes, we back in 15. And then we'll be up front with a laptop to do the last session.