 Thank you. We have several hands up Mike is it and then Howard suggest very briefly from the PMI perspective again Having been at the last couple of working groups. They are they are Constantly cognizant of the fact that diversity is is a critical element in the design of it but there to refine the the the debate at this point is that representative Cohort might serve some social and political Purposes, but the scientific questions might not be served It and so some refinement on exactly which subgroups would you want to go beyond representative and enrich? Yeah, a 12 percent African-American might be much while its representative wouldn't necessarily afford you the The right number to do good science. So that's one thing that would would You know, what how do you set those priorities? The second thing is is just the the any experience that that this group has in affecting the Specifics of enhancement that is desired would be where I think the the PMI could Could garner that could could gain from that kind of insight Howard So I agree with the comment that was just made in terms of really it's a sample size issue and and you know We've we've found that in ourselves So there are cancer studies that we have proportional representation from a population since the standpoint But it's underpowered to really draw any conclusion. So I definitely agree with that From a clinical trials standpoint, there's also a couple other issues that come up. I'm Gary puckering from the National Was the National Minority Health Quality Forum or whatever whatever it's called Has done some nice mapping showing that Clinical trials are not readily available in areas where there's greatest minority density or greatest diversity And so even though there might be a clinical trial available in in in Tampa It may not be where the groups are that it could Participate and therefore, you know, there's some some issues that could be solved if you will there Another thing is we have a paper coming out in the journal the National Cancer Institute where we looked at people who had art Who enrolled on clinical trials? Cooperative group trials and so they'd already agreed to be on a trial So they bypassed some of the historic issues and yet they the the non-white Patients were under represented in the genomic component and it turns out equally under represented in any of the correlative science And as we drilled down it was because they were treated at centers that were under resourced And so they didn't have the the expertise or the extra research nurse or whatever it might be to enroll on that extra trial And therefore didn't have the genomics and extra pits like that there So it had nothing to do with with It with with race per se it was there's more to do with economics in terms of the the resource Utility and the centers that were well resourced there was an equal representation in terms of White versus non-white so it was really came down to just the centers where they were treated and so I think there's some issues there and then lastly many of the of the Patients that are already of the areas that are represented with a high percentage of diversity are treated by physicians that may not be Hispanic or African-American or they might be Pakistani or there might be something else And so I think from an education clinical education standpoint. We need to be educating people who treat Areas with high diversity density not only the The clinicians that might happen to represent a minority group and so I think if we want to get to patients We need to treat who's treating them and not and and not otherwise Terry did you want to yeah, if I could Excellence points that that have been made so far and I echo everything that that Howard has said and all of you We didn't unfortunately have a chance to have venture or Craig introduced themselves But but events is responsible for diversity and health disparities. Do I get that right in NHGRI? Responsible with trying to help and self-solve them Yes Thank you, thank you for getting that right yeah And and Craig you've done this kind of kind of research in in Hispanic populations for decades as I understand it How three and a half well and and so that you know I think the question we need to ask because because many of the issues that have been raised. I think are common to Research in lots of different areas not not just genomics So what is about genomics that makes this particularly difficult or maybe it's not getting more difficult than than otherwise But I but I think when when I look at NHGRI so I came to NHGRI from the Heart Lung and Blood Institute at Heart Lung and Blood We had you know fairly consistently 25% or even greater representation particularly of African Americans because that's where the disease was I mean that you know Terrible hypertension and heart disease and stroke and that sort of thing And so so it was an area that we really pushed and we had a leadership that insisted and put systems in place to do That I came to NHGRI and and you couldn't get non European ancestry populations through study sections because you know They're they're diverse and that messes up the linkage maps and that that sort of thing And and I think you know we need to think carefully about why we've done so poorly And I would say poorly in in the genomics field particularly looking at GWAS You know we're doing a little bit better now and that we have Asian populations in a big way not from the US But but from Asia, but we still are I think underrepresented in other groups So you know there's a there's the dark legacy of eugenics and and you know mistrust of the way this information might be used against people to demonstrate inferiority you know there are there are other concerns about You know how to how to be sure that that the representation of People with different genomes then gets used for them rather than to inform the majority of oh we see this variant in you know Alaska Natives in a high Proportion so it must not be very important in the European ancestry population You know that may be a useful piece of information, but it's not the thing that will be useful to them So so as Vence said you know Can we identify health disparities questions that are really important that are genomic and and if we can do that I think that then we have the scientific argument along with all the other arguments for doing this kind of research And that would help us to sort of design those programs Yeah in the back When you mentioned the idea of using the difficulties of infrastructure for I'm sorry. Can you hear me now? Sorry when you speak about the infrastructure barriers related tours implementing some of these tools such as The use of the internet. I think you'll find out that there are opportunities that are available to overcome those There's several investigators that have actually Used up it overcome those barriers by utilizing technologies that are actually developed for that In addition one of the things we realized just for my own experience is having to work in working in purpose You know communities that don't have access to the infrastructure that have access to the knowledge to recruit them into Different protocols from NIH as related more towards they have a central person that they can talk to they no matter What the situation is there's one person they can pick up the phone and call and they actually develop a rapport with that person and You know best example I have is no dr. Powell dr. Tiffany Powell Yes, it's a shameful plug from my wife, but her program is dedicated towards doing exactly that you Developing mobile technology in areas that are impoverished to have the infrastructure and to get recruit underrepresented populations and she's gone to face faith based initiatives and It took a year or two to buy in for them because it needed the church to build it buy in but they are they've been on board And she's brought like a hundred and fifty people to the clinical center based off of that kind of simple structure of just she's always there and So I think what you'll find is if you utilize The one person kind of concept that they have that person to go to The barriers for the IT infrastructure those can be those tools are available to offset that and we've used some of those tools First hand so I know that they work You'll find that that will that could help as far as your recruitment because they the patients actually try to They just want to understand what's going on why it's helpful to them and they want somebody that they can talk to about that And that's all I have that for that So I think I would like to add and this builds a little off of I think what a lot of people have been saying but to synergies H3 Africa Because there isn't a barrier or opportunity up there that that they don't face tenfold and the scientific imperative to include the diversity is larger and harder and So at NHGRI we've been talking a lot about Working together on some of the community engagement strategies because they've come up with some really novel ways of doing things and I think that Trust relationship and building a trust relationship over time is a huge one as well as you know Identifying the appropriate community leaders that can help with that Sorry I don't recall from the introductions how many people here were involved in pediatrics or child health But just want to point out the particular salience of this issue for pediatrics. So Which is both a challenge and an opportunity so that we're very close to a majority minority society already in pediatrics I was just looking at the most recent numbers. I could find we're from 2009. We're 56 percent Why not Hispanic so that leaves 44 percent? You know again, we're very close and the 2015 numbers are probably even closer to a majority minority so on the one hand the the challenges of diversity and disparities are and the sort of limited knowledge bases and even bigger deal for Pediatrics than it is for other areas of genomic practice on the other hand There's an opportunity there because research that's done in pediatric populations is almost inevitably going to be more diverse or enroll More diverse populations than it would be if it were done in older and adult populations So pediatrics, I think is an opportunity to start to address this diversity challenge. We're all struggling with Yeah, so I wanted to get back to Terry's comment about You know the initial in the scientific community the initial, you know push for homogeneity and lack of diversity And so so I think one of the issues there is and what I think is a is a concrete Suggestion as well is the lack of real great or good methods to exploit at mixed Genomes for discovery and we find this now that in In our research through methods that are based on identity by descent and some other methods One can actually exploit sort of the diversity in the in in the population in in our biobank and one of the things that I think would be helpful is to think about ways to Increase funding for method development specifically addressing these issues new methods particularly as it comes to sequence data And associations and all of that in in at mixed populations I wanted to make a second point and that gets back to one item that you had on one of your slides and that is a community advisory board and one of the aspects that we found Incredibly helpful as part of our ignite project with his enrolling exclusively African Americans Because the genetic the genotype that is linked to our disease of interest is prevalent in in African American populations and not in others and What was incredibly helpful was that the community advisory board? Coming in very very early in the development of the study to the extent that they actually were almost writing the consent form So our community people have a different way of looking at the language that is used in the consent talking about genetics talking about genetic risk talking about data sharing and You know it was received with much greater I Was it was much more effective than what we typically use in in in the ordinary study So I think that's a very important point to bring out. How can community advisory boards and and others be? Engaged very very early and in a participatory way rather than At a later point when things are written and approved by IRB etc etc Carol. Oh, you've got him. Well, I was just gonna make trying to say something for ages And I was just gonna make the same point really is that if if you Bring the community of study with you right from the beginning even from beginning of Designing the research It's a particularly effective way of getting Better recruitment and better participation. It has to course not look token and it also Of course you have to hear criticism and those sorts of things which for some people can can be awkward but Comparisons are odious but We had the same problem with Aboriginal Australians involved in clinical trials And the only way around it was to give quite a lot of autonomy or Independence and ability to say difficult things to the researchers from the community advisory board right from the beginning and things have Changed extraordinarily after five or six years of it. I just wanted to very briefly amplify on your point with regard to pediatric populations So with the pediatric genetic diseases, we have two more whammy's on top of this one is that our databases have few Pathogenic variant annotations from minority populations and Second is that in genetic diseases in general one of the most powerful pieces of evidence We have in terms of likely pathogenicity is allele frequency accurate allele frequencies And for many of many minority populations, we wind up with hundreds or even thousands of variants That are novel or have very low allele frequencies just because they're under represented in our database Which means the quality of our interpretation today lags far behind in minority populations versus in northern Europeans I have a footnote to the pediatric Discussion and that is a phenomenon that that one of our pediatric geneticists thought she saw in her clinic And now we have really nice PCI based data that not on we have an increasingly biracial community not admixt but an increasingly number of biracial children at our hospital and it's it's it's very clear when you look at PCI plots by decade that in the last two decades We serve sort of a cloud blossoming right in the middle between blacks and whites So that that complicates Steve's data interpretation in an order of magnitude or two As as the country becomes not only more diverse But but the genetic ancestries become much more hard to define as just one of the major ancestry groups Sorry, who are you pointing at Carol? Yeah, hi to build on doctor Kingsmore's comment about just thought the disparate knowledge bases and non-European populations To the allele frequency issue I just wanted to highlight that the page program that was mentioned is going to be amassing allele frequency data on 50,000 non-European populations on about 8 to 10,000 variants that are in ClinVar and other loss of function Variants, so I think that will you know be just part of a resource that could be used And I also wanted to make the broader point that I think when thinking about variant interpretation and the underlying knowledge base You know the human the human association literature is very vast And I think it's it's wise to sort of keep the epi epi colleagues in the loop as well that that dialogue could be fostered Better as well So I also wanted to build on Steven's point about about the lack of data and the lack of ability to interpret variants in Non-European ancestry populations and when you think about it This is a disincentive to investigators to study those groups because you're gonna solve fewer cases and your you know Your applications will do poorer and review etc I wonder has has anyone really published that finding and shared this with community groups and others and said You know this is a huge problem in terms of using genomics in the care of your children and in you know In the care of this population and is there some way to get we recognize that communicate community advocacy can be very powerful So so I might ask Benson Craig and others who work with populations Is this a useful message or would this damage us further in the in the eyes of these groups? So I think this could be a useful message and it depends how it's delivered And one of the mark remarks I was gonna make was really kind of following up on this issue of trust and how Messages are communicated and how information is communicated and the trustworthiness of the researchers and those that are Communicating information. So I think if it's articulated in the right way, it could be very valuable At the end of the day, it's communities by and large and being in the community. There's no substitute for it We've done some 28,000 exams over the last 34 years and we've probably been in every one of those homes multiple times Not me because I'm not part of the community But the community workers have been there and it's made a huge difference and they're able to transmit that information and Get over some of the the biases that misunderstanding the other challenges that come the faith-based organizations become critical in this because there's a Cultural destiny, you know with family and genetic data and genetic data is is frightening and yet It's also very comforting that okay. I don't have to do anything about this and and overcoming some of those Feelings thoughts becomes an important aspect of that but at the end of the day, it's being in the community and being there for quite a while We'll see some changes. They'll put a plug for Texas. We're putting a medical school on the border between Texas and Mexico It will change what we do and how we access at least one population of a couple million people my comment isn't obviously to undermine the importance of increasing the diversity of People who are in the databases and for whom we have variants and frequency information But I also want to acknowledge that it's more challenging than just getting people into the data sets I do bone marrow transplantation in my clinical life And we all know that it's much harder to find an unrelated donor match for somebody who's from a Non-European background and part of that is because there aren't enough donors in the donor banks the cord blood registries But part of it is because the genetic diversity is so much greater Amongst people from non-European backgrounds or many non-European backgrounds and it is from European backgrounds And so part of it is really just a sort of population genetic diversity background that makes this challenging It's gonna be it's harder to solve than just getting more people into the databases in order to get their variant data so Terry and Eric I have a question for you so You know the the scientific imperative the social justice imperative those are both pretty clear a Lot of the solutions that people have had success with are Not things you would write necessarily into an R01 so community advisory groups and and those kinds of Building the trust in the communities That's not really something one would put necessarily in an independent sort of R01. Is it I mean what what are the mechanisms for? Having impact in this area from a fund of funding perspective is it new to be meeting proposals? What what kinds of things would people do? Yeah, I think maybe I'll take a swing at it You know we actually don't have a lot At least in in our you know science of medicine and effectiveness of health care Portfolio and we probably need them do we need dedicated programs just for for these kinds of questions? Do we need supplements that could provide these additional resources particularly in under-resourced settings where a lot of these patients are found? You know, I think all of those are possibilities and events, but what do you think I? Agree with that. I think supplements and ways to integrate and thinking about the two divisions the Genomic and medicine and the genomic and society are there ways to do some more integrative kind of research studies Where you're bringing these issues you're studying How do you engage with the community as part of your research as well as doing the basic genomics work? And so thinking about different kinds of ways and strategies may be a way that we all can think about how to really move this Conversation this direction of this research Mark Because mark is getting ready to speak and I can't let mark speak So actually, you know one of the things when when the page program population architecture using genomics and epidemiology Which was adding g was basically on top of large cohort studies that our colleagues and other institutes had funded One of the things that we looked at was this you know huge lack of data in diverse populations And we said all right this program is going to be all non European ancestry populations And maybe what we need to do is is more programs that are all non European ancestry because at least then they have a Little bit more of a level playing field I mean still the smaller groups are going to lose out to the bigger groups again because there's more data But but that may be an approach And vents it's fair and I'm also trying to prevent mark from speaking And it's fair to say this is exactly what vents is doing in a co-chairing role of one of the internal groups That's helping give sort of implementation Guidance advice about for the precision medicine initiative and I think all options all scenarios are on the table Of and there's because there's not a cookie cutter answer that to this and we know that and so but but this is being thought about again Not just at the NHGRI level, but even broader other initiative We think about these things for some common fund projects as well And certainly you're going to be thinking about it for the precision medicine initiative I was just I'm going to make one last comment is that other ICs are interested in working with us and this so that there are more collaborative relationships around this specific topic area That bring resources and expertise from other institutes in their interests Mark So this is a question. I know that in the grant proposals that there is frequently in the RFA is a call for you know training programs for Minority or other Investigators and and I still you know I see on there the lack of involvement of minority investigators and physicians and I also recognize the point that you're making is that it may not be at The investigator level that's the most important point of engagement with these communities but the question that I have is what has been the result of evaluation of the Intent to develop training within the RFAs has that resulted in in improvement in the number of investigators Clinician investigators that are coming into the space to to lead these trials I'm not I'm not sure I think I'll call on others at NIH to comment as well and maybe our See oh Kishel has left and so is Barb Connelly. Okay, so we're on our own Dean I don't know if you have a comment I think the training that we've tended to do within our program there hasn't been a lot I at least in the programs I've been involved in we have had some minority and diversity action plan Programs that have been more in the basic science programs There hasn't been a great deal and it's been very difficult to retain such folks in in science And and I think that's an NIH wide and a science-wide problem that I'm not sure that we can we can address necessarily but but I think to the degree that we can say Yes, this is an area of emphasis for us And and we really need to determine and figure out ways to increase genomic research in diverse populations I think that is within the realm of NHGRI and something that we can do how best to do it I think or is a testable question that we can look at I guess maybe to come back at that you know I can In some ways when you when you haven't had success and the question is as well, you know What haven't we gotten right and and I'm wondering if genomics would be a potential point of engagement in other words is this an area where there could be an exceptional quality to understanding You know these disparities and and inviting people to participate I I would think that there might be some sort of a Social research agenda around that question that that could be approachable I don't know that's just you know shooting off the forehead to events That at least isn't looking completely skeptical like he usually does when I talk so I know for the for the Sears I mean it was an important part of charge for the Sears to include training of minority You know students and it was very successful at our institution Hopefully going forward. I assume that that's going to be a part of the new RFAs that are out now to fund additional sear centers, but you know it You know made them make an effort in that area and fortunately They were extremely successful in no small part from some of the things that I mentioned before I Just want to add I know someone referenced that the Common Fund programs And I think this would be a really interesting conversation to have with Hannah Valentine and her team since they're coordinating these diversity workforce training programs across the NIH and I think they've got a Better sense of what's working and not working globally at the NIH and maybe this focused discussion with them on how could we engage More in the space of genomics as a small part of their bigger program would be a good starting point Any other comments? Oh? In the back. Yes, I will add in something really quickly. There are some training initiatives also going on with the big data-to-knowledge Program that many of you are probably familiar with that you may also be able to capitalize upon so I think for the five points that Terry wanted each of the panels to sort of sum up on critical knowledge gaps We've identified a number of gaps related to increasing diversity in the populations and health care systems and identified a number of barriers that recommended approaches in included a kind of leveraging other programs within NIH that are focused on These issues and it's really kind of an interesting blend of the social sciences and the genomic sciences We might have to do that in a more Unique but maybe in a more directed way kind of what Terry was saying where you might have a program that is very specific about You know only accepting research that that's going to work on some of these target populations And maybe increase the pool that way is that I interpret what you said correctly Terry? I think so Methods development for working with the data in ad mix populations. I forget who said that but that was one of the recommendations In terms of the training needs The training I think is is at all levels and an NHERI actually has been very proactive in in doing devoting funds for training for At all of these different levels and I think then the long term that's going to pay off but I think that needs to To continue and then the bedside to bench that virtuous cycle. I'm not sure I have anything to That I can sum up from this discussion on on that point But for the others that was my kind of summary take away from the discussion or anything else people wanted to add to those points I guess I just might add on the on the last one the bedside to bench You know as we start to see novel variants or previously undescribed variants in genes From ethnic minority populations that it is a great opportunity to start to think about function and And they represent new sites that we should be looking at in those proteins. So I I think that's a really interesting opportunity Got it. Did I miss anybody? Okay. Thank you All right. Thank you Carol and thank you everyone So we're we're now at this stage of kind of Summarizing and looking back at what we've what we've learned today. We thought it might be used. Are we at that stage? Yes, okay. I just got a a side-long glance from my coach here made me nervous, but At any rate, he'll I'll probably get several more of those before before we finish Yeah, so so I might ask Rita before you if you're heading up there because I was hoping you would switch to my screen And maybe she's going up there to do that. Oh, great. Okay. Thank you because we've I think we have the system down now Thank you much for for doing that. And do I need to share again? So I will go up to share and my screen Rita will stop sharing. Ah Beautiful. All right Thank you very much Rita And then if you could make it maximize the screen up there Yeah, you just make it make it big So so what we've tried to do a little bit on the fly and and we'll we'll obviously, you know, try to refine this feed it back to everyone as as You know after the meeting and and that and actually we'll have a dinner So, you know, we're sorry that we haven't arranged one of our our lavish Government dinners that you're familiar with For you We're leaving you on your own for dinner because the working group actually has to work over dinner And so so we'll be be heading off to do that Actually while I have the floor I might just mention that any of you with a domain name that ends in dot gov or dot mill need to pay for your lunch breakfast or snacks and Rita can tell you how to how to do that. So please be sure to get that done but with that with that now aside so in in terms of Evidence gaps what we've tried to do here with a little bit of Collusion with those on either side of me is try to identify some of the key points and in blue you can tell I love blue Are things that might be particularly important But they work just sort of our initial pass and if you want to blue in thing other things That would be fine or unblue sums that would be some that would be fine as well I thought that the statement that for implementation you need evidence and for evidence you need implementation is a really You know what a take-home message here. I'm not sure what we do with it other than worry Because it's it's you know, it's very difficult to get the implementation done But it probably also is a message to share with payers I have no doubt that that Jeff has tried to share that message with payers Jeff have you And have you gotten a response from them that's you know, let somebody else pay for the the implementation or It's always a work in progress. So and I actually put this discussion behind us I think at least I'm reinvigorated to try again One that seemed to resonate with folks was the idea the quality improvement projects Generally don't get published and is there a way to maximize sharing or learning from those and and there there should be some Way it would be novel and unusual. I think but that's we get the genome is to like to think we're in the novel and unusual space So so how can we engage those and maybe the HCSRN is a way to go about doing that. Maybe there are other ways But would folks agree that that's an opportunity we ought to we ought to at least try for when anyone disagree with that You only I mean it's only another half hour or so folks so stay with me So it's probably not even that long. I think the other thing I would just mention related to that since this has come up Several times relates to methods that the methods for qi research are different than the methods for traditional research and and there's the opportunity to really Adapt methods for the particular question that you're trying to answer So I think as we think about a repository potentially for methods that including qi methods would be good So Terry I would this is Julie over here. Um, I Guess my only comment on this would be to not present it in a way that limits it to Hsh whatever it is Because I think everybody can do that I mean everybody can everybody can use a qi approach and it's actually not very hard to publish Data you have to go through your IRB, but it's not really actually very hard to get approval to publish that and it may be that You know, there just needs to be better Education around that around how you publish those This is just an opportunity to say again that it rather than thinking about it as qi and research if we think about learning health care Infrastructures we don't need to make these sharp distinctions between the two we can just think of all them all as different ways to learn stuff Excellent point. Yeah, and I agree using the H what used to be the HMR and as sort of an example You're absolutely right. There are lots of groups that can do this Perhaps they're a group that we could start with because they are at least already organized and could help us point the way But there are others and and recognizing the uniqueness of methods for this research Also recognizing that again, you know for for the more laboratory oriented folks This may not seem like science and and you know, how do we deal with the areas that that don't come across? necessarily as being hard science and You know demonstrate their value in the in the genomic space, so We talked a bit about needing criteria for quality and types of evidence I didn't have the feeling that that was as as Resonating as it might be the fact that it's come up in several of these meetings You sort of made me make it blue But but what do people think is is this really one of those in the eye of the beholder and every study needs to be designed? For you know, whatever evidence you're you're or whoever you're trying to to convince basically So mark is nodding so that must be I must have hit it. I just I yeah, I guess based on my experience I just don't think that you know that there, you know There's a type of evidence that is just you know convincing to everyone And so it does need to understand you have to understand from the stakeholder What it is that they're looking for and some stakeholder groups like the payers aren't that forthcoming with what they're really looking for? So I Yes, I don't know if you have another slide on this But it but it seems like a related topic which is the opportunity for creating an unfunded mandate across all of the Networks that are generating evidence to agree on a certain types of evidence that they will collect and share so Seriously, I think that's that at least Should be reflected in my opinion is the is all the various groups from emerge to end site to Caesar and and others that are really working on some forms of evidence generation to coalesce on this topic Is that something that you know whether unfunded or funded is that in an area that that we really we could get the Programs to come together on and should get them. Well, you get what you pay for so an unfunded mandate will give you what you pay for I think that the the the idea of having a a either Basically, I'm a mandate to try to capture this kind of data in the in the projects where it fits best Are you know, there's a lot of natural interest and and gradually these can be baked in I mean the DB gap was this while wild crazy thing that may or may not ever happen a few years ago And now it's just part of Painful part of doing business, you know, so it becomes normal You know, and I think that's what where Jeff is leading a lot of these is if we started off It will become normal as we learn the rules and get it going Great, so mandate probably isn't the word that we want But but I don't have a chance to look up a synonym So so we'll put it in put it in quotes and and recognize that that may be some encouragement to try to do this across Across the programs for collecting and sharing and and this is a way that we can increase the impact of the programs presumably Skipping across the you know identifying payers needs and patient needs We heard an interesting idea about making sort of the testing equivalent of pharmacovigilent studies Is there a way to follow up the outcomes of testing? Is there a way to incentivize incentivize is a good way mandate across programs incentivize across programs But it incentivize either the testers or the people who paid for the testing to try to get some follow-up information Particularly in integrated healthcare systems, which is I understand do pay for a lot of this That might be a place that that would be useful for them and for us Can't we incentivize the patients? So what would be the incentives to the patient other than better health, I guess, but how would you do that provide? Feedback information surveys On the last bullet It's not clear what the role for NHGRI would be in that exactly. It's probably worth discussing but in the Evans Jarvik and Burke paper in the New England Journal last week They described that the FDA is undertaken something called the medical device surveillance system that looks at things like Hip hip replacements and other those types of devices and is is an equivalent of pharmacovigilence for devices So the question would be could Somehow NHGRI Influenced the FDA to take on a genomic medicine surveillance system that does the same thing and captures Data as Tesco, you know get get developed and marketed and figure out the way to capture the data Just like they're doing with those other programs Yes, so one way is to oh, I'm sorry Jeff Steve go ahead Go ahead and respond to Jeff and then I'll say You know one of the ways to get people to do things is to make it part of regulations that that tends to lead to some Unhappiness as well as but you know at least compliance if there were an easier way You know because I think the other challenge is getting that into regulations can be a years-long process But maybe I'm not understanding what you're suggesting. Well, I'm not gonna propose a solution here But I think the you know the easy thing is to engage the FDA in a discussion about this That this is an important driver for evidence generation and how do we work to in a partnership? Perhaps to do so but not be took prescriptive to say it should be regulated One thing to just keep in mind with this whole topic is evidence for what purpose and what set of decision-makers So the evidence for a guidelines panel might be different than evidence for a payer might be different that evidence for regulators So I think you need to have this conversation with respect to each set of decision-makers I was wondering following up on Jeff's point whether something like the vaccine safety data length funded by the CDC would be more analogous to what you're thinking which is more like a registry of People receiving the tests I'm not familiar with that program, but I really love the idea of creating registries It's really been informative in other areas of medicine where evidence generation is is also required for therapeutic decision-making So that's a concept we should insert into this discussion So part of in you know in clean gen there is the development of the Genome connect registry for people who have had genetic testing So that's something that's underway and and building could be a model for that Great all right. Thanks much. I think that was that's all we all we gleaned from evidence gaps Obviously, we'll be going back reviewing the videos and looking at our notes and that sort of thing But but at least for a first a first effort it sounds like we did pretty well with with that one In terms of variant interpretation I Won't read through all of these but the some that seemed to really be either novel or things that we really need to chew on one was bringing more basic scientists to the table to really help us in in Designing the research that's needed for variant interpretation and so that they can learn and this was carol suggestion in particular You know what what challenging clinical questions are being faced and how best to address them And I think we're also hearing from our more basic science colleagues that they they want their research to be relevant to disease and health I mean, that's one of the reasons they're in this business And they have ideas as well as to what might be useful and what might not and and having that dialogue could be a good thing so Seemed like that resonates reasonably well see heads heads nodding Facilitating data deposition through the possibility perhaps of covenants with a coverage with evidence development through the payers So so rather than focusing just on CMS for CED studies Could we work with the payers in a way and we've talked about before we've had some interactions with payers And could we now that the ground is softened a little bit and they're also recognizing that that they're gonna have to deal with this in one One way or another Could we perhaps try to do that kind of work? Is that something that we should make a priority or do you think? It's so unlikely that it's it's sort of barking up the wrong tree. What do you think I? Would tend to de-emphasize that not because it isn't critically important, but because there just is so little ability to engage with Payers in any sort of a centralized way and so you end up doing it You know as one offs And again There may be more of an opportunity through potential engagement with the more integrated systems that have payers as part of the delivery system Where you can have more rational conversations about the alignment of an implementation project and where in some cases like Geisinger There's actually investment from the health plan to the clinical system to do some of this in a cooperative way So I just it's our experience with trying to engage with the payers Has been that we can get them to the table and they can listen and they will talk but in terms of actually doing anything Going forward that's been where we've really had a lack of success so I would agree with mark on an individual investigator level, but as a As a health system or as a hospital I would disagree certainly our institution is are doing a number of these these pilots with The the five biggest insurers because that's our five of customers and they were too big for them to ignore And so they have to work with us and we want to work with them So it ends up being in some cases a very nice Relationship in some cases a very forced but necessary relationship, but it allows these kind of coverage with with evidence Projects to go forward and they're not all genomic either across cancer. So I think It can be possible, but it just won't be the usual won't be an R01 method So I think that's really interesting I and I wonder if there might be sense many sitting around the table Probably also come from institutions that are too big to be ignored I wonder if there's a one one opportunity might be to actually share the Specific projects that we're doing it around and some of the strategies that are being done so that we can apply the same Levers at our place. So if you you know if you do it at you know, you know the five million people at Northwestern and The four million people at Vanderbilt or whatever the number is and your place and all the other places around the table Then you might actually have an impact Some of it is like, you know, there's the blues are segmented and but at and as national I also some you know some of these are going to be better done regionally Mark, I think we're hearing some disagreement with you I know that's unusual for you to have face some disagreement agreement that is not easy, right? But the just strategically it made possible. Yeah, I mean, I guess the point I was making was again I was framing it from the perspective of prioritization for NHGRI funded research and I don't think I think Howard is that absolutely right that institutions that have Relationships with payers that are interested in this can make the push But it's it would be difficult for NHGRI as an institute to Engage payers in a way that they would be interested in in working that I think that's what I'm saying Yeah, but again, you know one of the things that NHGRI has done very well in the past is Coordinates of there were opportunities around genomic medicine issues To coordinate the people that typically sit around the table that might actually go along the way so it's against Howard code to go against a Howard but I'll go by my middle name for the next But but I think that if we were going to partner and we've had many conversations with insurers also I think the health care systems that are starting to deploy their ACO models and have resources that they're dumping in are more readily Going to be players in this because they're investing in this and the payers I think are the traditional payers are are still trying to figure out what they want to do with this and they're Tipping their toe into it, but I think getting some of these big systems and what the ACO's There's a real value structure for them down the road that they might be willing to come in and partner To to focus on how to do this and put real resources on the table that collectively you could think of it like a pre-compete Type structure, they're all gonna need to know how to do this. None of them have enough money They all have enough money, but they're not gonna spend that type of money to do this as as a proof of principle So I mean, I think that's a good opportunity Yeah, and I think that that's one of the reasons and again I don't I keep putting forward the health care systems research network is that because all of those groups are In that a ACO space and I mean they're the ones that are really doing it And they have that all that those is incentive alignment So I I think that's a good group to try and experiment with this were at your likelihood of success is higher Yeah Well, and maybe I could I could ask our colleagues in the division of policy communications and education Laura and Bob to Think a bit about you know one of the things that industry or I has done I think reasonably well is this convening power that we have and is there a way to sort of get get ahead of this without necessarily putting a lot of research dollars in it that we that we don't have but but could we in some way engage with them a Little more effectively. I don't know You know, it's just we know one thing where we are beginning to engage on the level of educating the staff at at insurers about genetics and genomics Part of the ISCC efforts, which is which is wonderful We also heard about the the idea of sort of cooperative sequencing groups like the cooperative oncology groups Which is an interesting idea from time to time. I've heard people talk about a lot of different kinds of cooperative blank groups Like the cooperative oncology groups and and so this might be a you know another one of those on the other hand It might be something for us to be thinking about so or maybe that's what we're already doing in some of our our existing studies But I guess it would be interesting for me to know from the group If that kind of convening would would best start with health systems convening health system directors or trying to convene all the blues or In different representatives of insurers, where would where would the best place to start be? So that harkens back I guess to the the previous point in terms of convening the pay or so we might ask marker or others Would be the best way to engage them or perhaps you can talk about that The trade organization is the American health insurance plans a hip but that's also not pulled in with hospitals and their directors just left So it'll be interesting to know how that will work but Again There hasn't been a lot of Energy around this and and I'm not I think we could reach out to them But whether they really would have any interest in in it's substantively engaging. I don't know I don't know who their medical director currently is But I mean we could we could the worst they can do is say we're not interested You could start with whatever the closest geographically Large a co a catable care organization is and try to work with you know They're medical director and to see whether certainly in the cancer space They they can't ignore this because the the testing is expensive If they do it send it out. It's about five grand patient They they and they won't get fully reimbursed and you're making decisions around a hundred thousand dollar a year cancer drugs So they all have to pay attention and if you're an ACO you definitely have to pay attention because you know You make you make one mistake and you sink the ship So so that I think they would be like they a good place to start just to get a feel for it If you want to do like a just once you know one such or one, you know one investigate Although by there's really I haven't found a payer yet That isn't interested in using genomic information when it's to avoid the use of a hundred thousand dollar a year medication They said don't confuse me with any additional evidence We like this answer that we don't have to treat 20 percent of the patients yet Okay, why don't we move on because we do have several other Panels to get through and and I'll just I know people are getting tired I'll just focus on the on the kind of the ones that we initially identified as being high priority So the we talked about clinical trials of added of the added value of whole genome To limited testing kind of building on the experience that we're developing in these small programs Those would be studies that I think we'd need to develop with our colleagues and other ic's Any disagreements that that this kind of work should be done other than maybe making this plural So we're not going to have just one that will do this one great big one Yeah And then then the idea of this of crowdsourcing rare variants for assessing action ability and finding Causes and treatments we heard about Matt might and and other patients Who have been patient advocates who have have used this very effectively is not a model that NHGRI has supported And our our colleague from PCORI will be back tomorrow But but obviously this is right up their alley in terms of patient You actually are supporting it in through ClinGen because the gene matcher Which is part of the ClinGen project is actually Start at being able to you know, send information around that's more at the gene level But there's no reason that that couldn't be leveraged around Variants as well. And so I think that you're funding some of the infrastructure that would allow that anyway So this is Matt might's Idea and I think it is something that could be funded in terms of ontologies. You've paid for ontologies in the past, which is Apparent oriented ontology. So one of the challenges for people that aren't medically trained and he uses us in his example Is what's the medical term for my child doesn't tear when they cry? Right, so I mean I think that's another place to be thinking about if we're going to really have patients trying to get Accurate data in they're going to write it with what they see and what they know So if there's a way to take that ontology and connect it to the hpo and other areas That could be something that would be hugely valuable and and that's something that matt's been talking about as a potential idea I would note that we're we're planning to have a discussion with some of the PCORI leadership And jeff that might be you know an area that we can raise with them in terms of you know potentials For for patient related kinds of work. Okay Uh also in in the the changing evidence or as evidence evolves kind of Range What's what would be the most effective way for clinicians to understand the meaning of variants? Particularly or especially variants of unknown significance Howard suggested, you know, maybe what we need is is like an unready an unrated x-ray Clinicians I mean I can I can go in my hospital and pull up an x-ray and try to read it myself But then you better believe I go and look at the radiologist report And and see what they actually know what the actual truth is So so a genome consult service is something that you do in your hospital. I believe to interpret these Maybe that's a direction we need to go in When people think of that Not getting a lot of traction Howard and I don't know But I think people are tired but All right In terms of metrics and and impact there was a lot of enthusiasm for common data elements a lot of Support for this this work being done in ignite which which I think those involved in ignite would note was a was a hard pull To to get done and is still a challenge to try to to incorporate in addition Measuring outcomes or identifying outcomes that are a value to lots of different stakeholders Something that we have not done terribly well I think Scientifically and things that get through review tend to be you know outcomes that will get you published in the new England journal and that's not necessarily what's a value to patients payers health care systems providers regulators etc Any disagreement with this being a sort of a key measure of metrics and impact In addition, we we heard You know might there be a way of designing systems This is a little bit off the topic But that's all right because it's a cool thing to guide the clinician to a specific test So kind of point them in the direction of this is the test that you should be doing whether those are automated tests Whether it's a genome consult service or or some other Kind of approach Can we do the research to determine when you should do that and and how best to do that? Is that a high research priority? Seeing some nods at least so okay great And engaging societies there in many ways. They're they're another stakeholder that need to be Engaged this idea of an implementation commons. I might ask whoever suggested that was that you mark Jeff You know Jeff had raised the concept And I we didn't press him on what exactly he meant by that but As I think about a commons. It's really you know an opportunity Where shared space where we're all you know using same tools same language same outcomes of the idea that It would build on what you had in the first slide that there would be common methodologies But it would go beyond that it would be an environment where different projects would be Sharing More about what they're doing and and the genomic medicine group this meeting in particular is Bringing together leadership of all of the different groups But this is really the first time we've had that type of a meeting and so you know god knows I don't think any of us want more meetings but the idea that you know having a Steering committee of steering committees where you could you could really on a regular basis share What is coming out of all the different projects and say oh that's something we need to be doing and and we could you know hammer out these You know the differences between programs to achieve more commonality The building on the the work that has been done to say well, let's have the EHR groups Talk together on a regular basis to do it more at a programmatic level That's at least we have conceptualized it. I don't know if that has anything to do with what how jeff is thinking of an implementation commons So sorry to hit you with that as you walk back in the room But we were we were a little curious as to as to what an implementation commons would look like if you can Yeah, that's right Well, I think I'll also see if julie or url and want to comment But I think in the ignite program, um, you know, whether it's implementing pharmacogenetics Genetic risk testing family history, what have you there are going to be some there are some common themes Um and challenges that we've all faced and that we have Begun to overcome or are overcoming as well as things that are pretty unique to those those areas and You know our intent is to write those up, of course for public dissemination, but As other programs which are doing similar implementation programs for genome sequencing and the like Could could contribute to that and make it a almost a repository perhaps with some metadata that would give an in a manual of operations essentially for The the the less genomically inclined institutions to know what they're going to need to do to Bring some of these technologies in could be quite valuable So this whole topic raises some actually quite fundamental sort of conceptual questions I don't know where those get addressed whether in this portfolio or in other portfolios, but um So the traditional measures of value would have to do with things as was discussed earlier like how long does somebody live quality of life Cost the traditional cost utility analysis can be reduced to those things But here we're talking about things like what's the value of ending a diagnostic odyssey What's the value of the reduction of uncertainty that comes with getting a diagnosis? Some of these things can't actually reduce be reduced I think to any of the traditional measures of value So we I think we need to think a little bit outside those usual measures to how do we value At least some of the outcomes of the genomic medicine at least where it stands today Great. Thanks. And I think mark had called on julie urban if you wanted to make any additional comments or are you good? Guys we're almost done. All right. So in terms of uh, EHR functionality Rex's group. There were a number of things raised I'm sorry. I didn't I didn't highlight any of of these forgive me And and perhaps they're all important. That's right. And we and we can't You know One that that seemed like we you know is a real opportunity It was this last one in terms of multiple training programs So so we heard there were a number of training programs now ongoing and in EHRs and and informatics and you know with the big new initiatives You know, couldn't we engage some of those trainees? We're we're dealing with I think some some really thorny problems that that could be solved informatically And we don't have the resources or tools to do that And and is there a way to work with them to say, you know, hey, they're you need a fellow's project Can can you work on one of these? Maybe that's something for us to think about a bit more So we can make that one. Make that one blue real quick here All right, and rex were there Others from your group. I know I'm putting you on the spot We've actually we have a a second one and then I'll ask you if there are any that we missed that we're we're critical um Something that seems to be a really genomic research question that that we should be able to address is when is the phenotype measure superior to Or does it add to information from the genotype and we heard a couple We heard you know one example of tpmt where where the assay because you don't know all of the variants you can measure them But you really want to know the activity as well Did I get that right mary? No, not quite. Well, I mean That's the limitation of every genetic test is that you don't know that you're finding all the Great Damaging variants. So I mean I just I just don't see It's always going to be true that knowing phenotype would be helpful The downside of phenotype is is that it usually changes over time And it might be artificially affected by biochemicals floating around in whatever tissue you're phenotyping Exactly so I don't again. I don't see why it really setting up an either or dichotomy seems a little unusual to me I mean we There's many examples where we'd like to know both if it's not too expensive. We want to know both. Hey, all right So let's un unblue that one. Yeah But although I might I might note that that hfv the hemochromatosis variant was one that that actually nh lbi and hgri Did an entire study on the air study trying to ask that question And it turned out the genotype really wasn't all that helpful because the penetrance was so low And and really what you want because there's gene environment interaction and other things now you disagree with that mary You look like or oh good. I was reading you're wrong The emerge data at least for the homozygous would contravene that argument there's much higher penetrance than was reported in the study and that May go to show the difference between a study versus a real world setting Um, so I mean those are those are important considerations and and you kind of put it on mary I think I was the one that opened my big mouth about the enzyme activity versus the The genotype but and I think mary is absolutely correct that you know You want what you want is what's most informative For the patient and I think sometimes because we're titled the genomic medicine working group that we focus in and And assume that because genomic is in our name that genomic is always the best But I think we need to be Of the belief that there may be situations where genomic isn't the best and it would be incumbent on us for To say you know in this case It's really not the genomic information. That's the most relevant. It is somewhat something else whether it be phenotype or Or some other intermediate biomarker or something I think in some cases when the phenotype is in some ways more informative than the genotype It's because that phenotype is closer to what is clinically actionable and clinically meaningful It is a it is a downstream endpoint. That's closer to what we can actually do something with In some cases the genotype information is too far away from that endpoint for us to effectively translate that knowledge into something that we can Act on in the clinic. So One of the caveats to this point could be As our understanding of how to translate that genetic information Into a more accurate phenotypic model of what's going on at a cellular and tissue level improves Our ability to utilize the genetic information to make these kind of predictions will also improve But right now we're being limited by our limited ability to do that translation as those models improve that'll change And I think you can imagine cases where actually knowing the genotype might cause you to look more carefully for phenotype in the future So, you know, there's that value as well It sounds as though we agree that this is an important topic Maybe it's not the most important topic when it comes to each our functionality But one that we so we're gonna we're gonna leave it as as a white one. Okay Great and and I I think we we did hear That the the value of precise phenotyping may vary depending on the frequency of the variant and perhaps It's more important with the rare variants than it than it has seemed to be with the common variants I'm I'm not sure that that has been proven yet But it's one that that seems to make sense. Would everyone agree that that's probably a direction We would we would agree on at least or something that needs to be demonstrated Or is that not a terribly important thing? We're going to be doing precise phenotyping anyway with rare variants And so so not something we need to highlight I think it's really important. I think we're talking about rare highly penetrant variants Where one or two variants probably are governing a complex phenotype medically complex patients And there I see it just every day as a real bottleneck for us Well, I think the the point that was made about the study earlier was that um, what we're really talking about is The the asymptote in other words how much phenotyping investment do you need to get the answer? That's good enough to do what you need to do And I think that this is a subset of the point that for things Where you have larger numbers of patients and and larger numbers of variants that You will tolerate a fair amount of noise But you know with things where there's very rare variants and you may need to invest more But I think it really is more research around the idea of You know, what is a good enough versus a perfect phenotype and what's the best way to To to use resources because you could expend infinite research It's a speed of light question you can keep spending resources and never actually get to the perfect phenotype I also think coming from the you know, the cancer genome atlas perspective one of the most remarkable things Oh, sorry. I'm back here in the corner um You know coming from the cancer genome atlas perspective the most To me that one of the most amazing things that that project has done is help redefine What do we mean when we say different types of cancer? And some of that starts to become using the molecular information to define the cancer But then I put my epidemiology back hat back on And I start to think well what happens if we go back to GWAS studies Defined by these different molecular types of cancer because it may be that the genetic Predispositions for these different cancers come into play and so I just wonder as genomic medicine goes forward Are there going to be similar? Redefinitions of disease or cat sub categorization of disease That you're going to need to have this more accurate phenotyping to be able to sort of see and research and capitalize on So that's another argument in the more precise phenotyping corner It perhaps another way of thinking of this is not just more precise phenotyping But also thinking about secondary phenotyping if you look for loss of immune loss of adaptive immune function That's quite common and it happens through a number of unrelated mechanisms But then if you look for unusual overlap like loss of adaptive immune system and no hair from birth turns out that is a rare Particular set of causes and the combination of phenotypes leads you to the correct cause of disease and the correct disease subset Cool. So so not only the precise phenotyping being what you're what you think you're looking for and sorry I can't see you mike, but but anyway, but also looking at sort of related things or secondary things. Okay Great. All right. So rex Did we sort of pretty much capture you think at least as a first pass the key things from your group? Okay, and then last but not least on the on the diversity issues a number of things that came up I think one that was that would be helpful to us at nhtri is to identify specific health disparities research questions related to genomics that we can Then you know study in a variety of different ways and you know, obviously there are Mendelian conditions that are unique to to Just about any ethnicity but but also when when looking specifically at health disparities how to address that And then the point being made that that you know diverse populations are particularly important in pediatrics I think thanks to steven others for making that point That that given that we're we're just about it a majority population in the is this under 18 under 12 Steve do you remember which what age group you're The way that the thing I looked at it was the 56 was 5 to 13 It's actually going to be even more and you know, it just gets more and more Minority as you get younger So it's like 58 percent in the 14 to 17 year old and then I kept going down by a couple of percentage point in each eight bracket Really? Wow Great All right, so so that seemed to be both of those seem to be high priority areas for diversity any disagreement Yes, ma'am Um on the last one. I don't know if you have this on the next slide But could the last one also include interpretation discovery and interpretation potentially? Uh, so on the on the last question So any better methods to utilize ethnic genomes for discovery and interpretation interpretation sure Discovery analytically and interpreting I'll fix it later. So, okay great. All right And then other things, you know I think it got some interesting points about giving community advisory boards boards not only a voice but you know a strong voice And and independence and autonomy and the ability to push back When needed against investigators maybe not against but at least To to be sure that they're heard We we talked a bit about Dedicated programs for non european ancestry populations for a whole variety of reasons Those have seemed to be successful in some settings and and maybe that's the you know We need to go further in that direction again event something that we could talk about Um, and then and then one kind of wonders is is there a way to make genomics? It's kind of a special draw Is there is there something unique about it that we could attract train other than the fact that it's super cool And and you know, everybody should be a training in genomics But but what is it about genomics that that would be particularly attractive to non european ancestry trainees And those seem to be areas that that were you know, kind of kind of real areas of opportunity any disagreement about the about those as being Useful areas to pursue Great We're just about at the end And five minutes early. I think so eight minutes early. All right. Fantastic So a reminder we'll be starting again at eight 30 tomorrow morning. Okay. Um, and yes, howard Could you send these around? Sure for this evening? So when we're less brain dead or at least when I'm less brain dead I can look at them again. Sure Sure, be happy to do that. So I'll just uh reply to to read his note to everybody So we'll reconvene at 8 30. Thank you all very much for for your attention and participation. We'll see you then