 So this is the part where we wrap everything up into a tidy bundle. Go ahead, Rick. So we've heard a lot over the last couple days, and we are going to try to use this time to see what additional pieces of it that we heard, separate pieces, can be brought together further. What we've heard so far is very interesting, I think has been very interesting, and very valuable. It is also very expansive, and so we're going to have to be thinking about that as we go forward. I just want to start asking about, I mean, now that you've heard the discussions of all sort of three major parts, if there are any other points that sort of apply across all three parts, we heard the importance of getting really accurate sequencing and developing those capabilities and how that's clearly important for just about everything we've talked about in the long term today. But I was wondering if there was anything else like that that has occurred now that you've listened to all three parts? And the other kind of question that I want to ask that's similar to that, I'd like to find what among all these three parts are useful to bring together scientifically. There might be none that could cut across all three parts, but anything that suggests where the scientific interactions have to happen between these three, between the sort of functional, the clinical, and the discovery parts. Some are obvious between discovery and clinical, and some are obvious between maybe discovery and a function, but is there anything else that ties across all three? So I'll just throw those out to start. And if I can just add to that. So this session here is to start to think of going from these strategic things that we've heard about to more tactical things. So as you think about your answers, think about what tactics you would use to implement some of these strategies. Eric, go ahead. So I'm struck that a thread that binds all three together, it's not a tactical component, but the thread that binds all three together is variant discovery or gene discovery for disease. That basically that's a first step, then taking that to mechanism and taking that to the clinic to improve diagnostic rate. Again, if you're looking for threads that tie all three together in a very clear way in a five second elevator speech, it just seems like discovery for health and disease is what drives this. And I also want to add that not, sorry I've got UW in a second. Add that also in bounds in this discussion are particular that if you imagine that you had three general kinds of programs that did the as we sort of do now, but maybe some modification of that or evolution of that, these are suggesting to us areas where we might have to build in interactions rather than having silos. So with that said, Debbie. I think the technology also binds them together. I mean really it came up a narrative, but putting the W back in whole genome is really important. Okay. I think that thinking about how to do that, and Heidi should get the credit for that one, but the moniker is true. I mean I do think the technology binds them being able to dissect variation at different levels and how it gets implemented in diagnostics even prenatal diagnostics is extremely important and so I think that ties together across the whole array of applications. And I think that's something that should always be in the mantra of genomics. That's the reason it started. And Jeff, did you have your hand up before? Yeah, I don't want to, we should continue this part of the discussion, but I want to come back to this integration question. If people have really creative ideas about how we can most effectively integrate rather than just add another 15 conference calls. Yep. And there was a hand up over here. Well I was just going to say another theme I think we've been hearing is longitudinal phenotype tracking among large numbers if not every one of the people who are getting sequenced for clinical reasons. Whether that's through a learning healthcare system or some other model. I think that's a theme we've been hearing throughout the several days. I just want to amplify that. I think that the longitudinal aspect of all that, as Cycler Robert said earlier about some of the most important things out of Cesar won't even be, haven't emerged yet, no pun intended. So I think the longitudinal aspect is critical in all of this. Data integration is of course a perennial binding issue. We haven't spent almost any time talking about biocomputing bioinformatics and I'm confused as to whether we've transcended that as an item because it permeates everything or if we've just forgotten about it for now. It's too hard. It does need its own discussion, but it has come up in key places and I think also the data integration is part of that as well unless you weren't including the data integration. Actually maybe we can use that for a second or two to talk about tactics then. What tactics should we be thinking about? Everybody talks about data integration, but when you think about all the different data types that we've got, we've heard maybe a dozen different functional assays that have been proposed by people, all of which the data is a different kind of beast. We've heard about lots of different structural variation, which I think is another topic because we haven't figured out yet how to measure all that structural variation. So what strategies or tactics do we have to integrate that? So you and I think your hands are up. I don't, I mean this is the thing. I think it's become so pervasive that we have stopped talking about it and that's why there are bioinformaticity around the table. I think that's a, at one level that's a good thing. I remind people that there's these very often, I describe this as there's these two sides of this. There's what I describe sometimes as blue collar, which is sort of making sure your data is straight, keeping it straight, keeping the metadata straight, making sure it flows correctly, making sure people can get access to it. That's very often quite engineering heavy and there I really feel that we keep having the set point of the teams. I know it's like the fourth time I've set it, but goddammit. The size of the teams need to be in sizes of five engineers at a time or something like that, rather than one or two people. And then we have this much more sophisticated what I describe as white collar problem and there I think it is and that is both the things associated with this big data world and making sure that NHGRI really is using and attracting the best people in this area is very, very important I think because I think there's a huge amount of problems that are sat on this side and there is no magic bullet. There's no magic thing that says, oh well if you only pulled out this method it would work. And so the most important thing I think is to keep, is to invest and fund in both of these areas correctly. And then NHGRI has a headache about coordinating with BD2K, with other ICs, with medical informatics. I don't think it should be a headache to coordinate with BD2K. I mean we should relish that as an opportunity and it gives us a much better leverage. The thing about bioinformaticians is they're capable of building things that immediately apply across the institutes at NIH and it's frustrating to have them to be things being located in different institutes in an inconsistent way. So I would strongly encourage that. And I would also exhort everyone to remind themselves that the white collar bioinformatics even describes it. Can we not call it white collar? Yeah let's not call it that. But bioinformatics at that level. I don't know bioinformaticians are part of the proletariat. Okay rejected your analogy but the deepest form of bioinformatics is a research enterprise. And it has to be supported as a research enterprise. Yeah I think specific incentives for data integration should be built in the form of RFAs or something like that because you can't expect necessarily different groups to continuously integrate or the data to integrate without specifically incentivizing that portion. So this is an area where I worry that we really have the wrong picture in our head. I think it's good this has been raised. I agree with Ewan and this is what I mean. First of all what we need in some cases is not new methods and technologies but we have sort of an organizational problem. Which is that we don't invest in interoperability we know the white collar whatever the high level bioinformatics tools are all non compatible. Some tools get used a lot but most people write their own. There's really no infrastructure or platform or whatever that the field relies upon. Even things that have been successful like file formats. I often hear people say well we have BAM we have VCF. The history of that is probably people know was there were no file format so at the 1000 Genomes Project Cold Spring Harbor we sat people in a room and said you have to come up with a file format and then there was no governance of it or evolution because it didn't belong to anyone or anything. That's now taken on by the global lines data working group to move that forward but we have a problem which is the picture in our head is often to do projects to people write a program and then we're surprised when it's not interoperable and the solution is not a monolithic approach. It's not to have the big database in the sky or to force everything together but we do need to change how we do it. The last point I'd make focusing more again on how do we learn from other areas where it's a virtuous cycle it's not how we work how they work. The other point I'd make is BD2K at least to date I think with Phil Bourne and maybe it will change does not see itself as I understand it in fixing this organizational problem or taking on some of the tasks that we talk about at least to date that will change has been focused on fundamental data science. That might be Eric Green who ran it up until now is nodding his head yes but so I worry a lot that the problems we're talking about are not generally problems of fundamental data science and then everyone goes oh BD2K will take care of that. I think it's highly unlikely as currently configured maybe Phil will change it because he's a great guy and he's starting but I just worried that in the discussion he goes yeah it's going to be BD2K is going to take care of that and there's no plans for BD2K to take care of that so no one's going to. Can NHGRI do to try to engage and fix that problem and NSF won't take care of it either just so I mean some of the things David may want to comment on others is again trying to figure out what limits people doing these things okay some of its incentives but incentives can be created by you know as people said grants require people to share we see with DB Gap for example that people are required to share but for a lot of architectural and regulatory and other reasons it doesn't actually just flow all that smoothly so one thing I don't want to be a broken record and this or suggest it's the only approach but trying to work on shared APIs try and use what the people use in other fields of processes to develop open APIs to let people iterate them to try and parse the problem so people can write tools that are interoperable and plug and play etc. David might want to comment more but there are some things and they are being working on the question of what can any share I do well it could support some of those things but it could encourage through its grantees to actually use such systems in some ways or measure whether or not data is flowing as opposed to just say did you check a box you know I don't want to make up on the fly what any share I can do but it's not going to be start with a new monolithic approach it's going to have to be figuring out that strategy. Sort of a meaningful use for genomic data. You know I think that this was what the meaningful use standards were supposed to do and didn't do but we actually need to look to other industries or other places because the whole point was APIs were supposed to be wrapped around this part of the meaningful use standards and it didn't happen. Doing deep bioinformatics in the context of one of the truly great challenges that we have discussed at this meeting is a tremendous opportunity. That's really you know you have to embrace the bioinformaticists as a key part of the team, fund them appropriately to get their part of that done and let them be a real team member with these great challenges. It's a simultaneously working out the new theory and the deep kinds of things that they would actually get credit for if they were in a computer science department. You got to do that but doing that in the context of a grand challenge problem that the NHGRI has identified so that you're developing the APIs and then you're working on deployed implementations of them at the same time so they're proven in the field and we're actually making great scientific progress with them. That's a huge opportunity so we just need to bring them into that. Give them the opportunity. I want to echo what David says but just to also say that the APIs are necessary but nowhere near sufficient to start on this. If you think about how Facebook or Twitter or Google work it's an incredibly well orchestrated and designed and built set of systems in order for you to have the functionality that you have in those settings. The set of problems that we're trying to challenge. I think this is the wrong analogy though. I think the analogy is we're trying to create the Internet. I think this is one of the huge problems we have as a field is that we think that what we're doing is creating things that sit on top of an Internet. Internet allows data to flow. Once data flows people can add layers on top of it of increasing value and right now data doesn't flow. Data is isolated and siloed. I think the reason why Facebook isn't a good analogy is Facebook is a big closed system there. That's not what we're talking about. I think it's a question of wrapping metadata tags around pieces of data and what's true about those sorts of things, XML type implementations, is they're extensible. They tend to be things that can be started lightly and grown and grown. I do think it would be worth getting sets of people from the tech industry in but Craig Mundy at Microsoft talks an awful lot about this about ways that you build a system that start light and grow rather than a Facebook kind of thing. There are a few of us in this room, myself is not one of them, who are deeply experienced. David Houser knows a lot about these things. I think the NHGRI could benefit from finding a handful of distributed tech advisors who do this routinely and have them come and look at what we're doing and say, yeah, we were in this position for certain things and we got out of it this way. My point about the meaningful use standards of the Office of the National Coordinator for Health Care Information Technology was there was a recommendation to do precisely this and it wasn't done and there were a set of people who tried to push them to do it. We could get them in and help us because there's an evolutionary path that doesn't have to be perfect at all. It's quite incremental but it would help. I'm not disagreeing with you, but in order to do the large scale analysis, people are going to want to build bigger systems in order to put that together and that's not going to, so the extraction of meaning from the data isn't going to come from the API but what comes from the API is the ability for third parties, the graduate student here, the postdoc there to write a great piece of code that plugs in and it unleashes great creativity because there's APIs. That's all. So I think one of the biggest challenges to integration is not data formats or APIs, but going back to the semantics of what we're talking about. So phenotypes, defining phenotypes in a way that the data you share is meaningful. You put a bunch of clinicians in a room and you ask them to define what schizophrenia is. You're going to get about a thousand different sort of descriptions about, so how do you integrate data on schizophrenia when the phenotype definitions being used to collect the data in the first place are so divergent. So I think semantics and metadata tags, as Eric said, I think that's really really critical to data integration. That's one. And then the other thing is how do we represent uncertainty about the data that we're generating so that it can be computed on. I think that's another critical issue that we really haven't addressed. This is a great topic. We have a few more but let's take two questions and then move on to another area. So I just want to sort of second a little bit what Carlos was saying. I mean you can sort of ask what comes first, the API of the standards, or then tackling the question, the research, or do you start to think about the question and the research and then think about the type of interfaces and APIs you need. And I would argue the latter. I mean I think you fundamentally integrate the data and put it together to get good people to work on it. You really have to have clear research questions, people really thinking about interesting problems and then they'll build all the interfaces. But I think you need that to motivate things. Yeah, but isn't that we've been doing for the last 25 years? No, I'm serious. I actually mean that in all seriousness, at least for myself. I just think here we're going through the same growing. It's been very revealing to me the conversations we've had with the CERN data group, which has grown up over the years around the LHC. And what's interesting is that they went through a period where the data stuff inside a CERN was sort of hidden inside of their projects and they didn't externalize it and they didn't think about it deeply. It was only in the 90s where they really sort of said, wait a second, we've really got to treat the data stuff as a separate part. And they made it, I mean it's in a very certain way, we don't have to copy it. But what they went through socially I think is what we are going through at this moment by saying we need a data infrastructure, which is data APIs and backends and all of this stuff, that allows us to do good science on top of it. And they viewed it in very much the same way as they talked about the ring as being a piece of infrastructure for physics in which you do great, clever physics. But the making of the ring is an engineering task. And so I just want to say that when we're drawing, I mean I just think we're growing up as a data science. This is part of our growing pains. There is stuff that we should carry forward from what we've done. So for example data openness, compared to many other sciences, we are naturally open in a way that many other sciences aren't, including high energy physics for example. So that's a good thing to take from our history. But there's some other things where we just got to leave it behind. Like our own kooky file formats, we've just got to leave it behind. And you know, so there's not a revolution or an evolution, we've just got to mature as a data science. And in particular as you, I agree entirely with you and I agree also what you said about the engineering teams can't be, you know, three to five people because we've tried that and what you do is you end up with lots of things that don't connect. I do think it's very informative and if NHRI hasn't done it, they should. Some of us have done it in other settings. Talk to a bunch of people who do work in tech or who have tried to get into genomics because they all have looked at it. They think it's an analogy. They just are perplexed and dumbfounded at how we do things. The utter absence of any standardized interfaces or any group that comes together and tries to does that, when we set up the global line, it's not to keep doing this, I can tell you what the Google's and Amazon's and all these people said was, oh now we might consider working in this field. One of the things that has stopped us in our tracks is that you could talk to 10 different people and hear 14 different things about how you should do it and that makes it a market failure. Okay, now whether, and I sure I can play an important role is I can't do it alone. But for example even just aligning that it will support things and also one of the things they tell us about standards is you can't have 16 sets of them and one of the things about our community is whenever you talk about this someone goes, yeah I'm doing it and someone else says we should have a meeting to talk about how to do it because I'm not doing it. And the problem is if you have many different sets of standards then no one can from the outside build anything and have the hope that it will be a market because in the absence of something to plug into, if every wall has a different plug no one can sell a toaster. And so any year I could that's what we're talking about guys. We're talking about like build anything because it doesn't plug into anything else. And so any year I could line up behind those things, try and line up BD2K behind them, not pick the winners but set conditions on which there can be a virtuous evolution of a market based on what actually works. So I think this has been a great discussion but there are a few other sort of common themes that maybe we should move on to. But one of the other common themes that I think we heard that crossed all three boundaries was this whole issue of we don't know how to yet capture all the kinds of genetic variation that exists in genomes. So we heard about today the idea of a telomere to telomere sequence. We heard about how to think about doing that across evolutionary space so that we understand a little bit more about what's conserved and what's not conserved. So thinking about that, what strategy or what tactic should NHGRI be thinking about to help advance that project of being able to capture all kinds of variation in genomes? Well I was just going to say to me that's a critical piece that goes across everything we've talked about. And one could certainly imagine taking the Mendelian project for example and taking the families that really look the most Mendelian for which no alteration has been found and to do these absolute platinum genomes on those families once we do them on normals and know what's missing. Because I think in the commercial space and the clinical lab space we compete on who's got the best coverage but what are we covering? We don't even know what genomic regions were not even sequencing. So I just think having incredibly well back to the W and whole genome that really informs the Mendelian project, it informs the CESAR projects, it cuts across all of them. Yeah I can just say from the 1000 Genome Structural Variation Group one of the things we're finding is we do need to encourage input of a lot of different technologies together and integrating that data properly and so anything we can do to encourage that from NHIR would be great. So what would that be though? Well so in our case it's for example incorporating data from packed bio reads, integrating optical analyses data for example PCR free DNA libraries etc. and then that with a huge amount of validation using orthogonal approaches as well and we're doing this actually in a U41 setting just as an FYI so it's working in a very concerted effort. I'm sorry Debbie, I want to pick up on something you said because I always think about the balance of what can be done in a standalone technology development program or center or grant what really benefits from being, having integrated different data types in more of a center that can do a bunch of things but may not focus on any particular tech development. So where's the right, what's the right way to balance that or to handle that? Do you absolutely need both or is it? Yeah I mean personally I think you do need both. You need to encourage the development of the new technologies and then when those become available to try to integrate that is and test them robustly as in the larger scale. So I guess another part of that question is so we're not looking at in the next four or five years predictably a magic bullet technology that's going to do the end-to-end chromosomal sequence saying nobody really has faith that that's going to happen and to the extent you need it. I mean I'm going to add a little bit of perspective I guess on this. We have spent quite a bit of time with the packed bio one platform and certainly there are regions of the genome that we still can't assemble with packed bio technology. We've looked at this specifically we know where those regions are but the regions that we can access and assemble in a routine fashion as opposed to a targeted mom and pop operation where we go after each of the difficult regions is really diminished significantly and I'm actually quite excited by the potential not just for packed bio for any of these long read technologies it's something that we brought up over and over again over the last ten years. The importance of long reads in terms of comprehensively accessing genetic variation still remains really high but I think if we had an increase from let's say 30 kilobase reads to 50 or 100 kilobase reads we really could be talking about I think we're at a transformational position right now in terms of de novo assembly but it would be another catapult that where NHGRI could really invest in really advancing and helping advance sequencing technologies to get us to that next level. I don't think we're ten years away I don't even think we're five years away. The technology exists now to increase by two orders of magnitude the M50 contact line. We could get the gold genome within three to five years and we didn't do that. That I understood but I understood you were asking for the platinum genome and I was wondering about the difference between the gold and Evans 90% of the source transmission. Evan made a distinction here. Evan made a good distinction right so the gold genome is going to be good enough without every base of every center mirror to do a lot of stuff and we need to go after that aggressively and we can get it in three to five years. And the gold genome gets how many, what percentage of your remaining insertions that you're worried about? All. Gets all. So what it gets it gets all the eukromatic variants without with irrespective of whether they're inversions insertions, deletions, irrespective of size or complexity if Evan's happy with it we'll all be happy. Okay so telomere to telomere is I mean that's a technology goal and that will take some time but you know you said three to five years for a gold genome. You know it like kind of was alluded to earlier this morning with PacBio I think it's reasonable to do a gold genome now and it's more expensive than an aluminum genome but it's not intractable. We're talking about ten thousand dollars. Well let's even say it's fifty thousand dollars but I think you know there's a I don't think this paper's come up today yet but a recent paper from Han Bruner and Joris Weltman I think in nature kind of illustrates how much we're missing in exomes when we I don't think we had a real appreciation for until you sequence a genome and you see how much you're actually missing and I don't think we have an appreciation now quantitatively for how much we're missing even in the genomes that we sequence and you know doing fifty or a hundred PacBio genomes would go a long way towards serving that goal and not waiting three years to do it but just going ahead and finding out what happens and that's something we could do tomorrow right and it would also serve the goal of supporting a second technology other than aluminum. Yeah I think this is a good thing to shoot at I think it's a good thing to use program announcements for that rather than huge efforts to do it there's a lot of creative ideas out there PacBio's a good one maybe there are a bunch of other ones too but right now at study sections new assembly programs don't get reviewed well because there is a sense on the part of reviewers that well it's a solved problem or kind of a boring problem or filling in some of this stuff is kind of not so interesting. I think you merely had a program announcement you'd begin to get some RO1s maybe not to do fifty but give me ten good genomes like that which at these prices you could do I think there's a lot of creativity high seeding to be able to jump over things other kinds of interesting long range technologies and it would be good to get a bunch of creativity what I wouldn't want to see is a single monolithic large project to get ten or twenty or fifty gold standard genomes what we actually need is a way to really be able to turn the crank and make them and I think they're a program announcement that signals to study sections fund some of this stuff could be a good thing So just to say and you know Oxford Nanopore is the other technology here and for a long time they've not seen a change in signal as the read length goes through and it really does look like it's limited by sample prep so the interesting problem there's a problem there's things about tuning up the chemistry and the readout system but in fact some of these challenges are going to be about the up front process of sample prep and delivery to the poor rather than the actual process of reading it so that telomere to telomere view might not be quite so crazy actually if you can get the sample prep in a five year time scale which is kind of awesome I mean I think this discussion is a really important one and I agree it's not just packed bio and it's not an either or it's not like we wouldn't do Illumina or use other technologies but I think one thing that has been touched on a couple of times is what's the best way to do this my sense is there's still a great advantage to having the large scale genome centers be involved in these activities largely because they've been involved in a lot of large scale sequencing these are to generate for example 250 flow cells of data form or smart cells from a packed bio does take some time it takes months and there's processing involved there's management there's annotation of sequences there's algorithm developments there's software developments and I think in terms of the mechanism for this if we're going to proceed in this direction is to involve maybe small groups of individuals in conjunction with large scale genome centers to really pull this off because we do need still muscle this fits perfectly with the seven characteristics Adam that you mentioned in the beginning you know in terms of scale and consortia and so on but there's like a really it shouldn't be too big and it should be kind of a pilot in some respects but at the same time you want to involve enough muscle and enough expertise to get the job done so this is something that came up in the comparative genomics and evolution breakout this whole concept of organizing this around I don't know an encode type consortium where you have technology development which is aimed at this 10k de novo genome where you have analytical development which is about assembly alignment algorithms you know element discovery and where you also have production of let's do a bunch of seeds for you know for all of these programs to work on and these seeds can be a bunch of different species a bunch of different populations a bunch of different individuals which can be serving as gold standard references so when you put all that together as a vision I think you will get buy in from both the technologists and the computational folks as well as the sort of reference both species and populations just a little point there if we add if we keep the data producers being the genome centers we have the Bermuda standards which means the data get out there they get out there and they become accessible immediately which will then create the whole proliferation of new algorithm development and software development I think there's a real benefit to not having that be localized to a group but to be distributed in terms of data production so the quality is there this is an important issue for the Mendel projects as well I think there's nothing more frustrating to have all of the family reagents in hand and still not solve the family and so we're always wondering whether it's somewhere in regions that we can't access and I would also urge Evan mentioned this morning that would be great to have this kind of resource for at least 50 different individuals from around the world because we get samples from as the point was made earlier from all over the world so having this kind of gold or platinum genome from a lot of different populations at least one representative of a lot of different populations would be quite helpful I think so can I transition slightly but build on a point that was just made so I'm going to refer to what I'll call the Bustamanti matrix which we saw this morning so what is the right balance you know this raises the whole question of what's the right balance versus a large scale project versus U54 slash U01 type metric versus an R01 and so getting the right balance of passionate PIs whose necks are on the line versus the cost efficiency and production standards of a large scale center and then the probably somewhat in between of the U54s and U01s what's the right way to think what's the right balance of the portfolio for again thinking about strategy or I'm sorry tactics going forward Adam you said you thought it was about 25 to 30 percent R01s right now 10 to 15 so what and then factor into that the use of program announcements that can actually guide the direction so what are people's thoughts about what the right balance of that is conversation killer so I think on a meta level that the NHGRI has actually been doing this experiment for the last decade or more to try to figure out how to do science at scale by mixing these various types of things the large scale consortium projects the directed projects the R01s and given the outputs of NHGRI funded science I think the experiments have been arguably largely successful but that doesn't mean that what has been done in the past should guide what's done in the future so I think probably there's no direct answer to the question that you asked as to what the right balance here is but the way forward is to be responsive and to probably keep experimenting between large projects R01s and as you and described earlier you know when or at least be comfortable making a bet to take something into a large scale sort of consortium project as it gets to the tipping point so it can be pushed into that and obviously there it won't be perfect but I think it's something to keep experimenting and something that NHGRI has been good at experimenting with and getting good outcomes over time at the risk of being accused of putting the shoe on the other foot I agree with the comment that was just made but as a follow on to that one has to measure the outcome and I'm wondering does NHGRI track this these experiments in a way saying you know if I add 20% of this or 30% of that what do I tend to get out over the next few years What's the metric? It's really hard to measure It's a hard metric but having been on the other side of that equation I've been asked to come up with metrics It's so context dependent I can think of a number of times when we've had R01s that came in and were really stretching the boundaries of what an R01 was and actually ended up being a U grant later on because that was most appropriate for that scale of effort So something that hasn't been touched on and NHGRI has done very well and I want to make sure it's mentioned is NHGRI has provided strong project support in the form of very strong program officers who often could steer and support a project without necessarily controlling the funds and so for example like the 1000 Genomes Project which has been a coalition of the willing with like almost no dedicated funding not really a strong governance model but if you ask why did that project succeed there was a coalition of funders who did support individual activities but then there was Lisa Brooks and the team she worked with who really were a glue that held that project together and I've worked with many institutes and like you can have a mechanism that's supposedly a very collaborative mechanism but where the inmates, the animals in the cage are clawing at each other and no zookeepers actually having them go in the same direction mechanism would suggest they're all going to work together and then I've seen others where no one knows where the funding's coming from and yet everybody works together really well and that's a testament to good project leadership and I think that's something in my experience that NHGRI, no offense to any of the other institutes around the table NHGRI does better than anyone and so what is the action item like value, support and recruit and retain people who are really strong program officers who don't try to control and dictate but do actually add value and you've done that well and you should keep doing it. So I always wanted to be a zookeeper when I was a kid. Which animal? Which animal, Adam? I won't say which animal. I agree with you David and actually it is although it's not an overt consideration I talk about it from time to time about models that scale to allow that I will also add that although sometimes we love to take credit for anything good that happens that in fact a lot of the amount of management things need, amount of people time things need very much depends on the individuals who were involved and also on the institutions, expectations of the institutes that may be substantive partners in it and I think those variables are actually larger than what you said but both are really important. Thank you. Debbie. I want to change topics again. Is that okay? Education, it hasn't come up. I have to have to bring up education and we have to train more people in genomics. We need to continue that trend. We probably need to double what we're doing in education in genomics as it takes the mainstream and I don't see us going there. I also think we need to change the people at the table. I see lots of gray here. I want to see lots of young in the future right? More young. It's younger than it was the last time we all met. I have to say that. I'm very happy. But I want to see more young people at the table too. So I'll just start it there. Education is really important and I know genome has always been interested and supportive of it but I think we have to be even more supportive in this climate. So what's the strategy to fix it? I mean we need to increase training at all levels. I mean we're not, we don't have enough training grants, we don't have enough slots on training grants, we don't have enough anything on training grants. We're not diversifying as much as we should be on training grants. Where do we need to go to do that? I don't know. Genomics tries harder than I think most groups do. But we cannot give up and we have to move forward and improve this. I always jump on this bandwagon when it gets brought up because I think it's incredibly important and at most of our training programs today focus on bringing physicians into science and funding their research. But the reverse of that, I get, you know, I've run a training program in clinical molecular genetics to bring people to do clinical genomics for the last seven years. I now have applications. I got 65 for one slot last cycle and these are incredibly talented, largely PhDs, interested in moving into the clinical translational space. And there's no they're all excited, young, energetic, and I can take one or two of them. And this is the same with the other, you know, handful of programs across the US. So, and all the programs complain because there is no source of funding for the PhD training in the clinical space, the reverse of it. So I think this is incredibly important, you know, opportunity to harness the young, you know, PhDs that could move into this space where we do something right. So I'll get David and then Jim, but you know, one of the things this flies in the face of has been a lot of fairly high profile papers in the last year or so that have talked about the fact that you know, maybe we're training too many people and it may not be we're training too many people in the area of genomics, but we're training too many people in the area of life sciences. So I just think as we think about that, we've got to think about this as an environment where this issue has been raised, I think, at the fairly highest of levels of our sort of scientific community. So, David. Yeah, I would also like to mention medical students. We've had a lot of disparaging remarks in the last two days about how physicians can use what they are able to interpret out of these data that we're generating. And going forward we have to have a much more sophisticated physician user base, I would argue. And if NHGRI could provide, I don't know, modules of information related to clinical genomics that could be used or co-opted by medical schools to educate their students, I think that would be highly desirable. It's like making an investment for a long time. So I just want to amplify what Heidi and Debbie said. And it's not just altruism. The presence of trainees and having a critical mass of trainees really vitalizes programs and I think propels success and tackling new things. I agree that as far as bang for the buck in propelling progress, it's probably a really, really good one. Mike. And I'd just like to respond to Rex's comment. Yes, this comment is always made, are we training too many people? In genomics we are not training too many people and particularly on the quantitative computational side, these guys have lots and lots of job options. Many of them don't do postdocs. I mean there is a huge demand for this area. It is increasing and it's not decreasing and if we look at what data is going to be coming up over the next period of time, who's going to be analyzing these data? Who's going to be interpreting and helping us go forward if we're not training enough and we're not, absolutely not in the quantitative sciences. I want to add my voice to that chorus and also raise a concern about the current move afoot on training grants. Which is to use them as mechanisms that are distributed across as many institutions as possible. So there's this sort of discussion about breaking up training grants into smaller and smaller slots and this is certainly what you've heard at NIGMS. I just think it's incredibly bad thinking. The places that are most successful in producing research should be the places that are doing the, it should be sort of proportional to the training, right? Or the places should come up with training programs that have the right kind of argument for it. And I don't think that view is going to produce the best out of trainees nationally. Mike? Here, here. Sorry, one other comment and strongly agreeing with Carlos. My wife has had training grants with four and six people and ours is not huge at Michigan in genome science. It's now 13, it's been 10, sorry, eight. With eight to 10 to 13 you can start to build a critical mass with four or six or eight, four or six. There's just no way you can do anything interdisciplinary and have any kind of critical mass of people. We can't get too small, we can't get too spread out. We need our programs to be big enough that these students are interacting not just with a strong faculty but a diverse set of fellow trainees across a range of disciplines. It's incredibly important. So what is the impediment for increasing the amount, the number of trainees? I mean in training grant slots. I mean in terms of the dollars it's not huge amounts of money in the grand scheme of things and I'm actually appalled by this largely because this is the first in my life I've been asked to take on a training grant and at the same time I've been asked to reduce the number of slots by half over a five-year window which I think is unfair because we produce and others do great trainees and there's a demand for them. So I don't understand why this is such an issue and why NHGRI if they've been good couldn't be better in this regard. Having taken on some of Eichler's trainees, I'd like to vouch for him. He does produce him very good. So I think we've had a really good discussion on a variety of areas. There actually are probably a few more that we could have but we're running out of time so are there sort of any last-minute burning issues that anyone would like to get on the table? Okay, great. Eric, I think you're up.