 So, welcome back. We're going to begin again. After those two very stimulating surveys of the landscape, now we turn to our colleagues in the federal government. Our colleagues in the federal government are well aware of the issues that we've been talking about and have been developing innovative approaches to rethink federal funding for research and training. So this morning we'll hear more about the perspectives and responses of the NIH and the NSF. As we did earlier, we'll hear from both speakers, and then we will open it up for questions. Our first speaker will be Patricia Lebowski. Dr. Lebowski earned her PhD in biology from Wesleyan University, did her postdoctoral training at Vanderbilt. She then joined the faculty at the University of Pennsylvania and then returned to Vanderbilt as a tenure professor. In 2012, she joined the Office of Strategic Coordination in the Office of the Director at the NIH. And in that role, she's part of a trans-NIH strategy to enhance training opportunities for early career scientists. Her remarks about the NIH will be followed by Richard Boone, who will speak about the NSF. Dr. Boone received his PhD in forest soils from the University of Massachusetts at Amherst. He later joined the faculty at the University of Alaska Fairbanks in the Department of Biology and Wildlife and the Institute of Arctic Biology. In 2011, he joined the National Science Foundation, where he's currently the Acting Deputy Division Director in the Division of Graduate Education. So thank you so much to both of you for being here today, and I will turn it over to Trish Lebowski. Okay, great. Can you hear me okay? Maybe? All right, great. So thanks for giving me the opportunity to come talk to you today. And what I'm going to tell you about are two experimental programs that are in an office that I work in. As Janet said, it's the Office of Strategic Coordination in the Office of the Director, and that's the home of the NIH Common Fund. And I'm sure a lot of you know about the NIH Common Fund, but I have to tell you just a tiny little bit about the Common Fund before I can then discuss these programs. So you may have all heard of the NIH roadmap about a little over 10 years ago, was set up by Zerhouni. And then when NIH was reauthorized in 2006, they established our division, whoops, going the wrong way, established our division there and set aside a portion of money for the NIH Common Fund. And the important thing for this audience is that this money was set aside for research that's very goal-oriented and produces some sort of deliverable at the end of it, okay? It's not R01 discovery sort of research. It's something that's more goal-oriented, and the other thing is that it has to be catalytic and really help the whole community, not just the people that get the money. That'll be important. So here, just some buzzwords that we use in our office about the programs that we set up. They have to be transformative. They really have to move the needle. They have to change things in the community, okay? Exceptionally high and broadly applicable impact, not just, again, for the folks that get the money, but for the whole community. And I think that's really important here with what we're talking about. They're catalytic. They're short-term, goal-driven. So they're not for a very long-term sort of project. There's something short, five years usually, sometimes ten years, depending on the science and sometimes in between. They have to be value-added to everyone at the NIH. So we're sort of disease agnostic. We won't fund something that's just, for example, diabetes or just cancer, but we want something that crosses the whole NIH mission and obviously workforce development. It's easy to see how that fits in. There's a high level of trans-NIH coordination. I'll show you a slide later with all the people that are working on these programs and care about them. And one other member of our trans-NIH working group, Jonathan Wiest, is here, too, from the NCI, which is rare that two feds could come to a meeting. And I don't know how we pulled it off, but don't tell. And there's supposed to be novel. These programs have to do something different. And again, move the needle. So this is just a snapshot of our current programs. And this pointer doesn't work, but the blue dot over at about nine o'clock is one of the programs I'm going to talk about today, Strengthening the Biomedical Research Workforce. And you all know it as these best awards. I'm going to talk about that in a moment. I'm also going to touch just briefly on something over here at about three o'clock, addressing the diversity the NIH funded workforce. So those are two experimental programs, and I really want to stress experimental, okay? But then we do a lot of other things, many that you've heard of, you know, the human microbiome, epigenomics, knockout mouse phenotyping, things like that. So I should stress that we're not a training office by any means. That's not our major focus. But it's an important focus, okay? Now we've already heard all this already, so I'm going to go quickly. A lot of this was built on the advisory committee to the director, the ACD Biomedical Workforce Report, BMW Report, and that was chaired by Shirley Tillman and Sally Rocky. And one of the important things that came out of it was this bottom bullet here is really the message that we need to help develop sustainable, diverse workforce in the U.S. That will inform decisions about training the optimal number of people for the appropriate types of position. Yes. Uh-oh. Oh, a pointer. Cool. Thanks. There. Okay, great. So you've seen these numbers already this morning, but I'm just going to spend just a tiny amount of time emphasizing how fuzzy they are. The ones that are in green here, which are mostly about graduate education, are the ones that we have a fair amount of confidence about. And this is on the web, so you can go get this chart if you want to. Just go to the ACD. I forget exactly where it is, but it's on the web. The yellow ones, this is important, of all the graduate students that stay in the United States, 30% of them skip a postdoc right there. The verb skip, right, is exactly what Keith was talking about last hour. And 70% go on to do a postdoc, and that's moderately firm numbers. The yellow numbers are slightly squishy, okay? But the ones that are really bad are the ones in red, where we're just kind of guessing. Here's a range, 37,000 to 68,000, who said this morning, boy, if you did an experiment where you said those were the results, I think the reviewers would laugh you out of the room, right? Well, those are the results. However, the important thing here, 43% approximately, of the folks go into academic research or training, only 23% of those are in tenure track. The main area that for many years, we thought we were training our students in postdocs to do. And I just have to put in a non sequitur. As Janet mentioned, I did run a lab and I trained students in postdocs for 15 years. So I've sat where you're sitting. So 23% of all the trainees end up in that career track. So I think one of the important things, and I'm delighted that we haven't heard it yet this morning, I'm going to use the A word. We don't use the word alternative careers, okay? Let's make a pact, right? Let's not use that. I just said it as something not to say, teaching your kids swear words, right? Okay, so these are the numbers that they pulled together from that report. And these are the numbers that we base this program on. So the conclusions, launching this traditional independent academic research career is more difficult than it used to be, perhaps. I actually don't know if that's true, but it's a long training time, relatively low early career postdoc salaries make the career less attractive. And I think this bullet is important that the current training programs, while they may do a fabulous job teaching graduate students how to think scientifically and creatively, I hope, they might not prepare them for anything other than that one career. So one result from the biomedical, from the BMW report is this program that I'm going to talk to you about today. So it's this common fun program. We call it strengthening the biomedical research workforce program, and a component of that within it are these best words. So our point is we want to expand the versatility of the things that these students, not that we want to train them to do all these things. We don't want to take a student out of the lab and train them to be a tech transfer expert, but we want to provide them with the information and maybe a little tiny bit of onsite information or direct experience as to what they could do with their PhD, okay? We don't want to train a bunch of, I'm using tech transfer again as an example, we want to just give them the information they would need to maybe make that choice, what's right for them and then how to go about doing it if that's what they choose to do, okay? So I'm going to circle back around to our office. Remember I mentioned human microbiome, epigenomics, all these sort of things. So in order to have a program fit in the common fund, it has to do something different, okay? One of the things it might do is provide new tools, technology, data approaches. Really what we're trying to do with this program is to get that culture change to happen. I think it's kind of important that the NIH is saying that we do not think that the one career path is the only career path. I think that's important. If nothing else, this program doesn't do anything else. I think the fact that people can point to that is important. So the other thing is direct exposure. Maybe internships, maybe something shorter. I think the meaningful exposure is important. So these trainees know what it means to be a tech transfer expert or something like that, okay? And not just say, well maybe I'll do it if I fail and don't get the academic job. So that's the working knowledge of these opportunities and the information to be able to go do that. The other thing is that these programs, both the workforce one and the one I'll talk about later, diversity, are experiments. And with any good experiment, you have to figure out if it works. So there's a huge evaluation component within each program, so each site is evaluating their own program, and then we also are coming together and we have a cross-site evaluation across all the C17 sites to figure out what works, what doesn't, and for whom. And then we have to make these tested approaches widely available. So we have to share all the information, and this is very important. People keep asking me, what are they doing? Well, you have in the back of your program, it's not printed, but on the list of links that you have from the meeting. You have a list of what everybody's doing. Everybody wrote a one or two page summary. And the other thing that this program will do, it's going to enable institutions to put up infrastructure, at least in the beginning. But because these are catalytic short term approaches, five year grants non-renewable, it's up to the investigators to figure out how to sustain these, and up to the institutions to take ownership of these once it's proven what are the right things to do. So these are DP7 awards. What the heck is that, right? R01, U01, R25, you know those things. What's DP7? Well, you don't know it because I made it up, all right? So it's a director's award, and it's not a defined mechanism that you've heard of before. However, they're research awards, okay, and that's really important. They're not training grants. And this was, boy, was this a tough part for everybody as they were applying for these awards. We don't support trainees stipends. That's hard, because you want to think about getting the trainees out of the lab for a short amount of time who pays for that. They're five years non-renewable. They have to establish these new things that they're going to do, and then they have to evaluate them, and I can't stress that enough. They have to work with the NIH, with me, with our contractor, Madeline Wallace, from Winrose Vision, who's here somewhere. I don't know where you are. On a cross-site evaluation, they've got to share the results. And we've also been working together as a consortium really well to understand what people are doing and what works and what doesn't. And personally, I like to hear what doesn't work. And we're getting to that, where we're finally starting to know each other well enough that we're sharing what doesn't work. So who's we? Here's we. We made ten awards in 2013, and we followed it up with seven more awards. We didn't have enough money to make any more awards in 2014. Again, these are five-year awards. So they're scattered around the US. We have some in the middle here. These coast and west coast are represented. And these were peer-reviewed. Peer-reviewed, probably a lot of you sat on those study sections. So they were, as anything, as any other grant, they were peer-reviewed. And you all, the reviewers, got to decide really who got the award. They're called BEST. And that acronym stands for Broadening Experiences in Scientific Training. Not a name I came up with myself, but I think the key is broadening, okay? And we were not really interested in supporting a program that trained people only for tech transfer, for example, or maybe even only for industry. But the idea is that they get the information that it's very broad. There's no, it's not a vocational jobs program, okay? I think that's really important, too. So what are these guys doing? And spend a little bit of time on this. You have the abstracts in the back, but to kind of summarize, a lot of it is career development skills, understanding what they could do other than the academic tract, figuring out what are the right things for them to do. There's a lot of self-assessment and reflection and awareness and things like that. Using IDPs, things like networking, job search skills. Because if you don't just go into those postdocs that are relatively easy to find, you don't know what else to do. And your mentor doesn't have the background to help you with it. Professional development skills, communication, giving talks, writing, things that are gonna help the trainee regardless of what career path they choose. And I think that's really important, too. These are not remedial programs. These are not programs that should be set up for those trainees that don't want to do the academic route and don't know what else to do, okay? These are for everyone. Experiential learning, these are the internships, externships, job shadowing, whatever form they take. And mentorship, so the mentorship outside of probably that primary mentor. And a lot of places are setting up alumni, mentoring, peer mentoring, things like that. So one of the things that's important to the NIH, and I hope to all of you, is that not every university is the same. You walk around University of Michigan and go, wow, this place is huge. Not every place is that big. So what we have done is there's a whole bunch of different models that are happening in this best program. And these are all being tested, right? So some of them are using a small cohort model where there's only a small number of students or postdocs getting the treatment, the experimental treatment. Some of them are abroad exposure with some things required, but not everything required. I mentioned alumni mentoring as a model, as a primary way to help these trainees sort out what they wanna do. There's a number of different career paths that are being focused upon. I said that wrong. Anyway, academia is included, it's always included. It's 23% after all, you can't cut it out. Communication, government, things like what I do. I'd like to think this is an acceptable career outcome for my PhD school. I think it is. Law, business, industry, biotech, pharmaceuticals, etc. We're trying to assay and look at what the job market is doing. There's one group that's teaming up with a software company to do exactly that, to test the waters. What do, if you just look at job ads, what is being asked for? What are the immediate jobs that are out there? And we know that might not be the same thing in five years or ten years. But at least we have some data, what does it look like now? So there's multiple programs being set up. Some were for, not very many, for the grad students only. Actually, I think that's changed now. Maybe that one place is including postdocs. Some that have postdocs, scholars only, and others that have both. Most of them have both. There's many different kinds of institutions. Some are med school, some are not. Some have med schools and a school of arts and sciences that might be on the same campus or different campuses. And how does it work with communication and sharing resources and building these programs? There's one that has two institutions together. I didn't point it out on the slide, but Emery is teaming up with Georgia Tech. This is what we at the NIH thought would be a huge part of the program. And I think almost all of these have this to different degrees. Our partnerships outside of academics. So who else, I mean, the research mentor doesn't know how to do it. Well, we need partners out there to kind of show people how to do it. And they're partnering with industry, biotech, their own business schools, their own law schools sometimes, government. Each site has external and internal advisory boards. So people that are really on the outside looking in that can help. Because again, we don't want this to be all focused just from one point of view, that doesn't make sense. And then there's this consortia influence, which I think is really important. As I said, the group gets together once a year physically. They also talk once a month on the phone. People present, we talk about issues, etc. I'm coming to visit everybody, which is exhausting. I was traveling all last week, that's my wine. The other thing is that we have this external panel of experts that the NIH has pulled together to kind of watch and see what's going on. They don't have a horse in the race directly, right? So, and they're giving their opinions to these people, too, about what's important, what they don't see enough of, what they maybe see too much of and think that we've already done that. So there's a variety of different approaches and things that are going on. And I hope there's not too many variables for a good experiment. But I think it's a good experiment and I think we're gonna see some outcomes, I hope. So here are some outcomes. We started this evaluation with this giant logic model that wouldn't fit in this room, okay? And it took us a while to get down to this, I think, pretty straightforward matrix. And it's things that we already heard this morning. So one of the desired impacts, sort of like a logic model, is that the ideal would be to enhance the trainee's agency to make these career decisions, their confidence. The fact that when they're ready to go, they know what to do and they don't default into a postdoc. So we're measuring all these different concepts and we're doing this in various different ways, okay? We also, and I think this is the other thing we already heard, we'd like to reduce the time to their first real job, okay? If you don't need the postdoc, don't do it. We would, NIH would love to see the postdoc reserved for only those people that want to do intensive research. If you're gonna write for the New York Times, Science, whatever, you probably do not need to do a postdoc and that's really critical. And I still, a lot of my colleagues at the NIH still say, well, when I'm hiring a program analyst, I only look at people with postdoc experience, why? So we're really pushing that and trying to see if we can move that needle. So we're getting that in various different ways. And then finally, the sustainability issue, as I mentioned, this is a five-year program, no renewals. So who's gonna pay for all this afterwards? And that's one of the jobs of the PIs for these awards is to try to figure that out. I think hearing for data is so important to be able to really show what your training programs are doing, where people are going, your outcomes data. Put it on the web, there's gonna be a big push from that, from NIGMS, I would say, in the next year or so, so get ready. You're gonna have to have the data on the web. And that's probably easy for you to do for your individual T32 people that are a fraction of all the folks you train. But think really hard about that because I think that train's coming. What is I gonna say? So the data, yeah, showing the data is gonna be really important. But I would like to suggest, and maybe put it out there, that if training programs don't include some of these activities at some level, they're not gonna be as competitive as the other university down the street that does have it. So I put that out as a challenge that I think it's gonna be important. And I'm not talking about a best grant. I'm talking about, cuz that's only, some of them, three and a half more years and they're done. I'm talking about the activities, okay? So again, this large cross-site evaluation is being conducted in collaboration with us at the NIH. There's a little subgroup of our working group that cares about evaluation, as well as a contractor, which is Winrose Vision and Madeleine Wallace, who you can talk to. She's here somewhere. All right. Now, here's some short term, some long term, sorry, consortial goals. And I think that this is right in line with what we heard this morning. The hope is that these institutions are gonna value a commitment to developing high level of research skills as well as this career development exercise. It's really important we don't wanna take away from the research that's happening, and that's critical. I was at Denver last week, and Inga Wethes is here somewhere. She's the PI of that award. They had the coolest new course they were doing in critical thinking, taught by someone in the Department of Philosophy. It was great to really think, forcing scientists to articulate why was that your hypothesis and why does that result only support those conclusions. It was, I wanted to take it. We imagine that there'll be a lot of these career development offices established at many places as a way to centralize these activities. I think truth in recruiting is gonna be very important. I mentioned already data. Show your data on the web, as we heard this morning. If you don't care about your postdocs, why are you doing that? They're not just employees. The tested models are gonna be available. We're gonna roll out a consortial website this summer soon, and there's already some publications started. There's a paper in Science Translational Medicine. I think it's out this week. And then we have another one that's coming soon. So keep an eye out, look for these. You can go to our website. We'll have them on there if you wanna figure out what's going on, what people are doing. So this is the main goal, I think, is that the trainees are gonna have an increased amount of confidence to be able to do what they wanna do, and not default into a postdoc. There's also a couple, three things I wanted to point out that I think can help everybody here. One is what I already said, that NIH is recognizing that the traditional research intensive career is not the only way that trainees can contribute to the workforce. But there's two other little things here that are really important. So how many of you guys do T32s, right? Everybody, yeah. So it's really important to note that the review criteria in the T32 FOA have changed to include non-research intensive careers. You are asked in there to explain what you do in your program to do that training. And if you're reviewing those T32s, please keep that in mind, okay? Because there's often a block between review and program that we're not allowed to talk in those meetings. So I'm telling you now, make sure you notice that if you're reviewing. The other thing that's really important too is the Office of Management and Budget, OMB, has recognized the dual role of postdocs. So what does that mean? They have recognized that postdocs are not merely employees, but they're also trainees, and should be afforded the time that they can go do trainee sort of stuff, okay? So hopefully this gets away from the idea that the mentor owns the postdoc and you are absolutely allowed to chain them to the bench. You have to let them go to a career exploration seminar or something like that. Doesn't mean you have to let them do a ten-week internship. And I completely understand that. But they should be able to go to a grant writing seminar or something like that. So these are important. This is just a snapshot of this relatively large working group that's running this. And the thing that's important to take away here are all these little letters, they're everybody's names. Is that people from all across the NIH are watching this experiment, are very interested in how it's gonna come out, same as you, I'm sure. And there will be a lot of notice that these activities will probably become expected in training grant applications and things like that. I'm not telling you that every training grant has to do all of this. That'd be silly, right? But if your university has this sort of thing set up in this philosophy for the trainees, I think those training grants are gonna be reviewed more positively. All right, now I'm gonna switch gears really quickly. And I didn't bring my cell phone up here to take, oh boy, okay, really quickly. Another program in our office which is focused on diversity. A couple years ago, Donna Ginther, Wally Schaefer, Reynard Kington published that the R01 profile, so if you look at who's getting R01s, it does not mirror the makeup of the US population, right? African-Americans, black or African-Americans 15%, Asian, Hispanic, white, all right? Well, this doesn't mirror the US population at all. So the idea is to do something about that. There is another ACD working group that focused on this. There's huge reports you can read. But what I'm gonna talk about just here for a minute is this diversity program consortium, which is another experimental training consortium. Most of you have probably heard of it. Have you heard of BUILD? I heard a couple of haunts, yeah, I think people know of it as that. But it's more than just BUILD, and that's really important. It's three very integrated initiatives. BUILD, I'm gonna explain these in a minute. CEC and NRMN, this is an evaluation component, and this is a mentoring network. And I thought David Burgess who's leading this was supposed to be here, but I didn't see him in the program, so I don't know if he made it. However, the goal is to enhance diversity in the workforce through development implementation, assessment, and then dissemination. Very similar to what I was talking about earlier, but with a focus on diversity. These BUILD is Building Infrastructure Leading to Diversity. These are given to under-resourced institutions that are reaching out with partnerships to their research intensive neighbors providing. Well, they're doing a number of different things that I'm not gonna go through. The National Research Mentoring Network, I'm gonna give you a link in a minute so you can check that out. And this coordination and evaluation center, this evaluation is critical. This is, again, the experimental part of this whole program. So the vision is that this should provide innovation and flexibility. It's a lot of money, okay? And the idea is to release that burden and see what people can do with it if they have enough money. It's an experiment, it's gonna be evaluated. It's incredibly collaborative. So these are what we call U mechanisms which means the NIH is intimately involved and can help steer the consortium and can help steer awards. And the other thing, of course, is that this is gonna be transformative. The expectation is that it will be transformative. This is just a snapshot of BUILD awards across the US along with the mentoring network and the CEC and they're listed here. And here's the net, or the website. I urge you all to look at NRMN, stands for National Research Mentoring Network. So nrmnet.net, if you could just go on there, take a look around. They're looking for mentors, if you're interested in this. They're happy to have more mentors registered. And then the CEC is at UCLA. So I've gotta go fast. So the idea is to broaden the perspective of all the trainees in order to set these different research priorities. Recruitments and retention, of course, improve not only the quality of research training but mentoring. And it's really for all trainees, not just for people from underrepresented groups. But one of the goals, a lot of the research that these folks will be taking part in will be to address and eliminate health disparities. Not all of it, but much of it. And then finally, I already said this, to recruit and retain also clinical research participants from different ethnic backgrounds as well. So this is a slightly bigger, but equally, I think, diverse group of folks that are keeping in their eye on this. And again, it's run by a number of different people that I see directors, institute and center directors here. Hannah Valentine recently came to the NIH and this is one of the things she's leading up at the NIH. And again, this very large working group of people. And many of these folks are project scientists that are actually tightly collaborative with the various sites and are going there and working with them. So I think it should be exciting. All right, here's my last slide. This is important. You know this, it's obvious. Somebody said the mothership, which is kind of scary. But NIH obviously has a commitment to the most appropriate workforce to maintain research and health in the US. This best program, the key is really broadening experiences for the trainees and letting them figure out what they want to do early in their graduate careers, hopefully eliminate that default postdoc. One postdoc that on a site visit recently called it the snooze button. I think that's probably sadly true for a lot of people. The diversity program is trying to figure out what kind of training works and for whom within different environments. And both of these programs are experimental. So they're going to be evaluated very rigorously to figure out what works. They're all time limited and very goal oriented. And all the programs are going to share their data, all the data with the community and really show what's happening after these sort of interventions or these experiments. So I'm going to stop there and hand this over to Rich and come back for questions later. Well, thanks very much Trish. It's nice to be able to represent the National Science Foundation. And I think that what Trish's talk shows is that the federal government's responding to drivers of change. And we ourselves are drivers of change, obviously. So I'd like to cover three things today. We've heard a lot about the drivers, but I couldn't hold back. Dave's English he said, no data slides. Well, I couldn't, there are three that I want to show that haven't been shown this morning that have influenced my thinking about graduate education, certainly. So I have three data slides, Dave. And also mention some of the other drivers, but no slides for those. And then highlight what I've identified as five innovations in STEM graduate education at the National Science Foundation. And then lastly, a challenge to you all and the audience. We can change things of the federal government, but you represent enormous change potential here in the audience. Students, administrators, and faculty members. So here's some of the drivers of change that are informing the conversation about STEM graduate education at the National Science Foundation right now. We saw some data this morning. There have been many national reports in the last five years. The peer reviewed literature, some authors of those papers are in the audience right here that influence my thinking certainly about graduate education and the needs. Engagement with stakeholders outside the foundation. And then also engagement with other federal agencies. NIH and NSF have been talking a lot over the past five years about the needs in STEM graduate education. So the agencies are talking with each other right now, which I think is a major positive. So here are my three data slides, Dave. Which I think people haven't seen this before. Every three years or so, over 700,000 scientists and engineers in the country are given a survey. And some in the audience may be part of that survey group. And we're asked various questions. But one question is, what is your primary work activity? And here are the results last time that this is conducted by NORC through the National Science Center in Engineering Statistics at NSF. But the results are quite surprising, the primary work activity. So this includes all STEM PhDs, engineering and science, etc. And what's really remarkable to me is that the primary work activity that we are trained for as graduate students, which is, I forget this button right, which is right here, represents relatively little of what we do. It's not our primary activity, even for faculty members in the audience. And you know that, I think. We spend a lot of time teaching, advising, administration, service, etc. But we engage in a relatively small proportion of our time in basic research, even though that's how we were trained as graduate students. Total non-R&D, in terms of primary activities, represents the majority of how STEM PhDs spend their time. And yet we're not trained in that at all. So here's one of my data slides. To me, this is really an important one. We've heard about time to degree. It takes what, two years for an MBA, three years for a law degree, four years for a medical degree. And Europe, it takes about three to four years for a PhD. And I always hear, oh, this is not my institution. I'm sorry, let me go back, sorry. Michigan's in the left here. So here's Michigan up here. I don't know what your data are in terms of time to degree. 5.5. 5.5, so you're trying to bring this number down, which is great. But it's taken in this country, there are about 100 institutions per column, by the way, about 300 total institutions that grant a PhD in STEM in this country. Time to degree is between seven and nine years. And Paul Estefan talks about deferred earnings, obviously. Well, there's a lot of deferred earnings when it takes you such a long time to complete a PhD. And the numbers haven't changed a great deal, obviously, over the past five years or so. They're not going down much further, which I think is a major problem if you're a graduate student. And it's a problem for this country. The last data slide. The mismatch between the number of PhDs produced per year in STEM versus the availability of faculty positions. So the dashed line represents down here the number of available faculty slots in STEM per year. Balance is around, but it averages maybe 3,600 faculty openings a year in STEM. And this dashed line shows the production of PhDs over time per year. And it's approaching about a 10 to 1 ratio between production and availability of faculty positions. The solid lines are the cumulative numbers over here. But to me, that mismatch is really striking. I don't like the word alternative, but the bottom line is that becoming a faculty member is now the alternative career pathway. If you want to use that terminology, it's the alternative career pathway because for 9 out of 10, basically, PhD recipients, they are not likely to get a tenure-track position based upon these data. So some take-home points in the data. This, to me, is the most stunning. Most employ doctoral, science and engineers work outside academia. I didn't show that data slide. And most do not engage in R&D as their primary work activity. Time to degree, 7 to 9 years right now, far too long. And the ratio of available faculty slots to PhDs produced is about 1 to 10. These are some of the recommendations that we've seen. And I'm happy to know that NIH, and I think NSF, is responding to some of these recommendations. So you've seen many of these things before. Expanding career pathways or at least awareness of alternative careers for graduate students. Internships, greater attention to mentoring, not just supervising, but mentoring. Greater attention to career counseling for a variety of career pathways. And these are some of the recommendations that we've seen before. Also, how to increase diversity. Linkage between undergraduate research programs and graduate programs. Or between masters and PhD programs, bridge programs. We know they work very well. So these drivers are informing changes in STEM graduate education at the National Science Foundation. Of course, the agency itself is a positive agent for change. So five innovations at NSF, I'm happy to report. And I've seen these during my time there. I've been at NSF now for about three and a half years. So I have five innovations. And the first is our new traineeship program, which has succeeded the IGERT program. Which people know about. IGERT Spinner was around from 1998 until 2014. The last awards were made in 2014. The program has now been sunsetted. It's always a pleasant word. NSF sunsets programs. So NSF has sunsetted IGERT. And IGERT has been replaced now with the new NSF research traineeship program that was launched in 2014. We announced the first eight awards in the NRT program just early in April, a month ago or so. And I want to go over some of those key features. And hopefully these are being responsive to some of the needs that have been identified already this morning. It focuses like IGERT on interdisciplinary training. Even though research is becoming more and more interdisciplinary, graduate training for the most part is not interdisciplinary. So NRT requires that all projects have to have an interdisciplinary research foundation to them. The development of new models for graduate education within a traineeship context. Preparation for multiple career pathways. Importantly, the extension of the benefits of the program to people that are not identified as trainees. We want many more students to benefit from NRT than was the case in the IGERT program, for example. We've heard a lot about evaluation. It's essential for IGERT. We want the evaluation to be formative as well as summative. We've required that the evaluator in every project be a member of the core leadership team. Underscore in the importance of evaluation for the program. We want these experiments, and they are experiments to some degree, not to stop at the campus borders, but to be shared nationally. We support both masters and PhD students. About 75% of the graduate students in this country are master students. But we don't give them enough attention. They are essential to the STEM enterprise of this country. And then lastly, changing priority themes, research themes. The first priority theme was data-enabled science and engineering. That's the theme this year again. The theme may well be different next year. In terms of award sides, $3 million awards over a five-year period. We made eight awards in FY14. We expect to make about 10 awards in FY15. And yes, the due date is looming, but there will be another due date in early 2016. This is an important question, and it varies by agency, interestingly. We talk about common definitions. There is not a common addition of traineeship across all federal agencies. So here's the definition of a traineeship for the NRT program. It focuses on students, the development of their technical competency, but there's also a focus on their broader development as STEM professionals, including the possibility that they will have a research-related career will not end up in academia, but will still be part of the national STEM enterprise. So it's a greater focus on their overall development, a focus on mentoring and counseling, vocational counseling, but also ensuring that they are being prepared for a research career, and then lastly, a major focus on professional skills that are often referred to as transferable professional skills. This varies across agency as well. What is a trainee? NRT trainees as students who are simply accepted into the NRT program. This can include students that are on a stipend. It can include some students that are off stipend that are receiving support from other sources, an RA ship, for example, a TA ship, a graduate research fellowship, any combination of funding. Most PhD students receive funding from three to four sources over their degree. So we've defined trainee broadly so that any student at a university who's accepted by the university into the traineeship program is a trainee so that as many students as possible can benefit from this program, not just the select few that are receiving a stipend. They have to be in either a master's or a PhD program that is research-based and will lead to a master's thesis or a PhD dissertation. So we did announce the first eight awards last month. I don't have the map, unfortunately, but a press release went out in early April. We were very happy with our first eight awardees and we held our PI orientation meeting just a week ago in D.C. And some of the more innovative approaches, again, they're all based in a research foundation that stresses interdisciplinary research and training. But some of the things that we've seen, an adoption of an approach done by some medical schools apparently, I'm not a medical person, but I understand that modular courses are more common. So modular courses that can be customized to the students' needs. Students at one site are going to be making available online teaching materials so that they will be able to disseminate basically the results of their work to the community broadly across the country and beyond potentially. Citizen science projects, interdisciplinary team projects among the students. A heavy emphasis on science communication training, we emphasize that a great deal in our solicitation. And in some cases, one of the NRT-PI's pairing up with the School of Journalism and also with the theater department. And then lastly, co-advisors, including a co-advisor from outside of academia who has a position from a non-academic sector, a non-research sector, that can provide them with informed information about having a career outside the academy or outside research. In 2015, we released another solicitation that broadened the scope of the NRT program to include something that we're really excited about, the innovations in graduate education track. Smaller award size. This is meant to be much more experimental, really pushing the envelope in graduate education. Smaller scale projects that might focus on a single discipline, multiple disciplines, a thematic area, were applied to any aspect of graduate education, such as mentoring, counseling. So we're really pushing the envelope with the IGE track. We want test bed projects. Some of them will definitely fail. And for us as a program, that's success, because we know that failure is essential for some of these. We're hoping that some will succeed, obviously. Award size is a bit smaller, $3,000 to $500,000 over the three-year period. We expect in the first year to make between 14 and 20 awards. And in the president's proposed FY16 budget, the IGE track is included. So this is highly experimental, and we're hoping we see some wonderful ideas from the PIs. Okay, two programs that are innovative. Another innovation is an expansion of opportunities through a program that many people know well, the Graduate Research Fellowship Program. So this is a program that people know, I think, quite well. NSF now supports about 8,600 GRF fellows in the country. The study state will rise to $10,000, because now there are 2,000 awards given per year. Very handsome awards, $34,000 per year, plus the cost of educational allowance. So here's an innovation now in the GRF program. Graduate Research Opportunities worldwide. For any GRF fellow, you are eligible to apply for a $5,000 supplemental award, essentially, to conduct research abroad. And now we have 22 countries that are signed on, and the equivalent of NSF at these countries is involved, for all the countries, except for those in blue. And for the countries in blue, these represent opportunities that have been established through coordination with the U.S.A.I.D. office. We're quite excited about this. The experiences can expand between two months and a full year. Another opportunity for GRF fellows, an internship opportunity. Again, a $5,000 supplement to the GRF fellowship. An allowance to work with a federal agency. These are mission agencies, obviously. But here are current partners. The internships can range from 10 months up to a full year. Again, $5,000 in addition to this stipend that you're receiving. People may not know it, but the Directorate of Education and Human Resources, EHR, funds research on education in graduate education. We fund a lot of research on K-12, undergraduates in formal science, but we also fund research on graduate education. It's a wide open field right now. And we've made a few awards that are noteworthy, ones that the Council of Graduate Schools might mention. But another award two years ago to a nonprofit organization that does a lot of training in science communication. And Compass, the organization, just released a report a month ago on this topic, mapping the pathways to integrate science communication training into STEM graduate education. What are the best practices? What can the federal government do to catalyze inclusion of this as a standard element of graduate education? What can universities do? They did a very nice landscape analysis of all the different training activities around the country, how many students receive this type of training or not, and they've given a number of recommendations. And I'll highlight their five major recommendations. We want to expand access. Even though we know students want this, the majority of STEM graduate students don't get formal science communication training, other than the chance to give a talk potentially. Now they don't get much iteration, feedback. They don't know how to put together a video or a professional blog, for example. So we think access should be expanded. Secondly, a community of practice. There is tremendous variability in how this is being taught around the country. There aren't standards. As part of that, then we have to define what some of the competencies are in science communication, both talking to a technical audience, a lay audience, putting together an effective blog site, putting together an effective video. We don't really have a good definition of what the core competencies are or how to measure them. Evaluation. We've heard that a lot this morning. It's really key. How can we evaluate whether a certain approaches to science communication training are effective or not? Beyond a weekend or a semester. Are they effective a year later? We simply don't know. And then lastly, what are the incentives for a faculty member to teach a course like this or for a graduate student to take one? Engagement with stakeholders. We're much more outward looking than we were when I started at NSF three and a half years ago. And there are many stakeholders. Faculty members, administrators, graduate deans, council graduate schools, for example. But there's one stakeholder group that we don't think about enough. And some of those stakeholders are in this audience right now. And that is the graduate students. And I'm really proud of this. This was an effort led by a triple A's fellow Kate Stoll who was at NSF from 2013 to 14 and led what was called the Innovation and Graduate Education Challenge. A simple idea. Ask graduate students nationally what are the problems and what are some innovative solutions. Some of you may have participated. We received 526 entries from students around the country. 48 of the 50 states. We had some great ideas. And all these ideas I should mention are available on the website for the Division of Graduate Education at NSF. Are there some new ideas for graduate education? Some innovations? The grad students have some fantastic ideas. So they were asked to identify what are the critical issues and what are their ideas for solutions for innovations in graduate education? We've done a careful text analysis of the 526 entries. And this is what has risen to the top is what graduate students nationally are asking for. They're not getting now or not getting enough of it. And you can see many things that we've heard about this morning. A major interest in basically improving science communication, prep for alternative careers, etc. There's a lot of focus on professional skills development, mentoring and counseling and also interdisciplinarity. These are the things that graduates are asking for. They're not concerned, interestingly to me, about funding so much. They're concerned about their futures because they've seen the data. They know that the chances of getting a tenure-track faculty position are pretty low. They want to prepare themselves for the type of professional future they're more likely to have. So students are an incredibly valuable source. We don't ask students enough for their feedback or their ideas. We should do it more. And then the last innovation, and I'll end with a challenge to the group here, strategic planning. I think this is the first time that NSF is doing this. NSF is now developing what I believe is its first strategic plan for STEM graduate education. It's a major step. Effort's been underway now for about three months. I don't know the forecast time for completion, but I would guess it's about three months from now. It's a major thing for the National Science Foundation what are the goals of the National Science Foundation in supporting STEM graduate education? I think every agency that supports grad students that have something like this. We're asking lots of questions. This is our funding blend currently in terms of how much money supports research assistantships, how much money supports traineeships, and how much money supports the Graduate Research Fellowship Program. And of course, the elephant in the room is the 80% research assistantships. And currently, NSF does not say anything about the expected educational or training outcomes of students who are on an RA ship. I should say we also don't say anything currently about students the outcomes expected for students who are on a fellowship. Should we or should we not? Should this blend be different? Should more of the pie go to traineeships, for example? I am a program officer with the NRT program. Of course, I would say yes because I see the benefits. We don't say anything about what the federal government expects the outcomes to be for those students that are receiving an RA. Is that a problem or not? And you can also see the amount of money going to traineeships because I care a lot about NRT. And NSF's budget per year is a bit over $7 billion. It invests about $800 million a year on graduate students, one way or the other, but still a relatively small proportion of the total allocation for funding for graduate students who's going to traineeships. Okay, so here's the message for you all in the audience. Trish and I have talked about how federal government can incentivize change and we are powerful change agents obviously with funding, policy programs, etc. But ultimately, the source of change in graduate education is right here in the room, really. The students, the administrators, and the faculty members. The university is the home for STEM graduate education. It's really where change occurs. So I have eight points eight for faculty and administrators and eight for students. So here's my challenge to you all and this might set the stage nicely for the breakout sessions afterwards. I won't read them all. But here eight challenge, eight points, I recommend recommendations I have for all the faculty members and the administrators in the room who care about changing STEM graduate education. We've heard a lot about data. We don't have enough. We need to report information on how long it does take time to get a degree. What happens for our students afterwards? Our students satisfied. A paper came out not too long ago that shows that students joy in doing science goes down during their graduate programs. That's a problem. That was done, has that been done at your university, for example? And why is joy going down? It should. It's why most of us went into science because it was exciting. In terms of faculty administration, seek input from the people that employ your graduates. And I think a valuable population are those that have been out, the students that have been out for about three years or so. They're fresh enough, they have a perspective, but they also have been in the workforce for a while. This is essential, especially given that they're going to have non-academic careers, many of them will, certainly. And you can easily argue that these skills are valuable to a professor as well, of course. I think we all have to have a view of what's good for the country in terms of STEM grad education. We can't think about just what's good for the University of Michigan or Boise State University or my home institution, University of Alaska Fairbanks. We have a national responsibility here. Lastly, face any actions on evidence and outcomes. Why are you doing what you're doing? And then lastly, a challenge for students that are here. And I might include postdocs in this group as well. Take ownership of your education. And some of the students in this room and postdocs have already done so. Now I saw a person in the room who was on the Boston Globe Front page a few months ago taking ownership for the outcomes of postdoctoral education. You are amazingly powerful. You are a change agent. So take initiative is my point. Engage with faculty administrators. They will listen to you. They're worried about the state of graduate education as well. Consider internships. Consider having people on your committee or a co-advisor who was outside of academia outside the research arena but still in a research-related profession. So you have an idea what it takes to work for, say, the Bureau of Land Management, USGS, NIH or the National Science Foundation. And then lastly, once you graduate, yes, do serve as ambassadors and resources for your program. And I think that's the last slide. I'll be happy to take any questions. And I think Trish is going to come up as well. Dave, we're a little over time. We have two hours for lunch. Oh, two hours for lunch. Great. Start over here. Why don't we take back 10 minutes to the questions. I was very happy to see the word master here because I haven't seen it anywhere in the program and I'm very happy that you now start master student funding in the NSF. In my field, which is interdisciplinary bioinformatics, a lot of the students, about 40%, come in with both a bachelor and a master. And oftentimes, like biology bachelor with a master in biostatistics or something like that, that is actually the rule. I wonder whether this is something we should discuss of whether we shouldn't make more of our students first go through a master before committing to a five to eight year PhD. As someone from Germany, I also would say that your comparison isn't quite fair with Europe because the three and a half year PhD in Europe usually has, as a starting point, something much closer to a master than a bachelor. So I've always said our five and a half years are equivalent to the three and a half years in Europe because the two years that it usually takes to a master diploma or whatever it's called there. But I wonder whether we should have a discussion about some of these alternative or non-academic tracks being served quite well with a master rather than a full blown five to eight year PhD. Thank you. I don't think there was a really a question there, but I'll just say that at least with the programs we were developing, it was sort of outside the scope of what we were supposed to look at. However, I can say that many of the universities that we're working with do have professional science masters and things like that. And I completely agree if you don't need a PhD to do the work, you probably shouldn't go to grad school and if you can get where you want to go with a master degree, that's great. And I think a lot of us look at the fact that we're urging people to expose their students to these possibilities early in graduate school. It would be great if they got this training back when they were in undergrad before they made the commitment of which grad schools to go to or whether they need a master's or a PhD. I think it's an important point. Hi, my name is Bob Fuller. I'm here at University of Michigan and the director of our interdepartmental cell and molecular biology program. So I have a couple of comments which are not so much questions but meant to maybe provoke a little bit of discussion. So many of the things that you talked about, Dr. Lobonsky, that are being funded by the best programs can be done in institutions with their institutional resources. And here, Rackham Graduate School has an office of career development and so forth. And I wanted to point out that in reference to what Dr. Boone said, that it's actually very important to engage the students in all of these things. I think it's actually the most important thing. It's something I've learned. So I'd actually direct those of you who are going to the workshops to take notice of a workshop that will be led by Aaron Goldstrom which will talk about a set of career workshops that were developed in effect from bottom up rather than top down by the students themselves. Students are not children. Students are adults. They have very clear ideas of what their career aspirations and interests are. And given the resources and it doesn't really necessarily take a lot of resources. They can explore these in ways that are really quite surprising and really change the culture of an entire graduate program. So that's one thing. The second is internships may be a really good idea. And within NIH training programs, I think that perhaps the only one that really requires internships are the biotechnology training programs in general. But they could be a very useful aspect of all of the T32s. And not every student would necessarily benefit or want to do an internship. But the problem is absence of funding. Often when these internships are within industry, companies are not willing to put the money forward to entertain a student for three months. If NIH were able to, within the context of the existing T32s for students to supply funds for a fraction of the trainees for three-month internships, it would be a tremendous innovation. I think it would help a lot because I think there are a lot of students who are interested in doing those sorts of things and we simply don't have the resources to provide that because that gets a little expensive. It's going to cost you $10,000 for a summer, for example, per student. So... I agree with you. I think something that I've seen among the best consortium that I've seen that really is working well is that it's really on the student or postdoc to kind of figure out how they can shape that internship to bring something back to the lab that benefits the lab. And if the internships can be worked that way, number one, you're going to have faculty buy in that's amazing because they're going to want to send their trainees out there to bring this back. And then the other thing is then you don't have to worry about a funding source. They're being paid. They're still a student and they're still working toward the completion of their work, right? Where it gets tricky is when they have to take a leave of absence, especially if they're international. So that's a problem. But I've also seen a lot of companies are happy to put a little money toward this, lots of them. So I wouldn't... Yeah, I wouldn't dismiss that out of hand. I might mention too, at NSF we've seen internships leading to research collaborations between the faculty member and the internship sponsor. Okay, and just Erin Goldstrom's presentation will be in workshop two. The back over here. Yeah. Hi, I'm Chris Pickett with the American Society for Biochemistry and Molecular Biology. I have a question about the diversity topics that you brought up, the BUILD program, the NRMN program. It's my understanding that those programs tend to engage people before they get to the grad student postdoc. Yeah, most of the BUILDs, the focus, well, not entirely. The BUILD program is focused at the undergraduate level. Yes. But the networking is supposed to serve all career stages from undergrad, grad, postdoc, junior, and even senior faculty. So that's supposed to cast a much wider net. But that kind of gets back to what we were saying about if you can get the students earlier, it's even better. Sure. I guess my question is about the biases that are inherent in the system. You know, the biases in hiring practices, the biases in peer review against women and underrepresented minorities. I'm wondering if you could comment on what the NSF and NIH are doing about those issues in the system, you know, aside from the BUILD and NRMN programs. Oh, additional programs. Well, I mean, I don't understand your question. Can you talk a little slower? You know, there are biases inherent in the program. They get the report that came out that showed that there is a bias against African-Americans getting R01 grants. Right. You know, these kinds of biases are fairly well known in the community and they may act to prevent underrepresented minorities and women from progressing through the career. They may be graduate students, but they see these biases already inherent in the system and they choose not to continue. So I'm wondering what the NIH and NSF are doing to address these biases to the extent that they can. Oh, directly, directly address the biases. Well, there's a lot of work going on to address the bias in peer review, which is one aspect of what you're talking about, implicit bias. We're unfortunately, we're kind of constrained in that we have to focus our programs that we're lucky we could stretch to the undergraduate and beyond. I would argue that a lot of this needs to happen earlier. And unfortunately, we're not allowed to tackle things earlier. I don't know if you want to say something about that. Nothing to add? Yeah. Yeah, thank you. Victor Dorita, University of Michigan. Richard, you showed a very interesting graph. I know David asked you not to show data, but the data was really interesting where you showed production of PhDs and then the faculty positions. What would that graph look like? Do you know if you put not just faculty positions, but all other careers for which PhDs would be needed? I mean, how much would we get closer to that supply line? Well, this issue came up this morning a bit, too, you know, the unemployment rate for some PhDs, you know, has always been around three to four percent, basically, so people are being employed many of them are employed in a job that we would regard as being within the greater STEM enterprise, I'll say that. So what you're saying is if you put all the PhDs on, if you put those other jobs out there, you would be at 98 percent of that supply line? Well, I should be 98, because some PhDs in STEM leave STEM. We know that, but it's a small percentage actually that actually do. The majority stay in STEM. They may not be in academia or doing research, but they are still, they still have a STEM position of some kind. Hi. I'm Chris Berger. I'm director of graduate education at the University of Vermont College of Medicine, and I wanted to just get clarification on the new guidelines for the T32s, because on one hand, we're being encouraged to provide better training for people to make career choices, which very often students, even in the middle of their graduate career are still not clear about. But one of the issues always have been with the review of the T32s while it may be encouraged now to have these programs to expose the students to this. The outcomes are really supposed to be about people getting into research careers. Really? That's not what it says. What? That's what it, that's... Yeah, so that's where I'm going to challenge you. So that, that is the way it used to read. And that was my point is that the words have been changed now. But has it changed on the review panels? Well, that, you are absolutely correct. And who are the reviewers? Right? Yeah, but I mean this, this, this... And I think that's critical. So it's up to you all to write your grant that way, right? As it says in the review criteria, we're expected to do X, Y, and Z. And look, here are our programs and look here are the data. So you write it that way. And when you sit on study section, you review them that way. So I'm sorry, but I have to throw that challenge back to you all because I can't do it. There is this wall in between program and review. I'm not allowed to talk to the reviewers. I'm not the, hey guys, we want this. I can't do that. So I have to push that back on you all. You're the reviewers and you've got to bring that up and show people that. And it's right there in writing. And let me tell you, to change that sentence, that was a lot of work. So please use it, okay? I know I'm being a little flip, but seriously, we can't do any more than what we can do is to just change the words that are there. Well, I mean, I bring this up because I think that has to be said that it really needs to change at the review panel. So it doesn't matter what those, I, you know, you can say that, but I know a lot of the review process is still focused on outcomes. Is it research or not? Yeah. And we're working to change that. I mean, this has only been there for what, less than a year? Jonathan, I can't remember exactly when it changed. So this is fairly new. But, you know, the SRO at the study section should note that. The chair of the study section should note it. And honestly, if any of you, the next time you're on one of the review sessions, please do your job as an ambassador to change things. Thank you for bringing it up. It's really helpful. Over here. Hi, Rudy Jimenez from the University of Texas at Austin. Kind of want to go back to the comment that you made where people prefer someone who's had post-doc experience or a professor as an expert. Has that become a concern as maybe from the report before or the presentation before where if they have those tracks as a PhD where they have experience in business or law, do they see that as being a challenge? And I asked that because I'm one of the ones that went from PhD straight to my STEM coordinator position, not by choice, but because my post-doc was taken away from me when the sequester happened. And luckily I took ownership of my education and was able to sit on boards of a couple, some national organizations and was able to have the kind of skills that I needed for my current position. So do you think that with this new idea of training the PhD into having those different kind of career options, will the fields change their way of thinking that they can take a PhD that doesn't have post-doc experience or someone that's not a professor or tenure tracker? Right. I mean, I think that's the hope which I hate that word, right, because it's slow. You know, culture change is not fast. And I expect that to change, but it's going to be slow. And so I think that we all have to work to make a change. Keith? I served on the diversity workforce committee as well as the BMW committee. I know. And there was one piece of data that was really staggering to I think the whole committee, which is that the number of PhDs awarded to underrepresented minorities in this country over the last decade averages 504 per year. That is 10 per state per year across all of the sciences. The distressing thing about that piece of data, aside from the data itself, is that there was sort of a palpable relaxation in the committee that said, oh, this is not our problem. There's no pool. So this is really a problem that K through 12 people should be worried about and we're not doing that. So I would push back on that. I tried to without success, but now I can do it again. And that is to say that the scientific community needs to get into work our way down into the K through 12 system and begin to enliven the hope and excitement amongst across the whole demographic in this country, which is changing about the excitement of science and the federal agencies could be involved in helping to support those efforts and a lot of them are involved in it. So I think the default that this is not our problem because we don't do undergraduates or you know do K through 12 is not true. Well, I think I meant a default in that we're not allowed, the NIH isn't allowed to put any money into that, right? But that does not mean that our trainees can't go out and change things, you know? And it does not mean that we can't support an undergrad or a grad student or a postdoc that's gonna end up being a teacher in a K through 12 situation or maybe that just goes and does some work in a K 12 environment. Those sort of things are incredibly important. So I agree with you but we have to we have to change that. So more to come on that topic and many others. So first, please join me in thanking our panelists. For those of you who are just with us for this morning, you will want to know that the rest of the group is going to be working over the next two days to reach consensus on some critical steps that will move us forward and a white paper summarizing the results of these sessions will be posted on the conference website after the meeting. What's more, we are planning a follow-up convening in May of 2016 to keep this work going to explore the implementation of both short and long-term solutions. So mark your calendars for May of 16 to come back and join us in that conversation. Now, for those of you who are registered for the rest of the meeting, the next step is lunch. So go out the back of the auditorium through the lobby out the front doors and turn right. Lunch is available in the building next door, the building that has big arched windows and there are box lunches there waiting for you. We will reconvene back in this building at two o'clock up on the fourth floor and to reach the fourth floor there are elevators in both wings of the building and both of them will take you where you need to go. So thank you all very much for joining us this meeting. Again, thank you so much to all four of our speakers who got us off to such a great start. Thanks.