 Welcome everybody. Welcome to the Future Trends Forum. I'm Brian Alexander. I'm your host, chief cadherter and founder of the forum and we have a great guest today on a great topic and I'm really looking forward to our conversation. What we're going to be talking about today is what seems like a very wonky issue, a very technical issue, but it's one that really has crucial importance for all dimensions of higher education. We're talking about metrics, the numbers that we use in order to determine all kinds of things. What classes we should take, what colleges we should go to, how to fund higher education and what we have today with us is a couple of authors, at least one of them, who have published a new book on metrics that matter and they take a critical eye on everything from return on investment to US News and World Reports and they ask us to rethink the metrics we use. If we cite the old American adage that what gets measured gets managed then we need to change what gets measured so we can change how we organize higher education. Now to join us I'm delighted to bring from Yale University Professor Zachary Blemmer. Zach is an assistant professor who works in the School of Business and Management and he is one of the authors of this fascinating new book. Professor Blemmer, welcome. Hey thanks so much Brian, thanks to all of you for being here. Oh it's a pleasure, it's a pleasure. How are you doing today and where are you today? I'm doing great. I am in my apartment in New Haven, Connecticut and I'm really excited to be talking about this work with you all. Excellent, excellent. Well a key question and I'll ask before somebody else does is do you have any snow? I think as a couple of other people mentioned in the chat it is unseasonably warm here in New Haven. It's about 70 degrees so if anything I should probably go open a window. Wow, wow, wow, that's amazing. Well if you open a window just make sure there aren't any loud arrests going on outside. Yeah exactly that's uh yeah I've only lived in New Haven six months and I'm learning the town very quickly that's a big part of it. I bet you are, I bet you are. Well welcome to to that position. Zach we have a tradition on the forum we're asking people to introduce themselves by talking about the future not about their past and I'm really curious what are you going to be working on for the next year? What projects, what courses, what writing, what topics are going to be top of mind for you? Sure so you know you all I suppose got somewhat unlucky of the five authors who put together this book that tricks that matter. I'm the sort of technical economist and so you're going to have to deal with me for the next 50 minutes. Oh no! Yeah exactly so you know that's the it's the luck of the draw so so so I'm an economist I primarily think about the mobility and equity ramifications of higher education policy so I'm sort of working in three different areas going forward for the next year. The first is working on a couple of projects related to undergraduate admissions policies, race-based affirmative action, and it's race-neutral alternatives, trying to understand how various policies you know including recent policies like test optional and test free admissions change the composition of students and have long-run implications for targeted students. The second area I'm thinking about is in policies that restrict access to certain college majors even after students have enrolled, thinking about GPA and other sort of essay or selective restriction policies on departments like engineering and other generally more lucrative fields of study, and I'm thinking about the long-run ramifications for students of being excluded from their most preferred fields of study. And then the third is looking at a longitudinal project building a number of databases of early 20th century higher education in the United States. Understanding how higher education has evolved over time and then how changes in access across institutions and access across college majors have changed kids basically the mobility ramifications of America's higher education sector for lower income students. So you mean socioeconomic mobility or geographical mobility? So the degree to which lower income kids go to college, what colleges they go to, what they study in college, and how access to these various programs has shifted over time. Excellent, excellent. That all sounds terrific and the first one may have great timing depending on the Supreme Court's ruling this summer and fall. Yes, I started working on affirmative action projects about five years ago not knowing the political ramifications of the work. Sometimes these things line up and maybe we'll circle back and bring you back to talk about that depending on what happens. Sure, sure. I should emphasize the sort of the structure of the book that we've put together is really intended to translate a lot of work that I and my colleagues have done in economics and other quantitative fields, trying to study the ramifications of different educational choices and now trying to translate that knowledge to kids and parents about whether and where to go to college and what to study when they're in school. Well that's actually a good way to start and friends if you're new to the forum by the way. What usually happens is I ask our poor guests a couple of fumbling questions and they cut loose and show off their brilliance and then I turn over to Mike to all of you. So one question I did want to ask just to start off with this. This isn't an usual book. It's a five authors I think, but you're all co-authors. You don't have individual chapters or anything. How did that come out? Yeah, so unfortunately my compatriot Chris Newfield couldn't join us today. I hope that he would be here to sort of share the stage with me, but Chris was the initial motivator for this project. He organized a group through the University of California Humanities Research Institute in UC Irvine and basically there was like a selective admissions procedure by which people were admitted into a residential research fellowship in the spring of 2018 which gives you a sense of how long it takes to write books. The group of us got together and then talked over how to best think through this issue of the metrics that are used by universities and students to guide resources and then either what metrics could be alternatively used on the university side or alternatively used on the student side to make better choices. And so the five of us we span the academic disciplines. Chris Newfield works in the English department and in critical university studies. There are a number of qualitative social scientists and then two quantitative social scientists. Ashish Mehta is also an economist and myself. And so the five of us I think came from a number of different disciplinary perspectives and did our best to in a sense translate through multiple stages. Ashish and I would go out and find a number of studies and then sort of talk them over with the group. The group would also sort of from a start bring together as much work as we could talk it through in these sort of like endless discussions a fun part of residential research fellowships as we got to spend three months really very much face to face talking every day about this work. And we ended up with this design where while we each had individual metrics that we were sort of the lead on and trying to explain what the metric was how it was currently being used what alternatives were available and to make some recommendations ultimately for parents and kids about alternative metrics or non-metrics that would help them make better choices. We each took leads but then we did a lot of rewriting of each other's chapters. It was very much a team effort for which reason we decided not to sort of have our names on each individual chapter. Fundamentally the five of us all wrote all of the chapters in the book. That's a fascinating synthesis and what a great low-tech front-loaded process of just being face to face. It was tremendously fun and hopefully we produced something that can be a value to others. We certainly enjoyed producing it. Well it's definitely a terrific book. The editor our wonderful friend and great great publisher Great Britain just threw a link to it in the chat so please please definitely grab a copy of this book. It's incredibly lucid. It feels like one synthetic voice by the way. You guys really succeeded in that. Well let me ask you just a couple of starting questions about this. In fact here let me play with the video a little bit so that we can all be face to face. One of the challenges you put to us is that a lot of the metrics that we currently use are are badly badly flawed and you look at return on investment numbers, you look at university rankings, institutional selectivity, the actual price versus the actual payment, what happens with student debt and then especially wages connected to college majors and you break these down and you have great criticisms where you say that the the data underlying these is often flawed and the analyses are as bad as preposterous and most of your book is tearing these down which is brilliant and then building up what you think are alternatives. I'm just curious if you could break down what are what are some of the problems with these well-known and widely used stats? Sure yeah so and there's a lot of examples in the book. Maybe I'll just choose one of sort of popularly known university rankings as sort of I think an example of the kind of thing we're trying to do in this book. You know rankings are designed to take in all sorts of different information about universities and then produce a sausage that is helpful to students but I think is often really unclear in these rankings is exactly who the consumer is supposed to be of rankings. After all a lot of different people have very different interests in what to get out of a university both on the student side but then also as you think about graduate students and professors and university administrators and all of the various parties who have an interest in comparing and contrasting different schools. Our focus was on students. We were just trying to think through how should students decide if they have a choice between two different universities where they want to spend their late teens and early 20s. Where the chapter begins is just trying to look at research that has examined what happens when two students who are observably similar choose two universities one of which is like slightly higher ranked than the other and there have been now a number of research studies trying to study outcomes for these students that essentially say no difference in outcomes between these students in terms of educational attainment at the undergraduate graduate level or wages earned in the labor market as well as a number of other potential non economic outcomes for these students. So the place to start is to notice that despite the fact that rankings really highlight small differences between schools the 15th ranked school seems very different from the 20th ranked school. They're really hiding tremendous similarity in student outcomes across these institutions. They said there are differences in student outcomes across institutions so the book highlights a really I think interesting study by Josh Goodman and Sarah Cahodes looking at a scholarship program called the Adams Scholarship in Massachusetts that in tendency pulled students by virtue of financial aid into relatively less selective universities. Attended to pull kids into into lower less selective public universities and away from more selective and for what it's worth higher ranked private universities and you did see pretty substantial educational differences between these students getting pulled into a relatively lower graduation rate institution led these students themselves to become less likely to earn college degrees within four or six years. And so what the chapter develops is this idea that if all your goal is to like figure out what's a good index for longer run educational and labor market outcomes on the basis of going from one university to another. It turns out you don't really need this sausage of rankings this tremendous amount of information pulled in from various sources and for various purposes nor do you need tremendous other information about schools. It turns out just that the university's graduation rate does a really good job of indexing what happens to students themselves when they get pulled between universities and as a result is a much more helpful metric for students to focus on as they're trying to decide between two different universities just focusing on what is the four or six year graduation rate of the two schools. Now in a lot of cases the graduation rate differences between schools that are very similarly ranked are basically identical. So if you look at top private universities these all have graduation rates over 90% extremely high rates. There is no fundamental difference in outcomes between these institutions and the graduation rate reflects that. But if instead you look at differences between say in the California context between local comprehensive public universities like the CSU system and more selective universities like the University of California campuses at San Diego or Davis or Irvine it really does look like student outcomes differ very substantially and causally at these different schools and that's reflected in graduation rates where a school like UC Irvine has a 50% graduation rate and the average of the CSU campuses is about a 50% graduation rate. So these differences are substantial and whenever those differences exist it's worth students paying attention to them and we don't want to obfuscate the fact that there are differences in student outcomes across schools but there's no reason to highlight small differences like it's done by university rankings. That's a huge, well I mean you know the US News report rankings are so so powerful I mean some academics disdain them but we live and die by them and yet what you've isolated is a far simpler far clearer much more accessible measurement that you could just pull out easily for any institution. And I want to emphasize there's now a long series of I think very high quality studies in economics looking at quasi experiments cases where there's effective randomization of whether a student goes to one or another school and these experiments have now repeatedly demonstrated that graduation rates seem to do a really good job of indexing both educational and labor market gains to higher graduation rate institutions. Oh this is fascinating I mean this is fascinating and it's kind of an almost a David and Goliath comparison to think about the huge apparatus that goes into US National World Report and you're just pulling out one thread and that's the one that shines the brightest. Let me ask you for one more of these numbers which is about tuition sticker price and we've talked about this trend on the forum before for quite a long time that the pricing for higher education is very non-transparent that we have a published sticker price and that's what gets all the media attention that's what everybody focuses on and yet tuition discounts are so steep now up to 50 plus and many institutions depending on institution type that that what students actually pay is often lower. How should we handle this? How should we be thinking about that price metric? Sure so again the structure of all these chapters is very similar we begin by just describing the metric that is presently very widely available so if you just you know google right now I don't know Columbia University tuition a number will come up I think it's getting close to $80,000 a year between room board and tuition and that number scares off a lot of families who either believe that various universities are more expensive than others on the basis of their sticker price or choose potentially not to apply to universities at all as a result of the potentially high cost of enrolling at those schools. So we begin by just defining what this metric is and then as you said you know we point out that the large majority of students enrolling at these universities pay substantially less than the posted tuition price. We have a growing body of evidence that sticker prices scare a lot of kids off especially from high cost institutions but just the fact that there exist high cost institutions seem to scare kids off even from lower cost schools but nevertheless you know this is the number that's widely apparent and so we make a number of recommendations in the chapter of what parents and students can try to look for to get a better sense of what college would actually cost to them and one recent example I think has been the my intuition net price calculator so lots of universities have that price calculators on their universities I think they can be very difficult to manipulate and use in a lot of cases students don't have a lot of their parents financial information that's required to complete these calculators simpler calculators like my intuition which is a project by Phil Levine an economist at University and is now available at over 70 college and universities in the US it's much simpler version of this I think there's only six questions that students have to answer before being added a range of tuitions that that they could be expected to pay at these different institutions this is a much more helpful piece of information to students there are other examples tuition fit is one and I think other universities use other tools but the idea is just to try to find higher quality information that is more decision relevant as students are making choices about where and whether to go well thank you that's that's a clear and and and exciting answer and by the way friends you may have seen a kind of black flicker here then this is one of our cats this is Hunter the he has decided to join us in order to you know this is this is the way things go the cats want to show off let me let me follow Hunter and step back a little bit and ask for all of your questions and we already have some and if you haven't read the book that's a quite all right I mean you have your assignment ahead of you but but please respond to what Zach has been talking about that also asked your questions about the overall topic of metrics the first one I want to bring up is Lynn Sibulski I'll bring up a video question from her let's see if she's oh there you are hello Lynn hello hello can you hear me perfectly always a win um Zachary thank you for coming on today for context I'm a college financial aid nerd um that's my career and that's also my area of kind of advocacy and and trying to improve the industry right one thing that I am working on with my colleagues is the idea that the financial aid office doesn't always have the correct information um for example there are lots of resources that say hey if you want information on scholarships and outside scholarships contact your financial aid office within higher education institutions as I'm sure you're aware there are different methods of distributing scholarships especially the outside scholarships and in some cases a student is saying hey I'm doing the math I should get some money back and the school says oh hey now that you've won some outside scholarships I'm going to take away your merit scholarships and give it to someone else can you please speak to scholarships in general in terms of student outcomes and how invested schools and private companies should be in giving out scholarships yeah sure absolutely so uh you know tuition discounts and the the obfuscation around tuition discounts make it really difficult for kids to make informed decisions about whether to go to school and where to go to school but I think uh some some recent research by sue denarski and others has really shined a light on the best ways that schools can use scholarship programs to encourage either more kids to attend college uh or to uh to promote outcomes for those students so uh this recent study I think it came out last year in the american economic review and we discuss it at some length in the book focuses on the hail scholarship at the university of michigan which was sort of an interesting policy the hail scholarship actually didn't have any money associated with it uh the way it worked is that the university of michigan went out of its way to to identify students in the beginning of their senior years who they believed on the basis of uh fafsa information and other family income information would be eligible for a full scholarship if they applied to the university of michigan and were admitted and then we went through the full financial aid process and and so they they informed those students directly congratulations you've been given a hail scholarship that means if you get into the university of michigan we're going to guarantee that you are going to attend the university of michigan tuition free what they saw was a large increase in the number of these kids who just sent an application to the university of michigan in the first place a big number of them got in and many of them attended the university of michigan but again like I said it didn't net cost the university of michigan any additional funds they were they would have already guaranteed those funds to these students because they were already eligible for financial aid under the university of michigan's traditional financial aid policies what this did was provide information to students earlier where they were actually making important choices about whether and where to attend college or at least to apply to college and just providing this information a little bit earlier was extremely impactful on whether these kids enrolled at any college at all and in particular if they attended the university of michigan now you can see that i've kind of twisted your question here really what you were asking about was these different agencies inside of the institution and where i'm trying to push the question is thinking about how these agencies can better be used to serve student interests and i think the hail scholarship gave just one nice example of this while bureaucratic challenges make it i think really difficult either to move money around or to to make as transparent as possible to students what it would cost for them to enroll at universities these kinds of relatively simple policies that just provide more information when it's being used by students can be really meaningful for for long run outcomes of this thank you well sorry to partially answer the question but but at least you know as we're thinking about sort of a student oriented version of this i think that's that's one way in which these scholarships can be really impactful well then thank you for the great question and please keep being a financial aid nerd we need more of you and thank you so much zach for that for that terrific and self-aware answer friends if you're new to the forum this is an example of the text of a video question excuse me so you can just join us and you can tell we're very welcoming and i even host people who don't have beards so now let me let me bring up some of the text questions that have come up and this is one from our dear friend rocks and rizkin and she asks this when is the most sensitive time age or grade for high school students and parents to read your book and do you know if high schools have easy access to this critical information yeah yeah it's a great question so the book is very much targeted toward students in their junior and senior years and parents of students in their junior and senior years the first chapter of the book is about return on investment indicators which are the kinds of metrics that students who were just deciding whether to go to college in the first place might look at as they're weighing you know the relative returns in an economic sense of going to college versus entering the labor market directly just getting a job straight out of high school so the first chapter is very much targeted toward that kind of student and then the book sort of proceeds from that point so the last couple of chapters of the book are about college major choice we have a chapter about average wages by college major helping kids choose between majors that have higher or lower pay and then college mobility rankings and thinking about you know the kids access to college majors and the degree to which universities can provide mobility to relatively lower income students so I think there's I don't I think the earliest age that we're sort of targeting the book to is people who are just starting to think about the college process younger kids I guess to the degree that they have to start investing in the kinds of materials that are used by college for admission by colleges for admission at least might find some of this information helpful but I really think it's juniors and seniors who will find the most relevant for the degree that to this question of whether kids have access to this information you know the best I think that we're able to do is you know sort of send this book out into the world raise publicity to try to help kids get this information I think it really calls attention to the academic work that's being summarized in this book so right so fundamentally this book is not doing any you know tremendously new work what it's doing is translating a lot of work by you know many of my colleagues and others who have done relatively technical work on the relationship between educational choices and long-run outcomes of students that is wholly inaccessible to the public and the book is an attempt to make this information more accessible you know I think the the sort of variety of authors and backgrounds that we brought to the book hopefully did a helpful job in sort of translating this work to the public but and so you know we're sort of now in the process of making sure the public has access to this information as best they can well and we're happy to do our part to spread the word Roxanne is just down the road from you by the way she's in Connecticut as well and Roxanne thank you for the really really good question so that's an example of a text question so again just go look at the bottom of the screen where that white band is and go to the question mark if you want to type in a text question like Roxanne's or click the raised hand if you want to beam on stage like Lynn we have questions already in the pipeline Zach so I want to make sure they get a chance our dear friend and fellow author Tom Hames asks one of his typically deep questions and this is do our metrics measure content over skills when skills are really the determining factor and success post college so I think there are a couple of different ways of taking this question but I think that the sort of the dominant thread that I'm seeing in here is just wondering about how universities choose which students to admit into a university so typically this the sort of distinction between content and skill I think is made in the context of like you know thinking about standardized tests and grades as being two different kinds of information that are used by universities in making admissions decisions and I think the as a result of sort of relevant answers that question depends on the objective of a university so in the case of private universities that might have a variety of objectives but I think largely come down to that they're looking to produce alumni who are very successful in society I agree if there's some sense in which just identifying the kids who you know whether or not they they learned a lot in high school why the kids who just have tremendous capabilities maybe as shown on a standardized test might be the right kids to admit in that context because these are kids who you know long after graduating high school might be able to take advantage of those capabilities in the labor market whether or not they went to that university at public universities I think that the context is a lot less clear so not really clear exactly what the objective of a public university should be in admission and moreover you know as students are making this choice between public and private university and sort of the same question arises of like what is the student's objective in these different contexts so the degree that public universities are intended to identify the kids who will most benefit from their enrollment I think we now have growing evidence to suggest that standardized tests and these sort of measures of skill do a very core job of identifying these like potentially high value ad students whereas measures of content like grades do at least a somewhat better job though still imperfect in identifying which students would really succeed at public universities I sorry if I twisted your question in a different direction than you intended it but at least I think I think that's what's most relevant here well it's a fascinating question and it's a very very rich answers act and Tom if you want to follow up just click the raised hand and we'd be glad to beam you on stage we have more questions and I want to make sure everyone gets a chance to to ask our friend Glenn McGee who is always always has a laser-like look at the data and Glenn asks interesting questions this hasn't come up in our conversation so far metrics that matter needs to include under-employment or mal-employment which ends currently a 43% can this be a proxy for credential inflation yeah so I think this is a super interesting question there are a lot of people who hold college degrees who nevertheless work in jobs that don't require a college degree from an economic perspective there's a certain notion by which you know this is somewhat concerning right you know there's a sense in which people aren't taking advantage of the college degree that they hold though I think in many cases that's often not clear so even within narrow occupational groups it nevertheless remains the case that wages for college graduates are higher than wages for non-college graduates this indicates that the college degree is still providing productivity enhancement that people are still providing added value in the workplace on the basis of their college degree that isn't necessarily just captured by their occupational title but that you know that piece aside I think it also calls into account you know just questions of exactly how long-run economic outcomes for different kinds of educational attainment are constructed so to give one example you know if you think about like the average wages in different college majors you know how do we calculate those average wages well we only look at the people who hold jobs and then we take average wages among the employee now that's including both the you know so called over employed and under employed but if everyone gets thrown into the same bucket but notice that one thing it doesn't include it doesn't include is the unemployed right so so one thing that's happening when you look at average wage by major statistics is you're you're not paying attention to the fact that whether or not some majors might lead to relatively high wages that doesn't necessarily mean they lead to employment at all or employment in the first place if you look at unemployment rates they differ pretty widely by college major sort of interestingly currently the highest unemployment rate major in the US is linguistics and and so as a result you know as you're thinking through like what is the relative economic value of a linguistics major now I really don't mean to emphasize that the only value of a linguistics major is economic but to the degree that you're sort of interested in the economic value of linguistics you might want to sort of temper your expectations just on the basis of average wage statistics because while they do account for these measures like these employment like under employment they're not accounting for unemployment altogether so in this chapter on average wages by major we go through a number of examples like this of sort of challenges in just the direct computation of the economic value of relative of different college majors again not just you know not stating that this is the only value of majors but to the degree that students are interested in this these are things that probably they should be taking into account that's interesting that's a very very different way of thinking about this by the way glenn first of all thank you for for bringing this up and we really appreciate this and also glenn just shared here I'll share this twice a fascinating tool this is from the census department taking a look at links between STEM degrees and what people actually study and I'll just copy and paste this in the chat too because it's a it's an elegant visualization and a fascinating one I haven't been to this website but I suspect what it shows is that you'll see that people who earn degrees very rarely work in the sort of most natural industry associated with that degree so to give an example from my own field no more than 20 percent of people with economics degrees are working in industries that have really anything to do with economics training which does mean that these degrees aren't providing value it's just that you feel they're providing value in a lot of areas you might not expect them to well there's there's a limit to the signaling function of the degree in terms of its content we also have another question from president george print steiner or point steiner who has had a fantastic career and he asked a very very direct and powerful question how could you find those important state or university policies yeah so I I think there are a number of cases where this can be helpful so again I'm just going to take one example in particular so the final chapter of the book thinks about mobility rankings between different universities so I think what people might be most aware of in this context are numbers that have been put out by Opportunity Insights Raj Chetty's group through their mobility report cards which indicate which schools have the highest share of low income students that end up succeeding later in life so one thing the book discusses in this final chapter on these mobility rankings is that the mobility rankings don't necessarily conform to the schools where lower income students would be best off and in fact I think it's often not the case that lower income students would be best off going to these low or these high mobility institutions so for context the high the highest mobility institutions in the US tend to be in the California State University system and in the CUNY system in New York these are schools that do enroll very large numbers of low income students and their graduates have surprising success given the high schools and family backgrounds that their students are coming from but the reason that they're indicated here as being called high mobility is not that their causal effect on low income kids is is supremely positive but instead because they enroll a very large number of low income kids the more low income kids you enroll the greater degree to which you are considered a high mobility institution as a result low income kids who are making choices between institutions should not be looking at mobility rankings actually not relevant information to low income kids as they're trying to decide which schools will most benefit them in terms of educational and labor market gains as I mentioned earlier an indicator like the graduation rate is a much better indicator for low income students to try to determine which schools that they should be choosing however on the side of policymakers these mobility rankings are extremely helpful these mobility rankings are indicating which schools could be most benefited by federal and state dollars to promote the lower income students enrolling at those schools so these are schools that have student bodies who are likely extremely susceptible and amenable to additional funding they tend to be substantially underfunded schools and so I think these mobility rankings while sort of you know presented publicly as if useful to students are in fact primarily and maybe solely valuable to policymakers as they're making funding and allocation decisions about where to to you provide greater support to university administration and to student bodies well George thank you for that great question and and Zach you just gave a terrific outline of what could be done with this in the check letting me build on this and asks quote any thoughts on how more complete data can be inputted into the policy space what would happen then after including for things like under employment effects of credential inflation yeah so again just to choose one particular example going back to average wages by college major so you'll take two majors at a given university say you know geology and history and you can now go and you look up in the college scorecard the average wages of people who earn one or another of these majors you know they present these wages three years after graduating which is very short a lot of these kids might still be in graduate school or haven't yet you know properly fallen into their long-run career trajectory but this provides a little bit of information and what you would see is that on average geology majors earn somewhat more than history majors in their in their early careers and persistently throughout their lives and so you can imagine on the one hand students looking at this information and making choices you know I wanted to be a history major but now I see that geology majors earn more money I think that that could open doors for me later in life and so I wouldn't want to say no to this opportunity and you can imagine policymakers looking at this and say oh we should do our best to expand these majors you know make as many states available maybe encourage students into these more lucrative fields of study because it might be in their best interest financially to choose these more lucrative college majors yeah so I think it's a sort of a plausible thing for people to believe on either side and what it but it turns out that I think that's that can be very misleading to students and so I just want to talk you through I think a really fascinating the most fascinating study that I'm aware of in terms of the long run outcomes of choosing one or another college major for students and this is a study that was conducted by Magnum Obstatt who was an economist at the University of Chicago it was published in 2016 and it focuses on this sort of interesting setting in the country of Norway where Magna is from and so Magna and his co-authors noticed that Norway has this application system into universities where instead of applying directly to universities what kids do is they send in to the federal government a rank order list of their most preferred university major pairs so you say okay you're my first choice is I want to be a computer science major at the University of Oslo and if I can't get into the computer science major at the University of Oslo then my second choice is to be a history major at Bergen and my third choice is to be a computer science major at Bergen etc kids send in these ranked order preferences and then there's a centralized admission scheme kids are admitted into the most preferred of their programs that still has a slot available for people of their high school GPA and test score so the better you do in high school the more access you have and so you know the top programs get allocated to the top students and then sort of all the way down kids get into the best program that they would be able to get into but they're only admitted to a single program so magna and his co-authors look at this and say here's a really cool opportunity to study what happens when say there's a kid who really wants to go into say geology but there's no slots available for students with their high school grades in geology so they get bumped down to their second choice say history major instead and then there's also a different kind of student a student who really wanted to get into their most preferred history major but that's not available to them those slots were already taken up for kids of their type and so they got pushed into geology instead and what the study shows it follows both of these pairs of students for the subsequent 10 years and so follows them into the labor market looks at how much money they're making in the Norwegian labor market and shows that in both cases these students are are losers in terms of long-run labor market outcomes if they get their second choice relative to their first choice now one of those isn't surprising getting bumped from geology to history I think it would surprise no one to learn that that kid ends up earning less money later in life after all they're not earning one of these lucrative STEM majors but the other is more surprising the kid gets bumped from history into geology you might have thought would get a wage advantage but that's not what happens actually they end up with relatively lower wages in their late 20s then they would have had if they'd had access to the history major their first choice major the authors take this as evidence of what they call comparative advantage but what we can just think of as a relative lucrativeness or or a financial advantage in studying what you most prefer to study it looks like at least in cases where the relative returns to different universe different majors are relatively small having you know just having access to your most preferred major being able to study what you want confers with it long-run labor market benefits even if the average wages at major are somewhat lower than your sort of fallback major the reason I think to pay attention to this is that this is a reason for policymakers to not try to push kids into STEM fields and away from fields that have somewhat similar but lower wages that would be a missed opportunity for kids to take advantage of their most preferred field which would lead to better life outcome overall that that's fascinating that's a fascinating study thank you thank you for drawing attention and summarizing that and and for using that in the book friends we only have about 13 minutes left and I want to make sure that everyone gets a chance to ask their questions so again on the bottom of the screen just click the raised hand if you want to be up here on stage with with Zach and I you don't have to have a lot of books in the background you can still join us or just type the the question mark hit that and you can type in your question in the chat box there while people are thinking about this I had a question specifically about the student debt chapter which is interesting because that's a metric which is which is outstanding I mean that's a you know the total amount of debt is huge and there's a lot of discussion about different types and and in your conclusion I'm not sure if I followed it I mean your conclusion seemed to be that there's lots of different kinds of debt some of the debt pays off pretty well and that it's a it didn't seem to be a good replacement metric for us to pay attention to yeah what do I'm missing or what did you conclude on that no so I think that's exactly right I think our struggle with student debt metrics is that fundamentally what was being reflected in student debt metrics had very little to do with actual financial decisions making you know being being provided to students instead fundamentally the decision that students were making was about tuition and debt was one of the many ways that they could choose to pay that tuition and so what we sort of end up concluding in our chapter on debt is that rather than paying much attention to say average debt loads coming out of different institutions or overall average debt in the US economy these are choices that kids and families have to make their choices that are primarily being made even before kids choose where to go to college once they finally have financial aid information from institutions and are able to make informed choices about the relative costs and benefits of different schools you know we don't think that these metrics on debt are sort of adding additional information over and above what kids can learn about the actual costs of different schools and so I agree this is the one chapter where we're not clearly pointing students in some other direction in fact this information that we just don't think is very helpful to students and instead they should be paying attention just directly to the relative costs of going to one or another school or studying one or another field oh interesting so this is a metric that is kind of uh that kind of blots out other more useful metrics even though it is accurate what it describes at a macro level it's not going to be very helpful at the micro individual level yeah that's exactly and actually that really calls something out very important so again from the perspective of policymakers who are thinking about how university education is financed in the U.S. these debt metrics are extremely helpful so you know the Federal Reserve Bank of New York publishes quarterly statistics on just aggregate debt in the U.S. economy and provides a lot of information about what kinds of kids hold that debt what schools are generating that debt where in the U.S. the debt sits these are really important pieces of information for policymakers who are thinking through changes in state funding of higher education changes in the allocation of financial aid etc but they aren't very helpful pieces of information for students who are choosing whether and where to go to college that's fascinating because that's something that I know I pay a great deal of attention to but if we go back to Roxanne's question on the question of an individual student then we can have much more actionable much more relevant data well thank you thank you Zach that's a really good answer thank you it clarifies a lot we have a question from Shelby Rosengarten at St. Petersburg College in Florida and as usual for everyone from Florida we send our best wishes for your survival during this time and Shelby asks the question of how does this apply to community colleges was that part of your study yes it was many community college graduates successfully transferred out and don't bury the same weight of debt yeah yeah so it's a fantastic question and I think really highlights the value of metrics like for your graduation rate though it should be emphasized that it can be somewhat difficult to get graduation rate measures for community colleges so so one thing we do in the book is right so we as we're talking about the the kinds of institutions where kids can enroll and long run outcomes have been rolling at these different institutions community colleges are very much part of that story they are substantially lower cost on average and their open access so students would in any case be able to attend these schools but if you were to say you know try to measure a six-year graduation rate for community college so what do I mean by that I mean taking the kids who enroll at this community college coming out of high school and maybe not including anyone who just intends to uh to you know say go to community college for a year and then into the labor market or people who have already been in the labor market for some years and come back to community college uh before you know just to get some uh professional training before returning instead just focusing on the kids who come straight in from high school and then you can just ask the question okay what share of those kids have earned a college degree within six years from any four-year institution in the U.S. or if it's community college that offers BAs which is now growing in popularity in some states have kids who show up at this community college earned a BA from either the community college or somewhere else. What you see is that there's a wide range in the success of community college students in eventually attaining four-year degrees which it should be emphasized are far greater economic value than just the associates degree that most community colleges offer though it should nevertheless be emphasized that associates degrees confer economic returns that not going to college at all do not. What you would see so we've measured these six-year graduation rates for most of the California community colleges and there's a range of community colleges from something like 10 to 50 percent which is to say 10 to 50 percent of kids coming into the community college straight from high school have earned a college degree somewhere within six years of enrolling at this community college. Now for those community colleges at a 50 percent level that's actually not so different from many of the sort of the less selective or relatively open or access public four-year universities in the U.S. and in particular in California in fact it's substantially higher than six-year graduation rates at many of the non-selective four-year public universities in the U.S. that being said it's substantially lower than the graduation rate of what you might think of as like relatively more selective public universities certainly almost the entire set of public research universities and so I think the takeaway here is on the one hand community college does save money that's important but to the degree that students have access to relatively higher graduation rate institutions in the four-year sector that's probably worth it in terms of at least long-run economic outcomes for students and in terms of likelihood of actually attaining a college degree relative to just going to the community college. Well that's and that covers so much ground and then brings up the whole question of transfer rates and credit you know mobility and all of this. Thank you, thank you. So Shelby thank you for bringing community colleges and thank you Zach for such an incredibly powerful answer. We have one more question and we'll go back to Lynn Sibolski who asks a text question here and Lynn asks for both of us Zach and Ryan how much value do the higher-ups policy makers at higher educations place on the research finding in your work? Is it hard to get their attention or do they actively seek out your expertise? Well Zach you go first, you go first. So I'll give two answers to this question maybe one more promising than the other so I'll start with a less promising example I mentioned toward the beginning of this conversation. This study in the Massachusetts setting in which for about 15 years the Massachusetts state legislature has provided the Adams Scholarship to the top 25 percent of kids coming out of each Massachusetts high school. The Adams Scholarship guarantees full tuition cost at all public universities in the state of Massachusetts and so this I think really excellent paper by by Josh Goodman and Sarah Cahody is a couple of years ago showed that the net effect of the Adams Scholarship is actually to decrease students' likelihood of earning a college degree within six years. Why is that? Because the public Massachusetts university system especially compared to many other states public university systems but especially compared to the private university system in the state of Massachusetts is not doing a very good job of graduating many of its students they have relatively low graduation rates and so the net effect of the Adams Scholarship is actually to encourage in-state students into relatively lower graduation rate public institutions where they are less likely to earn a college degree than the other white than the schools primarily private schools where these students would have otherwise enrolled. This is frustrating but to this question about policymaking the paper was published in 2014 it's now almost 10 years later and the Massachusetts government still provides Adams Scholarships to students so we still believe that the net effect of the Adams Scholarship is a decreased graduation rate this has not stopped policymakers from awarding these scholarships. I think you know to give sort of one second example as I mentioned at the beginning of the call I've been doing a lot of work recently about race-based affirmative action I put a number of papers on the subject over the last couple of years and because of the expected Supreme Court decision in June will lead many public and private universities to have to change their undergraduate admissions policies I've now been receiving a lot of calls from admissions offices and other university administrators sort of hunting for alternatives and I think it's been a space where actually because universities have so little experience thinking about what it would mean to conduct race-neutral admissions policies this has been a place where my sense is that academics have been very influential in shaping how selective public and private universities can share policies that have been implemented in benign states that have already banned race-based affirmative action and can try to get a better sense of just what's available to them and how they can move forward as a result of these likely radical changes in how they admit students. I'm well thank you for that answer Zach I'm just that that out of the story just depresses me I haven't heard that and that's so god that's that's so sad the answer question very very quickly then because I'm here in my facility to roll rather than present a roll I get a demand for my services I get flown out or video in to talk to trustees to some state governments some national governments and people buy my books and and people watch videos like this so I have some influence and share my research I would like to have more I'd like to sharpen my research but there's some traction on the other hand well I'll hold back from saying more because I don't get too far off topic but thank you for asking thank you for asking I'm as the host though I do want to assert one bit of privilege which is to ask a final question and Zach if you could wave the proverbial magic wand and and get everybody in higher education and everyone paying attention to higher education to switch the metrics you know from the from the preposterous and ill-suited data that you described and criticized to the alternatives that you talked about instead what would if that happened and then you were to press fast forward on the on the video and it's 10 years later what what changes would that make in college and university life great so I think as as we were putting this book together thinking about how we should make these recommendations we had two goals in mind and I think it's the same two goals that would occur in the scenario that you imagine so if people on the student side students and parents were to pay more attention to the information that we're making suggestions about and less attention to the many kinds of metrics that kids are primarily paying attention to so what this would do is allow students to make more informed decisions about whether and where to go to college and what to study so what would the net effect of that be well given that on average informational quality among students and parents is substantially lower among lower income and otherwise disadvantaged high school students I think what you would see is a net inflow of lower income and otherwise disadvantaged students into both less and more selective universities so there's both an inflow into the higher education system altogether because I think these metrics would indicate to those students the very substantial value of holding a college degree in the US in the 21st century but also would would push lower income students toward what are actually higher quality institutions for those students schools that are able to educate those students get them degrees and then lead them to successful or relatively more successful lives so I think it would sort of help reorient the informational sector among parents and students that would primarily benefit people who have relatively poorer information about the higher education system I'll very quickly say in the last 30 seconds here that I think it would also help reorient university priorities away from you the book is filled with these anecdotes of like dumb ways that universities use to either increase their rankings or make them look more prestigious or put or lead kids to make really problematic decisions about what they want to study and how they want to spend their years in college I think the metrics that we've chosen are in a sense incentive compatible if you were to incentivize schools along these metrics I think this would improve the quality of education that these schools are providing to their students wow well that's that's a magic wand we need to get waving I think and I hope in this past hour of very energetic and genial conversation I hope we've waved that wand a little bit further Zach we are unfortunately out of time and I appreciate your your eye on the clock what's the best way to keep up with you and your work can we just email you out of the blue or are you active in social media how do we how do we keep up with the next step in the lean reverse yeah so you'll see so you know all five of the authors on this book have public profiles it's easy to find me on my website I don't have a regular newsletter though I'm publishing a couple of papers a year and they'll be posted to the website but also please feel free it's just you know first name dollast name Zach redoublinger at Yale but EVU I'd be very happy to talk to any of you about any of these issues well fantastic well good luck first of all on on all on on this book in the world I think I'm so honored that we had a chance to spend time talking with you everybody this is a book I think it's it's very short it's incredibly readable it's just a a primer I think on how to proceed I just hope that we all get a chance to take a look at it and Zach all best with your research projects coming up thank you so much and enjoy the warm weather while you can cheers the same to you all thank you Zach thank you all friends for the great questions for today thank you all for the for the thoughtful ways of interrogating and thinking about metrics let's take a look quickly at what's coming up we have a session coming up on edtech and labor and still more to come you can find that on our website if you want to keep talking about these measures is this the right way to think about return on investment is the right way to connect students with the right majors just use a hashtag ftte and throw that at twitter or at mastone or wherever we happen to be if you want to go back and look at similar topics including our previous two appearances by chris newfield just go to tinyurl.com slash ftf archive and above all thank you all for your time today it is as always splendid thinking about the future of high red in your company i hope you're all well if you're in a spot getting spring i hope spring springs with a great deal of delight in the meantime everybody take care we'll see you next time online bye