 Welcome back, everyone. Contrary to appearances, we do have two commentators, they're sitting right here. No, they're actually sitting in the audience so they can see the slides. The slides from yesterday having garnered a very good reputation. I actually dreamt about them. Four o'clock in the morning, all I could see was synapses. Anyway, welcome back, everyone, and thank you very much for attending today's lecture, whether in person or on Zoom. I'm Rebecca McClennan. I'm on the faculty of the history department, and I'm also serving on the Tanner Lectures Organizing Committee this year. Today, Professor Caroline Hoxby will be presenting the second lecture in her Tanner lecture series, The Fork in the Road, the imperative of investing in adolescent education. Now yesterday, Professor Hoxby was very persuasive in arguing for the existence of a fork in the road of children's cognitive development, one that accelerates not so much in the first three or four years of a child's life, though it does a little, as most likely assume, but around the age of 10 and a half. Now, Professor Hoxby, I wish I had heard your wonderful lecture two years ago before I started writing preposterously large texts for my toddler San Francisco preschool. But building on this theme, today's lecture is entitled Smart Money. Educational investments in adolescents earn higher returns. We'll take a quick break after Professor Hoxby has further enlightened us, and then I'll introduce today's distinguished commentators, and after that we'll move to discussion. Thank you. Please welcome Professor Hoxby. Tendents on the Zoom. I'm going to see whether I can do better with the clicker today than I managed to do yesterday. So again, I'm just showing you some... I'm hoping... It's not on. Thank you, Jane. Synapses to get started on the neuroscience, but mostly today we're going to be talking about tests of whether adolescents are able to learn more than students at other ages, and so that's kind of the reason for the title of the talk today. Sorry, this one goes a little bit slowly. Is this image heavy? Okay, so Smart Money. Educational investments in adolescents earn higher returns. So I want to start with a very brief recap. I'm going to try to keep it brief, but just so people who weren't here at the lecture yesterday know where we are. So the key points were first that increasing the share of Americans with advanced cognitive skills is crucial in my opinion for social cohesion, for advancing economic opportunity, for reducing inequality, and for decreasing geographic and political polarization, and possibly also for other things like reducing crime or improving health. Now, recent neuroscience and neuropsychology suggest that adolescence is a key period for frontal cortex development. Since it is this frontal part of the brain that is thought to be responsible for advanced cognitive skills, at least disproportionately, we should expect some of these skills to really take off during adolescence. And in fact, I showed you this chart yesterday, the measures of children's cognition, and you can see that in early adolescence, this period right here starting at about 10.5 and going to about 15.5, but generally speaking that period, you can really see children who start gaining advanced cognitive skills, their skills take off on a sort of high trajectory, and other children, their skills really stagnate during this period of time, and then they tend to flatline thereafter, and they gain very little, if any, skill in their late teen years. So in other words, early adolescence offers both great opportunities to learn, but also risk of not learning. After early adolescence, I showed you some charts that looked like this that show that students' skill trajectories start to harden, and this is consistent with what we would expect from neuroscience, specifically the pruning on myelination processes that occur in adolescence would sort of harden up these trajectories. I also showed you a couple of maps of the United States. This is one that I happen to like a lot. I think it expresses political frustration and resentment of elites that may stem from the economic fatalism that some people feel when they realize that they have attained young adulthood without acquiring advanced cognitive skills, and they may be shut out of remunerative and fulfilling careers. So this one shows you what people think scientists think about climate change. It's important to say it's not what they think about climate change. And I concluded yesterday's lecture by saying that it's one thing to try to identify early adolescence as a key time when students should start developing advanced cognitive skills. It's one thing to conclude that as a matter of logic or by looking at the neuroscience, but it's another thing to demonstrate that we actually have realistic educational interventions that would be especially productive if they were implemented in early adolescence. So today's lecture is all about those demonstrations. I'm going to leave this up for right now and tell you a little story about how I became interested in adolescence now that we know where we are going forward. So I teach an undergraduate class. It's called the Economics of Education. And in it, we regularly examine recent research on early childhood education, everything from the very famous Abbasidarian and Perry preschool randomized control trials that are now almost 50 years old to recent research on the federal Head Start program. And every year, I teach my students the theory of endogenous skill growth, which posits that children who acquire skills sufficiently early benefit more from each later educational experience, creating trajectories that diverge further and further apart with age. Now, this theory predicts that positive early childhood interventions, such as Abbasidarian, which believe it or not, enrolled children at four months of age on average, or Perry preschool another very small but intensive program, or Head Start, that's the big federal program, a very big program, should launch children on a permanently steeper trajectory of skill growth. In fact, though, the evidence looks quite different. The effects of these programs are evident during the intervention period itself, when you're enrolled in Abbasidarian or Perry preschool or Head Start. But then they tend to die out fairly quickly by second grade, typically, sometimes even first grade. And the students and I would always find the theory of endogenous skill growth compelling and intuitive. We would think to ourselves, this makes sense. This comports with our own personal experiences. So we would finish up the lectures on early childhood education with a sense of class-wide frustration, including myself. And after class, I would inevitably field many questions along the lines of, if the theory is so compelling and intuitive, and we all think it makes sense, why isn't there stronger evidence for it? The other experience that forced me to pay attention to early adolescence was my research on charter schools in New York City. In this research, some of which we will see later, but very briefly, I studied more than 100 charter schools using an experimental design that makes use of application lotteries. Essentially, the experiment compared students who were and were not given seats in charter schools purely because of their randomly assigned lottery numbers. I'll talk more about that a little later. Now, some of the charter schools start with pre-kindergarten or kindergarten, and then they end with an elementary grade, let's say grade 5 or grade 6, and some charter schools start with grade 5 and end with grade 8, so they're charter middle schools, as it were, and some charter schools are more like charter high schools. They start typically with grade 9. Now, the many economists who work on early childhood had led me to strongly expect that if a charter school was going to be a success, it would be more successful if it started with the earliest possible grade, thereby launching its pupils on a higher endogenous growth trajectory of skills. I was really baffled by the people who wanted to lead charter schools that started at grade 5. I thought, you're just setting yourself up for failure. You should have started your charter school with kindergarten or pre-kindergarten. Why are you starting with grade 5? I was even more concerned because the data showed very clearly that schools that started with 5th graders had applicants who were already struggling and already falling behind in the regular public schools at the time that they applied to the charter school. I thought, this is too late for these students. They're never going to get the full benefit of a successful educational intervention or a successful charter school if the charter school is successful. But when I computed the results in this charter school study, it was immediately obvious that these grade 5 to 8 schools, were not only generating greater skill gains than other charter schools, their gains were often much greater. I reported the results, but I really just didn't know what to make of them or how to explain the differences. I also became aware that there was a reasonably large Mathematica study that's a nonprofit research organization. It was conducting a large randomized control trial on charter schools, and it was coming up with very similar results about the middle charter schools doing an unusually good job. So one day I was walking back to my office after having taught my early childhood classes and having heard all of my students' frustration, and I thought about the charter school evidence and I thought about the early childhood evidence, and I realized that when I looked at education data, which is of course something I do ordinarily in life, that the fork in the road did not appear as early as I had expected it to occur, something like kindergarten or pre-kindergarten, but rather seemed to advance itself mainly in middle school. And I also recalled that a sizable share of students appear to get stuck at the middle school level of skills. This is typified by the many students who are still struggling to learn Algebra I in their freshman year of college, even though they have been taking it in variously named math classes since the seventh grade. Indeed Algebra I is the most commonly taken college course in the United States and freshmen who fail to master it often have to drop out of college, not just because it's a prerequisite for majors like engineering or science or something along those lines, but it's also a prerequisite for many remunerative community college majors, just dental hygiene for instance or a lot of nursing majors. So anyway, this concatenation of thoughts about the charter schools, the middle school, the charter middle schools, the early childhood education and thinking about these people who get stalled in these middle school levels of skill made me rush off to the library which luckily is very close to my office and grab a whole stack of books and articles on neuroscience and adolescence and that was the genesis of these lectures. So I just grabbed the books and I had a wonderful time taking a month or so off from economics and reading neuroscience and neuropsychology instead. So if adolescence is indeed an age of opportunity, this is a phrase coined by Lauren Steinberg, then it also has a big advantage vis-a-vis early childhood. Early childhood interventions are plagued by logistics. Many adults are needed to provide infants or toddlers with adequate care. Children of that age, think of these four-month-olds, need help with basic tasks such as eating, toileting and dressing. They usually have to be dropped off and picked up. They cannot just ride on school bus with other older children and many parents are just uncomfortable with handling their toddler over to others by unless the setting is intimate and familiar to them. Adolescents on the other hand are quite different in terms of logistics. They're fairly independent people. They're already in school for six hours a day. So in other words, if we can find educational interventions that improve their cognitive skills, they are a captive audience. Now the next thing that I want to talk about is the current neglect of adolescent education, which might surprise you very much given the fact that this key opportunity time and a time that's a risky time, it's unfortunate that adolescent grades are by far the most neglected in terms of teaching. The neglect in my opinion is unintentional and it derives from some basic phenomenon. Class size is the largest in the middle school grades. That's about grades 6 to 8. So when I say middle school grades, I keep thinking grades 6 to 8, usually. This is partly because of the fascination with early childhood education combined with the fact that the logistics for smaller kids are just more difficult. So schools tend to put them in smaller classes. High school students are also in small classes, but for a very different reason. It's really because the curriculum starts to break up in high school. So it's not just that there's science, but there's chemistry and there's physics and there's earth science and biology. So classes get split up into smaller and smaller units. The same thing could be true of history. There's US history, European history, world history, and so forth. So evidence on class size. Sorry, this is one of my cognitive skill graphs showing you that early adolescence seems to be the time of divergence. This is average class size by grade for the state of North Carolina, which happens to have amazing administrative data. So those are the data that I'm using for this. And I'm going to keep using the North Carolina data for a little while for the rest of the lecture. So you'll see more North Carolina data. But if we look at average class size by grade, you can see that kindergarten has about 14 children in each class, very small. The rest of the elementary grades from kindergarten to grade 5, it's usually 20 or fewer students in a class. Then we have the middle school grades 6, 7 and 8 and they are the middle school grades in the state of North Carolina. And you'll notice that all the class sizes are 27 or 30 even. Okay, so much, much bigger class sizes. And then we have a decline in class size as we get higher and higher into the high school grades with the smallest class sizes being in 11th and 12th grade. So I'm going to show you from now on and going to cluster some of those grades together, both to keep the graphs a little bit neater and also because there are some data reasons for doing that into which I will not go. But so this is basically the same chart as my last chart. And I'll keep using the same colors all the time. Elementary will always be green, middle school will always be pink and high school will always be blue. So it's just a way of collapsing the data to make it a little bit easier to read. So not only is class size the largest for middle school teachers but their compensation is lower than that of elementary and high school teachers. So you can see again green, pink, blue and the pink bar is the lowest and that's because middle school teachers are paid less than elementary and high school teachers. This is not intentional but it appears to be due to the fact that teachers find it taxing and difficult to work with adolescents. Probably due to the brain development that makes them especially plastic, cognitively at that age or it could just be puberty and the social transformation that they're experiencing. In any case teachers who teach middle school grades are more likely to report problems such as student apathy, student mental health issues or students being belligerent. They're less likely to say they are satisfied with their current job or satisfied with teaching and so it should not come as a surprise that teachers often move to an elementary or high school as soon as a vacancy arises in one of them and as a result it's middle schools that are always seeing vacancies occur and having it half of time filling them. Now most public school teachers as you may know are paid in the United States according to a scale that depends only on their years of seniority and their highest educational degree like whether you have a bachelor's or master's degree. This is called lock step pay. Thus middle schools have lower paid teachers because they constantly need to fill their vacancies with more first year teachers or rookies or teachers who have little or no seniority in the district or teachers who have been taking years off. So that's what's shown on this chart this is the percentage of teachers who are rookies in elementary middle and high schools and you can see that you have more rookies in the elementary school sorry the middle schools and this is the percentage of teachers with no experience in this district so they're completely new to the curriculum. They may not know things as well and you can see again that's higher in middle schools and it's because more vacancies arise in middle schools. It's not just a problem for having very little experience with teaching it's also a problem for teacher quality. Although teacher value added research which I am going to talk about in a moment finds little increase in teachers effectiveness after they have three or four years of experience rookies are consistently found to be less effective and so are teachers who have no experience in their district. In other words middle school teachers not only encounter larger classes and less pay middle school students not only encounter larger classes and less well paid teachers. They also face teachers who are more likely to be struggling with instruction and who are dissatisfied with their current job. Middle school teachers are also less likely to have a graduate degree as shown here you can see that graduate degrees are quite disproportionately higher for people who are high school teachers and this is not a particular surprise. Remember that the middle school teachers are more likely to be rookies and since so many teachers in the United States actually get their master's degree they're just less likely to have a graduate degree. Well that means they're going to be paid less because of the lock step pay which gives you an automatic pay promotion if you get a master's degree and all of these phenomena add up to this figure which I think is very dramatic. So what this is showing you is the amount amount spent per pupil on teacher compensation and you can see that it is much much lower for middle school students than it is for elementary school students or for high school students. In fact it's less than half of what it is for high school students. So why is this it's all of these forces combining it's the bigger classes it's the less experienced teachers who aren't getting the seniority pay it's teachers who don't have a graduate degree so they're not getting that bump in pay and that means that what we have is middle school students actually being unintentionally neglected in terms of the resources that they get so at this crucial point in time where their brains are plastic and they're trying to make transitions to more advanced cognitive skills they're also being comparatively neglected. Alright so regardless of the reasons why this happens it's clearly counter productive in the sense that I tend to think that middle school teachers really may need combat pay of some type not the other way around because they're dealing with more risky harder to teach harder to teach children. Okay so now that we have established that early adolescents are relatively neglected people I want to return to the central goal of this lecture that is conducting tests that can generate plausibly causal evidence where early adolescents are especially affected by successful educational interventions and I had four criteria for the natural or policy experiments I'm going to use those two words interchangeably that I employ I want to test interventions that one are specific to a grade or an age so that I can compare results across ages I can compare what happens to an adolescent versus someone who's in elementary or high school and my third criteria is the intervention should be something that you could apply at any grade again this is so that I can compare results across grades in other words if it's some kind of trigonometry intervention you obviously aren't going to apply that to third graders so it can't be an intervention of that type the three my third criteria was that I wanted to test interventions that have typically been found in the past particularly significant effects that's because there's just no point in comparing results across grades if all of the results are null or extremely noisy and my fourth criterion was that I wanted to test interventions that allowed me to examine later effects such as a student's achievement at the end of high school or going to college after all given that age only moves in one direction we keep all getting older sad but it's true I see the effects of an intervention that is applied to ninth graders and then see oh now that I've affected them as ninth graders what would have happened to their third grade test scores they're already past the third grade but I can test the same intervention if I apply it to third graders and ninth graders and then look at both of them at the very end of high school and say where did it make the most difference now it turns out that I had to rack my brain to come up with interventions to build all of these criteria and I actually only came up with three that I think are at all good and the first one is what I think is the most important one it's about a teacher's value added the idea is quite simple it's having a teacher with high versus low value added in other words ability to teach students more materials, ability to teach students more skills would I want to have a teacher when I was in middle school who has high value added versus elementary school versus high school I'm not going to get a high value added teacher every year so where would I like to concentrate them if I had the choice my second which I hope you will find relatively fun is being exposed to a curriculum that is more challenging cognitively for instance if a school district is going to introduce a new testing or a curricular regime that has richer cognitive content and it's going to do that across all grades should students want to see that happen when they're still in middle school as opposed to say high school and then I'm going to talk about attending a successful charter school going back to this story with which I began so if students are only going to be able to secure a charter school seat in one lottery they can maybe win in elementary school maybe they could win in middle school maybe they could win in high school where would you want to win the most okay I'm going to spend most of my time on this value added because I think it's especially important partly because of who teachers are and partly because it's really the best policy experiment just to be clear so in my first natural experiment I'm going to test whether having a teacher with relatively high value added is especially important in the middle school grades so the way you want to think about this here is we could conjecture what if we could just concentrate on the most effective teachers in the middle school grades perhaps we could pay them more to teach in those grades or maybe we could reduce class size in those grades to make those jobs more appealing we could also have other amenities for them for instance occasional sabbaticals to attend graduate school or do something like that something to make them want to take those jobs and stay in those jobs and compared to the curricular changes in the charter schools I am going to focus more on the teacher because it's not only the most informative but in my opinion it's the most relevant to policy there is now a very extensive body of research that shows that individual teachers matter and that they differ substantially in their value added even within the same school and the same grade therefore any policy maker could propose to pay them or hire them differently as mentioned earlier to induce them to teach particular grades now I'm going to use the most validated method of estimating teachers value added and it may seem a little complicated when I describe it but in fact it is intuitive if we compare the problem to a simple randomized controlled trial which is what the method mimics suppose that there are two third grade teachers Smith and Jones they have classrooms located right next to one another and every year the school principal randomly assigns 20 students to Smith and 20 students to Jones something like that then Smith and Jones are essentially conducting a very small experiment every year 20 getting the Smith treatment 20 getting the Jones treatment and so we can answer the question if we did this for a while we could answer the question do students systematically gain more skills in Smith's classroom than in Jones's classroom now of course in anyone given a year Smith or Jones might get an unlucky draw of students for whom learning is difficult and even if we try to control for each student's characteristics such as poverty, race, ethnicity and gender and try to control for their prior achievement before coming into Smith's or Jones's classrooms there will probably be some unobservable differences in their two classes tendency to learn but if you repeat the experiment for several school years then we can iron out this problem simply because of the randomization it's not going to be that Smith is always lucky or that Jones is always unlucky so as long as you have multiple years of data think of them as multiple little experiments on the same teachers and you have students prior achievement and their characteristics and you have an accurate measure of achievement for each year then we can calculate a teacher's value with reasonable precision now those who enjoy statistics can listen to this brief description of the method and others can tune it out so I'm going to give you what the method is but if you don't like statistics you can just put your fingers in your ears okay calculating the value added of teachers I told you the method is more complicated than my Smith Jones example is you regress a student's test score on his or her prior test scores a different outcome by the way it does not have to be test scores it just happens that people usually describe it in terms of test scores and also in their predetermined characteristics such as poverty, race, and gender actually regressing them on the predetermined characteristics often doesn't matter very much as long as you have some prior test scores you take the residuals from this regression and you sum them at the level of the teacher by year by class then you take each sum of residuals for the same teacher but in different years and the prediction from this last regression is the teacher's value added this is the method that is most validated in actual randomized control trials where researchers are able to assign students to teachers and it also produces something called the correct shrinkage which really just means that luck is not going to dominate your estimate of teachers value added Smith always being lucky or Jones always being unlucky okay so I'm going to apply this method to the North Carolina data that I've already mentioned and then having obtained an estimate of each teacher's value added I'm going to do a pretty simple exercise so what I do is I'm going to plot the distribution of teacher value added I'm going to do it differently from math and reading by grade and I left out some grades to keep this chart from getting too busy and I'm going to always show and I made a mistake so it should say grade 3 is in the light blue or cyan then it should be grades 6, 7 and 8 not 5, 6 and 7 so keep in mind it's always 6 through 8 are the middle school grades and they're all plotted in a kind of orangey color and it doesn't matter you can hardly tell the three lines apart anyway so I just plotted them all in sort of the same color spectrum and then I'm also going to plot the distribution of grade 12 so what do we see when we look at this chart the third grade distribution and I should say by the way it's a mechanical fact that teacher value added is always pretty much centered around zero that's part of just we know that Smith may be better than Jones but teacher value added is about their relative ability to teach students skills and get students to learn about you don't want to take the numbers on the bottom are less important than the shapes of the distribution that's how maybe you should think about it and these distributions are smoothed a little bit as my fellow economists will recognize those graphs come from a program that we all like to use a lot ok so if we look at the distribution for third grade teacher value added you'll see it's centered on zero and a lot of the mass or the density build up right around zero and it's a quite narrow distribution so what's the interpretation of this it means that if I get an unusually good third grade teacher that is better than getting one who's unusually bad but the difference between an unusually good teacher and an unusually bad teacher in the third grade is not all that great in terms of the value added that I'm going to get out of her classroom now if you look at grades 6, 7 and 8 in this sort of orangey colors you can see that these distributions are much more spread out they're still centered around zero but the density or the mass is much lower than it is in the light blue distribution for the third graders and they're just more spread out distributions so now if I happen to get unlucky with say a grade 6 teacher or grade 7 teacher I would be way down here ok and if I get lucky I could be way up here so middle school teachers have bigger values than value added than third grade teachers do so your luck in being assigned a good teacher or the bad teacher say the less good teacher as a student matters more and one standard deviation below the mean is just much more meaningful then finally the 12th grade distribution that's the one that's in purple and that shows you that the 12th grade distribution is more spread out than the third grade distribution but it's kind of between the distributions for the third grade and the distributions for grades 6, 7 and 8 this is consistent with some hardening of the students trajectory of cognitive skills and later adolescence as I discussed in yesterday's lecture I would put it this way it's just harder for a 12th grade teacher to change a student's skill much in either a positive or negative direction than it is to change the cognitive skill of a 6th, 7th or 8th grader now one might worry that teachers are assigned to positions in such a way that middle school teachers are simply less similar for some reason in their efficacy for instance if you think about all of these rookie teachers you might say well they're just less similar than teachers who have greater seniority so to test this possibility this next figure which looks just like the previous one so don't worry about that is based on by grade differences and value added for the same teacher so the way to think about it is this if I teach sometimes in the sixth grade and then sometimes I'm teaching in the fourth grade or something like that then I'm only looking within that same teacher so it is not about assignment of some teachers to the sixth grade or some teachers to the fourth grade a teacher has to be moving among the different grades included in this figure and you can see it looks tremendously like the previous one I mean I have to look carefully so I can tell them apart from one another but it is actually making a different use of the data it is looking within teachers so we don't have to worry about that potential assignment problem and in this figure of course what we can see is that the third grade distribution is much more narrow and has a lot of mass or density around zero and the teacher value added distribution for grades six through eight is the most spread out and grade twelve is some place in between so the next two figures I will show them just briefly are for reading this is between different teachers and this is within different teachers so it's just to let you know it's not just all about math reading teachers also differ significantly in their value added across reading now the next figures that I'm going to show you are going to do another exercise and they differ only in the outcome that I'm going to study so for instance I'm going to study outcomes at the end of high school like how well you do on college aptitude test as a student your SAT scores or your ACT scores sorry I should have silenced that before I got started so the next figures are going to show the result of this other exercise that's fairly different and they're just going to differ in which outcome I look at at the end of high school for instance your SAT or ACT scores or whether you enroll in a four-year college and then what I'm going to do is I'm going to say well does it matter when you got the teacher with the high value added for your outcomes at the end of high school and the figures are a little bit complex so it's worthwhile explaining the first one carefully and then once you understand it the others are all sort of the same they just the outcome just changes alright so here's the figure what we have is grades going across the bottom grade 3, 4, 5, 6, 7, 8 and North Carolina for reasons that I don't need to explain there is really no grade 9 testing grades 10 and 11 okay so what I want you to see here is I'm regressing a student's math or ACT scores they're all converted to the SAT scale so don't worry about that on her third grade teacher's value added in math then her fourth grade teacher's value added in math and so on so this first one is saying if I had a teacher who was better than average in value added in the third grade which does that make me have higher SAT or ACT scores at the end of high school okay same exercise but for the fourth grade fifth grade 6, 7 and 8 and then 10 and 11 okay now if a teacher's value added had the same effect on you regardless of the grade in which you encountered that teacher then all the bars would be of the same height okay they're obviously not all of the same height and most notably the ones for grades 7 and 8 are significantly taller than the ones for the other grades the elementary grades but also the two high school grades so it really looks like having a middle school teacher who has high value added is what's going to have the most effect and it's not just that you take the SAT at the end and so high school would matter more than middle school because you can see high school actually matters less so the figure demonstrates that a middle school teacher can change a student's later outcomes more than elementary or high school teachers and this evidence confirms the idea at least to me that students are especially plastic when developing advanced cognitive skills in middle school and having an effective teacher in those grades sets a student on a steeper trajectory with endogenous skill growth ultimately producing substantially higher SAT or ACT scores now I want to note the role of endogenous skill growth here something I emphasized a lot yesterday so let's look at the number 32.6 for 7th grade it does not mean that having a teacher who's sort of plus 1 in terms of a standard deviation raises students scores by SAT scores by 32.6 in any immediate way rather it's having a teacher who's better in the 7th grade allows you to learn a little bit more in the 8th grade and then you learn a little bit more in the 9th grade because you learned a little bit more in the 8th grade and the whole effect of that is the 32.6 points sort of it's like compound interest that's how I like to think about it you can see how it adds up over time so the next figure is similar to the last one except that it's going to look at teachers value added in reading and actually they're value added in reading the effect on your verbal SAT or ACT scores is even bigger than the effect of those the value added in math on math SAT or ACT scores so otherwise it's not very different and again I think endogenous growth is really playing a role here because those numbers are quite big so it's not it couldn't really all happen in one year it sort of needs a few years to have like compound interest to build up ACT and ACT scores are loved by some people and hated by others so there are other ways to measure whether students are taking taking on cognitive tasks that are truly challenging so the next couple of figures I'm going to look at college board advanced placement tests in important subjects because the college board advanced placement tests have the same curriculum and the same test across all the high schools in the United States so I don't have to worry about some grading standards being easier than others for instance this figure is showing that if you have a teacher who's a plus one in terms of standard deviation in seventh grade it raises your probability of taking AP calculus by 6.7% or if you have that teacher in eighth grade it raises your probability of taking calculus AP calculus by 8.8% now obviously they're not taking calculus in seventh and eighth grade it's that later on having had this teacher allows you to end up being more likely to do that there's a very similar result for AP science classes I'm not going to show you the figure it looks somewhat similar to that and they're not taking AP chemistry in the seventh or eighth grade either but if they're having had a better science teacher or a better math teacher in the seventh or eighth grade a teacher's value added in reading this is on taking AP English by the grade in which the teacher taught students and again it's actually bigger than it is for math which I find fascinating for instance having a high value added teacher in the seventh or eighth grade raises the probability that you take AP English probably in your 11th or 12th grade year by a little more than 10% okay so finally oh I have two more finally not finally let's look at GPAs as an outcome this is the effect of teacher value added in reading on your final GPA in high school by the grade in which the teacher taught students so again we see this disproportionate effect of the middle school grades on students final GPAs in high school and the math version is similar but I won't show it to you and then finally I wanted to consider the effect of having a high value added teacher on enrolling in a four year college so and that's because I think four year college is a very authentic and important outcome for many people going to colleges one of the things that changes their lives so this figure is showing you that having a teacher whose one standard deviation better in math in grades six, seven or eight raises your probability of enrolling in a college by 6.3 to 8.1 percentage points and by the way the alternatives to enrolling in a four year college are employment, the military enrolling at a one or two year technical or community college or just plain inactivity which is actually quite common among young people so interestingly encountering a teacher with value added reading of plus one has an even bigger effect than having math of plus one for college enrollment and the effects of seventh and eighth grade reading teachers are more than 16 percentage points more likely to go to a four year college that's an impressive amount again probably due to endogenous skill growth I think the impressive and facts of value added in reading on later outcomes demonstrate that the critical reading skills that we develop through reading and writing are just as if not more important for a person's long term outcomes like college attainment and this is consistent with the fact that an inability to process college level materials or read college level texts is a consistent stumbling block for the many students who regardless of their preferred major find it hard to absorb material and they end up dropping out of college for that reason. Okay now I'm going to switch gears we're going to stop looking at teacher value added and instead we're going to talk about introducing a cognitively more challenging curriculum. Now the state of Texas has a longstanding and fairly high stakes accountability system it's based on students test scores not only do the schools themselves get graded but students need to pass an exit exam in order to get a high school diploma in Texas and also promotion to a higher grade is somewhat dependent on having done well on the tests at the end of that school year although there's more discretion apart from the exit exam the final one. In addition Texas is only one of two states that select the textbooks for school so it has an unusual degree of influence over curriculum in practice though I think that curriculum is more often influenced by tests than by textbooks a teacher can ignore the new textbook that is put into her classroom and just decide they're always giving me new textbooks I don't want to have to learn how to do this with this new material I don't want to have to learn new problem sets I don't want to have to learn new readings I'm just going to ignore the new textbook because I know in two or three years there will be another new textbook that will come down the road but a teacher cannot ignore the skills that are being tested by the exam that her students will take at the end of the school year and typical a testing regime typically lasts at least a decade so the one that we're going to be talking about in Texas has been in place for more than a decade at this point so therefore schools and teachers paid a great deal of attention when they were switched starting in 2011 from the testing program known as the Texas assessment of knowledge and skills I'm just going to call it talks from now on so I don't have to keep saying that long phrase to a new program which is known as the state of Texas assessments of academic readiness or star so from talks is the earlier one and star is the later one now the switch is interesting because making the tests more cognitively challenging was the explicit motivation for the introduction of star to make the tests more cognitive cognitively challenging the creators of star designed questions that require critical thinking more often than the talks tested the star questions were deliberately constructed to make it hard for teachers to teach to the test or coach students in test taking strategies that tend to work well on a multiple choice exam star questions often ask students to provide an answer not just only guess over multiple choices or choose over multiple choices there's there are three short essays on the star exam each in a different format whereas talks had no essay requirement and to assess whether a student truly understands a concept this is a kind of subtle thing the star exams are timed why it's because time pressure makes it hard for a student to simply try out each answer on a multiple choice test but the talks exam you could take as much time as you liked so you could just try out each answer you didn't have to really deeply understand how the problem worked and we'll see an example of that in a moment and star tests also focus on the content that should have been taught in the grading question thereby assessing whether students are making progress vis-a-vis an ever more challenging curriculum and talks tested content that was often way below the grade level of the students and so an 8th grade student could possibly feel like he or she was doing quite well on the test even though he or she was only really answering the questions that were for 4th graders or something like that so to make these I made all these claims about this difference in these tests and it's going to help to examine questions from released prior year tests the Texas State Department of Education releases prior year tests not quite all the questions but you can download them very easily from their website I'm just going to show you a couple here so the first one is this was a tox math question for 7th graders and I'm going to keep showing you 7th grade questions because they seem most relevant so which list of integers is in order from least to greatest is it negative 42, negative 39, negative 4 40, 41 or is it 41, 40, negative 4 39, negative 42 so this is just not a 7th grade level question it's a very easy question that a lot of 4th or 5th graders can get right or here's another example of a tox math question for 7th graders this is really an arithmetic question it has nothing to do with algebra it simply requires you to divide 9 by 3 that's obviously 3 then square it and you get 9 multiply that by 5 you got 45 and add 5 so you come up with 50 as the answer to the question but these are easy math questions for a 7th grader in contrast if you look at this question for 7th graders you can see that this is basically an algebra question it also requires you to understand that this is a Cartesian diagram you have to understand negative and positive numbers 2 different axes x and y you have to understand how to think about slope of a line and you can see that what you have to choose among they are multiple choice but they're 4 different algebraic equations that represent that line it's a really harder question that requires more critical thinking now I told you that I hope we would get to something that's rather funny so I want to show you a tox 7th grade reading question so this is again the older test that's less demanding it focuses on an Austin, Texas festival known as Spamorama that's after that after that canned processed meat that we all know Spam we all know the contest has 2 divisions one for professional chefs and restaurant owners and one for amateur cooks in the amateur division everyone is welcome to show their stuff one contestant entered the contest Spamorama with a dish that was a mixture of cheddar cheese mayonnaise Spam and raisins the dishes poor rating at the contest did not deter this stubborn individual hoping to find a more accepting panel of judges the next year the contestant took the entry and brought it back the following year and keeping with the spirit of the event the judges decided to create a last place even if there were 100 entries award just for him so that's the paragraph that students have to read on the tox test and the question was one contestant in Spamorama froze his food entry because he planned to carve it he missed the entry deadline he wanted it to be eaten cold the reading comprehension question for a seventh grader if you look at stars question it's obviously much harder it's on a more serious subject as well it's about Sainsbury's which is a big grocery store chain in the United Kingdom sort of like the Safeway or A&P of the United Kingdom they're talking about food waste this is a longer article that is about three pages long and relatively complicated so I picked out just one paragraph and I hope you get the basic idea of just the one paragraph would be better to read the whole thing so in spite of Sainsbury's efforts a large volume of waste in other words food waste remains a machine grinds the waste into slushy goo which is then poured into giant silos called anaerobic digester these giant silos act like artificial stomachs inside microbes digest organic waste and produce methane bubbles the same thing happens to organic waste landfill and waste treatment plants the difference is that these anaerobic digester silos are tightly sealed so they can capture and store the methane the resulting biofuel can power vehicles or it can be burned to produce electricity Sainsbury's management estimates that it's ADs those are the anaerobic digesters can produce enough energy to power 2,500 homes for one year or it can make enough electricity to completely remove one of its stores so this is a public power grid so this just requires a higher level of understanding there are actually a few hard vocabulary words like anaerobic or biofuel in the reading example what is the most likely reason that the author wrote this selection so they're trying to get the student to think about what's the structure of the argument what argument is being made and you can read the answers for yourself I think the correct answer is to demonstrate that creativity can help to solve environmental problems but it's undoubtedly significantly harder question than the spanorama question I think we would all agree alright so to test whether the switch from tox to star affected students cognitive skill development I'm going to focus on two cohorts in particular the students who entered the ninth grade in 2011 that was the first year of star the experienced star the heavier more difficult exam throughout high school however this cohort did not experience any star driven more cognitively challenging curriculum in any of their middle school grades the second cohort that interests me is composed of the students who entered fifth grade in 2011 so they experienced the more cognitively challenging curriculum both in middle school and in high school and the difference between these two cohorts is not their experience of the star driven curriculum in high school because they both had that but only the second cohort experienced star driven cognitively challenging curriculum in the middle school so by comparing their later performance on the SAT we can test whether it is crucial to come up against challenging material in middle school unfortunately a simple comparison between these two cohorts is not all that straightforward and that's because the college board undertook a major redesign of the SAT starting in 2017 just with the worst possible timing so not only did the redesign change the scoring but also fear of taking a new test for which they had not prepared cause many students to take the exam in 2016 and when they ordinarily would have taken it in 2017 and in addition of course testing in 2020 was affected by the pandemic so I have done my absolute best to make sure using the concordance tables that everything is in the right scores they are rationalized with one another but I still need to have a control for Texas that did not experience the tox to star transition of curriculum but did experience the same external events like the redesign of the SAT so to find a control I'm going to adopt the method called synthetic controls this method combines all of the possible controls for Texas so in other words all of the possible states that could be controls for Texas to create a synthetic Texas and the weights on these other possible controls are optimized to match Texas's time pattern in SAT scores as well as possible in the pre-star period in other words in the tox period and the weights and the controls are then validated outside of the optimization period so it has to also do well out of the period with which you use to generate the weights the weights have to do well when you go out of sample and see that they can still be valid so that's a very important part of using synthetic controls so this all sounds a little complicated but it's actually very intuitive I love synthetic controls so Texas is a big state with a variety of cities and landscapes and industries some of Texas is a lot like Oklahoma it's neighbor to the north they share oil drilling and farming and some of that sort of thing some of Texas is a lot like Louisiana with similar refineries and ports on the Gulf Coast Texas is a lot like New Mexico they share a desert they share a lot of cattle ranching and so forth and so on Austin Texas has a lot of similarities to other cities that are dominated by a state house and a flagship public university Dallas has some similarities with other financial hubs I could go on and on what's the point here the point is that none of these other states would be a good control for Texas in and of itself but if we weight them all optimally we can create a synthetic Texas that walks talks looks and acts like Texas so we're going to be comparing Texas and synthetic Texas and a properly constructed synthetic control should show how Texas school students would have done in the absence of the switch from the talks based curriculum to the star based curriculum so I'm just going to show you one figure here it's a little bit complicated so first of all this is the period that in talks okay before star was put in and this is the period when the weights are constructed for the synthetic control the synthetic Texas which is in the dashed sort of purple line and Texas itself is in the green line okay so you can see both in the weight construction period and in the validation period before the switch of tests the synthetic Texas is doing a very good job of looking a lot like Texas right they sort of have the same things happening to their students test scores on the SAT the pink line is the last cohort that had no early adolescent grade 5 through 8 exposure to the new star exam and then the red line is the first cohort with full early adolescent exposure to the new star exam what you'll notice is that up until you get to the pink line the two lines the synthetic Texas and Texas track one another very well but then after the introduction of the star exam Texas is doing significantly better than synthetic Texas and that's the idea right that synthetic Texas is telling us what would have happened in Texas if they had not switched if they had not switched tests one thing I wanted to mention about tests like star is that they're a very manageable policy reform of course it's not that easy to construct a good test that's what psychometricians are for many educators have to get involved but it's actually quite cheap to construct a test testing is not expensive by the standards of school finance in the United States finally I told you this story early in the talk and I'll be very brief here to stay on time about charter schools and one set of facts that motivated me was the strong performance of charter schools that started grade 8 as their entry year and used grade started grade 5 as their entry year and used grade 8 as their exit year and these school students often made annual test gains that were 2-3 times the gains of schools with kindergarten entry or grade 9 entry and at the time as I said these results struck me as very counterintuitive because I expected that the schools that started with kindergarten would produce the most positive gains in student achievement I really had low expectations for schools that took in the negatively selected applicants at grade 5 because I thought of them as students who had missed the boat of strong early childhood education and were already struggling in the regular public schools so this was all based on my idea which was then popular in economics and continues to be very popular among economists that children are only very plastic in early childhood so to my mind the struggling 5th graders had already been fated to have a lower level of cognitive skills so my study used more than 100 New York City charter schools and it used lottery based methods I mentioned these before they're very simple students applied to one or more charter schools simply by filling out a one pager with their name, their address their parents contact information and so on it's called an application but it's really just basic information and once the charter school has received all the applications it holds a lottery in which every child has assigned a lottery number so for instance if they want to admit 60 kindergarteners the first 60 kids get a seat in the charter school and if you were below the cut off for your lottery number you would typically not get a seat there's a bit more to it than that because the lotteried in and the lotteried out students only want to use that lottery information to come up with estimates we don't want to use students who might have been picked off a waiting list or something like that so there are some bells and whistles but I won't go into the details this method is generally considered to be the gold standard because it's not easy to evaluate charter schools and lottery based studies tend to have similar findings across urban settings where the application lotteries are over subscribed so it's not as easy in rural areas because you might just not have too many students apply so this next figure shows you the effects of a year in charter school on math scores it doesn't really matter whether you look at the blue bars or the orange bars that's something about attrition in New York City charter schools I used all of the lottery based studies that I could find where it was clear what grade was being shown for achievement for instance the KIPP schools are charter management organization that ordinarily starts charter middle schools with the fifth grade and so they're covered by a lot of those middle bars and you can see that their kids are just learning at a faster rate in the middle schools the middle charter schools than they are learning in the high schools the elementary schools and for New York City where I can do it I know long term outcomes and it is also true that if you look at long term outcomes like post secondary attainment whether you go to college or not the effects are significantly bigger for students who were lotteried in in the fifth grade in fact the effects for them are almost four times as large as they are for students who were lotteried in in kindergarten at the beginning of this lecture I made the case that early adolescents early adolescents are relatively neglected they have a lot less spent on their instruction they're in larger classes they have teachers who are rookies and they have teachers who are eager to stop teaching them and start instructing older younger children although this relative neglect is unintentional in my opinion it is perilous because adolescents is a crucial period of opportunity and also a period of risk owing to the plasticity of advanced cognition at those ages and the gradual hardening of children's cognitive trajectories thereafter now neuroscience and neuropsychology were behind much of my motivation to test for exceptional plasticity in adolescents and I believe that neuroscience and neuropsychology have a fundamental quality that makes them different than the kind of evidence that I've shown you today from education they should really apply across the whole world it shouldn't be the case that neuroscience for American we just have different brains of people in other countries in fact I'm going to be doing some work in the future on the UK and I would be horrified if I found out that they had completely different brain development than we did so that's the really nice thing about neuroscience is that it's important for learning universal truths about cognition in my opinion what I like about the education experiments is that these natural or policy experiments that I described today are all realistic interventions that are deployed in some schools already and could be deployed in more schools for instance concentrating the most effective teachers in early adolescent education or higher pay and other rewards that induce effective teachers to apply to and stay in middle school classrooms more generally the importance of early adolescents makes me optimistic that the future of American education and Americans advanced cognitive skill growth if you think about it the idea that only very early childhood education is matters is somewhat disheartening I would be the first to agree and neuroscience would support the idea that the period from three months before you're born to age two or three is crucial for brain development and later life skills but it is disheartening to think that kindergartners might already be faded to a trajectory of low modest or high cognitive skills I just find it disparaging to think of writing off a child who is early in elementary school yet this is the implication if we focus exclusively on early childhood spur to endogenous skill growth to the detriment of adolescents and its important effects on advanced cognitive skills which are so important to our economy and our today's fast paced world and I know that Eric is going to be discussing some of the implications of this for our economy and just the larger picture so the last two bullet points are especially aimed at Eric I find hope in early adolescence precisely because it is a later period when society can have more influence over a child's educational experience no child masters abstract math or rigorous critical reasoning before early adolescence and I find this to be a good thing because it means that children enter this quest for advanced cognitive skills on a relatively even playing field I didn't say it wasn't even playing field but it might be more even than we suppose when I look out over the United States and other highly developed countries what tends to worry me the most is economic fatalism among young people who lack advanced cognitive skills and therefore feel that they face a bleak future owing to technology and globalization we need service workers who can perform important tasks that are challenging on dimensions other than cognition like strength or dexterity or nurturing or courage however the demand for these tasks is declining in highly developed economies while the demand for advanced cognitive skills is rising I am convinced though that many more children could be sat on a positive high growth trajectory towards advanced cognitive skills during the plastic period of their adolescence not all children will attain these advanced skills but the more we see adolescence as an age of opportunity the more likely we are to offer these children opportunities rather than neglect them thank you we're going to break for 10 minutes and come back and I'll introduce the commentators and following that we'll have a discussion thank you very much please join me in thanking Professor Hoxby again for her wonderful lecture our first commentator Eric Hurst joins us from the University of Chicago the faculty of which he joined in 1999 and where he is the Frank P and Marion R. D'Arcy Distinguished Service Professor of Economics and the John E. Jerk Faculty Fellow at the Booth School of Business Professor Hurst's luminous and voluminous scholarship focuses on housing markets labour markets and household financial behaviour thematically and methodologically ranges far and wide and over some of the most vexing problems of our day the welfare losses to society of racial discrimination and urban gentrification the economics of the Covid pandemic leisure inequality and the correlates and causes of the persistent if narrowing gender wage gap among the college educated the recipient of multiple research awards Professor Hurst has also found time to be a remarkable teacher earning no fewer than two not three but four teaching awards in his 20 odd years at the University of Chicago please join me now and extending a very warm welcome to Professor Hurst Thank you so much it is great to be in person with people that is not my children and it is wonderful and so as Caroline kind of foreshadowed I'm going to do two things over my two days so tomorrow I'm going to talk a little bit about the role of advanced cognitive skills versus just completed years of schooling is there something about specific skills versus just getting a degree that shows up in labor market outcomes so that's going to be a little bit tomorrow but today I wanted to set the stage and try to show why the importance of accumulating skills in general are becoming more and more important for the labor market that we live in today and so I want to do four things today in my comments the first of which is I'm going to show you some patterns about inequality along a variety of dimensions and how they've been evolving over time in the United States particularly starting in the early 1980s second I'm going to talk a little bit about potential drivers of these changes in inequality because again if we want to think about a solution we might want to think about what might the causes of be the third thing I'm going to try to do is then I'm going to set up and say well how does economies adjust to changing in demand for certain types of labor and you'll see why that's going to be important in the fourth part of my comments where I'm going to link it back to some of Caroline's talks over the last two days so that's kind of the big picture overview the component of where I'm going to link to Caroline's lectures it's going to come at the end but I'm going to try to create an umbrella where you can see the importance of this research for some of what I think are the challenges that we are facing in a socioeconomic way in the 21st century so let me just kind of start by a picture that you might be familiar with this is going to be a measure of income inequality in the United States it's the share of all income earned accruing to the top 10% of the population and so the United States in particular is seeing an increase in the share of income accruing to the top 10% whereas in current periods we're talking about almost half of all income is owned by the top 10% of the US distribution so now I'm going to focus a little bit more on breaking things out by different education groups and so today I'm going to show you a lot of patterns by people who have a bachelor's degree or more versus that people who have less than a bachelor's degree and just with that as background and again when I teach my students this it is often striking to them but just as a reminder that we are rare in the population in terms of having a bachelor's degree and so this shows the share of I'm showing you men right now I'll talk about women as we go through but the share of men 25 to 34 who have completed at least a bachelor's degree by the time they are in that age range and the couple things I want you to focus your eye towards is first the number that we're looking at is somewhere between 25 and 30% so most people in the United States don't have a bachelor's degree and again through the worlds we live in we tend to focus on our reflection and I just like us to remind us that our reflection is not the average by any dimension but the second more important thing I want you to show you is that the share for men for women you've seen much more steeper increase but the share for men has been relatively stagnant over multiple decades and you see a little bit we talked a little bit about this in passing yesterday about coming out of the Vietnam and the GI bill you see a little increase but basically from cohorts over the last few decades it's been relatively flat a little uptake recently so we have a group where the returns to skill is going up but yet the movement into skill at least by at least getting a bachelor's degree have been relatively stagnant until potentially recently and so this is what I focus some of my work on is trying to figure out the difference in behavior in the labor market between those with a bachelor's degree and those with less and then trying to put on a little bit more what particular skills are necessary but this is just things that you might have seen in the newspaper which is just the average earnings of someone with a bachelor's degree relative to someone without a bachelor's degree and how that's evolved over time so a number like 30% means that on average a typical person with a bachelor's degree is earning 30% more in a given year than someone a typical person without a bachelor's degree but the key thing I want you to focus on is notice that has been increasing over time so part of the inequality we're seeing is showing up in completed years of schooling where the earnings are growing for those with a bachelor's degree at a greater rate than those without a bachelor's degree and that brings me to this next part which again we're starting to get into socioeconomic issues that are kind of tangible the things that we're feeling I'm showing you here now the employment rate of men and again I'm focused on men 25 to 54 here without a bachelor's degree and so there's a couple things your eyes should be drawn to there's ups and downs over time those are due to things like recessions so when we're recession less people are working you can see in 2008 there was a lot less people working and in 1984 there was a lot less people working but I want you to focus your eyes on the downward trends when the things are peaking those peaks are getting lower and lower over time and so we're at a position where currently about 82% of men this is pre-COVID so I stopped it right pre-COVID 82% of men in these prime working years post you know college in getting pre-retirement getting are working where that number used to be something like you know around almost 90% so we get this downward drift in employment for a group of individuals with less than a bachelor's degree and if we want to compare them to the men with the bachelor's you just don't see that same downward trend you see slight trends but it used to be about 94 and now we're something like 93 but really a concentration at the bottom of the distribution the bulk of the distribution of employment rates employment rates that are falling over time and I promised some depressants stuff when we were talking yesterday was pretty depressing I'm going to go there right now with this picture this one is the one that that I find stark and I pause every time I'm giving in some of my own work or when I'm teaching this is the share of people again men 25 to 54 without a bachelor's degree who report working for weeks during the prior year so we have lots of government surveys that samples people working and they call them up on the phone do you have a job they say no they said did you work at all in the last year they say no and so this is the share who report no they're not working at all over the prior 52 weeks and what I want you to see is that historically it was about one out of 20 men in this age for a whole year that number is now one out of seven so there's a large chunk of our prime age population again men you see similar patterns for women not so much in terms of the increase but men with a you know less than a bachelor's degree sitting idle for long periods of time and that has changed relative to where we were you know 30 40 years ago okay so we have these shifts inequalities rising it's showing up in labor market outcomes in particular in the propensity to work what are driving some of these shifts and how do we kind of interpret it and so there's two stories and in Caroline touched on yesterday I'm gonna show you a little bit of evidence of them you know things you hear about in the newspaper such as automation and globalization so how do we think about changes in automation and how and changes in globalization and affecting these patterns that are shifting inequality over time okay I want to focus on one industry and I do a lot of work just thinking about this industry and you'll see why why it's so important but I'm gonna show you stuff about the manufacturing industry and how that's changed in the United States over time and so what I'm showing you here is just if we count the number of manufacturing jobs in the United States and how that has evolved over time and so in 1980 ish there was almost 20 million about 18 19 million manufacturing jobs in the United States so people who report working in the manufacturing industry and you can see that has been declining over time with an acceleration of that decline in the 2000s and so about 6 million manufacturing jobs disappeared from the United States between 2000 and 2010 and there's been essentially no rebound in that period afterwards so what are the two stories that people have that link to these decline in manufacturing jobs one is openness to trade and so there is a bunch of our colleagues who have worked in economics on a variety of kind of angles showing that opening up to trade and particularly trade with China increase the competitive forces in the manufacturing sector and manufacturing employment as a result increased but that's not all the story and I will say it's not even probably the predominant story because during the same time period manufacturing output went up so how are we getting more output with less workers we've been switching our manufacturing production from being labor intensive to being capital intensive or automation intensive and so that shift that has been incurring has been reducing not the demand for manufactured products produced in the United States that has changed some with trade but it has resulted in a decline in the manufacturing workers in the United States shifting towards machines and so this decline in manufacturing does not hit all schooling groups equally so let me show you some data now on the share of workers of different schooling levels and what percentage of those worked in manufacturing in given years so how do you read this picture it says if you are an individual with a high school degree or less in 1977 essentially one third of you worked in the manufacturing sector not only those who are working but across all individuals in the manufacturing sector for those with less than a high school degree nationally not just in Detroit nationally was about one third that number now is somewhere around 12% so we have this big sector that was important for workers particularly with a given skill level and that sector has shrunk and more importantly not only is it shrunk conditional on working in that sector they don't need as much low skilled workers low educated workers as they do they need people now to fix robots as opposed to work the line and so I'm going to show you lots of pictures like this more today tomorrow than today but I'm going to show you a couple today and I'm going to focus on a region that Caroline focused on a lot in her picture so if you take a remember from her lecture earlier today and from yesterday where these places where these advanced cognitive skills are kind of missing spatially in the United States those are also the same places for which manufacturing intensity is high it's not exactly the same and I'll show you some pictures and more tomorrow than again today but each one of these areas basically show the intensity of manufacturing production in that county in 2000 so the darker the red the more manufacturing intensive that area is this should not be surprising to any of you manufacturing is clustered in the United States around kind of the mid section and again just as a reminder this is Caroline picture she had it in pretty blue I don't know why I put it in black and white but you can see the places that are missing some of these advanced cognitive skills are exactly the exactly the same places now in some of my work I've actually shown that the places where manufacturing declined the most or where manufacturing was most present were also the same places that manufacturing declined the most are exactly the same places where non-employment rates have risen the most and so you see this area where manufacturing has shrunk in certain areas and those are the same areas where now male participation particularly concentrated for those without a bachelor's degree have fallen the most within the United States between these aggregate forces of automation which has particularly affected the manufacturing affects other sectors too but affects the manufacturing industry and that has caused you know these employment declines that we've seen in the aggregate data so I'm going to do one more picture before I get to some of you know how do we adjust to these shocks but this is kind of something that Caroline alluded to towards the end part of our discussion so what this is now is take a look think about every job in the U.S. economy okay filled with some people and if you go to those jobs some of us are teachers and some of us are professors and some of us are lawyers and some of us are shop clerks and some of us are manufacturers and then for each one of these jobs there's some skills that are needed and so there's government surveys that says if you are a sales clerk what type of tasks do you tend to do on a regular basis and then we could take those task measures and project them on some I'm going to use these words like advanced cognitive tasks those are kind of you know using Caroline's language or we could have things that are kind of routine tasks and so advanced cognitive tasks are things that are going to require a lot of thinking as part of our jobs complex problem solving so things like lawyers and professors tend to be higher in this type of task need than bank tellers or sales clerk and then there's some other types of jobs that again have these routine things that you do the same thing kind of in a repetitive way so my first job back in high school my first job I worked under the table at a golf course but I'm going to forget about that I'm just saying that might have got rid of my chance of being the Fed chair for saying that okay but my second job was working at Wendy's and so there you know you just kind of worked the cash register it was a very routine type of or you know there was not much variation I'd come up I hit a button and people would give me money I handed it back what this is showing though is that the demand in the US economy for skills that require complex and analytical tasks is rising over time in the jobs that require these routine tasks that are kind of repetitive you know manufacturing is a lot in those green lines is also declining over time so the skill mix for which the labor market is requiring has been shifting in a given particular way favoring these advanced cognitive skills that Caroline was talking about so this brings me now to kind of a bigger question which is how do economies adjust to the demands for you know occupations shifting over time what happens when some sectors go away that require some skills in new sectors are born that are require others and so you know I like to you know what I'm teaching we've gone through lots of disruptive periods in an economy over our you know centuries of existence and so if we go back a hundred years ago most of us were in the agriculture sector and in particular you know about one third of all men worked in agriculture in the beginning of 1900's and then a robot came along we call that robot a tractor but it was some automation nonetheless and transformed this manufacturing this agricultural sector but yet the U.S. economy weathered that disruptive shock right now there's very few of us that work in the agriculture sector relative to where we were before but you know that was true in 1950 60 70 but lots of people were still working so how do we respond when sectors disappear and other sectors grow and there's many kinds of margins of adjustment that we could think about and some of which are we could if jobs are in one place like in agriculture they tend to be in you know more rural areas in manufacturing tended to be in you know different rural areas or sometimes people have to move to the job so moving spatially is one way people could adjust for shocks but often times the sectors that are growing and the sectors that are shrinking have different skill mixes between them now it's not always the case because when we used to be in agriculture the skill tasks that people needed in agriculture translated roughly well to the skill tasks that were needed in manufacture my dad was a manufacturer my guess is if he was born 40 years before he would have been a farmer the skill mix that my father would have had translated nicely between those two sectors but now we're in periods of time when manufacturing is declining in the skills that the jobs that are growing are requiring these advanced cognitive skills and as a result we need some other type of adjustment to occur to fill the demands of the labor market that is moving today and that means acquiring some new skills and we could do that by potentially going to college or training programs or other forms of human capital adjustment and the differences those adjustments take time and so now again this is just the umbrella to get to this last part which is you know I think Caroline's work is so important because it's highlighting how we can think about one margin again we'll probably talk later on there's lots of other margins that we might want to think about one margin of how we can start moving the economy to help get the skills that we're going to be needing in the labor market today relative to 20 years ago or 30 years ago and so Caroline could have given the same lecture 20 or 30 years ago that where are the most elastic for skill equity I'm going to say that adjustment is even more important to think about getting it right today than it was in the past given the demands of the shifts in the demand for different skills that have been accumulating and so thinking about this is providing a way to basically say we used to focus a lot of our narrative discussion and Caroline emphasized this that we want to target people early so the Head Start program a lot of the government programs and we say maybe there are better places at least additional places where we want to focus some of our energy and so I think that's why the important so tomorrow I'll have some more specific thoughts about what skills versus with others the umbrella sense of why is this important because these big shifts that the economy is hitting we need to think about ways if there are frictions to skill acquisition what can we do as a society to help mitigate some of those frictions and so that's what my comments are on I look forward to more discussion that's all I got thank you very much Eric next it gives me great pleasure to introduce our next commentator Professor Sylvia Bungay of Berkeley's own Department of Psychology and the Helen Wills Neuroscience Institute and excuse me an expert in the cognitive and neural processes that support reasoning memory and goal directed behavior in humans particularly in those young humans we call children Professor Bungay directs Berkeley's building blocks of cognition laboratory her lab draws several disciplines including cognitive neuroscience developmental psychology and education research to study developmental changes and neural plasticity in the cognitive control and reasoning skills of children and adults and author or co-author of some 196 papers and I believe a brand new textbook on the fundamentals of developmental cognitive neuroscience is that out not out yet almost what's it called supply supply chain issues stuck in a boat in the bay that happened to my textbook a few months ago too she's one of 11 members of the multidisciplinary multi-university national scientific council on the developing child which strives to generate public will to close the gap between what we know and what we do to promote successful learning adaptive behavior and sound physical and mental health for all young children please join me in giving a warm welcome to Professor Bungay thank you Rebecca I'm truly honored to be here and I really want to thank the Tanner Lecture committee the ones I've met so far Jay and Kinch and Hannah and everybody else for including me I also want to thank Chancellor Christ who was here yesterday and Jane Fink for organizing this really fascinating lecture series and of course I want to congratulate Professor Hawksby Caroline on two really compelling inspiring lectures on a really important problem so today I want to begin by providing some additional context on the advanced cognitive skills that Caroline has been talking about then I'm going to talk about periods of brain plasticity during development these topics are really right up my alley as Rebecca was alluding to so Caroline defined advanced cognitive skills as those that require higher order reasoning they require a capacity to solve problems through logic think in the abstract, engage in critical thinking and derive general principles from a set of facts they are integrative, they are synthesizing they are important for planning end quote so higher order reasoning which is something I've studied for many years is the culmination of a number of mid-level cognitive abilities so for one working memory or the ability to keep information in mind and manipulate it interference suppression, the ability to ignore distractors focus on what's currently relevant and relational thinking which is not as widely studied but something that's really near and dear to my heart which is the ability to jointly consider several pieces of information that is to compare them or integrate them and this information could be premises it could be rules, it could be concepts anything like that so trying to build these skills is critical for several reasons as we've already talked about but one of them is that many students who are in school today are going to have careers that don't even exist yet drawing on discoveries that have not yet been made so we can't impart all the knowledge that they will require and secondly students today and raised in the information age they have a wealth of information at their fingertips what they need more than anything else is to develop the skills that are required to select relevant information evaluate it, integrate it and apply it in other words they must tap into what we think of as domain general cognitive skills that you can port from one situation to another we're going to come back to the development of these skills a little bit later after a brief overview of brain development and plasticity and I will say that tomorrow I'll talk about the fact that it's contentious as to whether we even have these domain general skills but coming from neuroscience I think we have evidence that we in fact do so as Caroline has already beautifully covered we see dramatic brain development in the early years this is from seven and a half weeks gestation months gestation through six years these are horizontal slices through the brain created with MRI and the growth here is obvious even to the naked eye so and this message of early brain development sometimes called the birth to three or the birth to five period has been used effectively as a rationale for increased investment in preschool and kindergarten in this country so as Caroline was talking about Head Start, the Episcidarian the Perry all of these programs but as we'll see in a moment beyond this dramatic growth in what's called gross anatomy, not because it's gross just because it's broad is we see that more detailed measures reveal that the brain is not fully developed at age five or six or even 10, 15, 20 it's still being sculpted by experience long after that so Caroline's message that we need to invest more in middle school education was really music to my ears because I've been making the same argument about neuroscience for many years that we cannot let our efforts to promote early child development and child brain development completely eclipse our ability to do so for later child and adolescent development so in what ways does the brain change after early childhood I'm going to expand a little bit here on Caroline's excellent summary but if you take nothing from this then just remember what Caroline said okay so here we have a 7 year old and a 30 year old and it's hard to see the differences between them aside from the fact that this particular 7 year old happens to have a larger head and sort of more misshapen but if you look more closely you see that the grey matter is a little bit disproportionately bigger in the 7 year old and the grey matter is where the neurons are concentrated along with the local connections among them or synapses as Caroline talked about now the brain forms well over 100 trillion synapses over the first few years of life and as Caroline showed it peaks around age 2 and then it drops drops drops through childhood and adolescence into adulthood and so it's eliminating the synapses that aren't being used that just aren't as relevant and this pruning of these excess connections is sculpting the brain in a way that allows a child's brain to adapt to its particular environment and this is absolutely critical this is why we have this protracted period is that it allows us to basically adapt in a way that humans are extremely adaptable to a wide variety of environments for better or for worse so importantly I want to make this point that although later childhood and adolescence are characterized more on balance by a loss of synapses in fact synapses are being created and removed, eliminated all the time as you're sitting here listening to this lecture it's just that more in general are being removed than created so there is plasticity okay so where is the 7 year old oh sorry so my analogy here is to Michelangelo who would take the slabs of marble and sculpt them into these beautiful things by taking away rather than adding because the 7 year old has more gray matter the 30 year old has more white matter which is made up of these long range tracts these white matter tracts that connect distant parts of the brain that are necessary for basically anything higher order any kind of higher order cognitive function requires these distant brain networks okay and so these are, it's called white matter because these long range tracts, the fibers are wrapped in this white fatty tissue called myelin that as Caroline mentioned promotes efficient communication between areas and so my analogy here is to the building of the Golden Gate Bridge in the mid 1930s and so here we have the structure the white matter tracts are growing and then in childhood and adolescence these tracts are being reinforced there's more myelin wrapping around these fibers and what that's doing is to support network traffic communication between these distant brain regions and then over childhood and adolescence we see that this traffic increases unfortunately for us that's kind of the situation we have today right here and that's already just one year after the bridge was built okay so we know though that the brain is not developing in a vacuum right we know that it's sculpted by experience this is a phenomenon called experience dependent brain plasticity as the author Will Durant once said although it's been misattributed to Aristotle we are what we repeatedly do and a question that people often ask is when in development is the brain most malleable that is when is it most sensitive to experience and the answer as Caroline mentioned is that it depends on the particular neurocognitive system that we're talking about so the brain networks that underlie basic senses like vision and hearing as well as basic emotional capacities like attachment get wired up in the first couple of years of life and they're very hard to change after that so for example if we have a child who's born with cataracts in both eyes and is not getting pattern visual light they'll be functionally blind if they aren't operated on because this whole circuit here that's going from the eyes all the way to the back of the brain the visual cortex isn't wired up properly and so ophthalmologists say that the operation should happen early ideally within the first eight weeks of life but certainly within the first couple of years after that that's a critical period after which it's really really hard to change anything you need the specific environmental inputs to lay down that brain architecture so to draw a common analogy once you've laid the basic foundation for building a house the foundation and the plumbing it's really hard to change so by comparison with these lower level skills we can think about something higher level like language this period of plasticity is not as early and it's not as bright of a line in the sand in fact for language there's a series of windows of plasticity that we know about from humans as well as from other model systems including song birds believe it or not but anyway so there are these multiple sensitive periods and one of the earlier ones is actually learning to pronounce words in another language as though you were a native speaker and if you're not exposed to that language within the first six years of life or something it's going to be very very hard after not impossible very hard on the other hand thankfully for those of us who are trying to learn other languages vocabulary and grammar you can acquire throughout your life it's just going to get harder than it is early on okay and so here you know once you've laid the foundation of a house you can still make changes to the building you can add or remove a wall or something but expanding the footprint or changing the plumbing is more difficult okay so what about advanced cognitive skills like reasoning and the academic capabilities that it supports these skills depend heavily on a core set of brain regions here the frontal lobes on the front of the brain that Caroline talked about as well as the parietal lobes that's sort of a loyal henchman interacting a lot with the frontal lobes and this is called the frontal parietal network okay and it's basically a swiss army knife it is helpful for any kind of cognitive tasks that you throw at it and so different parts slightly different parts overlapping parts of it have been associated with these different mid-level cognitive skills that I've talked about like working memory or relational thinking so when does this brain network mature and just how malleable is it to address this this question I want to show you some behavioral and brain imaging data on a task called the matrix reasoning task here so this is a task that's commonly found on IQ tests and what you need to be able to do is to solve this puzzle by identifying the missing piece and to do that of course you have to look at the relations among the items in both of these directions, dimensions here and it turns out that the correct answer is this one here but as you can see there are a lot of distractors and it's easiest to solve if you just kind of look for the correct answer so this involves relational thinking as well as the other skills I mentioned so this task is designed purposely so as not to rely as much on background knowledge so it's supposed to be more culturally fair and yet it's one of the strongest cognitive predictors that we have of future scholastic achievement and performance in challenging careers and so for example as I mentioned the ability to integrate relations between numbers or variables or concepts is essential for math so just to take math as an example if you think about fractions comparing fractions for example or you think about graphs as Caroline was mentioning relating the x and y axes or you think about equations or sets of equations and algebra that all involves relational thinking okay so now I'm going to show you performance over ages 6 to 21 on this task and so these are the reasoning scores up here as a function of age from 6 to 21 and what you can notice several things right one is this beautiful increase in performance on this task and there's an inflection point around age 12 and then it starts to plateau but there's also massive variability I mean you can find a 7 year old who's you know performing better than an 18 year old for example just massive variability that we want to be able to understand I should mention I should back up and say that each point is one child and then each line is that child tested at two time points often around a year and a half apart these are data that were collected by my lab and several others it's over 500 participants and if you imagine the brain scanning and all the cognitive testing that we do takes like you know 6 hours per participant so it's a huge amount of work to get here but I digress okay so let's see if there's something else I wanted to say alright the other thing I wanted to say is we talked yesterday about motivation to perform a task right that's going to differ on some day somebody might be more motivated than another day or something along those lines sure that's definitely going to contribute here nonetheless we see these clear trends and nonetheless it is a good predictor of academic outcomes okay so some some researchers have emphasized the high heritability of reasoning skills right that genes can explain some of this variance here and indeed if you take a bunch of children who are from homogeneous higher income backgrounds you see that what differentiates them the most is actually genetic but then if you look across a wide range of backgrounds you see that actually environment matters a ton right and so performance on this task does vary by socio-economic status it varies by number of years that you have attended school and there's a strong environmental in fact it's even increased over the decades actually as people's jobs have required more and more higher cognitive skills okay so in parallel with these findings we find that this frontal pridal network gets stronger throughout childhood and to a lesser extent in adolescence over among six to twelve year olds where there's the most dramatic increase in performance on this task we find that among the six to twelve year olds those who have stronger white matter connections actually they do better on this task not only that they also show increased growth in their trajectory of reasoning development over time and so it's laying down the scaffold literally seems to be supporting cognitive development okay so to conclude a brain system that supports higher level cognition is still under construction during early adolescence and is thought to be malleable and I'm going to talk quite a bit more about that part tomorrow and given what we know so far from neuroscience and from Caroline's compelling evidence it's up to us really to ensure that students are properly challenged at this critical junction so tomorrow I'll touch on some of the challenges and future directions and I hope you'll come back to the question so thank you so much to you all thank you very much Professor Vangay and we can move to questions now ah so response pardon me Professor Hoxby yeah would you mind responding I want to make sure people have a chance to ask questions but I want to thank both Eric and Sylvia for fantastic commentary and I agree with so much of it I'm having a hard time coming up with what I should say except thank you thank you thank you and thank you again let me just say something briefly about Eric's comments I think what he did was provide this really great overarching framework for you to think about why is it that certain types of jobs are in decline and other types of jobs other the jobs that have the routine skills tend to be in decline and the jobs that have advanced cognitive skills tend to be rising in terms of labor demand and then the something that I liked especially was thinking about not just the drivers what has happened automation which has changed manufacturing and changed the need for certain types of skills like for instance strength but also how the economy does adjust to shocks it's not as though we have the same economy that we had in 1910 I think that's a really important point but we're doing different types of adjustment now and we can see that our economy is trying to get us to adjust right that rising premium for people who go to college or university is it's the economy saying go go to college or university learn these skills you need to learn the sort of skills that people pick up in college or university and we're just not adjusting as fast as the economy wants us to adjust which is one of the reasons why we have a rising wage premium for people in college and I should emphasize an interesting thing about colleges in the United States is that almost all of the growth in college education has been at colleges that are not very selective so they're not the Berkeley's of this world or the Stanford's or the Princeton's or something else they are increasingly online schools, for profit schools which are having a lot of growth and they're just not producing the same quality of skills so in some ways that skill premium that we see going up or that college premium that we see going up really understates the true premium for having those more advanced skills because a lot of colleges are just not producing them very well anymore so I think if anything it understates the pressures and why is it that we have such frictions to skill acquisition for advanced cognitive skills but we didn't seem to have as many frictions to changing someone from an agricultural worker to a manufacturing worker I think in fact it is because these are very very different sorts of tasks right they just reward different types of things being strong being hardy being able to work maybe in the cold is very useful in both agriculture and manufacturing but it's not all that helpful if you're a lawyer sitting in an office in downtown San Francisco or something like that I also wanted to say a few words about Sylvia's comments which I find just incredibly helpful because she can think about these types of neuroscientific things in a way that's much more precise than I can and thinking about what it is that we do when we're using our cognitive skills it is about holding information in your head it's about integrating information it's about this sort of relational thinking selecting the information that you're going to decide is important and I think that all of these things are unusual in certain jobs but very usual in other jobs and that's so that's important I also I'm glad to hear that our synapses just going strong we can keep developing them over time so we don't stop all together but if you're not being put under pressure to develop new cognitive skills you're not going to develop them as much as if you're under new pressure to develop cognitive skills so I think that's a really important thing and I love the phrase pruning is sculpting your mind because that's really the way to think about it that's fantastic also experienced dependent brain plasticity I think is a very important concept that she talked about because it's this idea that your brain is plastic but if you don't really use it you're not going to have it be plastic in the ways that you want it to be plastic and that is why I think there are a lot of frictions to changing people from manufacturing non-college types of jobs routine types of jobs to jobs that require them to show these integrative and other types of skills because they're just not getting enough experience in the areas that they would need to have the right plasticity in those directions so thank you very much both of you this was just fantastic I learned a lot I hope everyone else learned a lot and I think we can take questions and do ask questions of Sylvia and Eric too well thank you very much for these enlightening lectures yesterday and today so I'm wondering do you have feedback from teachers and policymakers on that and if so what kind of feedback do you get from all this yes so I'm this sort of person who talks to policymakers quite a lot I talk to policymakers both at the federal level like the US Department of Education but also at State Departments of Education which in practice are almost more important because K-12 education and higher education are more controlled by the states in the United States than they are controlled by the federal government that's a kind of surprising thing higher education is a little different than K-12 but so yes what you need to do is get their attention and do it with simple studies that they can understand well enough to say maybe I should try this I think the fact that we do have 50 states is actually a great benefit because the state of California can decide we're going to go all in early childhood education and some other state can decide we need to put more emphasis on adolescence and then you have a little experiment going on to see which policy works better I think the states are actually pretty good about experimenting if you can explain why it is that your study makes sense I mean I think the reason why we have so much early childhood emphasis at this point is precisely that some researchers got to state policy makers and city policy makers school superintendents and things like that and got them convinced that this was really important so they are changeable for sure and I think also one of the things that I think is that they're often more impressed by things like neuroscience than they are impressed by us economists because you show better pictures I think you show better pictures so I think that also may inspire more confidence in them that this is about science this is not just about economist opinions about the labor market they tend to think of us as opinionated but not necessarily as scientific as you and yet it's so hard for us to actually get the funding to pull off these interventions they're really hard to do and really expensive and they're so small scale as I'll talk about more tomorrow so it's really frustrating but at the same time it's just so nice to have this complementary evidence from both of your fields that really are on a much bigger scale this is poorly formulated but I just wanted to ask Caroline to say a little bit about something that Sylvia mentioned which is heritability and just ask you about how you think of that as interacting and how you might address or need to address the question of heritability as we look at your map of Appalachia say and we think of a fair bit of geographical immobility there it seems like questions of heritability come to the fore and I just wonder if you've thought about it so heritability is a as you can imagine it's a somewhat controversial thing it's hard to often parse the evidence between what is explained by something I inherited abilities or what I've picked up along the way as an experience let's see certainly it is the case that you see that there are some areas of the United States where there's very little mobility either mobility in terms of geographic mobility people are not maybe you should be moving out and they're not because jobs are decreasing there but also just a lack of economic mobility that families have been poor for a very long time and don't seem to be changing very much now that being said I think that the role of heritability because I think you were talking about genetic heritability genetic heritability I think is quite overrated for a couple of different reasons first of all when we look at interesting studies of adoptees who get assigned essentially randomly to different sets of parents in the United States sort of it's a first come first serve system on some adoption systems you see the adoptees do not look that different from the from the children who are the biological children of the parents who adopt them so they tend to not score exactly the same way they do not do exactly the same thing in school but they're not that different and they're not genetically related to the people who adopted them so experience must account for quite a lot of things the other point that I wanted to make about this is and I think I mentioned this briefly in passing yesterday that economists like me are teaching undergraduates I understand where the undergraduates are coming from but the first time we show them a regression that tries to explain an adult's outcomes say an adult's earnings or an adult's educational attainment or things about whether they stay married or something like that family stability it turns out that our regressions no matter what we throw at the regression and that includes a lot of measures of aptitude measures of what your parents were like your parents education what do you live, what's your race and ethnicity what's your sex at birth or your gender depending on the survey we never really get r squared above 15% so that means 85% of an adult's outcomes we can't explain even after we've thrown all of these variables at it now my undergraduates always dislike this fact they find it very frustrating because they want to know how to fix the world and essentially what we're telling them when we say the regression has an r squared can explain 15% of the data is they feel very frustrated they're like well wait how do I fix everything I want to be able to fix everything and I say no no no it means it's a good thing because it means 85% of who you are is an individual and we can't explain you by just putting all of these variables in the regression I don't really want to live in a world where everyone can be explained by knowing a few things about the household in which they grew up and knowing some things about their parents education and their test scores or something like that I think 85% of individuals being individual is good so I guess that gets to heritability a little bit I had a question about the evidence on teacher value at it and I was curious I was unaware of this pattern by Grey Lull is fascinating I was wondering if you would see a similar pattern if you looked at some outcome that we could think of as a proxy for non-cognitive skills and whether we would expect to see the same pattern or would that be more concentrated for early childhood education Yeah I don't know the answer to that question because I haven't tried it I mean Conrad knows very well teachers value added in non-cognitive skills is often quite different than their value added in cognitive skills in fact some teachers are very good at teaching math or very good at teaching verbal skills but not good at both and there are teachers who are very good at keeping kids in school making sure kids don't have discipline problems getting them to go to college things like that and they are not necessarily the same set of teachers who are good at raising math scores and test scores in other subjects so my my speculation would be that it would not the pattern would not look that similar if we were talking about non-cognitive skills and the things that you would want to do first would probably be to look at disciplinary outcomes other things like that where you think teachers play a very important role right they're very important motivators but it's not that close to just being good at taking math tests right so and you also need of course non-cognitive skills that you can actually measure so that's one of the great difficulties with non-cognitive skills as a lot of them are quite difficult to measure but I expect that the pattern would look different Sylvia do you have a sense of whether it would look younger I would I might think it might look older actually yeah a little bit later in adolescence when when your social skills are still continuing to develop so Larry Steinberg shows that social development has an even more protracted developmental period than abstract cognitive skills yeah one of my favorite tests I was mentioning this yesterday is this test that they give on the NLSY which is a longitudinal survey that just is extremely easy questions on it and they're really boring and the only thing it's picking up is what's called coding speed because some people get very discouraged when taking the test and they just get bored and so they stop doing the coding and so they don't it's not actually a hard test at all but it's a very very good predictor of a lot of non-cognitive outcomes later in life because it's sort of picking up that someone is is very motivated right so we need more tests like that that are supposed to be picking up non-cognitive things and not just cognitive tests we have a lot of cognitive tests and then two few non-cognitive tests I think please join me in thanking all three of our thank you thank you