 Thank you very much Brian. After that terrific introduction I think I should just leave now. It's all kind of downhill from here but I am very pleased to be here. I have great admiration for both the Ford School of Public Policy and the Michigan School of Education. I have friends at both places and have learned a great deal from the work of scholars at both schools. So as Brian said I want to talk about patterns in US high school graduation rates. This is based on a paper that I wrote for an economics journal so it's you'll see it's more economics-y perhaps than it might have been if I had written it for a different kind of journal although I won't put any there's no mathematics senator or anything else. Okay so this is from a speech that President Obama gave in 2009. We have one of the highest high school dropout rates of any industrialized nation. So clearly a concern at the presidential level. Also if you look at how do we compare with other industrialized countries and high school graduation rates. Well in 1965 we rank first among the 28 or 30 OECD countries that includes most of the countries, large countries whom we trade with. Over the next 20 or 30 years many of these other OECD countries increase their high school graduation rate quite markedly. The net effect is that in 2008 America ranked 19th among a 28 OECD countries. Now why should we care? Why should we worry about this? Well one reason is that and this is a theme that's well developed in Claudia Golden and Larry Katz's terrific book called The Race Between Education and Technology is that rising educational attainments fueled the rapid economic growth during the first 70 years of the 20th century that made the kind of economic pie grow at a steady rate and that large economic pie allowed standards of living for virtually all Americans to rise. A second reason again is that education is the primary mechanism of upward social mobility in the United States and this idea that a child may grow up poor but his children won't grow up poor. Very much lies at the root of what America thinks of what's good about itself. This is dramatically threatened when we have stagnation in high school graduation rates. Something that many Americans are not aware of is if you talk to Europeans you know who will chastise Americans because we have higher rates of economic inequality than many European countries and most European countries have. The Americans will respond but say yes we have this high rate of socioeconomic mobility. The most recent data shows the rate of intergenerational socioeconomic mobility is lower in the US than it is in most European countries and the stagnation in educational attainment is a key part of that. So that's why we should care about this. Now what are the questions? I'm going to try and address four questions. The first is what are the important patterns in high school graduation rates? The second is what are the most compelling explanations for these and my guess is that's where a lot of the discussion will have with Q&A. What's known about effective strategies to increase high school graduation rates? Talk about that and then finally a kind of different more research question is what's the challenge in making policy evaluations more useful to the folks on the ground trying to increase the number of American youth who graduate from high school? How do we make research more valuable to that effort? So those are the four questions I want to try and address. Now just a few kind of words to start is this I'll be very brief on these methodological issues but clearly you know how do we measure high school graduation rates? Oh it turns out it's not so easy to think of how to do it. There basically are three ways of doing it each of which has problems and the problems are different which is one reason why we see very different estimates of high school graduation rates and these differences matter particularly for certain subgroups particularly minority group members. So one way of doing it is the so-called status completion rate. So that's kind of the most straightforward perhaps most transparent. That is for example if we have surveys of a nationally representative sample of Americans as we have and we'll talk about that in a minute we could for example take a national representative sample of 20 to 24 year olds and we can simply ask them what's the highest education you completed and from their answers we can say well okay some percentage of them graduated from high school or perhaps went beyond that. That would be our estimate of the status completion rate. The sample that you could the data set that's most frequently used for this is the current population survey that's a survey that's administered to about 60,000 American households every month that's where the information comes from on the monthly unemployment rate. So that's a common estimates very straightforward. Now what are the problems with that? Well one problem is it's a single individual who answers the questions for everyone in the household and so the question would be part is do they know but perhaps more important do they perhaps exaggerate a bit and do they now of course that's particularly important if groups with particular attributes are more likely to exaggerate or if over time people become more or less likely to exaggerate in other words so those would be sources of problems with our estimation of differences in high school graduation rates across groups or over time. Another problem of the current population survey is it doesn't include people who are in group quarters. It doesn't include people who are incarcerated and people who are in the military. The census does that but the census of course is only once every 10 years and that's why we now have the American Community Survey which gives us recent estimates of high school graduation rates but that also has its own problems. The second is a so-called the cohort graduation rate and this has been around for very long time for at least 40 years. The US Department of Education asks every state to report to it every year the number of people who are enrolled in public schools in the state at each grade level information that states get from local school districts. So you collect all that information and you know how many individuals there were who were in ninth grade in the country and that's in public schools and there also is reported how many students graduated. So you could at principle take the number of students who graduated in a particularly year divided by the number of students who started grade nine four years earlier and that would be an estimate of the high school graduation rate. That's done that's called a cohort graduation rate. Among the problems with that is we don't have any information from that and the 11% of Americans who attend private high schools. Of course if none of these surveys do we virtually anything about the 3% of Americans who are homeschooled. I mean that's just a complete black black hole. Also there are problems where the kids move across states as they do we can lose them. So there's a number of problems with that. Another bigger problem is the number of students in grade nine is not the same as the number of students who started grade nine at the beginning of the year. If you'll so in a minute kids repeating grade nine is a very common phenomenon particularly in recent years and particularly in urban high schools. So that's a problem in getting the denominator in this ratio. So the last thing is we can look at longitudinal surveys. My guess is in a number we've looked at the NCES longitudinal surveys such as high school and beyond, NELS 88 that follow large samples of American students over time and collect information on what happens to them. There are a number of surveys that do that. That's a terrific source of information. One problem with it is most of those surveys start when kids are in grade 10. The exception is of the National Education and Longitudinal Survey of the class of kids who were in eighth grade in 1988. That starts with kids in eighth grade. It's been shown that about a third of the kids in from who drop out of high school in this NELS 88 survey do so before the middle of grade 10. So they'd be totally missed from all these other surveys that start collecting data when kids are in grade 10. Now I know a relatively new source of information are the longitudinal databases collected by State Department of Education. The federal government has provided grants to a large number of states to develop longitudinal databases where they follow every child from the date they enter public schools in the state until they leave and use that to collect information on high school graduation rates. And that's a promising strategy. There are problems with that as to, for example, when kids leave to go to private schools or when kids leave the state or when kids simply disappear. And we'll see that. So these are basically, that's enough on methodology, three ways of doing it. The estimates that we get vary a bit. They tend to be lowest when we use this cohort graduation rate method. They tend to be highest when you use the status completion rate method. How much of that is people exaggerating the educational attainments of their children or parents remains to be seen. Okay, now the other big, the kind of 500-pound gorilla in estimating high school graduation rates is how do you count people who left high school subsequently took these eight hours of examinations in five different subject areas, achieved passing scores and awarded this alternative credential called the General Educational Development Credential or more typically called the GED. It's an important question because it's not a small number. It's basically half a million Americans get that credential every year. Now one, so it's increased if you look at people in their 20s who are quote high school completers, about 12% of them now are actually GED recipients. So clearly our estimates high school graduation rates are going to be enormously sensitive as to whether we count those high school graduates or the economist dropouts. It's going to be particularly the case. It's going to have a biggest impact on African Americans and minor and Hispanics because they're even more likely that are non-Hispanic whites to take this alternative route to high school completion. Now there's been a number of papers, some that my group has done, some that Jim Heckman, the Nobel Prize in Economics and his collaborators have written that have shown quite dramatically that individuals who get a GED on average don't fare as well in the labor market or in post-secondary education as to people with conventional high school diplomas. And for that reason in the numbers I'm going to show you to the extent that I can, I followed the NCS practice of counting GEDs as not as high school graduates. In these three methods, one of them, the status completion rate counts GEDs as high school graduates. It's not the case in your estimate of these cohorts graduation rates. So what you do with GEDs really matters. So if you're looking at numbers on high school graduation rates, you want to look at the footnotes right away. It's not in the footnote of the text it ought to be and say, how are they counting GEDs? It makes a huge difference in our estimates of high school graduation rates, particularly for minority group members. Okay, so the first question, what are the patterns? Well, the first pattern really, as I said, as I said before, is this extraordinary stagnation since the early 1970s. So these are estimates from three different data sources on high school graduation rates for U.S. from 1950 up through 2008. Now if we carried this back to the year 1900, which would be about here, it would be at 6% in 1900 according to Golden and Katz's book, climbed precipitously in the first 70 years of the 20th century has been just remarkably flat since the early 1970s, while again, the high school graduation rate in many other countries that we compete with has grown. And that's the reason why U.S. has fallen from 1st to 19th. So that's the first big pattern to explain, the stagnation. That's a bit tough to explain because, you know, it's kind of like understanding or trying to understand why the dog didn't bark. Tough question to answer. Okay. The second are these substantial and deeply troubling gaps by race ethnicity and less well-known by family income. And those are quite striking. The race ethnicity gaps are between 10 and 20 percentage points, depending on which method you use to estimate high school graduation rates, but they're substantial. The gap between if you rank American children by their socioeconomic status of their parents, the high school graduation rate of youths whose parents from the top quartile, the top 25%, is 31 points higher than the high school graduation rate of kids whose parents' socioeconomic status ranks in the bottom quartile. Just deeply troubling, particularly if we're concerned with socioeconomic mobility. And that's a value that I think really cuts across political ideologies in the United States. The idea that one may grow up poor, one's children will not. This deeply challenges that. The third and is growing gaps favoring females. Sudinarsky has written with Martha Bailey, another Michigan faculty member, a very nice paper on these gaps favoring females, particularly among college students and college enrollment and college graduation rates. And as Sue has pointed out, it's some sense tougher to think about than our gaps by socioeconomic status. Gaps socioeconomic status, you can think of lots of reasons why kids growing up in poor families are going to have a tougher time for kids growing up in African families, in affluent families. But gaps by gender, when the kids are growing up in the same families, as Sue has pointed out, we need to think more creatively in trying to understand those patterns. Okay, now, this is the only graph that's got, is this has many more numbers on it than it ought to, as my colleague and friend, I, John Willett, is quick to tell me. But I just want to explain it and point to a couple of patterns here because it'll help you to understand how, what, why certain patterns, particularly gaps by racismity and by gender, are very sensitive to a couple of decisions that you need to make. So what we have here is, this is information on for all kids who are rising ninth graders in Massachusetts public high schools in 2003. Then from this state longitudinal database, six and a half years later, so in the summer of 2010, we asked what their status is, and there's lots of possibilities. We, they could have graduated from high school. And we know that that's, that's good. That's the first column that we could have dropped out. And perhaps we've gotten a GED. That's a second column. They could still be enrolled in striking. There are a little less than 1% who still are going into a seventh year. They could have transferred either to a private school or out of the state. That's another column. They could have disappeared from the longitudinal database. Or now, so notice those three still enrolled transfer and missing. You don't know what happened to them in terms of ultimately the high school outcome. And that, that's a real problem. And depending on the quality of a state's data collection effort, people slip between those categories. Whether you know they've gone to a private school, you know, or you know they've gone from Michigan to Indiana, the is very dicey and tricky to get good information on. So basically, so if you look down, now, why does it matter? Well, if you notice for the on the red row there, which are for Hispanics, we started ninth grade in 2003. For almost 19% of them fall into this category. We don't know what they were doing six years. So how do you treat how do you count those people whom we don't know? Well, we could take them out of the denominator entirely. And if we do that, then we can come up with an estimate. But of course, we're missing a lot of what may be going on that we don't know about. So that's a key problem with these larger to the databases, people who simply who disappear, we don't know about them. And they may not be random at all. They may it may be, in some sense, the most troubled kids. Another thing to pay attention to. So the last two columns are graduation rates for those who we know what happened to them. What's the graduation rate in four years? That's the far right column. And what's the graduation rate in six years? That's the next to far right column. One thing you can see, for example, if you look at Hispanics, you can see among those folks for whom we know 55% graduated in four years, 65% graduated in six years. So here are estimates of high school graduation rates, particularly for minority group members, are going to be very sensitive to how many years we give them the graduate that really matters. And finally down at the very bottom on the right, these are our rate, these are our gap, our gender gaps within race ethnicity. And you see really pretty much the same thing. So if you look at these numbers, I promised Bonnie I wouldn't go too far from the mic here. So I'll come over and come back quickly. So for among African Americans, we've got a 16 point gap, gender gap on high school graduation rates measured to the graduate often called on time four years. And if we look at given six years, that gap is substantially smaller at 11%. So the African American, a much larger percentage of African American males and African American females take longer than four years to graduate. Okay, so again, this is kind of accounting stuff when you see again, estimates high school graduation rates, you don't only want to ask them how they're treating GEDs, you ought to ask, how are they, if it's from one student of data, what are they doing with the people who disappeared from the database? And also, how long did we give people to graduate in our estimates? So back to our patterns. So we had, we talked about the first three. The fourth pattern is that half of dropouts from American high schools went to high school in 14% of the nation schools. This is a Bob Balfair. So point he said, like the member at Hopkins. So they concentrated in schools, they're not all urban schools, many of them effect are in the rural South, but they are schools that overwhelmingly serve kids from poor families and kids of color. And that's a situation, in fact, that has gotten worse. And I'll talk about that in a few minutes. And the final point is this big variation across states in high school graduation rates, a nice topic for to understand. There's a lot of measurement error in this. So it's not easy to do it. But there are quite striking differences in high school graduation rates, even among, for example, a particular race, racial ethnic group, if I can find the right page here. Yes. For African American youths, the high school graduation rate in Indiana is 53%. And that's four year on time. It's 73% in Indiana. Well, why is that the case? Thank you. In Indiana's the low, Maryland's the high. Thank you, Brian and Sue. One other thing that I missed here. Another thing that really matters when you're when you're trying to estimate high school graduation rate for Hispanics. And that is, what do you do with recent immigrants? Do you include them in the sample? Or do you not? Well, why does it matter? Well, according to the American Community Survey in 2009, 7.5% of of individuals age 20 to 24 in the United States came to the United States within the previous 10 years. So 7.5% of 2020 falls are relatively recent immigrants. Okay. Of those 59% are Hispanic. Of those recent immigrants who are Hispanic, less than half among 2020 four year olds graduated from high school. And many of them came and never went to high school. Many of them appeared particularly in urban high schools as 10th through 11th graders. So how you count them has an enormous impact on your estimates high school graduation rates for Hispanics. And of course, that's going to impact on this variation across states enormously because Hispanics are heavily concentrated in particular states. Okay, one of the explanations. Now I want to just have one slide here that's kind of the economics perspective. Since I wrote this paper for an economics journal, one thing you might think would matter is how much more can you do high school graduates earn than dropouts? Because it makes sense if that has declining over time, you might think it's less, it pays off less while they graduate from high school that didn't the past. Another is, how good are the jobs? What do you lose if you stay in school instead of working? And there's very interesting work on this that shows this matters. For example, the work I know best was done looking at data from Appalachia in the mid 1970s. After the OPEC oil crisis, the price of coal rose quite dramatically. It affects a lot of minds reopened, increased output, because coal was more valuable. That created lots of employment in Western Pennsylvania and Ohio and that region. That led to an increase in the dropout rate. Because again, these young people had well paying jobs available. When the coal boom faded, the dropout rate faded as well. Kids went back to school. So this matters. But can this explain national patterns? Third is, what is a funny, basically, you can think of this as, do kids care about these long term trends? Or do they have trouble thinking beyond Friday? And clearly, if we think if those of us who have teenagers or had teenagers, this thinking only to the next Friday night is very pervasive. But we do need to ask, though, is this more true for kids with certain characteristics on others? Is it more true now than it was 20 years ago? Because if that's a stable aspect of being a teenager, that won't help us explain any of these patterns. And finally, and this turns out as all the action is, is what in funny economics language would be the non monetary cost of completing another year of high school. How much of a drag is it to complete high school? How much time does it take? How unpleasant a process is it? Now, if I would, if this were, if I were a sociologist or a psychologist, I would be breaking down that last category into lots of much finer grained categories. And it turns out that's where really all the action is, as we'll see. So basically, how important are each one of these? Well, as you'll see, the top two, which are the ones economists most familiar with, don't get you anywhere. In terms of explaining these patterns. And you can see this, these are information from the current population survey. These are mean hourly wages by gender by education group. The bottom curve is dropouts. The red line in the middle is terminal high school graduates, the top line a four year college graduates. I didn't put in the sum college because that's a messy thing to calculate. So notice, there's no evidence here that the gap between high school graduation, earnings of high school graduation, and the real wages, these are all net inflation of dropouts decline. That this gap did not get small. Now, in fact, if you look at a more fine grained way, a particular racial ethnic groups, you see it even more strikingly. So basically, what you have here is the ratio of the average hourly wage of dropouts, divided by the average hourly rage of high school graduates who didn't go to college. So if you notice in 1980, for whites, non Hispanics, that's the blue curve for males, it's about 0.88, which means on average, dropouts are 12% less in 1980, then did white male high school graduates. We noticed with the one exception of male Hispanics, all those lines are downward sloping, in effect. So in other words, for all these different groups, the relative wages of dropouts compared to high school graduates got worse. So by this, if this is all that's going on, this should tell us high school graduation rate should have increased. It shouldn't have stayed stagnant. You know, the key story is here, high graduate high school paid off more than it now than it did 30 years ago. So we don't get anywhere there. Also looking at the same graph, you could see the idea and going back one, the idea that the payoff to leaving school immediately and going to work was greater is just wrong. There's no trend of indicating that high school grad high school dropouts fared better in labor markets than they did in the past. And in fact, if we look even further, these are the employment, the population rates among dropouts by gender. And notice, particularly for males, that's the top line, notice that's breaking me downward decline. So the percentage of adult dropouts who are working at least report working has declined precipitously. So the idea of kids are dropping out because there's the great jobs in the coal mines just doesn't explain the patterns. No, not notice the myopia. So basically, we need to turn to this last category. So what's going on? How do we explain these patterns? Notice we got the stagnation, the dog that didn't bark, we need to explain, then we need to explain these gaps by gender, these gaps by rathesnicity. And one, so there's basically four parts to the story, as I see, and the four parts go together. So the first story is to think about, as you increase high school graduation rate, it becomes harder to increase it further. Why? Because the kids who are left are increasingly from poor families who enter school, enter high school with relatively weak skills. So that's a just a story keeps getting harder. It's harder to go from 80% to 90% than it was to go 50% to 60%. Now one you can see that is, if you look at what these are data from Massachusetts, they're purely descriptive. So the black lines on the left are evident are the magnitude of the the black white gap and high school graduation rates. So for example, among males, the black white gap and I think graduation rate not controlling for anything is about 18% points. So the average high school graduation rate for black males in Massachusetts is in that six year graduation rates is about 18% lower than that for white males. Then you say, Well, how about if we try and hold constant, the effects of low family income, which African Americans are more likely to come for low income families, we all content their eighth grade attendance. And most importantly, we all content their eighth grade test scores. And holding constant those notice that explains somewhere between two thirds and three quarters of the gap. So a big part of the story is these kids are entering high school, really behind the eight ball. Among those kids will ultimately drop out much more so than was the case 50 years ago. So that's the first story is it's getting harder because we're working with kids who are some sense have more strikes against them when they get to high school. They in a and you see this, my colleague custom snow studies reading, for example, that a lot of the reason kids have difficulty in getting out of ninth grade, we'll see in a minute, is they enter high school without being able to read well enough to make sense of science and social studies texts in middle school or high school. So the second story. So the first is that we're again, we're working with kids who are have more face more challenges as we as we move over time. Second is policy changes. Beginning in the mid 1970s, there was concern about the quality of American workforce. And that led to a really pretty much many waves of policies aimed at increasing the standards for high school diploma. First, it was minimum competency tests in the 70s, then came a nation at risk in 1983, report many of you know about that led to increase core requirements, number of science courses at the past, having to pass algebra one or algebra two. That was followed by exit examinations. Now 75% of the nation's kids must pass an exit examination typically taken in 10th grade in some states in 11th grade in reading mathematics in science now in Massachusetts as well in order to graduate. So we've increased those. And when you look now at there's been lots of research, Brian's done some of it. A lot of research on does increase in high school graduate in standards for high in requirements for high school diploma have an impact on high school graduation. If you look at it across all kids, the answer is pretty much no. But if you then look at how about if we focus on the kids who are most vulnerable, the kids who come from poor families, the kids who come from minority group families, which you see is the answer is yes. You do see that that these increasing standards make it harder and we'll see why in a minute. Now I want to be clear. I don't mean this as an attack on these attempts to increase high school graduation standards. In fact, in Massachusetts, the state has increased standards quite dramatically. And I think done it quite well. There are lots of positive benefits from that. And you see this in how America how Massachusetts high school students compare in both on international comparisons, and compared to other states. But there are these unanticipated consequences for the kids who enter high school with two and a half strikes against them. Now one way you see this is, these are percentage of Massachusetts first time ninth graders who repeat grade nine. And notice you basically got among African American, Hispanic and urban low income kids about a quarter are repeating grade nine. So for example, if you go to East Boston High School, which is a big high school in Hispanic area Boston, there's 1300 students in the school of those 1300 700 on ninth graders. Again, because they can't pass that core English and math courses. Now how about how do those kids fair these kids who repeat ninth grade? And the answer is not very well. This is the percentage of them kids who repeat ninth grade, who graduate within six years. And if you notice, for example, among Hispanics, it's under 30% for none of these groups is that under is it over 40%. So basically, failing repeating ninth grade reduces the probability you graduate from high school by a whole lot. So basically, you can just one. So basically, a combination is we're working with kids who enter high school with in a difficult situation. We've raised the standards now, these standards affect disproportionately these vulnerable kids. Yeah, is it really them repeating ninth grade that causes them to have such low graduation rates? Or is it more that it kind of shows that they wouldn't graduate because they have to repeat ninth grade? Oh, yeah, I didn't mean any. This is pure descriptive, not caused. But they've got this 10th grade exit exam ahead of them. If they get there, and and a large percentage of them don't pass that they can take it again and again. But they have to pass it. And the law and some of them take it as many as seven times. Now, the last so we got four stories. So we've got again, we're working with kids who are poorer. We've increased standards. The third is this increasing family income inequality increases, which has been a phenomenon in the United States over the last 30 years has increased the challenge for kids from low income families of meeting these nine now higher standards. Okay, and I want to this is this is a volume that was just published. That's the book cover. It just came out last week. Sudenowski and Martha Bailey have a very nice chapter in there. Brian Jacob and a colleague have a very nice paper in there. Brian Rowan has a very nice paper in there. So and so Michigan is very well represented in this volume. Basically, so how does this story work? Well, first of all, you can see this is the growth in income inequality in 1977. The families at the 20th percentile of income distribution earned a little less than $26,000 in 77 families in 2009, who are at the 20% tile earn had a bit higher income 27,000, but it's virtually the same. This is all in constant dollars. So basically, no change in the earnings are in the income of families at the 20th percentile. How about families at the 80th percentile 84,000 and change in 77 113,000 in 2009. Now, why does that matter? One reason it matters is that again, with these higher graduation standards, it poses challenges for a significant number of kids, families that have resources can hire tutors can do a variety of things to meet those higher standards. Poor kids can't. You see this, for example, this is expenditures on enrichment. And these in the in the 1970 to 73 poor families lowest 20% spent about $835 on enrichment activities. And this is things like summer camp trips, those kinds of things that takes kids to have outside of their normal experience. And wealthy families spend about 3500. Look now 2005 six, some of it increased for poor families, just an extraordinary increase for kids from relatively affluent families. Now, why do we care about that? Well, this is one reason why how kids acquire background knowledge. They've been the national parks, they they've been on trips, they've been New York, they see a whole bunch of which they just have a broader set of experience. Why does that matter? Because when reading changes from learning to read and decode words to reading to learn, as in, in middle school, science and social studies text, background knowledge is absolutely central to being able to make sense of textbooks. And the difficulty is many poor kids just haven't had the experiences to acquire this background knowledge. Also, this is again, these are gaps between kids from wealthy families and poor families in when they enter kindergarten is the white bars. And in fifth grade are the dark bars. Basically, we see kids from poor families enter school with lower reading achievement, enter school with gaps in school engagement, and are more likely to have mental health problems, and to have basically behavior problems. So they again, you see these gaps that are income related for kids entering school and the gaps don't get smaller. This is even I think more trouble. There's a recent work shows that a consequence of rising income and equality is we have more residential segregation by income than we had 30 years ago. The nice paper on this is Sean Reardon and a colleague in American Journal of Sociology about a year ago was published. So basically, and this get this, this phenomenon of basically families with resources moving to neighborhoods with the more people like them. And this is even the phenomenon is particularly striking among African American families. So the net effect is that leads since most American kids still go to neighborhood schools, it's led to more segregation by income in schools. And there's a paper in the volume that documents there's been an increase. So effectively, poor kids are going to schools where their classmates are overwhelmingly other poor kids. And that's the case more than now than it was 30 years ago. And you see this. And one way that plays out is altogether as growing gaps in skills between poor kids and rich kids. It was always a gap in 75. That's a 75 kids who were 14 that year. It was a little less than a standard deviation if 100 points of standard deviation. Notice over the next 30 years, it gets about a third large. And this is work by Sean Reardon also in this volume. And we also see growing gaps in educational attainments. That's Sue and Martha's paper in that volume as well. So all in all, basically, we see is that we've got this combination of higher standards, higher standards impact, particularly on kids with the most vulnerable, and the income inequality, growth in income inequality has made it harder for kids from low income families. And the last part of the story here is this growing role of the GED. GED has grown in importance. A number of states have reduced the minimum age. You can take the test from 18 to 16. There's Jim Heckman has a couple of nice papers showing that where the kids decide to drop out of school, and someone get that GED depends upon the relative difficulty of getting a GED versus getting a high school diploma. So it gets harder. We've got these rising standards. They impact particularly on poor kids. And the kids are more poor kids are more likely to take this alternative route, which unfortunately does not serve them well, either in preparation of post-secondary education, or in the labor market. Now, what do we do about this? I'm near done. Unfortunately, because the what what do we do about this is much briefer. Thank you five minutes. Thank you. All right, I will do it. One is we've got this problem of what's the evidence standard. And what I tried to do is look at evidence from well done evaluations that impacted on either drop pot rates, high school graduation rates. The net effect of that is we don't really don't have anything that talks that evidence on elementary school, because almost no studies of elementary school interventions follow kids long enough to have impacts on high school graduation rates. So the first thing that you see not a surprise is a terrific paper in this volume by a very good neuroscience is Charles Nelson on the importance on the on the neuroscience of the first years of kids lives and how those are particularly important years to intervene because of the rapid brain development. So that there's no question this is absolutely critical to intervene in a variety of ways, including preschool education and and also important are supporting families. They're not then these new poverty numbers that came up from the census yesterday are deeply troubling. There's now five studies with good causal designs that show when very low income families get more money, the kids do better in school. A lot of some one of those studies is done looking at expansions of the early income tax credit and how that helped families. So it so these are not correlational, they're causal, and they're quite compelling. And given this declining, we're seeing it in income for great many American families. It's a troubling pattern. The third is improving high middle schools and high school quality. Well, how do we do that? Well, we know they're it makes a difference. So there's no question, even when kids enter high school with two strikes against them, good high schools can make a difference. These are all studies with good causal designs that have shown better quality schools really make a difference. And I could talk about those as pick less studies in the Q&A if you want me to. Then the question is, of course, what does it mean to be a good high school serving poor kids? And that's where it gets tough. So if we look at one of the characteristics, the schools that have clear goals and high expectations, they have a sense of community, they have a culture in which a culture of respect, teachers for the kids, kids for the teachers, kids for the kids, teachers for the teachers, they they have skilled teachers who are really working together to make instruction more consistent. The last one's in parenthesis, because I think it's a relatively new phenomenon, they make careful use of student assessment results to figure out where kids particular skills are falling down and intervening quickly. Now, I don't think anybody I'm sure you would have your own language for this list, you probably add to things in the list. My guess is, is nobody who we really disagree with that list, that those things ought to be on it. Notice, however, those are characteristics of effective schools, they're not policy levers. So how do we get those things? And that's where it gets really tough. And here was his challenge of making policy evaluations more useful. Now, as many of you know, these super education sciences was started in 2002. Basically, the idea is this would be the NIH for education that would fund high quality research that would provide good information to people making decisions about schools about how to improve them. I guess, it's first director, Russ Whitehurst, recognized that education research had a very poor reputation, much of it deserved. And constant was really important to have very high quality standards. As a result of which, in its first six years, IES funded more than 100 randomized control trials, which is clearly the kind of goal standard for evaluations. Well, and I applaud that. But the challenge really is going forward, is that when you design randomized control trials, what are the kind of interventions that lend themselves to that kind of evaluation. They're having an extra period for kids who are falling behind, having an after school program, having a very discreet intervention that doesn't change kids fundamental experiences of what the school day is like for them. As a result of which, there are none of the evaluations that show of particular school based interventions that show striking impacts. Not a surprise. So the difficulty is we know what we want. We want more schools that look like this. The difficulty is the kind of evaluations we know best how to do. Don't don't map tightly the policies we can evaluate. Don't map tightly to these kinds of schools that we want to have. In particularly, you need to do a lot of things at once to create these kind of schools. That's much harder to evaluate. So I think really that's the challenge going forward. Like John Easton, the current director of Institute of Education Sciences, who spent a number of years directing the Chicago Consortium, Research Consortium, which I think is a fabulous operation, really faces challenge. How do we keep the rigor of IES funded research? So we really believe the results. But how do we recognize we need to be evaluating things more complicated than simply adding an after school program? I think that really is the challenge. If you ask me about Brockton as an example of that challenge, I can do that. But Sue has told me I should end. So this is where we've gone. Basically we saw five important patterns, four explanations. And what's known about them is we've learned some. We have an awful lot more we need to learn. We know what characterizes good high schools. We know them when we see them. It's much harder to figure out how to get there. Thank you very much. People want to make a comment. Ask a question. John, nice to see you here. Thanks for coming. You started out by comparing the United States and the rest of the world sort of got lost. So is there something that came comment one night from Paris in particular? You're really basically saying that the United States is always been kind of, this is my preference. There's always been two worlds. The world that our kids live in, the world that they have. And that must be, in some sense, less true. If that's the diagnosis here, then it suggests that it's less true in other words. Just comment on what did we learn from that comparison. Well, as I'm sure you know, I mean, internet comparisons are hazardous term because you can use, you can see many differences. But what do you learn from them about what US should do, given US as different institutions? One thing is you see all these other countries that do better have much better social safety nets than the US does. So that's the kids don't start out as far behind as poor kids do in the US. I mean, that's striking and you see this. Almost all, so that's, I think that's the easy one. They also have, teaching is a more respected occupation in most of these countries. They can be more selective in whom they bring into teaching. And they were a trickier term. Almost all of them have national curriculum and national exam systems. Now, why does that matter? Because in the United States, if you're teaching, if you're doing a pre-service training for elementary school mathematics teacher, you have to think about, well, you know, you might end up, so in a school where you'll be teaching Saxon math, which is a very traditional math, you might end up in a school where you're teaching everyday math is very constructed. So we can't focus on what textbooks you're going to use or even what's going to be really at the core of what you're asked to teach. Well, that's the situation in the United States. It's very hard to have good instruction. Now, what do we do about that? Given, you know, local control has been, you know, part of the American scene for a long time. We're moving slowly towards, you know, we have these common core standards, how, whether they will be good standards. So I think there are striking differences. The other area where there are more differences is more paths through high school in many of these countries. In particular, there are apprenticeships, which I think are important in creating opportunities. I think the evidence from the U.S. suggests this is, in other words, for many kids are asking, why should I be learning algebra? This doesn't connect to anything in my life. I see, I don't know anybody who has a professional job. Why should I be learning these things? In many other countries, a lot of those kids are spending a day a week in internships as part of a secondary school training. Now, in those Northern European countries, where internships and apprenticeships work well, they have strong labor unions. They have a government and businesses play a key role. It's a very different political system than the U.S. is. So what can the U.S. adopt and move from that? I think it's a challenge, but I see the evidence from, like, career academies and some of the small high schools that work as suggesting we have to make education more relevant for kids in high school, particularly for poor kids, connecting to experiences in no-class workplaces can be a part of the answer. We have a little evidence to support that. I guess it's saying on the international tech, how much or any kind of the blame for America on behind be attributed to our relatively short school year compared to several of the countries that really leave a problem ahead of us? It's a good question. I think there is evidence, growing evidence. In fact, we're Steve, so I had lunch with. Your mentor, Dave Marcotte, has done interesting work on the length of the school year, I believe, showing that a longer school year from NASA experiments in the U.S. made a difference. One thing when you look at these schools, whether they be charter schools or charter schools are public schools, you have to be careful about the language, or other public schools that are successful in serving poor kids who come to school with real learning deficits. They find ways to more instructional time. Rollin' Friar has a nice paper that's just come out about the Promise Academy of the Harlem Children's Zone. Kids start school day at 7.30 with breakfast. They have time for mediation before they start classes. They're there until 5.30. They're there on Saturday. They're there in the summer. So, basically, if kids start out with deficits, they need more time for instruction, and the time has to be used well. They've done a nice job of figuring out how to use it well. In many of these small schools of choice in New York City, who now are having to serve more kids with special education, they're saying, look, we can finally get your kids to be able to pass these reading exams, but it's going to take six years. We can add instructional time that way. So, I think, clearly, you got to have it. Instructional time is a necessary but not sufficient condition. You've got to figure out how to use it well. I had a question about the relationship between increasing qualifying tests and graduation rates. Do you know about any studies that have tried to look at whether the tests mean that students would have made some progress, but when you're forcing them to make a lot of progress, they drop out? Or you really just not giving us a degree to people with the same skills? So, before, if you shut up every day, even if you didn't learn anything, you still got a degree and now you don't. Great. Let me, I'm about 100% sure I used to. It is the case. I mean, it is the case. And a book David Cohen, one of the authors of, The Shopping Mall High School, now an old book is really, I think you've said it really nicely, you know. In the 1970s, before this kind of upgrading started, if you pretty much went to school most of the time, went to class most of the time and didn't misbehave badly, you got to graduate. And that really has changed in an important way. And I think, you know, it makes a lot of sense. You do want to have kids graduating with skills. But there are these consequences for the most at-risk kids. So there's two questions. And one is, does imposition of the exams lead kids to drop out? And there's some evidence. If you look at the most vulnerable groups, yes. Also, work we've done in Massachusetts, use my Grist and Discontinuity approach, if you fail the test and compare it to kids who look equally skilled, but just or a notch above it, are the kids who just fail more likely to drop out for the whole population? No. For urban kids and low income families, yes. And the gap is large, about eight percentage points. Yes, ma'am. At least from the little bit I know, disadvantage starts really early. Not even in kindergarten, maybe as early as two or three years old. And so you're so going back to your point of the most vulnerable families, they start out really behind really early. But then secondly, this idea of characteristics of individual schools and characteristics of school systems, because if you get one school to do really well, let's say for the most advantaged kids, but you still have all these other disadvantaged kids in the rest of the city or the rest of the state or the rest of the county that still don't have opportunities. So it's one thing to say, well, can we do some tweaking to make this school and this principle better, but to make the whole system better and to kind of equalize equal opportunity across the board is a lot more difficult. I don't know if you've thought about that question at all. Well, I think that's, you know, that's the very difficult question. In American literature, you know, in American writing about schools, there have been these lighthouse schools written about for very long time. You look at Charles Schubertman's book written, I believe, in 1972, Crisis in the Classroom. You see the descriptions of these schools that succeeded were all the school in very potable which were all the schools around them did not. But then if you go back five years later, typically the principal has moved on, the teachers have gotten kind of burned out and no less than husband's bridge. So the challenge had, we've had these rays of hope that have burned brightly in urban school systems for a very long time. What can we learn from those about how to create more effective systems of education? That's been what's the real struggle, because it's not doing simply one thing. It's the whole bunch of things and that creates a serious evaluation problem. But I think that is the challenge, to go beyond individual schools. And you've got to go beyond simply we need strong leadership. Yes, strong leadership is really good. It really matters. It's important. But how do we create a system where we have systematically strong leaders? How do we have also, you know, a challenge for many of the strong I-Charter schools is they start out with these very young people, 22, 23 years old, who weren't hard to teach. They are very good the first year. They work 24-7. They are incredibly dedicated. After two or three years, they really are making a difference. Then they get married and they have a child and they're like, and they can't do that anymore. Well, this is a big challenge. It's a challenge in these small schools of choice in New York. How do you move from a model that depends upon 24-7, a very young people to a sustainable model that still has the care for the kids, the high expectations, the improving instruction, but has doable jobs where people can have lives that continue to do them. I think the Charter, we were talking at lunch, the Charter Schools Networks face that. I think this is a pervasive problem. Okay. Yes, sir. Go ahead. The blue. I was actually wondering if you started to talk a little bit about when you were talking about Charter schools. But I was wondering if there's been evidence of like programs like Teach for America increasing the amount of graduation rates or if like possibly expanding something like that would has shown evidence of making it a fact. That's another controversial topic. The best thing always is there are selection problems that are pervasive because typically are not kids and teachers are not randomly assigned to each other, so you worry about the matching. The best study was done by Mathematical Policy Research, which is a good policy research from what they did is they found a number of elementary schools in which the principal, the deal of principal, the deal of the principal was, you can, first of all, we had to have at least two fourth grade, two fifth grade, two sixth grade. Principal can divide up the kids any way they want. But then there has to be a coin flip as to which of the fourth grade sections the Teach for America teacher gets and which the teacher was already in the fourth grade in that school of God. So we were randomizing teachers. That was done in a fairly large number of schools and the question is what was learned? Well, the outcome that is emphasized on the Teach for America website. And I'm a fan of Teach for America for what it does for the participants. I teach 12 or 15 Teach for America graduates every year. I love having them in class. They're smart. They're dedicated. Most of them are there because they have such a frustrated experience. They want to start a school and do it right. What you found was that the kids who are the Teach for America participant on average did a little better in math schools, no better in reading schools. But I only got the real story. When you look more carefully, and this is in the report but you have to look for it, at what were the backgrounds of the non TFA teachers in these schools where TFA teachers were placed. And the answer is they were about the nation's least well prepared teachers. They typically did not go to the University of Michigan. They over one only went to very low quality preparation programs. They had very low student teacher. So one way of saying it is, yes, the kids did what TFA teachers did better. But they did better than very ill prepared teachers. So what do you make of this? Well, it depends upon if you say, if we as a nation are going to accept that the alternative to TFA is that the kids most in need of our best teachers are going to get our worst teachers. Does TFA help? Absolutely. It helps a little bit. But if we as a nation, when you accept that premise of well, what we can do is TFA. Because when you talk to TFA teachers, how many TFA participants are out here? So let me so we at least we got three or four. My experience in talking with the many I've talked with would say they were better in their second year than in their first. They had a very difficult experience because they were somewhat resented by the other teachers in their schools. Because again, you know, because the TFA teachers went were very well educated. The other teachers in these schools were not because remember, we're a TFA teachers place their place in the schools that can't attract regular teachers. So the ones who are there are the ones who can't find another job. There are exceptions. There are some wonderful teachers in every school, but there are not enough of them. So I think, I think TFA is fabulous. One of our doctoral students did a very nice education showing that a structurally large percentage of TFA participants stay in education broadly defined as you are. And that's great. I think it's absolutely terrific. As a strategy to solve these enormous gaps, nice for graduation rates that we've seen. I don't think it does. Could you touch on the issue of year round education, given that 99 percent of the population is a non-farm employment? I mean, we have kids that are, you know, forget what they learned all summer. You know, it's a crazy system to have a school system that, you know, goes on vacation for three months a year, and their parents are working all year round if they have a job that is. I think more instructional time generally make a difference. I'd want to do the evaluation very carefully with a really good design because, you know, what you don't want to do is what you want to be sure it's a bit like, you know, the class size story. If you lower class size, but you don't use time in any different way, not much happens. Well, in that effect, he says, OK, I can slow down. Kids are going to write the same three essays over the year, but now I'll spread it over 11 months instead of over 8 months. So I think it creates opportunity. But we need to really think through what are the incentives to design, what are the incentives for kids with a personal education is to be sure that kids have a very different experience. And I potentially can do it. There are places that do it. But how do we map this to policies? I think it's a real challenge. Yeah, could that also be dangerous to like the school is a drag effect, too? Because I mean, as you said, like, if you're just kind of sitting in there and you're not teaching them any more than you would in the eight months, it's kind of just keeping them their institution that we're doing it. Well, this is why we know it can't be just more of the more of the same. I think that's this is we have to figure out to use time in ways that makes education more relevant for these kids. I mean, there are models. I mean, it's not that we don't not that we know nothing. There are things to be done, but it can't simply be more of the same. Is there any evidence of studies of community mobilization? Because it seems to be part of the problem. I do some work in Detroit is that the community support for education is very truncated and and I there's been some effort to organize the community to see to really get parents and community residents and businesses and support of education. Is there any of it is that that is helping? Well, you know, one of the hard things is random randomized control trials are terrific. At the end of the day, being able to say whatever the treatment was made a difference or did not. It basically though you can't if it's a complex intervention, you don't know the relative importance of different pieces of this. Now, these small schools that basically under Joel Klein in New York City, very large dysfunctional high schools in the poorest areas, South Bronx, for example, were broken up. Now we're closed. But what they did in a very interesting way is they had a competition to start small schools and that and but to start a small school, you had to have a community partner. And the community partner, for example, with things like East Side Settlement House was a community partner. But that's not all that was there. They did lots of other things as well. They had students got to choose their schools. You had kids who for the most part wanted to be there. They had purse. We had waited per student budgeting. So schools got the money for the kids. They got more money if they serve poor kids. They did a whole bunch of different things. Now the evaluation shows that kids who went to these who won a lottery to go to these small schools of choice fared significantly better than otherwise would have. But it's a package of things. And it has to be a package of things. Another piece of that package, again, is reorganizing finances at the central level. So this is the difficulty here is that systemic change requires a package of things. But it's very hard to figure out what are the important pieces. Why don't you give me one or two more questions and then we'll. Sure. So how about you? I'm just thinking about not knowing the data to compare some between the United States and the other Western European countries. But the picture that you've all given is that the Western European countries are doing so much better than we are. Now, it seems to me that the gap between the majority population and the minority population in Western European countries and the minority population is mostly Muslim has got to be just as large and just as problematic with respect to educational issues as we see in outcomes for blacks and Hispanics in the United States. So it must be that the overall population that is, you know, Arab, you know, Northern African is much smaller in those countries and it doesn't exert such a large influence in the total outcomes. Well, in our case, if you add 12 percent black Americans, 15 percent Hispanics, I mean, that's about 27 percent of the population. So they do make a whole lot of difference. So that that would have to be taken into account in the comparisons also, because that their inequality isn't any better than ours. You may be right. I mean, the the depends on which European country you're talking about. And they have much more immigration. They have a lot now. But I think that's why these international comparisons are so so so hazardous. But I think it's I think the point is is important one and we need to know more who are those minorities? How are they faring? How many are them? Last question. So you have your what have been the effects of the growing classroom sizes, the integration of elderly and very diverse children into those growing classroom sizes? Because nowadays, teachers are teaching 30 plus kids. I don't know why I was at this school and never was 20. And then you throw the other students on top of it. That's going to be very difficult for us. I think that's right. I mean, in our evidence on this, Brian and Sue may know more about the tail evidence on this than I do. I'm thinking about the handshake study that finds having special ed kids in the class did not do any and they did not have a negative impact on the non-disabled learning kids. But there's a question of the numbers. There's also a question does how much money comes with those disabled kids? Although and I think that's the key piece of it is in most education I talked, I spent July in New York taught educators in small schools. And they said, you know, it's a real challenge. We got to figure out how to do it. And it takes time. What they're doing in these small schools is they're having more they teachers get the certification to deal with kids with special needs. And if they do that and if these kids come with this weighted per student funding with enough extra money to pay to the health they feel they can do it. They're going to have to keep those kids for a couple of years but they need the money and they need the time. So it's very easy if you do the same things it's not going to do it. I should probably stop but thank you very much.