 Okay. Well, let's get started. I'm Larry Michelle. I'm the president of the Economic Policy Institute and the director of the Education and Policy Research Program at EPI. So welcome. You're in for a real treat. We have some great panel, some real expertise, and some really solid analysis that breaks new ground, both comparing education outcomes across countries and with states, both cross-sectionally and over time. Can't get much better than that. And this is basically, you know, the stuff of what we are confronted with all the time in the press. And I think, you know, there's a lot of insights in this about what to believe, what not to believe, and what we ought to focus on. It's my job just to lay out the logistics. We're going to have our moderator, Mary Beth Markline, run the show. The speakers will speak. And then Mary Beth will ask some questions, will be some answers and discussion. And that'll be opened up for questions and answers, questions from the audience, answers from the panel. The logistics are, if you want to look for the bathrooms, they're back that way. There's a backdoor. Go that way so you don't walk through the videoing. This is on videoing online. Hello, online audience. This will be available as a video soon after the program, probably sometime today with our crack crew from EPI. And with that, let me just introduce Mary Beth, who was for many years a reporter at USA Today covering higher education, now reports for University World News, writing about international education. She has the great benefit of having returned to go to grad school, which is a very admirable and tough thing to do at George Mason University. And we're very pleased to have Mary Beth here to moderate the program. Thank you very much. Well, so welcome, everybody. It's great to see you all. I think the timing of this report is really interesting given that the NAEP scores just came out. And I'm sure our panelists will want to make a little bit of, give us a few of their insights about what to make of the scores. But beyond that, I'm going to just kind of do quick introductions and then let the presenters begin. So Martin Carnoy is the Vita Jax Professor of Education and Economics at Stanford University. And he's a research associate here at the Economic Policy Institute. He is teaching comparative education this fall at Stanford. And let's see. Oh, killed an electrical engineering degree from Caltech and a Ph.D. in Economics from University of Chicago. Emma Garcia is an education economist here at the Economic Policy Institute, specializing in the economics of education and education policy. After they give their presentations, Jim Harvey is going to offer some feedback and comments. He is the director of National Superintendent's Roundtable, an association of 80 school superintendents from 30 states. He also just received his Ph.D., he informed me, and his dissertation eventually turned into what is called the iceberg effect. So in January 2015, he was the lead author of school performance in context, the iceberg effect, which explored school performance in the United States and eight comparable nations. Unfortunately, Bill Schmidt could not make it today. He had flight troubles, so he's not going to be able to make it in time to our panel. So I am hoping that the audience will sort of crowdsource his questions and come up with all kinds of good things to say, good things to ask. So with that, I will turn it over to Martin and thank you all for being here. I will turn this over to Emma. We are dividing the presentation into two pieces. I'll cover the first is an alternative to Martin in a few minutes. Before we get started, it's very important for the three co-authors of this paper to thank the very many people who helped us finalizing and writing this report. And I should start by the within education team here at EPI, Larry Michel and Elaine Weiss, who is sitting in the back. They helped in the final stages of this paper and also in the very initial ones requesting the report in a way. Second of all, we received very helpful comments from three external reviewers. Professors Henry Levin, Sunny Latt and David Berliner, and their guidance was very, very helpful for shaping the contents and the message of our study. And I'll also like to thank Mike McCarthy, who has been working with us for a few weeks editing the report. The research assistance helping with the data and the communication teams within EPI helping in putting together the event, disseminating and many, many other things. Finally, to Mary Beth for joining us today as the moderator and to Jim Harvey. I'm really looking forward to the conversation with you, the discussion and also with the audience. So since we want to keep much the most of our time for the conversation and the discussion, let me just go straight to the main points that this paper does. We make three main points in this study. The first one is the following. Although international tests such as the PISA and teams show that US students perform below students in other countries, this does not mean that US students are not making greater academic progress than students in many of the high scoring countries. These country averages in a way hide the great variation in performance that exists within each specific countries. Variation across student subgroups, variation across regions, etc. So we need to be very aware of what the country average means. The second point that we make has to do with the fact that lessons or education reforms that come from looking into what other countries are doing are not often valid or applicable to US education. We'll provide a few examples in a few minutes, but the main idea is that it's very challenging to craft education policy based on international comparisons. The third point is that because there is not such a thing as a US educational system, we should be looking at performance of students in 51 different state systems or educational systems that deliver, fund, regulate education within the country. Moreover, since students in some of these states are performing as high as students in the highest scoring countries, we would do better if we took lessons for how to improve education policy if we looked at what these states are doing. So just a few data results and comments on each of the points I'll cover the first two and I'll turn it over to Martin to focus on the by-state analysis. There's indeed important signs of increasing and high performance within the US. If we look at variation depending on the socio-economic background of students or the family academic resources as we're doing these studies in our study, we can see that this advantage SES US students looking into the US as a whole have made larger gains over time in both piece and teams and those gains have been larger than gains done by the same low SES growth in other countries, especially in the higher scoring countries. Low SES students in some states bringing in the variation across regions have made very large gains in math which is traditionally the weak subject for US students and those gains have been also have been larger than in other countries as well. In some specific states high SES students example, Connecticut, Minnesota, Massachusetts, North Carolina, Indiana and Colorado perform at least as well in math again as students in Quebec, England or Finland. There's more evidence of high and improving performance and just bringing in more dimensions of variation here. Students in Massachusetts and Connecticut will use the data from PISA 2012 in reading perform roughly the same as in Canada, Finland, Korea, Poland or Ireland and they score higher than in France, Germany or the UK. This will be looking at the average performance of students in each of these states. Overtime which is one of the most important dimensions in our study. It will look at the period 1999-2011 using data from teams, mathematics, students in Massachusetts, Minnesota and North Carolina made gains at least as large as in Finland, Korea or the UK. Why it's important or challenging or difficult to learn from what other countries are doing and apply that to education policy within the US. What we want to say here is that educational reforms suggested from high scoring countries are not based on compelling enough evidence on why some countries score higher or have improved over time. So that's the main obstacle and limitation. In addition, the social and educational context in which education outcomes are produced in the different countries are very different across states, across countries. Looking into some examples that tend to be used as examples, role models of education. For example, Finland offered its students five years of preschool. It has a low poverty rate and it has a much higher equity across students than the US. And Korea families invest a lot of time and resources into cram and private tutoring. Another reason why it's hard to import the reforms from these countries is that some of those reforms are just not relevant to the United States characteristics. For example, Germany and Poland track a viability students starting in high school and some other countries do it even earlier. This is a very complex topic and there has been a lot of conversation and a lot of misutilization of international tests in a way. And a review of the critics, we wanted to announce that Martin Carnoy has just published with NEPC the National Education Policy Center a paper that reviews many of these critics and this paper has been released as of today. This is the end of my presentation and I'll turn it over to Martin to see what potential solution additional analysis we can do to learn more about education policy in the country. Thanks Emma and thanks to all the people at EPI for helping us get this out so quickly. So to summarize so far the main argument that we're putting forth that Emma laid out is that we do have a much more progress in the U.S. education than is usually focused on and I just want to make the point that we'll discuss later that we've had a decline in scores announced on Wednesday in the NAPE but if you caught the beginning of the headline what the shock was was this was the first time in 25 years that there was a decline in math scores and as far as I can tell everybody who's been telling us about how terribly we're doing on the international test has failed to tell us that we've made this tremendous increases in math score in the United States on our national test. So the second point that we want to do is it's much more useful since we have such high variation in the United States to shift the discussion to variation within our own country and to understand what we can learn. The main point of our paper is how do we learn from what has happened over a long period of time in the United States in among the different states what can we learn from those states that have done well over time particularly well and these happen to be states that compare very well with other countries so what can we learn from these comparisons with the U.S. States? Well we have data over a long period of time for our fourth and eighth grade mathematics and fourth and eighth grade reading and what we did this is very difficult to do this but we took state adjusted scores for the NAEP has been taken by all states since 2003 it's mandated under I believe no child left behind that they have to take the NAEP. But before then a large number of states also took the NAEP all the way back to 1990. So what we did was we did we adjusted the scores using the individual data the individual data set the micro data we adjusted the scores Emma worked on this for a year adjusted all these scores and corrected so called corrected adjusted the scores for a lot of variables of individual differences among the family backgrounds of the kids whether they were in ELL courses whether in special ed courses and we also adjusted for the school composition so on the idea that schools that concentrate kids who are very poor concentrate kids who are better off that those schools they have peer effects so we corrected for that. We also did an adjustment for some of the teacher variables that are available in the NAEP. So we took these adjusted scores for eighth grade math from 92 to 2013 and we did fourth grade math and reading and eighth grade reading from 2003 to 2013 and then we asked the question why do some states do better than others and we tried to find out at these what we had left the scores that we had left once we adjusted for these different variables that probably do not have to do so much with the quality of schooling once we try to take that out do these differences that are left are they related to other things and to keep things simple we're just going to focus on eighth grade math in this presentation so much of the variation as you might guess in the all the scores math and reading are due to socioeconomic differences among the students and socioeconomic composition of the schools. Once we took all that out and we looked at eighth grade math we found out that from 92 to 2013 the states that were in the top 10 gaining states they made gains on the NAEP that were twice as large as the bottom 10 states 1.6 points a year versus 0.8 points a year. If you take that over 20 years it's a huge difference it's a difference of one standard deviation it's a one standard deviation of the NAEP score over the I'm sorry half a standard deviation over the 20 years that's huge half a standard deviation you'll never find an intervention that will produce that larger change in among the entire literature on this so the next thing we did was to compare states neighboring states that had big differences between them that look very similar to each other on all other grounds so here's Massachusetts in Connecticut and you can see that after 2003 Massachusetts first of all their paths are pretty similar before 2003 and after 2003 all of a sudden Massachusetts starts to make big gains now they both go down I just extended it to 2015 making a kind of correcting using the same corrector as in 2013 but the dash line represents the observed score and the solid lines represent the adjusted scores for kids differences among the kids in the two states now it's all corrected to the U.S. average so you can see that both Massachusetts in Connecticut are below the observed scores because these are high socioeconomic states these are states that are pretty rich states so correcting them to the U.S. mean on who the kids are knocks them both down but that's less important than seeing these trajectories and you can see that the gain between in 10 years is a half a standard deviation Massachusetts exceeds Connecticut by half a standard deviation these are just neighboring states now we can learn far more from this comparison than worrying about why Connecticut didn't do as well as Finland makes much more sense to just look at a neighboring state and look at what their policies are now here are California and Texas okay and what's interesting about this is that I'm sorry this is not California and Texas some it's not California take this is what some of it's the wrong it's the wrong grab no it's not California Texas what yeah it's the wrong grab yeah it's okay I do we have California Texas way down yeah I can show you some other states are you interested in this at all yeah okay so so here's the people love these kind of comparisons that's why we did them so and they actually have have policy sense that's what's really nice about first of all it's over a long period of time it's 20 years so here is Minnesota and Iowa neighboring states okay and you can see I didn't extend this one but Iowa goes down a little bit Minnesota goes down more but the fact is that again starting in about 2000 Minnesota starts to make bigger gains now the difference between these two states is not as great as between Minnesota between Massachusetts and Connecticut but we know that Minnesota did had different education policies in Iowa I was sort of a laissez faire state you know they didn't want to ever test kids at all they gave the Iowa basic skills test eighth grade that's it and then finally they had to test because of no child left behind but basically they have been a laissez faire state very similar states here's North Carolina Kentucky and Tennessee and what's interesting about this by the way is in 2015 Tennessee does continue to go up a little bit and North Carolina drops a lot and Kentucky drops a little bit so now Tennessee is actually higher than Kentucky so what first of all why did North Carolina go up so much in the 1990s well my answer is Lamar Hunt I was pretty clear he put in a lot so what did Lamar Hunt do well no people have studied that but they haven't studied in a systematic way to see if those policies actually were then adopted by his other states Kentucky has been very much of a reform state and they have made gains but you can see that Kentucky's scores did go up and have been going up as much as North Carolina in the later period but somehow there's still a big difference North Carolina scores among the highest states in the United States it's not a rich state I mean it's not a it's very diverse state so here's New York and New Jersey we have the same thing Governor Christie should love this graph hello probably has very little to do with him but it seems things started back much further back so anyway the main point of all this is to say these kind of comparisons are much more interesting than the the ones that are usually made international they have much more sense because you can actually look at these states and look at all our states and see these are more or less going to some kind of school system face the same kind of labor markets for teachers the training programs for teachers are very similar so what have they done what are these states done and we just want to shift attention in that direction so one of the interesting things that I want to add at the end here is to say that even these residuals even these state differences correcting for all these variables that when we try to figure out at a first cut what might explain this what is this correlated with across states and one of the things that's interesting is that it is these differences that are left once you correct for all these other variables they still the poverty of the state the child poverty in the state is still correlated with what's left so actually poverty seems to hit how well kids do at three levels one at their individual level poor kids at the individual level don't do as well in school on these tests as kids that are from better off families secondly if you go to a school which is high concentration of kids in poverty if you go to that kind of school also a big effect versus a school that has few kids in poverty and now if you live in a state which has a high level of child poverty both the high income kids and the low income kids are going to do worse than in a state in which there's low levels of poverty just living in that kind of state so three levels the poverty effects how well schools do how well kids do in school and accountability the strength of the accountability is also correlated with how well the kids do I'm sorry to say that I'm not a great fan of accountability but the fact is that it is positively correlated and significantly correlated so states with you know the strong accountability like Texas North Carolina states like that they did better on average okay things that were not correlated things were not correlated were teachers union the I guess it's the strength of the teachers unions measured by an EPI index of strength isn't it no it's the proportion of the union of CDA yeah of members cobbled by collective bargaining okay there's quite a variation in the United States on this so that's not significant neither is unfortunately spending per student and the third one was oh the percentage of adults with higher education in the state okay so poverty yes not high education of the parents okay I've talked to a long already but so we haven't been able to explain why Massachusetts did better than Connecticut or everybody will have ideas here why this is true but we I just want to tell you everybody has ideas about all this but nobody's done a systematic analysis to do these comparisons that's the point it's going to have to be qualitative but it's going to have to be convincing and not just off the top of the head just like with these 2015 scores I saw at least 10 different reasons why these 2015 scores went down but it's not systematic and just to tell you I think we truly believe that the purpose of our paper was to get people to focus on this and to unravel this puzzle because this is going to be much more interesting the answer to this to these questions is much more interesting I mean it's less romantic to fly to Massachusetts or to Texas than to fly to Finland or to Singapore but the answers are going to be much more obvious flying to Massachusetts and Texas than going off to there okay I think that's thank you very much thank you very much Martin and Emma and Mary Beth and I want to start by thanking Larry, Michelle and the EPI for the opportunity to participate in this panel I really think this is a very significant piece of scholarship and it adds considerably to the discussion about what's going on in American education so I want to congratulate Martin and Emma and also Tatiana Cavanson who apparently is in Moscow and I hope she's able to view this as we stream it so I want to make five points very briefly and the first is to acknowledge that we have very real problems in our schools none of us can pretend that we don't I will contend that we've done a reasonably good job with the students in our schools that the schools were originally designed to educate but now we face a very new challenge and I think it's an unprecedented challenge in the world for the first time in the history of the United States the majority of students in American public schools are both low income and children of color and I don't know anyone else that's contending with that and this is a very real problem that we need to get on with and it's not helped frankly by hyperventilating about highly questionable international assessments you know we need to get into the meat of the challenges that faces not the challenges that draw people's attention on the front page of the newspapers. The second thing we need to understand is a point that Martin and Emma I think made very clearly the American educational system is not one system at all it's 51 different systems of education made up of 50 different states and the District of Columbia and if you add in the possessions we can we can start to get very quickly up to 55 or 56. Now people who understand American schools are involved with them on a daily basis take this for granted we think it's so obvious that we don't even need to state it but unfortunately those who who are more sort of concerned with national systems don't understand the dynamics of how this system works and therefore the more sensible ways to develop policy solutions to address these challenges and I might say it's not only 51 different systems we also have some 13,000 different local school districts implementing the decisions of these 51 different systems so the implementation challenges are also huge. In that light I think there's an argument to be made that we should follow Canada's approach specifically with respect to the TIMS assessments the trends in international math and science survey. Canada doesn't report national results under TIMS instead we get provinces reporting their work and as I understand it from this report of Martins and Emma's at least some seven states have taken have participated in state level assessments in either PISA or TIMS over the years so there's some precedent I think for that in the United States and I would like to encourage the US Department of Education and any of their officials who are here today along with any state officials to take some of the millions and billions of dollars that they've been spending on assessment and direct them towards encouraging state level participation in these international assessments and reporting on a state level basis I think that would be extremely useful. Third I'm very impressed with the manner in which bringing it back home focuses on a particular set of nations and Mary Beth mentioned that I developed a report called school performance in context the iceberg effect and it focused on eight nations and the United States that altogether account for more than 50% of global domestic product so naturally you won't be surprised to learn that when Martin and Emma's report focuses on eight states plus the United States that are either high scoring or post industrial nations since it so closely tracks sort of the thinking that was in my report I want to applaud them for being so smart so and I think there's very good reason to focus on a more limited set of nations if you're going to be in the international comparison business what can we possibly hope to learn from a small principality like Lichtenstein with 5600 students when we're worried about our 56 million students what do we hope to learn from dictatorships like Kazakhstan and Shanghai which have five year plans for their educational systems when we have trouble actually getting people in the United States to agree on a common core of what we want people to learn and why would we think that we're going to learn very much from a religious monarchy where the monarch appoints the parliament and he appoints all of the ministers in his or her government I guess they're all he's for the most part I think so why are we worried about what's going on and cutter and other other other religious monarchies I think high achieving and post industrial nations are the appropriate level of comparison for the United States this paper's construction of a family academic resources index is far indexed Martin was talking about creates a proxy for poverty and disadvantagement just as and just as the iceberg effect argued that you have to take students life circumstances into account I think this far adjustment is a very valuable introduction to the conversation and I want to emphasize what Martin said towards the end of his his findings about this tri level effect of poverty is first time I've seen this I think this is a significant new finding to quote from the report the effect of poverty on education performance is a three level effect in addition to the well documented impact of individual and school level poverty state level poverty put students in all socioeconomic levels at additional educational disadvantage and I think this is very significant I saw a report come out recently from Rutgers University that sort of cross indexed ethnicity and race against this this issue of child poverty and it turns out that African American students or students of color generally in the United States are ten times as likely as white students to be living in communities of concentrated poverty and surely we as a nation need to be paying attention to that as we think about our educational challenges and finally I love the pairing of the states that Martin just just ran through and I hope state policymakers will pay a lot of attention to it and you know in some ways this data about the states is sort of like data to back up the gossip that we all have about the states but it's it's it's real data the paper presents a very complicated analysis overall it's different tests different international tests different subjects different years different nations in different assessments frequently cross-cut by the socioeconomic measure that Martin and Emma have introduced into things so this is not light reading that you're going to take to the beach with you but it is very reassuring to know that some states American states are competitive with the highest achieving nations in some subjects and grades especially when socioeconomic status is taken into account but and so there's you know this complex analysis is accompanied by this the brilliant simplicity really of the final portion of the paper that pairs neighboring states with each other and Martin just went through them so we find Massachusetts paired with Connecticut New York with New Jersey California with Texas Minnesota with Iowa and North Carolina with Kentucky and Tennessee so there's a really rich diversity of states look that in this analysis and I would hope that state policymakers would look at that because I think it's in comparisons such as this that we're going to find out why states whether they started out low or high scoring on NAEP made significant gains at both the low end of the scale and the upper end of the scale and in contrast to the political uses to what NAEP is put I want to say that the NAEP database is one of the most significant national assets particularly at the state level that we have in this country for for understanding what's going on in our schools we also find that strong accountability works Martin just went through that the child poverty is a huge issue and now we know it's a three-part issue and that union strength seems to be irrelevant in terms of outcomes and we can't really what we might all like to bash unions and particularly sometimes superintendents go to war with their unions we can't blame unions for the situation that we find so let me conclude by quoting directly from bringing it back home quote the lessons embedded in how these states increase student achievement in the past two decades are much more relevant to improving student outcomes in other US states than looking to high-scoring countries with social political and educational histories that differ markedly from the US experience and I think that says it very well and it sums up the report in a nutshell thank you now am I on is my you can hear me okay okay thank you so much this is really really interesting and I do have a few questions that I want to ask right away and then we'll open up to everybody and their solutions and their their contributions I can tell you that if I was still working at USA Today and I brought this report to my editors and told them what it said what they would want me to write about is that some states are showing great improvement over this this idea of the improvement that's happening and it's such a good newsy story that I almost wonder why hasn't anybody done this before I mean it sort of seems like we should have been looking at this a long time ago so I'm intrigued by that especially because of the you say the brilliant simplicity so that's one question is how is it that we're just starting to disaggregate this data at this stage my second question is as somebody who's studying how the United States interacts with the rest of the world I wonder about the message we might this might be sending on some level that that we have nothing to learn from other countries and I hate and I and I would hesitate to I would like to have you sort of clarify that a little bit okay so whoever wants to respond I'll say something about I think it's growing the number of analysis that start disaggregating the data as Jim mentioned yes we produced a sophisticated analysis which was not so complicated to do when you have the support and I forgot to mention this at the beginning from people like Emmanuel Cicali who is here from the NCES who was willing to help and answer any questions that made the analysis less complicated to us so thank you Emmanuel I didn't mention at the beginning so I think it's it's because there's a growing demand in or growing awareness of how these disaggregated data can inform us can help us we are shaping our questions in a new direction and therefore we are shaping our analysis in a new direction this is for me it's one of the the the lessons as we paid countries for learning I think we also need to do a better job paving our questions with the data that we have and the US is perhaps a privileged place to do analysis on because it has all these information and before perhaps Martin adds to what I am saying I don't think this report should be interpreted as saying international comparisons are not valid they are valid for specific purposes they are they haven't they haven't been well used for the goals that we are discussing here so it's they they have a lot of interest we do international comparisons all the time it's more about the utilization of the comparisons I think you had two questions there one is why is the good news story not a usual story and my my theory is that we are edu masochists and we not we but there is a significant group of people who apparently gain some political advantage by bashing the US educational system the the question of why nobody has done it before is that sometimes the obvious is the most interesting and in fact there are many countries like the United States Jim mentioned Canada Canada on the Tim's as he mentioned only reports a provincial data does not just not report a Canada data now on the PISA they report Canada but you can get the proof you can get provincial data on the PISA also if you have a friend in Canada who will give it to you I'm serious it isn't just published like that it isn't on the regular available data the NAEP data have been available at the individual level and people have used it for various stuff but they have used it usually just one year they've done analysis of one year it's it's a lot of work and we can attest to this it's a lot of work particularly if you want to go before 2003 where it's not one data set is a data set for each state and you got to combine it you got to mesh all those states so it's it's not easy if you want to if you want to go before 2003 so it's a lot of work the that's why we didn't do it for fourth grade reading and eight grade reading in fourth grade math because so we only went back to 2003 so it is a lot of work to do that so maybe it's a question of academic laziness but I think rather it's because sometimes the most obvious thing that there are big differences within a country we're doing the same thing for Brazil which is 27 states that we can get national test data for Brazil from 1999 all the way to 2013 and we've done that and there's huge variation in Brazil the Germans will not release their land data there's their state data they don't release it because they're worried about the north-south difference the northern states do much worse than the southern states and so the states are very I just had this explained to me by a German colleague the states are very proprietous about their data we don't do that we're a free access data society so it's just a question of how hard you want to work I think when once we realize what we were seeing we said my goodness you know this is so obvious we should be looking at this stuff and and looking in versus looking out and there is a lot of good news here a lot of good news until this year the test scores have gone up regularly by the way even in 2015 a number of states made positive gains and not only that but the fourth-grade reading in general didn't go up significantly but didn't go down more of this will yield a lot of policy insights that we didn't have before if I could chime in on those two questions too I first became interested in international assessments in a real way around 1998 when IEA the people who produce Tim's first issued a fourth-grade reading assessment called Pearl's progress in international reading literacy which is still ongoing and that first assessment showed that the United States in fourth-grade reading was second out of 28 countries second out of 28 countries Sweden was number one that piece of good news was buried on page 17 of the Washington Post but every time the Tim's data came out showing us somewhere in the middle of the pack we got all of this pearl clutching about the future of the United States being at risk so the bad news appears on the front page and the good news if it appears at all is buried in the back sections because I think there is this well-funded echo chamber of failure about American schools and it feeds into perceptions that people have about oh yeah I remember when I was in school that was terrible why has nobody done this before I think Martin did a good job of explaining it this is very very hard work and you need some appreciation for the complexity of the work before you can even begin to do the analysis and most much of I won't say most of but much of the research that we see being reported in the newspapers much of it is advocacy masquerading as research and I say that about some of the things I've produced myself by in large nonprofits small nonprofits and universities don't have the resources that they can pour into these kinds of complex analyses and that's why I think we're seeing it here for the first time I do have one more question with my graduate student head on what would critics of this or people who disagree with with your the points you're making what would the critics say about your methodology do you have any any areas that you think they might poke holes in or try to poke holes in well I I presented this rough roughly speaking the results to a to a group in Australia about two months ago and a very well known economist of education who's among the edge of masochists was there my colleague at Stanford Eric Hanna check and he thought it was a great analysis so I mean terms just of methodologically I mean he had one point to make to do something a little bit different but it didn't make any difference so I think the main critique will be this it won't be of the methodology it will be that we are looking for good news when the news is generally bad and I think it's fair to say as Jim said I want to reiterate there is a lot to do in American education not every state is Massachusetts I'm just talking about 8th grade math here is Massachusetts or Texas or North Carolina and Minnesota there are these high scoring states not every state is that kind of state so there are a lot of states that are scoring quite low I mean there are lots of other high scoring states too as I said the top 10 you'll be these are corrected for differences in the in the family academic resources of the kids in the schools but even so the top 10 include by the way the gainers the game we focus a lot on gains I think it's it's also very important to focus on gains and not on level among the gainers I don't think I think Emma did point this out among the gainers are states that started quite low and states that started quite high normally when you look at gains you think that most of the gains will be those who start low because that's where the gains are to be made but that's not the case it's a mixture so you have Louisiana DC Hawaii all states that started quite low three of them then you have Vermont Massachusetts high scores that are up there too so the point is that we should also be looking separately perhaps at why the what the differences were in policies of states that started low and had big gains versus states that started high did they use different policies that they have a different ways of approaching educational improvement so the main critique will be you know on average the US still does badly in mathematics so that'll be the critique you know and the critique and in and the biggest critique by the way which we point out is that on average the kids that do the worst relative to high scoring countries are those who are advantaged in mathematics advanced social class kids do do worse in mathematics relative to other countries than our disadvantage kids our disadvantage kids on average have made much bigger gains than the advantage kids on these international tests and also score a lot lower relative to other countries so I think their critiques to be made I mean they're saying you know we're still as a country not doing great in education but so that's not our point our point is that first of all we're making gains that have not been talked about and even on the international tests and secondly we have a lot of educational systems real systems not the US system which is not a real system real systems that are making tremendous progress and so why aren't we looking at those systems instead of lamenting all the time how terrible the whole country is doing which is a point but it's not helpful to figure out how to do better the point is how do you do better how do you keep doing better where do you look where do you look for examples well Finland's just not example Korea's just not example American kids are not going to spend six hours a day outside of school studying and doing cram schools I just talked to a student of mine a Korean American born in the United States you know how she spent her summers every year going to cram schools from 7th grade through 12th grade cram schools every summer my grandkids go to soccer camp those kids go to cram schools and what do they teach them in cram schools I asked her where did you learn in cram schools take tests how to take tests basically test after test after how do you how do you negotiate these tests oh yeah okay by sending my kids every year to learn how to do tests they're going to do better on tests but it's nothing to do the quality of education has to do with what and these parents in Korea are spending $8,000 a year on each kid to send them to tutoring in cram schools that's six it's the average cost in Korea of schooling is $5,000 a year per kid now at $13,000 they're spending more or less what Massachusetts spends on school and I'm a do you have anything to add I I'm not sure if this could be a critique or so what question but one of the things that I will report does not include is the why right as Martin was saying we we we propose how to do we we provide a systematic way of looking for those answers but up to this point I'm just getting ready for the why is from people but there's no analysis of of or answer to why is it so I see that as both potentially a limitation of you did all of these you are ready to answer those questions but we don't really know well that's a great way to segue into audience questions and that hand up one of first I think there's a microphone I want to go back to the question that was asked by Mary Beth earlier why is it that nobody has done this type of analysis before at the department we have this rich data available and we strongly encourage the type of analysis we really want we really want researchers to go in the data and make hypothesis slice and dice it the way that they think fits and then do the research and publish we strongly encourage it up of research that's the point that I want to make in Emmanuel can I I'd like to add a sentence to what you said a question when are we gonna get the 2015 individual level data well I hope it won't be two years from now well I can I know I know I hit mine was an encouraging question oh there was a question from the front here you I guess mathematician great school math and I like a possible name a I think Bill Schmidt who's not here would totally agree with you he also said exactly the same thing that in Minnesota in Massachusetts they made real effort to write real math standards and that this had an impact he he I can represent his view by it's saying that he really believes that the main thing that explains most of this is opportunity to learn which translated means a better curriculum in mathematics that allows the teachers to teach kids better math the only thing I'd add to that is that it's easier to do that in states like Minnesota and Massachusetts than it is in in high-poverty states where teachers may not be quite as well prepared as in Massachusetts in Minnesota to do this hard to do this higher standard curriculum now the teachers are now better teachers actually have to pass and the rest of you are going to say something quickly about Finland yes well both of them does not have a good math professors engineering colleges in Finland are complaining loudly that students are showing up not knowing basic algebraic and written the calculations and why is it that they appear first on pizza anybody want to say anything about that and then we've got another some more questions from others I'm not so good into looking for education when I say the native data say that's only about a third or 40% of students are efficient or above there's a one one word to describe that and that's absurd and that calls for major big time improvement in the education system and these small gains that they measured over the past whatever 20 years they're too small I mean we need some way to jump so that the number of proficient you know I still feel it needs to be two out of three I do have a comment on that the one word to describe the native proficiency figures is the day proficiency figures are nonsense the nape the nape proficiency definition was set politically not by not by analysts and it was deliberately set to make the schools look bad as early as the 1990s people in NCES when this proficiency benchmark was first set stated very clearly that the nape proficiency benchmark was not great level that people who you and I would normally consider to be proficient by the common understanding of that term could not meet the nape proficiency benchmark because it was an aspirational level so they have twisted all meaning out of the term proficient and misled people in the public including professors of education about the true results so my one word response is that that those results are nonsensical and I say that having the greatest respect for the technical people who administer nape and having the greatest respect for nape as a as an assessment but the proficiency benchmark was set by a group of politicians who sit on the national assessment governing board they are not assessment experts they are not mathematicians they are not economists they are not researchers and they were setting a political level not at not at not an analytical level I've gone on a two-grader lens good morning friends and colleagues my name is Saret Kaminsky with the American Federation of Teachers and I just want to say thank you so much for this valuable research and it's very interesting to see these links to poverty on our website at go.aft.org forward slash equity we've actually mapped trends in child poverty by county for every state and also the district of Columbia so you can take a look at that work there I also really appreciate your comments about unions representing 1.6 members we also find that high-performing countries have strong labor management collaborations so appreciate your comments I do have a question we've found that the US is the second lowest rated OECD country in resource allocation and I was wondering if the panel could please comment on your findings on resource allocation and disparities between districts and how stakeholders can do better in advocating for equitable funding and funding reform well we didn't find a relationship between spending for student and these residuals however let me comment about about what's happened over the last since 2003 because I just looked at it this morning the average spending per student in the United States has not risen since 2003 corrected for inflation so one could argue if you look at these data I don't know if you know this argument maybe this is just sort of esoteric in group stuff but there has been an argument among the the agimacicus that the we've been spending a lot more on education and getting no gains and they usually throw up the reading scores to an ape reading scores to show that to show the increasing cost per student but no gains if you look at these data in math and reading over the last 10 years you can see gains in both and the math gains are very very large the reading gains are appreciable but not large so if you look at that and the fact that cost per student have not gone up over the past 10 years you can make exactly the opposite argument now that in fact we're getting much higher productivity with it no increase in resources and for either reading or math and so you can just turn that argument on its head and make it Larry will appreciate this make it exactly analogous to the fact that productivity has gone up in the labor force but wages have not gone up so this exactly analogous in the education sector we know the teacher salaries in real terms aren't going up we know that cost per student are not going up and yet productivity seems to be rising and then the question is on the 2015 data it's already been mentioned by at the NCS conference yesterday the presentation of the data on Wednesday that in fact one of the factors may be that when you do not spend more over a long period of time like ten twelve years then you may affect test scores even though we don't find that relationship I live in a state that clearly has been affected by the very slow growth of spending over many years our California test scores have relative to the rest of the country have not done very well and it's a question of whether oh it may be not from year to year or from period to period small periods but if you do this for a long period of time you probably do affect the quality of education and although we don't find evidence of that not all statistical analysis picks up all the inherent things that are happening I actually have just are you Lily hi we've had classes together okay we're gonna divide the answer on this so I'll do the accountability part back in in the early in the late 1990s early 2000s the consortium for policy research and education which is now located at the University of Pennsylvania a woman named Margaret Gertz and her colleagues developed an account called every state and found out what they were doing now this was pre no child left behind and they established on based on their data Susanna Loeb and I developed a five rubric of five items and based on what states were doing whether they were testing how often they were testing all the way up to whether they had a high school exit exam which I had to pass in order to graduate and based on that you got a score of zero to five so states like Nebraska and Iowa got zero and states like Texas I believe Maryland few other states got fives at that time so those change with no child left behind more and more states did this kind of stuff so in in this paper I must say and they've been papers written since that have tried to measure accountability and they will continue to do so and they will get better better measures of what would be called strength of accountability so based on that original index we adjusted it over time to try to take account of the fact there was another one done in 2002 and then people did one later to try to take care of no child left behind so we used in we use these I think three three points in time of accountability among the states to in our analysis so just strength of accountability so for the second question when we propose we did it in in this light and it's better justified in the paper is a combination of qualitative and quantitative work and that some going to mention elements that you would consider more qualitative or more quantitative in a way we are advocating for a change in the way in which the questions from policy to research and vice versa are framed and these includes elements of sitting in around a table policy research teachers and all participants in the in the education sector or the provision of education that ask questions such as how is the curriculum aligned with what we want to do implementation of common core this party is in funding equity within the state preschool availability for kids a number of reasons that need to be argued there needs to be a theory behind the reasons that are suggested and therefore that work we consider more qualitative work the identification of the things that might be behind the trends and the high performance of some states versus other states that should be tested in the end with quality quantitative analysis so there can be a very long list of like Martin said when we saw the result the 2015 results he said there I saw 10 explanation so we see a point and we find 10 potential explanations we way what we are arguing here is to think of the 10 2015 100 explanations to test them with quality so if I could add something on the accountability issue I don't think any of us got into the field of education eager to be accountability experts but I think in this day and age you can't say that you don't favor accountability however I can say that I don't favor simplistic accountability of testing kids and thinking that that's the be all and the end all I think we need quite complicated accountability systems we have in state after state state legislatures thumbing their nose at court decisions calling for equitable financing of schools earlier this month our roundtable met here in Washington DC and we actually heard from a school principal in Las Vegas whose leads Whitney elementary school enrollment of 600 kids 85% of her enrollment is made up of children who are actually homeless and by the time she graduates these kids she's not quite sure where they're going many of them wind up working on the streets of of Las Vegas doubtless serving people who are calling for accountability the social circumstances and the poverty in which many children in the United States are being raised are just shameful if you look into them and most of us drive by it on interstate highways and we never see it and I would like some public officials to be accountable for properly serving these children not only in school but out of school as well I know you asked a question and there was a hand up right at the beginning that I somebody who raised their hand right in the beginning not okay all right so how many people have questions because we want to just keep going with questions or do you want to take all the questions and then we'll excellent please my question goes to your point that there's a correlation between state level poverty or the poverty of a state the wealth of the state and the student score child poverty child poverty my question is whether the opposite is also true and thinking of states or jurisdictions where there's a huge divide between the rich and the poor I'm from DC there's very wealthy people and there's very poor people and so did it translate for the poor children in comparatively wealthy states to help them living in a wealthier state or were they still stuck where they're at we'll have we'll get some several questions out so that we can sort of Jim I'm sorry I mixed up Lamar Alexander and Jim thank you and Lamar Hunter's Oilman wrong to characterize it as education masochist because that that suggests it's a reflection of somebody's personality but this is actually a very agenda you know that there are people who are always looking at how our schools are failing because they have a different policy agendas that they are so I think education masochist so you would call them education satyrs there are a couple more questions over here we'll just get them all out because that is also a factor citation for Mr. Harvey's book but there's a significant study University of California showing that expenditures do have an impact on long-term outcomes and it's going beyond the test scores I think that ought to be there. I need to do follow up on your other 10 or 15 other topics and hypothesis but is this a first of a series of papers? And one last question then we'll go. Oh dear. That's one other person over here. But it sounds like these are public schools are there private schools in Asia? Yes. Yes private and charter schools. Okay so. A lot of us well we know I think by the way that Massachusetts has been the intent of the analysis and knew the news about Massachusetts frankly. Remember in some countries some states were doing well. But the concern among some of us is what are the implications for these local public schools? All right so you have your questions. Anybody want to take a first crack? I'll take on the question whether we adjusted by race ethnicity with it and we also did it for the proportion of students of different races and ethnicities in the school. I'm saying this and I'll connect with my answer to the last question which is the fact that race is only an important factor as long as it correlates very strongly minority status correlates very strongly with low SES, low social class. So the important controls in our model are individual and school level social class, socio-economic status. In that and going to the last question that was posed to us we are obviously very concerned about inequities, inequalities that we know exist throughout the school life of an individual before children get to school and they persist after they leave school. What we believe and the point that I was trying to make during my presentation at the beginning of it is that if we just focus on a score on an average score both the success and the problems disappear in a way. They compensate one another and we have one average point that is not really informative. From the disaggregation, from the knowing which individual school and state variables are important for student performance we want to make the case that we can learn from the successes and apply them to some of the problems. In theory that's something easier to do than it is in practice but it should be through answering this kind of what education policy in different places is doing, what public policy in general looks like across places that we might be able to transfer some of the success to the highest needs that schools and students have. Jim and then we'll let Martin do the cleanup. I don't know if living in wealthy areas helps low income students. Actually I don't think it should but I don't know. My guess is that this has little or no relevance to making post-secondary education free but perhaps Martin can comment on that. Emma commented on the race and ethnicity issue. I'd like to address your issue about the achievement gap. Before this panel convened this morning we were talking in one of the offices and we discussed the fact that for a long time we've all known about Massachusetts. As you said, what I liked about this study was that we were introducing North Carolina and Texas into the discussion so I think this is a much richer discussion and opens up the possibility of investigating in the ways that Emma talked about what explains the success and whether it in fact does anything in terms of closing the achievement gap. One of the things that interested me in the round table report about the iceberg effect is we looked into the achievement gap as reported by OECD in these nine countries and we talk about the achievement gap as though the United States is the only country in the world that has such a thing but in fact it's apparent everywhere and the largest achievement gap turns out to be in Japan according to this OECD data. So that's a very complex issue that obviously requires a lot more study. I think not only here but internationally as well. Let me follow up on that and then you'll quickly with some other things. The achievement gap is a very complicated issue and I'll characterize it very simply. Massachusetts has a higher achievement gap between the lowest scoring, the lowest group, the disadvantaged kids and the most disadvantaged kids. Alabama has a much lower achievement gap between, much lower than between the highest scoring kids and the lowest scoring kids. Florida has a much lower achievement gap. But on the Tims, for example, in which nine states took the Tims in 2011, Alabama, Florida and Massachusetts among them, the lowest group, the most disadvantaged kids in Massachusetts scored the same as the most advantaged kids in Alabama. So this is the question that I would pose back to you. Would you rather have, if you're from a low family academic resource family, would you rather go to school in Massachusetts or would you rather go to school in Alabama? Your gap is lower in Alabama but you're scoring very high in Massachusetts and I think the answer is obvious. That's why I feel, and there's another reason why achievement gap is a, I wouldn't call it a nonsense variable but it is an interesting, no it's an interesting variable but it's an interesting thing to focus on but think about this. One of the highest, besides Japan, one of the highest achievement gaps is in Sweden and by the way Japan falls in the same category. If you score low on a test, on tests in the United States, in terms of your future income and social position, you're in a much worse situation than in Sweden. In Sweden doesn't make the difference in income and status because of the tremendous safety net in Sweden and the kinds of policies they have even under conservative governments, the difference for you is not great. First of all it's almost universal, higher education, post-secondary education and all these opportunities later on are very equal because there's equal income distribution, equal, more equal wealth distribution, more equal everything out the end, this is what EPI is so concerned about. In the United States we have a lower achievement gap than in Sweden but the consequences of this gap are much greater than in Sweden and we talk about test scores and we should be talking about test scores because they measure something about learning of kids in school but we should never stop talking about the link between these what's going on in schools and what's happening out in the economy. I would rather be getting low test scores in Sweden than in the United States. That's the point. Any last concluding remarks, Martin? I would just say one thing on race and ethnicity. One of the most interesting findings that we have which we didn't report here because we're going to do other papers on this is that once we control for the social class composition, this is what Emma was saying, once we control for the social class composition of schools the race composition of schools adds nothing and this is really important because there are people everywhere in the U.S. who focus on race composition of schools as if a high concentration of blocks in a school really makes a difference. It's their block and it turns out from what we're showing what we're indicating from these NAPE data that it's not the concentration of the ethnicity in the school it's the concentration of how it's a social class issue and I think that's very important. That doesn't mean the individual race still has an important effect on test scores, the individual's race, but not the concentration once you control for SES and I think that's really another important result like this poverty thing. The nice thing about doing this kind of analysis that all these kind of things bubble up as you do this in terms of trying to understand what goes into test scores. So we'll continue to work on these data and as soon as the 2015 data come out, we'll analyze them too. Thank you everybody, there's a lot of opportunities to talk some more.