 Please welcome Brian Jacob. Hello, good morning. It's great to see so many folks here. Again, I'd like to thank all of our doctoral students. Okay, I thought for you I was going to have to bring it down a little bit. Down, I'm the middle. You can see how much in charge of anything I am. So, yeah, just again a thanks to all of the very hardworking students and postdocs and staff who have helped make this event possible. So, let's see. I'm going to jump right in. I think try to give some discussion of this. Our initial finding is the Michigan Merit Curriculum. I'm going to say this a few times as I go through the presentation, but one thing that's important to keep in mind is when we say the impacts or the effects, what we're really talking about here is just on the very first cohort of students that experienced the mandate of the curriculum. So, as you know, the first group for whom the curriculum was binding was those who were graduating in 2010-2011. And so because we're looking at things like high school graduation and not just four-year graduation but five-year graduation and trying to look at college enrollment, not enough time has passed for us to look at these outcomes for subsequent cohorts. So, I think this should be useful information, but we shouldn't take this as the final word on anything. Okay, so just a little context. Where did this come from? What is the context in which this Michigan Merit Curriculum was proposed and adopted here in Michigan? It was really in the context of stagnant high school achievement across the U.S. for the past 40 years. So, some statistics here. Achievement has barely moved, and you don't have to take my word for this. Wow, this is slow. So, these are NAIT, National Assessment of Education and Progress, Achievement Trends in Math. Look at the line for 17-year-olds on the top. Compared to 1973, those students in 2008 were virtually the same. And reading, this looks at reading, and this shows you for 17-year-olds and reading, doesn't just give you the average or the median, which is the 50th percentile line in the middle, but looks at the very lowest achieving students and even the very highest achieving students at the 90th percentile. And you see, these numbers have barely budged at all. And so, I think this is people, this data is made possible by Jack Buckley and all the folks at NCES. And this helps us understand the lack of improvement. I think that was one of the motivating factors in the adoption of the MMC. High school graduation rates have not risen much nationally. So, this is something, first of all, they've been very hard to measure, surprisingly. But in the last five or so years, researchers have kind of looked more carefully and found ways to properly account for various kind of mobility and so forth and found figures that look like this. This is based on high school graduating classes. If you go back, I mean, for males, comparing the high school graduates of 1966 to 2006, that is 40 years. And we are at the same place. Female's been some improvement, but that's been mostly in the last, you know, five or so years. So, people look at these trends and I think are motivated to change. BA attainment is leveling off despite the large economic returns to college. So, some figures at SUE and others have put together before. You look, the red line is some college completed. The blue line is BA's, completed BA's. And that certainly was growing up over time. The x-axis here, the horizontal axis is actually year of birth. So, it's a little bit different. But if you look at the last, you know, five or six years on the blue line, you'll see there's not been much change in BA attainment rates across the US. So, okay. And then, as many of you have heard in the news, kind of the US performs about average internationally and that has some people worried as kind of economics becomes more of a global marketplace. What are some of the common policies that people, states, districts have tried to improve high school performance? When you go back and look at this, I was actually a bit surprised at how many different types of reforms there have been at the elementary level. When you look at the high school level and what kind of reforms have been tried, it's much more limited. There's been some organizational structural changes. There's some work, especially in earlier decades on career technical education, some dropout prevention programs. But by far, the most kind of common reform to stagnating high school achievement has been to raise standards in some form or another, most prominently high school graduation requirements. So, and this is not new. And one thing that I think it's important to understand the context of the MMC is that we've had policies like this for a long time, starting in the 1970s, minimum competency exams of basic skills that were starting to be required for graduation. The 1980s, for those of you who are kind of active in education at that time, you probably remember the new basics, which is the push for all states to raise the number of required courses to four years of English and three years each of math, science and social studies. And many states and districts did that. High school exit exams. And then no child left behind. I guess the last decade, our country's been mostly focused on no child left behind from the biggest perspective, which I think most people believe is focused largely on elementary schools. And there hasn't been as much national attention on high schools. That is changing, obviously. So what is the rationale for higher standards? One, and these are things that come up not only in national debates, but if you go back and look at the record in Michigan in 2005, 2006, when the MMC was passed, you'll find kind of these arguments being made among advocates. One is that it ensures students are prepared for college and are jobs in the 21st century economy. It kind of adds more coherence to the education system. One of the, I think, primary intended benefits for many is to reduce some of that variation Sue showed us before. We know there are big gaps in high school graduation and college enrollment even within districts and certainly even within high schools to some extent that might be due to variation in course taking and if we can kind of formalize and improve the course taking of some students that weren't taking rigorous courses in the past, maybe this will help reduce some of the gaps later on. The thoughts that kind of implementing these new requirements would spur other changes in curriculum, pedagogy, teacher or professional development. Now, what are some of the concerns with higher standards? There certainly was a concern voiced at the time and continually that they may lead to increased dropout rates particularly among disadvantaged students. People have cited the reduced opportunity to take electives and CTE courses. On the other hand, I know talking with some folks at the State Department and various districts that there really are still kind of substantial opportunities for some elective taking, but that's certainly been mentioned as one issue. There's a concern about unintended consequences. Certainly some mention of things like cheating. One thing that has come up anecdotally is kind of relably of courses now that everyone has to take and pass Algebra 2. Well, maybe they are, but Algebra 2 might not be exactly like we think Algebra 2 would be and this is something that we don't have much evidence on now but as part of kind of the second phase of our research project we are embarking on a large study of high school transcripts and we have and will be contacting probably many people in the room to try to encourage your high schools to work with us and helping us look at student high school transcripts and to shed better light on course taking changes as a result of the MMC. One concern is that this requires substantial capacity on the part of teachers in schools that sharply increase the demand for math and science teachers and that's a difficult area to fill even in the best of times. Now we're going to have even higher a higher bar and to what extent is the state prepared for that. And then there might be some concern that this reform really won't have much of an impact perhaps positively or negatively on students that were already taking these courses and we still need something for those students. So can we learn anything from past efforts by other states and districts that have adopted similar policies? Well there's been a good bit of research on high school exit exams these exams students take in 10th or 11th grade that are required for high school graduation. I mean I've done some of this work, some other folks in the room have done some. I think it's fair to say that there's been little promise of these policies they have increased drop out rate slightly among low achieving groups and don't show much measurable improvement in student achievement among other groups. There is somewhat more evidence to support raising graduation requirements there has been some evidence that of economic payoffs for taking higher level math and science courses. So that certainly was I think one of the thoughts behind the Michigan AmeriCurriculum. There have been very few policies thus far that have attempted something exactly the same as the Michigan AmeriCurriculum the only the policies that we have evidence on are two policies one in Chicago and one in Charlotte Mecklenburg, North Carolina where they were really requiring all students to take Algebra I and even Mecklenburg requiring them to take it earlier in 8th grade as opposed to 9th or 10th grade. These the results of these studies have not been too positive they led to reductions in student achievement. The authors of these kind of evaluations have cited two potential hypotheses which I think are important for us to keep in mind when thinking about the Michigan AmeriCurriculum. One is these students that were kind of quote unquote forced into the classes under these policies may have been over placed not sufficiently prepared for the courses and that teachers may not have been prepared to teach these courses or these students in these courses and so this is something that we might think about when we're interpreting some of the results for the AmeriCurriculum. Okay, so this is a chart showing other states that have adopted policies similar to Michigan. So these are states that have required students high school graduation classes to pass at least Algebra I in geometry. You can see Kansas starting with the high school graduating class of 2009. Oklahoma in 2010. Michigan is among the first of these states but not the very first. One of the next steps in some of our research here is to do some comparisons comparing Michigan's experience to the experience of these other states but again when we use national data sets to look at this there just hasn't been enough time elapsed to get college enrollment, high school graduation from many cohorts for these various states. So we'll be certainly using data in coming years to try to examine variation across states and hopefully learn from that. So what Michigan high school reform? I think many of you, most people in the room are familiar with this but just so we're all on the same page. Michigan passed a set of reforms in spring 2006 that were based on a set of new K-12 education standards. One was the introduction of the new high school exam, the Michigan Merit exam, part of which was the ACT. And that's nice for research purposes because it gives us another kind of common metric and one that's used across the country to track high school student achievement. And as opposed to the past, we're looking at ACT scores, one would always be concerned, well not every student takes the ACT and of course when you're looking at ACT scores but that's only the more college inclined students. The set of data that we're looking at starting with the ninth graders in 0405 all of these students were required to take the MME and thus the ACT so when you look at the test taking rates they're really quite high and don't change much over time so we don't have that issue of selective test taking that we might have had in the past. Michigan Merit curriculum, this is what we all know of, we'll be talking a bit more about that in a slide or two. And then the Promise Scholarship, as Barbara mentioned this was discontinued, certainly was a very important reform. Initially we'll be doing a little bit of research on that in the coming years as well, this was part of the package of reforms that was passed. So the Michigan Merit curriculum, a set of more rigorous college prep courses that were required for graduation. Okay, so the ones that you mostly hear about are the math credits, so I think the innovation of the Michigan Merit curriculum was not the fact that four math classes were required but which classes specifically were required. So students were required to take and complete algebra one, algebra two, geometry, plus one other course. Science, biology, and then chemistry or physics plus one additional course and some of the other requirements that folks know of are by understanding is that the math and science, kind of what anecdotally and based on the data we have were really the biggest changes relative to what had been happening in the past. To give us a sense of how big those changes were, there was a survey done by the superintendent in 2005 of districts to understand what districts were requiring on their own prior to the passage of the MMC. And you'll see that algebra one, only 32% of responding districts were requiring even algebra one. Once we get down to algebra two, only 12% of responding districts were requiring algebra two. And we have some other data that I'll show at the end. Among those districts that weren't requiring it, a very small fraction of students were taking these kind of upper level math and science courses. So this is just to underscore that this was a big change. This is a big change that students and teachers throughout the state have been grappling with over the past few years. And so making it even more important to try to understand how this reform is playing out and what the impacts are. As a gentleman mentioned earlier, there is a kind of personal curriculum that allows students to graduate in ways that are not exactly like those outlined before. My understanding is it's relatively limited. There were initial plans for statewide and of course exams, those I think for funding reasons, perhaps for other reasons were never progressed to fully to completion. And then, so I think one of the other interesting things about this policy, which is distinct from some in other states, is the end of course exams that students are required to pass to indicate that they've met the Michigan AmeriCurriculum can be developed by schools and districts themselves, and the schools and districts themselves can set their own standards. This is very different than other states where there is a centralized state algebra one exam where they centralized passing cutoff and so that's another kind of unique feature of the AmeriCurriculum. Okay, so finally getting to some of the current study and some fun figures like Sue has. So what are we doing here? The MSR, our consortium is engaged and have two different pieces to this research project. One is a statistical analysis of student achievement, high school completion and college going. This is the work that I'll be presenting to you today. The second though is this Michigan high school transcript study which I think is going to be at least as, if not more interesting and informative than the statistical analysis where we're doing some more intensive data collection in 150 randomly selected high schools throughout the state. And the goal there is to measure the implementation of the Michigan AmeriCurriculum and to try to help uncover some of the underlying mechanisms, how students and teachers were responding, what some of the challenges were, what some of the successes have been. But today we are going to stick to the statistical analysis for someone like me that feels very comfortable and natural so I'm happy about that. As Sue mentioned, we are focusing, this analysis focuses on ninth grade cohorts. This is students starting ninth grade in 0405, up through students starting ninth grade in 0809. One thing that we're going to be doing throughout this that is not so much shown in the figures but is underlying all the estimates and results that we show you and everything that we do, we've been controlling for students eighth grade math scores and demographics and other school characteristics. So one thing that I think everyone is aware of is that Michigan, like the rest of the country, has experienced an economic downturn over the past five, 10 years. And you can see that very easily in the data, I think as Ken will show later on, the fraction of students eligible for free and reduced-priced lunch in the state has jumped considerably. And so we're trying to account for those changes using some of the information we have in this rich administrative data set. We know that otherwise there had been some improvement in student meep scores over time, perhaps better preparation among elementary school students in Michigan. Again, by looking at students eighth grade math scores and controlling for that, we're trying to take into account those changes. The research design we're using is something about an interrupted time series. I will illustrate that in one of our slides coming up. The outcomes are the ones we've mentioned before, high school completion, college enrollment, high school achievement. And I think the one big caveat is that all of the estimated effects that you're going to be seeing here are based solely on this very first cohort of students to experience the Michigan Merit Curriculum. For these ninth graders that were falling over time, we track them and out of all the different outcomes that students could experience, we group them into four categories for simplicity. Graduated, that is the official graduation, so it is not a... But it could include folks with personal curriculum, but it is not some other high school completion measure. It is kind of the official graduation indicator. Whether they're still enrolled in the Michigan Public High School, that's the other outcome. Whether they dropped out. And again, this is kind of the official, somewhat narrow definition of dropping out that is used kind of in some, some, but not all Seppi reports. And others, the category of folks that either transferred to a private school, left the state, or had an unknown status, meaning kind of their district and school didn't quite know what happened to them. And note this kind of other category is relatively small. Probably one reason for that is we're starting with students who are in ninth grade in Michigan Public High Schools, so it's not as if we're not capturing a lot of the turning and exit that you see up through ninth grade. So what do we see here? What do these pluses and minuses and zeros mean? These are meant to show you the effect. So this is kind of the effect of the Michigan AmeriCurriculum. So if you first look at all students, four-year graduation. This is whether they graduate high school within four years of their ninth grade year. That negative sign meant there was a reduction. So we'd say the Michigan AmeriCurriculum reduced the likelihood of four-year kind of on-time high school graduation overall. This was a small, very small effect to kind of give you a rough guideline. It kind of went from 72% to about 70%. So it was kind of small change for that group. Still enrolled, you'll see that that actually increased. And over four years, in the four-year time period, no change, zeros mean no change, no effect on dropping out or kind of being in this other category. I think one thing that's going to be, I think the most interesting thing, and hopefully the thing that jumps out when looking at this table, is if you look at top quartile students, for most of our analysis, what we've done is we group students into four groups based on their pre-high school performance. This is kind of taking into account their eighth grade standardized achievement scores, their poverty level, a whole host of factors, and using that information kind of have quartiles of students, we are mostly in this presentation going to be focusing on the top and the bottom quartile, because that really illustrates the spectrum of effects. If you look at the two middle quartiles, they kind of roughly mirror. The third quartile roughly is similar to the fourth, and the second is similar to the first. So to clarify, we just look at the top and the bottom. For the top quartile, you see a bunch of zeros. So really, that's telling you that the Michigan Merrick curriculum did not have any impact, positive or negative, for the highest performing students entering Michigan high schools. This is in terms of their graduation enrollment dropout. We'll see, we'll later, I'm going to show you the achievement effects. But for bottom quartile students, we see there were some impacts. We see reductions in high school graduation, and then we see an increase kind of in all of the other categories. You know, they were less likely to graduate, and what were they doing instead of graduating? Well, some of them were still enrolled. Some of them had dropped out, and some were in this kind of other category, which can include a bunch of different things. So let's go to the figures, and hopefully this will kind of clarify some things. This is a four-year graduation by initial achievement quartile. You'll see the bottom line is kind of for the lowest quartile students, and the top line for the top quartile. So even in 2005, I should say for the ninth graders in 2005, who were six, seven, eight, the high school class in theory of 0708, the four-year graduation rate for this group was only about 40%, up to about 90% for the top quartile students. And so what we're doing in this analysis is tracking the changes over time for these different cohorts of students. So for the top quartile students, you see pretty much even trends, 90%, 90%, 90%. And it doesn't really change much. This red line here distinguishes the class for whom the Michigan AmeriCurriculum was binding. So you'll see this, the group here in 2008. These are freshmen in 2008. So if you count forward, that means they're sophomores in 09, juniors in 10, seniors in 2011. So this was really the group, the first class for whom the requirement was binding. Not much change in this group. Going down to the bottom quartile, you'll see that from 05 to 0607, there was slight increases in the high school graduation rate of this group of students. And then in 08 for this group, it declines noticeably. So one thing we're going to be, just to kind of give you an intuition, what should we have expected this 2008 high school graduation rate to be? Well, you might think if the trends in the state, this kind of slight, modest improvement among the bottom quartile students had continued, you might expect the line to be there. And then what we are doing, in the essence of what researchers refer to as the interrupted time series, you say, well, we might expect if our trends continued as before, the 08 cohort to be here, we see them down here where the dark triangle is. This little square bracket here shows you the difference between where we expected them to be and where they ended up. And this illustrates kind of a negative impact on high school completion. They are below, the black triangle is below where we expected them to be. So throughout the analysis, when I present the results, you can kind of think about this intuition here. One thing we also did is in some cases, if we weren't sure about the prior trends, we just compared the 2007 versus 2008. So this shows you kind of the four-year graduation reductions. So where are these students? What happened to them? Well, this is an indication of whether they were still enrolled in a Michigan public high school after four years. So this is in the fall of what would have been their fifth year in high school. And so we see here increases. You know, it's going down here and then it kind of bumps back up again. And so as illustrated by this graph, after four years, what largely was happening is students were extending their stay in high school. We didn't see any large changes in dropouts right after the fourth year. We didn't see many changes in kind of this other category. Basically fewer people were officially graduating, but they were back in high school in that fifth year, presumably trying to make it through a lot of the additional requirements. So that was, I think, if we want to take away something for the four-year high school completion results, that's the main takeaway. Five year. This is after five years. And so you notice we only have this last cohort, 2008 is the last group year. That's because five-year graduation for them was June 2012. And that's kind of the latest data that's available. We see reductions in this group among the bottom quartile. Again, notice for the top quartile or even the top two quartiles, not much changed at all. We see still enrolled. Some increases among students that are still enrolled. And then four-year dropout rates. I'm going to skip through that. Fifty-year dropout rates. So this is at the end of five years. What do we see? Again, now you're going to have to switch your eyes. For dropout rates, the bottom lines are the highest performing students because they have the lowest dropout rates. This top line here is this bottom quartile of students. Right here, we see, you know, for the 2007 cohort of ninth graders, it was about 11%. And then for the 2008, it went up to about 12 or 13%. So it was an increase of about two percentage points relative to an 11% baseline that's kind of the increase. Again, among this bottom quartile of students. So you'll see throughout this, one of our main takeaways is very big differences based on the prior student preparation, which is something we probably shouldn't be surprised about based on earlier policies like this. Are there any questions on this? Like, I know I've been talking just nonstop for a while here. So there's two things. One is to break these students up into quartiles, it's largely using their eighth grade math test score. Now, it does take into account some things like whether they're free and reduced price lunch. It's got a general measure of achievement and socioeconomic status kind of, you know, combined. But it might be easiest to think of as just quartiles of eighth grade math achievement. So, any questions on, yep? Let's call them the Michigan Community College Association again. I'm just a quick question. So can you talk, in terms of the... I may be answering my own question here. I'm trying to get a sense of the number of students overall. How many students are we talking about in cohorts? Let's see, each cohort of ninth graders is about, let's say, 120,000, 125,000. And so 1% of that would be 1,300. Is that right, Sue? You have to check my math. So 1% point would be 1,300. And so if we're saying a 2% point increase in dropout would be 2,600. Thank you. Yes, sir. I was interested to know if you track that amongst IEP students or students with learning disabilities. We certainly could. Again, like Sue, we have that data. My guess is sometime in the last two months among the 150 tables we did, I have that somewhere I don't remember it off the top of my head. I just know from personal experience and amongst my peers, I think that gap from the four to five-year kids, that's primarily that group. At least that's been our experience. So this could, a lot of this, that could be one of the reasons why this bottom quartile is showing, if you're saying that the largest of facts might be for students with IEPs, there's probably, I think there is a large overlap between students with an IEP and students with kind of lower eighth grade math scores. So I think that's, a lot of them are probably contained in this upper line here. Although you're right, it's not focusing explicitly on them. Interestingly enough, we can keep them until they're 26, but it counts against our graduation rate where they don't finish before. Right. Okay, good. Ishmael from Macomb Community College. I was wondering whether you have found any regional variations in terms of districts? Yeah, this is, that's a great question. We have not done that yet. So this is what we did as we had done kind of results overall, and then we started by looking separately based on some kind of student subgroups. And we did this a few different ways, but I think we decided the most informative way to show this was based on their prior preparation. I think the next step is to be looking separately by districts, regions along the line that kind of Sue mentioned, to understand whether there are particular districts that are, you know, appearing to be more successful or districts that are having kind of more challenges. I mean that's, we haven't done that yet, but we certainly will. Julian Wilson, Portage Public Schools Kalamazoo Resa. So let's go back to 1A. 1A, yep. So we've got approximately 125,000 ninth grade students in this cohort in, let's say, 2005. Call us. I don't get my kitchen math and swag in here. So pretty, pretty constant in the state of Michigan we graduate. We have cohorts of about 125,000 students a year. So yeah, I mean it's been going down a little bit the last five years, but 125-30. Okay, so the cohort in 2005 is at ninth grade. And then is that, that they graduated in 08, or is that strictly a ninth grade cohort? No, so the 05 dot, the dot here that is the 2005, let's say the 2005 black diamond or black triangle, that is the fraction of students who entered ninth grade in 0405 who were in the bottom quartile of their entering high school class. So it's taking ninth graders in 0405, the bottom quartile of those ninth graders in 0405 and then looking, following them four years later, so that would be about 6 or 7 or 8. So looking at by June 08, what fraction had graduated high school? And so 40% by June 08, 40% of that group had graduated high school. Michigan Public High School. Extension of the MME. Thank you. Okay. So let me kind of push ahead now. I have some other results that are going to show on high school achievements. So maybe I'll show those and then we can come back to answer other questions as well. So now we're going to look at test scores by student achievement test scores. And so we're going to look separately by subjects. First of all, this figure is all the ACT scores. The figure on the left is for the top quartile students. The figure on the right is the bottom quartile students. The numbers have been standardized. So for those of you that are familiar with these, these are standard deviation units or kind of effect sizes. And so let's look at the top quartile. So for the top quartile students, you see there were steady increases prior to the MMC in all subjects. The top line is math. This is kind of the... And again, if you want to get your head around this, ninth graders in 05, if they're taking an 11th grade, this would be the 2007 test-taking group going up. And then not much, a slight improvement from 07 to 08. But if you look at this, it looks like it's kind of roughly following the same pattern as before. So actually when you do add the controls and do the statistical models, what you come out with, there's not much effect on math, a slight positive effect for the highest achieving students on their ACT score in math. But then look down the next two lines or science and reading. Here you see much more noticeable increases between the 07 cohort and the 08 cohort. And when you, again, run this through the statistical machinery, you get modest... I would say modest, meaning bigger than small, but smaller than huge, improvements in reading and science achievement. And so this suggests for the top quartile students, the Michigan Merit Curriculum was increasing science achievement, increasing reading achievement, as measured by the ACT, had maybe a tiny increase in math, but not so much in math. The big, one thing, the noticeable difference is writing. You see ACT writing scores kind of going along mostly, even a little bit upward and then dropping down dramatically after the MMC. So we see a substantial reduction in writing achievement associated with the MMC for the top quartile of students. For the bottom quartile of students, we see some similar pattern for writing. This top line is writing right here, it goes here, it goes way down. We see not much changes for any of the other subjects. So relatively even trends, so really not much positive or negative achievement effects for the bottom quartile. Okay, here's what we do looking at the MME test scores. One interesting pattern of results that we found that you might notice if you look carefully here is that in general the trends and the improvement among Michigan high school students, both prior to the MMC and during the MMC period was larger for the ACT scores than it was for the overall MME scores. Now that's a little bit tricky to think about because we know the MME, part of the MME is the ACT. So that is a bit confusing to think about, but I think I'd be curious to hear people's thoughts or hypotheses about differences between ACT and MME at some point. Here again we see, I think the pattern of results is for the top quartile student maybe some slight improvement, certainly a noticeable improvement in science. For the bottom quartile students not much, not many, no positive effects and we see some, this is kind of for reading, maybe math, maybe a negative effect there. And so maybe the easiest way to summarize this is here. For the top quartile students we see positive effects for the ACT in science, math and reading, but negative effects in writing. And for the bottom quartile students we see the zero, remember the zeroes are no effects, the negative is the minus signs are negative effects and the positive are, plus signs are positive effects. If you are kind of, if you are thinking yourself, okay what's the bottom line, tell me in one sentence what does this mean, why is this, there's all sorts of different things here, and I think after wrestling with the data for many months and talking it over, I think the bottom line is that it is a mixed pattern of results. There is not one clean, simple answer that is true for every type of student in the state. I think in general we see kind of more positive effects for the upper quartile students in terms of these achievement test scores, which is consistent with the, you know, kind of better outcomes for them in terms of the high school completion. We see some more positive effects for science than for math, which I think was, we certainly didn't predict that, and that's kind of interesting and worthy of further explanation. And we see across the board for all types of students, substantial decreases in writing ACT writing scores. So why don't I stop there now for some more questions about this? Let's compare to the interrupted time series. Is this positive gain in these subject areas above and beyond what you would expect as far as the trend line goes, or is it just increases? When you see the summaries here, these are plus or minus or zero relative to what you would expect. So a plus for the ACT science doesn't just mean the 2008 was higher than the 2007. It means it was significantly higher than you even would have predicted it to be based off of that. Mary Beth Dan, U of M. So I have a question. Did you look at the effects or try to control it all for, I would expect that there were any increased rigor in terms of classes yet from the change in terms of that. But I also wondered, what about educational financing in the state? What effect did that have and do you think it shows up? I understand the increased amount of test prep is interesting. I work in schools all over the place and many of them have been increasing the ACT test prep, especially at lower achieving schools. And it's fascinating to see that this might be an interesting result for that too. Those are both good questions. We definitely at this point have not looked into kind of any of those deeper mechanisms. And I think that the test prep is the type of thing that we're hoping to get from the high school transcript study when we go, which is going to include some interviews with principals and teachers, not simply the collection of transcripts. So I think we're going to, hopefully some of that will come out in this kind of second big piece of the project. But that's an excellent point. Was any data gathered for the top and bottom quartile with regard to radiation rates for those going to community colleges? You know, we, not yet, although that is definitely on the agenda. One of the, we have some college enrollment results. And I think the college enrollment results largely near the high school completion. See, not much change at all among the top group. The bottom quartile group, we see reductions in college completion, which is not surprising because we see reductions in high school graduation. But I think we've been a bit more leery to even kind of start looking at the college enrollment because it's been such a short period of time. We know that a lot of students even before weren't graduating in four years, there's a comment from the audience that usually for low income students you see college enrollment more two years after. So we're definitely going to be looking at all sorts of college outcomes including remediation. I think that will probably be the next the next set of analyses we do. What are the, what are the hypotheses behind the writing scores going down for all students? I mean, I think this is I think some of the hypotheses that we've thought about on our end. I'm sure people here who are actually working in schools every day will have an even better sense. But our initial thought had been largely a reallocation of time and effort and resources away from writing to some of the new requirements on the MMC. So I think it's What were the requirements because as well? Well, I mean the English requirements there were four years, it was four years of English but almost every district before was requiring students to take four years of some English. There were maybe small changes in English and kind of the social studies history, but if you go back and look at the new requirements relative to the old state but also relative to what districts were doing on their own, you see that the primary changes that were being required in the MMC were changes in the math and science. And so I mean that result of the writing is consistent. There's other studies that show reductions in achievement among kind of low stakes subjects. So even when you look at elementary schools under NCLB and you see improvements in reading and math, in some places in some cases you see slight reductions in science and social studies and the hypothesis there is teachers are kind of shifting the focus kind of toward the higher stakes subjects. Evan Hordeig from Looking at figure four would I be interpreting that correctly to say that in 2009 the reduction more than offset the gains from 2008? So yeah, so this is certainly true that's kind of the this is the raw data kind of the average achievement of this group. This doesn't have any of the kind of controls for eighth grade ability or the eighth grade achievement scores of the other factors. I think when we look at this in some of the models when we looked at this the 2009 group didn't look as looked a little bit worse than the 2008 group but not as much as this picture would make you think yeah, but this is I don't have the exact numbers off the top of my head but the 2009 group was not quite as positive as the 2008 group Brian Glenn Nelson from Ann Arbor Schoolport I just want to put a little more emphasis on Mary Beth's question about finances that I think the estimates you get on your control variable or variables with respect to resources available or funding might be of at least as much interest as those you get on the MMC that in Michigan we've had this general decline in funding but it's been different across districts because of the move towards equalization so you have variation among districts as well as the statewide trend and to me at least it would be very interesting even though it's probably going to be called a control variable what the parameter estimate on that is in six years from now maybe a lot more significant policy than the stuff around MMC yeah, that's right so this is one thing that and I think Glenn is right he's pointing out the importance of economic conditions and funding levels irrespective of what the particular high school graduation requirements are at the time which I think is a good point I think the point that Sue wanted me to remind people about is that in kind of the analysis, this interrupted time series it kind of understands and kind of takes into account any smooth or constant change over time so you looked at the trends and when we were looking at the change relative to the prior trend because we know that Michigan today is different than Michigan was five or seven or ten years ago I mean in some positive ways in some less than positive ways in terms of the economic climate and none of those kind of the general trending of the economy one way or another shouldn't substantially influence the estimate we get using this the research design we're using now so Dana Dyson University of Michigan Flint, I have a question perhaps that I don't know, maybe it's more of a statement I don't know, but it seems that we just are going to have students in the state that are going to continue to decline in terms of their academic achievement based on this particular reform when you have in the bottom quartile students who are either the MMC is either not having a fact or it's having a negative effect those predictions for the future seem very negative that said we know that there are unintended consequences for policy all the time and could you please speak on the fact that this suggests that in a few years we're going to have 21-year-old students who are going to be in the same facility with 14-year-olds one thing that one of the big take-aways for us among the research team was as I mentioned before the variation the way in which the policy seems to have differentially affected different subgroups of students and I think clearly the most from my perspective the most troubling aspect of the policy is the effects on the bottom quartile of students I think I think that's actually the benefit of some of this early analysis, early research that we can hopefully some of what we're doing here some of the more depth qualitative work that others are doing and continue analyses we can try to really pinpoint what Sue said the choke points and the pipeline or the problems with this particular policy itself and try to help schools and districts adjust and help their students even more the one last thing I want to show to highlight the importance of this kind of transcript study that we're embarked on is one of the things that we'd like to do in addition to looking at the high school completion and achievement outcomes is looking at course taking and obviously the best way to do this is to look at the high school transcripts what we have here while we are still working on the high school transcript component of the study we are able to look at the ACT data so as you know students that take the ACT which is now all test taking students in Michigan high schools are required to fill out a questionnaire where they answer lots of questions one set of questions they answer are about what courses they have taken or plan to take by the time they graduate and this data should be taken with a large grain of salt because A it is student self reports and B courses that may contain certain content might not be labeled the same thing across schools but kind of even even with a large grain of salt we thought it was sufficiently interesting to put these in the presentation the yellow highlighted rows are kind of the fraction of students the percent of students who report taking the courses that would have met taking or will have or would have taken they report that they have taken or will eventually take the courses required for meeting the Michigan Merit Curriculum and so we see in the top row for these classes of 0809 2010 about 30 percent of students were reporting taking algebra one, algebra two, geometry and one other math course this increased to 38 percent in 2011 so we do see some change even among these very you know rough and approximate student self reports you know and we see the same thing below for science kind of the fraction of students taking biology plus chemistry or physics goes from 60, 70 percent to 80 percent but so I think we do see some change which kind of I think tells me that there's some information in these figures but one thing that we see is you know the number 38 percent in 2011 for the fraction taking classes that would meet the MMC 38 percent is very different than 100 percent and even if we take into account the personal curriculum and even if we take into account the large grain of salt of students self reports I think this is suggestive that there's kind of been differential implementation across schools and districts in the state and I think this kind of illustrates the importance of the second component of the project we're doing which is the transcript study where we can actually get in and talk to teachers and principals look at transcripts and really try to understand what was going on in the schools so this is the final slide conclusions in ports of examining impacts separately by generation level potential slippage in implementation will schools grow in their ability to implement over time hopefully the transcript and related data work will shed some light on that there's some variation across subjects which I think is kind of interesting in puzzling the math versus science I think is particularly kind of interesting in puzzling to me and then the broader implications for other states as you saw before Michigan is just one of many states and more than half the states so far have passed policies very very similar to the Michigan American curriculum so many many states in the coming years will be implementing similar reforms and then the common core is going to present a roughly similar challenge probably many other challenges as well but kind of a similar challenge at the high school level in terms of more advanced course taking requirements so with that thank you very much