 So I have the task of kind of coming to the last part of this and I have a few things that I would like to talk to you about. The first thing is that as you had heard this is the beginning. This is really the just the beginning. So if we were going to think about what's next on the horizon, well one thing that we're really interested in as we had indicated earlier is about the Michigan Promise Scholarship. As we explained this was sunsetted but nonetheless again it represents one of those natural experiments given the fact that we have all this information on the students over time because we're able to connect it to the National Student Clearinghouse data that we really are going to be able to look at this perhaps in ways that it's never been done before and Sue will be leading that effort. The next thing is that we gave you a little bit of just what I would call a touch of kind of thinking about things and how things vary on the basis of income and race and ethnicity but we're you know that group of researchers that really want to go deep on some of these questions. We certainly are interested if there are issues related to gender or also issues is interested in questions about age and that at a larger level school and district characteristics and what about those school and district characteristics. So here again we're looking at not only the student but the context in which the student is in and we will be able to do different kinds of comparisons by schools and by districts and then we're going to be able to do something that we think is really quite novel in the sense that we're going to be able to look at course taking and its effect on achievement and on post-secondary attendance and really persistence as well if we have we're hoping we'll get those kind of data and we'll put those and add that to our file as well. And then finally as we explain there are a lot of things here that aren't answered so we have to be able to go deeper into our schools and figure out are there school organization and personnel changes that occurred that were in response to the MMC or that predicated the MMC. So how do you go about doing something like that? Well the best way really to think about doing that is to go deep into schools. Now we have a lot of high schools in the state so we can't go into every high school in the state. It's one of the reasons why we have the National Center for Education Statistics because in the 1940s there's some people that came up with some really good ways of what we call sampling so we don't have to go in and talk to every single school but we can draw a sample of schools and that sample of schools will give us a generalizable probability sample of all the schools in the state. So how do you do that? Well you got to look at all the schools in the state and you have to figure out mm-hmm well if we were going to try and build a generalizable sample of schools we need to know something about who's in those schools. So what's the race and ethnicity of those schools? Where are those schools located? How big? How small are those schools? What kind of grade levels do they have? Is there some sort of unique configuration of those schools that in fact makes them somehow different that makes them either in a generalizable sample or out a generalizable sample? So when you're building this sort of probability sample, you have to do what's called you got to stratify. You have to stratify on these kinds of factors to be able to build a probability sample of a number of schools in the state and then you have to worry about well how many of those schools do you need for it to really be a powerful sample that in fact when you got through with it you could say aha this looks like something that would really hold up for all the other kinds of high schools that were in the state and if we just wanted to look at urban schools would we have enough urban schools with low-income minority students in them to be able to make any kinds of differences across them be real so could we have enough power to be able to say something about those schools? Well what we did is we went and we took all of those schools and we built this massive database about all of our schools and then we went to some expert samplers. In our case it was a firm called Westat and we worked with the Westat firm in developing 150 schools actually we have more than 150 because we knew that much as we are good researchers and much as we have a fabulous team of people working with us that there were going to be some school districts that just we're going to say we're too busy we can't do this it's too much burden we're sorry or we closed and we're no longer there anymore or we merged and we're no longer there anymore so that that put us in a position where we knew that we needed what we call extra schools to be able to be used as replacement schools to what we were doing so eventually we drew this hundred and fifty schools now how does this hundred and fifty schools correspond to what we've been talking about all day long which is our transcript study. Well the first thing is that think about it and think about what Brian was telling you what we're trying to do here we have an interrupted time series so that means we have to be able to look at what kids were taking before the Michigan Merit Curriculum and what kids are taking after the Michigan Merit Curriculum that means that we have to collect course data from kids all the way back to 2003 so we have enough data points that we can look at to be able to look at what happened in 2003 2004 5 6 until now we have the Michigan Merit Curriculum and now what's happening to students after that so that means we have to go to schools and we have to go into those data banks where in fact we can get all of that information on students that went to that school from 2003 now why do we need to do this well fundamentally the first thing that has to happen is something which we call fidelity of implementation so what's fidelity of implementation essentially that is what everybody in this room has been talking about are they really teaching algebra or are they teaching something else so the most important thing that we are using this transcript study for and we're trying to get textbooks and we're trying to get syllabi and of course we were going to get and of course assessments but we found out that was a lot more difficult than we thought so we are not getting and of course assessments although we are trying to get whatever of course assessment information we can but the idea here is really to see in fact are we and are the students taking more advanced level courses and this could be either a change in title or a change in content or a change in both course and content and textbook or not so we really have to look very deeply to understand was there actually a change in what was being called algebra and if in fact we're asking for four years of a particular subject area were the students being advanced in that particular subject area now I think that all of us in this room know in fact that when we try and we know that this does happen the people are in fact saying that they're teaching geometry and they're not really teaching geometry and Jack talked about it this is data that we know has existed for a long time we've looked at this in some other kinds of studies at health is a big national study and they were Ken had a very big role in that and they looked at the data there one of our graduates one of our postdocs right now was doing some work actually at Penn the University of Pennsylvania again looking inside these courses and what are they seeing and they're seeing in fact that the course title and what actually gets taught in the course are very different and that work really started with as Jack mentioned before Bill Schmidt because he was really started to do this work in the elementary school right around Tim's so that's the first thing has to do with fidelity of implementation but there's a second reason why we have to do this what happens if with all these great methods and we turns out that courses had no difference but the kids scores went up anyway could happen right I mean we don't know Brian just told us about you know one cohort you know maybe when we look in 2012 we're gonna see something very different by having the course information we're able to be able to understand was it really the courses or not and this becomes really a fundamental question that we have to ask statistically to find out if this is really something that's going to make a difference was it the courses or wasn't it the courses so here we're going to be able to get the mechanisms by which this is going to occur now I am going to pull this down you know what the sticky yes okay there we are so here we are with the face of the person who is clearly the most important person that is going to help us with all of this now I just want you to think about and you can ask questions and her team is here because Karen is smiling right there at me so we're going to talk to you about all your FERPA concerns you know whether or not we have issues of confidentiality we'll talk to you about any kinds of questions that you want to raise about the transcript study but please try and think about the transcript study in the context of the MMC evaluation as something that is really critical critical in order to reach what Jack has said and for us to be able to be much more definitive in our response in terms of did the Michigan Merrick curriculum really make a difference now I'm going to take a little step aside before I ask for your questions because I would be remiss if I didn't talk to you a little bit more about what this hundred and fifty school sample is able to do when Vanessa was a graduate student she came to me and said wouldn't it be great if Michigan had a longitudinal sample of schools like we have for the national educational longitudinal study in fact we could call it mouse the Michigan education longitudinal study and I thought to myself what a great idea we're going to have all of this massive kind of data but we couldn't really get inside a lot of these schools so when someone raised the question before about Flint Michigan and does this mean that they're going to be kids in school that are going to be 21 we know they're going to be low-income schools that are graduating to graduate a lot of students and they're going to send them on to college but we have to figure out what was it about those schools that really made a difference what were the mechanisms in that school that really supported students this hundred and fifty school sample can be used for a lot of things and I would charge Jack and I would charge other states and say find a way to sample in your in your states to build these kinds of things because these are ways that you can move in and you can look at schools in ways that you don't have to burden every person in the state now what are we going to do for our schools our hundred and fifty schools we're going to do a lot but one thing that we promise that we will do is very is something very similar to what the Chicago Consortium on School research did and that is we're going to build for you school reports things that you want to know about your schools we're going to tell you and we will work with you and we will show you that in ways that are transparent and easily understood so that you can take it back to your school boards you can take it back to your personnel committees and you'll have information about your schools that you wouldn't otherwise have including where did your alumni go what happened to them how long did they persist in post-secondary education so I'm happy to say that my part in this discussion is now over not my part in the partnership or the collaboration I suppose we're not really partners we're more collaboration than partners so we're all over partners as well but it's really a consortium and to say I really thank you very much but we're here to entertain questions and I'm happy to field whatever you might want to ask right now so if you could please all just raise your hand and we'll be happy to answer your questions yes you've talked about getting the you've talked about securing syllabi books but I wonder in many districts since a sixth grade math teacher is not teaching the same thing in six middle schools in that district rarely are districts coordinating that how are you going to be able to pull apart what is teacher ability and that piece in the in the classroom with that door shut and what actually I know where you're going I know no I know where you're going and it's a very important question and I think that Tom Kane and a group of people that the Gates Foundation has funded is going to be able to answer that question in a way that we are not going to be able to answer that question so the Gates Foundation has something called the MEC project there's 69,000 hours of video of teachers they're looking very hard at effective teachers they've built teams of researchers who actually have coded that video information about effective math teachers at both the elementary and the secondary level and that kind of information is that goes very deep in terms of what people are actually doing in their classrooms when they shut the door regardless of them textbook that they're using or the syllabi that they say that they're covering or the homework assignments that they're giving that you're going to actually be able to see what their teacher is doing in that classroom and how well he or she is structuring the lesson whether they're doing scaffolding or they're not and how else that they're doing that at sixth grade and all the way up through high school and for those of you in here that are interested if you got your degree after 2007 your PhD degree after 2007 you can apply to analyze that data through the National Academy of Education and I would suggest that you contact Brian Rowan at the University of Michigan because he's the person that's in charge of building scholarships for people to be able to look at that data yes it's called the MET project that MET I'm trying to think I don't have my head and I'm like I have to my head is so on the Michigan Merit Curriculum right this minute that I'm not translating it but measuring effective teaching that's it measuring effective teachers sorry measuring effective teachers and it's funded by the Gates Foundation yes please do you have a concern about when you're doing the transcript study that you might only get districts willing to participate that actually do have their curriculum better aligned with what you might expect an algebra 1 class to look like for example and maybe not a lot of districts that may not have the rigor there or the the exact curriculum that it should be well we built the sample with the hope and expectation that it would be representative of the state hopefully we will go after every school district that we have on the file and we will of course look and we won't be able to do that from the beginning but Karen do you want to say anything about how districts have been responding she's really has a magic touch here so I think that it's important for her to say a few words thanks very much Barbara Barbara and I are not related by the way but I'd be happy if we were I'm really I'm privileged really to be part of this project and I have a real enthusiasm for talking to each and every one of the schools that are in our sample and I have had the opportunity to do that already with probably about 25 schools so we're ramping up to reach out to all of the schools that are in the sample it's a good way to get a good look at Michigan that's for sure and the other thing is that we find that we have to have an individualized approach with each school that we have in our sample now everybody's records are in the same shape are all available in electronic format for example so I just want to assure that if you do get a phone call or someone in your district or school gets a phone call from us that we do want to talk with you individually about how to work with you to obtain this data and what that means for the school and what it means to take part in the study so I do appreciate everyone's attention to it too please one thing is important two things are important to note I mean this is common in most research studies but if your school or district hasn't been part of it part of one you may not know one all the data will be confidential and anonymous you a district that doesn't feel like they've done as well as they would like to there will be nothing in the paper saying district X we found after analyzing this is the lowest district in the state on the I mean all of it I mean kind of the doctoral students and researchers will you look at it but when it kind of comes out and we present results they'll be kind of in general format kind of for the state of this or for you know districts and you know kind of you know with these characteristics but always in kind of large enough groups so that no one district and certainly no one school is ever singled out the other thing to keep in mind is that we luckily through the funding throughout yes we have funding to compensate schools if there's any kind of time required to pull some of this data together and our initial investigations have shown that most almost all schools and districts have the trans most of their transcript data even going back for many years in some form of electronic files but if you are worried that oh no it's ours are on this kind of drive that has to be transferred over here to do this and where we're gonna find time for someone to do that you won't have to we have Karen and Karen has dozens of people that will gladly do that and that's where the funding through IS IS is important because we know that school folks don't have the time or money to do some of this themselves so two two important things to keep in mind and we should say also that what we're again we're working with NCES on this as well because what we're trying to do with the coding of the transcripts and the course titles is to make that transparent so that the kind of coding scheme that we put together will be usable for other states especially as we move towards the common standards that our hope and expectation and we've been doing just a little bit of experimenting right now with some transcript data that we currently have we've been doing some reliability studies in other words you know if two people you know look at it do they come up with the same kind of code for the same course one thing that has happened since I first started working with transcript data from the 1980s is that there actually is less of the variability that there was at the time if you recall when Brian put up that thing called a nation at risk and so there has been some kind of movement especially within the last six or seven years towards more common titles again that doesn't mean that they're teaching that but it certainly is the case that we're moving towards more common titles so we're thinking right now that that's going to be not as difficult a task and we're hoping that the codes that we put together will be codes that other states will be able to use and all of our decision rules for where we put a particular course what kinds of decision rules we have to make one of the things that we're finding out right now is much too at least my surprise there's still a lot of schools that have three trimesters instead of two semesters and so you actually have three semesters of a particular course over the course of the year instead of two so how do you change that and how do you make them equivalent so that you're actually looking at you know a certain half of the year in algebra one and then the second half of the year in algebra one so we have to be able to have decision rules on that and that will become one of the things that we hope again will be will be able to make more transparent yes this is a bit of a broader question when I think about consortium work and I think it's amazing that this consortium work is happening the way it is I'm wondering if you've thought about partnering a little more intentionally with the ISDs in Michigan since we now have these data systems that you know we've been working very hard to have the school consortiums working with us as ISDs and I think we have access to a lot of the data maybe down a little bit closer to the student than maybe you realize or maybe you know that I'm just wondering what your thoughts are in terms of the larger consortium work and that possibility and yes yes yes well I know but I want to I but I want to talk a little bit more about that because I think so that's okay but if I'm if I'm if I'm missing something or I'm saying something wrong correct me please but when we were thinking about first doing this transcript study actually Brian Rowan is the kind of person that's in charge of this and his initial image was that we were going to go to the ISDs and we went to in fact we did an ISD study last summer where in fact we went to all the ISDs and we were trying to find out where the transcript data were and also how many the ISDs collected and of course assessments or had their own course and of course assessments and we found a lot of variability among the ISDs a lot more a variability than we had expected and we still are of course in the situation of wanting to work very much with the ISDs because we know that some of the transcripts are held with the ISDs we also know that some of the and of course information is with the ISDs and we also know that the ISDs can be great partners with us in having districts and schools participate in the transcript study and this is a way also that if you have particular information that you want to share with us that we can append to our already existing files that that would be a great way to move and get more information I just had a question there's a lot of talk right now about moving to the Common Core and how to meet all efficiently and maybe the integration is a really great way to do a project based learning are you taking those kinds of models into consideration and you're a hundred and fifty talking about variability in the hundred and fifty schools that you're sampling yes and the other thing is we talked about a lot about standards based reporting versus grading and and of course and so you're going to see a little bit of that at least probably and but there's a lot of interest in it because it's an efficient means and it puts students into an inquiry mode instead of the traditional course and and thank you very much for your comments so let me just talk a little bit about what we're going to be doing with the hundred and fifty schools because it isn't just getting the transcripts and it's not just you know collecting things one of the things that we would very much like to do with I think you heard before is we'd like to interview the teachers and the principals and I know that Brian and Ken both have real interests in how we push forward on this so probably one of the first things that we'll start doing is in addition to sitting and building models that make sense in terms of trying to understand what's actually going on in the schools as teachers have in fact and principals have made school organizational patterns change to meet the Michigan American curriculum we'll also want to get some other kinds of information so probably we'll be holding what we call focus groups with teachers and principals and talking through some of these issues about what kinds of things we really need to be looking at and looking at more closely what kinds of questions we really need to be asking to get at some of these things it's very clear that it's much harder to deal with project data when we're looking at the transcript data we're of course looking at Carnegie units we're looking at course titles and we're looking at grades so that in fact that really stops if in fact no letter grade if it's just a pass fail of course if it's just a pass fail we'll have that information as well but there might be other kinds of things that we really need to know about and there might be some things that it might be worthwhile to talk to people that are collecting transit transcript data that they nearly need to make some sort of amendments I think that the superintendent really pointed out and I think it's very important for all of us to remember that schools are dynamic places they don't stand still their fluid their grading practices change things change what they consider to be part of curriculum makes a difference and we have to be attentive to those kinds of things and we have to be responsive to those kinds of things and they may in fact make it for us to have to look at other kinds of things but right now this is sort of the plan that we're going with and if things change dramatically we will of course make changes ourselves we wanted to get you all real early in the morning when you are all bright and have lots to say fill you with information not you know and we've kind of figured that we would end a little sooner than later and we did it quite deliberately because we really wanted to make sure that we got our message out and it didn't become one where you know we got into a whole lot of issues that we really couldn't answer very well but I would like to say that the issue about staying connected we have a website and we are open we answer questions and I would like to say that please you know come to our website come to us talk to us talk to Karen this will be as good a project as all of us can make it and I hope you all will participate and before we close it's important that I say one little thing which is that Brian has been in sunny California on sabbatical this year he's been very active and we've been skyping with him regularly so he's not off the hook and as you could tell he's very much involved in what we're doing but Sue has really had to shoulder the majority of the work associated with this meeting which in my mind has turned out to be a great success and Julie and Christina and Steve and Rachel and Beth and of course our fabulous team of researchers but I think so we all owe you a special round of applause and thank you very much