 I'm Sudanarski, I'm a professor here at the Ford School of Education and co-director of the Education Policy Initiative here at Ford. It's a pleasure to have all of you here today. Just to say quickly, the Education Policy Initiative is a program of coordinated activities. We try to be coordinated, sometimes we're clumsy, designed to apply rigorous research methodology to inform education policy issues as well as to disseminate best practices of education reform to local, state, and national policy makers. And we've got a series of initiatives underway. A new one that we're excited about this summer that you guys might want to think about for future summers. If you're a first year student is an internship with the state. We have Michigan data fellows who will be doing work with the State Department of Education, helping them analyze data and understand evidence to drive education policy. And it even pays. So we train graduate students and others to conduct cutting-edge research. And we facilitate discussion about education reform here at the university and in the larger community. So today we're very excited to have Rebecca Maynard, who is university trustee chair and distinguished professor of education and social policy at Pennsylvania, University of Pennsylvania. She recently served as commissioner of the National Center for Education Evaluation and Regional Assistance at the Institute for Education Sciences. Institute for Education Sciences funds many of the activities we have going on here in the education policy initiative. Rebecca is a leading expert in program evaluation, including the design and conduct to randomize controlled trials in the areas of education and social policy. Throughout her career at the University of Pennsylvania, she has provided evaluation support to numerous public schools and community-based organizations, as well as overseeing national evaluations of teen pregnancy prevention, home visitation programs, and vocational prep programs. Today, Rebecca is going to be discussing her research focused upon using evidence to guide policy and practice in education. Her presentation will be followed at about five o'clock by a Q&A session. And I want to thank Susan Collins, dean of the Ford School, and our other distinguished guests, and to thank the Charles H. and Susan Gessner Fund for their generous support of this event. Thanks very much. Thank you. It's a real pleasure to be here. This is my second trip to Michigan, and I just think this is a wonderful campus. It just has a great feel. And thank you all for coming to late afternoon sessions. I don't know Midwest seems to be different than the East, but our folks are sort of all clearing out by now. So we'll see if we can keep you awake. So I want to talk a little bit about why the push for more and better use of evidence. This is something that in my time in Washington, I was there for a little over two years. We spent a good part of our energy worrying about how to make better use of the evidence that we had and how to get more of that good evidence. And one of the reasons that this is really important is because education is really big business. We spend a lot of money in this country on education. We have 75 million students. It's about a quarter of the population. That's pre-K through higher ed. We've got 10 million employees in education. That's a sizable number. So it goes education. So it goes a lot of the workforce. Three and a half million of those individuals are K-12 teachers. We've got a million higher ed faculty and a lot of support staff. We're exceeding $800 billion annually, about 60% of this for K-12 education. And that's about 6% of GDP. That's been pretty stable, but going up a tad in recent years. So this is one reason that's important. Another reason it's important to think about evidence and making smart decisions in this field is the landscape is continually changing. And we've got preschool. Most of our kids now in preschool. Pre-school has doubled since 1980. Our performance in K-12 education is just about where it was in 1970. Some of the standard benchmarks of performance. We've got a little bit of a progress in our dropout rate, but still pretty high and much higher than we'd like to have it. We've seen a near doubling of the percent of students with disabilities. Students with disabilities consume a lot of additional resources. We do have a pretty strong commitment in this country to serving those students in appropriate settings, but the numbers are going up. We've had a sharp increase, really sharp increase in the English language learner population and a massive increase in the diversity of those languages. I know you've got schools out here with 30, 40 languages being spoken in the schools. How do you manage that? That's a new challenge. And we can't leave those kids behind. We've had a sharp increase in post-secondary enrollments, pre- and post-BA. And a lot of this is going into the community colleges, so there's a shift in the allocation of these students. And we've had increasing per-pupil costs and declining budgets, so this is lots of tensions around that. Adding to all of this some of the common-sense strategies that we have had for addressing some of these challenges in our education system seem to have failed. We've had steady increase in early education programs, but we're not seeing kids higher proportions of kids ready for kindergarten. We've had expenditures per pupil rising sharply, but we haven't seen performance going up. We've had pupil-teacher ratios and pupil-staff ratios. I always get this one, which way am I going here? They're getting more favorable, so falling. And performance hasn't been moving up with that. Teacher salaries have increased substantially beginning in the 1980s. We had a big erosion in teacher salaries, but we sort of fixed that in the 80s and into the 90s. It's leveling off a bit now, but that didn't do the trick either. We just thought we needed to pay the best and the brightest to come into the profession. We've had massive increases in the funding for special education, partly driven by the numbers, partly driven by commitments and new ideas for how to serve that population. We have lots and lots of programs and lots of research going on and how to deal with the ELL population. We're still seeing that group tracking very poorly. And we've tried the old accountability through No Child Left Behind. So we've sort of upped the ante and changed the expectations there. So we've tried a lot of tricks and we're not sort of making any progress. So the next thing I think we should be trying is a little more reliance on evidence in our policymaking. And we have some examples out there right now. Tiered Evidence grantmaking is one of the new fangle things and we're putting about $140 million, $150 million a year for the last three years into investing in innovation. Now innovation may be not quite what we're investing in, but one of the nice things about this tiered evidence grantmaking is that we are charging people, the government is charging people to look at evidence before they submit their ideas. So what do we already know? And they're also charging them to make a commitment to generating some new evidence about how effective whatever it is they got funded to do was. Did it have the intended outcome? So we've got the investing in innovation, which is following national priorities in terms of the types of strategies we're funding. And we've started putting $100 million a year or so into our early learning challenge grants. So this is money to improve the preschool education system through a similar kind of competitive grantmaking where the applicants are charged with drawing down the evidence and committing to, actually in this one I have to say, after much fighting we really didn't get a lot of evidence coming back as a result of the grants. There will be some, but not, we don't have the same quality of research built into these grants as we do in investing in innovation. There's a big push in government. If you haven't heard about this one, this is the pay for success, the social impact fund and social impact bonds following the British model. This is basically charging the service providers to set their goals, set their targets, and the government will pay you if you achieve them. So that's got its own challenges. Are we setting targets that we know we can meet and that really aren't improving outcomes and we'll get paid for it? Or are we setting real targets that are going to raise the bar and give us better outcomes? And we have started with having some evidence-linked policy waivers. So one of these has been some waivers around that allow expanded eligibility for Pell grants. So that's one that affects directly the higher eds. We're hoping there's some evaluation that goes on with that and there are some waivers that are being given out under the Elementary and Secondary Education Act. So states can apply to the government on regular cycles to modify or have exceptions to some of the rules in return for a plan that is somewhat innovative, that promises to improve outcomes and that will return some evidence telling us whether or not in fact those outcomes are improving. There are some facilitating conditions to support this move towards greater reliance on evidence in education. One is an expansion and an improvement, I hope, in the access to evaluation and technical assistance. Some of you may know, others of you probably don't know that there are 10 regional educational laboratories that together collectively are charged with providing technical assistance and capacity building for all states and territories of the United States. These labs are funded at about $5 million a year through the Institute of Education Sciences. They all have strong research capabilities and resources to partner with state and local education agencies to improve their capacity to generate evidence and to use evidence with the goal of improving outcomes. There are a number of federally managed and funded evaluations that are continually ongoing. One of the things that happened with the establishment of the Institute of Education Sciences is that the number of such evaluations and the quality of such evaluations has improved quite considerably. I know that you folks out here have a number of projects that are funded through IES and these are all highly competitive projects. For those of you who don't appreciate the hard work and the prestige and importance of these, these are very precious grants and the research is really important to expanding our ability to act on evidence. And another facility in condition with tight budgets and tough trade-offs, this is actually an opportunity, I think, to provide some incentive to work smarter, to work smarter. And I'll give you one of my favorite examples at the end. I'm going to save it for the end of where lots of money could have been saved with no loss in productivity. Okay, so one of the things that we worry about is improving the inventory and the use of evidence. So how do we do that? I'm going to talk about four examples here using existing evidence smartly and I'm going to talk a little bit from my own experience of how we have done this and can do more of this. I'm going to talk a little bit about conducting rigorous studies to inform future policy or practice and improving access to credible evidence about the effectiveness of programs, policies and practices. One of the issues is if you've got to make a decision, it could take you a very long time to go through the education literature and sort the wheat from the chaff here and know how do you know if you're not an education scientist or how do you know what's good and what's not. And then I'll talk a little bit about making it easier to design good program and policy evaluations. So one of the comments I want is learning smart use of evidence and this was, I call it my mid-career awakening when I went into the Clinton administration thinking I was pretty smart and had done a lot of evaluation and I was charged with doing the part of his welfare reform plan that was coming up with a cost estimate for childcare. We were going to have changed the work requirements and welfare as we know it and we were going to put in high quality childcare so that all these mothers could go to work and we could reduce welfare and the next generation would be set to go. And so the approach here was one that we have been using for years in just normal policy and program management which is simulation modeling and we've done a lot of this where if you have a set of program and policy rules that are driven by demographics for example and income, things like food stamps you can easily simulate what's going to happen if you change the eligibility rules for food stamps you know how many people there are the family sizes and the income and so on so you can just pretty well work this one out. This was a little more complicated in childcare because we have some behavioral economics that come into this we have the choices we get to make on the public policy front we've got the demographics and the profile of children and what welfare is going to do to birth rates and marital stability and all these other things and then we have the cost accounting part so the one that's really tough in here is the behavioral side of this what is the response pattern going to be? So this is when I sort of went and looked at all the great studies that I've been working in the field of childcare so I thought I knew all the studies and as I sort of revealed this morning to a group we hadn't asked necessarily the right questions or reported the right information to carry out this policy analysis in the best way so we did the best we could so we got our unit cost by setting we got some cross-sectional estimates from the labor field about who would engage in school work or training and whether it would be full-time or part-time and for those in and out of home activities the numbers and ages of their kids but wow there were a lot of unknowns in this here model and then we had to sort of have some dynamics in here because people get jobs and lose jobs and changing numbers and ages of children so this got pretty messy pretty fast and the guilt on my shoulders is that I Clinton did not take my childcare estimates which were much much lower and probably much more accurate than the ones he put out and I think probably he lost the welfare reform plan because he had a number on childcare that was too high the administration came out estimating about 90% of these women were going to be using childcare was not going to be the reality okay so second aspect here is testing program effectiveness and how we should go about doing this and if you read the literature and I'll give you some statistics later the majority of the research out there that purports to be telling you how effective something is doesn't really tell you how effective something is it tells you differences or changes or it's not a good estimate of the of truth there so the test for me was when I took on the challenge of doing the impact evaluation for the title five abstinence education program and I got sort of a lot of chuckles from people because a lot of people just thought this was like a really stupid policy and whether it was a stupid policy or not a stupid policy it seemed like it was important to know because this was policy this was kids lives were being changed for better or worse or not at all so the assignment that I have was to determine whether these abstinence education programs which were funded to the tune of $50 million a year or something like that were beneficial so was this really a way to protect the health and well-being of adolescents or was it harmful or was it just money down the drain and so the approach here was to do impact evaluations for model programs and in doing this we looked hard at what the logic model was for the program this is sort of important I'm a big believer that you have to see it and you have to think hard about is this how it's going to work so in this model we sort of had because this was also I don't know how many of you remember anybody here remember those battles over abstinence only or comprehensive sex ed yeah they were really right so the first meeting I went to on this you could just shout it out of town for this because of course I took on despite the fact that I was a researcher I became not really but people looked at me as if I must believe this is the best thing since sliced bread as a researcher I'm just about the science so the logic of this is that you have these kids who are sitting in these schools and most of them are getting some kind of health and sex education and they're going out with their friends at night and they're getting some information there and their parents are talking to them are not talking to them who knows what's going on and out at the end there they have some behavior so they do or don't engage in sexual activity if they have sex they do or don't use contraception they do or don't do it with a lot of people or a few people and depending upon those choices then they're maybe going to have sexually transmitted infections or STDs or they're going to get pregnant if they get pregnant they might have an abortion they might not have an abortion and so we dropped this policy change in the middle of this and one side the abstinence only folks only wanted to know did they abstain or did they tell you they're going to abstain because you know we all know that we're not going to go watch them all the time so but when you look at the logic model like this there are sort of lots of things up there right that there were some you know people can not care about sex but they can care about contraception they can care you know they can not care about sex if you're using contraception they can I think we would all agree that sexually transmitted diseases are not good we probably most of us would agree that teen pregnancies are not good you know the abortions we probably very divided on how we think about that so we could and then in the middle here we've got sort of some knowledge of sexual health and some values and intentions in here now imagine if what we do one argument is if we give kids a clear message they will that's the best chance of getting to them to the end game but that means we can't tell them anything about contraception we can't we can tell them a lot about STIs STDs we probably should stay clear of the abortion or tell them only that it's bad now we've got some kids out there who maybe get that message but then they somehow get someplace and have sex and so they may end up not using contraception because they never heard of it whereas if we hadn't done this abstinence thing they might have heard of it so we might have actually we've got sort of two things going on here we could change the probability of having sex and we could also change the probability of using contraception and they probably go in opposite directions if the program is working right so who knows we could get some good things and some bad things or all good or all bad so that said to me that I was only going to do this study if we went for truth I wanted to get the best estimates I could of the likely impacts on all of those outcomes that we just showed you good, the bad and the ones we can disagree about and depending upon truth whatever we learned about this I wanted to be able to feedback something to that logic model to sort of say okay so this is really how it works if you teach them about abstinence yeah some of them will not do it more if you will get a reduction in teen sexual activity maybe they won't have as many partners but if they have sex they're not going to have contraception or whatever it is I wanted to be able to not only know what happened on those outcomes out there but I wanted to understand a little bit more about what was going on in the middle never cared so much about understanding inside the black box is in this one because it was it's really important I mean the kids are at the end so there's an answer to this this is good, bad, or indifferent so the strategy was I would only agree to do this study because I believe that is the only way we can control for self-selection and get a good unbiased estimate of impacts and we did parallel studies of four programs that varied by age group approach and counterfactual so we had a mix here and the findings some of you probably read to the bottom changed attitudes and intentions at least as reported to us did not change behaviors four programs zero, zero, zero, zero okay? on all of those outcomes over there it didn't matter so whatever is going on out there was bigger now if we had not done this experiment I can tell you that the people who liked the answer and we probably would have gotten some positive you know we would have gotten some we wouldn't have gotten all zeros people who liked it would have said I did a great study and the people who didn't like it those of you who didn't like whatever I found are going to tell me that my study was flawed that's what would happen, right? that would be the truth so this was really important to do this study so that right or wrong Jeff Smith and Sudanarsky they're going to stand behind me and say you should take these results seriously so the reason we did this experiment, stakes were high kids health and welfare were at stake it's a politically charged issue and findings from any other design were just going to be fodder for the political debates and battles here and the setting was right there was insufficient capacity to treat all students okay this was a demonstration project $50 million a year that wasn't going to do everything for all the students everywhere so we could have just put the money out there instead what we said you want the money help us learn something and if you're right, we'll learn that if you're wrong we're not going to chastise you for it we're going to celebrate the fact that you helped us learn that you were wrong and we hope that you will feed that back into your programming so that my argument to the people who loved this abstinence was you don't have to give up on abstinence if this doesn't work but if this doesn't work that's what you're trying to achieve and you don't want to work 16 hours a day doing something that's not having any impact or having the opposite impact okay third thing I want to talk about is how we organize evidence for efficient and smart use so how do we know what works what doesn't work too often we want to look just for what works and we don't want to look at what doesn't work and when don't we know I mean sometimes we just don't know we never ask the question and there's nothing wrong with that that doesn't make us bad people it just means we need to make a decision absent evidence and we ought to know that that's what we're doing we're making a smart decision based on you know our hunch our beliefs whatever so what we need is systems for searching sifting, rating and summarizing evidence and education and that's started in the medical field the medical field had something called the Cochrane collaboration your doctors my doctors all use this they want to know how effective a certain drug is they log into the internet and go to the Cochrane collaboration so about maybe now 15 years ago folks in education started the Campbell collaboration which was education, social welfare crime and justice they modeled on the same thing which is to do little sifting and sorting and summarizing of evidence this got picked up when IES was founded Russ Whitehurst was the first director took on a mission of creating something called the What Works Clearinghouse and my only wish is that it would say what works and doesn't work but it's catchier if we say the WWC or the What Works Clearinghouse and this is now I think a very useful source of information that it's not perfect yet and it's going to be continuing to evolve but it is certainly an improvement over what we had and establishes some standards so just to set the stage I want to show you the results of a study that we did under the sort of bridging the Campbell and What Works Clearinghouse where we did a full scan of literature over 20 years which is sort of what's typical for more than 20 years you have to think hard about whether the context is still right and for some things 20 years may be too long and what all of these models have in common is that they use consistent standards of evidence that are specific to study design they review and code the studies so that everything is sort of easily retrievable if you ever tried to read a lot of journals you get the stuff every which where footnotes tables or missions and then we have products that include like the original database which until recently was not public, intervention reports practice guides and technical reports and tools so there's a lot of stuff up there that will help both help feed information make it readily accessible and help those of us who are producing information do a better job of designing our studies and reporting our studies and sharing our information so in this study that we did of college access program we did this sort of as I said in parallel we looked at literature since 1990 we used the what works evidence standards we coded the data in a similar way to what works and here's just an interesting fact so we went to the library we got 2044 hits this is studies that sort of looked like they would be impact studies the title and the and or the abstract would lead you to believe they're going to give you some estimate of the effectiveness of a study some of them we screened out because they just weren't once we looked a little closer they weren't that 200 past face relevance after we sort of read it we sort of said yeah this is what it is go down here in the little corner down here 13 studies were what I mean we didn't go for gold standard studies these were studies that we could say you know probably have some causal validity so we should pay attention to the results that's an incredible bunch of studies in time and energy that have been put into doing research and publishing research because this is published up I didn't pull it out of people's drawers that I didn't answer the question even remotely or I would not want my policy maker acting on this I would not want my children's education to be based on this research so what we got not surprisingly is these are the centers of these little boxes are the point estimates of the impacts and the little bars on the side give you a 95% confidence interval around that so for those of you who are comfortable with confidence intervals so this just shows you that there's a range of impact estimates up and down down here this is the big federal program no effect and now we won't let it be evaluated again that's a big always a big political fight over whether or not that program can be evaluated even though it serves less than 7% of the eligible students okay so it's lots of room out there to know whether this is a good thing okay so what we've done with this is we now have a shared evidence platform called the what works clearing house then this is contracted out the IES contracts this out there are now two nodes to this there used to be one but all the data from your perspective all the data in one place it has sort of three three parts to it it has the individual studies of intervention effectiveness so this is the source this is what the researchers produce then the clearing house takes it sorts it sifts it finds the evidence that is credible by some standards that I think you know we have stood the test of time they've been out there now for about eight years and they've had a lot of fighting over them over the years but I think they've settled down people are not you know we all have our own judgment about whether they're too tough but they sort of seem to be striking a balance and so we have these studies get reviewed according to a consistent protocol and put into a nice neat format and there's a library of these studies and one of the things I feel the best about at my time in Washington is these studies are now available to you you have to request them but you can take these coded studies out and reuse them if you are doing a literature review and then there are a series of reports that summarize the results of the evidence there are single study reports that's another thing that we instituted we used to when we just reviewed one study we used to just put it in the file drawer and we'll give you a little blurb on your computer that tells you what what we found but now we actually publish the results of those single study reviews we tell you whether the evidence is credible or not how highly credible it is we tell you what was found in that and give you the descriptive background on it we also have in their intervention reports which summarize all the evidence on a given intervention so something like you know cognitive tutor or read 180 you can go in and you can find all the evidence that the clearing house found up to a certain point sorted by whether or not it's credible how if it's credible how credible it is and then the findings summarized and then we've turned some of that into practice guides so that the practice guides are intended to be a bridge between the the evidence on what it does and doesn't work and what one might do if you are actually in the classroom or in the school building having to make decisions on a daily basis so one of the most useful parts of the practice guides I think is telling you where there is no evidence to support a particular policy but practitioners think this is the best way to go but adopt this policy or practice knowing that we have not demonstrated this with evidence and then there are a lot of tools in there one of the one of the most recent things that was added to the what works site and this is the turning point between not being useful and being useful was something called the find what works tool we've got all these we're supposed to be doing this to support policymaking and I'm getting calls from the help committee and they just want to know what the most effective education policies are rank them 1 to 50 so we finally got them to understand that we needed to be able to queer this database in a little more nuanced way but we've done it in a way that I think is very supportive of policy so now what you can do you can go in and you can create summaries of the evidence of all the studies that we have that have been reviewed and summarized you can filter it by topic so do you want to know about math do you want to know about reading do you want to know about a particular outcome like dropping out of school or going to college or achieving proficiency or being well behaved in the classroom you can sort it by grade so you may be focused on preschoolers or you may be focused on high schoolers so there's a large part of this database that is not relevant to you depending upon what your perspective is you can sort by findings you could say I want all the stuff on populations that have positive effects or negative effects you may be looking for the negative effects and you can also look for it by extent of evidence you say I only want to look at interventions that have been tested to have a large body of evidence and that have been shown to improve either math or reading so you can put in more complex scenarios here and this will also give you when you do this you probably can't see this but in these circles here there are a lot of different population groups it will also tell me over in the far side how many studies there are in that bucket so you get a little snippet of that and then you can pull up and get a quick summary of what the evidence shows so that's what's going on over here so we've got this is the results for general academic achievement and it will just you can sort it by the size of the effects so you can sort it by the extent of evidence and if this were live we could just scroll down and see all the studies all the interventions so what is in here is within each of these headings this one's lessens in character that is a branded intervention so it's summarized all the studies of lessens in character and measure which is general academic achievement and it will tell you that in this case there's a positive effect but there's only a small body of evidence so we don't have very many studies but it is positive so the last thing I want to talk about is building a culture of evidence guided policy and practice because this is really where my passion is I just think we are spending too much and the stakes are too high to not know so here's just my favorite example I promise you an example West Virginia schools acted without evidence here so if you could read this and you can't read this but it says use of math software in West Virginia schools doesn't add up then it says the US Department of Education finds no discernible effects in raising test scores state reviews are mixed and what if this is the state of Virginia went and bought a computer based math curriculum for all the high schools in the state and they put it in place and they trained the teachers I don't know it may work in Virginia, West Virginia but in the studies that have been conducted by talented researchers published and passed the what works clearing house standards for causal inference evidence the bottom line is there's no evidence that this program works so what could it seems like the first thing you could have done is they could have gone to the clearing house now in fairness to them we didn't have the what works tool up and running until so it would have been hard for them to do what we can do now 18 months ago maybe even less than that but now they could certainly do that they could have contacted, we also have comprehensive centers the Department of Education has comprehensive centers which are also charged with providing support to state and local education agencies or regional education lab which are intended to provide this kind of support could have contacted one of you folks and sort of asked you what you knew what the evidence was even if West Virginia found the evidence and they said no effects they could have decided that they're different and this is a good fit for their schools but they wouldn't really know that they wouldn't really know that so what would have been a smarter thing to do maybe try it out they could have tried it out in some schools or with some classes and they could have gone to their local university or gone to the rail and gotten somebody to help them learn from their experience my guess is they did not such a good job training all the teachers in how to use this, doing it all at once they might have done a better job if they'd done a quarter of them or 10% of them and if they did one more thing they selected the schools that are going to be the early adopters with a flip of a coin they could have learned something really important they could have learned something about they could have learned several things important they could have learned number one whether or not this is actually improving math achievement for their students before they go and buy lots more Carnegie it's the Carnegie Cognitive Tutor they could have they could have also gotten some purchase on how to do the training efficiently they might have been able to improve the training most of us the first time we go out and do some kind of training go home and so to say I think I could have done that better let me incorporate all those questions that I was asked so it just feels like this is a case where a couple of steps here they could have saved a lot of money I think they spent $20 million on that okay okay so building a culture that encourages creation and use of evidence so using evidence and decision making I would say number one look for reliable sources search for evidence and use phased implementation when you when you can't it just it stuns me that we can't get Congress phased implementation we did this actually in welfare reform we did some of the what we did is when we did the big welfare reforms of when I was at 96 we allowed a lot of waivers so states could come in and say I've got a better idea so if you let me do this the quid pro quo was the feds let them change the rules but they had to do an experimental evaluation of it so they couldn't do it with everybody you know relax the mandatory work requirements for example but the quid pro quo was you can do that but you have to do an experiment and if it isn't effective if what you're doing is not better than our new policy, you have to stop doing it so we could do more of that in education it would seem so smart to do that and we could do strategic pilot testing we could just roll out programs I don't know why the state of Maine gave every seventh grader a notebook computer all at once same year you know didn't think about things like what the kids were going to do with those internet connected laptop computers, seventh graders right? they didn't think about what was going to happen in the households when there's fighting between mom and dad about getting what's going to happen when the computer gets dropped how are the teachers going to use these computers every student in the state of Maine all at once what if they had phased that in their investment would have been made over a number of years they would have learned some things about how to implement it, how to bring parents into the process well go figure there are some challenges in doing local evaluations and not having big national firms I mean nobody can afford to do these multi-million dollar evaluations at IES funds but state governments local school districts could do a whole lot more you don't have to have a study with 3,000 observations for it to be useful because if you did a study in one school with 50 observations and it was well designed and you did it in another school and another school and you could put those together right? in the state of Michigan you have all the same tests for all the kids data is cheap right? you got the outcome measures coming in already all you have to do is sort of roll these things out in a way that you can get reliable comparison of treated and untreated kids okay I'm going to just these are some things that are a little bit more I'm going to skip those I think I just want to end with a couple of examples of high value, low cost evaluations and I think this is where the future is I think we can be doing a lot more evaluations that are more narrowly focused you're not going to know everything about everybody but Pell Grant experimental initiatives here's where we're changing the rules on Pell Grants well what if we didn't change the rules for everybody or we changed the rules for everybody but we allow some backing out of that for people who have better ideas the rule is we do this randomly increase the amount increase the duration, forgiveness policies whatever it is we're just starting to design or when I left we were just starting to design some experimental initiatives that would be around allowing some flexibility in the rules changes but the quid pro quo is you got to study it the FAFSA completion study that Bridgetary Long and Bettinger did randomly assigning families to get H&R block to pre-populate the FAFSA form for eligible students that was very low cost because all they had to do was plug into the national database to get the outcome measures positive parenting program this is where they randomly assigned counties and then you can plug into administrative data to get outcome measures so you don't get all the details of what went on and how it was implemented but sometimes you can add that on especially if you can engage graduate students or regional labs or get a foundation to add the icing New York City teacher incentive program where you randomly assign schools to either get these incentives or not get these incentives all we had to do was watch whether the teachers move more whether the students do better all the data where they are to monitor the outcomes so it is possible to do this but it would be really smart of us to just get on the train here and have more partnerships with schools and the higher eds and make better use of this administrative data so let me stop there and give us time for questions if anybody has any questions yes so I like this idea about creating a culture of having a great culture of having a great culture but you mentioned what evidence-based culture could do but you didn't offer any proposals for having a school about creating evidence-based culture in an agency let's say I'm the new governor of West Virginia and I've seen your talk and I'm outraged that we wasted all this money on this math program and I want to create an evidence-based culture of my State Department of Education what would you tell that person to do I would I wouldn't be in your office a lot and I'd want to encourage you to sort of introduce me to all when you've got these new problems or initiatives I think we ought to be both expecting and to some extent incenting people one thing actually key to having an evidence-based culture is to make it safe for people to participate in evaluations I think everybody who goes out and does something that they think is going to be like buying that software somebody thought that was a good idea so if we did the if we had not bought it every place and we had done the study and we found out that was right it really wasn't a good idea I don't want those people who thought it was a good idea for they had good reasons I don't want them to feel like you know I think they're stupid or incompetent or irresponsible I want them to what would disappoint what I want to do is I want to encourage them to take that knowledge as just that knowledge and then use that in the next decision that they make you know to to I mean don't hunker down and say by gosh I want to do this so I think we have to make it there has to be a give back and the give back is you know you don't have to do this all at once be you'll learn something and you know I'm still on your team you know I still want to help and I think too often we think if I created the program you know if I'm the one who came up with this abstinence only program I'd better be right how am I going to feel when it's wrong I'm going to it's got to be okay to be you know I thought it was a good idea respect me for that and and when you when I want to start pushing back is if somebody you know refuses to accept the evidence as real evidence and learn from it I I like to put the I think we're putting the child at the center and sort of saying this is not about you or me or our careers this is about we're all trying to get the best for the kid so let's try to learn and if it works build on it if it doesn't work it was a good idea somebody thought it was a good idea somebody funded it so you know it's more than one person right so let's not get in a fight about that let's sort of move on figure out if not that then what so I don't know if that answers your question but it's hard I think it's really hard but I think researchers have a part I think we have a part to play in this we have been arrogant we have been inattentive to what the you know the the policy makers and the program people need you know it was like I mean how many researchers do you know who don't publish the results if they don't come out the way they wanted they keep redoing their model or put it in the file drawer I mean this is a group that is not a random sample of researchers in this room so I think we are under representing that group but the publication bias is pretty large out there there are journals that won't publish papers if they're not significant findings in the right direction I can tell you if I went to family planning perspectives and tried to publish an article that said abstinence education worked which they wouldn't even look at how I did the study that would be that would be a no-go I guess that would be that they have this evidence-based culture thing happening at the national level in some countries that don't it's a small end problem so it's easy to sort of speculate and say well you know they do have this in common and they don't have that in common but you could use state level variation yep across states you could use department level variation within a state maybe under the Bush 2 administration arguably education department was really enthusiastic and very serious by evidence department of labor by the changes so I think that's a good a good point we would go to policy makers with these briefs and say here's a great practice guide what I really want to know about is question acts the question acts wouldn't be represented in the law enforcement clearing house so I'm wondering if you can talk a little bit about the two-way street of dissemination in order to frame the conversation as well as responding to the needs of the questions that are really relevant to or pertinent to practice furniture yeah that's a really great question and it is actually one of the roles that the labs are supposed to be facilitating that making sure one of their charges is to do needs sensing so that they know what information is front and center at the local level I think part of this is we as researchers have to be more engaged with our clients so when I take on a new project now and I learned this halfway through I didn't or maybe three-quarters of the way through where my career I didn't do this at the outset but it's always a partnership I mean I want to I feel like I have to sit down with you and have a conversation about what you need to know what makes this worthwhile for you as well as I'm going to have a slightly bigger I care about that but I also care about the larger landscape and what would be useful to places beyond my local my local sites I also think there's an education that has to go on with the practitioners to accept that sometimes the answer is we really don't know so your judgment is as good as any evidence I can give you the research hasn't been done and that's I think it's better to tell you that if that's the true than it is to give you some evidence that isn't reliable a lot of evidence is not reliable you know people put stuff out there and it's promotional you know just poorly designed not good evidence and I think having that honesty about what we do know what we don't know and then sort of coming back and sort of saying well I know this is what you want to know but I've done a scan there is no evidence one way or the other here's what we can tell you we might want to think about building some in as you that would be how I would approach that I bring the perspective of school board researcher this is very interesting and I think very helpful to try it out wondering if there is another component to it I have a model on my mind so I'll say the model the cooperative extension there's one of the really astounding institutions that changed the entire sector in combination with the research enterprise and one of the key things in that was that the extension people along with scientists got local farmers to try things and I as a school board member to that amount I trust going to remember school or district and seeing that word talk with people more than I do a paper even if it's on some clearing house website that says this is terrific 1800 people say it's terrific I still need it to work because I have that skepticism that you concluded your last answer and I think there's something valid going next door and seeing if we really didn't get the rocks and everybody else's we didn't so maybe I should change more I think that's a great reference I think the extension service in an ideal world the labs would be they would be like the regional offices of the extension and we folks your local universities and would be more like the agents who are going to be working with the schools to help them when they try these new things out actually do it in a way that they can sense whether or not the rust is really there sometimes we don't see the outcomes that we see maybe the demographics of the schools are changing the performance is going up or down depending upon which way the currents are going so it does help so your analogy of the agents who are actually working with the farmers to do the studies the experiments if you will that is exactly what I think would be so helpful and it would build these conversations we researchers are much better researchers when we are engaged with the practitioners when they are on our team it's not me coming in and studying your school it is me coming in sitting down with you and we together are going to address the issues that you have and I have to do it in a way that is sensitive to what's going on in the school and that will give us evidence that you can believe so I've got some trust building and we as researchers have trust building to do and a lot to learn because the textbooks take us so far they are part of the training but they are not all of the training that's a good sometimes you'll know that and sometimes you won't so I think that's a useful comment and yeah I think that's a very useful yeah so I think yes West Virginia I particularly appreciate you I don't know if that decision will be after I left I would like to share my time there that I struggle that we would face in trying to bring a regular conversation that's a deliberate process and take a good amount of time in a state policy environment it can often be very time and so if a governor decides to take an initiative and wants an answer for them in six weeks and we're going to push ahead with this new policy initiative we really have to know well we'll get back to 18 months that is a way to address the needs so be interested in your insights on navigating the needs for a deliberate process and having good solid evidence for an intervention with the often time bound needs of policy makers I think that's one of the reasons that I think it's so important to have some of this stuff pre sifted and sorted so you know these all of these if you go to the clearing house the reports are different ages some of them will be back to you know 2006 some of them will be 2012 so there may still be a little bit to do but I think the more if we get all of us putting our evidence in there okay and we start contributing to that then it's really quick for you to go and respond number one what do we know already that doesn't take you a long time and if we don't know what is needed I think it's okay to say we don't know and I think that's better than sort of saying you know here's some not so good research that maybe as wrong as it is right I think that's the part that there is a lot of stuff out there masquerading as science and it is not really scientific you know those people in this room would not look at this and want their children's education being guided by that evidence so I think there's a I think we have to make it okay for people to do innovative things the world has to go on we can't do nothing and be sanguine about things but I would just encourage them to try to if you're going to take a bold step and do something be open to learning from doing that and it doesn't you know especially when you can do it at low cost with the new data systems and the standardizing of tests and so on you can get those top level outcomes really at very low cost in a very timely way if you build these partnerships draw on the rails draw on the talents in this room you know it's and there are you know people all over the country now 10 years ago there were not people all over the country who understood these issues and who could help a local education organization do a low cost study they just they just weren't they were concentrated in a few firms and a few large universities we now have lots of folks like you who have had some of this experience in a regional lab or in a university or at a research firm who and we're training people through things like the IES pre-doc program and this seminar series at Iran so I think there's a lot more capacity out there now to do that I just don't think we should mix up good evidence with not good evidence I think it's okay to say we don't have it yes I want to pick up on your earlier question in response to a number of questions you've been kind of playing out a theory based upon your experience in collaborating with a variety of policy makers and I'm wondering how you how we might go about learning from that theory that you're playing out based on your own practice in an evidence based way how do you learn about the ways in which researchers productively collaborate with policy makers system in different different contexts how do we learn about that well there is an initiative that is jointly sponsored by WT Grant Foundation and Spencer going on now which is on use of evidence and there are I don't know five or six teams somebody who may know better than I do I was involved in it when it was first starting and I was doing work with WT Grant I've seen some of the early results of that I think it's a you know it's it's hard work I think it's going to take us a while but it's picking up on Jeff's point too I think we need to we need to learn more about that I can tell you that it is getting a lot lot easier for people to to engage with schools, school personnel policy makers and embark on these kinds of knowledge development knowledge use enterprises so it's you know the number of people here at Michigan who have been out doing research with the schools says that you know it is getting out there the art and the strategies I think part of it is experience on both sides so I know a lot of times it helps one of my first things I do is I sort of make sure that I connect with the policy maker the practitioner that I'm communicating with on doing something I want them to know other people who have worked with me in similar capacities and how we have worked so sharing those kinds of experiences and relating other I mean there's probably not many unique situations somebody could you know our objections to the evidence generation or process that somebody could throw at me that I couldn't relate to some other encounter and have some strategy that we had made work I'm asking that very practice that you're describing right now you're theorizing about relationships well that's a good question I don't know you'd have to have more than the end of me to do that but I think there are enough of us out there doing this now that I think you could somebody could probably do that they could probably create a sample of us and how we are approaching it and when we're being more successful and less successful but I don't know it's a good dissertation topic for somebody do you have any suggestions or you just raise it as a question I think it's a good question and bottling it to sort of share with people too I think is another thing that is important I think the IES pre-doc program has been one way that people have been getting the art if you will the art of this science out there because we do a lot of communication and sharing of experiences sitting on technical work groups so we learn what other people what problems they encounter and how they get around it but you rightly identified we aren't formally studying it in that way we're not studying the negotiating there are these efforts as I said to study the use of evidence and what distinguishes when it is being used and for what purposes and how do we facilitate that process yes right in the south with your colleague and one just to answer some of your questions they are doing some research partnerships and studies right now so they just launched that but the other the other thing I was going to say is that when researchers come to these state education agencies or when they're trying to get ideas about how to succeed one of the questions they ask themselves is is the context like mine similar to what Glenn was saying the school board member of there you know I want to see it in action is this in a setting that looks like how do I translate these research findings into my context so there's a lot of so the researchers that are sitting at the table or at the table when they're discussing these issues are have more access to applying research at the moment of the when the questions are arising at the table and they're trying to do that translation process so one question I had is whether you've thought about those issues as a context because that's really the way on there on their thinking process that's always a key part of wanting a well designed and executed study would certainly have a good bit of contextual information as part of it the Clearinghouse does describe the context so if you find something that looks interesting to you or something that's counter you wanted to do it but it says no work you can actually read in the reports that they have where they've summarized what the context is so this was you know a large urban poor school district or this was a suburban school district or this was a place that had a highly paid teacher so you can get a lot of that context there it's not perfect but that's part of how deep and how broad is the evidence you're drawing on and again knowing whether you have it or don't have it so I would say it would be foolish to say go to the Clearinghouse if it's thumbs up or thumbs down you should use that as a piece of evidence and it has been through some screen and you know in some areas you know more and other areas less about the context in which the research was done you know we have some areas of research where there have been large multi-site evaluations so the Upward Bound study the Federal Evaluation of Upward Bound and Job Corps were both nationally representative samples okay so again there's some heterogeneity in the results so maybe you are at the extreme of the overall results are pretty modest to know maybe you're different and maybe this would work for you and maybe you look like that outlier that had the positive effects you know there's no study is going to give you all of that but I think if you you know go into the reports and scan them I think you will find some of that there and you won't be left you know just out at sea without an ore I mean I think it depends you know there are policy makers I mean there's a teacher is a policy maker in the classroom and a principal is a policy maker at another level and so you know there are all these layers and I think sometimes people you know if you're making the decision in West Virginia they were making the decision for the state so that's a pretty broad policy context you might have you know there might be different decisions for the high performing school and the suburbs versus the inner city it depends I think where is the decision making occurring and then the context maybe broader or narrower depending upon that there's more of this type of investigation going on one of the constraints has been in education the question of selection or the ethics of who's going to participate in certain research the unit of analysis is not usually well it's a classroom possibly and if this classroom is randomly assigned and this one is in control these parents aren't too excited about it it seems that that's a constraint on participation and then a constraint on you also expressed a concern about phase implementation that if a district is going to do something you think enough of it to do it you better do it for everyone do you find that diminishing at all is that becoming less of a constraint or is that a concern I think it's always a concern you know you're always balancing if you're a policymaker or you know heading the school those are always concerns I think there's a we're gaining some appreciation of the fact that you're experimenting when you when you change from this to that and if the more we can sort of see that often times changing from this to that is not getting us where we want to go and sometimes we break the system so one of my favorite examples of breaking the system is California did two things at once they glommed on to the Tennessee Star experiment findings of smaller class size and then they went to some research on bilingual education and they sort of did these two things at once and they totally broke their system because they didn't have enough bilingual teachers to staff the smaller class sizes you know who was served well by that as opposed to sort of saying you know this is a you know a five year proposition let's do bilingual in half the schools and smaller class size in half the schools and then we will gradually you know get enough teachers to round this out and let's learn because maybe just maybe the Tennessee Star experiments in one place don't generalize to California maybe they don't it was one study one time one place instead California just something that gave them a horrible headache and so I do think that there is getting to be you know there is still that knee jerk reaction of you know you're going to give those kids the computers the notebooks but you're not going to give my kids the notebooks but you know there are always choices being made and so yeah I think it's I think it's lessening I think the resistance is lessening but you know it's still going to be there yes um-hmm um-hmm um um um um um um um um um um um um um um um um um um um um So that's a really good question, how do we know where the big gaps are so that we can work in those fields. One thing that Clearinghouse didn't do in the first five years and has now does routinely, which is if they search for evidence on a topic and come up dry, they publish a NEL review and they will tell you that they've looked for this and there is no evidence. You can also go in and easily sort of see all the places where you can search on interventions or strategies or population groups where there is minimal evidence and that will sort of be a key to sort of a whole there. It's a little more nuanced when you get to things like context and stuff like that because that's a little more uneven. One of the things that the Clearinghouse has done and I think the profession is sort of doing on its own too is there's an emerging sort of I think a standard of design and reporting on studies. So one of the things that was published fairly recently is reporting guidelines for causal inference studies and one of the things is we would want you to tell us certain things about the context of the study. And so by saying that up front that triggers maybe I wouldn't have thought about English language learners as being sort of something I should be noting because I am so immersed in English language learners that I just sort of think this is who I study and I don't but making that front and center sort of saying yes I should tell you whether or not I'm dealing with these population groups or is it urban or rural or older or younger all those kinds of things. So it's not perfect I think you've you know it's an area to be worked on but I think there is I think you will find some evidence in here to support that and I also think you know networks like this you know just the you know your colleagues in this room will have just a wealth of information that would be helpful networking is really just a good thing to do find out what others know and pick their brains. Yes. So perhaps you touch on it so don't answer it if that's the case. Jeff was talking about how to create culture within policy makers like make them more like dead driven and basically use evidence to support their implementation but it seems to me that we're like focusing a lot on the policy maker and not that much on the researcher in the sense of often times well it's really hard for a researcher like it's not that fancy to replicate a study it's not that fancy to replicate the same idea in a different context and even if you have one single experiment that gives you strong evidence that something is effective you will probably want to test it. Even the same experiment in the same setting I mean like this as a robust tech it will be nice to have more evidence and I guess I guess in academia there's I mean like as you were saying like you're no you're not going to publish there are many journals that won't publish an idea that where you have significant effects right. So how have you talked about how to create culture even within researchers of a like there might be I mean like there's there's some there's some definitely like some positive results of how to describe the idea I guess I know that nobody's gonna publish the thesis but I mean like someone needs to do this is this is a really important I'm glad you raised this I didn't even think to to to mention it but you know sitting here in this grand university there is something called tenure and it works against as you say I mean it works it works against some of what you know I is important for creating an evident evidence-based culture and feeding it with good evidence because we don't that we tend not to value null findings negative findings or replications to the level that we do. So I hope and believe that some of what is going on in things like the this project on use of evidence that WG Grant has been sponsoring the the IES pre-doc and post-doc programs that are sort of actually rewarding or providing incentives for universities to place more value on this applied research I hope that that will start to change I think in in in some institutions it is changing the how people think about what it takes to get to get tenure and to reward people and to publish things there are some journals now that are much more willing and value good research regardless of results which is what it what it should be I mean in in the in the hard sciences in the in the agricultural you know extension service they they would let you know if the fertilizer burned the crop you know I mean that would be you know there would be no question about that but in education somehow if it didn't if we don't get that positive in those stars it's you know we did what was I what was I doing right so I you know I I don't know how to accelerate this but I do hope that it is changing and and it does need to